<!DOCTYPE html>
kube-apiserver Encryption Configuration (v1) | Kubernetes
Documentation
Kubernetes Blog
Training
Partners
Community
Case Studies
Versions
Release Information
v1.26
v1.25
v1.24
v1.23
v1.22
English
中文 (Chinese)
Home
Available Documentation Versions
Getting started
Learning environment
Production environment
Container Runtimes
Installing Kubernetes with deployment tools
Bootstrapping clusters with kubeadm
Installing kubeadm
Troubleshooting kubeadm
Creating a cluster with kubeadm
Customizing components with the kubeadm API
Options for Highly Available Topology
Creating Highly Available Clusters with kubeadm
Set up a High Availability etcd Cluster with kubeadm
Configuring each kubelet in your cluster using kubeadm
Dual-stack support with kubeadm
Installing Kubernetes with kOps
Installing Kubernetes with Kubespray
Turnkey Cloud Solutions
Best practices
Considerations for large clusters
Running in multiple zones
Validate node setup
Enforcing Pod Security Standards
PKI certificates and requirements
Concepts
Overview
Kubernetes Components
The Kubernetes API
Working with Kubernetes Objects
Understanding Kubernetes Objects
Kubernetes Object Management
Object Names and IDs
Labels and Selectors
Namespaces
Annotations
Field Selectors
Finalizers
Owners and Dependents
Recommended Labels
Cluster Architecture
Nodes
Communication between Nodes and the Control Plane
Controllers
Leases
Cloud Controller Manager
About cgroup v2
Container Runtime Interface (CRI)
Garbage Collection
Containers
Images
Container Environment
Runtime Class
Container Lifecycle Hooks
Windows in Kubernetes
Windows containers in Kubernetes
Guide for scheduling Windows containers in Kubernetes
Workloads
Pods
Pod Lifecycle
Init Containers
Disruptions
Ephemeral Containers
User Namespaces
Downward API
Workload Resources
Deployments
ReplicaSet
StatefulSets
DaemonSet
Jobs
Automatic Clean-up for Finished Jobs
CronJob
ReplicationController
Services, Load Balancing, and Networking
Service
Ingress
Ingress Controllers
EndpointSlices
Network Policies
DNS for Services and Pods
IPv4/IPv6 dual-stack
Topology Aware Hints
Networking on Windows
Service ClusterIP allocation
Service Internal Traffic Policy
Topology-aware traffic routing with topology keys
Storage
Volumes
Persistent Volumes
Projected Volumes
Ephemeral Volumes
Storage Classes
Dynamic Volume Provisioning
Volume Snapshots
Volume Snapshot Classes
CSI Volume Cloning
Storage Capacity
Node-specific Volume Limits
Volume Health Monitoring
Windows Storage
Configuration
Configuration Best Practices
ConfigMaps
Secrets
Resource Management for Pods and Containers
Organizing Cluster Access Using kubeconfig Files
Resource Management for Windows nodes
Security
Overview of Cloud Native Security
Pod Security Standards
Pod Security Admission
Pod Security Policies
Security For Windows Nodes
Controlling Access to the Kubernetes API
Role Based Access Control Good Practices
Good practices for Kubernetes Secrets
Multi-tenancy
Kubernetes API Server Bypass Risks
Security Checklist
Policies
Limit Ranges
Resource Quotas
Process ID Limits And Reservations
Node Resource Managers
Scheduling, Preemption and Eviction
Kubernetes Scheduler
Assigning Pods to Nodes
Pod Overhead
Pod Scheduling Readiness
Pod Topology Spread Constraints
Taints and Tolerations
Scheduling Framework
Dynamic Resource Allocation
Scheduler Performance Tuning
Resource Bin Packing
Pod Priority and Preemption
Node-pressure Eviction
API-initiated Eviction
Cluster Administration
Certificates
Managing Resources
Cluster Networking
Logging Architecture
Metrics For Kubernetes System Components
System Logs
Traces For Kubernetes System Components
Proxies in Kubernetes
API Priority and Fairness
Installing Addons
Extending Kubernetes
Compute, Storage, and Networking Extensions
Network Plugins
Device Plugins
Extending the Kubernetes API
Custom Resources
Kubernetes API Aggregation Layer
Operator pattern
Tasks
Install Tools
Install and 

Set Up kubectl on Linux
Install and Set Up kubectl on macOS
Install and Set Up kubectl on Windows
Administer a Cluster
Administration with kubeadm
Certificate Management with kubeadm
Configuring a cgroup driver
Reconfiguring a kubeadm cluster
Upgrading kubeadm clusters
Upgrading Windows nodes
Migrating from dockershim
Changing the Container Runtime on a Node from Docker Engine to containerd
Migrate Docker Engine nodes from dockershim to cri-dockerd
Find Out What Container Runtime is Used on a Node
Troubleshooting CNI plugin-related errors
Check whether dockershim removal affects you
Migrating telemetry and security agents from dockershim
Generate Certificates Manually
Manage Memory, CPU, and API Resources
Configure Default Memory Requests and Limits for a Namespace
Configure Default CPU Requests and Limits for a Namespace
Configure Minimum and Maximum Memory Constraints for a Namespace
Configure Minimum and Maximum CPU Constraints for a Namespace
Configure Memory and CPU Quotas for a Namespace
Configure a Pod Quota for a Namespace
Install a Network Policy Provider
Use Antrea for NetworkPolicy
Use Calico for NetworkPolicy
Use Cilium for NetworkPolicy
Use Kube-router for NetworkPolicy
Romana for NetworkPolicy
Weave Net for NetworkPolicy
Access Clusters Using the Kubernetes API
Advertise Extended Resources for a Node
Autoscale the DNS Service in a Cluster
Change the default StorageClass
Change the Reclaim Policy of a PersistentVolume
Cloud Controller Manager Administration
Configure a kubelet image credential provider
Configure Quotas for API Objects
Control CPU Management Policies on the Node
Control Topology Management Policies on a node
Customizing DNS Service
Debugging DNS Resolution
Declare Network Policy
Developing Cloud Controller Manager
Enable Or Disable A Kubernetes API
Encrypting Secret Data at Rest
Guaranteed Scheduling For Critical Add-On Pods
IP Masquerade Agent User Guide
Limit Storage Consumption
Migrate Replicated Control Plane To Use Cloud Controller Manager
Namespaces Walkthrough
Operating etcd clusters for Kubernetes
Reconfigure a Node's Kubelet in a Live Cluster
Reserve Compute Resources for System Daemons
Running Kubernetes Node Components as a Non-root User
Safely Drain a Node
Securing a Cluster
Set Kubelet parameters via a config file
Share a Cluster with Namespaces
Upgrade A Cluster
Use Cascading Deletion in a Cluster
Using a KMS provider for data encryption
Using CoreDNS for Service Discovery
Using NodeLocal DNSCache in Kubernetes Clusters
Using sysctls in a Kubernetes Cluster
Utilizing the NUMA-aware Memory Manager
Verify Signed Kubernetes Artifacts
Configure Pods and Containers
Assign Memory Resources to Containers and Pods
Assign CPU Resources to Containers and Pods
Configure GMSA for Windows Pods and containers
Configure RunAsUserName for Windows pods and containers
Create a Windows HostProcess Pod
Configure Quality of Service for Pods
Assign Extended Resources to a Container
Configure a Pod to Use a Volume for Storage
Configure a Pod to Use a PersistentVolume for Storage
Configure a Pod to Use a Projected Volume for Storage
Configure a Security Context for a Pod or Container
Configure Service Accounts for Pods
Pull an Image from a Private Registry
Configure Liveness, Readiness and Startup Probes
Assign Pods to Nodes
Assign Pods to Nodes using Node Affinity
Configure Pod Initialization
Attach Handlers to Container Lifecycle Events
Configure a Pod to Use a ConfigMap
Share Process Namespace between Containers in a Pod
Use a User Namespace With a Pod
Create static Pods
Translate a Docker Compose File to Kubernetes Resources
Enforce Pod Security Standards by Configuring the Built-in Admission Controller
Enforce Pod Security Standards with Namespace Labels
Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller
Monitoring, Logging, and Debugging
Troubleshooting Applications
Debug Pods
Debug Services
Debug a StatefulSet
Determine the Reason for Pod Failure
Debug Init Containers
Debu

g Running Pods
Get a Shell to a Running Container
Troubleshooting Clusters
Resource metrics pipeline
Tools for Monitoring Resources
Monitor Node Health
Debugging Kubernetes nodes with crictl
Auditing
Debugging Kubernetes Nodes With Kubectl
Developing and debugging services locally using telepresence
Windows debugging tips
Manage Kubernetes Objects
Declarative Management of Kubernetes Objects Using Configuration Files
Declarative Management of Kubernetes Objects Using Kustomize
Managing Kubernetes Objects Using Imperative Commands
Imperative Management of Kubernetes Objects Using Configuration Files
Update API Objects in Place Using kubectl patch
Managing Secrets
Managing Secrets using kubectl
Managing Secrets using Configuration File
Managing Secrets using Kustomize
Inject Data Into Applications
Define a Command and Arguments for a Container
Define Dependent Environment Variables
Define Environment Variables for a Container
Expose Pod Information to Containers Through Environment Variables
Expose Pod Information to Containers Through Files
Distribute Credentials Securely Using Secrets
Run Applications
Run a Stateless Application Using a Deployment
Run a Single-Instance Stateful Application
Run a Replicated Stateful Application
Scale a StatefulSet
Delete a StatefulSet
Force Delete StatefulSet Pods
Horizontal Pod Autoscaling
HorizontalPodAutoscaler Walkthrough
Specifying a Disruption Budget for your Application
Accessing the Kubernetes API from a Pod
Run Jobs
Running Automated Tasks with a CronJob
Coarse Parallel Processing Using a Work Queue
Fine Parallel Processing Using a Work Queue
Indexed Job for Parallel Processing with Static Work Assignment
Job with Pod-to-Pod Communication
Parallel Processing using Expansions
Handling retriable and non-retriable pod failures with Pod failure policy
Access Applications in a Cluster
Deploy and Access the Kubernetes Dashboard
Accessing Clusters
Configure Access to Multiple Clusters
Use Port Forwarding to Access Applications in a Cluster
Use a Service to Access an Application in a Cluster
Connect a Frontend to a Backend Using Services
Create an External Load Balancer
List All Container Images Running in a Cluster
Set up Ingress on Minikube with the NGINX Ingress Controller
Communicate Between Containers in the Same Pod Using a Shared Volume
Configure DNS for a Cluster
Access Services Running on Clusters
Extend Kubernetes
Configure the Aggregation Layer
Use Custom Resources
Extend the Kubernetes API with CustomResourceDefinitions
Versions in CustomResourceDefinitions
Set up an Extension API Server
Configure Multiple Schedulers
Use an HTTP Proxy to Access the Kubernetes API
Use a SOCKS5 Proxy to Access the Kubernetes API
Set up Konnectivity service
TLS
Configure Certificate Rotation for the Kubelet
Manage TLS Certificates in a Cluster
Manual Rotation of CA Certificates
Manage Cluster Daemons
Perform a Rolling Update on a DaemonSet
Perform a Rollback on a DaemonSet
Networking
Adding entries to Pod /etc/hosts with HostAliases
Validate IPv4/IPv6 dual-stack
Extend kubectl with plugins
Manage HugePages
Schedule GPUs
Tutorials
Hello Minikube
Learn Kubernetes Basics
Create a Cluster
Using Minikube to Create a Cluster
Interactive Tutorial - Creating a Cluster
Deploy an App
Using kubectl to Create a Deployment
Interactive Tutorial - Deploying an App
Explore Your App
Viewing Pods and Nodes
Interactive Tutorial - Exploring Your App
Expose Your App Publicly
Using a Service to Expose Your App
Interactive Tutorial - Exposing Your App
Scale Your App
Running Multiple Instances of Your App
Interactive Tutorial - Scaling Your App
Update Your App
Performing a Rolling Update
Interactive Tutorial - Updating Your App
Example: Configuring a Java Microservice
Externalizing config using MicroProfile, ConfigMaps and Secrets
Interactive Tutorial - Configuring a Java Microservice
Configuring Redis using a ConfigMap
Security
Apply Pod Security Standards at the Cluster Level
Apply Pod Security Standards at t

he Namespace Level
Restrict a Container's Access to Resources with AppArmor
Restrict a Container's Syscalls with seccomp
Stateless Applications
Exposing an External IP Address to Access an Application in a Cluster
Example: Deploying PHP Guestbook application with Redis
Stateful Applications
StatefulSet Basics
Example: Deploying WordPress and MySQL with Persistent Volumes
Example: Deploying Cassandra with a StatefulSet
Running ZooKeeper, A Distributed System Coordinator
Services
Connecting Applications with Services
Using Source IP
Reference
Glossary
API Overview
Kubernetes API Concepts
Server-Side Apply
Client Libraries
Kubernetes Deprecation Policy
Deprecated API Migration Guide
Kubernetes API health endpoints
API Access Control
Authenticating
Authenticating with Bootstrap Tokens
Certificate Signing Requests
Admission Controllers
Dynamic Admission Control
Managing Service Accounts
Authorization Overview
Using RBAC Authorization
Using ABAC Authorization
Using Node Authorization
Mapping PodSecurityPolicies to Pod Security Standards
Webhook Mode
Kubelet authentication/authorization
TLS bootstrapping
Validating Admission Policy
Well-Known Labels, Annotations and Taints
Audit Annotations
Kubernetes API
Pod
PodTemplate
Deployment
StatefulSet
ControllerRevision
DaemonSet
Job
CronJob
HorizontalPodAutoscaler
PriorityClass
PodScheduling v1alpha1
ResourceClaim v1alpha1
ResourceClaimTemplate v1alpha1
ResourceClass v1alpha1
Service Resources
Service
Endpoints
EndpointSlice
Ingress
IngressClass
Config and Storage Resources
ConfigMap
Secret
Volume
PersistentVolumeClaim
PersistentVolume
StorageClass
VolumeAttachment
CSIDriver
CSINode
CSIStorageCapacity
Authentication Resources
ServiceAccount
TokenRequest
TokenReview
CertificateSigningRequest
Authorization Resources
LocalSubjectAccessReview
SelfSubjectAccessReview
SelfSubjectRulesReview
SubjectAccessReview
SelfSubjectReview v1alpha1
ClusterRole
ClusterRoleBinding
Role
RoleBinding
Policy Resources
LimitRange
ResourceQuota
NetworkPolicy
PodDisruptionBudget
Extend Resources
CustomResourceDefinition
MutatingWebhookConfiguration
ValidatingWebhookConfiguration
ValidatingAdmissionPolicy v1alpha1
Cluster Resources
Node
Namespace
Event
APIService
Lease
RuntimeClass
FlowSchema v1beta3
PriorityLevelConfiguration v1beta3
Binding
ComponentStatus
ClusterCIDR v1alpha1
Common Definitions
DeleteOptions
LabelSelector
ListMeta
LocalObjectReference
NodeSelectorRequirement
ObjectFieldSelector
ObjectMeta
ObjectReference
Patch
Quantity
ResourceFieldSelector
Status
TypedLocalObjectReference
Other Resources
ValidatingAdmissionPolicyBindingList v1alpha1
Common Parameters
Instrumentation
Service Level Indicator Metrics
Node metrics data
Kubernetes Metrics Reference
Kubernetes Issues and Security
Kubernetes Issue Tracker
Kubernetes Security and Disclosure Information
Official CVE Feed
Node Reference Information
Kubelet Checkpoint API
Kubelet Device Manager API Versions
Articles on dockershim Removal and on Using CRI-compatible Runtimes
Networking Reference
Protocols for Services
Ports and Protocols
Virtual IPs and Service Proxies
Setup tools
Kubeadm
kubeadm init
kubeadm join
kubeadm upgrade
kubeadm config
kubeadm reset
kubeadm token
kubeadm version
kubeadm alpha
kubeadm certs
kubeadm init phase
kubeadm join phase
kubeadm kubeconfig
kubeadm reset phase
kubeadm upgrade phase
Implementation details
Command line tool (kubectl)
kubectl Cheat Sheet
kubectl Commands
kubectl
JSONPath Support
kubectl for Docker Users
kubectl Usage Conventions
Component tools
Feature Gates
Feature Gates (removed)
kubelet
kube-apiserver
kube-controller-manager
kube-proxy
kube-scheduler
Configuration APIs
Client Authentication (v1)
Client Authentication (v1beta1)
Event Rate Limit Configuration (v1alpha1)
Image Policy API (v1alpha1)
kube-apiserver Admission (v1)
kube-apiserver Audit Configuration (v1)
kube-apiserver Configuration (v1)
kube-apiserver Configuration (v1alpha1)

kube-apiserver Encryption Configuration (v1)
kube-proxy Configuration (v1alpha1)
kube-scheduler Configuration (v1)
kube-scheduler Configuration (v1beta2)
kube-scheduler Configuration (v1beta3)
kubeadm Configuration (v1beta2)
kubeadm Configuration (v1beta3)
kubeconfig (v1)
Kubelet Configuration (v1)
Kubelet Configuration (v1alpha1)
Kubelet Configuration (v1beta1)
Kubelet CredentialProvider (v1)
Kubelet CredentialProvider (v1alpha1)
Kubelet CredentialProvider (v1beta1)
WebhookAdmission Configuration (v1)
Scheduling
Scheduler Configuration
Scheduling Policies
Other Tools
Mapping from dockercli to crictl
Contribute
Suggesting content improvements
Contributing new content
Opening a pull request
Documenting for a release
Blogs and case studies
Reviewing changes
Reviewing pull requests
For approvers and reviewers
Localizing Kubernetes documentation
Participating in SIG Docs
Roles and responsibilities
PR wranglers
Documentation style overview
Content guide
Style guide
Diagram guide
Writing a new topic
Page content types
Content organization
Custom Hugo Shortcodes
Updating Reference Documentation
Quickstart
Contributing to the Upstream Kubernetes Code
Generating Reference Documentation for the Kubernetes API
Generating Reference Documentation for kubectl Commands
Generating Reference Pages for Kubernetes Components and Tools
Advanced contributing
Viewing Site Analytics
Docs smoke test page
Kubernetes Documentation
Reference
Package v1 is the v1 version of the API.
Resource Types
EncryptionConfiguration
EncryptionConfiguration stores the complete configuration for encryption providers.
Field
Description
apiVersion
string
apiserver.config.k8s.io/v1
kind
string
resources
[Required]
[]ResourceConfiguration
resources is a list containing resources, and their corresponding encryption providers.
AESConfiguration
Appears in:
ProviderConfiguration
AESConfiguration contains the API configuration for an AES transformer.
Field
keys
[]Key
keys is a list of keys to be used for creating the AES transformer.
Each key has to be 32 bytes long for AES-CBC and 16, 24 or 32 bytes for AES-GCM.
IdentityConfiguration
IdentityConfiguration is an empty struct to allow identity transformer in provider configuration.
KMSConfiguration
KMSConfiguration contains the name, cache size and path to configuration file for a KMS based envelope transformer.
Field
string
apiVersion of KeyManagementService
name
string
name is the name of the KMS plugin to be used.
cachesize
int32
cachesize is the maximum number of secrets which are cached in memory. The default value is 1000.
Set to a negative value to disable caching.
endpoint
string
endpoint is the gRPC server listening address, for example "unix:///var/run/kms-provider.sock".
timeout
meta/v1.Duration
timeout for gRPC calls to kms-plugin (ex. 5s). The default is 3 seconds.
Key
SecretboxConfiguration
Key contains name and secret of the provided key for a transformer.
Field
name
string
name is the name of the key to be used while storing data to disk.
secret
string
secret is the actual key, encoded in base64.
ResourceConfiguration
ProviderConfiguration stores the provided configuration for an encryption provider.
Field
aesgcm
aesgcm is the configuration for the AES-GCM transformer.
aescbc
aescbc is the configuration for the AES-CBC transformer.
secretbox
secretbox is the configuration for the Secretbox based transformer.
identity
identity is the (empty) configuration for the identity transformer.
kms
[Required

]
kms contains the name, cache size and path to configuration file for a KMS based envelope transformer.
ResourceConfiguration stores per resource configuration.
Field
resources
[]string
resources is a list of kubernetes resources which have to be encrypted.
providers
[]ProviderConfiguration
providers is a list of transformers to be used for reading and writing the resources to disk.
eg: aesgcm, aescbc, secretbox, identity.
SecretboxConfiguration contains the API configuration for an Secretbox transformer.
Field
keys
[]Key
keys is a list of keys to be used for creating the Secretbox transformer.
Each key has to be 32 bytes long.
This page is automatically generated.
If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.
Feedback
Was this page helpful?
Yes
No
Thanks for the feedback. If you have a specific, answerable question about how to use Kubernetes, ask it on
Stack Overflow
.
Open an issue in the GitHub repo if you want to
report a problem
or
suggest an improvement
.
Last modified August 24, 2022 at 10:22 AM PST:
Config API for v1.25 (57f5790d52)
Create an issue
(auto-generated page)
Print entire section
Key
Home
Blog
Training
Partners
Community
© 2023 The Kubernetes Authors | Documentation Distributed under
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our
Trademark Usage page
ICP license: 京ICP备17074266号-3

Get a Shell to a Running Container | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
한국어 (Korean)
日本語 (Japanese)
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

all and Set Up kubectl on Linux
Debug Pods
Debug Init Contain

ers
Debug Running Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standa

rds at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v1

alpha1)
Tasks
This page shows how to use
kubectl exec
to get a shell to a
running container.
Before you begin
You need to have a Kubernetes cluster, and the kubectl command-line tool must
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a
cluster, you can create one by using
minikube
or you can use one of these Kubernetes playgrounds:
Killercoda
Play with Kubernetes
Getting a shell to a container
In this exercise, you create a Pod that has one container. The container
runs the nginx image. Here is the configuration file for the Pod:
application/shell-demo.yaml
:
v1
kind
:
Pod
metadata
:
name
:
shell-demo
spec
:
volumes
:
-
name
:
shared-data
emptyDir
:
{}
containers
:
-
name
:
nginx
image
:
nginx
volumeMounts
:
-
name
:
mountPath
:
/usr/share/nginx/html
hostNetwork
:
true
dnsPolicy
:
Default
Create the Pod:
kubectl apply -f https://k8s.io/examples/application/shell-demo.yaml
Verify that the container is running:
kubectl get pod shell-demo
Get a shell to the running container:
kubectl
exec
--stdin --tty shell-demo -- /bin/bash
Note:
The double dash (
--
) separates the arguments you want to pass to the command from the kubectl arguments.
In your shell, list the root directory:
# Run this inside the container
ls /
In your shell, experiment with other commands. Here are
some examples:
# You can run these example commands inside the container
ls /
cat /proc/mounts
cat /proc/1/maps
apt-get update
apt-get install -y tcpdump
tcpdump
apt-get install -y lsof
lsof
apt-get install -y procps
ps aux
ps aux | grep nginx
Writing the root page for nginx
Look again at the configuration file for your Pod. The Pod
has an
emptyDir
volume, and the container mounts the volume
at
.
In your shell, create an
index.html
file in the
directory:
echo
'Hello shell demo'
&gt; /usr/share/nginx/html/index.html
In your shell, send a GET request to the nginx server:
# Run this in the shell inside your container
apt-get install curl
curl http://localhost/
The output shows the text that you wrote to the
file:
Hello shell demo
When you are finished with your shell, enter
exit
.
exit
# To quit the shell in the container
Running individual commands in a container
In a

n ordinary command window, not your shell, list the environment
variables in the running container:
kubectl
exec
shell-demo env
Experiment with running other commands. Here are some examples:
kubectl
exec
shell-demo -- ps aux
kubectl
exec
shell-demo -- ls /
kubectl
exec
shell-demo -- cat /proc/1/mounts
Opening a shell when a Pod has more than one container
If a Pod has more than one container, use
--container
or
-c
to
specify a container in the
command. For example,
suppose you have a Pod named my-pod, and the Pod has two containers
named
main-app
and
helper-app
. The following command would open a
shell to the
main-app
container.
kubectl
exec
-i -t my-pod --container main-app -- /bin/bash
Note:
The short options
-i
and
-t
are the same as the long options
--stdin
and
--tty
What's next
Read about
Feedback
Yes
No
.
or
.
Last modified April 26, 2022 at 12:30 AM PST:
Reorg the monitoring task section (#32823) (f26e8eff23)
Edit this page
Create child page
Home
Blog
Training
Partners
Community
CC BY 4.0

Limit Storage Consumption | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on Linux
I

nstall and Set Up kubectl on macOS
Debug Pods
Get a Shel

l to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespace Level
Restri

ct a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption

 Configuration (v1)
Tasks
This example demonstrates how to limit the amount of storage consumed in a namespace.
The following resources are used in the demonstration:
,
,
and
.
minikube
To check the version, enter
kubectl version
.
Scenario: Limiting Storage Consumption
The cluster-admin is operating a cluster on behalf of a user population and the admin wants to control
how much storage a single namespace can consume in order to control cost.
The admin would like to limit:
The number of persistent volume claims in a namespace
The amount of storage each claim can request
The amount of cumulative storage the namespace can have
LimitRange to limit requests for storage
Adding a
to a namespace enforces storage request sizes to a minimum and maximum. Storage is requested
via
. The admission controller that enforces limit ranges will reject any PVC that is above or below
the values set by the admin.
In this example, a PVC requesting 10Gi of storage would be rejected because it exceeds the 2Gi max.
:
v1
kind
:
metadata
:
name
:
storagelimits
spec
:
limits
:
-
type
:
max
:
storage
:
2Gi
min
:
storage
:
1Gi
Minimum storage requests are used when the underlying storage provider requires certain minimums. For example,
AWS EBS volumes have a 1Gi minimum requirement.
StorageQuota to limit PVC count and cumulative storage capacity
Admins can limit the number of PVCs in a namespace as well as the cumulative capacity of those PVCs. New PVCs that exceed
either maximum value will be rejected.
In this example, a 6th PVC in the namespace would be rejected because it exceeds the maximum count of 5. Alternatively,
a 5Gi maximum quota when combined with the 2Gi max limit above, cannot have 3 PVCs where each has 2Gi. That would be 6Gi requested
for a namespace capped at 5Gi.
:
v1
kind
:
metadata
:
name
:
storagequota
spec
:
hard
:
persistentvolumeclaims
:
"5"
requests.storage
:
"5Gi"
Summary
A limit range can put a ceiling on how much storage is requested while a resourc

e quota can effectively cap the storage
consumed by a namespace through claim counts and cumulative storage capacity. The allows a cluster-admin to plan their
cluster's storage budget without risk of any one project going over their allotment.
Feedback
Yes
No
.
or
.
Last modified February 03, 2021 at 2:28 PM PST:
Apply suggestions from code review (c4ef1d4b81)
Summary
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes API health endpoints | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubect

l on Linux
Debug Pods
Debug Running Pod

s
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespace 

Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserve

r Encryption Configuration (v1)
Reference
The Kubernetes
API server
provides API endpoints to indicate the current status of the API server.
This page describes these API endpoints and explains how you can use them.
API endpoints for health
The Kubernetes API server provides 3 API endpoints (
healthz
,
livez
and
readyz
) to indicate the current status of the API server.
The
healthz
endpoint is deprecated (since Kubernetes v1.16), and you should use the more specific
livez
and
readyz
endpoints instead.
The
livez
endpoint can be used with the
--livez-grace-period
flag
to specify the startup duration.
For a graceful shutdown you can specify the
--shutdown-delay-duration
flag
with the
/readyz
endpoint.
Machines that check the
healthz
/
livez
/
readyz
of the API server should rely on the HTTP status code.
A status code
200
indicates the API server is
healthy
/
live
/
ready
, depending on the called endpoint.
The more verbose options shown below are intended to be used by human operators to debug their cluster or understand the state of the API server.
The following examples will show how you can interact with the health API endpoints.
For all endpoints, you can use the
verbose
parameter to print out the checks and their status.
This can be useful for a human operator to debug the current status of the API server, it is not intended to be consumed by a machine:
curl -k https://localhost:6443/livez?verbose
or from a remote host with authentication:
kubectl get --raw
=
'/readyz?verbose'
The output will look like this:
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
healthz check passed
The Kubernetes API server also supports to exclude specific checks.
The query parameters can also be combined like in this example:
curl -k
'https://localhost:6443/readyz?verbose&am

p;exclude=etcd'
The output show that the
etcd
check is excluded:
[+]ping ok
[+]log ok
[+]etcd excluded: ok
[+]shutdown ok
Individual health checks
FEATURE STATE:
Kubernetes v1.26 [alpha]
Each individual health check exposes an HTTP endpoint and can be checked individually.
The schema for the individual health checks is
/livez/&lt;healthcheck-name&gt;
where
livez
and
readyz
and be used to indicate if you want to check the liveness or the readiness of the API server.
The
&lt;healthcheck-name&gt;
path can be discovered using the
verbose
flag from above and take the path between
[+]
and
ok
.
These individual health checks should not be consumed by machines but can be helpful for a human operator to debug a system:
curl -k https://localhost:6443/livez/etcd
Feedback
Yes
No
.
or
.
Last modified November 25, 2021 at 9:41 AM PST:
fix: typo in health-checks.md (af2a60b57f)
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-apiserver Configuration (v1alpha1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kub

ectl on Linux
Debug Pods
Debug Running 

Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespa

ce Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apise

rver Encryption Configuration (v1)
Reference
Package v1alpha1 is the v1alpha1 version of the API.
AdmissionConfiguration
EgressSelectorConfiguration
TracingConfiguration
AdmissionConfiguration provides versioned configuration for admission controllers.
Field
string
apiserver.k8s.io/v1alpha1
kind
string
plugins
[]AdmissionPluginConfiguration
Plugins allows specifying a configuration per admission control plugin.
EgressSelectorConfiguration provides versioned configuration for egress selector clients.
Field
string
kind
string
egressSelections
[]EgressSelection
connectionServices contains a list of egress selection client configurations
TracingConfiguration provides versioned configuration for tracing clients.
Field
string
kind
string
endpoint
string
Endpoint of the collector that's running on the control-plane node.
The APIServer uses the egressType ControlPlane when sending data to the collector.
The syntax is defined in https://github.com/grpc/grpc/blob/master/doc/naming.md.
Defaults to the otlpgrpc default, localhost:4317
The connection is insecure, and does not support TLS.
samplingRatePerMillion
int32
SamplingRatePerMillion is the number of samples to collect per million spans.
Defaults to 0.
AdmissionPluginConfiguration
AdmissionPluginConfiguration provides the configuration for a single plug-in.
Field
name
string
Name is the name of the admission controller.
It must match the registered admission plugin name.
path
string
Path is the path to a configuration file that contains the plugin's
configuration
k8s.io/apimachinery/pkg/runtime.Unknown
Configuration is an embedded configuration object to be used as the plugin's
configuration. If present, it will be used instead of the path to the configuration file.
Connection
EgressSelection
Connection provides the configuration for a single egress selection client.
Field
proxyProtocol
ProtocolType
Protocol is the protocol used to connect from client to the konnectivity server.
transport
Transport
Transport defines the transport configurations we use to dial to the konnectivity server.
This is required if ProxyProtocol is HTTPConnect or G

RPC.
EgressSelection provides the configuration for a single egress selection client.
Field
name
string
name is the name of the egress selection.
Currently supported values are "controlplane", "master", "etcd" and "cluster"
The "master" egress selector is deprecated in favor of "controlplane"
connection
connection is the exact information used to configure the egress selection
(Alias of
string
)
ProtocolType is a set of valid values for Connection.ProtocolType
TCPTransport
Transport
TCPTransport provides the information to connect to konnectivity server via TCP
Field
url
string
URL is the location of the konnectivity server to connect to.
As an example it might be "https://127.0.0.1:8131"
tlsConfig
TLSConfig
TLSConfig is the config needed to use TLS when connecting to konnectivity server
TLSConfig
TLSConfig provides the authentication information to connect to konnectivity server
Only used with TCPTransport
Field
caBundle
string
caBundle is the file location of the CA to be used to determine trust with the konnectivity server.
Must be absent/empty if TCPTransport.URL is prefixed with http://
If absent while TCPTransport.URL is prefixed with https://, default to system trust roots.
clientKey
string
clientKey is the file location of the client key to be used in mtls handshakes with the konnectivity server.
Must be configured if TCPTransport.URL is prefixed with https://
clientCert
string
clientCert is the file location of the client certificate to be used in mtls handshakes with the konnectivity server.
Transport
Transport defines the transport configurations we use to dial to the konnectivity server
Field
tcp
TCP is the TCP configuration for communicating with the konnectivity server via TCP
ProxyProtocol of GRPC is not supported with TCP transport at the moment
Requires at least one of TCP or UDS to be set
uds
UDSTransport
UDS is the UDS configuration for communicating with the konnectivity server via UDS
Transport
UDSTransport provides the information to connect to konnectivity server via UDS
Field
udsName
string
UDSName is the name of the unix domain socket to connect to konnectivity server
This does not use a unix:// prefix. (Eg: /etc/srv/kubernetes/konnectivity-server/konnectivity-server.socket)
Feedback
Yes
No
.
or
.
TLSConfig
Transport
Home
Blog
Training
Partners
Community
CC BY 4.0

StatefulSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

l and Set Up kubectl on Windows
Debug Pods
Troubleshooting 

Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resources with

 AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configurat

ion (v1alpha1)
Reference
StatefulSet represents a set of pods with consistent identities.
apiVersion: apps/v1
import "k8s.io/api/apps/v1"
StatefulSet represents a set of pods with consistent identities. Identities are defined as:
Network: A single stable DNS and hostname.
Storage: As many VolumeClaims as requested.
The StatefulSet guarantees that a given network identity will always map to the same storage identity.
: apps/v1
kind
: StatefulSet
metadata
(
)
Standard object's metadata. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
(
StatefulSetSpec
)
Spec defines the desired identities of pods in this set.
status
(
StatefulSetStatus
)
Status is the current status of Pods in this StatefulSet. This data may be out of date by some window of time.
A StatefulSetSpec is the specification of a StatefulSet.
serviceName
(string), required
serviceName is the name of the service that governs this StatefulSet. This service must exist before the StatefulSet, and is responsible for the network identity of the set. Pods get DNS/hostnames that follow the pattern: pod-specific-string.serviceName.default.svc.cluster.local where "pod-specific-string" is managed by the StatefulSet controller.
selector
(
), required
selector is a label query over pods that should match the replica count. It must match the pod template's labels. More info:
https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
template
(
PodTemplateSpec
template is the object that describes the pod that will be created if insufficient replicas are detected. Each pod stamped out by the StatefulSet will fulfill this Template, but have a unique identity from the rest of the StatefulSet. Each pod will be named with the format &lt;statefulsetname&gt;-&lt;podindex&gt;. For example, a pod in a StatefulSet named "web" with index number "3" would be named "web-3".
replicas
(int32)
replicas is the desired number of replicas of the given Template. These are replicas in the sense that they are instantiations of the same Template, but individual replicas also have a consistent identity. If unspecified, defaults to 1.
updateStrategy
(StatefulSetUpdateStrategy)
updateStrategy indicates the StatefulSetUpdateStrategy that will be employed to update Pods in the StatefulSet when a revision is made to Template.
StatefulSetUpdateStrategy indicates the strategy that the StatefulSet controller will use to perform updates. It includes any additional paramet

ers necessary to perform the update for the indicated strategy.
updateStrategy.type
(string)
Type indicates the type of the StatefulSetUpdateStrategy. Default is RollingUpdate.
updateStrategy.rollingUpdate
(RollingUpdateStatefulSetStrategy)
RollingUpdate is used to communicate parameters when Type is RollingUpdateStatefulSetStrategyType.
RollingUpdateStatefulSetStrategy is used to communicate parameter for RollingUpdateStatefulSetStrategyType.
updateStrategy.rollingUpdate.maxUnavailable
(IntOrString)
The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding up. This can not be 0. Defaults to 1. This field is alpha-level and is only honored by servers that enable the MaxUnavailableStatefulSet feature. The field applies to all pods in the range 0 to Replicas-1. That means if there is any unavailable pod in the range 0 to Replicas-1, it will be counted towards MaxUnavailable.
IntOrString is a type that can hold an int32 or a string. When used in JSON or YAML marshalling and unmarshalling, it produces or consumes the inner type. This allows you to have, for example, a JSON field that can accept a name or number.
updateStrategy.rollingUpdate.partition
(int32)
Partition indicates the ordinal at which the StatefulSet should be partitioned for updates. During a rolling update, all pods from ordinal Replicas-1 to Partition are updated. All pods from ordinal Partition-1 to 0 remain untouched. This is helpful in being able to do a canary based deployment. The default value is 0.
podManagementPolicy
(string)
podManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. The default policy is
OrderedReady
, where pods are created in increasing order (pod-0, then pod-1, etc) and the controller will wait until each pod is ready before continuing. When scaling down, the pods are removed in the opposite order. The alternative policy is
Parallel
which will create pods in parallel to match the desired scale without waiting, and on scale down will delete all pods at once.
revisionHistoryLimit
(int32)
revisionHistoryLimit is the maximum number of revisions that will be maintained in the StatefulSet's revision history. The revision history consists of all revisions not represented by a currently applied StatefulSetSpec version. The default value is 10.
volumeClaimTemplates
([]
)
volumeClaimTemplates is a list of claims that pods are allowed to reference. The StatefulSet controller is responsible for mapping network identities to claims in a way that maintains the identity of a pod. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. A claim in this list takes precedence over any volumes in the template, with the same name.
minReadySeconds
(int32)
Minimum number of seconds for which a newly created pod should be ready without any of its container crashing for it to be considered available. Defaults to 0 (pod will be considered available as soon as it is ready)
persistentVolumeClaimRetentionPolicy
(StatefulSetPersistentVolumeClaimRetentionPolicy)
persistentVolumeClaimRetentionPolicy describes the lifecycle of persistent volume claims created from volumeClaimTemplates. By default, all persistent volume claims are created as needed and retained until manually deleted. This policy allows the lifecycle to be altered, for example by deleting persistent volume claims when their stateful set is deleted, or when their pod is scaled down. This requires the StatefulSetAutoDeletePVC feature gate to be enabled, which is alpha. +optional
StatefulSetPersistentVolumeClaimRetentionPolicy describes the policy used for PVCs created from the StatefulSet VolumeClaimTemplates.
persistentVolumeClaimRetentionPolicy.whenDeleted
(string)
WhenDeleted specifies what happens to PVCs 

created from StatefulSet VolumeClaimTemplates when the StatefulSet is deleted. The default policy of
Retain
causes PVCs to not be affected by StatefulSet deletion. The
Delete
policy causes those PVCs to be deleted.
persistentVolumeClaimRetentionPolicy.whenScaled
(string)
WhenScaled specifies what happens to PVCs created from StatefulSet VolumeClaimTemplates when the StatefulSet is scaled down. The default policy of
Retain
causes PVCs to not be affected by a scaledown. The
Delete
policy causes the associated PVCs for any excess pods above the replica count to be deleted.
ordinals
(StatefulSetOrdinals)
ordinals controls the numbering of replica indices in a StatefulSet. The default ordinals behavior assigns a "0" index to the first replica and increments the index by one for each additional replica requested. Using the ordinals field requires the StatefulSetStartOrdinal feature gate to be enabled, which is alpha.
StatefulSetOrdinals describes the policy used for replica ordinal assignment in this StatefulSet.
ordinals.start
(int32)
start is the number representing the first replica's index. It may be used to number replicas from an alternate index (eg: 1-indexed) over the default 0-indexed names, or to orchestrate progressive movement of replicas from one StatefulSet to another. If set, replica indices will be in the range:
[.spec.ordinals.start, .spec.ordinals.start + .spec.replicas).
If unset, defaults to 0. Replica indices will be in the range:
[0, .spec.replicas).
StatefulSetStatus represents the current state of a StatefulSet.
replicas
(int32), required
replicas is the number of Pods created by the StatefulSet controller.
readyReplicas
(int32)
readyReplicas is the number of pods created for this StatefulSet with a Ready Condition.
currentReplicas
(int32)
currentReplicas is the number of Pods created by the StatefulSet controller from the StatefulSet version indicated by currentRevision.
updatedReplicas
(int32)
updatedReplicas is the number of Pods created by the StatefulSet controller from the StatefulSet version indicated by updateRevision.
availableReplicas
(int32)
Total number of available pods (ready for at least minReadySeconds) targeted by this statefulset.
collisionCount
(int32)
collisionCount is the count of hash collisions for the StatefulSet. The StatefulSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.
conditions
([]StatefulSetCondition)
Patch strategy: merge on key
type
Represents the latest available observations of a statefulset's current state.
StatefulSetCondition describes the state of a statefulset at a certain point.
conditions.status
Status of the condition, one of True, False, Unknown.
conditions.type
Type of statefulset condition.
conditions.lastTransitionTime
(Time)
Last time the condition transitioned from one status to another.
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.
conditions.message
(string)
A human readable message indicating details about the transition.
conditions.reason
(string)
The reason for the condition's last transition.
currentRevision
(string)
currentRevision, if not empty, indicates the version of the StatefulSet used to generate Pods in the sequence [0,currentReplicas).
updateRevision
(string)
updateRevision, if not empty, indicates the version of the StatefulSet used to generate Pods in the sequence [replicas-updatedReplicas,replicas)
observedGeneration
(int64)
observedGeneration is the most recent generation observed for this StatefulSet. It corresponds to the StatefulSet's generation, which is updated on mutation by the API Server.
StatefulSetList
StatefulSetList is a collection of StatefulSets.
: apps/v1
kind
: StatefulSetList
metadata
(
ListMeta
)
Standard list's metadata. More info:
https://git.k8s.io

/community/contributors/devel/sig-architecture/api-conventions.md#metadata
items
([]
Items is the list of stateful sets.
Operations
get
read the specified StatefulSet
HTTP Request
GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
Parameters
name
(
in path
): string, required
name of the StatefulSet
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
401: Unauthorized
get
read status of the specified StatefulSet
GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind StatefulSet
GET /apis/apps/v1/namespaces/{namespace}/statefulsets
namespace
(
in path
namespace
allowWatchBookmarks
(
in query
): boolean
continue
(
in query
): string
continue
fieldSelector
(
in query
): string
labelSelector
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
resourceVersion
(
in query
): string
resourceVersionMatch
(
in query
): string
timeoutSeconds
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/apps/v1/statefulsets
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a StatefulSet
POST /apis/apps/v1/namespaces/{namespace}/statefulsets
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
fieldManager
(
in query
): string
fieldValidation
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
): Accepted
update
replace the specified StatefulSet
PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified StatefulSet
PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified StatefulSet
PATCH /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
field

Validation
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified StatefulSet
PATCH /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a StatefulSet
DELETE /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
gracePeriodSeconds
(
in query
): integer
pretty
(
in query
): string
pretty
propagationPolicy
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
deletecollection
delete collection of StatefulSet
DELETE /apis/apps/v1/namespaces/{namespace}/statefulsets
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Last modified December 09, 2022 at 9:15 AM PST:
API Reference multi-pages for v1.26 (366ba7f62c)
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Running Kubernetes Node Components as a Non-root User | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Tools
Debug Pods
Debug In

it Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Secur

ity Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configu

ration (v1alpha1)
Tasks
Kubernetes v1.22 [alpha]
This document describes how to run Kubernetes Node components such as kubelet, CRI, OCI, and CNI
without root privileges, by using a
user namespace
.
This technique is also known as
rootless mode
.
Note:
This document describes how to run Kubernetes Node components (and hence pods) as a non-root user.
If you are just looking for how to run a pod as a non-root user, see
SecurityContext
.
Your Kubernetes server must be at or later than version 1.22.
.
Enable Cgroup v2
Enable systemd with user session
Configure several sysctl values, depending on host Linux distribution
Ensure that your unprivileged user is listed in
/etc/subuid
and
/etc/subgid
Enable the
KubeletInUserNamespace
feature gate
Running Kubernetes inside Rootless Docker/Podman
kind
kind
supports running Kubernetes inside Rootless Docker or Rootless Podman.
See
Running kind with Rootless Docker
.
minikube
minikube
also supports running Kubernetes inside Rootless Docker or Rootless Podman.
See the Minikube documentation:
Rootless Docker
Rootless Podman
Running Kubernetes inside Unprivileged Containers
Note:
This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the
content guide
before submitting a change.
More information.
sysbox
Sysbox
is an open-source container runtime
(similar to "runc") that supports running system-level workloads such as Docker
and Kubernetes inside unprivileged containers isolated with the Linux user
namespace.
See
Sysbox Quick Start Guide: Kubernetes-in-Docker
for more info.
Sysbox supports running Kubernetes inside unprivileged containers without
requiring Cgroup v2 and without the
feature gate. It
does this by exposing specially crafted
/proc
and
/sys
filesystems inside
the container plus several other advanced OS virtualization techniques.
Running Rootless Kubernetes directly on a host
Note:
K3s
K3s
exp

erimentally supports rootless mode.
See
Running K3s with Rootless mode
for the usage.
Usernetes
Usernetes
is a reference distribution of Kubernetes that can be installed under
$HOME
directory without the root privilege.
Usernetes supports both containerd and CRI-O as CRI runtimes.
Usernetes supports multi-node clusters using Flannel (VXLAN).
See
the Usernetes repo
Manually deploy a node that runs the kubelet in a user namespace
This section provides hints for running Kubernetes in a user namespace manually.
Note:
This section is intended to be read by developers of Kubernetes distributions, not by end users.
Creating a user namespace
The first step is to create a
.
If you are trying to run Kubernetes in a user-namespaced container such as
Rootless Docker/Podman or LXC/LXD, you are all set, and you can go to the next subsection.
Otherwise you have to create a user namespace by yourself, by calling
unshare(2)
with
CLONE_NEWUSER
.
A user namespace can be also unshared by using command line tools such as:
unshare(1)
RootlessKit
become-root
After unsharing the user namespace, you will also have to unshare other namespaces such as mount namespace.
You do
not
need to call
chroot()
nor
pivot_root()
after unsharing the mount namespace,
however, you have to mount writable filesystems on several directories
in
the namespace.
At least, the following directories need to be writable
in
the namespace (not
outside
the namespace):
/etc
/run
/var/logs
/var/lib/kubelet
/var/lib/cni
/var/lib/containerd
(for containerd)
/var/lib/containers
(for CRI-O)
Creating a delegated cgroup tree
In addition to the user namespace, you also need to have a writable cgroup tree with cgroup v2.
Note:
Kubernetes support for running Node components in user namespaces requires cgroup v2.
Cgroup v1 is not supported.
If you are trying to run Kubernetes in Rootless Docker/Podman or LXC/LXD on a systemd-based host, you are all set.
Otherwise you have to create a systemd unit with
Delegate=yes
property to delegate a cgroup tree with writable permission.
On your node, systemd must already be configured to allow delegation; for more details, see
cgroup v2
in the Rootless
Containers documentation.
Configuring network
Note:
The network namespace of the Node components has to have a non-loopback interface, which can be for example configured with
slirp4netns
,
VPNKit
, or
lxc-user-nic(1)
.
The network namespaces of the Pods can be configured with regular CNI plugins.
For multi-node networking, Flannel (VXLAN, 8472/UDP) is known to work.
Ports such as the kubelet port (10250/TCP) and
NodePort
service ports have to be exposed from the Node network namespace to
the host with an external port forwarder, such as RootlessKit, slirp4netns, or
socat(1)
.
You can use the port forwarder from K3s.
See
Running K3s in Rootless Mode
for more details.
The implementation can be found in
the
pkg/rootlessports
package
of k3s.
Configuring CRI
The kubelet relies on a container runtime. You should deploy a container runtime such as
containerd or CRI-O and ensure that it is running within the user namespace before the kubelet starts.
containerd
CRI-O
Running CRI plugin of containerd in a user namespace is supported since containerd 1.4.
Running containerd within a user namespace requires the following configurations.
version =
2
[plugins.
"io.containerd.grpc.v1.cri"
]
# Disable AppArmor
disable_apparmor =
true
# Ignore an error during setting oom_score_adj
restrict_oom_score_adj =
true
# Disable hugetlb cgroup v2 controller (because systemd does not support delegating hugetlb controller)
disable_hugetlb_controller =
true
[plugins.
.containerd]
# Usin

g non-fuse overlayfs is also possible for kernel &gt;= 5.11, but requires SELinux to be disabled
snapshotter =
"fuse-overlayfs"
[plugins.
.containerd.runtimes.runc.options]
# We use cgroupfs that is delegated by systemd, so we do not use SystemdCgroup driver
# (unless you run another systemd in the namespace)
SystemdCgroup =
false
The default path of the configuration file is
/etc/containerd/config.toml
.
The path can be specified with
containerd -c /path/to/containerd/config.toml
.
Running CRI-O in a user namespace is supported since CRI-O 1.22.
CRI-O requires an environment variable
_CRIO_ROOTLESS=1
to be set.
The following configurations are also recommended:
[crio]
storage_driver =
"overlay"
# Using non-fuse overlayfs is also possible for kernel &gt;= 5.11, but requires SELinux to be disabled
storage_option = [
"overlay.mount_program=/usr/local/bin/fuse-overlayfs"
]
[crio.runtime]
# We use cgroupfs that is delegated by systemd, so we do not use "systemd" driver
cgroup_manager =
"cgroupfs"
/etc/crio/crio.conf
.
crio --config /path/to/crio/crio.conf
.
Configuring kubelet
Running kubelet in a user namespace requires the following configuration:
:
kubelet.config.k8s.io/v1beta1
kind
:
KubeletConfiguration
featureGates
:
:
true
cgroupDriver
:
When the
feature gate is enabled, the kubelet ignores errors
that may happen during setting the following sysctl values on the node.
vm.overcommit_memory
vm.panic_on_oom
kernel.panic
kernel.panic_on_oops
kernel.keys.root_maxkeys
kernel.keys.root_maxbytes
.
Within a user namespace, the kubelet also ignores any error raised from trying to open
/dev/kmsg
.
This feature gate also allows kube-proxy to ignore an error during setting
RLIMIT_NOFILE
.
The
feature gate was introduced in Kubernetes v1.22 with "alpha" status.
Running kubelet in a user namespace without using this feature gate is also possible
by mounting a specially crafted proc filesystem (as done by
Sysbox
), but not officially supported.
Configuring kube-proxy
Running kube-proxy in a user namespace requires the following configuration:
:
kubeproxy.config.k8s.io/v1alpha1
kind
:
KubeProxyConfiguration
mode
:
"iptables"
# or "userspace"
conntrack
:
# Skip setting sysctl value "net.netfilter.nf_conntrack_max"
maxPerCore
:
0
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_established"
tcpEstablishedTimeout
:
0s
# Skip setting "net.netfilter.nf_conntrack_tcp_timeout_close"
tcpCloseWaitTimeout
:
0s
Caveats
Most of "non-local" volume drivers such as
nfs
and
iscsi
do not work.
Local volumes like
local
,
hostPath
,
emptyDir
,
configMap
,
secret
, and
downwardAPI
are known to work.
Some CNI plugins may not work. Flannel (VXLAN) is known to work.
For more on this, see the
Caveats and Future work
page
on the rootlesscontaine.rs website.
See Also
rootlesscontaine.rs
Rootless Containers 2020 (KubeCon NA 2020)
Usernetes
Running K3s with rootless mode
KEP-2033: Kubelet-in-UserNS (aka Rootless mode)
Items on this page refer to third party products or projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for those third-party products or projects. See the
CNCF website guidelines
You should read the
before proposing a change that adds an extra third-party link.
Feedback
Yes
No
.
or
.
Last modified July 01, 2022

 at 11:24 AM PST:
kubelet-in-userns.md: minikube now supports Rootless Podman (f62276e924)
Third party content advice
kind
minikube
sysbox
K3s
Usernetes
Caveats
See Also
Home
Blog
Training
Partners
Community
CC BY 4.0

Understanding Kubernetes Objects | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Español (Spanish)
Português (Portuguese)
Bahasa Indonesia
Русский (Russian)
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Ku

bernetes API Aggregation Layer
Tasks
Debug Pods
Deb

ug a StatefulSet
Auditing
Run Jobs
TLS
Tutorials
Security
App

ly Pod Security Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Audit Configuration

 (v1)
Concepts
Overview
This page explains how Kubernetes objects are represented in the Kubernetes API, and how you can
express them in
.yaml
format.
Understanding Kubernetes objects
Kubernetes objects
are persistent entities in the Kubernetes system. Kubernetes uses these
entities to represent the state of your cluster. Specifically, they can describe:
What containerized applications are running (and on which nodes)
The resources available to those applications
The policies around how those applications behave, such as restart policies, upgrades, and fault-tolerance
A Kubernetes object is a "record of intent"--once you create the object, the Kubernetes system
will constantly work to ensure that object exists. By creating an object, you're effectively
telling the Kubernetes system what you want your cluster's workload to look like; this is your
cluster's
desired state
.
To work with Kubernetes objects--whether to create, modify, or delete them--you'll need to use the
. When you use the
kubectl
command-line
interface, for example, the CLI makes the necessary Kubernetes API calls for you. You can also use
the Kubernetes API directly in your own programs using one of the
.
Object spec and status
Almost every Kubernetes object includes two nested object fields that govern
the object's configuration: the object
spec
and the object
status
.
For objects that have a
spec
, you have to set this when you create the object,
providing a description of the characteristics you want the resource to have:
its
.
The
status
describes the
current state
of the object, supplied and updated
by the Kubernetes system and its components. The Kubernetes
control plane
continually
and actively manages every object's actual state to match the desired state you
supplied.
For example: in Kubernetes, a Deployment is an object that can represent an
application running on your cluster. When you create the Deployment, you
might set the Deployment
spec
to specify that you want three replicas of
the application to be running. The Kubernetes system reads the Deployment
spec and starts three instances of your desired application--updating
the status to match your spec. If any of those instances should fail
(a status change), the Kubernetes system responds to the difference
between spec and status by making a correction--in this case, starting
a repl

acement instance.
For more information on the object spec, status, and metadata, see the
Kubernetes API Conventions
.
Describing a Kubernetes object
When you create an object in Kubernetes, you must provide the object spec that describes its
desired state, as well as some basic information about the object (such as a name). When you use
the Kubernetes API to create the object (either directly or via
kubectl
), that API request must
include that information as JSON in the request body.
Most often, you provide the information to
kubectl
in a .yaml file.
kubectl
converts the information to JSON when making the API
request.
Here's an example
.yaml
file that shows the required fields and object spec for a Kubernetes Deployment:
application/deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
nginx-deployment
spec
:
selector
:
matchLabels
:
app
:
nginx
replicas
:
2
# tells deployment to run 2 pods matching the template
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
nginx:1.14.2
ports
:
-
containerPort
:
80
One way to create a Deployment using a
.yaml
file like the one above is to use the
kubectl apply
command
in the
kubectl
command-line interface, passing the
.yaml
file as an argument. Here's an example:
kubectl apply -f https://k8s.io/examples/application/deployment.yaml
The output is similar to this:
deployment.apps/nginx-deployment created
Required fields
In the
.yaml
file for the Kubernetes object you want to create, you'll need to set values for the following fields:
- Which version of the Kubernetes API you're using to create this object
kind
- What kind of object you want to create
metadata
- Data that helps uniquely identify the object, including a
name
string,
UID
, and optional
namespace
spec
- What state you desire for the object
The precise format of the object
spec
is different for every Kubernetes object, and contains
nested fields specific to that object. The
Kubernetes API Reference
can help you find the spec format for all of the objects you can create using Kubernetes.
For example, see the
spec
field
for the Pod API reference.
For each Pod, the
.spec
field specifies the pod and its desired state (such as the container image name for
each container within that pod).
Another example of an object specification is the
spec
field
for the StatefulSet API. For StatefulSet, the
.spec
field specifies the StatefulSet and
its desired state.
Within the
.spec
of a StatefulSet is a
template
for Pod objects. That template describes Pods that the StatefulSet controller will create in order to
satisfy the StatefulSet specification.
Different kinds of object can also have different
.status
; again, the API reference pages
detail the structure of that
.status
field, and its content for each different type of object.
Learn more about the following:
Pods
which are the most important basic Kubernetes objects.
objects.
in Kubernetes.
Kubernetes API overview
which explains some more API concepts.
kubectl
and
kubectl commands
.
Feedback
Yes
No
.
or
.
Last modified November 13, 2022 at 9:26 PM PST:
updated /kubernetes-objects.md (25aa28ff6a)
Home
Blog
Training
Partners
Community
CC BY 4.0

Cluster Administration | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Français (French)
Italiano (Italian)
Deutsch (German)
This is the multi-page printable view of this section.
Click here to print
.
Return to the regular view of this page
.
Lower-level detail relevant to creating or administering a Kubernetes cluster.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
The cluster administration overview is for anyone creating or administering a Kubernetes cluster.
It assumes some familiarity with core Kubernetes
concepts
.
Planning a cluster
See the guides in
Setup
for examples of how to plan, set up, and configure
Kubernetes clusters. The solutions listed in this article are called
distros
.
Note:
Not all distros are actively maintained. Choose distros which have been tested with a recent
version of Kubernetes.
Before choosing a guide, here are some considerations:
Do you want to try out Kubernetes on your computer, or do you want to build a high-availability,
multi-node cluster? Choose distros best suited for your needs.
Will you be using
a hosted Kubernetes cluster
, such as
Google Kubernetes Engine
, or
hosting your own cluster
?
Will your cluster be
on-premises
, or
in the cloud (IaaS)
? Kubernetes does not directly
support hybrid clusters. Instead, you can set up multiple clusters.
If you are configuring Kubernetes on-premises
, consider which
networking model
fits best.
Will you be running Kubernetes on
"bare metal" hardware
or on
virtual machines (VMs)
?
Do you
want to run a cluster
, or do you expect to do
active development of Kubernetes project code
?
If the latter, choose an actively-developed distro. Some distros only use binary releases, but
offer a greater variety of choices.
Familiarize yourself with the
components
needed to run a cluster.
Managing a cluster
Learn how to
manage nodes
.
Learn how to set up and manage the
resource quota
for shared clusters.
Securing a cluster
Generate Certificates
describes the steps to
generate certificates using different tool chains.
Kubernetes Container Environment
describes
the environment for Kubelet managed containers on a Kubernetes node.
describes
how Kubernetes implements access control for its own API.
explains authentication in
Kubernetes, including the various authentication options.
Authorization
is separate from
authentication, and controls how HTTP calls are handled.
Using Admission Controllers
explains plug-ins which intercepts requests to the Kubernetes API server after authentication
and authorization.
Using Sysctls in a Kubernetes Cluster
describes to an administrator how to use the
sysctl
command-line tool to set kernel parameters
.
Auditing
describes how to interact with Kubernetes'
audit logs.
Securing the kubelet
Control Plane-Node communication
Optional Cluster Services
DNS Integration
describes how to resolve
a DNS name directly to a Kubernetes service.
Logging and Monitoring Cluster Activity
explains how logging in Kubernetes works and how to implement it.
1 - Certificates
To learn how to generate certificates for your cluster, see
.
2 - Managing Resources
You've deployed your application and exposed it via a service. Now what? Kubernetes provides a number of tools to help you manage your application deployment, including scaling and updating. Among the features that we will discuss in more depth are
configuration files
and
labels
.
Organizing resource configurations

Many applications require multiple resources to be created, such as a Deployment and a Service. Management of multiple resources can be simplified by grouping them together in the same file (separated by
---
in YAML). For example:
application/nginx-app.yaml
:
v1
kind
:
Service
metadata
:
name
:
my-nginx-svc
labels
:
app
:
nginx
spec
:
type
:
LoadBalancer
ports
:
-
port
:
80
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
labels
:
app
:
nginx
spec
:
replicas
:
3
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Multiple resources can be created the same way as a single resource:
kubectl apply -f https://k8s.io/examples/application/nginx-app.yaml
service/my-nginx-svc created
deployment.apps/my-nginx created
The resources will be created in the order they appear in the file. Therefore, it's best to specify the service first, since that will ensure the scheduler can spread the pods associated with the service as they are created by the controller(s), such as Deployment.
also accepts multiple
-f
arguments:
kubectl apply -f https://k8s.io/examples/application/nginx/nginx-svc.yaml -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml
It is a recommended practice to put resources related to the same microservice or application tier into the same file, and to group all of the files associated with your application in the same directory. If the tiers of your application bind to each other using DNS, you can deploy all of the components of your stack together.
A URL can also be specified as a configuration source, which is handy for deploying directly from configuration files checked into GitHub:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/application/nginx/nginx-deployment.yaml
Bulk operations in kubectl
Resource creation isn't the only operation that
kubectl
can perform in bulk. It can also extract resource names from configuration files in order to perform other operations, in particular to delete the same resources you created:
kubectl delete -f https://k8s.io/examples/application/nginx-app.yaml
deployment.apps
"my-nginx"
deleted
service
"my-nginx-svc"
deleted
In the case of two resources, you can specify both resources on the command line using the resource/name syntax:
kubectl delete deployments/my-nginx services/my-nginx-svc
For larger numbers of resources, you'll find it easier to specify the selector (label query) specified using
-l
or
--selector
, to filter resources by their labels:
kubectl delete deployment,services -l
app
=
nginx
deleted
service
deleted
Because
kubectl
outputs resource names in the same syntax it accepts, you can chain operations using
$()
or
xargs
:
kubectl get
$(
kubectl create -f docs/concepts/cluster-administration/nginx/ -o name | grep service
)
kubectl create -f docs/concepts/cluster-administration/nginx/ -o name | grep service | xargs -i kubectl get
{}
NAME           TYPE           CLUSTER-IP   EXTERNAL-IP   PORT
(
S
)
AGE
my-nginx-svc   LoadBalancer   10.0.0.208   &lt;pending&gt;     80/TCP       0s
With the above commands, we first create resources under
examples/application/nginx/
and print the resources created with
-o name
output format
(print each resource as resource/name). Then we
grep
only the "service", and then print it with
.
If you happen to organize your resources across several subdirectories within a particular directory, you can recursively perform the operations on the subdirectories also, by specifying
--recursive
or
-R
alongside the
--filename,-f
flag.
For instance, assume there is a directory
project/k8s/development
that holds all of the
manifests
needed for the development environment, organized by resource type:
├── configmap

│   └── my-configmap.yaml
├── deployment
│   └── my-deployment.yaml
└── pvc
    └── my-pvc.yaml
By default, performing a bulk operation on
will stop at the first level of the directory, not processing any subdirectories. If we had tried to create the resources in this directory using the following command, we would have encountered an error:
kubectl apply -f project/k8s/development
error: you must provide one or more resources by argument or filename
(
.json|.yaml|.yml|stdin
)
Instead, specify the
or
-R
flag with the
flag as such:
kubectl apply -f project/k8s/development --recursive
configmap/my-config created
deployment.apps/my-deployment created
persistentvolumeclaim/my-pvc created
The
flag works with any operation that accepts the
flag such as:
kubectl {create,get,delete,describe,rollout}
etc.
The
flag also works when multiple
-f
arguments are provided:
kubectl apply -f project/k8s/namespaces -f project/k8s/development --recursive
namespace/development created
namespace/staging created
If you're interested in learning more about
kubectl
, go ahead and read
.
Using labels effectively
The examples we've used so far apply at most a single label to any resource. There are many scenarios where multiple labels should be used to distinguish sets from one another.
For instance, different applications would use different values for the
app
label, but a multi-tier application, such as the
guestbook example
, would additionally need to distinguish each tier. The frontend could carry the following labels:
labels
:
app
:
guestbook
tier
:
frontend
while the Redis master and slave would have different
tier
labels, and perhaps even an additional
role
label:
labels
:
app
:
guestbook
tier
:
backend
role
:
master
and
labels
:
app
:
guestbook
tier
:
backend
role
:
slave
The labels allow us to slice and dice our resources along any dimension specified by a label:
kubectl apply -f examples/guestbook/all-in-one/guestbook-all-in-one.yaml
kubectl get pods -Lapp -Ltier -Lrole
NAME                           READY     STATUS    RESTARTS   AGE       APP         TIER       ROLE
guestbook-fe-4nlpb             1/1       Running
0
1m        guestbook   frontend   &lt;none&gt;
guestbook-fe-ght6d             1/1       Running
0
guestbook-fe-jpy62             1/1       Running
0
guestbook-redis-master-5pg3b   1/1       Running
0
1m        guestbook   backend    master
guestbook-redis-slave-2q2yf    1/1       Running
0
1m        guestbook   backend    slave
guestbook-redis-slave-qgazl    1/1       Running
0
my-nginx-divi2                 1/1       Running
0
29m       nginx       &lt;none&gt;     &lt;none&gt;
my-nginx-o0ef1                 1/1       Running
0
kubectl get pods -lapp
=
guestbook,role
=
slave
NAME                          READY     STATUS    RESTARTS   AGE
guestbook-redis-slave-2q2yf   1/1       Running
0
3m
guestbook-redis-slave-qgazl   1/1       Running
0
3m
Canary deployments
Another scenario where multiple labels are needed is to distinguish deployments of different releases or configurations of the same component. It is common practice to deploy a
canary
of a new application release (specified via image tag in the pod template) side by side with the previous release so that the new release can receive live production traffic before fully rolling it out.
For instance, you can use a
track
label to differentiate different releases.
The primary, stable release would have a
track
label with value as
stable
:
name
:
frontend
replicas
:
3
...
labels
:
app
:
guestbook
tier
:
frontend
track
:
stable
...
image
:
gb-frontend:v3
and then you can create a new 

release of the guestbook frontend that carries the
track
label with different value (i.e.
canary
), so that two sets of pods would not overlap:
name
:
frontend-canary
replicas
:
1
...
labels
:
app
:
guestbook
tier
:
frontend
track
:
canary
...
image
:
gb-frontend:v4
The frontend service would span both sets of replicas by selecting the common subset of their labels (i.e. omitting the
track
label), so that the traffic will be redirected to both applications:
selector
:
app
:
guestbook
tier
:
frontend
You can tweak the number of replicas of the stable and canary releases to determine the ratio of each release that will receive live production traffic (in this case, 3:1).
Once you're confident, you can update the stable track to the new application release and remove the canary one.
For a more concrete example, check the
tutorial of deploying Ghost
.
Updating labels
Sometimes existing pods and other resources need to be relabeled before creating new resources. This can be done with
kubectl label
.
For example, if you want to label all your nginx pods as frontend tier, run:
kubectl label pods -l
app
=
nginx
tier
=
fe
pod/my-nginx-2035384211-j5fhi labeled
pod/my-nginx-2035384211-u2c7e labeled
pod/my-nginx-2035384211-u3t6x labeled
This first filters all pods with the label "app=nginx", and then labels them with the "tier=fe".
To see the pods you labeled, run:
kubectl get pods -l
app
=
nginx -L tier
NAME                        READY     STATUS    RESTARTS   AGE       TIER
my-nginx-2035384211-j5fhi   1/1       Running
0
23m       fe
my-nginx-2035384211-u2c7e   1/1       Running
0
23m       fe
my-nginx-2035384211-u3t6x   1/1       Running
0
23m       fe
This outputs all "app=nginx" pods, with an additional label column of pods' tier (specified with
-L
or
--label-columns
).
For more information, please see
labels
and
.
Updating annotations
Sometimes you would want to attach annotations to resources. Annotations are arbitrary non-identifying metadata for retrieval by API clients such as tools, libraries, etc. This can be done with
kubectl annotate
. For example:
kubectl annotate pods my-nginx-v4-9gw19
description
=
'my frontend running nginx'
kubectl get pods my-nginx-v4-9gw19 -o yaml
apiVersion: v1
kind: pod
metadata:
annotations:
description: my frontend running nginx
...
annotations
and
document.
Scaling your application
When load on your application grows or shrinks, use
kubectl
to scale your application. For instance, to decrease the number of nginx replicas from 3 to 1, do:
kubectl scale deployment/my-nginx --replicas
=
1
deployment.apps/my-nginx scaled
Now you only have one pod managed by the deployment.
app
=
nginx
NAME                        READY     STATUS    RESTARTS   AGE
0
30m
To have the system automatically choose the number of nginx replicas as needed, ranging from 1 to 3, do:
kubectl autoscale deployment/my-nginx --min
=
1
--max
=
3
horizontalpodautoscaler.autoscaling/my-nginx autoscaled
Now your nginx replicas will be scaled up and down as needed, automatically.
kubectl scale
,
kubectl autoscale
and
horizontal pod autoscaler
document.
In-place updates of resources
Sometimes it's necessary to make narrow, non-disruptive updates to resources you've created.
It is suggested to maintain a set of configuration files in source control
(see
configuration as code
),
so that they can be maintained and versioned along with the code for the resources they configure.
Then, you can use
to push your configuration changes to the cluster.
This command will compare the version of the configuration that you're pushing with the previous version and apply the changes you've made, without overwriting any automated changes to properties you haven't specified.
kubectl apply -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml
deployment.app

s/my-nginx configured
Note that
attaches an annotation to the resource in order to determine the changes to the configuration since the previous invocation. When it's invoked,
does a three-way diff between the previous configuration, the provided input and the current configuration of the resource, in order to determine how to modify the resource.
Currently, resources are created without this annotation, so the first invocation of
will fall back to a two-way diff between the provided input and the current configuration of the resource. During this first invocation, it cannot detect the deletion of properties set when the resource was created. For this reason, it will not remove them.
All subsequent calls to
, and other commands that modify the configuration, such as
kubectl replace
and
kubectl edit
, will update the annotation, allowing subsequent calls to
to detect and perform deletions using a three-way diff.
Alternatively, you may also update resources with
:
kubectl edit deployment/my-nginx
This is equivalent to first
get
the resource, edit it in text editor, and then
apply
the resource with the updated version:
kubectl get deployment my-nginx -o yaml &gt; /tmp/nginx.yaml
vi /tmp/nginx.yaml
# do some edit, and then save the file
kubectl apply -f /tmp/nginx.yaml
deployment.apps/my-nginx configured
rm /tmp/nginx.yaml
This allows you to do more significant changes more easily. Note that you can specify the editor with your
EDITOR
or
KUBE_EDITOR
environment variables.
document.
kubectl patch
You can use
to update API objects in place. This command supports JSON patch,
JSON merge patch, and strategic merge patch. See
and
.
Disruptive updates
In some cases, you may need to update resource fields that cannot be updated once initialized, or you may want to make a recursive change immediately, such as to fix broken pods created by a Deployment. To change such fields, use
replace --force
, which deletes and re-creates the resource. In this case, you can modify your original configuration file:
kubectl replace -f https://k8s.io/examples/application/nginx/nginx-deployment.yaml --force
deployment.apps/my-nginx deleted
deployment.apps/my-nginx replaced
Updating your application without a service outage
At some point, you'll eventually need to update your deployed application, typically by specifying a new image or image tag, as in the canary deployment scenario above.
kubectl
supports several update operations, each of which is applicable to different scenarios.
We'll guide you through how to create and update applications with Deployments.
Let's say you were running version 1.14.2 of nginx:
kubectl create deployment my-nginx --image
=
with 3 replicas (so the old and new revisions can coexist):
kubectl scale deployment my-nginx --current-replicas
=
1
--replicas
=
3
To update to version 1.16.1, change
.spec.template.spec.containers[0].image
from
to
nginx:1.16.1
using the previous kubectl commands.
That's it! The Deployment will declaratively update the deployed nginx application progressively behind the scene. It ensures that only a certain number of old replicas may be down while they are being updated, and only a certain number of new replicas may be created above the desired number of pods. To learn more details about it, visit
Deployment page
.
Learn about
how to use
kubectl
for application introspection and debugging
.
See
Configuration Best Practices and Tips
.
3 - Cluster Networking
Networking is a central part of Kubernetes, but it can be challenging to
understand exactly how it is expected to work. There are 4 distinct networking
problems to address:
Highly-coupled container-to-con

tainer communications: this is solved by
Pods
and
localhost
communications.
Pod-to-Pod communications: this is the primary focus of this document.
Pod-to-Service communications: this is covered by
Services
.
External-to-Service communications: this is also covered by Services.
Kubernetes is all about sharing machines between applications. Typically,
sharing machines requires ensuring that two applications do not try to use the
same ports. Coordinating ports across multiple developers is very difficult to
do at scale and exposes users to cluster-level issues outside of their control.
Dynamic port allocation brings a lot of complications to the system - every
application has to take ports as flags, the API servers have to know how to
insert dynamic port numbers into configuration blocks, services have to know
how to find each other, etc. Rather than deal with this, Kubernetes takes a
different approach.
To learn about the Kubernetes networking model, see
here
.
How to implement the Kubernetes network model
The network model is implemented by the container runtime on each node. The most common container runtimes use
Container Network Interface
(CNI) plugins to manage their network and security capabilities. Many different CNI plugins exist from many different vendors. Some of these provide only basic features of adding and removing network interfaces, while others provide more sophisticated solutions, such as integration with other container orchestration systems, running multiple CNI plugins, advanced IPAM features etc.
See
this page
for a non-exhaustive list of networking addons supported by Kubernetes.
The early design of the networking model and its rationale, and some future
plans are described in more detail in the
networking design document
.
4 - Logging Architecture
Application logs can help you understand what is happening inside your application. The
logs are particularly useful for debugging problems and monitoring cluster activity. Most
modern applications have some kind of logging mechanism. Likewise, container engines
are designed to support logging. The easiest and most adopted logging method for
containerized applications is writing to standard output and standard error streams.
However, the native functionality provided by a container engine or runtime is usually
not enough for a complete logging solution.
For example, you may want to access your application's logs if a container crashes,
a pod gets evicted, or a node dies.
In a cluster, logs should have a separate storage and lifecycle independent of nodes,
pods, or containers. This concept is called
cluster-level logging
.
Cluster-level logging architectures require a separate backend to store, analyze, and
query logs. Kubernetes does not provide a native storage solution for log data. Instead,
there are many logging solutions that integrate with Kubernetes. The following sections
describe how to handle and store logs on nodes.
Pod and container logs
Kubernetes captures logs from each container in a running Pod.
This example uses a manifest for a
Pod
with a container
that writes text to the standard output stream, once per second.
debug/counter-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
busybox:1.28
args
:
[/bin/sh, -c,
'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done'
]
To run this pod, use the following command:
kubectl apply -f https://k8s.io/examples/debug/counter-pod.yaml
The output is:
pod/counter created
To fetch the logs, use the
kubectl logs
command, as follows:
kubectl logs counter
The output is similar to:
0: Fri Apr  1 11:42:23 UTC 2022
1: Fri Apr  1 11:42:24 UTC 2022
2: Fri Apr  1 11:42:25 UTC 2022
You can use
kubectl logs --previous
to retrieve logs from a previous instantiation of a container.
If your pod has multiple containers, specify which container's logs you want to access by
appending a container name to the command, with a
-c
flag, like so:
kubectl

 logs counter -c count
See the
documentation
How nodes handle container logs
A container runtime handles and redirects any output generated to a containerized application's
stdout
and
stderr
streams.
Different container runtimes implement this in different ways; however, the integration with the kubelet is standardized
as the
CRI logging format
.
By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node,
all corresponding containers are also evicted, along with their logs.
The kubelet makes logs available to clients via a special feature of the Kubernetes API. The usual way to access this is
by running
.
Log rotation
Kubernetes v1.21 [stable]
You can configure the kubelet to rotate logs automatically.
If you configure rotation, the kubelet is responsible for rotating container logs and managing the logging directory structure.
The kubelet sends this information to the container runtime (using CRI),
and the runtime writes the container logs to the given location.
You can configure two kubelet
configuration settings
,
containerLogMaxSize
and
containerLogMaxFiles
,
using the
kubelet configuration file
.
These settings let you configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
When you run
as in
the basic logging example, the kubelet on the node handles the request and
reads directly from the log file. The kubelet returns the content of the log file.
Note:
Only the contents of the latest log file are available through
.
For example, if a Pod writes 40 MiB of logs and the kubelet rotates logs
after 10 MiB, running
returns at most 10MiB of data.
System component logs
There are two types of system components: those that typically run in a container,
and those components directly involved in running containers. For example:
The kubelet and container runtime do not run in containers. The kubelet runs
your containers (grouped together in
pods
)
The Kubernetes scheduler, controller manager, and API server run within pods
(usually
static Pods
).
The etcd component runs in the control plane, and most commonly also as a static pod.
If your cluster uses kube-proxy, you typically run this as a
DaemonSet
.
Log locations
The way that the kubelet and container runtime write logs depends on the operating
system that the node uses:
Linux
Windows
On Linux nodes that use systemd, the kubelet and container runtime write to journald
by default. You use
journalctl
to read the systemd journal; for example:
journalctl -u kubelet
.
If systemd is not present, the kubelet and container runtime write to
.log
files in the
/var/log
directory. If you want to have logs written elsewhere, you can indirectly
run the kubelet via a helper tool,
kube-log-runner
, and use that tool to redirect
kubelet logs to a directory that you choose.
You can also set a logging directory using the deprecated kubelet command line
argument
--log-dir
. However, the kubelet always directs your container runtime to
write logs into directories within
/var/log/pods
.
For more information on
, read
.
By default, the kubelet writes logs to files within the directory
C:\var\logs
(notice that this is not
C:\var\log
).
Although
is the Kubernetes default location for these logs, several
cluster deployment tools set up Windows nodes to log to
C:\var\log\kubelet
instead.
If you want to have logs written elsewhere, you can indirectly
However, the kubelet always directs your container runtime to write logs within the
directory
C:\var\log\pods
.
, read
.
For Kubernetes cluster components that run in pods, these write to files inside
the
/var/log
directo

ry, bypassing the default logging mechanism (the components
do not write to the systemd journal). You can use Kubernetes' storage mechanisms
to map persistent storage into the container that runs the component.
For details about etcd and its logs, view the
etcd documentation
.
Again, you can use Kubernetes' storage mechanisms to map persistent storage into
the container that runs the component.
Note:
If you deploy Kubernetes cluster components (such as the scheduler) to log to
a volume shared from the parent node, you need to consider and ensure that those
logs are rotated.
Kubernetes does not manage that log rotation
.
Your operating system may automatically implement some log rotation - for example,
if you share the directory
/var/log
into a static Pod for a component, node-level
log rotation treats a file in that directory the same as a file written by any component
outside Kubernetes.
Some deploy tools account for that log rotation and automate it; others leave this
as your responsibility.
Cluster-level logging architectures
While Kubernetes does not provide a native solution for cluster-level logging, there are several common approaches you can consider. Here are some options:
Use a node-level logging agent that runs on every node.
Include a dedicated sidecar container for logging in an application pod.
Push logs directly to a backend from within an application.
Using a node logging agent
You can implement cluster-level logging by including a
node-level logging agent
on each node. The logging agent is a dedicated tool that exposes logs or pushes logs to a backend. Commonly, the logging agent is a container that has access to a directory with log files from all of the application containers on that node.
Because the logging agent must run on every node, it is recommended to run the agent
as a
DaemonSet
.
Node-level logging creates only one agent per node and doesn't require any changes to the applications running on the node.
Containers write to stdout and stderr, but with no agreed format. A node-level agent collects these logs and forwards them for aggregation.
Using a sidecar container with the logging agent
You can use a sidecar container in one of the following ways:
The sidecar container streams application logs to its own
stdout
.
The sidecar container runs a logging agent, which is configured to pick up logs from an application container.
Streaming sidecar container
By having your sidecar containers write to their own
stdout
and
stderr
streams, you can take advantage of the kubelet and the logging agent that
already run on each node. The sidecar containers read logs from a file, a socket,
or journald. Each sidecar container prints a log to its own
stdout
or
stderr
stream.
This approach allows you to separate several log streams from different
parts of your application, some of which can lack support
for writing to
stdout
or
stderr
. The logic behind redirecting logs
is minimal, so it's not a significant overhead. Additionally, because
stdout
and
stderr
are handled by the kubelet, you can use built-in tools
like
.
For example, a pod runs a single container, and the container
writes to two different log files using two different formats. Here's a
manifest for the Pod:
admin/logging/two-files-counter-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
args
:
- /bin/sh
- -c
- &gt;
i=0;
while true;
do
echo "$i: $(date)" &gt;&gt; /var/log/1.log;
echo "$(date) INFO $i" &gt;&gt; /var/log/2.log;
i=$((i+1));
sleep 1;
done
:
-
name
:
varlog
mountPath
:
/var/log
volumes
:
-
name
:
varlog
emptyDir
:
{}
It is not recommended to write log entries with different formats to the same log
stream, even if you managed to redirect both components to the
stdout
stream of
the container. Instead, you can create two sidecar containers. Each sidecar
container could tail a particular log file from a shared volume and then redirect
the logs to its own
s

tdout
stream.
Here's a manifest for a pod that has two sidecar containers:
admin/logging/two-files-counter-pod-streaming-sidecar.yaml
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
args
:
- /bin/sh
- -c
- &gt;
i=0;
do
sleep 1;
done
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
count-log-1
image
:
args
:
[/bin/sh, -c, 'tail -n+1 -F /var/log/1.log']
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
count-log-2
image
:
args
:
[/bin/sh, -c, 'tail -n+1 -F /var/log/2.log']
:
-
name
:
varlog
mountPath
:
/var/log
volumes
:
-
name
:
varlog
emptyDir
:
{}
Now when you run this pod, you can access each log stream separately by
running the following commands:
kubectl logs counter count-log-1
0: Fri Apr  1 11:42:26 UTC 2022
1: Fri Apr  1 11:42:27 UTC 2022
2: Fri Apr  1 11:42:28 UTC 2022
...
kubectl logs counter count-log-2
Fri Apr  1 11:42:29 UTC 2022 INFO 0
Fri Apr  1 11:42:30 UTC 2022 INFO 0
Fri Apr  1 11:42:31 UTC 2022 INFO 0
...
If you installed a node-level agent in your cluster, that agent picks up those log
streams automatically without any further configuration. If you like, you can configure
the agent to parse log lines depending on the source container.
Even for Pods that only have low CPU and memory usage (order of a couple of millicores
for cpu and order of several megabytes for memory), writing logs to a file and
then streaming them to
stdout
can double how much storage you need on the node.
If you have an application that writes to a single file, it's recommended to set
/dev/stdout
as the destination rather than implement the streaming sidecar
container approach.
Sidecar containers can also be used to rotate log files that cannot be rotated by
the application itself. An example of this approach is a small container running
logrotate
periodically.
However, it's more straightforward to use
stdout
and
stderr
directly, and
leave rotation and retention policies to the kubelet.
Sidecar container with a logging agent
If the node-level logging agent is not flexible enough for your situation, you
can create a sidecar container with a separate logging agent that you have
configured specifically to run with your application.
Note:
Using a logging agent in a sidecar container can lead
to significant resource consumption. Moreover, you won't be able to access
those logs using
because they are not controlled
by the kubelet.
Here are two example manifests that you can use to implement a sidecar container with a logging agent.
The first manifest contains a
ConfigMap
to configure fluentd.
admin/logging/fluentd-sidecar-config.yaml
:
v1
kind
:
ConfigMap
metadata
:
name
:
fluentd-config
data
:
fluentd.conf
:
|
&lt;source&gt;
type tail
format none
path /var/log/1.log
pos_file /var/log/1.log.pos
tag count.format1
&lt;/source&gt;
type tail
path /var/log/2.log
pos_file /var/log/2.log.pos
tag count.format2
&lt;match **&gt;
type google_cloud
&lt;/match&gt;
Note:
In the sample configurations, you can replace fluentd with any logging agent, reading
from any source inside an application container.
The second manifest describes a pod that has a sidecar container running fluentd.
The pod mounts a volume where fluentd can pick up its configuration data.
admin/logging/two-files-counter-pod-agent-sidecar.yaml
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
args
:
- /bin/sh
- -c
- &gt;
i=0;
do
sleep 1;
done
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
count-agent
image
:
registry.k8s.io/fluentd-gcp:1.30
env
:
-
name

:
FLUENTD_ARGS
value
:
-c /etc/fluentd-config/fluentd.conf
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
config-volume
mountPath
:
/etc/fluentd-config
volumes
:
-
name
:
varlog
emptyDir
:
{}
-
name
:
configMap
:
name
:
Exposing logs directly from the application
Cluster-logging that exposes or pushes logs directly from every application is outside the scope of Kubernetes.
Read about
Kubernetes system logs
customise the termination message
that Kubernetes records when a Pod fails
5 - Metrics For Kubernetes System Components
System component metrics can give a better look into what is happening inside them. Metrics are
particularly useful for building dashboards and alerts.
Kubernetes components emit metrics in
Prometheus format
.
This format is structured plain text, designed so that people and machines can both read it.
Metrics in Kubernetes
In most cases metrics are available on
/metrics
endpoint of the HTTP server. For components that
doesn't expose endpoint by default it can be enabled using
--bind-address
flag.
Examples of those components:
kubelet
In a production environment you may want to configure
Prometheus Server
or some other metrics scraper to periodically gather these metrics and make them available in some
kind of time series database.
Note that
kubelet
also exposes metrics in
/metrics/cadvisor
,
/metrics/resource
and
/metrics/probes
endpoints. Those metrics do not
have same lifecycle.
If your cluster uses
RBAC
, reading metrics requires
authorization via a user, group or ServiceAccount with a ClusterRole that allows accessing
/metrics
:
rbac.authorization.k8s.io/v1
kind
:
metadata
:
name
:
prometheus
rules
:
-
nonResourceURLs
:
-
"/metrics"
verbs
:
- get
Metric lifecycle
Alpha metric → Stable metric → Deprecated metric → Hidden metric → Deleted metric
Alpha metrics have no stability guarantees. These metrics can be modified or deleted at any time.
Stable metrics are guaranteed to not change. This means:
A stable metric without a deprecated signature will not be deleted or renamed
A stable metric's type will not be modified
Deprecated metrics are slated for deletion, but are still available for use.
These metrics include an annotation about the version in which they became deprecated.
For example:
Before deprecation
# HELP some_counter this counts things
# TYPE some_counter counter
some_counter 0
After deprecation
# HELP some_counter (Deprecated since 1.15.0) this counts things
Hidden metrics are no longer published for scraping, but are still available for use. To use a
hidden metric, please refer to the
Show hidden metrics
section.
Deleted metrics are no longer published and cannot be used.
As described above, admins can enable hidden metrics through a command-line flag on a specific
binary. This intends to be used as an escape hatch for admins if they missed the migration of the
metrics deprecated in the last release.
The flag
show-hidden-metrics-for-version
takes a version for which you want to show metrics
deprecated in that release. The version is expressed as x.y, where x is the major version, y is
the minor version. The patch version is not needed even though a metrics can be deprecated in a
patch release, the reason for that is the metrics deprecation policy runs against the minor release.
The flag can only take the previous minor version as it's value. All metrics hidden in previous
will be emitted if admins set the previous version to
. The too
old version is not allowed because this violates the metrics deprecated policy.
Take metric
A
as an example, here assumed that
A
is deprecated in 1.n. According to metrics
deprecated policy, we can reach the following conclusion:
In release
1.n
, th

e metric is deprecated, and it can be emitted by default.
In release
1.n+1
, the metric is hidden by default and it can be emitted by command line
show-hidden-metrics-for-version=1.n
.
In release
1.n+2
, the metric should be removed from the codebase. No escape hatch anymore.
If you're upgrading from release
1.12
to
1.13
, but still depend on a metric
A
deprecated in
1.12
, you should set hidden metrics via command line:
--show-hidden-metrics=1.12
and remember
to remove this metric dependency before upgrading to
1.14
Disable accelerator metrics
The kubelet collects accelerator metrics through cAdvisor. To collect these metrics, for
accelerators like NVIDIA GPUs, kubelet held an open handle on the driver. This meant that in order
to perform infrastructure changes (for example, updating the driver), a cluster administrator
needed to stop the kubelet agent.
The responsibility for collecting accelerator metrics now belongs to the vendor rather than the
kubelet. Vendors must provide a container that collects metrics and exposes them to the metrics
service (for example, Prometheus).
The
DisableAcceleratorUsageMetrics
disables metrics collected by the kubelet, with a
timeline for enabling this feature by default
.
Component metrics
kube-controller-manager metrics
Controller manager metrics provide important insight into the performance and health of the
controller manager. These metrics include common Go language runtime metrics such as go_routine
count and controller specific metrics such as etcd request latencies or Cloudprovider (AWS, GCE,
OpenStack) API latencies that can be used to gauge the health of a cluster.
Starting from Kubernetes 1.7, detailed Cloudprovider metrics are available for storage operations
for GCE, AWS, Vsphere and OpenStack.
These metrics can be used to monitor health of persistent volume operations.
For example, for GCE these metrics are called:
cloudprovider_gce_api_request_duration_seconds { request = "instance_list"}
cloudprovider_gce_api_request_duration_seconds { request = "disk_insert"}
cloudprovider_gce_api_request_duration_seconds { request = "disk_delete"}
cloudprovider_gce_api_request_duration_seconds { request = "attach_disk"}
cloudprovider_gce_api_request_duration_seconds { request = "detach_disk"}
cloudprovider_gce_api_request_duration_seconds { request = "list_disk"}
kube-scheduler metrics
Kubernetes v1.21 [beta]
The scheduler exposes optional metrics that reports the requested resources and the desired limits
of all running pods. These metrics can be used to build capacity planning dashboards, assess
current or historical scheduling limits, quickly identify workloads that cannot schedule due to
lack of resources, and compare actual usage to the pod's request.
The kube-scheduler identifies the resource
requests and limits
configured for each Pod; when either a request or limit is non-zero, the kube-scheduler reports a
metrics timeseries. The time series is labelled by:
namespace
pod name
the node where the pod is scheduled or an empty string if not yet scheduled
priority
the assigned scheduler for that pod
the name of the resource (for example,
cpu
)
the unit of the resource if known (for example,
cores
)
Once a pod reaches completion (has a
restartPolicy
of
Never
or
OnFailure
and is in the
Succeeded
or
Failed
pod phase, or has been deleted and all containers have a terminated state)
the series is no longer reported since the scheduler is now free to schedule other pods to run.
The two metrics are called
kube_pod_resource_request
and
kube_pod_resource_limit
.
The metrics are exposed at the HTTP endpoint
/metrics/resources
and require the same
authorization as the
/metrics
endpoint on the scheduler. You must use the
--show-hidden-metrics-for-version=1.20
flag to expose these alpha stability metrics.
Disabling metrics
You can explicitly turn off metrics via command line flag
--disabled-metrics
. This may be
desired if, for example, a metric is causing a performance pro

blem. The input is a list of
disabled metrics (i.e.
--disabled-metrics=metric1,metric2
).
Metric cardinality enforcement
Metrics with unbounded dimensions could cause memory issues in the components they instrument. To
limit resource use, you can use the
--allow-label-value
command line option to dynamically
configure an allow-list of label values for a metric.
In alpha stage, the flag can only take in a series of mappings as metric label allow-list.
Each mapping is of the format
&lt;metric_name&gt;,&lt;label_name&gt;=&lt;allowed_labels&gt;
where
&lt;allowed_labels&gt;
is a comma-separated list of acceptable label names.
The overall format looks like:
--allow-label-value &lt;metric_name&gt;,&lt;label_name&gt;='&lt;allow_value1&gt;, &lt;allow_value2&gt;...', &lt;metric_name2&gt;,&lt;label_name&gt;='&lt;allow_value1&gt;, &lt;allow_value2&gt;...', ...
Here is an example:
--allow-label-value number_count_metric,odd_number='1,3,5', number_count_metric,even_number='2,4,6', date_gauge_metric,weekend='Saturday,Sunday'
Read about the
Prometheus text format
for metrics
See the list of
stable Kubernetes metrics
Kubernetes deprecation policy
6 - System Logs
System component logs record events happening in cluster, which can be very useful for debugging.
You can configure log verbosity to see more or less detail.
Logs can be as coarse-grained as showing errors within a component, or as fine-grained as showing
step-by-step traces of events (like HTTP access logs, pod state changes, controller actions, or
scheduler decisions).
Klog
klog is the Kubernetes logging library.
klog
generates log messages for the Kubernetes system components.
For more information about klog configuration, see the
Command line tool reference
.
Kubernetes is in the process of simplifying logging in its components.
The following klog command line flags
are deprecated
starting with Kubernetes 1.23 and will be removed in a future release:
--add-dir-header
--alsologtostderr
--log-backtrace-at
--log-dir
--log-file
--log-file-max-size
--logtostderr
--one-output
--skip-headers
--skip-log-headers
--stderrthreshold
Output will always be written to stderr, regardless of the output format. Output redirection is
expected to be handled by the component which invokes a Kubernetes component. This can be a POSIX
shell or a tool like systemd.
In some cases, for example a distroless container or a Windows system service, those options are
not available. Then the
binary can be used as wrapper around a Kubernetes component to redirect
output. A prebuilt binary is included in several Kubernetes base images under
its traditional name as
/go-runner
and as
in server and
node release archives.
This table shows how
invocations correspond to shell redirection:
Usage
POSIX shell (such as bash)
kube-log-runner &lt;options&gt; &lt;cmd&gt;
Merge stderr and stdout, write to stdout
2&gt;&amp;1
(default behavior)
Redirect both into log file
1&gt;&gt;/tmp/log 2&gt;&amp;1
kube-log-runner -log-file=/tmp/log
Copy into log file and to stdout
2&gt;&amp;1 | tee -a /tmp/log
kube-log-runner -log-file=/tmp/log -also-stdout
Redirect only stdout into log file
&gt;/tmp/log
kube-log-runner -log-file=/tmp/log -redirect-stderr=false
Klog output
An example of the traditional klog native format:
I1025 00:15:15.525108       1 httplog.go:79] GET /api/v1/namespaces/kube-system/pods/metrics-server-v0.3.1-57c75779f-9p8wg: (1.512ms) 200 [pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format 10.56.1.19:51756]
The message string may contain line breaks:
I1025 00:15:15.525108       1 example.go:79] This is a message
which has a line break.
Structured Logging
Kubernetes v1.23 [beta]
Warning:
Migration to structured log messages is an ongoing process. Not all log messages are structured in
this version. When parsing log files, you must also handle unstructured log messages.
Log formatting and value serialization are subject to c

hange.
Structured logging introduces a uniform structure in log messages allowing for programmatic
extraction of information. You can store and process structured logs with less effort and cost.
The code which generates a log message determines whether it uses the traditional unstructured
klog output or structured logging.
The default formatting of structured log messages is as text, with a format that is backward
compatible with traditional klog:
&lt;klog header&gt; "&lt;message&gt;" &lt;key1&gt;
=
"&lt;value1&gt;" &lt;key2&gt;="&lt;value2&gt;" ...
Example:
I1025 00:15:15.525108       1 controller_utils.go:116] "Pod status updated" pod
=
"kube-system/kubedns" status="ready"
Strings are quoted. Other values are formatted with
%+v
, which may cause log messages to
continue on the next line
depending on the data
.
I1025 00:15:15.525108       1 example.go:116] "Example" data="This is text with a line break\nand \"quotation marks\"." someInt=1 someFloat=0.1 someStruct={StringField: First line,
second line.}
Contextual Logging
Kubernetes v1.24 [alpha]
Contextual logging builds on top of structured logging. It is primarily about
how developers use logging calls: code based on that concept is more flexible
and supports additional use cases as described in the
KEP
.
If developers use additional functions like
WithValues
or
WithName
in
their components, then log entries contain additional information that gets
passed into functions by their caller.
Currently this is gated behind the
StructuredLogging
feature gate and
disabled by default. The infrastructure for this was added in 1.24 without
modifying components. The
component-base/logs/example
command demonstrates how to use the new logging calls and how a component
behaves that supports contextual logging.
$
cd
$GOPATH
/src/k8s.io/kubernetes/staging/src/k8s.io/component-base/logs/example/cmd/
$
go run . --help
...
--feature-gates mapStringBool  A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
AllAlpha=true|false (ALPHA - default=false)
AllBeta=true|false (BETA - default=false)
ContextualLogging=true|false (ALPHA - default=false)
$
go run . --feature-gates
ContextualLogging
=
true
...
I0404 18:00:02.916429  451895 logger.go:94] "example/myname: runtime" foo="bar" duration="1m0s"
I0404 18:00:02.916447  451895 logger.go:95] "example: another runtime" foo="bar" duration="1m0s"
The
example
prefix and
foo="bar"
were added by the caller of the function
which logs the
runtime
message and
duration="1m0s"
value, without having to
modify that function.
With contextual logging disable,
and
WithName
do nothing and log
calls go through the global klog logger. Therefore this additional information
is not in the log output anymore:
$
=
false
...
I0404 18:03:31.171945  452150 logger.go:94] "runtime" duration="1m0s"
I0404 18:03:31.171962  452150 logger.go:95] "another runtime" duration="1m0s"
JSON log format
Kubernetes v1.19 [alpha]
Warning:
JSON output does not support many standard klog flags. For list of unsupported klog flags, see the
.
Not all logs are guaranteed to be written in JSON format (for example, during process start).
If you intend to parse logs, make sure you can handle log lines that are not JSON as well.
Field names and JSON serialization are subject to change.
The
--logging-format=json
flag changes the format of logs from klog native format to JSON format.
Example of JSON log format (pretty printed):
{
"ts"
:
1580306777.04728
,
"v"
:
4
,
"msg"
:
"Pod status updated"
,
"pod"
:{
"name"
:
"nginx-1"
,
"namespace"
:
"default"
},
"status"
:
"ready"
}
Keys with special meaning:
ts
- timestamp as Unix time (required, float)
v
- verbosity (only for info and not for error messages, int)
err
- error string (optional, string)
msg
- message (required, string)
List of components currently supporting JSON format:
kube-co

ntroller-manager
kubelet
Log verbosity level
The
-v
flag controls log verbosity. Increasing the value increases the number of logged events.
Decreasing the value decreases the number of logged events. Increasing verbosity settings logs
increasingly less severe events. A verbosity setting of 0 logs only critical events.
Log location
There are two types of system components: those that run in a container and those
that do not run in a container. For example:
The Kubernetes scheduler and kube-proxy run in a container.
The kubelet and
container runtime
do not run in containers.
On machines with systemd, the kubelet and container runtime write to journald.
Otherwise, they write to
.log
/var/log
directory.
System components inside containers always write to
.log
/var/log
directory,
bypassing the default logging mechanism.
Similar to the container logs, you should rotate system component logs in the
/var/log
In Kubernetes clusters created by the
kube-up.sh
script, log rotation is configured by the
logrotate
tool.
The
logrotate
tool rotates logs daily, or once the log size is greater than 100MB.
Kubernetes Logging Architecture
Read about
Read about
Read about
deprecation of klog flags
Conventions for logging severity
7 - Traces For Kubernetes System Components
System component traces record the latency of and relationships between operations in the cluster.
Kubernetes components emit traces using the
OpenTelemetry Protocol
with the gRPC exporter and can be collected and routed to tracing backends using an
OpenTelemetry Collector
.
Trace Collection
For a complete guide to collecting traces and using the collector, see
Getting Started with the OpenTelemetry Collector
.
However, there are a few things to note that are specific to Kubernetes components.
By default, Kubernetes components export traces using the grpc exporter for OTLP on the
IANA OpenTelemetry port
, 4317.
As an example, if the collector is running as a sidecar to a Kubernetes component,
the following receiver configuration will collect spans and log them to standard output:
receivers
:
otlp
:
protocols
:
grpc
:
exporters
:
# Replace this exporter with the exporter for your backend
logging
:
logLevel
:
debug
service
:
pipelines
:
traces
:
receivers
:
[otlp]
exporters
:
[logging]
Component traces
kube-apiserver traces
The kube-apiserver generates spans for incoming HTTP requests, and for outgoing requests
to webhooks, etcd, and re-entrant requests. It propagates the
W3C Trace Context
with outgoing requests
but does not make use of the trace context attached to incoming requests,
as the kube-apiserver is often a public endpoint.
Enabling tracing in the kube-apiserver
To enable tracing, enable the
APIServerTracing
on the kube-apiserver. Also, provide the kube-apiserver with a tracing configuration file
with
--tracing-config-file=&lt;path-to-config&gt;
. This is an example config that records
spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:
:
apiserver.config.k8s.io/v1alpha1
kind
:
# default value
#endpoint: localhost:4317
:
100
For more information about the
struct, see
API server config API (v1alpha1)
.
kubelet traces
Kubernetes v1.25 [alpha]
The kubelet CRI interface and authenticated http servers are instrumented to generate
trace spans. As with the apiserver, the endpoint and sampling rate are configurable.
Trace context propagation is also configured. A parent span's sampling decision is always respected.
A provided tracing configuration sampling rate will apply to spans without a parent.
Enabled without a configured endpoint, the default OpenTelemetry Collector receiver address of "localhost:4317" is set.
Enabling tracing in the kubelet
To enable tracing, ena

ble the
KubeletTracing
on the kubelet. Also, provide the kubelet with a
tracing configuration
.
This is an example snippet of a kubelet config that records spans for 1 in 10000 requests, and uses the default OpenTelemetry endpoint:
:
kind
:
:
:
true
tracing
:
:
100
Stability
Tracing instrumentation is still under active development, and may change
in a variety of ways. This includes span names, attached attributes,
instrumented endpoints, etc. Until this feature graduates to stable,
there are no guarantees of backwards compatibility for tracing instrumentation.
Read about
8 - Proxies in Kubernetes
This page explains proxies used with Kubernetes.
Proxies
There are several different proxies you may encounter when using Kubernetes:
The
kubectl proxy
:
runs on a user's desktop or in a pod
proxies from a localhost address to the Kubernetes apiserver
client to proxy uses HTTP
proxy to apiserver uses HTTPS
locates apiserver
adds authentication headers
The
apiserver proxy
:
is a bastion built into the apiserver
connects a user outside of the cluster to cluster IPs which otherwise might not be reachable
runs in the apiserver processes
client to proxy uses HTTPS (or http if apiserver so configured)
proxy to target may use HTTP or HTTPS as chosen by proxy using available information
can be used to reach a Node, Pod, or Service
does load balancing when used to reach a Service
The
kube proxy
:
runs on each node
proxies UDP, TCP and SCTP
does not understand HTTP
provides load balancing
is only used to reach services
A Proxy/Load-balancer in front of apiserver(s):
existence and implementation varies from cluster to cluster (e.g. nginx)
sits between all clients and one or more apiservers
acts as load balancer if there are several apiservers.
Cloud Load Balancers on external services:
are provided by some cloud providers (e.g. AWS ELB, Google Cloud Load Balancer)
are created automatically when the Kubernetes service has type
usually supports UDP/TCP only
SCTP support is up to the load balancer implementation of the cloud provider
implementation varies by cloud provider.
Kubernetes users will typically not need to worry about anything other than the first two types. The cluster admin
will typically ensure that the latter types are set up correctly.
Requesting redirects
Proxies have replaced redirect capabilities. Redirects have been deprecated.
9 - API Priority and Fairness
Kubernetes v1.20 [beta]
Controlling the behavior of the Kubernetes API server in an overload situation
is a key task for cluster administrators. The
has some controls available
(i.e. the
--max-requests-inflight
and
--max-mutating-requests-inflight
command-line flags) to limit the amount of outstanding work that will be
accepted, preventing a flood of inbound requests from overloading and
potentially crashing the API server, but these flags are not enough to ensure
that the most important requests get through in a period of high traffic.
The API Priority and Fairness feature (APF) is an alternative that improves upon
aforementioned max-inflight limitations. APF classifies
and isolates requests in a more fine-grained way. It also introduces
a limited amount of queuing, so that no requests are rejected in cases
of very brief bursts. Requests are dispatched from queues using a
fair queuing technique so that, for example, a poorly-behaved
controller
need not
starve others (even at the same priority level).
This feature is designed to work well with standard controllers, which
use informers and react to failures of API requests with exponential
back-off, and other clients that also work this way.
Caution:
Some requests classified as "long-running"—such as remote
command execution or log tailing—are not subject to 

the API
Priority and Fairness filter. This is also true for the
flag without the API Priority and Fairness
feature enabled. API Priority and Fairness
does
apply to
watch
requests. When API Priority and Fairness is disabled,
watch
requests
are not subject to the
limit.
Enabling/Disabling API Priority and Fairness
The API Priority and Fairness feature is controlled by a feature gate
and is enabled by default. See
Feature
Gates
for a general explanation of feature gates and how to enable and
disable them. The name of the feature gate for APF is
"APIPriorityAndFairness". This feature also involves an
API Group
with: (a) a
v1alpha1
version and a
v1beta1
version, disabled by default, and
(b)
v1beta2
and
v1beta3
versions, enabled by default. You can
disable the feature gate and API group beta versions by adding the
following command-line flags to your
invocation:
\
--feature-gates
=
APIPriorityAndFairness
=
false
\
--runtime-config
=
flowcontrol.apiserver.k8s.io/v1beta2
=
false,flowcontrol.apiserver.k8s.io/v1beta3
=
false
\
# …and other flags as usual
Alternatively, you can enable the v1alpha1 and v1beta1 versions of the API group
with
--runtime-config=flowcontrol.apiserver.k8s.io/v1alpha1=true,flowcontrol.apiserver.k8s.io/v1beta1=true
.
The command-line flag
--enable-priority-and-fairness=false
will disable the
API Priority and Fairness feature, even if other flags have enabled it.
Concepts
There are several distinct features involved in the API Priority and Fairness
feature. Incoming requests are classified by attributes of the request using
FlowSchemas
, and assigned to priority levels. Priority levels add a degree of
isolation by maintaining separate concurrency limits, so that requests assigned
to different priority levels cannot starve each other. Within a priority level,
a fair-queuing algorithm prevents requests from different
flows
from starving
each other, and allows for requests to be queued to prevent bursty traffic from
causing failed requests when the average load is acceptably low.
Priority Levels
Without APF enabled, overall concurrency in the API server is limited by the
flags
and
. With APF enabled, the concurrency limits
defined by these flags are summed and then the sum is divided up among a
configurable set of
priority levels
. Each incoming request is assigned to a
single priority level, and each priority level will only dispatch as many
concurrent requests as its particular limit allows.
The default configuration, for example, includes separate priority levels for
leader-election requests, requests from built-in controllers, and requests from
Pods. This means that an ill-behaved Pod that floods the API server with
requests cannot prevent leader election or actions by the built-in controllers
from succeeding.
The concurrency limits of the priority levels are periodically
adjusted, allowing under-utilized priority levels to temporarily lend
concurrency to heavily-utilized levels. These limits are based on
nominal limits and bounds on how much concurrency a priority level may
lend and how much it may borrow, all derived from the configuration
objects mentioned below.
Seats Occupied by a Request
The above description of concurrency management is the baseline story.
In it, requests have different durations but are counted equally at
any given moment when comparing against a priority level's concurrency
limit. In the baseline story, each request occupies one unit of
concurrency. The word "seat" is used to mean one unit of concurrency,
inspired by the way each passenger on a train or aircraft takes up one
of the fixed supply of seats.
But some requests take up more than one seat. Some of these are
list
requests that the server estimates will return a large number of
objects. These have been found to put an exceptionally heavy burden
on the server, among requests that tak

e a similar amount of time to
run. For this reason, the server estimates the number of objects that
will be returned and considers the request to take a number of seats
that is proportional to that estimated number.
Execution time tweaks for watch requests
API Priority and Fairness manages
watch
requests, but this involves a
couple more excursions from the baseline behavior. The first concerns
how long a
watch
request is considered to occupy its seat. Depending
on request parameters, the response to a
watch
request may or may not
begin with
create
notifications for all the relevant pre-existing
objects. API Priority and Fairness considers a
watch
request to be
done with its seat once that initial burst of notifications, if any,
is over.
The normal notifications are sent in a concurrent burst to all
relevant
watch
response streams whenever the server is notified of an
object create/update/delete. To account for this work, API Priority
and Fairness considers every write request to spend some additional
time occupying seats after the actual writing is done. The server
estimates the number of notifications to be sent and adjusts the write
request's number of seats and seat occupancy time to include this
extra work.
Queuing
Even within a priority level there may be a large number of distinct sources of
traffic. In an overload situation, it is valuable to prevent one stream of
requests from starving others (in particular, in the relatively common case of a
single buggy client flooding the kube-apiserver with requests, that buggy client
would ideally not have much measurable impact on other clients at all). This is
handled by use of a fair-queuing algorithm to process requests that are assigned
the same priority level. Each request is assigned to a
flow
, identified by the
name of the matching FlowSchema plus a
flow distinguisher
— which
is either the requesting user, the target resource's namespace, or nothing — and the
system attempts to give approximately equal weight to requests in different
flows of the same priority level.
To enable distinct handling of distinct instances, controllers that have
many instances should authenticate with distinct usernames
After classifying a request into a flow, the API Priority and Fairness
feature then may assign the request to a queue. This assignment uses
a technique known as
shuffle sharding
, which makes relatively efficient use of
queues to insulate low-intensity flows from high-intensity flows.
The details of the queuing algorithm are tunable for each priority level, and
allow administrators to trade off memory use, fairness (the property that
independent flows will all make progress when total traffic exceeds capacity),
tolerance for bursty traffic, and the added latency induced by queuing.
Exempt requests
Some requests are considered sufficiently important that they are not subject to
any of the limitations imposed by this feature. These exemptions prevent an
improperly-configured flow control configuration from totally disabling an API
server.
Resources
The flow control API involves two kinds of resources.
PriorityLevelConfigurations
define the available priority levels, the share of the available concurrency
budget that each can handle, and allow for fine-tuning queuing behavior.
are used to classify individual inbound requests, matching each to a
single PriorityLevelConfiguration. There is also a
v1alpha1
version
of the same API group, and it has the same Kinds with the same syntax and
semantics.
PriorityLevelConfiguration
A PriorityLevelConfiguration represents a single priority level. Each
PriorityLevelConfiguration has an independent limit on the number of outstanding
requests, and limitations on the number of queued requests.
The nominal oncurrency limit for a PriorityLevelConfiguration is not
specified in an absolute number of seats, but rather in "nominal
concurrency shares." The total concurrency limit for the API Server is
distributed among the existing PriorityLev

elConfigurations in
proportion to these shares, to give each level its nominal limit in
terms of seats. This allows a cluster administrator to scale up or
down the total amount of traffic to a server by restarting
with a different value for
(or
), and all
PriorityLevelConfigurations will see their maximum allowed concurrency
go up (or down) by the same fraction.
Caution:
In the versions before
v1beta3
the relevant
PriorityLevelConfiguration field is named "assured concurrency shares"
rather than "nominal concurrency shares". Also, in Kubernetes release
1.25 and earlier there were no periodic adjustments: the
nominal/assured limits were always applied without adjustment.
The bounds on how much concurrency a priority level may lend and how
much it may borrow are expressed in the PriorityLevelConfiguration as
percentages of the level's nominal limit. These are resolved to
absolute numbers of seats by multiplying with the nominal limit /
100.0 and rounding. The dynamically adjusted concurrency limit of a
priority level is constrained to lie between (a) a lower bound of its
nominal limit minus its lendable seats and (b) an upper bound of its
nominal limit plus the seats it may borrow. At each adjustment the
dynamic limits are derived by each priority level reclaiming any lent
seats for which demand recently appeared and then jointly fairly
responding to the recent seat demand on the priority levels, within
the bounds just described.
Caution:
With the Priority and Fairness feature enabled, the total concurrency limit for
the server is set to the sum of
and
. There is no longer any distinction made
between mutating and non-mutating requests; if you want to treat them
separately for a given resource, make separate FlowSchemas that match the
mutating and non-mutating verbs respectively.
When the volume of inbound requests assigned to a single
PriorityLevelConfiguration is more than its permitted concurrency level, the
type
field of its specification determines what will happen to extra requests.
A type of
Reject
means that excess traffic will immediately be rejected with
an HTTP 429 (Too Many Requests) error. A type of
Queue
means that requests
above the threshold will be queued, with the shuffle sharding and fair queuing techniques used
to balance progress between request flows.
The queuing configuration allows tuning the fair queuing algorithm for a
priority level. Details of the algorithm can be read in the
enhancement proposal
, but in short:
Increasing
queues
reduces the rate of collisions between different flows, at
the cost of increased memory usage. A value of 1 here effectively disables the
fair-queuing logic, but still allows requests to be queued.
queueLengthLimit
allows larger bursts of traffic to be
sustained without dropping any requests, at the cost of increased
latency and memory usage.
Changing
handSize
allows you to adjust the probability of collisions between
different flows and the overall concurrency available to a single flow in an
overload situation.
Note:
A larger
handSize
makes it less likely for two individual flows to collide
(and therefore for one to be able to starve the other), but more likely that
a small number of flows can dominate the apiserver. A larger
handSize
also
potentially increases the amount of latency that a single high-traffic flow
can cause. The maximum number of queued requests possible from a
single flow is
handSize * queueLengthLimit
.
Following is a table showing an interesting collection of shuffle
sharding configurations, showing for each the probability that a
given mouse (low-intensity flow) is squished by the elephants (high-intensity flows) for
an illustrative collection of numbers of elephants. See
https://play.golang.org/p/Gi0PLgVHiUg
, which computes this table.
Example Shuffle Sharding Configurations
HandSize
Queues
1 elephant
4 elephants

16 elephants
12
32
4.428838398950118e-09
0.11431348830099144
0.9935089607656024
10
32
1.550093439632541e-08
0.0626479840223545
0.9753101519027554
10
64
6.601827268370426e-12
0.00045571320990370776
0.49999929150089345
9
64
3.6310049976037345e-11
0.00045501212304112273
0.4282314876454858
8
64
2.25929199850899e-10
0.0004886697053040446
0.35935114681123076
8
128
6.994461389026097e-13
3.4055790161620863e-06
0.02746173137155063
7
128
1.0579122850901972e-11
6.960839379258192e-06
0.02406157386340147
7
256
7.597695465552631e-14
6.728547142019406e-08
0.0006709661542533682
6
256
2.7134626662687968e-12
2.9516464018476436e-07
0.0008895654642000348
6
512
4.116062922897309e-14
4.982983350480894e-09
2.26025764343413e-05
6
1024
6.337324016514285e-16
8.09060164312957e-11
4.517408062903668e-07
FlowSchema
A FlowSchema matches some inbound requests and assigns them to a
priority level. Every inbound request is tested against every
FlowSchema in turn, starting with those with numerically lowest ---
which we take to be the logically highest ---
matchingPrecedence
and
working onward. The first match wins.
Caution:
Only the first matching FlowSchema for a given request matters. If multiple
FlowSchemas match a single inbound request, it will be assigned based on the one
with the highest
. If multiple FlowSchemas with equal
match the same request, the one with lexicographically
smaller
name
will win, but it's better not to rely on this, and instead to
ensure that no two FlowSchemas have the same
.
A FlowSchema matches a given request if at least one of its
rules
matches. A rule matches if at least one of its
subjects
and
at least
one of its
resourceRules
or
nonResourceRules
(depending on whether the
incoming request is for a resource or non-resource URL) matches the request.
For the
name
field in subjects, and the
verbs
,
apiGroups
,
resources
,
namespaces
, and
fields of resource and non-resource rules,
the wildcard
*
may be specified to match all values for the given field,
effectively removing it from consideration.
A FlowSchema's
distinguisherMethod.type
determines how requests matching that
schema will be separated into flows. It may be
either
ByUser
, in which case one requesting user will not be able to starve
other users of capacity, or
ByNamespace
, in which case requests for resources
in one namespace will not be able to starve requests for resources in other
namespaces of capacity, or it may be blank (or
distinguisherMethod
may be
omitted entirely), in which case all requests matched by this FlowSchema will be
considered part of a single flow. The correct choice for a given FlowSchema
depends on the resource and your particular environment.
Defaults
Each kube-apiserver maintains two sorts of APF configuration objects:
mandatory and suggested.
Mandatory Configuration Objects
The four mandatory configuration objects reflect fixed built-in
guardrail behavior. This is behavior that the servers have before
those objects exist, and when those objects exist their specs reflect
this behavior. The four mandatory objects are as follows.
The mandatory
exempt
priority level is used for requests that are
not subject to flow control at all: they will always be dispatched
immediately. The mandatory
exempt
FlowSchema classifies all
requests from the
system:masters
group into this priority
level. You may define other FlowSchemas that direct other requests
to this priority level, if appropriate.
catch-all
priority level is used in combination with
the mandatory
catch-all
FlowSchema to make sure that every request
gets some kind of classification. Typically you should not rely on
this catch-all configuration, and should create your own catch-all
FlowSchema and PriorityLevelConfiguration (or use the suggested
global-default
priority level that is installed by default) as
appropriate. Because it is not expected to be used normally, the
mandatory
catch-all
priority level has a v

ery small concurrency
share and does not queue requests.
Suggested Configuration Objects
The suggested FlowSchemas and PriorityLevelConfigurations constitute a
reasonable default configuration. You can modify these and/or create
additional configuration objects if you want. If your cluster is
likely to experience heavy load then you should consider what
configuration will work best.
The suggested configuration groups requests into six priority levels:
The
node-high
priority level is for health updates from nodes.
The
system
priority level is for non-health requests from the
system:nodes
group, i.e. Kubelets, which must be able to contact
the API server in order for workloads to be able to schedule on
them.
The
leader-election
priority level is for leader election requests from
built-in controllers (in particular, requests for
endpoints
,
configmaps
,
or
leases
coming from the
system:kube-controller-manager
or
system:kube-scheduler
users and service accounts in the
kube-system
namespace). These are important to isolate from other traffic because failures
in leader election cause their controllers to fail and restart, which in turn
causes more expensive traffic as the new controllers sync their informers.
The
workload-high
priority level is for other requests from built-in
controllers.
The
workload-low
priority level is for requests from any other service
account, which will typically include all requests from controllers running in
Pods.
The
priority level handles all other traffic, e.g.
interactive
kubectl
commands run by nonprivileged users.
The suggested FlowSchemas serve to steer requests into the above
priority levels, and are not enumerated here.
Maintenance of the Mandatory and Suggested Configuration Objects
Each
independently maintains the mandatory and
suggested configuration objects, using initial and periodic behavior.
Thus, in a situation with a mixture of servers of different versions
there may be thrashing as long as different servers have different
opinions of the proper content of these objects.
Each
makes an initial maintenance pass over the
mandatory and suggested configuration objects, and after that does
periodic maintenance (once per minute) of those objects.
For the mandatory configuration objects, maintenance consists of
ensuring that the object exists and, if it does, has the proper spec.
The server refuses to allow a creation or update with a spec that is
inconsistent with the server's guardrail behavior.
Maintenance of suggested configuration objects is designed to allow
their specs to be overridden. Deletion, on the other hand, is not
respected: maintenance will restore the object. If you do not want a
suggested configuration object then you need to keep it around but set
its spec to have minimal consequences. Maintenance of suggested
objects is also designed to support automatic migration when a new
version of the
is rolled out, albeit potentially with
thrashing while there is a mixed population of servers.
Maintenance of a suggested configuration object consists of creating
it --- with the server's suggested spec --- if the object does not
exist. OTOH, if the object already exists, maintenance behavior
depends on whether the
kube-apiservers
or the users control the
object. In the former case, the server ensures that the object's spec
is what the server suggests; in the latter case, the spec is left
alone.
The question of who controls the object is answered by first looking
for an annotation with key
apf.kubernetes.io/autoupdate-spec
. If
there is such an annotation and its value is
true
then the
kube-apiservers control the object. If there is such an annotation
and its value is
false
then the users control the object. If
neither of those condtions holds then the
metadata.generation
of the
object is consulted. If that is 1 then the kube-apiservers control
the object. Otherwise the users control the object. These rules were
introduced in rel

ease 1.22 and their consideration of
is for the sake of migration from the simpler
earlier behavior. Users who wish to control a suggested configuration
object should set its
annotation
to
false
.
Maintenance of a mandatory or suggested configuration object also
includes ensuring that it has an
annotation that accurately reflects whether the kube-apiservers
control the object.
Maintenance also includes deleting objects that are neither mandatory
nor suggested but are annotated
apf.kubernetes.io/autoupdate-spec=true
.
Health check concurrency exemption
The suggested configuration gives no special treatment to the health
check requests on kube-apiservers from their local kubelets --- which
tend to use the secured port but supply no credentials. With the
suggested config, these requests get assigned to the
FlowSchema and the corresponding
priority level,
where other traffic can crowd them out.
If you add the following additional FlowSchema, this exempts those
requests from rate limiting.
Caution:
Making this change also allows any hostile party to then send
health-check requests that match this FlowSchema, at any volume they
like. If you have a web traffic filter or similar external security
mechanism to protect your cluster's API server from general internet
traffic, you can configure rules to block any health check requests
that originate from outside your cluster.
priority-and-fairness/health-for-strangers.yaml
:
kind
:
metadata
:
name
:
health-for-strangers
spec
:
:
1000
priorityLevelConfiguration
:
name
:
exempt
rules
:
-
:
-
:
-
"/healthz"
-
"/livez"
-
"/readyz"
verbs
:
-
"*"
subjects
:
-
kind
:
Group
group
:
name
:
system:unauthenticated
Diagnostics
Every HTTP response from an API server with the priority and fairness feature
enabled has two extra headers:
X-Kubernetes-PF-FlowSchema-UID
and
X-Kubernetes-PF-PriorityLevel-UID
, noting the flow schema that matched the request
and the priority level to which it was assigned, respectively. The API objects'
names are not included in these headers in case the requesting user does not
have permission to view them, so when debugging you can use a command like
kubectl get flowschemas -o custom-columns
=
"uid:{metadata.uid},name:{metadata.name}"
kubectl get prioritylevelconfigurations -o custom-columns
=
to get a mapping of UIDs to names for both FlowSchemas and
PriorityLevelConfigurations.
Observability
Metrics
Note:
In versions of Kubernetes before v1.20, the labels
flow_schema
and
priority_level
were inconsistently named
flowSchema
and
priorityLevel
,
respectively. If you're running Kubernetes versions v1.19 and earlier, you
should refer to the documentation for your version.
When you enable the API Priority and Fairness feature, the kube-apiserver
exports additional metrics. Monitoring these can help you determine whether your
configuration is inappropriately throttling important traffic, or find
poorly-behaved workloads that may be harming system health.
apiserver_flowcontrol_rejected_requests_total
is a counter vector
(cumulative since server start) of requests that were rejected,
broken down by the labels
(indicating the one that
matched the request),
(indicating the one to which
the request was assigned), and
reason
. The
reason
label will be
have one of the following values:
queue-full
, indicating that too many requests were already
queued,
concurrency-limit
, indicating that the
PriorityLevelConfiguration is configured to reject rather than
queue excess requests, or
time-out
, indicating that the request was still in the queue
when its queuing time limit expired.
apiserver_flowcontrol_dispatched_requests_total
is a counter
vector (cumulative since server start) of requests that be

gan
executing, broken down by the labels
(indicating the
one that matched the request) and
one to which the request was assigned).
apiserver_current_inqueue_requests
is a gauge vector of recent
high water marks of the number of queued requests, grouped by a
label named
request_kind
whose value is
mutating
or
readOnly
.
These high water marks describe the largest number seen in the one
second window most recently completed. These complement the older
apiserver_current_inflight_requests
gauge vector that holds the
last window's high water mark of number of requests actively being
served.
apiserver_flowcontrol_read_vs_write_current_requests
is a
histogram vector of observations, made at the end of every
nanosecond, of the number of requests broken down by the labels
phase
(which takes on the values
waiting
and
executing
) and
mutating
and
readOnly
). Each observed value is a ratio, between 0 and 1, of a
number of requests divided by the corresponding limit on the number
of requests (queue volume limit for waiting and concurrency limit
for executing).
apiserver_flowcontrol_current_inqueue_requests
is a gauge vector
holding the instantaneous number of queued (not executing) requests,
and
.
apiserver_flowcontrol_current_executing_requests
holding the instantaneous number of executing (not waiting in a
queue) requests, broken down by the labels
and
.
apiserver_flowcontrol_request_concurrency_in_use
holding the instantaneous number of occupied seats, broken down by
the labels
and
.
apiserver_flowcontrol_priority_level_request_utilization
is a
histogram vector of observations, made at the end of each
phase
waiting
and
executing
) and
. Each observed value is a ratio, between 0 and 1,
of a number of requests divided by the corresponding limit on the
number of requests (queue volume limit for waiting and concurrency
limit for executing).
apiserver_flowcontrol_priority_level_seat_utilization
is a
nanosecond, of the utilization of a priority level's concurrency
limit, broken down by
. This utilization is the
fraction (number of seats occupied) / (concurrency limit). This
metric considers all stages of execution (both normal and the extra
delay at the end of a write to cover for the corresponding
notification work) of all requests except WATCHes; for those it
considers only the initial stage that delivers notifications of
pre-existing objects. Each histogram in the vector is also labeled
with
phase: executing
(there is no seat limit for the waiting
phase).
apiserver_flowcontrol_request_queue_length_after_enqueue
is a
histogram vector of queue lengths for the queues, broken down by
the labels
and
, as sampled by the
enqueued requests. Each request that gets queued contributes one
sample to its histogram, reporting the length of the queue immediately
after the request was added. Note that this produces different
statistics than an unbiased survey would.
Note:
An outlier value in a histogram here means it is likely that a single flow
(i.e., requests by one user or for one namespace, depending on
configuration) is flooding the API server, and being throttled. By contrast,
if one priority level's histogram shows that all queues for that priority
level are longer than those for other priority levels, it may be appropriate
to increase that PriorityLevelConfiguration's concurrency shares.
apiserver_flowcontrol_request_concurrency_limit
is the same as
apiserver_flowcontrol_nominal_limit_seats
. Before the
introduction of concurrency borrowing between priority levels, this
was always equal to
apiserver_flowcontrol_current_limi

t_seats
(which did not exist as a distinct metric).
holding each priority level's nominal concurrency limit, computed
from the API server's total concurrency limit and the priority
level's configured nominal concurrency shares.
apiserver_flowcontrol_lower_limit_seats
is a gauge vector holding
the lower bound on each priority level's dynamic concurrency limit.
apiserver_flowcontrol_upper_limit_seats
the upper bound on each priority level's dynamic concurrency limit.
apiserver_flowcontrol_demand_seats
is a histogram vector counting
observations, at the end of every nanosecond, of each priority
level's ratio of (seat demand) / (nominal concurrency limit). A
priority level's seat demand is the sum, over both queued requests
and those in the initial phase of execution, of the maximum of the
number of seats occupied in the request's initial and final
execution phases.
apiserver_flowcontrol_demand_seats_high_watermark
holding, for each priority level, the maximum seat demand seen
during the last concurrency borrowing adjustment period.
apiserver_flowcontrol_demand_seats_average
holding, for each priority level, the time-weighted average seat
demand seen during the last concurrency borrowing adjustment period.
apiserver_flowcontrol_demand_seats_stdev
holding, for each priority level, the time-weighted population
standard deviation of seat demand seen during the last concurrency
borrowing adjustment period.
apiserver_flowcontrol_target_seats
is a gauge vector holding, for
each priority level, the concurrency target going into the borrowing
allocation problem.
apiserver_flowcontrol_seat_fair_frac
is a gauge holding the fair
allocation fraction determined in the last borrowing adjustment.
apiserver_flowcontrol_current_limit_seats
holding, for each priority level, the dynamic concurrency limit
derived in the last adjustment.
apiserver_flowcontrol_request_wait_duration_seconds
is a histogram
vector of how long requests spent queued, broken down by the labels
(indicating which one matched the request),
(indicating the one to which the request was
assigned), and
execute
(indicating whether the request started
executing).
Note:
Since each FlowSchema always assigns requests to a single
PriorityLevelConfiguration, you can add the histograms for all the
FlowSchemas for one priority level to get the effective histogram for
requests assigned to that priority level.
apiserver_flowcontrol_request_execution_seconds
vector of how long requests took to actually execute, broken down by
the labels
(indicating which one matched the request)
and
assigned).
apiserver_flowcontrol_watch_count_samples
is a histogram vector of
the number of active WATCH requests relevant to a given write,
broken down by
and
.
apiserver_flowcontrol_work_estimated_seats
is a histogram vector
of the number of estimated seats (maximum of initial and final stage
of execution) associated with requests, broken down by
and
.
apiserver_flowcontrol_request_dispatch_no_accommodation_total
is a
counter vec of the number of events that in principle could have led
to a request being dispatched but did not, due to lack of available
concurrency, broken down by
and
. The
relevant sorts of events are arrival of a request and completion of
a request.
Debug endpoints
When you enable the API Priority and Fairness feature, the
serves the following additional paths at its HTTP[S] ports.
/debug/api_priority_and_fairness/dump_priority_levels
- a listing of
all the priority levels and the current state of each. You can fetch like this:
kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
Prior

ityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests,
workload-low,      0,            true,   false,       0,               0,
global-default,    0,            true,   false,       0,               0,
exempt,            &lt;none&gt;,       &lt;none&gt;, &lt;none&gt;,      &lt;none&gt;,          &lt;none&gt;,
catch-all,         0,            true,   false,       0,               0,
system,            0,            true,   false,       0,               0,
leader-election,   0,            true,   false,       0,               0,
workload-high,     0,            true,   false,       0,               0,
/debug/api_priority_and_fairness/dump_queues
- a listing of all the
queues and their current state. You can fetch like this:
kubectl get --raw /debug/api_priority_and_fairness/dump_queues
PriorityLevelName, Index,  PendingRequests, ExecutingRequests, VirtualStart,
workload-high,     0,      0,               0,                 0.0000,
workload-high,     1,      0,               0,                 0.0000,
workload-high,     2,      0,               0,                 0.0000,
...
leader-election,   14,     0,               0,                 0.0000,
leader-election,   15,     0,               0,                 0.0000,
/debug/api_priority_and_fairness/dump_requests
- a listing of all the requests
that are currently waiting in a queue. You can fetch like this:
kubectl get --raw /debug/api_priority_and_fairness/dump_requests
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher,       ArriveTime,
exempt,            &lt;none&gt;,         &lt;none&gt;,     &lt;none&gt;,              &lt;none&gt;,                &lt;none&gt;,
system,            system-nodes,   12,         0,                   system:node:127.0.0.1, 2020-07-23T15:26:57.179170694Z,
In addition to the queued requests, the output includes one phantom line
for each priority level that is exempt from limitation.
You can get a more detailed listing with a command like this:
'/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1'
PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher,       ArriveTime,                     UserName,              Verb,   APIPath,                                                     Namespace, Name,   APIVersion, Resource, SubResource,
system,            system-nodes,   12,         0,                   system:node:127.0.0.1, 2020-07-23T15:31:03.583823404Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
system,            system-nodes,   12,         1,                   system:node:127.0.0.1, 2020-07-23T15:31:03.594555947Z, system:node:127.0.0.1, create, /api/v1/namespaces/scaletest/configmaps,
For background information on design details for API priority and fairness, see
the
.
You can make suggestions and feature requests via
SIG API Machinery
or the feature's
slack channel
.
10 - Installing Addons
Note:
Add-ons extend the functionality of Kubernetes.
This page lists some of the available add-ons and links to their respective installation instructions. The list does not try to be exhaustive.
Networking and Network Policy
ACI
provides integrated container networking and network security with Cisco ACI.
Antrea
operates at Layer 3/4 to provide networking and security services for Kubernetes, leveraging Open vSwitch as the networking data plane.
Calico
is a networking and network policy provider. Calico supports a flexible set of networking options so you can choose the most efficient option 

for your situation, including non-overlay and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts, pods, and (if using Istio &amp; Envoy) applications at the service mesh layer.
Canal
unites Flannel and Calico, providing networking and network policy.
Cilium
is a networking, observability, and security solution with an eBPF-based data plane. Cilium provides a simple flat Layer 3 network with the ability to span multiple clusters in either a native routing or overlay/encapsulation mode, and can enforce network policies on L3-L7 using an identity-based security model that is decoupled from network addressing. Cilium can act as a replacement for kube-proxy; it also offers additional, opt-in observability and security features.
CNI-Genie
enables Kubernetes to seamlessly connect to a choice of CNI plugins, such as Calico, Canal, Flannel, or Weave.
Contiv
provides configurable networking (native L3 using BGP, overlay using vxlan, classic L2, and Cisco-SDN/ACI) for various use cases and a rich policy framework. Contiv project is fully
open sourced
. The
installer
provides both kubeadm and non-kubeadm based installation options.
Contrail
, based on
Tungsten Fabric
, is an open source, multi-cloud network virtualization and policy management platform. Contrail and Tungsten Fabric are integrated with orchestration systems such as Kubernetes, OpenShift, OpenStack and Mesos, and provide isolation modes for virtual machines, containers/pods and bare metal workloads.
Flannel
is an overlay network provider that can be used with Kubernetes.
Knitter
is a plugin to support multiple network interfaces in a Kubernetes pod.
Multus
is a Multi plugin for multiple network support in Kubernetes to support all CNI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
OVN-Kubernetes
is a networking provider for Kubernetes based on
OVN (Open Virtual Network)
, a virtual networking implementation that came out of the Open vSwitch (OVS) project. OVN-Kubernetes provides an overlay based networking implementation for Kubernetes, including an OVS based implementation of load balancing and network policy.
Nodus
is an OVN based CNI controller plugin to provide cloud native based Service function chaining(SFC).
NSX-T
Container Plug-in (NCP) provides integration between VMware NSX-T and container orchestrators such as Kubernetes, as well as integration between NSX-T and container-based CaaS/PaaS platforms such as Pivotal Container Service (PKS) and OpenShift.
Nuage
is an SDN platform that provides policy-based networking between Kubernetes Pods and non-Kubernetes environments with visibility and security monitoring.
Romana
is a Layer 3 networking solution for pod networks that also supports the
API.
Weave Net
provides networking and network policy, will carry on working on both sides of a network partition, and does not require an external database.
Service Discovery
CoreDNS
is a flexible, extensible DNS server which can be
installed
as the in-cluster DNS for pods.
Visualization &amp; Control
Dashboard
is a dashboard web interface for Kubernetes.
Weave Scope
is a tool for graphically visualizing your containers, pods, services etc. Use it in conjunction with a
Weave Cloud account
or host the UI yourself.
Infrastructure
KubeVirt
is an add-on to run virtual machines on Kubernetes. Usually run on bare-metal clusters.
The
node problem detector
runs on Linux nodes and reports system issues as either
Events
or
Node conditions
.
Legacy Add-ons
There are several other add-ons documented in the deprecated
cluster/addons
Well-maintained ones should be linked to here. PRs welcome!
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks.

 For a list of trademarks of The Linux Foundation, please see our

APIService | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

and Set Up kubectl on Windows
Debug Pods
Troubleshooting Cl

usters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resources with A

ppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configuratio

n (v1alpha1)
Reference
APIService represents a server for a particular GroupVersion.
apiVersion: apiregistration.k8s.io/v1
import "k8s.io/kube-aggregator/pkg/apis/apiregistration/v1"
APIService represents a server for a particular GroupVersion. Name must be "version.group".
: apiregistration.k8s.io/v1
kind
: APIService
metadata
(
)
spec
(
APIServiceSpec
)
Spec contains information for locating and communicating with a server
status
(
APIServiceStatus
)
Status contains derived information about an API server
APIServiceSpec contains information for locating and communicating with a server. Only https is supported, though you are able to disable certificate verification.
groupPriorityMinimum
GroupPriorityMininum is the priority this group should have at least. Higher priority means that the group is preferred by clients over lower priority ones. Note that other versions of this group might specify even higher GroupPriorityMininum values such that the whole group gets a higher priority. The primary sort is based on GroupPriorityMinimum, ordered highest number to lowest (20 before 10). The secondary sort is based on the alphabetical comparison of the name of the object. (v1.bar before v1.foo) We'd recommend something like: *.k8s.io (except extensions) at 18000 and PaaSes (OpenShift, Deis) are recommended to be in the 2000s
versionPriority
VersionPriority controls the ordering of this API version inside of its group. Must be greater than zero. The primary sort is based on VersionPriority, ordered highest to lowest (20 before 10). Since it's inside of a group, the number can be small, probably in the 10s. In case of equal version priorities, the version string will be used to compute the order inside a group. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10.
caBundle
([]byte)
Atomic: will be replaced during a merge
CABundl

e is a PEM encoded CA bundle which will be used to validate an API server's serving certificate. If unspecified, system trust roots on the apiserver are used.
group
(string)
Group is the API group name this server hosts
insecureSkipTLSVerify
(boolean)
InsecureSkipTLSVerify disables TLS certificate verification when communicating with this server. This is strongly discouraged. You should use the CABundle instead.
service
(ServiceReference)
Service is a reference to the service for this API server. It must communicate on port 443. If the Service is nil, that means the handling for the API groupversion is handled locally on this server. The call will simply delegate to the normal handler chain to be fulfilled.
ServiceReference holds a reference to Service.legacy.k8s.io
service.name
(string)
Name is the name of the service
service.namespace
(string)
Namespace is the namespace of the service
service.port
(int32)
If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility.
port
should be a valid port number (1-65535, inclusive).
version
(string)
Version is the API version this server hosts. For example, "v1"
APIServiceStatus contains derived information about an API server
([]APIServiceCondition)
type
Map: unique values on key type will be kept during a merge
Current service state of apiService.
APIServiceCondition describes the state of an APIService at a particular point
Status is the status of the condition. Can be True, False, Unknown.
Type is the type of the condition.
(Time)
(string)
Human-readable message indicating details about last transition.
(string)
Unique, one-word, CamelCase reason for the condition's last transition.
APIServiceList
APIServiceList is a list of APIService objects.
kind
: APIServiceList
metadata
(
ListMeta
)
Standard list metadata More info:
items
([]
Items is the list of APIService
get
read the specified APIService
GET /apis/apiregistration.k8s.io/v1/apiservices/{name}
name
(
in path
name of the APIService
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified APIService
GET /apis/apiregistration.k8s.io/v1/apiservices/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind APIService
GET /apis/apiregistration.k8s.io/v1/apiservices
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create an APIService
POST /apis/apiregistration.k8s.io/v1/apiservices
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
):

 Created
202 (
update
replace the specified APIService
PUT /apis/apiregistration.k8s.io/v1/apiservices/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified APIService
PUT /apis/apiregistration.k8s.io/v1/apiservices/{name}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified APIService
PATCH /apis/apiregistration.k8s.io/v1/apiservices/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified APIService
PATCH /apis/apiregistration.k8s.io/v1/apiservices/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete an APIService
DELETE /apis/apiregistration.k8s.io/v1/apiservices/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of APIService
DELETE /apis/apiregistration.k8s.io/v1/apiservices
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Last modified December 08, 2021 at 9:34 AM PST:
Reference docs for kubernetes 1.23 (02445930e5)
get
get
list
create
upd

ate
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Security | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Security
1:
2:
3:
4:
1 - Apply Pod Security Standards at the Cluster Level
Note
This tutorial applies only for new clusters.
Pod Security admission (PSA) is enabled by default in v1.23 and later, as it has
graduated to beta
.
Pod Security
is an admission controller that carries out checks against the Kubernetes
when new pods are
created. This tutorial shows you how to enforce the
baseline
Standard at the cluster level which applies a standard configuration
to all namespaces in a cluster.
To apply Pod Security Standards to specific namespaces, refer to
Apply Pod Security Standards at the namespace level
.
If you are running a version of Kubernetes other than v1.26,
check the documentation for that version.
Install the following on your workstation:
KinD
kubectl
Choose the right Pod Security Standard to apply
lets you apply built-in
with the following modes:
enforce
,
audit
, and
warn
.
To gather information that helps you to choose the Pod Security Standards
that are most appropriate for your configuration, do the following:
Create a cluster with no Pod Security Standards applied:
kind create cluster --name psa-wo-cluster-pss --image kindest/node:v1.24.0
Creating cluster "psa-wo-cluster-pss" ...
✓ Ensuring node image (kindest/node:v1.24.0) 🖼
✓ Preparing nodes 📦  
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
Set kubectl context to "kind-psa-wo-cluster-pss"
You can now use your cluster with:

kubectl cluster-info --context kind-psa-wo-cluster-pss

Thanks for using kind! 😊
Set the kubectl context to the new cluster:
Kubernetes control plane is running at https://127.0.0.1:61350

CoreDNS is running at https://127.0.0.1:61350/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Get a list of namespaces in the cluster:
kubectl get ns
NAME                 STATUS   AGE
default              Active   9m30s
kube-node-lease      Active   9m32s
kube-public          Active   9m32s
kube-system          Active   9m32s
local-path-storage   Active   9m26s
Use
--dry-run=server
to understand what happens when different Pod Security Standards
are applied:
Privileged
kubectl label --dry-run
=
server --overwrite ns --all
\
pod-security.kubernetes.io/enforce
=
privileged
namespace/default labeled
namespace/kube-node-lease labeled
namespace/kube-public labeled
namespace/kube-system labeled
namespace/local-path-storage labeled
Baseline
=
\
=
baseline
Warning: existing pods in namespace "kube-system" violate the new PodSecurity enforce level "baseline:latest"
Warning: etcd-psa-wo-cluster-pss-control-plane (and 3 other pods): host namespaces, hostPath volumes
Warning: kindnet-vzj42: non-default capabilities, host namespaces, hostPath volumes
Warning: kube-proxy-m6hwf: host namespaces, hostPath volumes, privileged
Restricted
=
server -

-overwrite ns --all
\
=
restricted
Warning: existing pods in namespace "kube-system" violate the new PodSecurity enforce level "restricted:latest"
Warning: coredns-7bb9c7b568-hsptc (and 1 other pod): unrestricted capabilities, runAsNonRoot != true, seccompProfile
Warning: etcd-psa-wo-cluster-pss-control-plane (and 3 other pods): host namespaces, hostPath volumes, allowPrivilegeEscalation != false, unrestricted capabilities, restricted volume types, runAsNonRoot != true
Warning: kindnet-vzj42: non-default capabilities, host namespaces, hostPath volumes, allowPrivilegeEscalation != false, unrestricted capabilities, restricted volume types, runAsNonRoot != true, seccompProfile
Warning: kube-proxy-m6hwf: host namespaces, hostPath volumes, privileged, allowPrivilegeEscalation != false, unrestricted capabilities, restricted volume types, runAsNonRoot != true, seccompProfile
Warning: existing pods in namespace "local-path-storage" violate the new PodSecurity enforce level "restricted:latest"
Warning: local-path-provisioner-d6d9f7ffc-lw9lh: allowPrivilegeEscalation != false, unrestricted capabilities, runAsNonRoot != true, seccompProfile
From the previous output, you'll notice that applying the
Pod Security Standard shows no warnings
for any namespaces. However,
baseline
and
standards both have
warnings, specifically in the
Set modes, versions and standards
In this section, you apply the following Pod Security Standards to the
latest
version:
baseline
standard in
enforce
mode.
warn
and
audit
mode.
The
baseline
Pod Security Standard provides a convenient
middle ground that allows keeping the exemption list short and prevents known
privilege escalations.
Additionally, to prevent pods from failing in
, you'll exempt the namespace
from having Pod Security Standards applied.
When you implement Pod Security Admission in your own environment, consider the
following:
Based on the risk posture applied to a cluster, a stricter Pod Security
Standard like
might be a better choice.
Exempting the
namespace allows pods to run as
in this namespace. For real world use, the Kubernetes project
strongly recommends that you apply strict RBAC
policies that limit access to
, following the principle of least
privilege.
To implement the preceding standards, do the following:
Create a configuration file that can be consumed by the Pod Security
Admission Controller to implement these Pod Security Standards:
mkdir -p /tmp/pss
cat &lt;&lt;EOF &gt; /tmp/pss/cluster-level-pss.yaml 
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "baseline"
      enforce-version: "latest"
      audit: "restricted"
      audit-version: "latest"
      warn: "restricted"
      warn-version: "latest"
    exemptions:
      usernames: []
      runtimeClasses: []
      namespaces: [kube-system]
EOF
Note:
pod-security.admission.config.k8s.io/v1
configuration requires v1.25+.
For v1.23 and v1.24, use
v1beta1
.
For v1.22, use
v1alpha1
.
Configure the API server to consume this file during cluster creation:
cat &lt;&lt;EOF &gt; /tmp/pss/cluster-config.yaml 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        extraArgs:
          admission-control-config-file: /etc/config/cluster-level-pss.yaml
        extraVolumes:
          - name: accf
            hostPath: /etc/config
            mountPath: /etc/config
            readOnly: false
            pathType: "Direc

toryOrCreate"
  extraMounts:
  - hostPath: /tmp/pss
    containerPath: /etc/config
    # optional: if set, the mount is read-only.
    # default false
    readOnly: false
    # optional: if set, the mount needs SELinux relabeling.
    selinuxRelabel: false
    # optional: set propagation mode (None, HostToContainer or Bidirectional)
    # see https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation
    # default None
    propagation: None
EOF
Note:
If you use Docker Desktop with KinD on macOS, you can
add
/tmp
as a Shared Directory under the menu item
Preferences &gt; Resources &gt; File Sharing
.
Create a cluster that uses Pod Security Admission to apply
these Pod Security Standards:
kind create cluster --name psa-with-cluster-pss --image kindest/node:v1.24.0 --config /tmp/pss/cluster-config.yaml
Creating cluster "psa-with-cluster-pss" ...
  ✓ Ensuring node image (kindest/node:v1.24.0) 🖼 
  ✓ Preparing nodes 📦  
  ✓ Writing configuration 📜 
  ✓ Starting control-plane 🕹️ 
  ✓ Installing CNI 🔌 
  ✓ Installing StorageClass 💾 
 Set kubectl context to "kind-psa-with-cluster-pss"
 You can now use your cluster with:

 kubectl cluster-info --context kind-psa-with-cluster-pss

 Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
Point kubectl to the cluster
kubectl cluster-info --context kind-psa-with-cluster-pss
Kubernetes control plane is running at https://127.0.0.1:63855
 CoreDNS is running at https://127.0.0.1:63855/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

 To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Create the following Pod specification for a minimal configuration in the default namespace:
cat &lt;&lt;EOF &gt; /tmp/pss/nginx-pod.yaml
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - image: nginx
      name: nginx
      ports:
        - containerPort: 80
EOF
Create the Pod in the cluster:
kubectl apply -f /tmp/pss/nginx-pod.yaml
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
 pod/nginx created
Clean up
Run
kind delete cluster --name psa-with-cluster-pss
and
kind delete cluster --name psa-wo-cluster-pss
to delete the clusters you
created.
Run a
shell script
to perform all the preceding steps at once:
Create a Pod Security Standards based cluster level Configuration
Create a file to let API server consume this configuration
Create a cluster that creates an API server with this configuration
Set kubectl context to this new cluster
Create a minimal pod yaml file
Apply this file to create a Pod in the new cluster
2 - Apply Pod Security Standards at the Namespace Level
Note
Pod Security admission (PSA) is enabled by default in v1.23 and later, as it
. Pod Security Admission
is an admission controller that applies
when pods are created. In this tutorial, you will enforce the
baseline
Pod Security Standard,
one namespace at a time.
You can also apply Pod Security Standards to multiple namespaces at once at the cluster
level. For instructions, refer to
Apply Pod Security Standards at the cluster level
.
KinD
kubectl
Create cluster
Create a
KinD
cluster as follows:
kind create cluster --name psa-ns-level --image kindest/

node:v1.23.0
Creating cluster "psa-ns-level" ...
 ✓ Ensuring node image (kindest/node:v1.23.0) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-psa-ns-level"

kubectl cluster-info --context kind-psa-ns-level

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
Kubernetes control plane is running at https://127.0.0.1:50996
CoreDNS is running at https://127.0.0.1:50996/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Create a namespace
Create a new namespace called
example
:
kubectl create ns example
namespace/example created
Apply Pod Security Standards
Enable Pod Security Standards on this namespace using labels supported by
built-in Pod Security Admission. In this step we will warn on baseline pod
security standard as per the latest version (default value)
kubectl label --overwrite ns example
\
pod-security.kubernetes.io/warn
=
baseline
\
pod-security.kubernetes.io/warn-version
=
latest
Multiple pod security standards can be enabled on any namespace, using labels.
Following command will
enforce
the
baseline
Pod Security Standard, but
warn
and
audit
for
Pod Security Standards as per the latest
version (default value)
\
=
baseline
\
pod-security.kubernetes.io/enforce-version
=
latest
\
=
\
=
latest
\
pod-security.kubernetes.io/audit
=
\
pod-security.kubernetes.io/audit-version
=
latest
Verify the Pod Security Standards
Create a minimal pod in
example
namespace:
cat
&lt;&lt;EOF &gt; /tmp/pss/nginx-pod.yaml
kind: Pod
metadata:
name: nginx
spec:
containers:
- image: nginx
ports:
- containerPort: 80
EOF
Apply the pod spec to the cluster in
example
kubectl apply -n example -f /tmp/pss/nginx-pod.yaml
pod/nginx created
default
kubectl apply -n default -f /tmp/pss/nginx-pod.yaml
Output is similar to this:
The Pod Security Standards were applied only to the
example
namespace. You could create the same Pod in the
default
namespace
with no warnings.
Clean up
Run
kind delete cluster --name psa-ns-level
to delete the cluster created.
Run a
to perform all the preceding steps all at once.
Create KinD cluster
Create new namespace
Apply
baseline
Pod Security Standard in
enforce
mode while applying
Pod Security Standard also in
warn
and
audit
mode.
Create a new pod with the following pod security standards applied
3 - Restrict a Container's Access to Resources with AppArmor
Kubernetes v1.4 [beta]
AppArmor is a Linux kernel security module that supplements the standard Linux user and group based
permissions to confine programs to a limited set of resources. AppArmor can be configured for any
application to reduce its potential attack surface and provide greater in-depth def

ense. It is
configured through profiles tuned to allow the access needed by a specific program or container,
such as Linux capabilities, network access, file permissions, etc. Each profile can be run in either
enforcing
mode, which blocks access to disallowed resources, or
complain
mode, which only reports
violations.
AppArmor can help you to run a more secure deployment by restricting what containers are allowed to
do, and/or provide better auditing through system logs. However, it is important to keep in mind
that AppArmor is not a silver bullet and can only do so much to protect against exploits in your
application code. It is important to provide good, restrictive profiles, and harden your
applications and cluster from other angles as well.
Objectives
See an example of how to load a profile on a node
Learn how to enforce the profile on a Pod
Learn how to check that the profile is loaded
See what happens when a profile is violated
See what happens when a profile cannot be loaded
Make sure:
Kubernetes version is at least v1.4 -- Kubernetes support for AppArmor was added in
v1.4. Kubernetes components older than v1.4 are not aware of the new AppArmor annotations, and
will
silently ignore
any AppArmor settings that are provided. To ensure that your Pods are
receiving the expected protections, it is important to verify the Kubelet version of your nodes:
kubectl get nodes -o
=
jsonpath
=
$'{range .items[*]}{@.metadata.name}: {@.status.nodeInfo.kubeletVersion}\n{end}'
gke-test-default-pool-239f5d02-gyn2: v1.4.0
gke-test-default-pool-239f5d02-x1kf: v1.4.0
gke-test-default-pool-239f5d02-xwux: v1.4.0
AppArmor kernel module is enabled -- For the Linux kernel to enforce an AppArmor profile, the
AppArmor kernel module must be installed and enabled. Several distributions enable the module by
default, such as Ubuntu and SUSE, and many others provide optional support. To check whether the
module is enabled, check the
/sys/module/apparmor/parameters/enabled
file:
cat /sys/module/apparmor/parameters/enabled
Y
If the Kubelet contains AppArmor support (&gt;= v1.4), it will refuse to run a Pod with AppArmor
options if the kernel module is not enabled.
Note:
Ubuntu carries many AppArmor patches that have not been merged into the upstream Linux
kernel, including patches that add additional hooks and features. Kubernetes has only been
tested with the upstream version, and does not promise support for other features.
Container runtime supports AppArmor -- Currently all common Kubernetes-supported container
runtimes should support AppArmor, like
Docker
,
CRI-O
or
.
Please refer to the corresponding runtime documentation and verify that the cluster fulfills
the requirements to use AppArmor.
Profile is loaded -- AppArmor is applied to a Pod by specifying an AppArmor profile that each
container should be run with. If any of the specified profiles is not already loaded in the
kernel, the Kubelet (&gt;= v1.4) will reject the Pod. You can view which profiles are loaded on a
node by checking the
/sys/kernel/security/apparmor/profiles
file. For example:
ssh gke-test-default-pool-239f5d02-gyn2
"sudo cat /sys/kernel/security/apparmor/profiles | sort"
apparmor-test-deny-write (enforce)
apparmor-test-audit-write (enforce)
docker-default (enforce)
k8s-nginx (enforce)
For more details on loading profiles on nodes, see
Setting up nodes with profiles
.
As long as the Kubelet version includes AppArmor support (&gt;= v1.4), the Kubelet will reject a Pod
with AppArmor options if any of the prerequisites are not met. You can also verify AppArmor support
on nodes by checking the node ready condition message (though this is likely to be removed in a
later release):
=
jsonpath
=
'{range .items[*]}{@.metadata.name}: {.status.conditions[?(@.reason=="KubeletReady")].message}{"\n"}{end}'
gke-test-default-pool-239f5d02-gyn2: kubelet is posting ready status. AppArmor enabled
gke-test-default-pool-239f5d02-x1kf: kubelet is postin

g ready status. AppArmor enabled
gke-test-default-pool-239f5d02-xwux: kubelet is posting ready status. AppArmor enabled
Securing a Pod
Note:
AppArmor is currently in beta, so options are specified as annotations. Once support graduates to
general availability, the annotations will be replaced with first-class fields (more details in
Upgrade path to GA
).
AppArmor profiles are specified
per-container
. To specify the AppArmor profile to run a Pod
container with, add an annotation to the Pod's metadata:
container.apparmor.security.beta.kubernetes.io/&lt;container_name&gt;
:
&lt;profile_ref&gt;
Where
&lt;container_name&gt;
is the name of the container to apply the profile to, and
specifies the profile to apply. The
profile_ref
can be one of:
runtime/default
to apply the runtime's default profile
localhost/&lt;profile_name&gt;
to apply the profile loaded on the host with the name
&lt;profile_name&gt;
unconfined
to indicate that no profiles will be loaded
See the
API Reference
for the full details on the annotation and profile name formats.
Kubernetes AppArmor enforcement works by first checking that all the prerequisites have been
met, and then forwarding the profile selection to the container runtime for enforcement. If the
prerequisites have not been met, the Pod will be rejected, and will not run.
To verify that the profile was applied, you can look for the AppArmor security option listed in the container created event:
kubectl get events | grep Created
22s        22s         1         hello-apparmor     Pod       spec.containers{hello}   Normal    Created     {kubelet e2e-test-stclair-node-pool-31nt}   Created container with docker id 269a53b202d3; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write]
You can also verify directly that the container's root process is running with the correct profile by checking its proc attr:
kubectl
exec
&lt;pod_name&gt; -- cat /proc/1/attr/current
k8s-apparmor-example-deny-write (enforce)
Example
This example assumes you have already set up a cluster with AppArmor support.
First, we need to load the profile we want to use onto our nodes. This profile denies all file writes:
#include &lt;tunables/global&gt;
profile k8s-apparmor-example-deny-write
flags
=(
attach_disconnected
)
{
#include &lt;abstractions/base&gt;
file,
# Deny all file writes.
deny /** w,
}
Since we don't know where the Pod will be scheduled, we'll need to load the profile on all our
nodes. For this example we'll use SSH to install the profiles, but other approaches are
discussed in
.
NODES
=(
# The SSH-accessible domain names of your nodes
gke-test-default-pool-239f5d02-gyn2.us-central1-a.my-k8s
gke-test-default-pool-239f5d02-x1kf.us-central1-a.my-k8s
gke-test-default-pool-239f5d02-xwux.us-central1-a.my-k8s
)
for
NODE in
${
NODES
[*]
}
;
do
ssh
$NODE
'sudo apparmor_parser -q &lt;&lt;EOF
profile k8s-apparmor-example-deny-write flags=(attach_disconnected) {
file,
deny /** w,
}
EOF'
done
Next, we'll run a simple "Hello AppArmor" pod with the deny-write profile:
pods/security/hello-apparmor.yaml
:
v1
kind
:
Pod
metadata
:
name
:
hello-apparmor
:
# Tell Kubernetes to apply the AppArmor profile "k8s-apparmor-example-deny-write".
# Note that this is ignored if the Kubernetes node is not running version 1.4 or greater.
container.apparmor.security.beta.kubernetes.io/hello
:
localhost/k8s-apparmor-example-deny-write
spec
:
:
-
name
:
hello
image
:
command
:
[
"sh"
,
"-c"
,
"echo 'Hello AppArmor!' &amp;&amp; sleep 1h"
]
kubectl create -f ./hello-apparmor.yaml
If we look at the pod events, we can see that the Pod container was created with the AppArmor
profile "k8s-apparmor-example-deny-write":
kubectl get events | grep hello-apparmor
14s        14s         1         hello-apparmor   Pod                                Normal    Schedul

ed   {default-scheduler }                           Successfully assigned hello-apparmor to gke-test-default-pool-239f5d02-gyn2
14s        14s         1         hello-apparmor   Pod       spec.containers{hello}   Normal    Pulling     {kubelet gke-test-default-pool-239f5d02-gyn2}   pulling image "busybox"
13s        13s         1         hello-apparmor   Pod       spec.containers{hello}   Normal    Pulled      {kubelet gke-test-default-pool-239f5d02-gyn2}   Successfully pulled image "busybox"
13s        13s         1         hello-apparmor   Pod       spec.containers{hello}   Normal    Created     {kubelet gke-test-default-pool-239f5d02-gyn2}   Created container with docker id 06b6cd1c0989; Security:[seccomp=unconfined apparmor=k8s-apparmor-example-deny-write]
13s        13s         1         hello-apparmor   Pod       spec.containers{hello}   Normal    Started     {kubelet gke-test-default-pool-239f5d02-gyn2}   Started container with docker id 06b6cd1c0989
We can verify that the container is actually running with that profile by checking its proc attr:
kubectl
exec
hello-apparmor -- cat /proc/1/attr/current
Finally, we can see what happens if we try to violate the profile by writing to a file:
kubectl
exec
hello-apparmor -- touch /tmp/test
touch: /tmp/test: Permission denied
error: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1
To wrap up, let's look at what happens if we try to specify a profile that hasn't been loaded:
kubectl create -f /dev/stdin &lt;&lt;EOF
:
v1
kind
:
Pod
metadata
:
name
:
hello-apparmor-2
:
:
localhost/k8s-apparmor-example-allow-write
spec
:
:
-
name
:
hello
image
:
command
:
[
"sh"
,
"-c"
,
]
EOF
pod/hello-apparmor-2 created
kubectl describe pod hello-apparmor-2
Name:          hello-apparmor-2
Namespace:     default
Node:          gke-test-default-pool-239f5d02-x1kf/
Start Time:    Tue, 30 Aug 2016 17:58:56 -0700
Labels:        &lt;none&gt;
Annotations:   container.apparmor.security.beta.kubernetes.io/hello=localhost/k8s-apparmor-example-allow-write
Status:        Pending
Reason:        AppArmor
Message:       Pod Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded
IP:
Controllers:   &lt;none&gt;
Containers:
  hello:
    Container ID:
    Image:     busybox
    Image ID:
    Port:
    Command:
      sh
      -c
      echo 'Hello AppArmor!' &amp;&amp; sleep 1h
    State:              Waiting
      Reason:           Blocked
    Ready:              False
    Restart Count:      0
    Environment:        &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dnz7v (ro)
Conditions:
  Type          Status
  Initialized   True
  Ready         False
  PodScheduled  True
Volumes:
  default-token-dnz7v:
    Type:    Secret (a volume populated by a Secret)
    SecretName:    default-token-dnz7v
    Optional:   false
QoS Class:      BestEffort
Node-Selectors: &lt;none&gt;
Tolerations:    &lt;none&gt;
Events:
  FirstSeen    LastSeen    Count    From                        SubobjectPath    Type        Reason        Message
  ---------    --------    -----    ----                        -------------    --------    ------        -------
  23s          23s         1        {default-scheduler }                         Normal      Scheduled     Successfully assigned hello-apparmor-2 to e2e-test-stclair-node-pool-t1f5
  23s          23s         1        {kubelet e2e-test-stclair-node-pool-t1f5}             Warning        AppArmor    Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded
Note the pod status is Pending, with a helpful error message:
Pod Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-write" is not loaded
. An event was also recorded with the same 

message.
Administration
Kubernetes does not currently provide any native mechanisms for loading AppArmor profiles onto
nodes. There are lots of ways to set up the profiles though, such as:
Through a
DaemonSet
that runs a Pod on each node to
ensure the correct profiles are loaded. An example implementation can be found
here
.
At node initialization time, using your node initialization scripts (e.g. Salt, Ansible, etc.) or
image.
By copying the profiles to each node and loading them through SSH, as demonstrated in the
Example
.
The scheduler is not aware of which profiles are loaded onto which node, so the full set of profiles
must be loaded onto every node. An alternative approach is to add a node label for each profile (or
class of profiles) on the node, and use a
node selector
to ensure the Pod is run on a
node with the required profile.
Disabling AppArmor
If you do not want AppArmor to be available on your cluster, it can be disabled by a command-line flag:
--feature-gates=AppArmor=false
When disabled, any Pod that includes an AppArmor profile will fail validation with a "Forbidden"
error.
Note:
Even if the Kubernetes feature is disabled, runtimes may still enforce the default profile. The
option to disable the AppArmor feature will be removed when AppArmor graduates to general
availability (GA).
Authoring Profiles
Getting AppArmor profiles specified correctly can be a tricky business. Fortunately there are some
tools to help with that:
aa-genprof
and
aa-logprof
generate profile rules by monitoring an application's activity and
logs, and admitting the actions it takes. Further instructions are provided by the
AppArmor documentation
.
bane
is an AppArmor profile generator for Docker that uses a
simplified profile language.
To debug problems with AppArmor, you can check the system logs to see what, specifically, was
denied. AppArmor logs verbose messages to
dmesg
, and errors can usually be found in the system
logs or through
. More information is provided in
AppArmor failures
.
Pod Annotation
Specifying the profile a container will run with:
key
:
Where
matches the name of a container in the Pod.
A separate profile can be specified for each container in the Pod.
value
: a profile reference, described below
Profile Reference
: Refers to the default runtime profile.
Equivalent to not specifying a profile, except it still
requires AppArmor to be enabled.
In practice, many container runtimes use the same OCI default profile, defined here:
https://github.com/containers/common/blob/main/pkg/apparmor/apparmor_linux_template.go
: Refers to a profile loaded on the node (localhost) by name.
The possible profile names are detailed in the
core policy reference
.
: This effectively disables AppArmor on the container.
Any other profile reference format is invalid.
Additional resources:
Quick guide to the AppArmor profile language
AppArmor core policy reference
4 - Restrict a Container's Syscalls with seccomp
Kubernetes v1.19 [stable]
Seccomp stands for secure computing mode and has been a feature of the Linux
kernel since version 2.6.12. It can be used to sandbox the privileges of a
process, restricting the calls it is able to make from userspace into the
kernel. Kubernetes lets you automatically apply seccomp profiles loaded onto a
node
to your Pods and containers.
Identifying the privileges required for your workloads can be difficult. In this
tutorial, you will go through how to load seccomp profiles into a local
Kubernetes cluster, how to apply them to a Pod, and how you can begin to craft
profiles that give only the necessary privileges to your container processes.
Learn how to load seccomp profiles on a node
Learn how to apply a seccomp profile to a container
Observe auditing of syscalls made 

by a container process
Observe behavior when a missing profile is specified
Observe a violation of a seccomp profile
Learn how to create fine-grained seccomp profiles
Learn how to apply a container runtime default seccomp profile
In order to complete all steps in this tutorial, you must install
kind
and
kubectl
.
This tutorial shows some examples that are still beta (since v1.25) and
others that use only generally available seccomp functionality. You should
make sure that your cluster is
configured correctly
for the version you are using.
The tutorial also uses the
curl
tool for downloading examples to your computer.
You can adapt the steps to use a different tool if you prefer.
Note:
It is not possible to apply a seccomp profile to a container running with
privileged: true
set in the container's
securityContext
. Privileged containers always
run as
Unconfined
.
Download example seccomp profiles
The contents of these profiles will be explored later on, but for now go ahead
and download them into a directory named
profiles/
so that they can be loaded
into the cluster.
audit.json
violation.json
fine-grained.json
pods/security/seccomp/profiles/audit.json
{
"defaultAction"
:
"SCMP_ACT_LOG"
}
pods/security/seccomp/profiles/violation.json
{
:
"SCMP_ACT_ERRNO"
}
pods/security/seccomp/profiles/fine-grained.json
{
:
,
"architectures"
: [
"SCMP_ARCH_X86_64"
,
"SCMP_ARCH_X86"
,
"SCMP_ARCH_X32"
],
"syscalls"
: [
{
"names"
: [
"accept4"
,
"epoll_wait"
,
"pselect6"
,
"futex"
,
"madvise"
,
"epoll_ctl"
,
"getsockname"
,
"setsockopt"
,
"vfork"
,
"mmap"
,
"read"
,
"write"
,
"close"
,
"arch_prctl"
,
"sched_getaffinity"
,
"munmap"
,
"brk"
,
"rt_sigaction"
,
"rt_sigprocmask"
,
"sigaltstack"
,
"gettid"
,
"clone"
,
"bind"
,
"socket"
,
"openat"
,
"readlinkat"
,
"exit_group"
,
"epoll_create1"
,
"listen"
,
"rt_sigreturn"
,
"sched_yield"
,
"clock_gettime"
,
"connect"
,
"dup2"
,
"epoll_pwait"
,
"execve"
,
"exit"
,
"fcntl"
,
"getpid"
,
"getuid"
,
"ioctl"
,
"mprotect"
,
"nanosleep"
,
"open"
,
"poll"
,
"recvfrom"
,
"sendto"
,
"set_tid_address"
,
"setitimer"
,
"writev"
],
"action"
:
"SCMP_ACT_ALLOW"
}
]
}
Run these commands:
mkdir ./profiles
curl -L -o profiles/audit.json https://k8s.io/examples/pods/security/seccomp/profiles/audit.json
curl -L -o profiles/violation.json https://k8s.io/examples/pods/security/seccomp/profiles/violation.json
curl -L -o profiles/fine-grained.json https://k8s.io/examples/pods/security/seccomp/profiles/fine-grained.json
ls profiles
You should see three profiles listed at the end of the final step:
audit.json  fine-grained.json  violation.json
Create a local Kubernetes cluster with kind
For simplicity,
kind
can be used to create a single
node cluster with the seccomp profiles loaded. Kind runs Kubernetes in Docker,
so each node of the cluster is a container. This allows for files
to be mounted in the filesystem of each container similar to loading files
onto a node.
pods/security/seccomp/kind.yaml
:
kind.x-k8s.io/v1alpha4
kind
:
Cluster
nodes
:
-
role
:
control-plane
extraMounts
:
-
hostPath
:
"./profiles"
containerPath
:
"/var/lib/kubelet/seccomp/profiles"
Download that example kind configuration, and save it to a file named
kind.yaml
:
curl -L -O https://k8s.io/examples/pods/security/seccomp/kind.yaml
You can set a specific Kubernetes version by setting the node's container image.
See
Nodes
within the
kind documentation about configuration for more details on this.
This tutorial assumes you are using Kubernetes v1.26.
As a beta feature, you can configure Kubernetes to use the profile that the
prefers by default, rather than falling back to
.
If you want to try that, see
enable the use of
RuntimeDefault
as the default seccomp profile for all workloads
before you continue.
Once you have a kind configuration in place, create the kind cluster with
that configuration:
kind create cluster --config
=
kind.yaml
After the new Kuberne

tes cluster is ready, identify the Docker container running
as the single node cluster:
docker ps
You should see output indicating that a container is running with name
kind-control-plane
. The output is similar to:
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                       NAMES
6a96207fed4b        kindest/node:v1.18.2   "/usr/local/bin/entr…"   27 seconds ago      Up 24 seconds       127.0.0.1:42223-&gt;6443/tcp   kind-control-plane
If observing the filesystem of that container, you should see that the
profiles/
directory has been successfully loaded into the default seccomp path
of the kubelet. Use
docker exec
to run a command in the Pod:
# Change 6a96207fed4b to the container ID you saw from "docker ps"
docker
exec
-it 6a96207fed4b ls /var/lib/kubelet/seccomp/profiles
You have verified that these seccomp profiles are available to the kubelet
running within kind.
Enable the use of
Kubernetes v1.25 [beta]
To use seccomp profile defaulting, you must run the kubelet with the
SeccompDefault
enabled
(this is the default). You must also explicitly enable the defaulting behavior for each
node where you want to use this with the corresponding
--seccomp-default
command line flag
.
Both have to be enabled simultaneously to use the feature.
If enabled, the kubelet will use the
seccomp profile by default, which is
defined by the container runtime, instead of using the
(seccomp disabled) mode.
The default profiles aim to provide a strong set
of security defaults while preserving the functionality of the workload. It is
possible that the default profiles differ between container runtimes and their
release versions, for example when comparing those from CRI-O and containerd.
Note:
Enabling the feature will neither change the Kubernetes
securityContext.seccompProfile
API field nor add the deprecated annotations of
the workload. This provides users the possibility to rollback anytime without
actually changing the workload configuration. Tools like
crictl inspect
can be used to
verify which seccomp profile is being used by a container.
Some workloads may require a lower amount of syscall restrictions than others.
This means that they can fail during runtime even with the
profile. To mitigate such a failure, you can:
Run the workload explicitly as
.
Disable the
feature for the nodes. Also making sure that
workloads get scheduled on nodes where the feature is disabled.
Create a custom seccomp profile for the workload.
If you were introducing this feature into production-like cluster, the Kubernetes project
recommends that you enable this feature gate on a subset of your nodes and then
test workload execution before rolling the change out cluster-wide.
You can find more detailed information about a possible upgrade and downgrade strategy
in the related Kubernetes Enhancement Proposal (KEP):
Enable seccomp by default
.
Kubernetes 1.26 lets you configure the seccomp profile
that applies when the spec for a Pod doesn't define a specific seccomp profile.
This is a beta feature and the corresponding
feature
gate
is enabled by
default. However, you still need to enable this defaulting for each node where
you would like to use it.
If you are running a Kubernetes 1.26 cluster and want to
enable the feature, either run the kubelet with the
command
line flag, or enable it through the
kubelet configuration
file
. To enable the
feature gate in
kind
, ensure that
kind
provides
the minimum required Kubernetes version and enables the
feature
in the kind configuration
:
kind
:
Cluster
:
:
:
true
nodes
:
-
role
:
image
:
kindest/node:v1.23.0@sha256:49824ab1727c04

e56a21a5d8372a402fcd32ea51ac96a2706a12af38934f81ac
kubeadmConfigPatches
:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
seccomp-default: "true"
-
role
:
worker
image
:
kindest/node:v1.23.0@sha256:49824ab1727c04e56a21a5d8372a402fcd32ea51ac96a2706a12af38934f81ac
:
- |
feature-gates: SeccompDefault=true
If the cluster is ready, then running a pod:
kubectl run --rm -it --restart
=
Never --image
=
alpine alpine -- sh
Should now have the default seccomp profile attached. This can be verified by
using
to run
for the container on the kind
worker:
docker
exec
-it kind-worker bash -c
\
'crictl inspect $(crictl ps --name=alpine -q) | jq .info.runtimeSpec.linux.seccomp'
{
:
,
: [
,
,
],
: [
{
"names"
: [
"..."
]
}
]
}
Create a Pod with a seccomp profile for syscall auditing
To start off, apply the
profile, which will log all syscalls of the
process, to a new Pod.
Here's a manifest for that Pod:
pods/security/seccomp/ga/audit-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
audit-pod
labels
:
app
:
audit-pod
spec
:
:
seccompProfile
:
type
:
Localhost
localhostProfile
:
profiles/audit.json
:
-
name
:
test-container
image
:
hashicorp/http-echo:0.2.3
args
:
-
"-text=just made some syscalls!"
:
allowPrivilegeEscalation
:
false
Note:
The functional support for the already deprecated seccomp annotations
seccomp.security.alpha.kubernetes.io/pod
(for the whole pod) and
container.seccomp.security.alpha.kubernetes.io/[name]
(for a single container)
is going to be removed with a future release of Kubernetes. Please always use
the native API fields in favor of the annotations.
Since Kubernetes v1.25, kubelets no longer support the annotations, use of the
annotations in static pods is no longer supported, and the seccomp annotations
are no longer auto-populated when pods with seccomp fields are created.
Auto-population of the seccomp fields from the annotations is planned to be
removed in a future release.
kubectl apply -f https://k8s.io/examples/pods/security/seccomp/ga/audit-pod.yaml
This profile does not restrict any syscalls, so the Pod should start
successfully.
kubectl get pod/audit-pod
NAME        READY   STATUS    RESTARTS   AGE
audit-pod   1/1     Running   0          30s
In order to be able to interact with this endpoint exposed by this
container, create a NodePort
Services
that allows access to the endpoint from inside the kind control plane container.
kubectl expose pod audit-pod --type NodePort --port
5678
Check what port the Service has been assigned on the node.
kubectl get service audit-pod
NAME        TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
audit-pod   NodePort   10.111.36.142   &lt;none&gt;        5678:32373/TCP   72s
Now you can use
curl
to access that endpoint from inside the kind control plane container,
at the port exposed by this Service. Use
to run the
curl
command within the
container belonging to that control plane container:
# Change 6a96207fed4b to the control plane container ID you saw from "docker ps"
docker
exec
-it 6a96207fed4b curl localhost:32373
just made some syscalls!
You can see that the process is running, but what syscalls did it actually make?
Because this Pod is running in a local cluster, you should be able to see those
in
/var/log/syslog
. Open up a new terminal window and
tail
the output for
calls from
http-echo
:
tail -f /var/log/syslog | grep
'http-echo'
You should already see some logs of syscalls made by
http-echo
, and if you
curl
the endpoint in the control plane container you will see more written.
Jul  6 15:37:40 my-machine kernel: [369128.669452] audit: type=1326 audit(1594067860.484

:14536): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=51 compat=0 ip=0x46fe1f code=0x7ffc0000
Jul  6 15:37:40 my-machine kernel: [369128.669453] audit: type=1326 audit(1594067860.484:14537): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=54 compat=0 ip=0x46fdba code=0x7ffc0000
Jul  6 15:37:40 my-machine kernel: [369128.669455] audit: type=1326 audit(1594067860.484:14538): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=202 compat=0 ip=0x455e53 code=0x7ffc0000
Jul  6 15:37:40 my-machine kernel: [369128.669456] audit: type=1326 audit(1594067860.484:14539): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=288 compat=0 ip=0x46fdba code=0x7ffc0000
Jul  6 15:37:40 my-machine kernel: [369128.669517] audit: type=1326 audit(1594067860.484:14540): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=0 compat=0 ip=0x46fd44 code=0x7ffc0000
Jul  6 15:37:40 my-machine kernel: [369128.669519] audit: type=1326 audit(1594067860.484:14541): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=270 compat=0 ip=0x4559b1 code=0x7ffc0000
Jul  6 15:38:40 my-machine kernel: [369188.671648] audit: type=1326 audit(1594067920.488:14559): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=270 compat=0 ip=0x4559b1 code=0x7ffc0000
Jul  6 15:38:40 my-machine kernel: [369188.671726] audit: type=1326 audit(1594067920.488:14560): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=202 compat=0 ip=0x455e53 code=0x7ffc0000
You can begin to understand the syscalls required by the
http-echo
process by
looking at the
syscall=
entry on each line. While these are unlikely to
encompass all syscalls it uses, it can serve as a basis for a seccomp profile
for this container.
Clean up that Pod and Service before moving to the next section:
kubectl delete service audit-pod --wait
kubectl delete pod audit-pod --wait --now
Create Pod with a seccomp profile that causes violation
For demonstration, apply a profile to the Pod that does not allow for any
syscalls.
The manifest for this demonstration is:
pods/security/seccomp/ga/violation-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
violation-pod
labels
:
app
:
spec
:
:
:
type
:
Localhost
:
profiles/violation.json
:
-
name
:
image
:
args
:
-
:
:
false
Attempt to create the Pod in the cluster:
kubectl apply -f https://k8s.io/examples/pods/security/seccomp/ga/violation-pod.yaml
The Pod creates, but there is an issue.
If you check the status of the Pod, you should see that it failed to start.
kubectl get pod/violation-pod
NAME            READY   STATUS             RESTARTS   AGE
violation-pod   0/1     CrashLoopBackOff   1          6s
As seen in the previous example, the
http-echo
process requires quite a few
syscalls. Here seccomp has been instructed to error on any syscall by setting
"defaultAction": "SCMP_ACT_ERRNO"
. This is extremely secure, but removes the
ability to do anything meaningful. What you really want is to give workloads
only the privileges they need.
Clean up that Pod before moving to the next section:
kubectl delete pod violation-pod --wait --now
Create Pod with a seccomp profile that only allows necessary syscalls
If you take a look at the
profile, you will notice some of the syscalls
seen in syslog of the first example where the profile set
"defaultAction": "SCMP_ACT_LOG"
. Now the profile 

is setting
,
but explicitly allowing a set of syscalls in the
"action": "SCMP_ACT_ALLOW"
block. Ideally, the container will run successfully and you will see no messages
sent to
syslog
.
The manifest for this example is:
pods/security/seccomp/ga/fine-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
fine-pod
labels
:
app
:
fine-pod
spec
:
:
:
type
:
Localhost
:
profiles/fine-grained.json
:
-
name
:
image
:
args
:
-
:
:
false
Create the Pod in your cluster:
kubectl apply -f https://k8s.io/examples/pods/security/seccomp/ga/fine-pod.yaml
kubectl get pod fine-pod
The Pod should be showing as having started successfully:
fine-pod   1/1     Running   0          30s
Open up a new terminal window and use
tail
to monitor for log entries that
mention calls from
http-echo
:
# The log path on your computer might be different from "/var/log/syslog"
Next, expose the Pod with a NodePort Service:
kubectl expose pod fine-pod --type NodePort --port
5678
Check what port the Service has been assigned on the node:
kubectl get service fine-pod
fine-pod    NodePort   10.111.36.142   &lt;none&gt;        5678:32373/TCP   72s
Use
curl
to access that endpoint from inside the kind control plane container:
docker
exec
You should see no output in the
syslog
. This is because the profile allowed all
necessary syscalls and specified that an error should occur if one outside of
the list is invoked. This is an ideal situation from a security perspective, but
required some effort in analyzing the program. It would be nice if there was a
simple way to get closer to this security without requiring as much effort.
kubectl delete service fine-pod --wait
kubectl delete pod fine-pod --wait --now
Create Pod that uses the container runtime default seccomp profile
Most container runtimes provide a sane set of default syscalls that are allowed
or not. You can adopt these defaults for your workload by setting the seccomp
type in the security context of a pod or container to
.
Note:
If you have the
enabled, then Pods use the
seccomp profile whenever
no other seccomp profile is specified. Otherwise, the default is
.
Here's a manifest for a Pod that requests the
seccomp profile
for all its containers:
pods/security/seccomp/ga/default-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
default-pod
labels
:
app
:
spec
:
:
:
type
:
:
-
name
:
image
:
args
:
-
"-text=just made some more syscalls!"
:
:
false
Create that Pod:
kubectl apply -f https://k8s.io/examples/pods/security/seccomp/ga/default-pod.yaml
kubectl get pod default-pod
default-pod 1/1     Running   0          20s
Finally, now that you saw that work OK, clean up:
kubectl delete pod default-pod --wait --now
You can learn more about Linux seccomp:
A seccomp Overview
Seccomp Security Profiles for Docker
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademark

s of The Linux Foundation, please see our

Kubelet CredentialProvider (v1alpha1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl

 on Linux
Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespace L

evel
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

 Encryption Configuration (v1)
Reference
CredentialProviderRequest
CredentialProviderResponse
CredentialProviderRequest includes the image that the kubelet requires authentication for.
Kubelet will pass this request object to the plugin via stdin. In general, plugins should
prefer responding with the same apiVersion they were sent.
Field
string
credentialprovider.kubelet.k8s.io/v1alpha1
kind
string
image
string
image is the container image that is being pulled as part of the
credential provider plugin request. Plugins may optionally parse the image
to extract any information required to fetch credentials.
CredentialProviderResponse holds credentials that the kubelet should use for the specified
image provided in the original request. Kubelet will read the response from the plugin via stdout.
This response should be set to the same apiVersion as CredentialProviderRequest.
Field
string
kind
string
cacheKeyType
PluginCacheKeyType
cacheKeyType indiciates the type of caching key to use based on the image provided
in the request. There are three valid values for the cache key type: Image, Registry, and
Global. If an invalid value is specified, the response will NOT be used by the kubelet.
cacheDuration
cacheDuration indicates the duration the provided credentials should be cached for.
The kubelet will use this field to set the in-memory cache duration for credentials
in the AuthConfig. If null, the kubelet will use defaultCacheDuration provided in
CredentialProviderConfig. If set to 0, the kubelet will not cache the provided AuthConfig.
auth
map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1alpha1.AuthConfig
auth is a map containing authentication information passed into the kubelet.
Each key is a match image string (more on this below). The corresponding authConfig value
should be valid for all images that match against this key. A plugin should set
this field to null if no valid credentials can be returned for the requested image.
Each key in the map is a pattern which can optionally contain a port and a path.
Globs can be used in the domain, but not in the port or the path. Globs are supported
as subdomains like '
.k8s.io' or 'k8s.
.io', and top-level-domains such as 'k8s.
'.
Matching partial subdomains like 'app
.k8s.io' is also

 supported. Each glob can only match
a single subdomain segment, so *.io does not match *.k8s.io.
The kubelet will match images against the key when all of the below are true:
Both contain the same number of domain parts and each part matches.
The URL path of an imageMatch must be a prefix of the target image URL path.
If the imageMatch contains a port, then the port must match in the image as well.
When multiple keys are returned, the kubelet will traverse all keys in reverse order so that:
longer keys come before shorter keys with the same prefix
non-wildcard keys come before wildcard keys with the same prefix.
For any given match, the kubelet will attempt an image pull with the provided credentials,
stopping after the first successfully authenticated pull.
Example keys:
123456789.dkr.ecr.us-east-1.amazonaws.com
*.azurecr.io
gcr.io
.
.registry.io
registry.io:8080/path
AuthConfig
AuthConfig contains authentication information for a container registry.
Only username/password based authentication is supported today, but more authentication
mechanisms may be added in the future.
Field
username
string
username is the username used for authenticating to the container registry
An empty username is valid.
password
string
password is the password used for authenticating to the container registry
An empty password is valid.
(Alias of
string
)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-proxy Configuration (v1alpha1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on 

Linux
Debug Pods
Get

 a Shell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security

Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Enc

ryption Configuration (v1)
Reference
KubeProxyConfiguration contains everything necessary to configure the
Kubernetes proxy server.
Field
string
kind
string
map[string]bool
featureGates is a map of feature names to bools that enable or disable alpha/experimental features.
bindAddress
string
bindAddress is the IP address for the proxy server to serve on (set to 0.0.0.0
for all interfaces)
healthzBindAddress
string
healthzBindAddress is the IP address and port for the health check server to serve on,
defaulting to 0.0.0.0:10256
metricsBindAddress
string
metricsBindAddress is the IP address and port for the metrics server to serve on,
defaulting to 127.0.0.1:10249 (set to 0.0.0.0 for all interfaces)
bindAddressHardFail
bool
bindAddressHardFail, if true, kube-proxy will treat failure to bind to a port as fatal and exit
enableProfiling
bool
enableProfiling enables profiling via web interface on /debug/pprof handler.
Profiling handlers will be handled by metrics server.
clusterCIDR
string
clusterCIDR is the CIDR range of the pods in the cluster. It is used to
bridge traffic coming from outside of the cluster. If not provided,
no off-cluster bridging will be performed.
hostnameOverride
string
hostnameOverride, if non-empty, will be used as the identity instead of the actual hostname.
clientConnection
ClientConnectionConfiguration
clientConnection specifies the kubeconfig file and client connection settings for the proxy
server to use when communicating with the apiserver.
iptables
KubeProxyIPTablesConfiguration
iptables contains iptables-related configuration options.
ipvs
KubeProxyIPVSConfiguration
ipvs contains ipvs-related configuration options.
oomScoreAdj
int32
oomScoreAdj is the oom-score-adj value for kube-proxy process. Values must be within
the range [-1000, 1000]
mode
ProxyMode
mode specifies which proxy mode to use.
portRange
string
portRange is the range of host ports (beginPort-endPort, inclusive) that may be consumed
in order to proxy service traffic. If unspecified (0-0) then ports will be randomly chosen.
conntrack
KubeProxyConntrackConfiguration
conntrack contains conntrack-related configuration options.
configSyncPeriod
configSync

Period is how often configuration from the apiserver is refreshed. Must be greater
than 0.
nodePortAddresses
[]string
nodePortAddresses is the --nodeport-addresses value for kube-proxy process. Values must be valid
IP blocks. These values are as a parameter to select the interfaces where nodeport works.
In case someone would like to expose a service on localhost for local visit and some other interfaces for
particular purpose, a list of IP blocks would do that.
If set it to "127.0.0.0/8", kube-proxy will only select the loopback interface for NodePort.
If set it to a non-zero IP block, kube-proxy will filter that down to just the IPs that applied to the node.
An empty string slice is meant to select all network interfaces.
winkernel
KubeProxyWinkernelConfiguration
winkernel contains winkernel-related configuration options.
showHiddenMetricsForVersion
string
ShowHiddenMetricsForVersion is the version for which you want to show hidden metrics.
detectLocalMode
LocalMode
DetectLocalMode determines mode to use for detecting local traffic, defaults to LocalModeClusterCIDR
detectLocal
DetectLocalConfiguration
DetectLocal contains optional configuration settings related to DetectLocalMode.
DetectLocalConfiguration contains optional settings related to DetectLocalMode option
Field
bridgeInterface
string
BridgeInterface is a string argument which represents a single bridge interface name.
Kube-proxy considers traffic as local if originating from this given bridge.
This argument should be set if DetectLocalMode is set to LocalModeBridgeInterface.
interfaceNamePrefix
string
InterfaceNamePrefix is a string argument which represents a single interface prefix name.
Kube-proxy considers traffic as local if originating from one or more interfaces which match
the given prefix. This argument should be set if DetectLocalMode is set to LocalModeInterfaceNamePrefix.
KubeProxyConntrackConfiguration contains conntrack settings for
the Kubernetes proxy server.
Field
int32
maxPerCore is the maximum number of NAT connections to track
per CPU core (0 to leave the limit as-is and ignore min).
min
int32
min is the minimum value of connect-tracking records to allocate,
regardless of conntrackMaxPerCore (set maxPerCore=0 to leave the limit as-is).
tcpEstablishedTimeout is how long an idle TCP connection will be kept open
(e.g. '2s'). Must be greater than 0 to set.
tcpCloseWaitTimeout is how long an idle conntrack entry
in CLOSE_WAIT state will remain in the conntrack
table. (e.g. '60s'). Must be greater than 0 to set.
KubeProxyIPTablesConfiguration contains iptables-related configuration
details for the Kubernetes proxy server.
Field
masqueradeBit
int32
masqueradeBit is the bit of the iptables fwmark space to use for SNAT if using
the pure iptables proxy mode. Values must be within the range [0, 31].
masqueradeAll
bool
masqueradeAll tells kube-proxy to SNAT everything if using the pure iptables proxy mode.
localhostNodePorts
bool
LocalhostNodePorts tells kube-proxy to allow service NodePorts to be accessed via
localhost (iptables mode only)
syncPeriod
syncPeriod is the period that iptables rules are refreshed (e.g. '5s', '1m',
'2h22m'). Must be greater than 0.
minSyncPeriod
minSyncPeriod is the minimum period that iptables rules are refreshed (e.g. '5s', '1m',
'2h22m').
KubeProxyIPVSConfiguration contains ipvs-related configuration
Field
Des

cription
syncPeriod is the period that ipvs rules are refreshed (e.g. '5s', '1m',
minSyncPeriod is the minimum period that ipvs rules are refreshed (e.g. '5s', '1m',
'2h22m').
scheduler
string
ipvs scheduler
excludeCIDRs
[]string
excludeCIDRs is a list of CIDR's which the ipvs proxier should not touch
when cleaning up ipvs services.
strictARP
bool
strict ARP configure arp_ignore and arp_announce to avoid answering ARP queries
from kube-ipvs0 interface
tcpTimeout
tcpTimeout is the timeout value used for idle IPVS TCP sessions.
The default value is 0, which preserves the current timeout value on the system.
tcpFinTimeout
tcpFinTimeout is the timeout value used for IPVS TCP sessions after receiving a FIN.
udpTimeout
udpTimeout is the timeout value used for IPVS UDP packets.
KubeProxyWinkernelConfiguration contains Windows/HNS settings for
Field
networkName
string
networkName is the name of the network kube-proxy will use
to create endpoints and policies
sourceVip
string
sourceVip is the IP address of the source VIP endoint used for
NAT when loadbalancing
enableDSR
bool
enableDSR tells kube-proxy whether HNS policies should be created
with DSR
rootHnsEndpointName
string
RootHnsEndpointName is the name of hnsendpoint that is attached to
l2bridge for root network namespace
forwardHealthCheckVip
bool
ForwardHealthCheckVip forwards service VIP for health check port on
Windows
LocalMode
(Alias of
string
)
LocalMode represents modes to detect local traffic from the node
ProxyMode
(Alias of
string
)
ProxyMode represents modes used by the Kubernetes proxy server.
Currently, two modes of proxy are available on Linux platforms: 'iptables' and 'ipvs'.
One mode of proxy is available on Windows platforms: 'kernelspace'.
If the proxy mode is unspecified, the best-available proxy mode will be used (currently this
is
iptables
on Linux and
kernelspace
on Windows). If the selected proxy mode cannot be
used (due to lack of kernel support, missing userspace components, etc) then kube-proxy
will exit with an error.
KubeSchedulerConfiguration
GenericControllerManagerConfiguration
ClientConnectionConfiguration contains details for constructing a client.
Field
kubeconfig
string
kubeconfig is the path to a KubeConfig file.
acceptContentTypes
string
acceptContentTypes defines the Accept header sent by clients when connecting to a server, overriding the
default value of 'application/json'. This field will control all connections to the server used by a particular
client.
contentType
string
contentType is the content type used when sending data to the server from this client.
qps
float32
qps controls the number of queries per second allowed for this connection.
burst
int32
burst allows extra queries to accumulate when a client is exceeding its rate.
DebuggingConfiguration
DebuggingConfiguration holds configuration for Debugging related features.
Field
bool
enableProfiling enables profiling via web interface host:port/debug/pprof/
enableContentionProfiling
bool
enableContentionProfili

ng enables lock contention profiling, if
enableProfiling is true.
LeaderElectionConfiguration
LeaderElectionConfiguration defines the configuration of leader election
clients for components that can run with leader election enabled.
Field
leaderElect
bool
leaderElect enables a leader election client to gain leadership
before executing the main loop. Enable this when running replicated
components for high availability.
leaseDuration
leaseDuration is the duration that non-leader candidates will wait
after observing a leadership renewal until attempting to acquire
leadership of a led but unrenewed leader slot. This is effectively the
maximum duration that a leader can be stopped before it is replaced
by another candidate. This is only applicable if leader election is
enabled.
renewDeadline
renewDeadline is the interval between attempts by the acting master to
renew a leadership slot before it stops leading. This must be less
than or equal to the lease duration. This is only applicable if leader
election is enabled.
retryPeriod
retryPeriod is the duration the clients should wait between attempting
acquisition and renewal of a leadership. This is only applicable if
leader election is enabled.
resourceLock
string
resourceLock indicates the resource object type that will be used to lock
during leader election cycles.
resourceName
string
resourceName indicates the name of resource object that will be used to lock
resourceNamespace
string
resourceName indicates the namespace of resource object that will be used to lock
Feedback
Yes
No
.
or
.
Last modified December 13, 2022 at 10:12 AM PST:
Config API update for 1.26 (6989fc3c5b)
LocalMode
ProxyMode
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeconfig (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
हिन्दी (Hindi)
Tiếng Việt (Vietnamese)
Polski (Polish)
Українська (Ukrainian)
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, Storage, and Net

working Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Contro

ller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Inter

active Tutorial - Configuring a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate Limit Configuration (v1al

pha1)
Reference
Config
AuthInfo
NamedAuthInfo
AuthInfo contains information that describes identity information. This is use to tell the kubernetes cluster who you are.
Field
client-certificate
string
ClientCertificate is the path to a client cert file for TLS.
client-certificate-data
[]byte
ClientCertificateData contains PEM-encoded data from a client cert file for TLS. Overrides ClientCertificate
client-key
string
ClientKey is the path to a client key file for TLS.
client-key-data
[]byte
ClientKeyData contains PEM-encoded data from a client key file for TLS. Overrides ClientKey
token
string
Token is the bearer token for authentication to the kubernetes cluster.
tokenFile
string
TokenFile is a pointer to a file that contains a bearer token (as described above). If both Token and TokenFile are present, Token takes precedence.
as
string
Impersonate is the username to impersonate. The name matches the flag.
as-uid
string
ImpersonateUID is the uid to impersonate.
as-groups
[]string
ImpersonateGroups is the groups to impersonate.
as-user-extra
map[string][]string
ImpersonateUserExtra contains additional information for impersonated user.
username
string
Username is the username for basic authentication to the kubernetes cluster.
password
string
Password is the password for basic authentication to the kubernetes cluster.
auth-provider
AuthProviderConfig
AuthProvider specifies a custom authentication plugin for the kubernetes cluster.
exec
ExecConfig
Exec specifies a custom exec-based authentication plugin for the kubernetes cluster.
extensions
[]NamedExtension
Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unknown fields
AuthInfo
AuthProviderConfig holds the configuration for a specified auth provider.
Field
name
string
No description provided.
config
map[string]string
Cluster
NamedCluster
Cluster contains information about how to communicate with a kubernetes cluster
Field
server
string
Server is the address of the kubernetes cluster (https://hostname:port).
tls-server-name
string
TLSServerName is used to check server certificate. If TLSServerName is empty, the hostname used to contact the

 server is used.
insecure-skip-tls-verify
bool
InsecureSkipTLSVerify skips the validity check for the server's certificate. This will make your HTTPS connections insecure.
certificate-authority
string
CertificateAuthority is the path to a cert file for the certificate authority.
certificate-authority-data
[]byte
CertificateAuthorityData contains PEM-encoded certificate authority certificates. Overrides CertificateAuthority
proxy-url
string
ProxyURL is the URL to the proxy to be used for all requests made by this
client. URLs with "http", "https", and "socks5" schemes are supported. If
this configuration is not provided or the empty string, the client
attempts to construct a proxy configuration from http_proxy and
https_proxy environment variables. If these environment variables are not
set, the client does not attempt to proxy requests.
socks5 proxying does not currently support spdy streaming endpoints (exec,
attach, port forward).
disable-compression
bool
DisableCompression allows client to opt-out of response compression for all requests to the server. This is useful
to speed up requests (specifically lists) when client-server network bandwidth is ample, by saving time on
compression (server-side) and decompression (client-side): https://github.com/kubernetes/kubernetes/issues/112296.
Context
NamedContext
Context is a tuple of references to a cluster (how do I communicate with a kubernetes cluster), a user (how do I identify myself), and a namespace (what subset of resources do I want to work with)
Field
cluster
string
Cluster is the name of the cluster for this context
user
string
AuthInfo is the name of the authInfo for this context
namespace
string
Namespace is the default namespace to use on unspecified requests
AuthInfo
ExecConfig specifies a command to provide client credentials. The command is exec'd
and outputs structured stdout holding credentials.
See the client.authentication.k8s.io API group for specifications of the exact input
and output format
Field
command
string
Command to execute.
args
[]string
Arguments to pass to the command when executing it.
env
[]ExecEnvVar
Env defines additional environment variables to expose to the process. These
are unioned with the host's environment, as well as variables client-go uses
to pass argument to the plugin.
string
Preferred input version of the ExecInfo. The returned ExecCredentials MUST use
the same encoding version as the input.
installHint
string
This text is shown to the user when the executable doesn't seem to be
present. For example,
brew install foo-cli
might be a good InstallHint for
foo-cli on Mac OS systems.
provideClusterInfo
bool
ProvideClusterInfo determines whether or not to provide cluster information,
which could potentially contain very large CA data, to this exec plugin as a
part of the KUBERNETES_EXEC_INFO environment variable. By default, it is set
to false. Package k8s.io/client-go/tools/auth/exec provides helper methods for
reading this environment variable.
interactiveMode
ExecInteractiveMode
InteractiveMode determines this plugin's relationship with standard input. Valid
values are "Never" (this exec plugin never uses standard input), "IfAvailable" (this
exec plugin wants to use standard input if it is available), or "Always" (this exec
plugin requires standard input to function). See ExecInteractiveMode values for more
details.
If APIVersion is client.authentication.k8s.io/v1alpha1 or
client.authentication.k8s.io/v1beta1, then this field is optional and defaults
to "IfAvailable" when unset. Otherwise, this field 

is required.
ExecEnvVar
ExecEnvVar is used for setting environment variables when executing an exec-based
credential plugin.
Field
name
string
value
string
(Alias of
string
)
ExecInteractiveMode is a string that describes an exec plugin's relationship with standard input.
Config
NamedAuthInfo relates nicknames to auth information
Field
name
string
Name is the nickname for this AuthInfo
user
AuthInfo
AuthInfo holds the auth information
Config
NamedCluster relates nicknames to cluster information
Field
name
string
Name is the nickname for this Cluster
cluster
Cluster
Cluster holds the cluster information
Config
NamedContext relates nicknames to context information
Field
name
string
Name is the nickname for this Context
context
Context
Context holds the context information
NamedExtension
Config
AuthInfo
Cluster
Context
Preferences
NamedExtension relates nicknames to extension information
Field
name
string
Name is the nickname for this Extension
extension
k8s.io/apimachinery/pkg/runtime.RawExtension
Extension holds the extension information
Config
Field
colors
bool
Feedback
Yes
No
.
or
.
Last modified December 20, 2022 at 1:52 PM PST:
Add Config API reference for kubeconfig (6bdc308785)
AuthInfo
Cluster
Context
Home
Blog
Training
Partners
Community
CC BY 4.0

Authorization Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up

 kubectl on macOS
Debug Pods
Get a Shell to a Running Co

ntainer
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's 

Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configuration (v

1)
Reference
LocalSubjectAccessReview checks whether or not a user or group can perform an action in a given namespace.
SelfSubjectAccessReview checks whether or the current user can perform an action.
SelfSubjectRulesReview enumerates the set of actions the current user can perform within a namespace.
SubjectAccessReview checks whether or not a user or group can perform an action.
SelfSubjectReview contains the user information that the kube-apiserver has about the user making this request.
ClusterRole is a cluster level, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding or ClusterRoleBinding.
ClusterRoleBinding references a ClusterRole, but not contain it.
Role
Role is a namespaced, logical grouping of PolicyRules that can be referenced as a unit by a RoleBinding.
RoleBinding references a role, but does not contain it.
Feedback
Yes
No
.
or
.
Last modified April 18, 2021 at 5:30 PM PST:
Add auto_gerated metadata (21a2715031)
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes Documentation | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Ext

ending Kubernetes
Tasks
Migrate from PodSecurityPolicy t

o the Built-In PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using

 MicroProfile, ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Client Authentication (v1

beta1)
Home
Kubernetes is an open source container orchestration engine for automating deployment, scaling, and management of containerized applications. The open source project is hosted by the Cloud Native Computing Foundation (
CNCF
).
Understand Kubernetes
Learn about Kubernetes and its fundamental concepts.
Overview
Pods
View Concepts
Try Kubernetes
Follow tutorials to learn how to deploy applications in Kubernetes.
Walkthrough the basics
Stateless Example: PHP Guestbook with Redis
Stateful Example: Wordpress with Persistent Volumes
View Tutorials
Set up a K8s cluster
Get Kubernetes running based on your resources and needs.
Install the kubeadm setup tool
Set up Kubernetes
Learn how to use Kubernetes
Look up common tasks and how to perform them using a short sequence of steps.
Install kubectl on Linux
Install kubectl on macOS
Install kubectl on Windows
Use the Web UI Dashboard
View Tasks
Training
Get certified in Kubernetes and make your cloud native projects successful!
View training
Look up reference information
Browse terminology, command line syntax, API resource types, and setup tool documentation.
Glossary
Kubeadm
Overview of API
View Reference
Contribute to the docs
Anyone can contribute, whether you're new to the project or you've been around a long time.
Start contributing to K8s
Documenting a feature for a release
Translating the docs
Download Kubernetes
Install Kubernetes or upgrade to the newest version.
About the documentation
This website contains documentation for the current and previous 4 versions of Kubernetes.
Last modified October 09, 2022 at 11:31 AM PST:
Changes-01 applied (16e02cd193)
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights res

erved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Kubernetes Issue Tracker | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

tall and Set Up kubectl on macOS
Debug Pods
Get a Shell 

to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict

 a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption C

onfiguration (v1)
Reference
To report a security issue, please follow the
Kubernetes security disclosure process
.
Work on Kubernetes code and public issues are tracked using
GitHub Issues
.
Official
list of known CVEs
(security vulnerabilities) that have been announced by the
Security Response Committee
CVE-related GitHub issues
Security-related announcements are sent to the
kubernetes-security-announce@googlegroups.com
mailing list.
Feedback
Yes
No
.
or
.
Last modified July 19, 2022 at 3:47 PM PST:
Fetch and Render CVE JSON feed (cafe6d258c)
Home
Blog
Training
Partners
Community
CC BY 4.0

Configuring Redis using a ConfigMap | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

stall and Set Up kubectl on Linux
Debug Pods
Debug Init Conta

iners
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Stan

dards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (

v1alpha1)
Tutorials
This page provides a real world example of how to configure Redis using a ConfigMap and builds upon the
task.
Create a ConfigMap with Redis configuration values
Create a Redis Pod that mounts and uses the created ConfigMap
Verify that the configuration was correctly applied.
minikube
.
The example shown on this page works with
kubectl
1.14 and above.
Understand
.
Real World Example: Configuring Redis using a ConfigMap
Follow the steps below to configure a Redis cache using data stored in a ConfigMap.
First create a ConfigMap with an empty configuration block:
cat
&lt;&lt;EOF &gt;./example-redis-config.yaml
kind: ConfigMap
metadata:
name: example-redis-config
data:
redis-config: ""
EOF
Apply the ConfigMap created above, along with a Redis pod manifest:
kubectl apply -f example-redis-config.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/pods/config/redis-pod.yaml
Examine the contents of the Redis pod manifest and note the following:
A volume named
config
is created by
spec.volumes[1]
The
key
and
path
under
spec.volumes[1].items[0]
exposes the
redis-config
key from the
example-redis-config
ConfigMap as a file named
redis.conf
on the
config
volume.
The
config
volume is then mounted at
/redis-master
by
spec.containers[0].volumeMounts[1]
.
This has the net effect of exposing the data in
data.redis-config
from the
ConfigMap above as
/redis-master/redis.conf
inside the Pod.
pods/config/redis-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
redis
spec
:
:
-
name
:
redis
image
:
redis:5.0.4
command
:
- redis-server
-
"/redis-master/redis.conf"
env
:
-
name
:
MASTER
value
:
"true"
ports
:
-
:
6379
resources
:
limits
:
cpu
:
"0.1"
:
-
mountPath
:
/redis-master-data
name
:
data
-
mountPath
:
name
:
config
volumes
:
-
name
:
data
emptyDir
:
{}
-
name
:
config
confi

gMap
:
name
:
items
:
-
key
:
path
:
Examine the created objects:
kubectl get pod/redis configmap/example-redis-config
You should see the following output:
pod/redis   1/1     Running   0          8s

NAME                             DATA   AGE
configmap/example-redis-config   1      14s
Recall that we left
key in the
ConfigMap blank:
kubectl describe configmap/example-redis-config
You should see an empty
key:
Name:         example-redis-config
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
Data
====
redis-config:
Use
to enter the pod and run the
redis-cli
tool to check the current configuration:
kubectl
exec
-it redis -- redis-cli
Check
maxmemory
:
127.0.0.1:6379&gt; CONFIG GET maxmemory
It should show the default value of 0:
1
)
"maxmemory"
2
)
"0"
Similarly, check
maxmemory-policy
:
127.0.0.1:6379&gt; CONFIG GET maxmemory-policy
Which should also yield its default value of
noeviction
:
1
)
"maxmemory-policy"
2
)
"noeviction"
Now let's add some configuration values to the
ConfigMap:
pods/config/example-redis-config.yaml
:
v1
kind
:
ConfigMap
metadata
:
name
:
data
:
:
|
maxmemory 2mb
maxmemory-policy allkeys-lru
Apply the updated ConfigMap:
Confirm that the ConfigMap was updated:
You should see the configuration values we just added:
Data
====
----
Check the Redis Pod again using
redis-cli
via
to see if the configuration was applied:
kubectl
exec
Check
maxmemory
:
It remains at the default value of 0:
1
)
2
)
"0"
Similarly,
remains at the
default setting:
Returns:
1
)
2
)
The configuration values have not changed because the Pod needs to be restarted to grab updated
values from associated ConfigMaps. Let's delete and recreate the Pod:
kubectl delete pod redis
Now re-check the configuration values one last time:
kubectl
exec
Check
maxmemory
:
It should now return the updated value of 2097152:
1
)
2
)
"2097152"
has also been updated:
It now reflects the desired value of
allkeys-lru
:
1
)
2
)
"allkeys-lru"
Clean up your work by deleting the created resources:
kubectl delete pod/redis configmap/example-redis-config
Learn more about
.
Feedback
Yes
No
.
or
.
Last modified May 30, 2022 at 4:42 PM PST:
Update link text to configure-pod-configmap.md (ca34038e9f)
Home
Blog
Training
Partners
Community
CC BY 4.0

Administer a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Learn common tasks for administering a cluster.
1:
1.1:
1.2:
1.3:
1.4:
1.5:
2:
2.1:
2.2:
2.3:
2.4:
2.5:
2.6:
3:
4:
4.1:
4.2:
4.3:
4.4:
4.5:
4.6:
5:
5.1:
5.2:
5.3:
5.4:
5.5:
5.6:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:
1 - Administration with kubeadm
1.1 - Certificate Management with kubeadm
Kubernetes v1.15 [stable]
Client certificates generated by
kubeadm
expire after 1 year.
This page explains how to manage certificate renewals with kubeadm. It also covers other tasks related
to kubeadm certificate management.
You should be familiar with
PKI certificates and requirements in Kubernetes
.
Using custom certificates
By default, kubeadm generates all the certificates needed for a cluster to run.
You can override this behavior by providing your own certificates.
To do so, you must place them in whatever directory is specified by the
--cert-dir
flag or the
certificatesDir
field of kubeadm's
ClusterConfiguration
.
By

 default this is
/etc/kubernetes/pki
.
If a given certificate and private key pair exists before running
,
kubeadm does not overwrite them. This means you can, for example, copy an existing
CA into
/etc/kubernetes/pki/ca.crt
and
/etc/kubernetes/pki/ca.key
,
and kubeadm will use this CA for signing the rest of the certificates.
External CA mode
It is also possible to provide only the
ca.crt
file and not the
ca.key
file (this is only available for the root CA file, not other cert pairs).
If all other certificates and kubeconfig files are in place, kubeadm recognizes
this condition and activates the "External CA" mode. kubeadm will proceed without the
CA key on disk.
Instead, run the controller-manager standalone with
--controllers=csrsigner
and
point to the CA certificate and key.
includes guidance on
setting up a cluster to use an external CA.
Check certificate expiration
You can use the
check-expiration
subcommand to check when certificates expire:
kubeadm certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Dec 30, 2020 23:36 UTC   364d                                    no
apiserver                  Dec 30, 2020 23:36 UTC   364d            ca                      no
apiserver-etcd-client      Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
apiserver-kubelet-client   Dec 30, 2020 23:36 UTC   364d            ca                      no
controller-manager.conf    Dec 30, 2020 23:36 UTC   364d                                    no
etcd-healthcheck-client    Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
etcd-peer                  Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
etcd-server                Dec 30, 2020 23:36 UTC   364d            etcd-ca                 no
front-proxy-client         Dec 30, 2020 23:36 UTC   364d            front-proxy-ca          no
scheduler.conf             Dec 30, 2020 23:36 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Dec 28, 2029 23:36 UTC   9y              no
etcd-ca                 Dec 28, 2029 23:36 UTC   9y              no
front-proxy-ca          Dec 28, 2029 23:36 UTC   9y              no
The command shows expiration/residual time for the client certificates in the
folder and for the client certificate embedded in the kubeconfig files used
by kubeadm (
admin.conf
,
controller-manager.conf
and
scheduler.conf
).
Additionally, kubeadm informs the user if the certificate is externally managed; in this case, the
user should take care of managing certificate renewal manually/using other tools.
Warning:
kubeadm
cannot manage certificates signed by an external CA.
Note:
kubelet.conf
is not included in the list above because kubeadm configures kubelet
for
automatic certificate renewal
with rotatable certificates under
/var/lib/kubelet/pki
.
To repair an expired kubelet client certificate see
Kubelet client certificate rotation fails
.
Warning:
On nodes created with
, prior to kubeadm version 1.17, there is a
bug
where you manually have to modify the
contents of
. After
finishes, you should update
to
point to the rotated kubelet client certificates, by replacing
and
with:
:
/var/lib/kubelet/pki/kubelet-client-current.pem
:
Automatic certificate renewal
kubeadm renews all the certificates during control plane
upgrade
.
This feature is designed for addressing the simplest use cases;
if you don't have specific requirements on certificate renewal and perform Kubernetes version
upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of

 keeping
your cluster up to date and reasonably secure.
Note:
It is a best practice to upgrade your cluster frequently in order to stay secure.
If you have more complex requirements for certificate renewal, you can opt out from the default
behavior by passing
--certificate-renewal=false
to
kubeadm upgrade apply
or to
kubeadm upgrade node
.
Warning:
Prior to kubeadm version 1.17 there is a
bug
where the default value for
--certificate-renewal
is
false
for the
command. In that case, you should explicitly set
--certificate-renewal=true
.
Manual certificate renewal
You can renew your certificates manually at any time with the
kubeadm certs renew
command.
This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in
.
After running the command you should restart the control plane Pods. This is required since
dynamic certificate reload is currently not supported for all components and certificates.
Static Pods
are managed by the local kubelet
and not by the API Server, thus kubectl cannot be used to delete and restart them.
To restart a static Pod you can temporarily remove its manifest file from
/etc/kubernetes/manifests/
and wait for 20 seconds (see the
fileCheckFrequency
value in
KubeletConfiguration struct
.
The kubelet will terminate the Pod if it's no longer in the manifest directory.
You can then move the file back and after another
period, the kubelet will recreate
the Pod and the certificate renewal for the component can complete.
Warning:
If you are running an HA cluster, this command needs to be executed on all the control-plane nodes.
Note:
certs renew
uses the existing certificates as the authoritative source for attributes (Common
Name, Organization, SAN, etc.) instead of the
kubeadm-config
ConfigMap. It is strongly recommended
to keep them both in sync.
provides the following options:
The Kubernetes certificates normally reach their expiration date after one year.
--csr-only
can be used to renew certificates with an external CA by generating certificate
signing requests (without actually renewing certificates in place); see next paragraph for more
information.
It's also possible to renew a single certificate instead of all.
Renew certificates with the Kubernetes certificates API
This section provides more details about how to execute manual certificate renewal using the Kubernetes certificates API.
Caution:
These are advanced topics for users who need to integrate their organization's certificate
infrastructure into a kubeadm-built cluster. If the default kubeadm configuration satisfies your
needs, you should let kubeadm manage certificates instead.
Set up a signer
The Kubernetes Certificate Authority does not work out of the box.
You can configure an external signer such as
cert-manager
,
or you can use the built-in signer.
The built-in signer is part of
.
To activate the built-in signer, you must pass the
--cluster-signing-cert-file
and
--cluster-signing-key-file
flags.
If you're creating a new cluster, you can use a kubeadm
configuration file
:
:
kubeadm.k8s.io/v1beta3
kind
:
controllerManager
:
extraArgs
:
cluster-signing-cert-file
:
cluster-signing-key-file
:
Create certificate signing requests (CSR)
See
Create CertificateSigningRequest
for creating CSRs with the Kubernetes API.
Renew certificates with external CA
This section provide more details about how to execute manual certificate renewal using an external CA.
To better integrate with external CAs, kubeadm can also produce certificate signing requests (CSRs).
A CSR represents a request to a CA for a signed certificate for a client.
In kubeadm terms, any certificate that would normally be signed by an on-disk CA can be produced
as a CSR instead. A CA, however, cannot be produced as a CSR.
You can cre

ate certificate signing requests with
kubeadm certs renew --csr-only
.
Both the CSR and the accompanying private key are given in the output.
You can pass in a directory with
--csr-dir
to output the CSRs to the specified location.
If
--csr-dir
is not specified, the default certificate directory (
) is used.
Certificates can be renewed with
.
As with
, an output directory can be specified with the
--csr-dir
flag.
A CSR contains a certificate's name, domains, and IPs, but it does not specify usages.
It is the responsibility of the CA to specify
the correct cert usages
when issuing a certificate.
In
openssl
this is done with the
openssl ca
command
.
In
cfssl
you specify
usages in the config file
.
After a certificate is signed using your preferred method, the certificate and the private key
must be copied to the PKI directory (by default
).
Certificate authority (CA) rotation
Kubeadm does not support rotation or replacement of CA certificates out of the box.
For more information about manual rotation or replacement of CA, see
manual rotation of CA certificates
.
Enabling signed kubelet serving certificates
By default the kubelet serving certificate deployed by kubeadm is self-signed.
This means a connection from external services like the
metrics-server
to a
kubelet cannot be secured with TLS.
To configure the kubelets in a new kubeadm cluster to obtain properly signed serving
certificates you must pass the following minimal configuration to
:
:
kind
:
---
:
kind
:
serverTLSBootstrap
:
true
If you have already created the cluster you must adapt it by doing the following:
Find and edit the
kubelet-config-1.26
ConfigMap in the
In that ConfigMap, the
kubelet
key has a
document as its value. Edit the KubeletConfiguration document to set
serverTLSBootstrap: true
.
On each node, add the
field in
/var/lib/kubelet/config.yaml
and restart the kubelet with
systemctl restart kubelet
The field
will enable the bootstrap of kubelet serving
certificates by requesting them from the
certificates.k8s.io
API. One known limitation
is that the CSRs (Certificate Signing Requests) for these certificates cannot be automatically
approved by the default signer in the kube-controller-manager -
kubernetes.io/kubelet-serving
.
This will require action from the user or a third party controller.
These CSRs can be viewed using:
kubectl get csr
NAME        AGE     SIGNERNAME                        REQUESTOR                      CONDITION
csr-9wvgt   112s    kubernetes.io/kubelet-serving     system:node:worker-1           Pending
csr-lz97v   1m58s   kubernetes.io/kubelet-serving     system:node:control-plane-1    Pending
To approve them you can do the following:
kubectl certificate approve &lt;CSR-name&gt;
By default, these serving certificate will expire after one year. Kubeadm sets the
field
rotateCertificates
to
true
, which means that close
to expiration a new set of CSRs for the serving certificates will be created and must
be approved to complete the rotation. To understand more see
Certificate Rotation
.
If you are looking for a solution for automatic approval of these CSRs it is recommended
that you contact your cloud provider and ask if they have a CSR signer that verifies
the node identity with an out of band mechanism.
Note:
Third party custom controllers can be used:
kubelet-csr-approver
Such a controller is not a secure mechanism unless

 it not only verifies the CommonName
in the CSR but also verifies the requested IPs and domain names. This would prevent
a malicious actor that has access to a kubelet client certificate to create
CSRs requesting serving certificates for any IP or domain name.
Generating kubeconfig files for additional users
During cluster creation, kubeadm signs the certificate in the
to have
Subject: O = system:masters, CN = kubernetes-admin
.
is a break-glass, super user group that bypasses the authorization layer (e.g. RBAC).
Sharing the
with additional users is
not recommended
!
Instead, you can use the
kubeadm kubeconfig user
command to generate kubeconfig files for additional users.
The command accepts a mixture of command line flags and
kubeadm configuration
options.
The generated kubeconfig will be written to stdout and can be piped to a file using
kubeadm kubeconfig user ... &gt; somefile.conf
.
Example configuration file that can be used with
--config
:
# example.yaml
:
kind
:
# Will be used as the target "cluster" in the kubeconfig
clusterName
:
"kubernetes"
# Will be used as the "server" (IP or DNS name) of this cluster in the kubeconfig
controlPlaneEndpoint
:
"some-dns-address:6443"
# The cluster CA key and certificate will be loaded from this local directory
:
"/etc/kubernetes/pki"
Make sure that these settings match the desired target cluster settings.
To see the settings of an existing cluster use:
kubectl get cm kubeadm-config -n kube-system -o
=
jsonpath
=
"{.data.ClusterConfiguration}"
The following example will generate a kubeconfig file with credentials valid for 24 hours
for a new user
johndoe
that is part of the
appdevs
group:
kubeadm kubeconfig user --config example.yaml --org appdevs --client-name johndoe --validity-period 24h
The following example will generate a kubeconfig file with administrator credentials valid for 1 week:
kubeadm kubeconfig user --config example.yaml --client-name admin --validity-period 168h
1.2 - Configuring a cgroup driver
This page explains how to configure the kubelet cgroup driver to match the container
runtime cgroup driver for kubeadm clusters.
You should be familiar with the Kubernetes
container runtime requirements
.
Configuring the container runtime cgroup driver
The
Container runtimes
page
explains that the
systemd
driver is recommended for kubeadm based setups instead
of the
cgroupfs
driver, because kubeadm manages the kubelet as a systemd service.
The page also provides details on how to set up a number of different container runtimes with the
systemd
driver by default.
Configuring the kubelet cgroup driver
kubeadm allows you to pass a
structure during
.
This
can include the
field which controls the cgroup
driver of the kubelet.
Note:
In v1.22, if the user is not setting the
field under
,
kubeadm
will default it to
systemd
.
A minimal example of configuring the field explicitly:
# kubeadm-config.yaml
kind
:
:
kubernetesVersion
:
v1.21.0
---
kind
:
:
:
systemd
Such a configuration file can then be passed to the kubeadm command:
kubeadm init --config kubeadm-config.yaml
Note:
Kubeadm uses the same
for all nodes in the cluster.
The
is stored in a
ConfigMap
object under the
Executing the sub commands
init
,
join
and
upgrade
would result in kubeadm
writing the
as a file under
and passing it to the local node kubelet.
Using the
cgroupfs
driver
To use
cgroupfs
and to prevent
from modifying the
cgroup driver on existing setups, you must be explicit
about its value. This applies to a case wh

ere you do not wish future versions
of kubeadm to apply the
systemd
See the below section on "
Modify the kubelet ConfigMap
" for details on
how to be explicit about the value.
If you wish to configure a container runtime to use the
cgroupfs
driver,
you must refer to the documentation of the container runtime of your choice.
Migrating to the
systemd
driver
To change the cgroup driver of an existing kubeadm cluster to
systemd
in-place,
a similar procedure to a kubelet upgrade is required. This must include both
steps outlined below.
Note:
Alternatively, it is possible to replace the old nodes in the cluster with new ones
that use the
systemd
driver. This requires executing only the first step below
before joining the new nodes and ensuring the workloads can safely move to the new
nodes before deleting the old nodes.
Call
kubectl edit cm kubelet-config -n kube-system
.
Either modify the existing
value or add a new field that looks like this:
:
systemd
This field must be present under the
kubelet:
section of the ConfigMap.
Update the cgroup driver on all nodes
For each node in the cluster:
Drain the node
using
kubectl drain &lt;node-name&gt; --ignore-daemonsets
Stop the kubelet using
systemctl stop kubelet
Stop the container runtime
Modify the container runtime cgroup driver to
systemd
Set
cgroupDriver: systemd
in
Start the container runtime
Start the kubelet using
systemctl start kubelet
Uncordon the node
using
kubectl uncordon &lt;node-name&gt;
Execute these steps on nodes one at a time to ensure workloads
have sufficient time to schedule on different nodes.
Once the process is complete ensure that all nodes and workloads are healthy.
1.3 - Reconfiguring a kubeadm cluster
kubeadm does not support automated ways of reconfiguring components that
were deployed on managed nodes. One way of automating this would be
by using a custom
operator
.
To modify the components configuration you must manually edit associated cluster
objects and files on disk.
This guide shows the correct sequence of steps that need to be performed
to achieve kubeadm cluster reconfiguration.
You need a cluster that was deployed using kubeadm
Have administrator credentials (
/etc/kubernetes/admin.conf
) and network connectivity
to a running kube-apiserver in the cluster from a host that has kubectl installed
Have a text editor installed on all hosts
Reconfiguring the cluster
kubeadm writes a set of cluster wide component configuration options in
ConfigMaps and other objects. These objects must be manually edited. The command
can be used for that.
The
command will open a text editor where you can edit and save the object directly.
You can use the environment variables
KUBECONFIG
and
to specify the location of
the kubectl consumed kubeconfig file and preferred text editor.
KUBECONFIG=/etc/kubernetes/admin.conf KUBE_EDITOR=nano kubectl edit &lt;parameters&gt;
Note:
Upon saving any changes to these cluster objects, components running on nodes may not be
automatically updated. The steps below instruct you on how to perform that manually.
Warning:
Component configuration in ConfigMaps is stored as unstructured data (YAML string).
This means that validation will not be performed upon updating the contents of a ConfigMap.
You have to be careful to follow the documented API format for a particular
component configuration and avoid introducing typos and YAML indentation mistakes.
Applying cluster configuration changes
Updating the
During cluster creation and upgrade, kubeadm writes its
in a ConfigMap called
in the
To change a particular option in the
you can edit the ConfigMap with this command:
kubectl edit cm -n kube-system kubeadm-config
The configuration is located under the
data.ClusterConfiguration

key.
Note:
The
includes a variety of options that affect the configuration of individual
components such as kube-apiserver, kube-scheduler, kube-controller-manager, CoreDNS, etcd and kube-proxy.
Changes to the configuration must be reflected on node components manually.
Reflecting
changes on control plane nodes
kubeadm manages the control plane components as static Pod manifests located in
the directory
/etc/kubernetes/manifests
.
Any changes to the
under the
apiServer
,
,
scheduler
or
etcd
keys must be reflected in the associated files in the manifests directory on a control plane node.
Such changes may include:
extraArgs
- requires updating the list of flags passed to a component container
- requires updated the volume mounts for a component container
*SANs
- requires writing new certificates with updated Subject Alternative Names.
Before proceeding with these changes, make sure you have backed up the directory
/etc/kubernetes/
.
To write new certificates you can use:
kubeadm init phase certs &lt;component-name&gt; --config &lt;config-file&gt;
To write new manifest files in
you can use:
kubeadm init phase control-plane &lt;component-name&gt; --config &lt;config-file&gt;
The
&lt;config-file&gt;
contents must match the updated
.
The
&lt;component-name&gt;
value must be the name of the component.
Note:
Updating a file in
will tell the kubelet to restart the static Pod for the corresponding component.
Try doing these changes one node at a time to leave the cluster without downtime.
Applying kubelet configuration changes
kubelet-config
in the
You can edit the ConfigMap with this command:
kubectl edit cm -n kube-system kubelet-config
data.kubelet
key.
Reflecting the kubelet changes
To reflect the change on kubeadm nodes you must do the following:
Log in to a kubeadm node
Run
kubeadm upgrade node phase kubelet-config
to download the latest
ConfigMap contents into the local file
/var/lib/kubelet/config.conf
Edit the file
/var/lib/kubelet/kubeadm-flags.env
to apply additional configuration with
flags
Restart the kubelet service with
Note:
Do these changes one node at a time to allow workloads to be rescheduled properly.
Note:
During
, kubeadm downloads the
from the
ConfigMap and overwrite the contents of
.
This means that node local configuration must be applied either by flags in
or by manually updating the contents of
after
, and then restarting the kubelet.
Applying kube-proxy configuration changes
in a ConfigMap in the
namespace called
.
This ConfigMap is used by the
DaemonSet in the
, you can edit the ConfigMap with this command:
kubectl edit cm -n kube-system kube-proxy
data.config.conf
key.
Reflecting the kube-proxy changes
Once the
ConfigMap is updated, you can restart all kube-proxy Pods:
Obtain the Pod names:
kubectl get po -n kube-system | grep kube-proxy
Delete a Pod with:
kubectl delete po -n kube-system &lt;pod-name&gt;
New Pods that use the updated ConfigMap will be created.
Note:
Because kubeadm deploys kube-proxy as a DaemonSet, node specific configuration is unsupported.
Applying CoreDNS configuration changes
Updating the CoreDNS Deployment and Service
kubeadm deploys CoreDNS as a Deplo

yment called
coredns
and with a Service
kube-dns
,
both in the
To update any of the CoreDNS settings, you can edit the Deployment and
Service objects:
kubectl edit deployment -n kube-system coredns
kubectl edit service -n kube-system kube-dns
Reflecting the CoreDNS changes
Once the CoreDNS changes are applied you can delete the CoreDNS Pods:
kubectl get po -n kube-system | grep coredns
New Pods with the updated CoreDNS configuration will be created.
Note:
kubeadm does not allow CoreDNS configuration during cluster creation and upgrade.
This means that if you execute
, your changes to the CoreDNS
objects will be lost and must be reapplied.
Persisting the reconfiguration
During the execution of
on a managed node, kubeadm might overwrite configuration
that was applied after the cluster was created (reconfiguration).
Persisting Node object reconfiguration
kubeadm writes Labels, Taints, CRI socket and other information on the Node object for a particular
Kubernetes node. To change any of the contents of this Node object you can use:
kubectl edit no &lt;node-name&gt;
During
the contents of such a Node might get overwritten.
If you would like to persist your modifications to the Node object after upgrade,
you can prepare a
and apply it to the Node object:
kubectl patch no &lt;node-name&gt; --patch-file &lt;patch-file&gt;
Persisting control plane component reconfiguration
The main source of control plane configuration is the
object stored in the cluster. To extend the static Pod manifests configuration,
patches
can be used.
These patch files must remain as files on the control plane nodes to ensure that
they can be used by the
kubeadm upgrade ... --patches &lt;directory&gt;
.
If reconfiguration is done to the
and static Pod manifests on disk,
the set of node specific patches must be updated accordingly.
Persisting kubelet reconfiguration
stored in
will be overwritten on
by downloading the contents of the cluster wide
ConfigMap.
To persist kubelet node specific configuration either the file
has to be updated manually post-upgrade or the file
can include flags.
The kubelet flags override the associated
options, but note that
some of the flags are deprecated.
A kubelet restart will be required after changing
or
.
Certificate management with kubeadm
Find more about kubeadm set-up
1.4 - Upgrading kubeadm clusters
This page explains how to upgrade a Kubernetes cluster created with kubeadm from version
1.25.x to version 1.26.x, and from version
1.26.x to 1.26.y (where
y &gt; x
). Skipping MINOR versions
when upgrading is unsupported. For more details, please visit
Version Skew Policy
.
To see information about upgrading clusters created using older versions of kubeadm,
please refer to following pages instead:
Upgrading a kubeadm cluster from 1.24 to 1.25
Upgrading a kubeadm cluster from 1.23 to 1.24
Upgrading a kubeadm cluster from 1.22 to 1.23
Upgrading a kubeadm cluster from 1.21 to 1.22
The upgrade workflow at high level is the following:
Upgrade a primary control plane node.
Upgrade additional control plane nodes.
Upgrade worker nodes.
Make sure you read the
release notes
carefully.
The cluster should use a static control plane and etcd pods or external etcd.
Make sure to back up any important components, such as app-level state stored in a database.
does not touch your workloads, only components internal to Kubernetes, but backups are always a best practice.

Swap must be disabled
.
Additional information
The instructions below outline when to drain each node during the upgrade process.
If you are performing a
minor
version upgrade for any kubelet, you
must
first drain the node (or nodes) that you are upgrading. In the case of control plane nodes,
they could be running CoreDNS Pods or other critical workloads. For more information see
Draining nodes
.
All containers are restarted after upgrade, because the container spec hash value is changed.
To verify that the kubelet service has successfully restarted after the kubelet has been upgraded,
you can execute
systemctl status kubelet
or view the service logs with
journalctl -xeu kubelet
.
Usage of the
--config
flag of
with
kubeadm configuration API types
with the purpose of reconfiguring the cluster is not recommended and can have unexpected results. Follow the steps in
instead.
Determine which version to upgrade to
Find the latest patch release for Kubernetes 1.26 using the OS package manager:
Ubuntu, Debian or HypriotOS
CentOS, RHEL or Fedora
apt update
apt-cache madison kubeadm
# find the latest 1.26 version in the list
# it should look like 1.26.x-00, where x is the latest patch
yum list --showduplicates kubeadm --disableexcludes=kubernetes
# it should look like 1.26.x-0, where x is the latest patch
Upgrading control plane nodes
The upgrade procedure on control plane nodes should be executed one node at a time.
Pick a control plane node that you wish to upgrade first. It must have the
file.
Call "kubeadm upgrade"
For the first control plane node
Upgrade kubeadm:
# replace x in 1.26.x-00 with the latest patch version
apt-mark unhold kubeadm
&amp;&amp;
\
apt-get install -y
kubeadm
=
1.26.x-00
\
apt-mark hold kubeadm
# replace x in 1.26.x-0 with the latest patch version
yum install -y kubeadm-1.26.x-0 --disableexcludes
=
kubernetes
Verify that the download works and has the expected version:
Verify the upgrade plan:
kubeadm upgrade plan
This command checks that your cluster can be upgraded, and fetches the versions you can upgrade to.
It also shows a table with the component config version states.
Note:
also automatically renews the certificates that it manages on this node.
To opt-out of certificate renewal the flag
For more information see the
certificate management guide
.
Note:
If
shows any component configs that require manual upgrade, users must provide
a config file with replacement configs to
via the
--config
command line flag.
Failing to do so will cause
to exit with an error and not perform an upgrade.
Choose a version to upgrade to, and run the appropriate command. For example:
# replace x with the patch version you picked for this upgrade
sudo kubeadm upgrade apply v1.26.x
Once the command finishes you should see:
[upgrade/successful] SUCCESS! Your cluster was upgraded to "v1.26.x". Enjoy!

[upgrade/kubelet] Now that your control plane is upgraded, please proceed with upgrading your kubelets if you haven't already done so.
Manually upgrade your CNI provider plugin.
Your Container Network Interface (CNI) provider may have its own upgrade instructions to follow.
Check the
addons
page to
find your CNI provider and see whether additional upgrade steps are required.
This step is not required on additional control plane nodes if the CNI provider runs as a DaemonSet.
For the other control plane nodes
Same as the first control plane node but use:
sudo kubeadm upgrade node
instead of:
sudo kubeadm upgrade apply
Also calling
and upgrading the CNI provider plugin is no longer needed.
Prepare the node for maintenance by marking it unschedulable and evicting the

 workloads:
# replace &lt;node-to-drain&gt; with the name of your node you are draining
kubectl drain &lt;node-to-drain&gt; --ignore-daemonsets
Upgrade kubelet and kubectl
Upgrade the kubelet and kubectl:
apt-mark unhold kubelet kubectl
\
kubelet
=
1.26.x-00
kubectl
=
1.26.x-00
\
apt-mark hold kubelet kubectl
yum install -y kubelet-1.26.x-0 kubectl-1.26.x-0 --disableexcludes
=
Restart the kubelet:
sudo systemctl daemon-reload
sudo systemctl restart kubelet
Bring the node back online by marking it schedulable:
# replace &lt;node-to-uncordon&gt; with the name of your node
kubectl uncordon &lt;node-to-uncordon&gt;
Upgrade worker nodes
The upgrade procedure on worker nodes should be executed one node at a time or few nodes at a time,
without compromising the minimum required capacity for running your workloads.
Upgrade kubeadm
\
kubeadm
=
1.26.x-00
\
=
For worker nodes this upgrades the local kubelet configuration:
Prepare the node for maintenance by marking it unschedulable and evicting the workloads:
\
kubelet
=
1.26.x-00
kubectl
=
1.26.x-00
\
=
Verify the status of the cluster
After the kubelet is upgraded on all nodes verify that all nodes are available again by running
the following command from anywhere kubectl can access the cluster:
kubectl get nodes
The
STATUS
column should show
Ready
for all your nodes, and the version number should be updated.
Recovering from a failure state
If
fails and does not roll back, for example because of an unexpected shutdown during execution, you can run
again.
This command is idempotent and eventually makes sure that the actual state is the desired state you declare.
To recover from a bad state, you can also run
kubeadm upgrade apply --force
without changing the version that your cluster is running.
During upgrade kubeadm writes the following backup folders under
/etc/kubernetes/tmp
:
kubeadm-backup-etcd-&lt;date&gt;-&lt;time&gt;
kubeadm-backup-manifests-&lt;date&gt;-&lt;time&gt;
kubeadm-backup-etcd
contains a backup of the local etcd member data for this control plane Node.
In case of an etcd upgrade failure and if the automatic rollback does not work, the contents of this folder
can be manually restored in
/var/lib/etcd
. In case external etcd is used this backup folder will be empty.
kubeadm-backup-manifests
contains a backup of the static Pod manifest files for this control plane Node.
In case of a upgrade failure and if the automatic rollback does not work, the contents of this folder can be
manu

ally restored in
. If for some reason there is no difference between a pre-upgrade
and post-upgrade manifest file for a certain component, a backup file for it will not be written.
How it works
does the following:
Checks that your cluster is in an upgradeable state:
The API server is reachable
All nodes are in the
Ready
state
The control plane is healthy
Enforces the version skew policies.
Makes sure the control plane images are available or available to pull to the machine.
Generates replacements and/or uses user supplied overwrites if component configs require version upgrades.
Upgrades the control plane components or rollbacks if any of them fails to come up.
Applies the new
CoreDNS
and
manifests and makes sure that all necessary RBAC rules are created.
Creates new certificate and key files of the API server and backs up old files if they're about to expire in 180 days.
does the following on additional control plane nodes:
Fetches the kubeadm
from the cluster.
Optionally backups the kube-apiserver certificate.
Upgrades the static Pod manifests for the control plane components.
Upgrades the kubelet configuration for this node.
does the following on worker nodes:
1.5 - Upgrading Windows nodes
Kubernetes v1.18 [beta]
This page explains how to upgrade a Windows node created with kubeadm.
minikube
Your Kubernetes server must be at or later than version 1.17.
.
Familiarize yourself with
the process for upgrading the rest of your kubeadm
cluster
. You will want to
upgrade the control plane nodes before upgrading your Windows nodes.
Upgrading worker nodes
From the Windows node, upgrade kubeadm:
# replace v1.26.0 with your desired version
curl.exe -Lo &lt;
path-to
-kubeadm.exe&gt;
"https://dl.k8s.io/v1.26.0/bin/windows/amd64/kubeadm.exe"
From a machine with access to the Kubernetes API,
prepare the node for maintenance by marking it unschedulable and evicting the workloads:
You should see output similar to this:
node/ip-172-31-85-18 cordoned
node/ip-172-31-85-18 drained
Upgrade the kubelet configuration
From the Windows node, call the following command to sync new kubelet configuration:
Upgrade kubelet and kube-proxy
From the Windows node, upgrade and restart the kubelet:
stop-service
kubelet
path-to
-kubelet.exe&gt;
"https://dl.k8s.io/v1.26.0/bin/windows/amd64/kubelet.exe"
restart-service
kubelet
From the Windows node, upgrade and restart the kube-proxy.
path-to
-kube-proxy.exe&gt;
"https://dl.k8s.io/v1.26.0/bin/windows/amd64/kube-proxy.exe"
Note:
If you are running kube-proxy in a HostProcess container within a Pod, and not as a Windows Service,
you can upgrade kube-proxy by applying a newer version of your kube-proxy manifests.
bring the node back online by marking it schedulable:
# replace &lt;node-to-drain&gt; with the name of your node
kubectl uncordon &lt;node-to-drain&gt;
2 - Migrating from dockershim
This section presents information you need to know when migrating from
dockershim to other container ru

ntimes.
Since the announcement of
dockershim deprecation
in Kubernetes 1.20, there were questions on how this will affect various workloads and Kubernetes
installations. Our
Dockershim Removal FAQ
is there to help you
to understand the problem better.
Dockershim was removed from Kubernetes with the release of v1.24.
If you use Docker Engine via dockershim as your container runtime and wish to upgrade to v1.24,
it is recommended that you either migrate to another runtime or find an alternative means to obtain Docker Engine support.
Check out the
container runtimes
section to know your options. Make sure to
report issues
you encountered
with the migration so the issues can be fixed in a timely manner and your cluster would be
ready for dockershim removal.
Your cluster might have more than one kind of node, although this is not a common
configuration.
These tasks will help you to migrate:
Check whether Dockershim removal affects you
Check out
to understand your options for an alternative.
There is a
GitHub issue
to track the discussion about the deprecation and removal of dockershim.
If you find a defect or other technical concern relating to migrating away from dockershim,
you can
report an issue
to the Kubernetes project.
2.1 - Changing the Container Runtime on a Node from Docker Engine to containerd
This task outlines the steps needed to update your container runtime to containerd from Docker. It
is applicable for cluster operators running Kubernetes 1.23 or earlier. This also covers an
example scenario for migrating from dockershim to containerd. Alternative container runtimes
can be picked from this
page
.
Note:
Install containerd. For more information see
containerd's installation documentation
and for specific prerequisite follow
the containerd guide
.
Replace
&lt;node-to-drain&gt;
with the name of your node you are draining.
Stop the Docker daemon
systemctl disable docker.service --now
Install Containerd
Follow the
guide
for detailed steps to install containerd.
Linux
Windows (PowerShell)
Install the
containerd.io
package from the official Docker repositories.
Instructions for setting up the Docker repository for your respective Linux distribution and
installing the
package can be found at
Getting started with containerd
.
Configure containerd:
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
Restart containerd:
sudo systemctl restart containerd
Start a Powershell session, set
$Version
to the desired version (ex:
$Version="1.4.3"
), and
then run the following commands:
Download containerd:
curl.exe -L https
:
//github.com/containerd/containerd/releases/download/v
$Version
/containerd-
$Version
-windows-amd64.tar.gz -o
containerd-windows
-amd64.tar.gz
tar.exe xvf .\
Extract and configure:
Copy-Item
-Path
".\bin\"
-Destination
"$Env:ProgramFiles\containerd"
-Recurse -Force
cd
$Env:ProgramFiles
\containerd\
.\containerd.exe config
default
|
Out-File
config.toml -Encoding ascii
# Review the configuration. Depending on setup you may want to adjust:
# - the sandbox_image (Kubernetes pause image)
# - cni bin_dir and conf_dir locations
Get-Content
config.toml
# (Optional - but highly recommended) Exclude containerd from Windows Defender Scans
Add-MpPreference
-ExclusionProcess
"$Env:ProgramFiles\containerd\containerd.exe"
Start containerd:
.\containerd.exe --register-service
Start-Service
contai

nerd
Configure the kubelet to use containerd as its container runtime
and add the containerd runtime to the flags.
--container-runtime=remote
and
--container-runtime-endpoint=unix:///run/containerd/containerd.sock
.
Users using kubeadm should be aware that the
kubeadm
tool stores the CRI socket for each host as
an annotation in the Node object for that host. To change it you can execute the following command
on a machine that has the kubeadm
file.
This will start a text editor where you can edit the Node object.
To choose a text editor you can set the
environment variable.
Change the value of
kubeadm.alpha.kubernetes.io/cri-socket
from
/var/run/dockershim.sock
to the CRI socket path of your choice (for example
unix:///run/containerd/containerd.sock
).
Note that new CRI socket paths must be prefixed with
unix://
ideally.
Save the changes in the text editor, which will update the Node object.
Restart the kubelet
Verify that the node is healthy
Run
kubectl get nodes -o wide
and containerd appears as the runtime for the node we just changed.
Remove Docker Engine
Note:
Finally if everything goes well, remove Docker.
CentOS
Debian
Fedora
Ubuntu
sudo yum remove docker-ce docker-ce-cli
sudo apt-get purge docker-ce docker-ce-cli
sudo dnf remove docker-ce docker-ce-cli
The preceding commands don't remove images, containers, volumes, or customized configuration files on your host.
To delete them, follow Docker's instructions to
Uninstall Docker Engine
.
Caution:
Docker's instructions for uninstalling Docker Engine create a risk of deleting containerd. Be careful when executing commands.
2.2 - Migrate Docker Engine nodes from dockershim to cri-dockerd
Note:
This page shows you how to migrate your Docker Engine nodes to use
cri-dockerd
instead of dockershim. You should follow these steps in these scenarios:
You want to switch away from dockershim and still use Docker Engine to run
containers in Kubernetes.
You want to upgrade to Kubernetes v1.26 and your
existing cluster relies on dockershim, in which case you must migrate
from dockershim and
is one of your options.
To learn more about the removal of dockershim, read the
FAQ page
.
What is cri-dockerd?
In Kubernetes 1.23 and earlier, you could use Docker Engine with Kubernetes,
relying on a built-in component of Kubernetes named
dockershim
.
The dockershim component was removed in the Kubernetes 1.24 release; however,
a third-party replacement,
, is available. The
adapter
lets you use Docker Engine through the
Container Runtime Interface
.
Note:
If you already use
, you aren't affected by the dockershim removal.
Before you begin,
Check whether your nodes use the dockershim
.
If you want to migrate to
so that you can continue using Docker
Engine as your container runtime, you should do the following for each affected
node:
Install
.
Cordon and drain the node.
Configure the kubelet to use
.
Restart the kubelet.
Verify that the node is healthy.
Test the migration on non-critical nodes first.
You should perform the following steps for each node that you want to migrate
to
.
installed and started on each

 node.
A
network plugin
.
Cordon and drain the node
Cordon the node to stop new Pods scheduling on it:
kubectl cordon &lt;NODE_NAME&gt;
Replace
&lt;NODE_NAME&gt;
with the name of the node.
Drain the node to safely evict running Pods:
kubectl drain &lt;NODE_NAME&gt;
\
--ignore-daemonsets
Configure the kubelet to use cri-dockerd
The following steps apply to clusters set up using the kubeadm tool. If you use
a different tool, you should modify the kubelet using the configuration
instructions for that tool.
Open
on each affected node.
Modify the
--container-runtime-endpoint
flag to
unix:///var/run/cri-dockerd.sock
.
The kubeadm tool stores the node's socket as an annotation on the
Node
object
in the control plane. To modify this socket for each affected node:
Edit the YAML representation of the
Node
object:
=
/path/to/admin.conf kubectl edit no &lt;NODE_NAME&gt;
Replace the following:
/path/to/admin.conf
: the path to the kubectl configuration file,
.
: the name of the node you want to modify.
Change
from
to
.
Save the changes. The
Node
object is updated on save.
To check whether the node uses the
endpoint, follow the
instructions in
Find out which runtime you use
.
The
flag for the kubelet should be
.
Uncordon the node to let Pods schedule on it:
kubectl uncordon &lt;NODE_NAME&gt;
Read the
dockershim removal FAQ
.
Learn how to migrate from Docker Engine with dockershim to containerd
.
2.3 - Find Out What Container Runtime is Used on a Node
This page outlines steps to find out what
the nodes in your cluster use.
Depending on the way you run your cluster, the container runtime for the nodes may
have been pre-configured or you need to configure it. If you're using a managed
Kubernetes service, there might be vendor-specific ways to check what container runtime is
configured for the nodes. The method described on this page should work whenever
the execution of
kubectl
is allowed.
Install and configure
kubectl
. See
section for details.
Find out the container runtime used on a Node
Use
kubectl
to fetch and show node information:
The output is similar to the following. The column
CONTAINER-RUNTIME
outputs
the runtime and its version.
For Docker Engine, the output is similar to this:
NAME         STATUS   VERSION    CONTAINER-RUNTIME
node-1       Ready    v1.16.15   docker://19.3.1
node-2       Ready    v1.16.15   docker://19.3.1
node-3       Ready    v1.16.15   docker://19.3.1
If your runtime shows as Docker Engine, you still might not be affected by the
removal of dockershim in Kubernetes v1.24.
Check the runtime
endpoint
to see if you use dockershim. If you don't use
dockershim, you aren't affected.
For containerd, the output is similar to this:
NAME         STATUS   VERSION   CONTAINER-RUNTIME
node-1       Ready    v1.19.6   containerd://1.4.1
node-2       Ready    v1.19.6   containerd://1.4.1
node-3       Ready    v1.19.6   containerd://1.4.1
Find out more information about container runtimes
on
page.
Find out what container runtime endpoint you use
The container runtime talks to the kubelet over a Unix socket using the
CRI
protocol
, which is based on the gRPC
framework. The kubelet acts as a client, and the runtime acts as the server.
In some cases, you might find it useful to know which socket your nodes use. For
example, with the removal of dockershim in Kubernetes v1.24 and later, you might
want to know whether you use Docker Engine with dockershim.
Note:
If you currently use Docker Engine in your nodes with
, you aren't
affected by the dockershim removal.
You can check which

 socket you use by checking the kubelet configuration on your
nodes.
Read the starting commands for the kubelet process:
tr \\0 ' ' &lt; /proc/"$(pgrep kubelet)"/cmdline
If you don't have
tr
or
pgrep
, check the command line for the kubelet
process manually.
In the output, look for the
--container-runtime
flag and the
flag.
If your nodes use Kubernetes v1.23 and earlier and these flags aren't
present or if the
flag is not
remote
,
you use the dockershim socket with Docker Engine.
If the
flag is present, check the socket
name to find out which runtime you use. For example,
is the containerd endpoint.
If you want to change the Container Runtime on a Node from Docker Engine to containerd,
you can find out more information on
migrating from Docker Engine to containerd
,
or, if you want to continue using Docker Engine in Kubernetes v1.24 and later, migrate to a
CRI-compatible adapter like
.
2.4 - Troubleshooting CNI plugin-related errors
To avoid CNI plugin-related errors, verify that you are using or upgrading to a
container runtime that has been tested to work correctly with your version of
Kubernetes.
About the "Incompatible CNI versions" and "Failed to destroy network for sandbox" errors
Service issues exist for pod CNI network setup and tear down in containerd
v1.6.0-v1.6.3 when the CNI plugins have not been upgraded and/or the CNI config
version is not declared in the CNI config files. The containerd team reports, "these issues are resolved in containerd v1.6.4."
With containerd v1.6.0-v1.6.3, if you do not upgrade the CNI plugins and/or
declare the CNI config version, you might encounter the following "Incompatible
CNI versions" or "Failed to destroy network for sandbox" error conditions.
Incompatible CNI versions error
If the version of your CNI plugin does not correctly match the plugin version in
the config because the config version is later than the plugin version, the
containerd log will likely show an error message on startup of a pod similar
to:
incompatible CNI versions; config is \"1.0.0\", plugin supports [\"0.1.0\" \"0.2.0\" \"0.3.0\" \"0.3.1\" \"0.4.0\"]"
To fix this issue,
update your CNI plugins and CNI config files
.
Failed to destroy network for sandbox error
If the version of the plugin is missing in the CNI plugin config, the pod may
run. However, stopping the pod generates an error similar to:
ERRO[2022-04-26T00:43:24.518165483Z] StopPodSandbox for "b" failed
error="failed to destroy network for sandbox \"bbc85f891eaf060c5a879e27bba9b6b06450210161dfdecfbb2732959fb6500a\": invalid version \"\": the version is empty"
This error leaves the pod in the not-ready state with a network namespace still
attached. To recover from this problem,
edit the CNI config file
to add
the missing version information. The next attempt to stop the pod should
be successful.
Updating your CNI plugins and CNI config files
If you're using containerd v1.6.0-v1.6.3 and encountered "Incompatible CNI
versions" or "Failed to destroy network for sandbox" errors, consider updating
your CNI plugins and editing the CNI config files.
Here's an overview of the typical steps for each node:
Safely drain and cordon the
node
.
After stopping your container runtime and kubelet services, perform the
following upgrade operations:
If you're running CNI plugins, upgrade them to the latest version.
If you're using non-CNI plugins, replace them with CNI plugins. Use the
latest version of the plugins.
Update the plugin configuration file to specify or match a version of the
CNI specification that the plugin supports, as shown in the following
"An
example containerd configuration
file"
section.
For
, ensure that you have installed the latest version (v1.0.0
or later) of the CNI loopback plugin.
Upgrade node components (for example, the kubelet) to Kubernetes v1.24
Upgrade to or install the most current version of the 

container runtime.
Bring the node back into your cluster by restarting your container runtime
and kubelet. Uncordon the node (
kubectl uncordon &lt;nodename&gt;
).
An example containerd configuration file
The following example shows a configuration for
runtime v1.6.x,
which supports a recent version of the CNI specification (v1.0.0).
Please see the documentation from your plugin and networking provider for
further instructions on configuring your system.
On Kubernetes, containerd runtime adds a loopback interface,
lo
, to pods as a
default behavior. The containerd runtime configures the loopback interface via a
CNI plugin,
loopback
. The
loopback
plugin is distributed as part of the
release packages that have the
cni
designation.
v1.6.0 and later includes a CNI v1.0.0-compatible loopback plugin as well as
other default CNI plugins. The configuration for the loopback plugin is done
internally by containerd, and is set to use CNI v1.0.0. This also means that the
loopback
plugin must be v1.0.0 or later when this newer version
is started.
The following bash command generates an example CNI config. Here, the 1.0.0
value for the config version is assigned to the
cniVersion
field for use when
invokes the CNI bridge plugin.
cat
&lt;&lt; EOF | tee /etc/cni/net.d/10-containerd-net.conflist
{
"cniVersion": "1.0.0",
"name": "containerd-net",
"plugins": [
{
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"promiscMode": true,
"ipam": {
"type": "host-local",
"ranges": [
[{
"subnet": "10.88.0.0/16"
}],
[{
"subnet": "2001:db8:4860::/64"
}]
],
"routes": [
{ "dst": "0.0.0.0/0" },
{ "dst": "::/0" }
]
}
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"externalSetMarkChain": "KUBE-MARK-MASQ"
}
]
}
EOF
Update the IP address ranges in the preceding example with ones that are based
on your use case and network addressing plan.
2.5 - Check whether dockershim removal affects you
The
component of Kubernetes allows to use Docker as a Kubernetes's
.
Kubernetes' built-in
component was removed in release v1.24.
This page explains how your cluster could be using Docker as a container runtime,
provides details on the role that
plays when in use, and shows steps
you can take to check whether any workloads could be affected by
removal.
Finding if your app has a dependencies on Docker
If you are using Docker for building your application containers, you can still
run these containers on any container runtime. This use of Docker does not count
as a dependency on Docker as a container runtime.
When alternative container runtime is used, executing Docker commands may either
not work or yield unexpected output. This is how you can find whether you have a
dependency on Docker:
Make sure no privileged Pods execute Docker commands (like
docker ps
),
restart the Docker service (commands such as
systemctl restart docker.service
),
or modify Docker-specific files such as
/etc/docker/daemon.json
.
Check for any private registries or image mirror settings in the Docker
configuration file (like
). Those typically need to
be reconfigured for another container runtime.
Check that scripts and apps running on nodes outside of your Kubernetes
infrastructure do not execute Docker commands. It might be:
SSH to nodes to troubleshoot;
Node startup scripts;
Monitoring and security agents installed on nodes directly.
Third-party tools that perform above mentioned privileged operations. See
for more information.
Make sure there is no indirect dependencies on dockershim behavior.
This is an edge case and unlikely to affect your application. Some tooling may be configured
to react to Docker-specific behaviors, for example, raise alert on specific metrics or search for
a specific log message as part of troubleshooting instructions.
If you have 

such tooling configured, test the behavior on test
cluster before migration.
Dependency on Docker explained
A
is software that can
execute the containers that make up a Kubernetes pod. Kubernetes is responsible for orchestration
and scheduling of Pods; on each node, the
kubelet
uses the container runtime interface as an abstraction so that you can use any compatible
In its earliest releases, Kubernetes offered compatibility with one container runtime: Docker.
Later in the Kubernetes project's history, cluster operators wanted to adopt additional container runtimes.
The CRI was designed to allow this kind of flexibility - and the kubelet began supporting CRI. However,
because Docker existed before the CRI specification was invented, the Kubernetes project created an
adapter component,
. The dockershim adapter allows the kubelet to interact with Docker as
if Docker were a CRI compatible runtime.
You can read about it in
Kubernetes Containerd integration goes GA
blog post.
Switching to Containerd as a container runtime eliminates the middleman. All the
same containers can be run by container runtimes like Containerd as before. But
now, since containers schedule directly with the container runtime, they are not visible to Docker.
So any Docker tooling or fancy UI you might have used
before to check on these containers is no longer available.
You cannot get container information using
docker ps
or
docker inspect
commands. As you cannot list containers, you cannot get logs, stop containers,
or execute something inside container using
.
Note:
If you're running workloads via Kubernetes, the best way to stop a container is through
the Kubernetes API rather than directly through the container runtime (this advice applies
for all container runtimes, not only Docker).
You can still pull images or build them using
docker build
command. But images
built or pulled by Docker would not be visible to container runtime and
Kubernetes. They needed to be pushed to some registry to allow them to be used
by Kubernetes.
Read
to understand your next steps
Read the
dockershim deprecation FAQ
article for more information.
2.6 - Migrating telemetry and security agents from dockershim
Note:
Kubernetes' support for direct integration with Docker Engine is deprecated and
has been removed. Most apps do not have a direct dependency on runtime hosting
containers. However, there are still a lot of telemetry and monitoring agents
that have a dependency on Docker to collect containers metadata, logs, and
metrics. This document aggregates information on how to detect these
dependencies as well as links on how to migrate these agents to use generic tools or
alternative runtimes.
Telemetry and security agents
Within a Kubernetes cluster there are a few different ways to run telemetry or
security agents. Some agents have a direct dependency on Docker Engine when
they run as DaemonSets or directly on nodes.
Why do some telemetry agents communicate with Docker Engine?
Historically, Kubernetes was written to work specifically with Docker Engine.
Kubernetes took care of networking and scheduling, relying on Docker Engine for
launching and running containers (within Pods) on a node. Some information that
is relevant to telemetry, such as a pod name, is only available from Kubernetes
components. Other data, such as container metrics, is not the responsibility of
the container runtime. Early telemetry agents needed to query the container
runtime
and
Kubernetes to report an accurate picture. Over time, Kubernetes
gained the ability to support multiple runtimes, and now supports any runti

me
that is compatible with the
container runtime interface
.
Some telemetry agents rely specifically on Docker Engine tooling. For example, an agent
might run a command such as
docker ps
or
docker top
to list
containers and processes or
docker logs
to receive streamed logs. If nodes in your existing cluster use
Docker Engine, and you switch to a different container runtime,
these commands will not work any longer.
Identify DaemonSets that depend on Docker Engine
If a pod wants to make calls to the
dockerd
running on the node, the pod must either:
mount the filesystem containing the Docker daemon's privileged socket, as a
volume
; or
mount the specific path of the Docker daemon's privileged socket directly, also as a volume.
For example: on COS images, Docker exposes its Unix domain socket at
/var/run/docker.sock
This means that the pod spec will include a
hostPath
volume mount of
.
Here's a sample shell script to find Pods that have a mount directly mapping the
Docker socket. This script outputs the namespace and name of the pod. You can
remove the
grep '/var/run/docker.sock'
to review other mounts.
kubectl get pods --all-namespaces
\
-o
=
jsonpath
=
'{range .items[*]}{"\n"}{.metadata.namespace}{":\t"}{.metadata.name}{":\t"}{range .spec.volumes[*]}{.hostPath.path}{", "}{end}{end}'
\
| sort
\
| grep
'/var/run/docker.sock'
Note:
There are alternative ways for a pod to access Docker on the host. For instance, the parent
directory
/var/run
may be mounted instead of the full path (like in
this
example
).
The script above only detects the most common uses.
Detecting Docker dependency from node agents
If your cluster nodes are customized and install additional security and
telemetry agents on the node, check with the agent vendor
to verify whether it has any dependency on Docker.
Telemetry and security agent vendors
This section is intended to aggregate information about various telemetry and
security agents that may have a dependency on container runtimes.
We keep the work in progress version of migration instructions for various telemetry and security agent vendors
in
Google doc
.
Please contact the vendor to get up to date instructions for migrating from dockershim.
Migration from dockershim
Aqua
No changes are needed: everything should work seamlessly on the runtime switch.
Datadog
How to migrate:
Docker deprecation in Kubernetes
The pod that accesses Docker Engine may have a name containing any of:
datadog-agent
datadog
dd-agent
Dynatrace
Migrating from Docker-only to generic container metrics in Dynatrace
Containerd support announcement:
Get automated full-stack visibility into
containerd-based Kubernetes
environments
CRI-O support announcement:
Get automated full-stack visibility into your CRI-O Kubernetes containers (Beta)
The pod accessing Docker may have name containing:
dynatrace-oneagent
Falco
Migrate Falco from dockershim
Falco supports any CRI-compatible runtime (containerd is used in the default configuration); the documentation explains all details.
falco
Prisma Cloud Compute
Check
documentation for Prisma Cloud
,
under the "Install Prisma Cloud on a CRI (non-Docker) cluster" section.
The pod accessing Docker may be named like:
twistlock-defender-ds
SignalFx (Splunk)
The SignalFx Smart Agent (deprecated) uses several different monitors for Kubernetes including
kubernetes-cluster
,
kubelet-stats/kubelet-metrics
, and
docker-container-stats
.
The
kubelet-stats
monitor was previously deprecated by the vendor, in favor of
kubelet-metrics
.
The
monitor is the one affected by dockershim removal.
Do not use the
with container runtimes other than Docker Engine.
How to migrate from dockershim-dependent agent:
Remove
from the list of
configured monitors
.
Note, keeping this monitor enabled with non-dockershim runtime will result in incorrect metrics
being re

ported when docker is installed on node and no metrics when docker is not installed.
Enable and configure
monitor.
Note:
The set of collected metrics will change. Review your alerting rules and dashboards.
The Pod accessing Docker may be named something like:
signalfx-agent
Yahoo Kubectl Flame
Flame does not support container runtimes other than Docker. See
https://github.com/yahoo/kubectl-flame/issues/51
3 - Generate Certificates Manually
When using client certificate authentication, you can generate certificates
manually through
easyrsa
,
openssl
or
cfssl
.
easyrsa
easyrsa
can manually generate certificates for your cluster.
Download, unpack, and initialize the patched version of
easyrsa3
.
curl -LO https://storage.googleapis.com/kubernetes-release/easy-rsa/easy-rsa.tar.gz
tar xzf easy-rsa.tar.gz
cd
easy-rsa-master/easyrsa3
./easyrsa init-pki
Generate a new certificate authority (CA).
--batch
sets automatic mode;
--req-cn
specifies the Common Name (CN) for the CA's new root certificate.
./easyrsa --batch
"--req-cn=
${
MASTER_IP
}
@`date +%s`"
build-ca nopass
Generate server certificate and key.
The argument
--subject-alt-name
sets the possible IPs and DNS names the API server will
be accessed with. The
MASTER_CLUSTER_IP
is usually the first IP from the service CIDR
that is specified as the
--service-cluster-ip-range
argument for both the API server and
the controller manager component. The argument
--days
is used to set the number of days
after which the certificate expires.
The sample below also assumes that you are using
cluster.local
as the default
DNS domain name.
./easyrsa --subject-alt-name
=
"IP:
${
MASTER_IP
}
,"
\
"IP:
${
}
,"
\
"DNS:kubernetes,"
\
"DNS:kubernetes.default,"
\
"DNS:kubernetes.default.svc,"
\
"DNS:kubernetes.default.svc.cluster,"
\
"DNS:kubernetes.default.svc.cluster.local"
\
--days
=
10000
\
build-server-full server nopass
Copy
pki/ca.crt
,
pki/issued/server.crt
, and
pki/private/server.key
to your directory.
Fill in and add the following parameters into the API server start parameters:
--client-ca-file
=
/yourdirectory/ca.crt
--tls-cert-file
=
/yourdirectory/server.crt
--tls-private-key-file
=
/yourdirectory/server.key
openssl
openssl
Generate a ca.key with 2048bit:
openssl genrsa -out ca.key
2048
According to the ca.key generate a ca.crt (use
-days
to set the certificate effective time):
openssl req -x509 -new -nodes -key ca.key -subj
"/CN=
${
MASTER_IP
}
"
-days
10000
-out ca.crt
Generate a server.key with 2048bit:
openssl genrsa -out server.key
2048
Create a config file for generating a Certificate Signing Request (CSR).
Be sure to substitute the values marked with angle brackets (e.g.
&lt;MASTER_IP&gt;
)
with real values before saving this to a file (e.g.
csr.conf
).
Note that the value for
is the service cluster IP for the
API server as described in previous subsection.
[ req ]
default_bits
=
2048
prompt
=
no
default_md
=
sha256
req_extensions
=
req_ext
distinguished_name
=
dn
[ dn ]
C
=
&lt;country&gt;
ST
=
&lt;state&gt;
L
=
&lt;city&gt;
O
=
&lt;organization&gt;
OU
=
&lt;organization unit&gt;
CN
=
[ req_ext ]
subjectAltName
=
@alt_names
[ alt_names ]
DNS.1
=
DNS.2
=
kubernetes.default
DNS.3
=
kubernetes.default.svc
DNS.4
=
kubernetes.default.svc.cluster
DNS.5
=
kubernetes.default.svc.cluster.local
IP.1
=
IP.2
=
&lt;MASTER_CLUSTER_IP&gt;
[ v3_ext ]
authorityKeyIdentifier
=
keyid,issuer:always
basicConstraints
=
CA:FALSE
keyUsage
=
keyEncipherment,dataEncipherment
extendedKeyUsage
=
serverAuth,clientAuth
=
Generate the certificate signing request based on the config file:
openssl req -new -key server.key -out server.csr -config csr.conf
Generate the server certificate using the ca.key, ca.crt and server.csr:
openssl x509 -req -

in server.csr -CA ca.crt -CAkey ca.key
\
-CAcreateserial -out server.crt -days
10000
\
-extensions v3_ext -extfile csr.conf -sha256
View the certificate signing request:
openssl req  -noout -text -in ./server.csr
View the certificate:
openssl x509  -noout -text -in ./server.crt
Finally, add the same parameters into the API server start parameters.
cfssl
cfssl
is another tool for certificate generation.
Download, unpack and prepare the command line tools as shown below.
Note that you may need to adapt the sample commands based on the hardware
architecture and cfssl version you are using.
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl_1.5.0_linux_amd64 -o cfssl
chmod +x cfssl
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssljson_1.5.0_linux_amd64 -o cfssljson
chmod +x cfssljson
curl -L https://github.com/cloudflare/cfssl/releases/download/v1.5.0/cfssl-certinfo_1.5.0_linux_amd64 -o cfssl-certinfo
chmod +x cfssl-certinfo
Create a directory to hold the artifacts and initialize cfssl:
mkdir cert
cd
cert
../cfssl print-defaults config &gt; config.json
../cfssl print-defaults csr &gt; csr.json
Create a JSON config file for generating the CA file, for example,
ca-config.json
:
{
"signing"
: {
"default"
: {
"expiry"
:
"8760h"
},
"profiles"
: {
: {
"usages"
: [
"signing"
,
"key encipherment"
,
"server auth"
,
"client auth"
],
"expiry"
:
"8760h"
}
}
}
}
Create a JSON config file for CA certificate signing request (CSR), for example,
ca-csr.json
. Be sure to replace the values marked with angle brackets with
real values you want to use.
{
"CN"
:
,
"key"
: {
"algo"
:
"rsa"
,
"size"
:
2048
},
"names"
:[{
"C"
:
"&lt;country&gt;"
,
"ST"
:
"&lt;state&gt;"
,
"L"
:
"&lt;city&gt;"
,
"O"
:
"&lt;organization&gt;"
,
"OU"
:
"&lt;organization unit&gt;"
}]
}
Generate CA key (
ca-key.pem
) and certificate (
ca.pem
):
../cfssl gencert -initca ca-csr.json | ../cfssljson -bare ca
Create a JSON config file for generating keys and certificates for the API
server, for example,
server-csr.json
. Be sure to replace the values in angle brackets with
real values you want to use. The
is the service cluster
IP for the API server as described in previous subsection.
{
"CN"
:
,
"hosts"
: [
"127.0.0.1"
,
"&lt;MASTER_IP&gt;"
,
"&lt;MASTER_CLUSTER_IP&gt;"
,
,
"kubernetes.default"
,
"kubernetes.default.svc"
,
"kubernetes.default.svc.cluster"
,
"kubernetes.default.svc.cluster.local"
],
"key"
: {
"algo"
:
"rsa"
,
"size"
:
2048
},
"names"
: [{
"C"
:
,
"ST"
:
,
"L"
:
,
"O"
:
,
"OU"
:
}]
}
Generate the key and certificate for the API server, which are by default
saved into file
server-key.pem
and
server.pem
respectively:
../cfssl gencert -ca
=
ca.pem -ca-key
=
\
--config
=
ca-config.json -profile
=
\
server-csr.json | ../cfssljson -bare server
Distributing Self-Signed CA Certificate
A client node may refuse to recognize a self-signed CA certificate as valid.
For a non-production deployment, or for a deployment that runs behind a company
firewall, you can distribute a self-signed CA certificate to all clients and
refresh the local list for valid certificates.
On each client, perform the following operations:
sudo cp ca.crt /usr/local/share/ca-certificates/kubernetes.crt
sudo update-ca-certificates
Updating certificates in /etc/ssl/certs...
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....
done.
Certificates API
API to provision
x509 certificates to use for authentication as documented
in the
Managing TLS in a cluster
task page.
4 - Manage Memory, CPU, and API Resources
4.1 - Configure Default Memory Requests and Limits for a Namespace
Define a default memory resource limit f

or a namespace, so that every new Pod in that namespace has a memory resource limit configured.
This page shows how to configure default memory requests and limits for a
namespace
.
A Kubernetes cluster can be divided into namespaces. Once you have a namespace that
has a default memory
limit
,
and you then try to create a Pod with a container that does not specify its own memory
limit, then the
assigns the default
memory limit to that container.
Kubernetes assigns a default memory request under certain conditions that are explained later in this topic.
minikube
You must have access to create namespaces in your cluster.
Each node in your cluster must have at least 2 GiB of memory.
Create a namespace so that the resources you create in this exercise are
isolated from the rest of your cluster.
kubectl create namespace default-mem-example
Create a LimitRange and a Pod
Here's a manifest for an example
.
The manifest specifies a default memory
request and a default memory limit.
admin/resource/memory-defaults.yaml
:
v1
kind
:
metadata
:
name
:
mem-limit-range
spec
:
limits
:
-
default
:
memory
:
512Mi
defaultRequest
:
memory
:
256Mi
type
:
Container
Create the LimitRange in the default-mem-example namespace:
kubectl apply -f https://k8s.io/examples/admin/resource/memory-defaults.yaml --namespace
=
default-mem-example
Now if you create a Pod in the default-mem-example namespace, and any container
within that Pod does not specify its own values for memory request and memory limit,
then the
applies default values: a memory request of 256MiB and a memory limit of 512MiB.
Here's an example manifest for a Pod that has one container. The container
does not specify a memory request and limit.
admin/resource/memory-defaults-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
default-mem-demo
spec
:
:
-
name
:
default-mem-demo-ctr
image
:
nginx
Create the Pod.
kubectl apply -f https://k8s.io/examples/admin/resource/memory-defaults-pod.yaml --namespace
=
View detailed information about the Pod:
kubectl get pod default-mem-demo --output
=
yaml --namespace
=
The output shows that the Pod's container has a memory request of 256 MiB and
a memory limit of 512 MiB. These are the default values specified by the LimitRange.
imagePullPolicy: Always
name: default-mem-demo-ctr
resources:
limits:
memory: 512Mi
requests:
memory: 256Mi
Delete your Pod:
kubectl delete pod default-mem-demo --namespace
=
What if you specify a container's limit, but not its request?
Here's a manifest for a Pod that has one container. The container
specifies a memory limit, but not a request:
admin/resource/memory-defaults-pod-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
default-mem-demo-2
spec
:
:
-
name
:
default-mem-demo-2-ctr
image
:
nginx
resources
:
limits
:
memory
:
"1Gi"
kubectl apply -f https://k8s.io/examples/admin/resource/memory-defaults-pod-2.yaml --namespace
=
kubectl get pod default-mem-demo-2 --output
=
=
The output shows that the container's memory request is set to match its memory limit.
Notice that the container was not assigned the default memory request value of 256Mi.
  limits:
    memory: 1Gi
  requests:
What if you specify a container's request, but not its limit?
Here's a manifest for a Pod that has one container. T

he container
specifies a memory request, but not a limit:
admin/resource/memory-defaults-pod-3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
default-mem-demo-3
spec
:
:
-
name
:
default-mem-demo-3-ctr
image
:
nginx
resources
:
requests
:
memory
:
"128Mi"
kubectl apply -f https://k8s.io/examples/admin/resource/memory-defaults-pod-3.yaml --namespace
=
View the Pod's specification:
kubectl get pod default-mem-demo-3 --output
=
=
The output shows that the container's memory request is set to the value specified in the
container's manifest. The container is limited to use no more than 512MiB of
memory, which matches the default memory limit for the namespace.
  limits:
    memory: 512Mi
  requests:
    memory: 128Mi
Note:
A
does
not
check the consistency of the default values it applies. This means that a default value for the
limit
that is set by
may be less than the
request
value specified for the container in the spec that a client submits to the API server. If that happens, the final Pod will not be scheduleable.
See
Constraints on resource limits and requests
Motivation for default memory limits and requests
If your namespace has a memory
configured,
it is helpful to have a default value in place for memory limit.
Here are three of the restrictions that a resource quota imposes on a namespace:
For every Pod that runs in the namespace, the Pod and each of its containers must have a memory limit.
(If you specify a memory limit for every container in a Pod, Kubernetes can infer the Pod-level memory
limit by adding up the limits for its containers).
Memory limits apply a resource reservation on the node where the Pod in question is scheduled.
The total amount of memory reserved for all Pods in the namespace must not exceed a specified limit.
The total amount of memory actually used by all Pods in the namespace must also not exceed a specified limit.
When you add a LimitRange:
If any Pod in that namespace that includes a container does not specify its own memory limit,
the control plane applies the default memory limit to that container, and the Pod can be
allowed to run in a namespace that is restricted by a memory ResourceQuota.
Clean up
Delete your namespace:
kubectl delete namespace default-mem-example
For cluster administrators
For app developers
4.2 - Configure Default CPU Requests and Limits for a Namespace
Define a default CPU resource limits for a namespace, so that every new Pod in that namespace has a CPU resource limit configured.
This page shows how to configure default CPU requests and limits for a
namespace
.
A Kubernetes cluster can be divided into namespaces. If you create a Pod within a
namespace that has a default CPU
limit
, and any container in that Pod does not specify
its own CPU limit, then the
CPU limit to that container.
Kubernetes assigns a default CPU
request
,
but only under certain conditions that are explained later in this page.
minikube
You must have access to create namesp

aces in your cluster.
If you're not already familiar with what Kubernetes means by 1.0 CPU,
read
meaning of CPU
.
kubectl create namespace default-cpu-example
.
The manifest specifies a default CPU request and a default CPU limit.
admin/resource/cpu-defaults.yaml
:
v1
kind
:
metadata
:
name
:
cpu-limit-range
spec
:
limits
:
-
default
:
cpu
:
1
:
cpu
:
0.5
type
:
Container
Create the LimitRange in the default-cpu-example namespace:
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults.yaml --namespace
=
default-cpu-example
Now if you create a Pod in the default-cpu-example namespace, and any container
in that Pod does not specify its own values for CPU request and CPU limit,
then the control plane applies default values: a CPU request of 0.5 and a default
CPU limit of 1.
does not specify a CPU request and limit.
admin/resource/cpu-defaults-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
default-cpu-demo
spec
:
:
-
name
:
default-cpu-demo-ctr
image
:
nginx
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults-pod.yaml --namespace
=
kubectl get pod default-cpu-demo --output
=
=
The output shows that the Pod's only container has a CPU request of 500m
cpu
(which you can read as “500 millicpu”), and a CPU limit of 1
cpu
.
These are the default values specified by the LimitRange.
name: default-cpu-demo-ctr
limits:
cpu:
"1"
requests:
cpu: 500m
specifies a CPU limit, but not a request:
admin/resource/cpu-defaults-pod-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
default-cpu-demo-2
spec
:
:
-
name
:
default-cpu-demo-2-ctr
image
:
nginx
resources
:
limits
:
cpu
:
"1"
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults-pod-2.yaml --namespace
=
View the
specification
of the Pod that you created:
kubectl get pod default-cpu-demo-2 --output=yaml --namespace=default-cpu-example
The output shows that the container's CPU request is set to match its CPU limit.
Notice that the container was not assigned the default CPU request value of 0.5
cpu
:
  limits:
    cpu: "1"
  requests:
    cpu: "1"
specifies a CPU request, but not a limit:
admin/resource/cpu-defaults-pod-3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
default-cpu-demo-3
spec
:
:
-
name
:
default-cpu-demo-3-ctr
image
:
nginx
resources
:
requests
:
cpu
:
"0.75"
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-defaults-pod-3.yaml --namespace
=
View the
kubectl get pod default-cpu-demo-3 --output=yaml --namespace=default-cpu-example
The output shows that the container's CPU request is set to the value you specified at
the time you created the Pod (in other words: it matches the manifest).
However, the same container's CPU limit is set to 1
cpu
, which is the default CPU limit
for that namespace.
  limits:
    cpu: "1"
  requests:
    cpu: 750m
Motivation for default CPU limits and requests
If your namespace has a CPU
it is helpful to have a default value in place for CPU limit.
Here are two of the restrictions that a CPU resource quota imposes on a namespace:
For every Pod that runs in the namespace

, each of its containers must have a CPU limit.
CPU limits apply a resource reservation on the node where the Pod in question is scheduled.
The total amount of CPU that is reserved for use by all Pods in the namespace must not
exceed a specified limit.
If any Pod in that namespace that includes a container does not specify its own CPU limit,
the control plane applies the default CPU limit to that container, and the Pod can be
allowed to run in a namespace that is restricted by a CPU ResourceQuota.
Clean up
kubectl delete namespace default-cpu-example
4.3 - Configure Minimum and Maximum Memory Constraints for a Namespace
Define a range of valid memory resource limits for a namespace, so that every new Pod in that namespace falls within the range you configure.
This page shows how to set minimum and maximum values for memory used by containers
running in a
namespace
.
You specify minimum and maximum memory values in a
object. If a Pod does not meet the constraints imposed by the LimitRange,
it cannot be created in the namespace.
minikube
Each node in your cluster must have at least 1 GiB of memory available for Pods.
kubectl create namespace constraints-mem-example
Here's an example manifest for a LimitRange:
admin/resource/memory-constraints.yaml
:
v1
kind
:
metadata
:
name
:
mem-min-max-demo-lr
spec
:
limits
:
-
max
:
memory
:
1Gi
min
:
memory
:
500Mi
type
:
Container
Create the LimitRange:
kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints.yaml --namespace
=
constraints-mem-example
View detailed information about the LimitRange:
kubectl get limitrange mem-min-max-demo-lr --namespace
=
constraints-mem-example --output
=
yaml
The output shows the minimum and maximum memory constraints as expected. But
notice that even though you didn't specify default values in the configuration
file for the LimitRange, they were created automatically.
limits:
  - default:
      memory: 1Gi
    defaultRequest:
    max:
    min:
      memory: 500Mi
    type: Container
Now whenever you define a Pod within the constraints-mem-example namespace, Kubernetes
performs these steps:
If any container in that Pod does not specify its own memory request and limit,
the control plane assigns the default memory request and limit to that container.
Verify that every container in that Pod requests at least 500 MiB of memory.
Verify that every container in that Pod requests no more than 1024 MiB (1 GiB)
of memory.
Here's a manifest for a Pod that has one container. Within the Pod spec, the sole
container specifies a memory request of 600 MiB and a memory limit of 800 MiB. These satisfy the
minimum and maximum memory constraints imposed by the LimitRange.
admin/resource/memory-constraints-pod.yaml
:
v1
kin

d
:
Pod
metadata
:
name
:
constraints-mem-demo
spec
:
:
-
name
:
constraints-mem-demo-ctr
image
:
nginx
resources
:
limits
:
memory
:
"800Mi"
requests
:
memory
:
"600Mi"
kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod.yaml --namespace
=
Verify that the Pod is running and that its container is healthy:
kubectl get pod constraints-mem-demo --namespace
=
kubectl get pod constraints-mem-demo --output
=
=
The output shows that the container within that Pod has a memory request of 600 MiB and
a memory limit of 800 MiB. These satisfy the constraints imposed by the LimitRange for
this namespace:
resources
:
limits
:
memory
:
800Mi
requests
:
memory
:
600Mi
kubectl delete pod constraints-mem-demo --namespace
=
Attempt to create a Pod that exceeds the maximum memory constraint
Here's a manifest for a Pod that has one container. The container specifies a
memory request of 800 MiB and a memory limit of 1.5 GiB.
admin/resource/memory-constraints-pod-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
constraints-mem-demo-2
spec
:
:
-
name
:
constraints-mem-demo-2-ctr
image
:
nginx
resources
:
limits
:
memory
:
"1.5Gi"
requests
:
memory
:
"800Mi"
Attempt to create the Pod:
kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod-2.yaml --namespace
=
The output shows that the Pod does not get created, because it defines a container that
requests more memory than is allowed:
Error from server (Forbidden): error when creating "examples/admin/resource/memory-constraints-pod-2.yaml":
pods "constraints-mem-demo-2" is forbidden: maximum memory usage per Container is 1Gi, but limit is 1536Mi.
Attempt to create a Pod that does not meet the minimum memory request
Here's a manifest for a Pod that has one container. That container specifies a
memory request of 100 MiB and a memory limit of 800 MiB.
admin/resource/memory-constraints-pod-3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
constraints-mem-demo-3
spec
:
:
-
name
:
constraints-mem-demo-3-ctr
image
:
nginx
resources
:
limits
:
memory
:
"800Mi"
requests
:
memory
:
"100Mi"
kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod-3.yaml --namespace
=
The output shows that the Pod does not get created, because it defines a container
that requests less memory than the enforced minimum:
Error from server (Forbidden): error when creating "examples/admin/resource/memory-constraints-pod-3.yaml":
pods "constraints-mem-demo-3" is forbidden: minimum memory usage per Container is 500Mi, but request is 100Mi.
Create a Pod that does not specify any memory request or limit
Here's a manifest for a Pod that has one container. The container does not
specify a memory request, and it does not specify a memory limit.
admin/resource/memory-constraints-pod-4.yaml
:
v1
kind
:
Pod
metadata
:
name
:
constraints-mem-demo-4
spec
:
:
-
name
:
constraints-mem-demo-4-ctr
image
:
nginx
kubectl apply -f https://k8s.io/examples/admin/resource/memory-constraints-pod-4.yaml --namespace
=
kubectl get pod constraints-mem-demo-4 --namespace
=
=
yaml
The output shows that the Pod's only container has a memory request of 1 GiB and a memory limit of 1 GiB.
How did that container get those values?
  limits:
  requests:
Because your Pod did not define any memory request and limit for that container, the cluster
applied a
default memory request and limit
from the LimitRange.
This means that the definition of that Pod shows those values. You can check it using
kubectl describe
:
# Look for the "Requ

ests:" section of the output
kubectl describe pod constraints-mem-demo-4 --namespace
=
At this point, your Pod might be running or it might not be running. Recall that a prerequisite
for this task is that your Nodes have at least 1 GiB of memory. If each of your Nodes has only
1 GiB of memory, then there is not enough allocatable memory on any Node to accommodate a memory
request of 1 GiB. If you happen to be using Nodes with 2 GiB of memory, then you probably have
enough space to accommodate the 1 GiB request.
kubectl delete pod constraints-mem-demo-4 --namespace
=
Enforcement of minimum and maximum memory constraints
The maximum and minimum memory constraints imposed on a namespace by a LimitRange are enforced only
when a Pod is created or updated. If you change the LimitRange, it does not affect
Pods that were created previously.
Motivation for minimum and maximum memory constraints
As a cluster administrator, you might want to impose restrictions on the amount of memory that Pods can use.
Each Node in a cluster has 2 GiB of memory. You do not want to accept any Pod that requests
more than 2 GiB of memory, because no Node in the cluster can support the request.
A cluster is shared by your production and development departments.
You want to allow production workloads to consume up to 8 GiB of memory, but
you want development workloads to be limited to 512 MiB. You create separate namespaces
for production and development, and you apply memory constraints to each namespace.
Clean up
kubectl delete namespace constraints-mem-example
4.4 - Configure Minimum and Maximum CPU Constraints for a Namespace
Define a range of valid CPU resource limits for a namespace, so that every new Pod in that namespace falls within the range you configure.
This page shows how to set minimum and maximum values for the CPU resources used by containers
and Pods in a
namespace
. You specify minimum
and maximum CPU values in a
object. If a Pod does not meet the constraints imposed by the LimitRange, it cannot be created
in the namespace.
minikube
Each node in your cluster must have at least 1.0 CPU available for Pods.
See
to learn what Kubernetes means by “1 CPU”.
kubectl create namespace constraints-cpu-example
:
admin/resource/cpu-constraints.yaml
:
v1
kind
:
metadata
:
name
:
cpu-min-max-demo-lr
spec
:
limits
:
-
max
:
cpu
:
"800m"
min
:
cpu
:
"200m"
type
:
Container
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints.yaml --namespace
=
constraints-cpu-example
kubectl get limitrange cpu-min-max-demo-lr --output
=
=
The output shows 

the minimum and maximum CPU constraints as expected. But
limits
:
-
default
:
cpu
:
800m
:
cpu
:
800m
max
:
cpu
:
800m
min
:
cpu
:
200m
type
:
Container
Now whenever you create a Pod in the constraints-cpu-example namespace (or some other client
of the Kubernetes API creates an equivalent Pod), Kubernetes performs these steps:
If any container in that Pod does not specify its own CPU request and limit, the control plane
assigns the default CPU request and limit to that container.
Verify that every container in that Pod specifies a CPU request that is greater than or equal to 200 millicpu.
Verify that every container in that Pod specifies a CPU limit that is less than or equal to 800 millicpu.
Note:
When creating a
object, you can specify limits on huge-pages
or GPUs as well. However, when both
default
and
are specified
on these resources, the two values must be the same.
Here's a manifest for a Pod that has one container. The container manifest
specifies a CPU request of 500 millicpu and a CPU limit of 800 millicpu. These satisfy the
minimum and maximum CPU constraints imposed by the LimitRange for this namespace.
admin/resource/cpu-constraints-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
constraints-cpu-demo
spec
:
:
-
name
:
constraints-cpu-demo-ctr
image
:
nginx
resources
:
limits
:
cpu
:
"800m"
requests
:
cpu
:
"500m"
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod.yaml --namespace
=
kubectl get pod constraints-cpu-demo --namespace
=
kubectl get pod constraints-cpu-demo --output
=
=
The output shows that the Pod's only container has a CPU request of 500 millicpu and CPU limit
of 800 millicpu. These satisfy the constraints imposed by the LimitRange.
resources
:
limits
:
cpu
:
800m
requests
:
cpu
:
500m
Delete the Pod
kubectl delete pod constraints-cpu-demo --namespace
=
Attempt to create a Pod that exceeds the maximum CPU constraint
CPU request of 500 millicpu and a cpu limit of 1.5 cpu.
admin/resource/cpu-constraints-pod-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
constraints-cpu-demo-2
spec
:
:
-
name
:
constraints-cpu-demo-2-ctr
image
:
nginx
resources
:
limits
:
cpu
:
"1.5"
requests
:
cpu
:
"500m"
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod-2.yaml --namespace
=
The output shows that the Pod does not get created, because it defines an unacceptable container.
That container is not acceptable because it specifies a CPU limit that is too large:
Error from server (Forbidden): error when creating "examples/admin/resource/cpu-constraints-pod-2.yaml":
pods "constraints-cpu-demo-2" is forbidden: maximum cpu usage per Container is 800m, but limit is 1500m.
Attempt to create a Pod that does not meet the minimum CPU request
CPU request of 100 millicpu and a CPU limit of 800 millicpu.
admin/resource/cpu-constraints-pod-3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
constraints-cpu-demo-3
spec
:
:
-
name
:
constraints-cpu-demo-3-ctr
image
:
nginx
resources
:
limits
:
cpu
:
"800m"
requests
:
cpu
:
"100m"
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod-3.yaml --namespace
=
That container is not acceptable because it specifies a CPU request 

that is lower than the
enforced minimum:
Error from server (Forbidden): error when creating "examples/admin/resource/cpu-constraints-pod-3.yaml":
pods "constraints-cpu-demo-3" is forbidden: minimum cpu usage per Container is 200m, but request is 100m.
Create a Pod that does not specify any CPU request or limit
specify a CPU request, nor does it specify a CPU limit.
admin/resource/cpu-constraints-pod-4.yaml
:
v1
kind
:
Pod
metadata
:
name
:
constraints-cpu-demo-4
spec
:
:
-
name
:
constraints-cpu-demo-4-ctr
image
:
vish/stress
kubectl apply -f https://k8s.io/examples/admin/resource/cpu-constraints-pod-4.yaml --namespace
=
kubectl get pod constraints-cpu-demo-4 --namespace=constraints-cpu-example --output=yaml
The output shows that the Pod's single container has a CPU request of 800 millicpu and a
CPU limit of 800 millicpu.
resources
:
limits
:
cpu
:
800m
requests
:
cpu
:
800m
Because that container did not specify its own CPU request and limit, the control plane
applied the
default CPU request and limit
from the LimitRange for this namespace.
At this point, your Pod may or may not be running. Recall that a prerequisite for
this task is that your Nodes must have at least 1 CPU available for use. If each of your Nodes has only 1 CPU,
then there might not be enough allocatable CPU on any Node to accommodate a request of 800 millicpu.
If you happen to be using Nodes with 2 CPU, then you probably have enough CPU to accommodate the 800 millicpu request.
kubectl delete pod constraints-cpu-demo-4 --namespace=constraints-cpu-example
Enforcement of minimum and maximum CPU constraints
The maximum and minimum CPU constraints imposed on a namespace by a LimitRange are enforced only
Motivation for minimum and maximum CPU constraints
As a cluster administrator, you might want to impose restrictions on the CPU resources that Pods can use.
Each Node in a cluster has 2 CPU. You do not want to accept any Pod that requests
more than 2 CPU, because no Node in the cluster can support the request.
You want to allow production workloads to consume up to 3 CPU, but you want development workloads to be limited
to 1 CPU. You create separate namespaces for production and development, and you apply CPU constraints to
each namespace.
Clean up
kubectl delete namespace constraints-cpu-example
4.5 - Configure Memory and CPU Quotas for a Namespace
Define overall memory and CPU resource limits for a namespace.
This page shows how to set quotas for the total amount memory and CPU that
can be used by all Pods running in a
namespace
.
You specify quotas in a
object.
minikube
You must have access to create namespaces 

in your cluster.
Each node in your cluster must have at least 1 GiB of memory.
kubectl create namespace quota-mem-cpu-example
Create a ResourceQuota
Here is a manifest for an example ResourceQuota:
admin/resource/quota-mem-cpu.yaml
:
v1
kind
:
metadata
:
name
:
mem-cpu-demo
spec
:
hard
:
requests.cpu
:
"1"
requests.memory
:
1Gi
limits.cpu
:
"2"
limits.memory
:
2Gi
Create the ResourceQuota:
kubectl apply -f https://k8s.io/examples/admin/resource/quota-mem-cpu.yaml --namespace
=
quota-mem-cpu-example
View detailed information about the ResourceQuota:
kubectl get resourcequota mem-cpu-demo --namespace
=
quota-mem-cpu-example --output
=
yaml
The ResourceQuota places these requirements on the quota-mem-cpu-example namespace:
For every Pod in the namespace, each container must have a memory request, memory limit, cpu request, and cpu limit.
The memory request total for all Pods in that namespace must not exceed 1 GiB.
The memory limit total for all Pods in that namespace must not exceed 2 GiB.
The CPU request total for all Pods in that namespace must not exceed 1 cpu.
The CPU limit total for all Pods in that namespace must not exceed 2 cpu.
See
Create a Pod
Here is a manifest for an example Pod:
admin/resource/quota-mem-cpu-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
quota-mem-cpu-demo
spec
:
:
-
name
:
quota-mem-cpu-demo-ctr
image
:
nginx
resources
:
limits
:
memory
:
"800Mi"
cpu
:
"800m"
requests
:
memory
:
"600Mi"
cpu
:
"400m"
kubectl apply -f https://k8s.io/examples/admin/resource/quota-mem-cpu-pod.yaml --namespace
=
Verify that the Pod is running and that its (only) container is healthy:
kubectl get pod quota-mem-cpu-demo --namespace
=
Once again, view detailed information about the ResourceQuota:
=
=
yaml
The output shows the quota along with how much of the quota has been used.
You can see that the memory and CPU requests and limits for your Pod do not
exceed the quota.
status:
  hard:
    limits.cpu: "2"
    limits.memory: 2Gi
    requests.cpu: "1"
    requests.memory: 1Gi
  used:
    limits.cpu: 800m
    limits.memory: 800Mi
    requests.cpu: 400m
    requests.memory: 600Mi
jq
tool, you can also query (using
JSONPath
)
for just the
used
values,
and
pretty-print that that of the output. For example:
=
quota-mem-cpu-example -o
jsonpath
=
'{ .status.used }'
| jq .
Attempt to create a second Pod
Here is a manifest for a second Pod:
admin/resource/quota-mem-cpu-pod-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
quota-mem-cpu-demo-2
spec
:
:
-
name
:
quota-mem-cpu-demo-2-ctr
image
:
redis
resources
:
limits
:
memory
:
"1Gi"
cpu
:
"800m"
requests
:
memory
:
"700Mi"
cpu
:
"400m"
In the manifest, you can see that the Pod has a memory request of 700 MiB.
Notice that the sum of the used memory request and this new memory
request exceeds the memory request quota: 600 MiB + 700 MiB &gt; 1 GiB.
kubectl apply -f https://k8s.io/examples/admin/resource/quota-mem-cpu-pod-2.yaml --namespace
=
The second Pod does not get created. The output shows that creating the second Pod
would cause the memory request total to exceed the memory request quota.
Error from server (Forbidden): error when creating "examples/admin/resource/quota-mem-cpu-pod-2.yaml":
pods "quota-mem-cpu-demo-2" is forbidden: exceeded quota: mem-cpu-demo,
requested: requests.memory=700Mi,used: requests.memory=600Mi, limited: requests.memory=1Gi
Discussion
As you have seen in this exercise, you can use a ResourceQuota to restrict
the memory request total for all Pods running in a n

amespace.
You can also restrict the totals for memory limit, cpu request, and cpu limit.
Instead of managing total resource use within a namespace, you might want to restrict
individual Pods, or the containers in those Pods. To achieve that kind of limiting, use a
.
Clean up
kubectl delete namespace quota-mem-cpu-example
4.6 - Configure a Pod Quota for a Namespace
Restrict how many Pods you can create within a namespace.
This page shows how to set a quota for the total number of Pods that can run
in a
Namespace
. You specify quotas in a
object.
minikube
kubectl create namespace quota-pod-example
Here is an example manifest for a ResourceQuota:
admin/resource/quota-pod.yaml
:
v1
kind
:
metadata
:
name
:
pod-demo
spec
:
hard
:
pods
:
"2"
kubectl apply -f https://k8s.io/examples/admin/resource/quota-pod.yaml --namespace
=
quota-pod-example
kubectl get resourcequota pod-demo --namespace
=
quota-pod-example --output
=
yaml
The output shows that the namespace has a quota of two Pods, and that currently there are
no Pods; that is, none of the quota is used.
spec
:
hard
:
pods
:
"2"
status
:
hard
:
pods
:
"2"
used
:
pods
:
"0"
Here is an example manifest for a
:
admin/resource/quota-pod-deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
pod-quota-demo
spec
:
selector
:
:
purpose
:
quota-demo
replicas
:
3
template
:
metadata
:
labels
:
purpose
:
spec
:
:
-
name
:
image
:
nginx
In that manifest,
replicas: 3
tells Kubernetes to attempt to create three new Pods, all
running the same application.
Create the Deployment:
kubectl apply -f https://k8s.io/examples/admin/resource/quota-pod-deployment.yaml --namespace
=
View detailed information about the Deployment:
kubectl get deployment pod-quota-demo --namespace
=
=
yaml
The output shows that even though the Deployment specifies three replicas, only two
Pods were created because of the quota you defined earlier:
spec
:
...
replicas
:
3
...
status
:
:
2
...
lastUpdateTime
:
2021-04-02T20:57:05Z
message: 'unable to create pods
:
pods "pod-quota-demo-1650323038-" is forbidden:
exceeded quota: pod-demo, requested: pods=1, used: pods=2, limited
:
pods=2'
Choice of resource
In this task you have defined a ResourceQuota that limited the total number of Pods, but
you could also limit the total number of other kinds of object. For example, you
might decide to limit how many
CronJobs
that can live in a single namespace.
Clean up
kubectl delete namespace quota-pod-example
Configure De

fault CPU Requests and Limits for a Namespace
5 - Install a Network Policy Provider
5.1 - Use Antrea for NetworkPolicy
This page shows how to install and use Antrea CNI plugin on Kubernetes.
For background on Project Antrea, read the
Introduction to Antrea
.
You need to have a Kubernetes cluster. Follow the
kubeadm getting started guide
to bootstrap one.
Deploying Antrea with kubeadm
Follow
Getting Started
guide to deploy Antrea for kubeadm.
Once your cluster is running, you can follow the
to try out Kubernetes NetworkPolicy.
5.2 - Use Calico for NetworkPolicy
This page shows a couple of quick ways to create a Calico cluster on Kubernetes.
Decide whether you want to deploy a
cloud
or
local
cluster.
Creating a Calico cluster with Google Kubernetes Engine (GKE)
Prerequisite
:
gcloud
.
To launch a GKE cluster with Calico, include the
--enable-network-policy
flag.
Syntax
gcloud container clusters create
[
CLUSTER_NAME
]
Example
gcloud container clusters create my-calico-cluster --enable-network-policy
To verify the deployment, use the following command.
kubectl get pods --namespace
=
The Calico pods begin with
calico
. Check to make sure each one has a status of
Running
.
Creating a local Calico cluster with kubeadm
To get a local single-host Calico cluster in fifteen minutes using kubeadm, refer to the
Calico Quickstart
.
5.3 - Use Cilium for NetworkPolicy
This page shows how to use Cilium for NetworkPolicy.
For background on Cilium, read the
Introduction to Cilium
.
minikube
.
Deploying Cilium on Minikube for Basic Testing
To get familiar with Cilium easily you can follow the
Cilium Kubernetes Getting Started Guide
to perform a basic DaemonSet installation of Cilium in minikube.
To start minikube, minimal version required is &gt;= v1.5.2, run the with the
following arguments:
minikube version
minikube version: v1.5.2
minikube start --network-plugin
=
cni
For minikube you can install Cilium using its CLI tool. To do so, first download the latest
version of the CLI with the following command:
curl -LO https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
Then extract the downloaded file to your
/usr/local/bin
directory with the following command:
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz
After running the above commands, you can now install Cilium with the following command:
cilium install
Cilium will then automatically detect the cluster configuration and create and
install the appropriate components for a successful installation.
The components are:
Certificate Authority (CA) in Secret
cilium-ca
and certificates for Hubble (Cilium's observability layer).
Service accounts.
Cluster roles.
Agent DaemonSet and an Operator Deployment.
After the installation, you can view the overall status of the Cilium deployment with the
cilium status
command.
See the expected output of the
status
command
here
.
The remainder of the Getting St

arted Guide explains how to enforce both L3/L4
(i.e., IP address + port) security policies, as well as L7 (e.g., HTTP) security
policies using an example application.
Deploying Cilium for Production Use
For detailed instructions around deploying Cilium for production, see:
Cilium Kubernetes Installation Guide
This documentation includes detailed requirements, instructions and example
production DaemonSet files.
Understanding Cilium components
Deploying a cluster with Cilium adds Pods to the
namespace. To see
this list of Pods run:
=
kube-system -l k8s-app
=
cilium
You'll see a list of Pods similar to this:
NAME           READY   STATUS    RESTARTS   AGE
cilium-kkdhz   1/1     Running   0          3m23s
...
A
cilium
Pod runs on each node in your cluster and enforces network policy
on the traffic to/from Pods on that node using Linux BPF.
to try out Kubernetes NetworkPolicy with Cilium.
Have fun, and if you have questions, contact us using the
Cilium Slack Channel
.
5.4 - Use Kube-router for NetworkPolicy
Kube-router
for NetworkPolicy.
You need to have a Kubernetes cluster running. If you do not already have a cluster, you can create one by using any of the cluster installers like Kops, Bootkube, Kubeadm etc.
Installing Kube-router addon
The Kube-router Addon comes with a Network Policy Controller that watches Kubernetes API server for any NetworkPolicy and pods updated and configures iptables rules and ipsets to allow or block traffic as directed by the policies. Please follow the
trying Kube-router with cluster installers
guide to install Kube-router addon.
Once you have installed the Kube-router addon, you can follow the
5.5 - Romana for NetworkPolicy
This page shows how to use Romana for NetworkPolicy.
Complete steps 1, 2, and 3 of the
.
Installing Romana with kubeadm
Follow the
containerized installation guide
for kubeadm.
Applying network policies
To apply network policies use one of the following:
Romana network policies
.
Example of Romana network policy
.
The NetworkPolicy API.
Once you have installed Romana, you can follow the
5.6 - Weave Net for NetworkPolicy
This page shows how to use Weave Net for NetworkPolicy.
Install the Weave Net addon
Follow the
Integrating Kubernetes via the Addon
guide.
The Weave Net addon for Kubernetes comes with a
Network Policy Controller
that automatically monitors Kubernetes for any NetworkPolicy annotations on all
namespaces and configures
iptables
rules to allow or block traffic as directed by the policies.
Test the installation
Verify that the weave works.
Enter the following command:
kubectl get pods -n kube-system -o wide
NAME                                    READY     STATUS    RESTARTS   AGE       IP              NODE
weave-net-1t1qg                         2/2       Running   0          9d        192.168.2.10    worknode3
weave-net-231d7                         2/2       Running   1          7d        10.2.0.17       worknodegpu
weave-net-7nmwt                         2/2       Running   3          9d        192.168.2.131   masternode
weave-net-pmw8w                         2/2       Running   0          9d        192.168.2.216   worknode2
Each Node has a weave Pod, and all Pods are
Running
and
2/2 READY
. (
2/2
means that each Pod has
weave
and
weave-npc
.)
Once you have installed the Weave Net addon, you can follow the
to try out Kubernetes NetworkPolicy. If you have any question, contact us at
#weave-community on Slack or Weave User Group
.
6 

- Access Clusters Using the Kubernetes API
This page shows how to access clusters using the Kubernetes API.
minikube
.
Accessing the Kubernetes API
Accessing for the first time with kubectl
When accessing the Kubernetes API for the first time, use the
Kubernetes command-line tool,
kubectl
.
To access a cluster, you need to know the location of the cluster and have credentials
to access it. Typically, this is automatically set-up when you work through
a
Getting started guide
,
or someone else set up the cluster and provided you with credentials and a location.
Check the location and credentials that kubectl knows about with this command:
kubectl config view
Many of the
examples
provide an introduction to using
kubectl. Complete documentation is found in the
kubectl manual
.
Directly accessing the REST API
kubectl handles locating and authenticating to the API server. If you want to directly access the REST API with an http client like
curl
or
wget
, or a browser, there are multiple ways you can locate and authenticate against the API server:
Run kubectl in proxy mode (recommended). This method is recommended, since it uses the stored apiserver location and verifies the identity of the API server using a self-signed cert. No man-in-the-middle (MITM) attack is possible using this method.
Alternatively, you can provide the location and credentials directly to the http client. This works with client code that is confused by proxies. To protect against man in the middle attacks, you'll need to import a root cert into your browser.
Using the Go or Python client libraries provides accessing kubectl in proxy mode.
Using kubectl proxy
The following command runs kubectl in a mode where it acts as a reverse proxy. It handles
locating the API server and authenticating.
Run it like this:
kubectl proxy --port
=
8080
&amp;
See
Then you can explore the API with curl, wget, or a browser, like so:
curl http://localhost:8080/api/
{
"versions"
: [
"v1"
],
"serverAddressByClientCIDRs"
: [
{
"clientCIDR"
:
"0.0.0.0/0"
,
"serverAddress"
:
"10.0.1.149:443"
}
]
}
Without kubectl proxy
It is possible to avoid using kubectl proxy by passing an authentication token
directly to the API server, like this:
Using
grep/cut
approach:
# Check all possible clusters, as your .KUBECONFIG may have multiple contexts:
kubectl config view -o
jsonpath
=
'{"Cluster name\tServer\n"}{range .clusters[*]}{.name}{"\t"}{.cluster.server}{"\n"}{end}'
# Select name of cluster you want to interact with from above output:
export
=
"some_server_name"
# Point to the API server referring the cluster name
APISERVER
=
$(
jsonpath
=
"{.clusters[?(@.name==\"
$CLUSTER_NAME
\")].cluster.server}"
)
# Create a secret to hold a token for the default service account
kubectl apply -f -
&lt;&lt;EOF
kind: Secret
metadata:
name: default-token
kubernetes.io/service-account.name: default
type: kubernetes.io/service-account-token
EOF
# Wait for the token controller to populate the secret with a token:
while
! kubectl describe secret default-token | grep -E
'^token'
&gt;/dev/null;
do
echo
"waiting for token..."
&gt;&amp;
2
sleep
1
done
# Get the token value
TOKEN
=
$(
kubectl get secret default-token -o
jsonpath
=
'{.data.token}'
| base64 --decode
)
# Explore the API with TOKEN
curl -X GET
$APISERVER
/api --header
"Authorization: Bearer
$TOKEN
"
--insecure
{
"kind"
:
"APIVersions"
,
: [
"

v1"
],
: [
{
:
,
:
}
]
}
The above example uses the
flag. This leaves it subject to MITM
attacks. When kubectl accesses the cluster it uses a stored root certificate
and client certificates to access the server. (These are installed in the
~/.kube
directory). Since cluster certificates are typically self-signed, it
may take special configuration to get your http client to use root
certificate.
On some clusters, the API server does not require authentication; it may serve
on localhost, or be protected by a firewall. There is not a standard
for this.
describes how you can configure this as a cluster administrator.
Programmatic access to the API
Kubernetes officially supports client libraries for
Go
,
Python
,
Java
,
dotnet
,
JavaScript
, and
Haskell
. There are other client libraries that are provided and maintained by their authors, not the Kubernetes team. See
client libraries
for accessing the API from other languages and how they authenticate.
Go client
To get the library, run the following command:
go get k8s.io/client-go@kubernetes-&lt;kubernetes-version-number&gt;
See
https://github.com/kubernetes/client-go/releases
to see which versions are supported.
Write an application atop of the client-go clients.
Note:
client-go defines its own API objects, so if needed, import API definitions from client-go rather than from the main repository. For example,
import "k8s.io/client-go/kubernetes"
is correct.
The Go client can use the same
kubeconfig file
as the kubectl CLI does to locate and authenticate to the API server. See this
example
:
package
main
import
(
"context"
"fmt"
"k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
)
func
main
() {
// uses the current context in kubeconfig
// path-to-kubeconfig -- for example, /root/.kube/config
config, _
:=
clientcmd.
BuildConfigFromFlags
(
""
,
"&lt;path-to-kubeconfig&gt;"
)
// creates the clientset
clientset, _
:=
kubernetes.
NewForConfig
(config)
// access the API to list pods
pods, _
:=
clientset.
CoreV1
().
Pods
(
""
).
List
(context.
TODO
(), v1.ListOptions{})
fmt.
Printf
(
"There are %d pods in the cluster\n"
,
len
(pods.Items))
}
If the application is deployed as a Pod in the cluster, see
Accessing the API from within a Pod
.
Python client
To use
, run the following command:
pip install kubernetes
. See
Python Client Library page
for more installation options.
The Python client can use the same
example
:
from
import
client, config
config
.
load_kube_config()
v1
=
client
.
CoreV1Api()
print
(
"Listing pods with their IPs:"
)
ret
=
v1
.
list_pod_for_all_namespaces(watch
=
False
)
for
i
in
ret
.
items:
print
(
"
%s
\t
%s
\t
%s
"
%
(i
.
status
.
pod_ip, i
.
metadata
.
namespace, i
.
metadata
.
name))
Java client
To install the
Java Client
, run:
# Clone java library
git clone --recursive https://github.com/kubernetes-client/java
# Installing project artifacts, POM etc:
cd
java
mvn install
See
https://github.com/kubernetes-client/java/releases
The Java client can use the same
example
:
package
io.kubernetes.client.examples
;
import
io.kubernetes.client.ApiClient
;
import
io.kubernetes.client.ApiException
;
import
io.kubernetes.client.Configuration
;
import
io.kubernetes.client.apis.CoreV1Api
;
import
io.kubernetes.client.models.V1Pod
;
import
io.kubernetes.client.models.V1PodList
;
import
io.kubernetes.client.util.ClientBuilder
;
import
io.kubernetes.client.util.KubeConfig
;
import
java.io.FileReader
;
import
java.io.IOException
;
/**
* A simple example of how to use the Java API from an application outside a kubernetes cluster
*
* &lt;p&gt;Easies

t way to run this: mvn exec:java
* -Dexec.mainClass="io.kubernetes.client.examples.KubeConfigFileClientExample"
*
*/
public
class
KubeConfigFileClientExample
{
public
static
void
main
(
String
[]
args
)
throws
IOException
,
ApiException
{
// file path to your KubeConfig
String kubeConfigPath
=
"~/.kube/config"
;
// loading the out-of-cluster config, a kubeconfig from file-system
ApiClient client
=
ClientBuilder
.
(
KubeConfig
.
loadKubeConfig
(
new
FileReader
(
kubeConfigPath
))).
build
();
// set the global default api-client to the in-cluster one from above
.
setDefaultApiClient
(
client
);
// the CoreV1Api loads default api-client from global configuration.
CoreV1Api api
=
new
CoreV1Api
();
// invokes the CoreV1Api client
V1PodList list
=
api
.
listPodForAllNamespaces
(
null
,
null
,
null
,
null
,
null
,
null
,
null
,
null
,
null
);
System
.
out
.
println
(
"Listing all pods: "
);
for
(
V1Pod item
:
list
.
getItems
())
{
System
.
out
.
println
(
item
.
getMetadata
().
getName
());
}
}
}
dotnet client
To use
dotnet add package KubernetesClient --version 1.6.1
See
dotnet Client Library page
for more installation options. See
https://github.com/kubernetes-client/csharp/releases
The dotnet client can use the same
example
:
using
System
;
using
k8s
;
namespace
simple
{
internal
class
PodList
{
private
static
void
Main(
string
[] args)
{
var
config = KubernetesClientConfiguration.BuildDefaultConfig();
IKubernetes client =
new
Kubernetes(config);
Console.WriteLine(
"Starting Request!"
);
var
list = client.ListNamespacedPod(
"default"
);
foreach
(
var
item
in
list.Items)
{
Console.WriteLine(item.Metadata.Name);
}
if
(list.Items.Count ==
0
)
{
"Empty!"
);
}
}
}
}
JavaScript client
To install
npm install @kubernetes/client-node
. See
https://github.com/kubernetes-client/javascript/releases
The JavaScript client can use the same
example
:
const
k8s
=
require(
'@kubernetes/client-node'
);
const
kc
=
new
k8s.KubeConfig();
kc.loadFromDefault();
const
k8sApi
=
kc.makeApiClient(k8s.CoreV1Api);
k8sApi.listNamespacedPod(
'default'
).then((res) =&gt; {
console.log(res.body);
});
Haskell client
See
https://github.com/kubernetes-client/haskell/releases
The
can use the same
example
:
exampleWithKubeConfig
::
IO
()
=
do
oidcCache
&lt;-
atomically
$
newTVar
$
Map
.
fromList
[]
(mgr, kcfg)
&lt;-
mkKubeClientConfig oidcCache
$
KubeConfigFile
"/path/to/kubeconfig"
dispatchMime
mgr
kcfg
(
CoreV1
.
listPodForAllNamespaces (
Accept
MimeJSON
))
&gt;&gt;=
print
7 - Advertise Extended Resources for a Node
This page shows how to specify extended resources for a Node.
Extended resources allow cluster administrators to advertise node-level
resources that would otherwise be unknown to Kubernetes.
minikube
.
Get the names of your Nodes
Choose one of your Nodes to use for this exercise.
Advertise a new extended resource on one of your Nodes
To advertise a new extended resource on a Node, s

end an HTTP PATCH request to
the Kubernetes API server. For example, suppose one of your Nodes has four dongles
attached. Here's an example of a PATCH request that advertises four dongle resources
for your Node.
PATCH /api/v1/nodes/&lt;your-node-name&gt;/status HTTP/1.1
Accept: application/json
Content-Type: application/json-patch+json
Host: k8s-master:8080
[
{
"op"
:
"add"
,
"path"
:
"/status/capacity/example.com~1dongle"
,
"value"
:
"4"
}
]
Note that Kubernetes does not need to know what a dongle is or what a dongle is for.
The preceding PATCH request tells Kubernetes that your Node has four things that
you call dongles.
Start a proxy, so that you can easily send requests to the Kubernetes API server:
In another command window, send the HTTP PATCH request.
Replace
&lt;your-node-name&gt;
with the name of your Node:
curl --header
"Content-Type: application/json-patch+json"
\
--request PATCH
\
--data
'[{"op": "add", "path": "/status/capacity/example.com~1dongle", "value": "4"}]'
\
http://localhost:8001/api/v1/nodes/&lt;your-node-name&gt;/status
Note:
In the preceding request,
~1
is the encoding for the character / in
the patch path. The operation path value in JSON-Patch is interpreted as a
JSON-Pointer. For more details, see
IETF RFC 6901
, section 3.
The output shows that the Node has a capacity of 4 dongles:
"capacity": {
  "cpu": "2",
  "memory": "2049008Ki",
  "example.com/dongle": "4",
Describe your Node:
kubectl describe node &lt;your-node-name&gt;
Once again, the output shows the dongle resource:
Capacity
:
cpu
:
2
memory
:
2049008Ki
example.com/dongle
:
4
Now, application developers can create Pods that request a certain
number of dongles. See
.
Extended resources are similar to memory and CPU resources. For example,
just as a Node has a certain amount of memory and CPU to be shared by all components
running on the Node, it can have a certain number of dongles to be shared
by all components running on the Node. And just as application developers
can create Pods that request a certain amount of memory and CPU, they can
create Pods that request a certain number of dongles.
Extended resources are opaque to Kubernetes; Kubernetes does not
know anything about what they are. Kubernetes knows only that a Node
has a certain number of them. Extended resources must be advertised in integer
amounts. For example, a Node can advertise four dongles, but not 4.5 dongles.
Storage example
Suppose a Node has 800 GiB of a special kind of disk storage. You could
create a name for the special storage, say example.com/special-storage.
Then you could advertise it in chunks of a certain size, say 100 GiB. In that case,
your Node would advertise that it has eight resources of type
example.com/special-storage.
Capacity
:
...
example.com/special-storage
:
8
If you want to allow arbitrary requests for special storage, you
could advertise special storage in chunks of size 1 byte. In that case, you would advertise
800Gi resources of type example.com/special-storage.
Capacity
:
...
:
800Gi
Then a Container could request any number of bytes of special storage, up to 800Gi.
Clean up
Here is a PATCH request that removes the dongle advertisement from a Node.

[
  {
    "op": "remove",
    "path": "/status/capacity/example.com~1dongle",
  }
]
Replace
\
\
--data
'[{"op": "remove", "path": "/status/capacity/example.com~1dongle"}]'
\
Verify that the dongle advertisement has been remov

ed:
kubectl describe node &lt;your-node-name&gt; | grep dongle
(you should not see any output)
For application developers
8 - Autoscale the DNS Service in a Cluster
This page shows how to enable and configure autoscaling of the DNS service in
your Kubernetes cluster.
minikube
.
This guide assumes your nodes use the AMD64 or Intel 64 CPU architecture.
Make sure
Kubernetes DNS
is enabled.
Determine whether DNS horizontal autoscaling is already enabled
List the
in your cluster in the kube-system
namespace
:
kubectl get deployment --namespace
=
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
...
dns-autoscaler            1/1     1            1           ...
...
If you see "dns-autoscaler" in the output, DNS horizontal autoscaling is
already enabled, and you can skip to
Tuning autoscaling parameters
.
Get the name of your DNS Deployment
List the DNS deployments in your cluster in the kube-system namespace:
kubectl get deployment -l k8s-app
=
kube-dns --namespace
=
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
...
coredns   2/2     2            2           ...
...
If you don't see a Deployment for DNS services, you can also look for it by name:
=
and look for a deployment named
coredns
or
kube-dns
.
Your scale target is
Deployment/&lt;your-deployment-name&gt;
where
&lt;your-deployment-name&gt;
is the name of your DNS Deployment. For example, if
the name of your Deployment for DNS is coredns, your scale target is Deployment/coredns.
Note:
CoreDNS is the default DNS service for Kubernetes. CoreDNS sets the label
k8s-app=kube-dns
so that it can work in clusters that originally used
kube-dns.
Enable DNS horizontal autoscaling
In this section, you create a new Deployment. The Pods in the Deployment run a
container based on the
cluster-proportional-autoscaler-amd64
image.
Create a file named
dns-horizontal-autoscaler.yaml
with this content:
admin/dns/dns-horizontal-autoscaler.yaml
kind
:
:
v1
metadata
:
name
:
kube-dns-autoscaler
namespace
:
---
kind
:
:
metadata
:
name
:
system:kube-dns-autoscaler
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"nodes"
]
verbs
:
[
"list"
,
"watch"
]
-
apiGroups
:
[
""
]
resources
:
[
"replicationcontrollers/scale"
]
verbs
:
[
"get"
,
"update"
]
-
apiGroups
:
[
"apps"
]
resources
:
[
"deployments/scale"
,
"replicasets/scale"
]
verbs
:
[
"get"
,
"update"
]
# Remove the configmaps rule once below issue is fixed:
# kubernetes-incubator/cluster-proportional-autoscaler#16
-
apiGroups
:
[
""
]
resources
:
[
"configmaps"
]
verbs
:
[
"get"
,
"create"
]
---
kind
:
:
metadata
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
rbac.authorization.k8s.io
---
:
apps/v1
kind
:
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
kubernetes.io/cluster-service
:
"true"
spec
:
selector
:
:
k8s-app
:
template
:
metadata
:
labels
:

k8s-app
:
spec
:
priorityClassName
:
system-cluster-critical
:
:
type
:
supplementalGroups
:
[
65534
]
fsGroup
:
65534
nodeSelector
:
kubernetes.io/os
:
linux
:
-
name
:
autoscaler
image
:
registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.4
resources
:
requests
:
cpu
:
"20m"
memory
:
"10Mi"
command
:
- /cluster-proportional-autoscaler
- --namespace=kube-system
- --configmap=kube-dns-autoscaler
# Should keep target in sync with cluster/addons/dns/kube-dns.yaml.base
- --target=&lt;SCALE_TARGET&gt;
# When cluster is using large nodes(with more cores), "coresPerReplica" should dominate.
# If using small nodes, "nodesPerReplica" should dominate.
- --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true,"includeUnschedulableNodes":true}}
- --logtostderr=true
- --v=2
tolerations
:
-
key
:
"CriticalAddonsOnly"
operator
:
"Exists"
serviceAccountName
:
In the file, replace
&lt;SCALE_TARGET&gt;
with your scale target.
Go to the directory that contains your configuration file, and enter this
command to create the Deployment:
kubectl apply -f dns-horizontal-autoscaler.yaml
The output of a successful command is:
deployment.apps/dns-autoscaler created
DNS horizontal autoscaling is now enabled.
Tune DNS autoscaling parameters
Verify that the dns-autoscaler
ConfigMap
exists:
kubectl get configmap --namespace
=
NAME                  DATA      AGE
...
dns-autoscaler        1         ...
...
Modify the data in the ConfigMap:
kubectl edit configmap dns-autoscaler --namespace
=
Look for this line:
linear
:
'{"coresPerReplica":256,"min":1,"nodesPerReplica":16}'
Modify the fields according to your needs. The "min" field indicates the
minimal number of DNS backends. The actual number of backends is
calculated using this equation:
replicas = max( ceil( cores × 1/coresPerReplica ) , ceil( nodes × 1/nodesPerReplica ) )
Note that the values of both
coresPerReplica
and
nodesPerReplica
are
floats.
The idea is that when a cluster is using nodes that have many cores,
dominates. When a cluster is using nodes that have fewer
cores,
dominates.
There are other supported scaling patterns. For details, see
cluster-proportional-autoscaler
.
Disable DNS horizontal autoscaling
There are a few options for tuning DNS horizontal autoscaling. Which option to
use depends on different conditions.
Option 1: Scale down the dns-autoscaler deployment to 0 replicas
This option works for all situations. Enter this command:
kubectl scale deployment --replicas
=
0
dns-autoscaler --namespace
=
deployment.apps/dns-autoscaler scaled
Verify that the replica count is zero:
kubectl get rs --namespace
=
The output displays 0 in the DESIRED and CURRENT columns:
NAME                                 DESIRED   CURRENT   READY   AGE
...
dns-autoscaler-6b59789fc8            0         0         0       ...
...
Option 2: Delete the dns-autoscaler deployment
This option works if dns-autoscaler is under your own control, which means
no one will re-create it:
kubectl delete deployment dns-autoscaler --namespace
=
deployment.apps "dns-autoscaler" deleted
Option 3: Delete the dns-autoscaler manifest file from the master node
This option works if dns-autoscaler is under control of the (deprecated)
Addon Manager
,
and you have write access to the master node.
Sign in to the master node and delete the corresponding manifest file.
The common path for this dns-autoscaler is:
/etc/kubernetes/addons/dns-horizontal-autoscaler/dns-horizontal-autoscaler.yaml
After the manifest file is deleted, the Addon Manager will delete the
dns-autoscaler Deployment.
Understanding how DNS horizontal autoscaling works
The cluster-proportional-autoscaler application is deployed separately from
the DNS service.
An autoscaler Pod 

runs a client that polls the Kubernetes API server for the
number of nodes and cores in the cluster.
A desired replica count is calculated and applied to the DNS backends based on
the current schedulable nodes and cores and the given scaling parameters.
The scaling parameters and data points are provided via a ConfigMap to the
autoscaler, and it refreshes its parameters table every poll interval to be up
to date with the latest desired scaling parameters.
Changes to the scaling parameters are allowed without rebuilding or restarting
the autoscaler Pod.
The autoscaler provides a controller interface to support two control
patterns:
linear
and
ladder
.
Read about
.
Learn more about the
implementation of cluster-proportional-autoscaler
.
9 - Change the default StorageClass
This page shows how to change the default Storage Class that is used to
provision volumes for PersistentVolumeClaims that have no special requirements.
minikube
.
Why change the default storage class?
Depending on the installation method, your Kubernetes cluster may be deployed with
an existing StorageClass that is marked as default. This default StorageClass
is then used to dynamically provision storage for PersistentVolumeClaims
that do not require any specific storage class. See
PersistentVolumeClaim documentation
for details.
The pre-installed default StorageClass may not fit well with your expected workload;
for example, it might provision storage that is too expensive. If this is the case,
you can either change the default StorageClass or disable it completely to avoid
dynamic provisioning of storage.
Deleting the default StorageClass may not work, as it may be re-created
automatically by the addon manager running in your cluster. Please consult the docs for your installation
for details about addon manager and how to disable individual addons.
Changing the default StorageClass
List the StorageClasses in your cluster:
kubectl get storageclass
NAME                 PROVISIONER               AGE
standard
(
default
)
kubernetes.io/gce-pd      1d
gold                 kubernetes.io/gce-pd      1d
The default StorageClass is marked by
(default)
.
Mark the default StorageClass as non-default:
The default StorageClass has an annotation
storageclass.kubernetes.io/is-default-class
set to
true
. Any other value
or absence of the annotation is interpreted as
false
.
To mark a StorageClass as non-default, you need to change its value to
false
:
kubectl patch storageclass standard -p
'{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'
where
standard
is the name of your chosen StorageClass.
Mark a StorageClass as default:
Similar to the previous step, you need to add/set the annotation
storageclass.kubernetes.io/is-default-class=true
.
kubectl patch storageclass gold -p
'{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
Please note that at most one StorageClass can be marked as default. If two
or more of them are marked as default, a
without
storageClassName
explicitly specified cannot be created.
Verify that your chosen StorageClass is default:
NAME             PROVISIONER               AGE
standard         kubernetes.io/gce-pd      1d
gold
(
default
)
PersistentVolumes
.
10 - Change the Reclaim Policy of a PersistentVolume
This 

page shows how to change the reclaim policy of a Kubernetes
PersistentVolume.
minikube
.
Why change reclaim policy of a PersistentVolume
PersistentVolumes can have various reclaim policies, including "Retain",
"Recycle", and "Delete". For dynamically provisioned PersistentVolumes,
the default reclaim policy is "Delete". This means that a dynamically provisioned
volume is automatically deleted when a user deletes the corresponding
PersistentVolumeClaim. This automatic behavior might be inappropriate if the volume
contains precious data. In that case, it is more appropriate to use the "Retain"
policy. With the "Retain" policy, if a user deletes a PersistentVolumeClaim,
the corresponding PersistentVolume will not be deleted. Instead, it is moved to the
Released phase, where all of its data can be manually recovered.
Changing the reclaim policy of a PersistentVolume
List the PersistentVolumes in your cluster:
kubectl get pv
NAME                                       CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM             STORAGECLASS     REASON    AGE
pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     10s
pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     6s
pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim3    manual                     3s
This list also includes the name of the claims that are bound to each volume
for easier identification of dynamically provisioned volumes.
Choose one of your PersistentVolumes and change its reclaim policy:
kubectl patch pv &lt;your-pv-name&gt; -p
'{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'
where
&lt;your-pv-name&gt;
is the name of your chosen PersistentVolume.
Note:
On Windows, you must
double
quote any JSONPath template that contains spaces (not single
quote as shown above for bash). This in turn means that you must use a single quote or escaped
double quote around any literals in the template. For example:
"{\"
spec\
":{\"
persistentVolumeReclaimPolicy\
":\"
Retain\
"}}"
Verify that your chosen PersistentVolume has the right policy:
pvc-b6efd8da-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim1    manual                     40s
pvc-b95650f8-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Delete          Bound     default/claim2    manual                     36s
pvc-bb3ca71d-b7b5-11e6-9d58-0ed433a7dd94   4Gi        RWO           Retain          Bound     default/claim3    manual                     33s
In the preceding output, you can see that the volume bound to claim
default/claim3
has reclaim policy
Retain
. It will not be automatically
deleted when a user deletes claim
.
.
PersistentVolumeClaims
.
References
Pay attention to the
.spec.persistentVolumeReclaimPolicy
field
of PersistentVolume.
11 - Cloud Controller Manager Administration
Kubernetes v1.11 [beta]
Since cloud providers develop and release at a different pace compared to the Kubernetes proj

ect, abstracting the provider-specific code to the
cloud-controller-manager
binary allows cloud vendors to evolve independently from the core Kubernetes code.
The
can be linked to any cloud provider that satisfies
cloudprovider.Interface
. For backwards compatibility, the
provided in the core Kubernetes project uses the same cloud libraries as
. Cloud providers already supported in Kubernetes core are expected to use the in-tree cloud-controller-manager to transition out of Kubernetes core.
Requirements
Every cloud has their own set of requirements for running their own cloud provider integration, it should not be too different from the requirements when running
. As a general rule of thumb you'll need:
cloud authentication/authorization: your cloud may require a token or IAM rules to allow access to their APIs
kubernetes authentication/authorization: cloud-controller-manager may need RBAC rules set to speak to the kubernetes apiserver
high availability: like kube-controller-manager, you may want a high available setup for cloud controller manager using leader election (on by default).
Running cloud-controller-manager
Successfully running cloud-controller-manager requires some changes to your cluster configuration.
and
MUST NOT specify the
--cloud-provider
flag. This ensures that it does not run any cloud specific loops that would be run by cloud controller manager. In the future, this flag will be deprecated and removed.
kubelet
must run with
--cloud-provider=external
. This is to ensure that the kubelet is aware that it must be initialized by the cloud controller manager before it is scheduled any work.
Keep in mind that setting up your cluster to use cloud controller manager will change your cluster behaviour in a few ways:
kubelets specifying
will add a taint
node.cloudprovider.kubernetes.io/uninitialized
with an effect
NoSchedule
during initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work. Note that in the event that cloud controller manager is not available, new nodes in the cluster will be left unschedulable. The taint is important since the scheduler may require cloud specific information about nodes such as their region or type (high cpu, gpu, high memory, spot instance, etc).
cloud information about nodes in the cluster will no longer be retrieved using local metadata, but instead all API calls to retrieve node information will go through cloud controller manager. This may mean you can restrict access to your cloud API on the kubelets for better security. For larger clusters you may want to consider if cloud controller manager will hit rate limits since it is now responsible for almost all API calls to your cloud from within the cluster.
The cloud controller manager can implement:
Node controller - responsible for updating kubernetes nodes using cloud APIs and deleting kubernetes nodes that were deleted on your cloud.
Service controller - responsible for loadbalancers on your cloud against services of type LoadBalancer.
Route controller - responsible for setting up network routes on your cloud
any other features you would like to implement if you are running an out-of-tree provider.
Examples
If you are using a cloud that is currently supported in Kubernetes core and would like to adopt cloud controller manager, see the
cloud controller manager in kubernetes core
.
For cloud controller managers not in Kubernetes core, you can find the respective projects in repositories maintained by cloud vendors or by SIGs.
For providers already in Kubernetes core, you can run the in-tree cloud controller manager as a DaemonSet in your cluster, use the following as a guideline:
admin/cloud/ccm-example.yaml
# This is an example of how to set up cloud-controller-manager as a Daemonse

t in your cluster.
# It assumes that your masters can run pods and has the role node-role.kubernetes.io/master
# Note that this Daemonset will not work straight out of the box for your cloud, this is
# meant to be a guideline.
---
:
v1
kind
:
metadata
:
name
:
namespace
:
---
:
kind
:
metadata
:
name
:
system:cloud-controller-manager
roleRef
:
apiGroup
:
kind
:
name
:
cluster-admin
subjects
:
-
kind
:
name
:
namespace
:
---
:
apps/v1
kind
:
DaemonSet
metadata
:
labels
:
k8s-app
:
name
:
namespace
:
spec
:
selector
:
:
k8s-app
:
template
:
metadata
:
labels
:
k8s-app
:
spec
:
:
:
-
name
:
# for in-tree providers we use registry.k8s.io/cloud-controller-manager
# this can be replaced with any other image for out-of-tree providers
image
:
registry.k8s.io/cloud-controller-manager:v1.8.0
command
:
- /usr/local/bin/cloud-controller-manager
- --cloud-provider=[YOUR_CLOUD_PROVIDER]
# Add your own cloud provider here!
- --leader-elect=true
- --use-service-account-credentials
# these flags will vary for every cloud provider
- --allocate-node-cidrs=true
- --configure-cloud-routes=true
- --cluster-cidr=172.17.0.0/16
:
# this is required so CCM can bootstrap itself
-
key
:
value
:
"true"
effect
:
# these tolerations are to have the daemonset runnable on control plane nodes
# remove them if your control plane nodes should not run pods
-
key
:
node-role.kubernetes.io/control-plane
operator
:
Exists
effect
:
-
key
:
node-role.kubernetes.io/master
operator
:
Exists
effect
:
# this is to restrict CCM to only run on master nodes
# the node selector may vary depending on your cluster setup
:
:
""
Limitations
Running cloud controller manager comes with a few possible limitations. Although these limitations are being addressed in upcoming releases, it's important that you are aware of these limitations for production workloads.
Support for Volumes
Cloud controller manager does not implement any of the volume controllers found in
as the volume integrations also require coordination with kubelets. As we evolve CSI (container storage interface) and add stronger support for flex volume plugins, necessary support will be added to cloud controller manager so that clouds can fully integrate with volumes. Learn more about out-of-tree CSI volume plugins
here
.
Scalability
The cloud-controller-manager queries your cloud provider's APIs to retrieve information for all nodes. For very large clusters, consider possible bottlenecks such as resource requirements and API rate limiting.
Chicken and Egg
The goal of the cloud controller manager project is to decouple development of cloud features from the core Kubernetes project. Unfortunately, many aspects of the Kubernetes project has assumptions that cloud provider features are tightly integrated into the project. As a result, adopting this new architecture can create several situations where a request is being made for information from a cloud provider, but the cloud controller manager may not be able to return that information without the original request being complete.
A good example of this is the TLS bootstrapping feature in the Kubelet. TLS bootstrapping assumes that the Kubelet has the ability to ask the cloud provider (or a local metadata service) for all its address types (private, public, etc) but cloud controller manager cannot set a node's address types without being initialized in the first place which requires that the kubel

et has TLS certificates to communicate with the apiserver.
As this initiative evolves, changes will be made to address these issues in upcoming releases.
To build and develop your own cloud controller manager, read
.
12 - Configure a kubelet image credential provider
Configure the kubelet's image credential provider plugin
Kubernetes v1.26 [stable]
Starting from Kubernetes v1.20, the kubelet can dynamically retrieve credentials for a container image registry
using exec plugins. The kubelet and the exec plugin communicate through stdio (stdin, stdout, and stderr) using
Kubernetes versioned APIs. These plugins allow the kubelet to request credentials for a container registry dynamically
as opposed to storing static credentials on disk. For example, the plugin may talk to a local metadata server to retrieve
short-lived credentials for an image that is being pulled by the kubelet.
You may be interested in using this capability if any of the below are true:
API calls to a cloud provider service are required to retrieve authentication information for a registry.
Credentials have short expiration times and requesting new credentials frequently is required.
Storing registry credentials on disk or in imagePullSecrets is not acceptable.
This guide demonstrates how to configure the kubelet's image credential provider plugin mechanism.
You need a Kubernetes cluster with nodes that support kubelet credential
provider plugins. This support is available in Kubernetes 1.26;
Kubernetes v1.24 and v1.25 included this as a beta feature, enabled by default.
A working implementation of a credential provider exec plugin. You can build your own plugin or use one provided by cloud providers.
Your Kubernetes server must be version v1.26.
.
Installing Plugins on Nodes
A credential provider plugin is an executable binary that will be run by the kubelet. Ensure that the plugin binary exists on
every node in your cluster and stored in a known directory. The directory will be required later when configuring kubelet flags.
Configuring the Kubelet
In order to use this feature, the kubelet expects two flags to be set:
--image-credential-provider-config
- the path to the credential provider plugin config file.
--image-credential-provider-bin-dir
- the path to the directory where credential provider plugin binaries are located.
Configure a kubelet credential provider
The configuration file passed into
is read by the kubelet to determine which exec plugins
should be invoked for which container images. Here's an example configuration file you may end up using if you are using the
ECR
-based plugin:
:
kubelet.config.k8s.io/v1
kind
:
CredentialProviderConfig
# providers is a list of credential provider helper plugins that will be enabled by the kubelet.
# Multiple providers may match against a single image, in which case credentials
# from all providers will be returned to the kubelet. If multiple providers are called
# for a single image, the results are combined. If providers return overlapping
# auth keys, the value from the provider earlier in this list is used.
providers
:
# name is the required name of the credential provider. It must match the name of the
# provider executable as seen by the kubelet. The executable must be in the kubelet's
# bin directory (set by the --image-credential-provider-bin-dir flag).
-
name
:
ecr
# matchImages is a required list of strings used to match against images in order to
# determine if this provider should be invoked. If one of the strings matches the
# requested image from the kubelet, the plugin will be invoked and given a chance
# to provide credentials. Images are expected to contain the registry domain
# and URL path.
#
# Each entry in matchImages is a pattern which can optionally contain a port and a path.
# Globs can be used in the domain, but not in 

the port or the path. Globs are supported
# as subdomains like '*.k8s.io' or 'k8s.*.io', and top-level-domains such as 'k8s.*'.
# Matching partial subdomains like 'app*.k8s.io' is also supported. Each glob can only match
# a single subdomain segment, so `*.io` does **not** match `*.k8s.io`.
#
# A match exists between an image and a matchImage when all of the below are true:
# - Both contain the same number of domain parts and each part matches.
# - The URL path of an imageMatch must be a prefix of the target image URL path.
# - If the imageMatch contains a port, then the port must match in the image as well.
#
# Example values of matchImages:
# - 123456789.dkr.ecr.us-east-1.amazonaws.com
# - *.azurecr.io
# - gcr.io
# - *.*.registry.io
# - registry.io:8080/path
matchImages
:
-
"*.dkr.ecr.*.amazonaws.com"
-
"*.dkr.ecr.*.amazonaws.cn"
-
"*.dkr.ecr-fips.*.amazonaws.com"
-
"*.dkr.ecr.us-iso-east-1.c2s.ic.gov"
-
"*.dkr.ecr.us-isob-east-1.sc2s.sgov.gov"
# defaultCacheDuration is the default duration the plugin will cache credentials in-memory
# if a cache duration is not provided in the plugin response. This field is required.
defaultCacheDuration
:
"12h"
# Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse
# MUST use the same encoding version as the input. Current supported values are:
# - credentialprovider.kubelet.k8s.io/v1
:
credentialprovider.kubelet.k8s.io/v1
# Arguments to pass to the command when executing it.
# +optional
args
:
- get-credentials
# Env defines additional environment variables to expose to the process. These
# are unioned with the host's environment, as well as variables client-go uses
# to pass argument to the plugin.
env
:
-
name
:
AWS_PROFILE
value
:
example_profile
The
providers
field is a list of enabled plugins used by the kubelet. Each entry has a few required fields:
name
: the name of the plugin which MUST match the name of the executable binary that exists
in the directory passed into
.
: a list of strings used to match against images in order to determine
if this provider should be invoked. More on this below.
: the default duration the kubelet will cache credentials in-memory
if a cache duration was not specified by the plugin.
: the API version that the kubelet and the exec plugin will use when communicating.
Each credential provider can also be given optional args and environment variables as well.
Consult the plugin implementors to determine what set of arguments and environment variables are required for a given plugin.
Configure image matching
The
field for each credential provider is used by the kubelet to determine whether a plugin should be invoked
for a given image that a Pod is using. Each entry in
is an image pattern which can optionally contain a port and a path.
Globs can be used in the domain, but not in the port or the path. Globs are supported as subdomains like
*.k8s.io
or
k8s.*.io
,
and top-level domains such as
k8s.*
. Matching partial subdomains like
app*.k8s.io
is also supported. Each glob can only match
a single subdomain segment, so
*.io
does NOT match
*.k8s.io
.
A match exists between an image name and a
matchImage
entry when all of the below are true:
The URL path of match image must be a prefix of the target image URL path.
Some example values of
patterns are:
gcr.io
*.*.registry.io
foo.registry.io:8080/path
Read the details about
in the
kubelet configuration API (v1) reference
.
Read the
kubelet credential provider API reference (v1)
.
13 - Configure Quotas for API Objects
This page shows how to configure quotas for API objects, including
Persisten

tVolumeClaims and Services. A quota restricts the number of
objects, of a particular type, that can be created in a namespace.
object.
minikube
.
kubectl create namespace quota-object-example
Here is the configuration file for a ResourceQuota object:
admin/resource/quota-objects.yaml
:
v1
kind
:
metadata
:
name
:
object-quota-demo
spec
:
hard
:
:
"1"
services.loadbalancers
:
"2"
services.nodeports
:
"0"
kubectl apply -f https://k8s.io/examples/admin/resource/quota-objects.yaml --namespace
=
quota-object-example
kubectl get resourcequota object-quota-demo --namespace
=
quota-object-example --output
=
yaml
The output shows that in the quota-object-example namespace, there can be at most
one PersistentVolumeClaim, at most two Services of type LoadBalancer, and no Services
of type NodePort.
status
:
hard
:
:
"1"
:
"2"
:
"0"
used
:
:
"0"
:
"0"
:
"0"
Create a PersistentVolumeClaim
Here is the configuration file for a PersistentVolumeClaim object:
admin/resource/quota-objects-pvc.yaml
:
v1
kind
:
metadata
:
name
:
pvc-quota-demo
spec
:
:
manual
accessModes
:
- ReadWriteOnce
resources
:
requests
:
storage
:
3Gi
Create the PersistentVolumeClaim:
kubectl apply -f https://k8s.io/examples/admin/resource/quota-objects-pvc.yaml --namespace
=
Verify that the PersistentVolumeClaim was created:
kubectl get persistentvolumeclaims --namespace
=
The output shows that the PersistentVolumeClaim exists and has status Pending:
NAME             STATUS
pvc-quota-demo   Pending
Attempt to create a second PersistentVolumeClaim
Here is the configuration file for a second PersistentVolumeClaim:
admin/resource/quota-objects-pvc-2.yaml
:
v1
kind
:
metadata
:
name
:
pvc-quota-demo-2
spec
:
:
manual
:
resources
:
requests
:
storage
:
4Gi
Attempt to create the second PersistentVolumeClaim:
kubectl apply -f https://k8s.io/examples/admin/resource/quota-objects-pvc-2.yaml --namespace
=
The output shows that the second PersistentVolumeClaim was not created,
because it would have exceeded the quota for the namespace.
persistentvolumeclaims "pvc-quota-demo-2" is forbidden:
exceeded quota: object-quota-demo, requested: persistentvolumeclaims=1,
used: persistentvolumeclaims=1, limited: persistentvolumeclaims=1
Notes
These are the strings used to identify API resources that can be constrained
by quotas:
String
API Object
"pods"
Pod
"services"
Service
"replicationcontrollers"
"resourcequotas"
"secrets"
Secret
ConfigMap
"persistentvolumeclaims"
"services.nodeports"
Service of type NodePort
"services.loadbalancers"
Service of type LoadBalancer
Clean up
kubectl delete namespace quota-object-example
Configure Minimum and Maximum Memory Constraints for a

 Namespace
14 - Control CPU Management Policies on the Node
Kubernetes keeps many aspects of how pods execute on nodes abstracted
from the user. This is by design.  However, some workloads require
stronger guarantees in terms of latency and/or performance in order to operate
acceptably. The kubelet provides methods to enable more complex workload
placement policies while keeping the abstraction free from explicit placement
directives.
minikube
.
If you are running an older version of Kubernetes, please look at the documentation for the version you are actually running.
CPU Management Policies
By default, the kubelet uses
CFS quota
to enforce pod CPU limits.  When the node runs many CPU-bound pods,
the workload can move to different CPU cores depending on
whether the pod is throttled and which CPU cores are available at
scheduling time. Many workloads are not sensitive to this migration and thus
work fine without any intervention.
However, in workloads where CPU cache affinity and scheduling latency
significantly affect workload performance, the kubelet allows alternative CPU
management policies to determine some placement preferences on the node.
The CPU Manager policy is set with the
--cpu-manager-policy
kubelet
flag or the
cpuManagerPolicy
field in
.
There are two supported policies:
none
: the default policy.
static
: allows pods with certain resource characteristics to be
granted increased CPU affinity and exclusivity on the node.
The CPU manager periodically writes resource updates through the CRI in
order to reconcile in-memory CPU assignments with cgroupfs. The reconcile
frequency is set through a new Kubelet configuration value
--cpu-manager-reconcile-period
. If not specified, it defaults to the same
duration as
--node-status-update-frequency
.
The behavior of the static policy can be fine-tuned using the
--cpu-manager-policy-options
flag.
The flag takes a comma-separated list of
key=value
policy options.
If you disable the
CPUManagerPolicyOptions
then you cannot fine-tune CPU manager policies. In that case, the CPU manager
operates only using its default settings.
In addition to the top-level
feature gate, the policy options are split
into two groups: alpha quality (hidden by default) and beta quality (visible by default).
The groups are guarded respectively by the
CPUManagerPolicyAlphaOptions
and
CPUManagerPolicyBetaOptions
feature gates. Diverging from the Kubernetes standard, these
feature gates guard groups of options, because it would have been too cumbersome to add a feature
gate for each individual option.
Changing the CPU Manager Policy
Since the CPU manager policy can only be applied when kubelet spawns new pods, simply changing from
"none" to "static" won't apply to existing pods. So in order to properly change the CPU manager
policy on a node, perform the following steps:
Drain
the node.
Stop kubelet.
Remove the old CPU manager state file. The path to this file is
/var/lib/kubelet/cpu_manager_state
by default. This clears the state maintained by the
CPUManager so that the cpu-sets set up by the new policy won’t 

conflict with it.
Edit the kubelet configuration to change the CPU manager policy to the desired value.
Start kubelet.
Repeat this process for every node that needs its CPU manager policy changed. Skipping this
process will result in kubelet crashlooping with the following error:
could not restore state from checkpoint: configured policy "static" differs from state checkpoint policy "none", please drain this node and delete the CPU manager checkpoint file "/var/lib/kubelet/cpu_manager_state" before restarting Kubelet
None policy
The
none
policy explicitly enables the existing default CPU
affinity scheme, providing no affinity beyond what the OS scheduler does
automatically.  Limits on CPU usage for
Guaranteed pods
and
Burstable pods
are enforced using CFS quota.
Static policy
The
static
policy allows containers in
Guaranteed
pods with integer CPU
requests
access to exclusive CPUs on the node. This exclusivity is enforced
using the
cpuset cgroup controller
.
Note:
System services such as the container runtime and the kubelet itself can continue to run on these exclusive CPUs.  The exclusivity only extends to other pods.
Note:
CPU Manager doesn't support offlining and onlining of
CPUs at runtime. Also, if the set of online CPUs changes on the node,
the node must be drained and CPU manager manually reset by deleting the
state file
cpu_manager_state
in the kubelet root directory.
This policy manages a shared pool of CPUs that initially contains all CPUs in the
node. The amount of exclusively allocatable CPUs is equal to the total
number of CPUs in the node minus any CPU reservations by the kubelet
--kube-reserved
or
--system-reserved
options. From 1.17, the CPU reservation list can be specified
explicitly by kubelet
--reserved-cpus
option. The explicit CPU list specified by
takes precedence over the CPU reservation specified by
and
. CPUs reserved by these options are taken, in
integer quantity, from the initial shared pool in ascending order by physical
core ID.  This shared pool is the set of CPUs on which any containers in
BestEffort
and
Burstable
pods run. Containers in
pods with fractional
CPU
requests
also run on CPUs in the shared pool. Only containers that are
both part of a
pod and have integer CPU
requests
are assigned
exclusive CPUs.
Note:
The kubelet requires a CPU reservation greater than zero be made
using either
and/or
or
when
the static policy is enabled. This is because zero CPU reservation would allow the shared
pool to become empty.
As
pods whose containers fit the requirements for being statically
assigned are scheduled to the node, CPUs are removed from the shared pool and
placed in the cpuset for the container. CFS quota is not used to bound
the CPU usage of these containers as their usage is bound by the scheduling domain
itself. In others words, the number of CPUs in the container cpuset is equal to the integer
CPU
limit
specified in the pod spec. This static assignment increases CPU
affinity and decreases context switches due to throttling for the CPU-bound
workload.
Consider the containers in the following pod specs:
spec
:
:
-
name
:
nginx
image
:
nginx
This pod runs in the
QoS class because no resource
requests
or
limits
are specified. It runs in the shared pool.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
Burstable
QoS class because resource
requests
do not
equal
limits
and the
cpu
quantity is not specified. It runs in the shared
pool.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
requests
:
memory
:
"100Mi"
cpu
:
"1"
Burstable
requests
do not
equal
limits
. It runs in the shared pool.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:

"200Mi"
cpu
:
"2"
requests
:
memory
:
"200Mi"
cpu
:
"2"
QoS class because
requests
are equal to
limits
.
And the container's resource limit for the CPU resource is an integer greater than
or equal to one. The
nginx
container is granted 2 exclusive CPUs.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"1.5"
requests
:
memory
:
"200Mi"
cpu
:
"1.5"
requests
limits
.
But the container's resource limit for the CPU resource is a fraction. It runs in
the shared pool.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
QoS class because only
limits
and
requests
are set equal to
limits
when not explicitly specified. And the
container's resource limit for the CPU resource is an integer greater than or
equal to one. The
nginx
Static policy options
You can toggle groups of options on and off based upon their maturity level
using the following feature gates:
default enabled. Disable to hide beta-level options.
default disabled. Enable to show alpha-level options.
You will still have to enable each option using the
kubelet option.
The following policy options exist for the static
CPUManager
policy:
full-pcpus-only
(beta, visible by default) (1.22 or higher)
distribute-cpus-across-numa
(alpha, hidden by default) (1.23 or higher)
align-by-socket
(alpha, hidden by default) (1.25 or higher)
If the
policy option is specified, the static policy will always allocate full physical cores.
By default, without this option, the static policy allocates CPUs using a topology-aware best-fit allocation.
On SMT enabled systems, the policy can allocate individual virtual cores, which correspond to hardware threads.
This can lead to different containers sharing the same physical cores; this behaviour in turn contributes
to the
noisy neighbours problem
.
With the option enabled, the pod will be admitted by the kubelet only if the CPU request of all its containers
can be fulfilled by allocating full physical cores.
If the pod does not pass the admission, it will be put in Failed state with the message
SMTAlignmentError
.
If the
policy option is specified, the static
policy will evenly distribute CPUs across NUMA nodes in cases where more than
one NUMA node is required to satisfy the allocation.
By default, the
will pack CPUs onto one NUMA node until it is
filled, with any remaining CPUs simply spilling over to the next NUMA node.
This can cause undesired bottlenecks in parallel code relying on barriers (and
similar synchronization primitives), as this type of code tends to run only as
fast as its slowest worker (which is slowed down by the fact that fewer CPUs
are available on at least one NUMA node).
By distributing CPUs evenly across NUMA nodes, application developers can more
easily ensure that no single worker suffers from NUMA effects more than any
other, improving the overall performance of these types of applications.
If the
policy option is specified, CPUs will be considered
aligned at the socket boundary when deciding how to allocate CPUs to a
container. By default, the
aligns CPU allocations at the NUMA
boundary, which could result in performance degradation if CPUs need to be
pulled from more than one NUMA node to satisfy the allocation. Although it
tries to ensure that all CPUs are allocated from the
minimum
number of NUMA
nodes, there is no guarantee that those NUMA nodes will be on the same socket.
By directing the
to explicitly align CPUs at the socket boundary
rather than the NUMA boundary, we are able to avoid such issues. Note, this
policy option is not compatible with
TopologyManager
single-numa-node
polic

y and does not apply to hardware where the number of sockets is greater
than number of NUMA nodes.
The
option can be enabled by adding
full-pcpus-only=true
to
the CPUManager policy options.
Likewise, the
distribute-cpus-across-numa=true
to the CPUManager policy options.
When both are set, they are "additive" in the sense that CPUs will be
distributed across NUMA nodes in chunks of full-pcpus rather than individual
cores.
The
policy option can be enabled by adding
align-by-socket=true
to the
policy options. It is also additive to the
and
15 - Control Topology Management Policies on a node
An increasing number of systems leverage a combination of CPUs and hardware accelerators to
support latency-critical execution and high-throughput parallel computation. These include
workloads in fields such as telecommunications, scientific computing, machine learning, financial
services and data analytics. Such hybrid systems comprise a high performance environment.
In order to extract the best performance, optimizations related to CPU isolation, memory and
device locality are required. However, in Kubernetes, these optimizations are handled by a
disjoint set of components.
Topology Manager
is a Kubelet component that aims to coordinate the set of components that are
responsible for these optimizations.
minikube
Your Kubernetes server must be at or later than version v1.18.
.
How Topology Manager Works
Prior to the introduction of Topology Manager, the CPU and Device Manager in Kubernetes make
resource allocation decisions independently of each other. This can result in undesirable
allocations on multiple-socketed systems, performance/latency sensitive applications will suffer
due to these undesirable allocations. Undesirable in this case meaning for example, CPUs and
devices being allocated from different NUMA Nodes thus, incurring additional latency.
The Topology Manager is a Kubelet component, which acts as a source of truth so that other Kubelet
components can make topology aligned resource allocation choices.
The Topology Manager provides an interface for components, called
Hint Providers
, to send and
receive topology information. Topology Manager has a set of node level policies which are
explained below.
The Topology manager receives Topology information from the
as a bitmask denoting
NUMA Nodes available and a preferred allocation indication. The Topology Manager policies perform
a set of operations on the hints provided and converge on the hint determined by the policy to
give the optimal result, if an undesirable hint is stored the preferred field for the hint will be
set to false. In the current policies preferred is the narrowest preferred mask.
The selected hint is stored as part of the Topology Manager. Depending on the policy configured
the pod can be accepted or rejected from the node based on the selected hint.
The hint is then stored in the Topology Manager for use by the
when making the
resource allocation decisions.
Enable the Topology Manager feature
Support for the Topology Manager requires
to be enabled.
It is enabled by default starting with Kubernetes 1.18.
Topology Manager Scopes and Policies
The Topology Manager currently:
Aligns Pods of all QoS classes.
Aligns the requested resources that Hint Provider provid

es topology hints for.
If these conditions are met, the Topology Manager will align the requested resources.
In order to customise how this alignment is carried out, the Topology Manager provides two
distinct knobs:
scope
and
policy
.
The
scope
defines the granularity at which you would like resource alignment to be performed
(e.g. at the
pod
or
container
level). And the
policy
defines the actual strategy used to
carry out the alignment (e.g.
best-effort
,
,
, etc.).
Details on the various
scopes
and
policies
available today can be found below.
Note:
To align CPU resources with other requested resources in a Pod Spec, the CPU Manager should be
enabled and proper CPU Manager policy should be configured on a Node.
See
control CPU Management Policies
.
Note:
To align memory (and hugepages) resources with other requested resources in a Pod Spec, the Memory
Manager should be enabled and proper Memory Manager policy should be configured on a Node. Examine
Memory Manager
documentation.
Topology Manager Scopes
The Topology Manager can deal with the alignment of resources in a couple of distinct scopes:
container
(default)
pod
Either option can be selected at a time of the kubelet startup, with
--topology-manager-scope
flag.
container scope
The
container
scope is used by default.
Within this scope, the Topology Manager performs a number of sequential resource alignments, i.e.,
for each container (in a pod) a separate alignment is computed. In other words, there is no notion
of grouping the containers to a specific set of NUMA nodes, for this particular scope. In effect,
the Topology Manager performs an arbitrary alignment of individual containers to NUMA nodes.
The notion of grouping the containers was endorsed and implemented on purpose in the following
scope, for example the
pod
scope.
pod scope
To select the
pod
scope, start the kubelet with the command line option
--topology-manager-scope=pod
.
This scope allows for grouping all containers in a pod to a common set of NUMA nodes. That is, the
Topology Manager treats a pod as a whole and attempts to allocate the entire pod (all containers)
to either a single NUMA node or a common set of NUMA nodes. The following examples illustrate the
alignments produced by the Topology Manager on different occasions:
all containers can be and are allocated to a single NUMA node;
all containers can be and are allocated to a shared set of NUMA nodes.
The total amount of particular resource demanded for the entire pod is calculated according to
effective requests/limits
formula, and
thus, this total value is equal to the maximum of:
the sum of all app container requests,
the maximum of init container requests,
for a resource.
Using the
pod
scope in tandem with
Topology Manager policy is specifically
valuable for workloads that are latency sensitive or for high-throughput applications that perform
IPC. By combining both options, you are able to place all containers in a pod onto a single NUMA
node; hence, the inter-NUMA communication overhead can be eliminated for that pod.
In the case of
policy, a pod is accepted only if a suitable set of NUMA nodes
is present among possible allocations. Reconsider the example above:
a set containing only a single NUMA node - it leads to pod being admitted,
whereas a set containing more NUMA nodes - it results in pod rejection (because instead of one
NUMA node, two or more NUMA nodes are required to satisfy the allocation).
To recap, Topology Manager first computes a set of NUMA nodes and then tests it against Topology
Manager policy, which either leads to the rejection or admission of the pod.
Topology Manager Policies
Topology Manager supports four allocation policies. You can set a policy via a Kubelet flag,
--topology-manager-policy
. There are four supported policies:
none
(default)
Note:
If Topology Manager is configured with the
pod
scope, the container, which 

is considered by
the policy, is reflecting requirements of the entire pod, and thus each container from the pod
will result with
the same
topology alignment decision.
none policy
This is the default policy and does not perform any topology alignment.
best-effort policy
For each container in a Pod, the kubelet, with
topology management policy, calls
each Hint Provider to discover their resource availability. Using this information, the Topology
Manager stores the preferred NUMA Node affinity for that container. If the affinity is not
preferred, Topology Manager will store this and admit the pod to the node anyway.
The
can then use this information when making the
resource allocation decision.
restricted policy
topology management policy, calls each
Hint Provider to discover their resource availability. Using this information, the Topology
preferred, Topology Manager will reject this pod from the node. This will result in a pod in a
Terminated
state with a pod admission failure.
Once the pod is in a
state, the Kubernetes scheduler will
not
attempt to
reschedule the pod. It is recommended to use a ReplicaSet or Deployment to trigger a redeploy of
the pod. An external control loop could be also implemented to trigger a redeployment of pods that
have the
Topology Affinity
error.
If the pod is admitted, the
single-numa-node policy
topology management policy,
calls each Hint Provider to discover their resource availability. Using this information, the
Topology Manager determines if a single NUMA Node affinity is possible. If it is, Topology
Manager will store this and the
resource allocation decision. If, however, this is not possible then the Topology Manager will
reject the pod from the node. This will result in a pod in a
state with a pod
admission failure.
not
attempt to
reschedule the pod. It is recommended to use a Deployment with replicas to trigger a redeploy of
the Pod.An external control loop could be also implemented to trigger a redeployment of pods
that have the
error.
Topology manager policy options
Support for the Topology Manager policy options requires
TopologyManagerPolicyOptions
You can toggle groups of options on and off based upon their maturity level using the following feature gates:
TopologyManagerPolicyBetaOptions
default disabled. Enable to show beta-level options. Currently there are no beta-level options.
TopologyManagerPolicyAlphaOptions
default disabled. Enable to show alpha-level options. You will still have to enable each option using the
The following policy options exists:
prefer-closest-numa-nodes
(alpha, invisible by default,
and
feature gates have to be enabled)(1.26 or higher)
If the
policy option is specified, the
and
policies will favor sets of NUMA nodes with shorter distance between them when making admission decisions.
You can enable this option by adding
prefer-closest-numa-nodes=true
to the Topology Manager policy options.
By default, without this option, Topology Manager aligns resources on either a single NUMA node or
the minimum number of NUMA nodes (in cases where more than one NUMA node is required). However,
the
is not aware of NUMA distances and does not take them into account when making admission decisions.
This limitation surfaces in multi-socket, as well as single-socket multi NUMA systems,

and can cause significant performance degradation in latency-critical execution and high-throughput applications if the
Topology Manager decides to align resources on non-adjacent NUMA nodes.
Pod Interactions with Topology Manager Policies
spec
:
:
-
name
:
nginx
image
:
nginx
requests
or
limits
are specified.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
Burstable
QoS class because requests are less than limits.
If the selected policy is anything other than
none
, Topology Manager would consider these Pod
specifications. The Topology Manager would consult the Hint Providers to get topology hints.
In the case of the
static
, the CPU Manager policy would return default topology hint, because
these Pods do not have explicitly request CPU resources.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
example.com/device
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
This pod with integer CPU request runs in the
requests
are equal
to
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
This pod with sharing CPU request runs in the
requests
are equal
to
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
example.com/deviceA
:
"1"
example.com/deviceB
:
"1"
requests
:
:
"1"
:
"1"
QoS class because there are no CPU and memory requests.
The Topology Manager would consider the above pods. The Topology Manager would consult the Hint
Providers, which are CPU and Device Manager to get topology hints for the pods.
pod with integer CPU request, the
static
CPU Manager policy
would return topology hints relating to the exclusive CPU and the Device Manager would send back
hints for the requested device.
pod with sharing CPU request, the
static
would return default topology hint as there is no exclusive CPU request and the Device Manager
would send back hints for the requested device.
In the above two cases of the
pod, the
none
CPU Manager policy would return default
topology hint.
pod, the
static
CPU Manager policy would send back the default
topology hint as there is no CPU request and the Device Manager would send back the hints for each
of the requested devices.
Using this information the Topology Manager calculates the optimal hint for the pod and stores
this information, which will be used by the Hint Providers when they are making their resource
assignments.
Known Limitations
The maximum number of NUMA nodes that Topology Manager allows is 8. With more than 8 NUMA nodes
there will be a state explosion when trying to enumerate the possible NUMA affinities and
generating their hints.
The scheduler is not topology-aware, so it is possible to be scheduled on a node and then fail
on the node due to the Topology Manager.
16 - Customizing DNS Service
This page explains how to configure your DNS
Pod(s)
and customize the
DNS resolution process in your cluster.
minikube
Your cluster must be running the CoreDNS add-on.
Your Kubernetes server must be at

 or later than version v1.12.
.
Introduction
DNS is a built-in Kubernetes service launched automatically
using the
addon manager
cluster add-on
.
Note:
The CoreDNS Service is named
kube-dns
in the
metadata.name
field.
The intent is to ensure greater interoperability with workloads that relied on
the legacy
kube-dns
Service name to resolve addresses internal to the cluster.
Using a Service named
kube-dns
abstracts away the implementation detail of
which DNS provider is running behind that common name.
If you are running CoreDNS as a Deployment, it will typically be exposed as
a Kubernetes Service with a static IP address.
The kubelet passes DNS resolver information to each container with the
--cluster-dns=&lt;dns-service-ip&gt;
flag.
DNS names also need domains. You configure the local domain in the kubelet
with the flag
--cluster-domain=&lt;default-local-domain&gt;
.
The DNS server supports forward lookups (A and AAAA records), port lookups (SRV records),
reverse IP address lookups (PTR records), and more. For more information, see
.
If a Pod's
dnsPolicy
is set to
default
, it inherits the name resolution
configuration from the node that the Pod runs on. The Pod's DNS resolution
should behave the same as the node.
But see
Known issues
.
If you don't want this, or if you want a different DNS config for pods, you can
use the kubelet's
--resolv-conf
flag. Set this flag to "" to prevent Pods from
inheriting DNS. Set it to a valid file path to specify a file other than
/etc/resolv.conf
for DNS inheritance.
CoreDNS
CoreDNS is a general-purpose authoritative DNS server that can serve as cluster DNS,
complying with the
DNS specifications
.
CoreDNS ConfigMap options
CoreDNS is a DNS server that is modular and pluggable, with plugins adding new functionalities.
The CoreDNS server can be configured by maintaining a
Corefile
,
which is the CoreDNS configuration file. As a cluster administrator, you can modify the
ConfigMap
for the CoreDNS Corefile to
change how DNS service discovery behaves for that cluster.
In Kubernetes, CoreDNS is installed with the following default Corefile configuration:
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
The Corefile configuration includes the following
plugins
of CoreDNS:
errors
: Errors are logged to stdout.
health
: Health of CoreDNS is reported to
http://localhost:8080/health
. In this extended syntax
lameduck
will make the process
unhealthy then wait for 5 seconds before the process is shut down.
ready
: An HTTP endpoint on port 8181 will return 200 OK,
when all plugins that are able to signal readiness have done so.
: CoreDNS will reply to DNS queries
based on IP of the Services and Pods. You can find
more details
about this plugin on the CoreDNS website.
ttl
allows you to set a custom TTL for responses. The default is 5 seconds.
The minimum TTL allowed is 0 seconds, and the maximum is capped at 3600 seconds.
Setting TTL to 0 will prevent records from being cached.
The
option is provided for backward compatibility with
kube-dns
.
pods verified
option, which returns an A record only if there exists a pod
in the same namespace with a matching IP.
The
pods disabled
option can be used if you don't use pod records.
: Metrics of CoreDNS are available at
http://localhost:9153/metrics
in the
Prometheus
format
(also known as OpenMetrics).
forward
: Any queries that are not within the Kubernetes
cluster domain are forwarded to predefined resolvers (/etc/resolv.conf).
cache
: This enables a frontend cache.
loop
: Detects simple forwarding loops and
halts the CoreDNS process if a loop is found.
reload
: Allows 

automatic reload of a changed Corefile.
After you edit the ConfigMap configuration, allow two minutes for your changes to take effect.
: This is a round-robin DNS loadbalancer
that randomizes the order of A, AAAA, and MX records in the answer.
You can modify the default CoreDNS behavior by modifying the ConfigMap.
Configuration of Stub-domain and upstream nameserver using CoreDNS
CoreDNS has the ability to configure stub-domains and upstream nameservers
using the
forward plugin
.
Example
If a cluster operator has a
Consul
domain server located at "10.150.0.1",
and all Consul names have the suffix ".consul.local". To configure it in CoreDNS,
the cluster administrator creates the following stanza in the CoreDNS ConfigMap.
consul.local:53 {
    errors
    cache 30
    forward . 10.150.0.1
}
To explicitly force all non-cluster DNS lookups to go through a specific nameserver at 172.16.0.1,
point the
forward
to the nameserver instead of
forward .  172.16.0.1
The final ConfigMap along with the default
Corefile
configuration looks like:
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
errors
health
}
forward . 172.16.0.1
cache 30
loop
reload
}
errors
cache 30
forward . 10.150.0.1
}
Note:
CoreDNS does not support FQDNs for stub-domains and nameservers (eg: "ns.foo.com").
During translation, all FQDN nameservers will be omitted from the CoreDNS config.
Read
17 - Debugging DNS Resolution
This page provides hints on diagnosing DNS problems.
minikube
Your cluster must be configured to use the CoreDNS
addon
or its precursor,
kube-dns.
Your Kubernetes server must be at or later than version v1.6.
.
Create a simple Pod to use as a test environment
admin/dns/dnsutils.yaml
:
v1
kind
:
Pod
metadata
:
name
:
dnsutils
namespace
:
default
spec
:
:
-
name
:
dnsutils
image
:
registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3
command
:
- sleep
-
"infinity"
imagePullPolicy
:
IfNotPresent
:
Always
Note:
This example creates a pod in the
default
namespace. DNS name resolution for
services depends on the namespace of the pod. For more information, review
.
Use that manifest to create a Pod:
kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
pod/dnsutils created
…and verify its status:
kubectl get pods dnsutils
NAME      READY     STATUS    RESTARTS   AGE
dnsutils   1/1       Running   0          &lt;some-time&gt;
Once that Pod is running, you can exec
nslookup
in that environment.
If you see something like the following, DNS is working correctly.
kubectl
exec
-i -t dnsutils -- nslookup kubernetes.default
Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      kubernetes.default
Address 1: 10.0.0.1
If the
nslookup
command fails, check the following:
Check the local DNS configuration first
Take a look inside the resolv.conf file.
(See
and
below for more information)
kubectl
exec
-ti dnsutils -- cat /etc/resolv.conf
Verify that the search path and name server are set up like the following
(note that search path may vary for different cloud providers):
search default.svc.cluster.local svc.cluster.local cluster.local google.internal c.gce_project_id.internal
nameserver 10.0.0.10
options ndots:5
Errors such as the following indicate a pr

oblem with the CoreDNS (or kube-dns)
add-on or with associated Services:
kubectl
exec

nslookup: can't resolve 'kubernetes.default'
or
kubectl
exec
Address 1: 10.0.0.10 kube-dns.kube-system.svc.cluster.local

Check if the DNS pod is running
Use the
kubectl get pods
command to verify that the DNS pod is running.
=
=
kube-dns
NAME                       READY     STATUS    RESTARTS   AGE
...
coredns-7b96bf9f76-5hsxb   1/1       Running   0           1h
coredns-7b96bf9f76-mvmmt   1/1       Running   0           1h
...
Note:
The value for label
k8s-app
is
kube-dns
for both CoreDNS and kube-dns deployments.
If you see that no CoreDNS Pod is running or that the Pod has failed/completed,
the DNS add-on may not be deployed by default in your current environment and you
will have to deploy it manually.
Check for errors in the DNS pod
Use the
command to see logs for the DNS containers.
For CoreDNS:
kubectl logs --namespace
=
=
kube-dns
Here is an example of a healthy CoreDNS log:
.:53
2018/08/15 14:37:17 [INFO] CoreDNS-1.2.2
2018/08/15 14:37:17 [INFO] linux/amd64, go1.10.3, 2e322f6
CoreDNS-1.2.2
linux/amd64, go1.10.3, 2e322f6
2018/08/15 14:37:17 [INFO] plugin/reload: Running configuration MD5 = 24e6c59e83ce706f07bcc82c31b1ea1c
See if there are any suspicious or unexpected messages in the logs.
Is DNS service up?
Verify that the DNS service is up by using the
kubectl get service
command.
kubectl get svc --namespace
=
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
...
kube-dns     ClusterIP   10.0.0.10      &lt;none&gt;        53/UDP,53/TCP        1h
...
Note:
The service name is
kube-dns
If you have created the Service or in the case it should be created by default
but it does not appear, see
debugging Services
for
more information.
Are DNS endpoints exposed?
You can verify that DNS endpoints are exposed by using the
kubectl get endpoints
command.
kubectl get endpoints kube-dns --namespace
=
NAME       ENDPOINTS                       AGE
kube-dns   10.180.3.17:53,10.180.3.17:53    1h
If you do not see the endpoints, see the endpoints section in the
For additional Kubernetes DNS examples, see the
cluster-dns examples
in the Kubernetes GitHub repository.
Are DNS queries being received/processed?
You can verify if queries are being received by CoreDNS by adding the
log
plugin to the CoreDNS configuration (aka Corefile).
The CoreDNS Corefile is held in a
ConfigMap
named
coredns
. To edit it, use the command:
kubectl -n kube-system edit configmap coredns
Then add
log
in the Corefile section per the example below:
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
log
errors
health
upstream
}
cache 30
loop
reload
}
After saving the changes, it may take up to minute or two for Kubernetes to propagate these changes to the CoreDNS pods.
Next, make some queries and view the logs per the sections above in this document. If CoreDNS pods are receiving the queries, you should see them in the logs.
Here is an example of a query in the log:
.:53
2018/08/15 14:37:15 [INFO] CoreDNS-1.2.0
2018/08/15 14:37:15 [INFO] linux/amd64, go1.10.3, 2e322f6
CoreDNS-1.2.0
2018/09/07 15:29:04 [INFO] plugin/reload: Running configuration MD5 = 162475cdf272d8aa601e6fe67a6ad42f
2018/09/07 15:29:04 [INFO] Reloading complete
172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 

54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s
Does CoreDNS have sufficient permissions?
CoreDNS must be able to list
service
and
endpoint
related resources to properly resolve service names.
Sample error message:
2022-03-18T07:12:15.699431183Z [INFO] 10.96.144.227:52299 - 3686 "A IN serverproxy.contoso.net.cluster.local. udp 52 false 512" SERVFAIL qr,aa,rd 145 0.000091221s
First, get the current ClusterRole of
system:coredns
:
kubectl describe clusterrole system:coredns -n kube-system
Expected output:
PolicyRule:
  Resources                        Non-Resource URLs  Resource Names  Verbs
  ---------                        -----------------  --------------  -----
  nodes                            []                 []              [get]
  endpoints                        []                 []              [list watch]
  namespaces                       []                 []              [list watch]
  pods                             []                 []              [list watch]
  services                         []                 []              [list watch]
  endpointslices.discovery.k8s.io  []                 []              [list watch]
If any permissions are missing, edit the ClusterRole to add them:
kubectl edit clusterrole system:coredns -n kube-system
Example insertion of EndpointSlices permissions:
...
- apiGroups:
  - discovery.k8s.io
  resources:
  - endpointslices
  verbs:
  - list
  - watch
...
Are you in the right namespace for the service?
DNS queries that don't specify a namespace are limited to the pod's
If the namespace of the pod and service differ, the DNS query must include
the namespace of the service.
This query is limited to the pod's namespace:
kubectl
exec
-i -t dnsutils -- nslookup &lt;service-name&gt;
This query specifies the namespace:
kubectl
exec
-i -t dnsutils -- nslookup &lt;service-name&gt;.&lt;namespace&gt;
To learn more about name resolution, see
.
Some Linux distributions (e.g. Ubuntu) use a local DNS resolver by default (systemd-resolved).
Systemd-resolved moves and replaces
with a stub file that can cause a fatal forwarding
loop when resolving names in upstream servers. This can be fixed manually by using kubelet's
flag
to point to the correct
resolv.conf
(With
systemd-resolved
, this is
/run/systemd/resolve/resolv.conf
).
kubeadm automatically detects
, and adjusts the kubelet flags accordingly.
Kubernetes installs do not configure the nodes'
files to use the
cluster DNS by default, because that process is inherently distribution-specific.
This should probably be implemented eventually.
Linux's libc (a.k.a. glibc) has a limit for the DNS
nameserver
records to 3 by
default and Kubernetes needs to consume 1
record. This means that
if a local installation already uses 3
s, some of those entries will
be lost. To work around this limit, the node can run
dnsmasq
, which will
provide more
entries. You can also use kubelet's
flag.
If you are using Alpine version 3.3 or earlier as your base image, DNS may not
work properly due to a known issue with Alpine.
Kubernetes
issue 30215
details more information on this.
See
Autoscaling the DNS Service in a Cluster
.
Read
18 - Declare Network Policy
This document helps you get started using the Kubernetes
NetworkPolicy API
to declare network policies that govern how pods communicate with each other.
Note:
be configured to communicate with your cluster. It is recommended t

o run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a
minikube
Your Kubernetes server must be at or later than version v1.8.
.
Make sure you've configured a network provider with network policy support. There are a number of network providers that support NetworkPolicy, including:
Antrea
Calico
Cilium
Romana
Weave Net
Create an
nginx
deployment and expose it via a service
To see how Kubernetes network policy works, start off by creating an
nginx
Deployment.
kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
Expose the Deployment through a Service called
nginx
.
kubectl expose deployment nginx --port=80
service/nginx exposed
The above commands create a Deployment with an nginx Pod and expose the Deployment through a Service named
nginx
. The
nginx
Pod and Deployment are found in the
default
kubectl get svc,pod
NAME                        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
service/kubernetes          10.100.0.1    &lt;none&gt;        443/TCP    46m
service/nginx               10.100.0.16   &lt;none&gt;        80/TCP     33s

NAME                        READY         STATUS        RESTARTS   AGE
pod/nginx-701339712-e0qfq   1/1           Running       0          35s
Test the service by accessing it from another Pod
You should be able to access the new
nginx
service from other Pods. To access the
nginx
Service from another Pod in the
default
namespace, start a busybox container:
kubectl run busybox --rm -ti --image=busybox:1.28 -- /bin/sh
In your shell, run the following command:
wget --spider --timeout
=
1
nginx
Connecting to nginx (10.100.0.16:80)
remote file exists
Limit access to the
nginx
service
To limit the access to the
nginx
service so that only Pods with the label
access: true
can query it, create a NetworkPolicy object as follows:
service/networking/nginx-policy.yaml
:
networking.k8s.io/v1
kind
:
metadata
:
name
:
access-nginx
spec
:
podSelector
:
:
app
:
nginx
ingress
:
-
from
:
-
:
:
access
:
"true"
The name of a NetworkPolicy object must be a valid
DNS subdomain name
.
Note:
NetworkPolicy includes a
which selects the grouping of Pods to which the policy applies. You can see this policy selects Pods with the label
app=nginx
. The label was automatically added to the Pod in the
nginx
Deployment. An empty
selects all pods in the namespace.
Assign the policy to the service
Use kubectl to create a NetworkPolicy from the above
nginx-policy.yaml
file:
kubectl apply -f https://k8s.io/examples/service/networking/nginx-policy.yaml
networkpolicy.networking.k8s.io/access-nginx created
Test access to the service when access label is not defined
When you attempt to access the
nginx
Service from a Pod without the correct labels, the request times out:
In your shell, run the command:
=
1
nginx
wget: download timed out
Define access label and test again
You can create a Pod with the correct labels to see that the request is allowed:
kubectl run busybox --rm -ti --labels="access=true" --image=busybox:1.28 -- /bin/sh
=
1
nginx
19 - Developing Cloud Controller Manager
The cloud-controller-manager is a Kubernetes
component
that embeds cloud-specific control logic. The cloud controller manager lets you link your
cluster into your cloud provider's API, and separates out the components that interact
with that cloud platform from components that only interact with your cluster.
By decou

pling the interoperability logic between Kubernetes and the underlying cloud
infrastructure, the cloud-controller-manager component enables cloud providers to release
features at a different pace compared to the main Kubernetes project.
Background
Since cloud providers develop and release at a different pace compared to the Kubernetes project, abstracting the provider-specific code to the
The Kubernetes project provides skeleton cloud-controller-manager code with Go interfaces to allow you (or your cloud provider) to plug in your own implementations. This means that a cloud provider can implement a cloud-controller-manager by importing packages from Kubernetes core; each cloudprovider will register their own code by calling
cloudprovider.RegisterCloudProvider
to update a global variable of available cloud providers.
Developing
Out of tree
To build an out-of-tree cloud-controller-manager for your cloud:
Create a go package with an implementation that satisfies
.
Use
main.go
in cloud-controller-manager
from Kubernetes core as a template for your
main.go
. As mentioned above, the only difference should be the cloud package that will be imported.
Import your cloud package in
main.go
, ensure your package has an
init
block to run
.
Many cloud providers publish their controller manager code as open source. If you are creating
a new cloud-controller-manager from scratch, you could take an existing out-of-tree cloud
controller manager as your starting point.
In tree
For in-tree cloud providers, you can run the in-tree cloud controller manager as a
DaemonSet
in your cluster. See
20 - Enable Or Disable A Kubernetes API
This page shows how to enable or disable an API version from your cluster's
.
Specific API versions can be turned on or off by passing
--runtime-config=api/&lt;version&gt;
as a
command line argument to the API server. The values for this argument are a comma-separated
list of API versions. Later values override earlier values.
The
runtime-config
command line argument also supports 2 special keys:
api/all
, representing all known APIs
api/legacy
, representing only legacy APIs. Legacy APIs are any APIs that have been
explicitly
deprecated
.
For example, to turn off all API versions except v1, pass
--runtime-config=api/all=false,api/v1=true
to the
.
Read the
full documentation
for the
component.
21 - Encrypting Secret Data at Rest
This page shows how to enable and configure encryption of secret data at rest.
minikube
Your Kubernetes server must be at or later than version 1.13.
.
etcd v3.0 or later is required
To encrypt a custom resource, your cluster must be running Kubernetes v1.26 or newer.
Configuration and determining whether encryption at rest is already enabled
The
process accepts an argument
--encryption-provider-config
that controls how API data is encrypted in etcd.
The configuration is provided as an API named
.
--encryption-provider-config-automatic-reload
boolean argument determines if the file set by
should be automatically reloaded if the disk contents change. This enables key rotation without API server restarts. An example configuration is provided below.
Caution:
IMPORTANT:
For high-availability configurations (wi

th two or more control plane nodes), the
encryption configuration file must be the same! Otherwise, the
component cannot
decrypt data stored in the etcd.
Understanding the encryption at rest configuration
:
kind
:
resources
:
-
resources
:
- secrets
- configmaps
- pandas.awesome.bears.example
providers
:
-
identity
:
{}
-
aesgcm
:
keys
:
-
name
:
key1
secret
:
c2VjcmV0IGlzIHNlY3VyZQ==
-
name
:
key2
secret
:
dGhpcyBpcyBwYXNzd29yZA==
-
aescbc
:
keys
:
-
name
:
key1
secret
:
-
name
:
key2
secret
:
-
secretbox
:
keys
:
-
name
:
key1
secret
:
YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
Each
resources
array item is a separate config and contains a complete configuration. The
resources.resources
field is an array of Kubernetes resource names (
resource
or
resource.group
that should be encrypted like Secrets, ConfigMaps, or other resources.
If custom resources are added to
and the cluster version is 1.26 or newer,
any newly created custom resources mentioned in the
will be encrypted.
Any custom resources that existed in etcd prior to that version and configuration will be unencrypted
until they are next written to storage. This is the same behavior as built-in resources.
See the
Ensure all secrets are encrypted
section.
The
providers
array is an ordered list of the possible encryption providers to use for the APIs that you listed.
Only one provider type may be specified per entry (
identity
or
aescbc
may be provided,
but not both in the same item).
The first provider in the list is used to encrypt resources written into the storage. When reading
resources from storage, each provider that matches the stored data attempts in order to decrypt the
data. If no provider can read the stored data due to a mismatch in format or secret key, an error
is returned which prevents clients from accessing that resource.
For more detailed information about the
struct, please refer to the
encryption configuration API
.
Caution:
If any resource is not readable via the encryption config (because keys were changed),
the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to
read that resource will fail until it is deleted or a valid decryption key is provided.
Providers
Providers for Kubernetes encryption at rest
Name
Encryption
Strength
Speed
Key Length
Other Considerations
identity
None
N/A
N/A
N/A
Resources written as-is without encryption. When set as the first provider, the resource will be decrypted as new values are written.
secretbox
XSalsa20 and Poly1305
Strong
Faster
32-byte
A newer standard and may not be considered acceptable in environments that require high levels of review.
aesgcm
AES-GCM with random nonce
Must be rotated every 200k writes
Fastest
16, 24, or 32-byte
Is not recommended for use except when an automated key rotation scheme is implemented.
aescbc
AES-CBC with
PKCS#7
padding
Weak
Fast
32-byte
Not recommended due to CBC's vulnerability to padding oracle attacks.
kms
Uses envelope encryption scheme: Data is encrypted by data encryption keys (DEKs) using AES-CBC with
PKCS#7
padding (prior to v1.25), using AES-GCM starting from v1.25, DEKs are encrypted by key encryption keys (KEKs) according to configuration in Key Management Service (KMS)
Strongest
Fast
32-bytes
The recommended choice for using a third party tool for key management. Simplifies key rotation, with a new DEK generated for each encryption, and KEK rotation controlled by the user.
Configure the KMS provider
.
Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
is the first provider, the first key is used for encryption.
Caution:
Storing the raw encryption key in the EncryptionConfig only moderately improves your security
posture, compared to no encryption. Please use
kms
provider for ad

ditional security.
identity
provider is used to protect secret data in etcd, which provides no
encryption.
was introduced to encrypt secret data locally, with a locally
managed key.
Encrypting secret data with a locally managed key protects against an etcd compromise, but it fails to
protect against a host compromise. Since the encryption keys are stored on the host in the
EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption
keys.
Envelope encryption creates dependence on a separate key, not stored in Kubernetes. In this case,
an attacker would need to compromise etcd, the
kubeapi-server
, and the third-party KMS provider to
retrieve the plaintext values, providing a higher level of security than locally stored encryption keys.
Encrypting your data
Create a new encryption config file:
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
aescbc
:
keys
:
-
name
:
key1
secret
:
&lt;BASE 64 ENCODED SECRET&gt;
-
identity
:
{}
To create a new Secret, perform the following steps:
Generate a 32-byte random key and base64 encode it. If you're on Linux or macOS, run the following command:
head -c
32
/dev/urandom | base64
Place that value in the
secret
field of the
struct.
Set the
flag on the
to point to
the location of the config file.
You will need to mount the new encryption config file to the
static pod. Here is an example on how to do that:
Save the new encryption config file to
/etc/kubernetes/enc/enc.yaml
on the control-plane node.
Edit the manifest for the
static pod:
/etc/kubernetes/manifests/kube-apiserver.yaml
similarly to this:
:
v1
kind
:
Pod
metadata
:
:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint
:
10.10.30.4
:
6443
creationTimestamp
:
null
labels
:
component
:
tier
:
name
:
namespace
:
spec
:
:
-
command
:
- kube-apiserver
...
- --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
# &lt;-- add this line
:
...
-
name
:
enc
mountPath
:
/etc/kubernetes/enc
readonly
:
true
...
volumes
:
...
-
name
:
enc
hostPath
:
path
:
type
:
DirectoryOrCreate
...
Restart your API server.
Caution:
Your config file contains keys that can decrypt the contents in etcd, so you must properly restrict
permissions on your control-plane nodes so only the user who runs the
can read it.
Verifying that data is encrypted
Data is encrypted when written to etcd. After restarting your
, any newly created or
updated Secret or other resource types configured in
should be encrypted
when stored. To check this, you can use the
etcdctl
command line
program to retrieve the contents of your secret data.
Create a new Secret called
secret1
in the
default
kubectl create secret generic secret1 -n default --from-literal
=
mykey
=
mydata
Using the
etcdctl
command line, read that Secret out of etcd:
ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C
where
[...]
must be the additional arguments for connecting to the etcd server.
ETCDCTL_API
=
3
etcdctl
\
--cacert
=
/etc/kubernetes/pki/etcd/ca.crt
\
--cert
=
/etc/kubernetes/pki/etcd/server.crt
\
--key
=
/etc/kubernetes/pki/etcd/server.key
\
get /registry/secrets/default/secret1 | hexdump -C
The output is similar to this (abbreviated):
00000000
2f
72
65
67
69
73
74
72
79
2f
73
65
63
72
65
74
|
/registry/secret
|
00000010
73
2f
64
65
66
61
75
6c
74
2f
73
65
63
72
65
74
|
s/default/secret
|
00000020
31
0a
6b
38
73
3a
65
6e
6

3
3a
61
65
73
63
62
63
|
1.k8s:enc:aescbc
|
00000030
3a
76
31
3a
6b
65
79
31
3a
c7
6c
e7
d3
09
bc
06
|
:v1:key1:.l.....
|
00000040
25
51
91
e4
e0
6c
e5
b1
4d
7a
8b
3d
b9
c2
7c
6e
|
%Q...l..Mz.=..|n
|
00000050
b4
79
df
05
28
ae
0d
8e
5f
35
13
2c
c0
18
99
3e
|
.y..(..._5.,...&gt;
|
[...]
00000110
23
3a
0d
fc
28
ca
48
2d
6b
2d
46
cc
72
0b
70
4c
|
#:..(.H-k-F.r.pL
|
00000120
a5
fc
35
43
12
4e
60
ef
bf
6f
fe
cf
df
0b
ad
1f
|
..5C.N`..o......
|
00000130
82
c4
88
53
02
da
3e
66
ff
0a
|
...S..&gt;f..
|
0000013a
Verify the stored Secret is prefixed with
k8s:enc:aescbc:v1:
which indicates
the
aescbc
provider has encrypted the resulting data. Confirm that the key name shown in
etcd
matches the key name specified in the
mentioned above. In this example,
you can see that the encryption key named
key1
is used in
etcd
and in
.
Verify the Secret is correctly decrypted when retrieved via the API:
kubectl get secret secret1 -n default -o yaml
The output should contain
mykey: bXlkYXRh
, with contents of
mydata
encoded, check
decoding a Secret
to completely decode the Secret.
Ensure all Secrets are encrypted
Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
The command above reads all Secrets and then updates them to apply server side encryption.
Note:
If an error occurs due to a conflicting write, retry the command.
For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
Rotating a decryption key
Changing a Secret without incurring downtime requires a multi-step operation, especially in
the presence of a highly-available deployment where multiple
processes are running.
Generate a new key and add it as the second key entry for the current provider on all servers
Restart all
processes to ensure each server can decrypt using the new key
Make the new key the first entry in the
keys
array so that it is used for encryption in the config
processes to ensure each server now encrypts using the new key
Run
to encrypt all
existing Secrets with the new key
Remove the old decryption key from the config after you have backed up etcd with the new key in use
and updated all Secrets
When running a single
instance, step 2 may be skipped.
Decrypting all data
To disable encryption at rest, place the
identity
provider as the first entry in the config
and restart all
processes.
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
aescbc
:
keys
:
-
name
:
key1
secret
:
Then run the following command to force decrypt all Secrets:
EncryptionConfiguration configuration API (v1)
.
22 - Guaranteed Scheduling For Critical Add-On Pods
Kubernetes core components such as the API server, scheduler, and controller-manager run on a control plane node. However, add-ons must run on a regular cluster node.
Some of these add-ons are critical to a fully functional cluster, such as metrics-server, DNS, and UI.
A cluster may stop working properly if a critical add-on is evicted (either manually or as a side effect of another operation like upgrade)
and becomes pending (for example when the cluster is highly utilized and either there are other pending pods that schedule into the space
vacated by the evicted critical add-on pod or the amount of resources available on the node changed for some other reason).
Note that marking a pod as critical is not meant to prevent evictions entirely; it only prevents the pod from becoming permanently unavailable.
A static pod marked as critical, can't be evicted. However, a non-static pods ma

rked as critical are always rescheduled.
Marking pod as critical
To mark a Pod as critical, set priorityClassName for that Pod to
or
system-node-critical
.
is the highest available priority, even higher than
.
23 - IP Masquerade Agent User Guide
This page shows how to configure and enable the
ip-masq-agent
.
minikube
.
The
configures iptables rules to hide a pod's IP address behind the cluster node's IP address. This is typically done when sending traffic to destinations outside the cluster's pod
CIDR
range.
Key Terms
NAT (Network Address Translation)
Is a method of remapping one IP address to another by modifying either the source and/or destination address information in the IP header. Typically performed by a device doing IP routing.
Masquerading
A form of NAT that is typically used to perform a many to one address translation, where multiple source IP addresses are masked behind a single address, which is typically the device doing the IP routing. In Kubernetes this is the Node's IP address.
CIDR (Classless Inter-Domain Routing)
Based on the variable-length subnet masking, allows specifying arbitrary-length prefixes. CIDR introduced a new method of representation for IP addresses, now commonly known as
CIDR notation
, in which an address or routing prefix is written with a suffix indicating the number of bits of the prefix, such as 192.168.2.0/24.
Link Local
A link-local address is a network address that is valid only for communications within the network segment or the broadcast domain that the host is connected to. Link-local addresses for IPv4 are defined in the address block 169.254.0.0/16 in CIDR notation.
The ip-masq-agent configures iptables rules to handle masquerading node/pod IP addresses when sending traffic to destinations outside the cluster node's IP and the Cluster IP range. This essentially hides pod IP addresses behind the cluster node's IP address. In some environments, traffic to "external" addresses must come from a known machine address. For example, in Google Cloud, any traffic to the internet must come from a VM's IP. When containers are used, as in Google Kubernetes Engine, the Pod IP will be rejected for egress. To avoid this, we must hide the Pod IP behind the VM's own IP address - generally known as "masquerade". By default, the agent is configured to treat the three private IP ranges specified by
RFC 1918
as non-masquerade
CIDR
. These ranges are 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. The agent will also treat link-local (169.254.0.0/16) as a non-masquerade CIDR by default. The agent is configured to reload its configuration from the location
/etc/config/ip-masq-agent
every 60 seconds, which is also configurable.
The agent configuration file must be written in YAML or JSON syntax, and may contain three optional keys:
nonMasqueradeCIDRs
: A list of strings in
CIDR
notation that specify the non-masquerade ranges.
masqLinkLocal
: A Boolean (true/false) which indicates whether to masquerade traffic to the
link local prefix
169.254.0.0/16
. False by default.
resyncInterval
: A time interval at which the agent attempts to reload config from disk.
For example: '30s', where 's' means seconds, 'ms' means milliseconds.
Traffic to 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16) ranges will NOT be masqueraded. Any other traffic (assumed to be internet) will be masqueraded. An example of a local destination from a pod could be its Node's IP a

ddress as well as another node's address or one of the IP addresses in Cluster's IP range. Any other traffic will be masqueraded by default. The below entries show the default set of rules that are applied by the ip-masq-agent:
iptables -t nat -L IP-MASQ-AGENT
RETURN     all  --  anywhere             169.254.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             172.16.0.0/12        /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
RETURN     all  --  anywhere             192.168.0.0/16       /* ip-masq-agent: cluster-local traffic should not be subject to MASQUERADE */ ADDRTYPE match dst-type !LOCAL
MASQUERADE  all  --  anywhere             anywhere             /* ip-masq-agent: outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
By default, in GCE/Google Kubernetes Engine, if network policy is enabled or
you are using a cluster CIDR not in the 10.0.0.0/8 range, the
will run in your cluster. If you are running in another environment,
you can add the
DaemonSet
to your cluster.
Create an ip-masq-agent
To create an ip-masq-agent, run the following kubectl command:
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/ip-masq-agent/master/ip-masq-agent.yaml
You must also apply the appropriate node label to any nodes in your cluster that you want the agent to run on.
kubectl label nodes my-node node.kubernetes.io/masq-agent-ds-ready
=
true
More information can be found in the ip-masq-agent documentation
here
In most cases, the default set of rules should be sufficient; however, if this is not the case for your cluster, you can create and apply a
ConfigMap
to customize the IP ranges that are affected. For example, to allow only 10.0.0.0/8 to be considered by the ip-masq-agent, you can create the following
ConfigMap
in a file called "config".
Note:
It is important that the file is called config since, by default, that will be used as the key for lookup by the
:
:
-
10.0.0.0
/8
:
60s
Run the following command to add the config map to your cluster:
kubectl create configmap ip-masq-agent --from-file
=
config --namespace
=
This will update a file located at
which is periodically checked every
and applied to the cluster node.
After the resync interval has expired, you should see the iptables rules reflect your changes:
Chain IP-MASQ-AGENT (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             10.0.0.0/8           /* ip-masq-agent: cluster-local
By default, the link local range (169.254.0.0/16) is also handled by the ip-masq agent, which sets up the appropriate iptables rules. To have the ip-masq-agent ignore link local, you can set
to true in the ConfigMap.
:
-
10.0.0.0
/8
:
60s
:
true
24 - Limit Storage Consumption
,
,
and
.
You need to have a Ku

bernetes cluster, and the kubectl command-line tool must
minikube
.
Adding a
via
:
v1
kind
:
metadata
:
name
:
spec
:
limits
:
-
type
:
max
:
storage
:
2Gi
min
:
storage
:
1Gi
:
v1
kind
:
metadata
:
name
:
spec
:
hard
:
:
"5"
:
"5Gi"
Summary
A limit range can put a ceiling on how much storage is requested while a resource quota can effectively cap the storage
25 - Migrate Replicated Control Plane To Use Cloud Controller Manager
Kubernetes v1.24 [stable]
component
By decoupling the interoperability logic between Kubernetes and the underlying cloud
As part of the
cloud provider extraction effort
, all cloud specific controllers must be moved out of the
. All existing clusters that run cloud controllers in the
must migrate to instead run the controllers in a cloud provider specific
.
Leader Migration provides a mechanism in which HA clusters can safely migrate "cloud specific" controllers between the
and the
via a shared resource lock between the two components while upgrading the replicated control plane. For a single-node control plane, or if unavailability of controller managers can be tolerated during the upgrade, Leader Migration is not needed and this guide can be ignored.
Leader Migration ca

n be enabled by setting
--enable-leader-migration
on
or
. Leader Migration only applies during the upgrade and can be safely disabled or left enabled after the upgrade is complete.
This guide walks you through the manual process of upgrading the control plane from
with built-in cloud provider to running both
and
. If you use a tool to deploy and manage the cluster, please refer to the documentation of the tool and the cloud provider for specific instructions of the migration.
It is assumed that the control plane is running Kubernetes version N and to be upgraded to version N + 1. Although it is possible to migrate within the same version, ideally the migration should be performed as part of an upgrade so that changes of configuration can be aligned to each release. The exact versions of N and N + 1 depend on each cloud provider. For example, if a cloud provider builds a
to work with Kubernetes 1.24, then N can be 1.23 and N + 1 can be 1.24.
The control plane nodes should run
with Leader Election enabled, which is the default. As of version N, an in-tree cloud provider must be set with
flag and
should not yet be deployed.
The out-of-tree cloud provider must have built a
with Leader Migration implementation. If the cloud provider imports
k8s.io/cloud-provider
and
k8s.io/controller-manager
of version v0.21.0 or later, Leader Migration will be available. However, for version before v0.22.0, Leader Migration is alpha and requires feature gate
ControllerManagerLeaderMigration
to be enabled in
.
This guide assumes that kubelet of each control plane node starts
and
as static pods defined by their manifests. If the components run in a different setting, please adjust the steps accordingly.
For authorization, this guide assumes that the cluster uses RBAC. If another authorization mode grants permissions to
and
components, please grant the needed access in a way that matches the mode.
Grant access to Migration Lease
The default permissions of the controller manager allow only accesses to their main Lease. In order for the migration to work, accesses to another Lease are required.
You can grant
full access to the leases API by modifying the
system::leader-locking-kube-controller-manager
role. This task guide assumes that the name of the migration lease is
cloud-provider-extraction-migration
.
kubectl patch -n kube-system role 'system::leader-locking-kube-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge
Do the same to the
system::leader-locking-cloud-controller-manager
role.
kubectl patch -n kube-system role 'system::leader-locking-cloud-controller-manager' -p '{"rules": [ {"apiGroups":[ "coordination.k8s.io"], "resources": ["leases"], "resourceNames": ["cloud-provider-extraction-migration"], "verbs": ["create", "list", "get", "update"] } ]}' --type=merge
Initial Leader Migration configuration
Leader Migration optionally takes a configuration file representing the state of controller-to-manager assignment. At this moment, with in-tree cloud provider,
runs
route
,
service
, and
cloud-node-lifecycle
. The following example configuration shows the assignment.
Leader Migration can be enabled without a configuration. Please see
Default Configuration
kind
:
LeaderMigrationConfiguration
:
controllermanager.config.k8s.io/v1
leaderName
:
controllerLeaders
:
-
name
:
route
component
:
kube-controller-ma

nager
-
name
:
service
component
:
-
name
:
component
:
Alternatively, because the controllers can run under either controller managers, setting
component
to
*
for both sides makes the configuration file consistent between both parties of the migration.
# wildcard version
kind
:
:
:
:
-
name
:
route
component
:
*
-
name
:
service
component
:
*
-
name
:
component
:
*
On each control plane node, save the content to
/etc/leadermigration.conf
, and update the manifest of
so that the file is mounted inside the container at the same location. Also, update the same manifest to add the following arguments:
to enable Leader Migration on the controller manager
--leader-migration-config=/etc/leadermigration.conf
to set configuration file
Restart
on each node. At this moment,
has leader migration enabled and is ready for the migration.
Deploy Cloud Controller Manager
In version N + 1, the desired state of controller-to-manager assignment can be represented by a new configuration file, shown as follows. Please note
component
field of each
changing from
to
. Alternatively, use the wildcard version mentioned above, which has the same effect.
kind
:
:
:
:
-
name
:
route
component
:
-
name
:
service
component
:
-
name
:
component
:
When creating control plane nodes of version N + 1, the content should be deployed to
. The manifest of
should be updated to mount the configuration file in the same manner as
of version N. Similarly, add
and
to the arguments of
.
Create a new control plane node of version N + 1 with the updated
manifest, and with the
flag set to
external
for
.
of version N + 1 MUST NOT have Leader Migration enabled because, with an external cloud provider, it does not run the migrated controllers anymore, and thus it is not involved in the migration.
Please refer to
for more detail on how to deploy
.
Upgrade Control Plane
The control plane now contains nodes of both version N and N + 1. The nodes of version N run
only, and these of version N + 1 run both
and
. The migrated controllers, as specified in the configuration, are running under either
of version N or
of version N + 1 depending on which controller manager holds the migration lease. No controller will ever be running under both controller managers at any time.
In a rolling manner, create a new control plane node of version N + 1 and bring down one of version N + 1 until the control plane contains only nodes of version N + 1.
If a rollback from version N + 1 to N is required, add nodes of version N with Leader Migration enabled for
back to the control plane, replacing one of version N + 1 each time until there are only nodes of version N.
(Optional) Disable Leader Migration
Now that the control plane has been upgraded to run both
and
of version N + 1, Leader Migration has finished its job and can be safely disabled to save one Lease resou

rce. It is safe to re-enable Leader Migration for the rollback in the future.
In a rolling manager, update manifest of
to unset both
and
--leader-migration-config=
flag, also remove the mount of
, and finally remove
. To re-enable Leader Migration, recreate the configuration file and add its mount and the flags that enable Leader Migration back to
.
Starting Kubernetes 1.22, Leader Migration provides a default configuration suitable for the default controller-to-manager assignment.
The default configuration can be enabled by setting
but without
.
For
and
, if there are no flags that enable any in-tree cloud provider or change ownership of controllers, the default configuration can be used to avoid manual creation of the configuration file.
Special case: migrating the Node IPAM controller
If your cloud provider provides an implementation of Node IPAM controller, you should switch to the implementation in
. Disable Node IPAM controller in
of version N + 1 by adding
--controllers=*,-nodeipam
to its flags. Then add
nodeipam
to the list of migrated controllers.
# wildcard version, with nodeipam
kind
:
:
:
:
-
name
:
route
component
:
*
-
name
:
service
component
:
*
-
name
:
component
:
*
-
name
:
nodeipam
-
component
:
*
Read the
Controller Manager Leader Migration
enhancement proposal.
26 - Namespaces Walkthrough
help different projects, teams, or customers to share a Kubernetes cluster.
It does this by providing the following:
A scope for
Names
.
A mechanism to attach authorization and policy to a subsection of the cluster.
Use of multiple namespaces is optional.
This example demonstrates how to use Kubernetes namespaces to subdivide your cluster.
minikube
.
Prerequisites
This example assumes the following:
You have an
existing Kubernetes cluster
.
You have a basic understanding of Kubernetes
Pods
,
Services
, and
.
Understand the default namespace
By default, a Kubernetes cluster will instantiate a default namespace when provisioning the cluster to hold the default set of Pods,
Services, and Deployments used by the cluster.
Assuming you have a fresh cluster, you can inspect the available namespaces by doing the following:
kubectl get namespaces
NAME      STATUS    AGE
default   Active    13m
Create new namespaces
For this exercise, we will create two additional Kubernetes namespaces to hold our content.
Let's imagine a scenario where an organization is using a shared Kubernetes cluster for development and production use cases.
The development team would like to maintain a space in the cluster where they can get a view on the list of Pods, Services, and Deployments
they use to build and run their application. In this space, Kubernetes resources come and go, and the restrictions on who can or cannot modify resources
are relaxed to enable agile development.
The operations team would like to maintain a space in the cluster where they can enforce strict procedures on who can or cannot manipulate the set of
Pods, Services, and Deployments that run the production site.
One pattern this or

ganization could follow is to partition the Kubernetes cluster into two namespaces:
development
and
production
.
Let's create two new namespaces to hold our work.
Use the file
namespace-dev.yaml
which describes a
admin/namespace-dev.yaml
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
name
:
Create the
namespace using kubectl.
kubectl create -f https://k8s.io/examples/admin/namespace-dev.yaml
Save the following contents into file
namespace-prod.yaml
admin/namespace-prod.yaml
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
name
:
And then let's create the
kubectl create -f https://k8s.io/examples/admin/namespace-prod.yaml
To be sure things are right, let's list all of the namespaces in our cluster.
kubectl get namespaces --show-labels
NAME          STATUS    AGE       LABELS
default       Active    32m       &lt;none&gt;
development   Active    29s       name=development
production    Active    23s       name=production
Create pods in each namespace
A Kubernetes namespace provides the scope for Pods, Services, and Deployments in the cluster.
Users interacting with one namespace do not see the content in another namespace.
To demonstrate this, let's spin up a simple Deployment and Pods in the
We first check what is the current context:
:
v1
clusters
:
-
cluster
:
:
REDACTED
server
:
https://130.211.122.180
name
:
lithe-cocoa-92103_kubernetes
contexts
:
-
context
:
cluster
:
user
:
name
:
current-context
:
kind
:
Config
preferences
:
{}
users
:
-
name
:
user
:
:
REDACTED
:
REDACTED
token
:
65rZW78y8HbwXXtSXuUw9DbP4FLjHi4b
-
name
:
lithe-cocoa-92103_kubernetes-basic-auth
user
:
password
:
h5M0FtUUIflBSdI7
username
:
admin
kubectl config current-context
The next step is to define a context for the kubectl client to work in each namespace. The value of "cluster" and "user" fields are copied from the current context.
kubectl config set-context dev --namespace
=
\
--cluster
=
\
--user
=
kubectl config set-context prod --namespace
=
\
--cluster
=
\
--user
=
By default, the above commands adds two contexts that are saved into file
.kube/config
. You can now view the contexts and alternate against the two
new request contexts depending on which namespace you wish to work against.
To view the new contexts:
:
v1
clusters
:
-
cluster
:
:
REDACTED
server
:
name
:
contexts
:
-
context
:
cluster
:
user
:
name
:
-
context
:
cluster
:
namespace
:
user
:
name
:
dev
-
context
:
cluster
:
namespace
:
user
:
name
:
prod
:
kind
:
Config
:
{}
users
:
-
name
:
user
:
:
REDACTED
:
REDACTED
token
:
-
name
:
user
:
password
:
username
:
admin
Let's switch to operate in the
kubectl config use-context dev
You can verify your current context by doing the following:
dev
At this point, all requests we make to the Kubernetes cluster from the command line are scop

ed to the
Let's create some contents.
admin/snowflake-deployment.yaml
:
apps/v1
kind
:
metadata
:
labels
:
app
:
snowflake
name
:
snowflake
spec
:
replicas
:
2
selector
:
:
app
:
snowflake
template
:
metadata
:
labels
:
app
:
snowflake
spec
:
:
-
image
:
registry.k8s.io/serve_hostname
:
Always
name
:
snowflake
Apply the manifest to create a Deployment
kubectl apply -f https://k8s.io/examples/admin/snowflake-deployment.yaml
We have created a deployment whose replica size is 2 that is running the pod called
snowflake
with a basic container that serves the hostname.
kubectl get deployment
NAME         READY   UP-TO-DATE   AVAILABLE   AGE
snowflake    2/2     2            2           2m
app
=
snowflake
NAME                         READY     STATUS    RESTARTS   AGE
snowflake-3968820950-9dgr8   1/1       Running   0          2m
snowflake-3968820950-vgc4n   1/1       Running   0          2m
And this is great, developers are able to do what they want, and they do not have to worry about affecting content in the
Let's switch to the
namespace and show how resources in one namespace are hidden from the other.
kubectl config use-context prod
The
namespace should be empty, and the following commands should return nothing.
Production likes to run cattle, so let's create some cattle pods.
kubectl create deployment cattle --image
=
registry.k8s.io/serve_hostname --replicas
=
5
cattle       5/5     5            5           10s
app
=
cattle
NAME                      READY     STATUS    RESTARTS   AGE
cattle-2263376956-41xy6   1/1       Running   0          34s
cattle-2263376956-kw466   1/1       Running   0          34s
cattle-2263376956-n4v97   1/1       Running   0          34s
cattle-2263376956-p5p3i   1/1       Running   0          34s
cattle-2263376956-sxpth   1/1       Running   0          34s
At this point, it should be clear that the resources users create in one namespace are hidden from the other namespace.
As the policy support in Kubernetes evolves, we will extend this scenario to show how you can provide different
authorization rules for each namespace.
27 - Operating etcd clusters for Kubernetes
etcd is a consistent and highly-available key value store used as Kubernetes' backing store for all cluster data.
If your Kubernetes cluster uses etcd as its backing store, make sure you have a
back up
plan
for those data.
You can find in-depth information about etcd in the official
.
minikube
.
Run etcd as a cluster of odd members.
etcd is a leader-based distributed system. Ensure that the leader
periodically send heartbeats on time to all followers to keep the cluster
stable.
Ensure that no resource starvation occurs.
Performance and stability of the cluster is sensitive to network and disk
I/O. Any resource starvation can lead to heartbeat timeout, causing instability
of the cluster. An unstable etcd indicates that no leader is elected. Under
such circumstances, a cluster cannot make any changes to its current state,
which implies no new pods can be scheduled.
Keeping etcd clusters stable is critical to the stability of Kubernetes
clusters. Therefore, run etcd clusters on dedicated machines or isolated
environments for
guaranteed resource requirements
.
The minimum recommended version

 of etcd to run in production is
3.2.10+
.
Resource requirements
Operating etcd with limited resources is suitable only for testing purposes.
For deploying in production, advanced hardware configuration is required.
Before deploying etcd in production, see
resource requirement reference
.
Starting etcd clusters
This section covers starting a single-node and multi-node etcd cluster.
Single-node etcd cluster
Use a single-node etcd cluster only for testing purpose.
Run the following:
etcd --listen-client-urls
=
http://
$PRIVATE_IP
:2379
\
--advertise-client-urls
=
http://
:2379
Start the Kubernetes API server with the flag
--etcd-servers=$PRIVATE_IP:2379
.
Make sure
PRIVATE_IP
is set to your etcd client IP.
Multi-node etcd cluster
For durability and high availability, run etcd as a multi-node cluster in
production and back it up periodically. A five-member cluster is recommended
in production. For more information, see
FAQ documentation
.
Configure an etcd cluster either by static member information or by dynamic
discovery. For more information on clustering, see
etcd clustering documentation
.
For an example, consider a five-member etcd cluster running with the following
client URLs:
http://$IP1:2379
,
http://$IP2:2379
,
http://$IP3:2379
,
http://$IP4:2379
, and
http://$IP5:2379
. To start a Kubernetes API server:
=
http://
$IP1
:2379,http://
$IP2
$IP3
$IP4
$IP5
:2379 --advertise-client-urls
=
http://
$IP1
$IP2
$IP3
$IP4
$IP5
:2379
Start the Kubernetes API servers with the flag
--etcd-servers=$IP1:2379,$IP2:2379,$IP3:2379,$IP4:2379,$IP5:2379
.
Make sure the
IP&lt;n&gt;
variables are set to your client IP addresses.
Multi-node etcd cluster with load balancer
To run a load balancing etcd cluster:
Set up an etcd cluster.
Configure a load balancer in front of the etcd cluster.
For example, let the address of the load balancer be
$LB
.
Start Kubernetes API Servers with the flag
--etcd-servers=$LB:2379
.
Securing etcd clusters
Access to etcd is equivalent to root permission in the cluster so ideally only
the API server should have access to it. Considering the sensitivity of the
data, it is recommended to grant permission to only those nodes that require
access to etcd clusters.
To secure etcd, either set up firewall rules or use the security features
provided by etcd. etcd security features depend on x509 Public Key
Infrastructure (PKI). To begin, establish secure communication channels by
generating a key and certificate pair. For example, use key pairs
peer.key
and
peer.cert
for securing communication between etcd members, and
client.key
and
client.cert
for securing communication between etcd and its
clients. See the
example scripts
provided by the etcd project to generate key pairs and CA files for client
authentication.
Securing communication
To configure etcd with secure peer communication, specify flags
--peer-key-file=peer.key
and
--peer-cert-file=peer.cert
, and use HTTPS as
the URL schema.
Similarly, to configure etcd with secure client communication, specify flags
--key-file=k8sclient.key
and
--cert-file=k8sclient.cert
the URL schema. Here is an example on a client command that uses secure
communication:
ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379 \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  member list
Limiting access of etcd clusters
After configuring secure communication, restrict the access of etcd cluster to
only the Kubernetes API servers. Use TLS authentication to do so.
For example, consider key pairs
k8sclient.key
and
k8sclient.cert
that are
trusted by the CA
etcd.ca
. When etcd is configured with
--client-cert-auth
along with TLS, it verifies the certificates from clients by using system CAs
or the CA passed in by
--trusted-ca-file
flag. Specif

ying flags
--client-cert-auth=true
and
--trusted-ca-file=etcd.ca
will restrict the
access to clients with the certificate
.
Once etcd is configured correctly, only clients with valid certificates can
access it. To give Kubernetes API servers the access, configure them with the
flags
--etcd-certfile=k8sclient.cert
,
--etcd-keyfile=k8sclient.key
and
--etcd-cafile=ca.cert
.
Note:
etcd authentication is not currently supported by Kubernetes. For more
information, see the related issue
Support Basic Auth for Etcd v2
.
Replacing a failed etcd member
etcd cluster achieves high availability by tolerating minor member failures.
However, to improve the overall health of the cluster, replace failed members
immediately. When multiple members fail, replace them one by one. Replacing a
failed member involves two steps: removing the failed member and adding a new
member.
Though etcd keeps unique member IDs internally, it is recommended to use a
unique name for each member to avoid human errors. For example, consider a
three-member etcd cluster. Let the URLs be,
member1=http://10.0.0.1
,
member2=http://10.0.0.2
, and
member3=http://10.0.0.3
. When
member1
fails,
replace it with
member4=http://10.0.0.4
.
Get the member ID of the failed
member1
:
etcdctl --endpoints
=
http://10.0.0.2,http://10.0.0.3 member list
The following message is displayed:
8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379
91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379
fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379
Do either of the following:
If each Kubernetes API server is configured to communicate with all etcd
members, remove the failed member from the
--etcd-servers
flag, then
restart each Kubernetes API server.
If each Kubernetes API server communicates with a single etcd member,
then stop the Kubernetes API server that communicates with the failed
etcd.
Stop the etcd server on the broken node. It is possible that other
clients besides the Kubernetes API server is causing traffic to etcd
and it is desirable to stop all traffic to prevent writes to the data
dir.
Remove the failed member:
etcdctl member remove 8211f1d0f64f3269
Removed member 8211f1d0f64f3269 from cluster
Add the new member:
etcdctl member add member4 --peer-urls
=
http://10.0.0.4:2380
Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4
Start the newly added member on a machine with the IP
10.0.0.4
:
export
ETCD_NAME
=
"member4"
export
ETCD_INITIAL_CLUSTER
=
"member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380"
export
ETCD_INITIAL_CLUSTER_STATE
=
existing
etcd
[
flags
]
members, add the newly added member to the
flag, then
start the Kubernetes API server that was stopped in step 2. Then
configure Kubernetes API server clients to again route requests to the
Kubernetes API server that was stopped. This can often be done by
configuring a load balancer.
For more information on cluster reconfiguration, see
etcd reconfiguration documentation
.
Backing up an etcd cluster
All Kubernetes objects are stored on etcd. Periodically backing up the etcd
cluster data is important to recover Kubernetes clusters under disaster
scenarios, such as losing all control plane nodes. The snapshot file contains
all the Kubernetes states and critical information. In order to keep the
sensitive Kubernetes data safe, encrypt the snapshot files.
Backing up an etcd cluster can be accomplished in two ways: etcd built-in
snapshot and volume snapshot.
Built-in snapshot
etcd supports built-in snapshot. A snapshot may either be taken from a live
member with the
etcdctl snapshot save
command or by copyin

g the
member/snap/db
file from an etcd
data directory
that is not currently used by an etcd process. Taking the snapshot will
not affect the performance of the member.
Below is an example for taking a snapshot of the keyspace served by
$ENDPOINT
to the file
snapshotdb
:
=
3
$ENDPOINT
snapshot save snapshotdb
Verify the snapshot:
=
3
etcdctl --write-out
=
table snapshot status snapshotdb
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
| fe01cf57 |       10 |          7 | 2.1 MB     |
Volume snapshot
If etcd is running on a storage volume that supports backup, such as Amazon
Elastic Block Store, back up etcd data by taking a snapshot of the storage
volume.
Snapshot using etcdctl options
We can also take the snapshot using various options given by etcdctl. For example
=
3
etcdctl -h
will list various options available from etcdctl. For example, you can take a snapshot by specifying
the endpoint, certificates etc as shown below:
=
3
=
https://127.0.0.1:2379
\
--cacert
=
&lt;trusted-ca-file&gt; --cert
=
&lt;cert-file&gt; --key
=
&lt;key-file&gt;
\
snapshot save &lt;backup-file-location&gt;
where
trusted-ca-file
,
cert-file
and
key-file
can be obtained from the description of the etcd Pod.
Scaling out etcd clusters
Scaling out etcd clusters increases availability by trading off performance.
Scaling does not increase cluster performance nor capability. A general rule
is not to scale out or in etcd clusters. Do not configure any auto scaling
groups for etcd clusters. It is highly recommended to always run a static
five-member etcd cluster for production Kubernetes clusters at any officially
supported scale.
A reasonable scaling is to upgrade a three-member cluster to a five-member
one, when more reliability is desired. See
for information on how to add members into an existing cluster.
Restoring an etcd cluster
etcd supports restoring from snapshots that are taken from an etcd process of
the
major.minor
version. Restoring a version from a
different patch version of etcd also is supported. A restore operation is
employed to recover the data of a failed cluster.
Before starting the restore operation, a snapshot file must be present. It can
either be a snapshot file from a previous backup operation, or from a remaining
.
=
3
etcdctl --endpoints 10.2.0.9:2379 snapshot restore snapshotdb
Another example for restoring using etcdctl options:
=
3
etcdctl snapshot restore --data-dir &lt;data-dir-location&gt; snapshotdb
Yet another example would be to first export the environment variable
export
=
3
For more information and examples on restoring a cluster from a snapshot file, see
etcd disaster recovery documentation
.
If the access URLs of the restored cluster is changed from the previous
cluster, the Kubernetes API server must be reconfigured accordingly. In this
case, restart Kubernetes API servers with the flag
--etcd-servers=$NEW_ETCD_CLUSTER
instead of the flag
--etcd-servers=$OLD_ETCD_CLUSTER
. Replace
$NEW_ETCD_CLUSTER
and
$OLD_ETCD_CLUSTER
with the respective IP addresses. If a load balancer is
used in front of an etcd cluster, you might need to update the load balancer
instead.
If the majority of etcd members have permanently failed, the etcd cluster is
considered failed. In this scenario, Kubernetes cannot make any changes to its
current state. Although the scheduled pods might continue to run, no new pods
can be scheduled. In such cases, recover the etcd cluster and potentially
reconfigure Kubernetes API servers to fix the issue.
Note:
If any API servers are running in your cluster, you should not attempt to
restore insta

nces of etcd. Instead, follow these steps to restore etcd:
stop
all
API server instances
restore state in all etcd instances
restart all API server instances
We also recommend restarting any components (e.g.
,
,
kubelet
) to ensure that they don't rely on some
stale data. Note that in practice, the restore takes a bit of time. During the
restoration, critical components will lose leader lock and restart themselves.
Upgrading etcd clusters
For more details on etcd upgrade, please refer to the
etcd upgrades
Note:
Before you start an upgrade, please back up your etcd cluster first.
28 - Reconfigure a Node's Kubelet in a Live Cluster
Kubernetes v1.22 [deprecated]
Caution:
The
Dynamic Kubelet Configuration
feature is deprecated in 1.22 and removed in 1.24.
Please switch to alternative means distributing configuration to the Nodes of your cluster.
allowed you to change the configuration of each
kubelet
in a running Kubernetes cluster,
by deploying a
ConfigMap
and configuring
each
Node
to use it.
Please find documentation on this feature in
earlier versions of documentation
.
Migrating from using Dynamic Kubelet Configuration
There is no recommended replacement for this feature that works generically
across various Kubernetes distributions. If you are using managed Kubernetes
version, please consult with the vendor hosting Kubernetes for the best
practices for customizing your Kubernetes. If you are using
kubeadm
, refer to
.
In order to migrate off the Dynamic Kubelet Configuration feature, the
alternative mechanism should be used to distribute kubelet configuration files.
In order to apply configuration, config file must be updated and kubelet restarted.
See the
for information.
Please note, the
DynamicKubeletConfig
feature gate cannot be set on a kubelet
starting v1.24 as it has no effect. However, the feature gate is not removed
from the API server or the controller manager before v1.26. This is designed for
the control plane to support nodes with older versions of kubelets and for
satisfying the
Kubernetes version skew policy
.
29 - Reserve Compute Resources for System Daemons
Kubernetes nodes can be scheduled to
Capacity
. Pods can consume all the
available capacity on a node by default. This is an issue because nodes
typically run quite a few system daemons that power the OS and Kubernetes
itself. Unless resources are set aside for these system daemons, pods and system
daemons compete for resources and lead to resource starvation issues on the
node.
The
kubelet
exposes a feature named 'Node Allocatable' that helps to reserve
compute resources for system daemons. Kubernetes recommends cluster
administrators to configure 'Node Allocatable' based on their workload density
on each node.
minikube
Your Kubernetes server must be at or later than version 1.8.
.
Your Kubernetes server must be at or later than version 1.17 to use
the kubelet command line option
to set an
explicitly reserved CPU list
.
Node Allocatable
'Allocatable' on a Kubernetes node is defined as the amount of compute resources
that are available for pods. The scheduler does not over-subscribe
'Allocatable'. 'CPU', 'memory' and 'ephemeral-storage' are supported as of now.
Node Allocatable is exposed as part of
v1.Node
object in the API and as part
of
kubectl describe node
in the CLI.
Resources can be reserved fo

r two categories of system daemons in the
kubelet
.
Enabling QoS and Pod level cgroups
To properly enforce node allocatable constraints on the node, you must
enable the new cgroup hierarchy via the
--cgroups-per-qos
flag. This flag is
enabled by default. When enabled, the
kubelet
will parent all end-user pods
under a cgroup hierarchy managed by the
kubelet
.
The
kubelet
supports manipulation of the cgroup hierarchy on
the host using a cgroup driver. The driver is configured via the
--cgroup-driver
flag.
The supported values are the following:
cgroupfs
is the default driver that performs direct manipulation of the
cgroup filesystem on the host in order to manage cgroup sandboxes.
systemd
is an alternative driver that manages cgroup sandboxes using
transient slices for resources that are supported by that init system.
Depending on the configuration of the associated container runtime,
operators may have to choose a particular cgroup driver to ensure
proper system behavior. For example, if operators use the
systemd
cgroup driver provided by the
runtime, the
kubelet
must
be configured to use the
systemd
cgroup driver.
Kube Reserved
Kubelet Flag
:
--kube-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]
:
--kube-reserved-cgroup=
kube-reserved
is meant to capture resource reservation for kubernetes system
daemons like the
kubelet
,
,
, etc.
It is not meant to reserve resources for system daemons that are run as pods.
is typically a function of
pod density
on the nodes.
In addition to
cpu
,
memory
, and
ephemeral-storage
,
pid
may be
specified to reserve the specified number of process IDs for
kubernetes system daemons.
To optionally enforce
on kubernetes system daemons, specify the parent
control group for kube daemons as the value for
--kube-reserved-cgroup
kubelet
flag.
It is recommended that the kubernetes system daemons are placed under a top
level control group (
runtime.slice
on systemd machines for example). Each
system daemon should ideally run within its own child control group. Refer to
the design proposal
for more details on recommended control group hierarchy.
Note that Kubelet
does not
create
if it doesn't
exist. Kubelet will fail if an invalid cgroup is specified. With
systemd
cgroup driver, you should follow a specific pattern for the name of the cgroup you
define: the name should be the value you set for
,
with
.slice
appended.
System Reserved
:
--system-reserved=[cpu=100m][,][memory=100Mi][,][ephemeral-storage=1Gi][,][pid=1000]
:
--system-reserved-cgroup=
system-reserved
is meant to capture resource reservation for OS system daemons
like
sshd
,
udev
, etc.
should reserve
memory
for the
kernel
too since
kernel
memory is not accounted to pods in Kubernetes at this time.
Reserving resources for user login sessions is also recommended (
user.slice
in
systemd world).
cpu
,
memory
, and
,
pid
may be
specified to reserve the specified number of process IDs for OS system
daemons.
on system daemons, specify the parent
control group for OS system daemons as the value for
--system-reserved-cgroup
kubelet flag.
It is recommended that the OS system daemons are placed under a top level
control group (
system.slice
on systemd machines for example).
Note that
kubelet
does not
create
exist.
kubelet
will fail if an invalid cgroup is specified. With
systemd
,
with
.slice
appended.
Explicitly Reserved CPU List
Kubernetes v1.17 [stable]
:
--reserved-cpus=0-3
KubeletConfiguration Flag
:
reservedSystemCpus: 0-3
reserved

-cpus
is meant to define an explicit CPU set for OS system daemons and
reserved-cpus
is for systems that do not intend to
define separate top level cgroups for OS system daemons and kubernetes system daemons
with regard to cpuset resource.
If the Kubelet
does not
have
and
,
the explicit cpuset provided by
will take precedence over the CPUs
defined by
and
options.
This option is specifically designed for Telco/NFV use cases where uncontrolled
interrupts/timers may impact the workload performance. you can use this option
to define the explicit cpuset for the system/kubernetes daemons as well as the
interrupts/timers, so the rest CPUs on the system can be used exclusively for
workloads, with less impact from uncontrolled interrupts/timers. To move the
system daemon, kubernetes daemons and interrupts/timers to the explicit cpuset
defined by this option, other mechanism outside Kubernetes should be used.
For example: in Centos, you can do this using the tuned toolset.
Eviction Thresholds
:
--eviction-hard=[memory.available&lt;500Mi]
Memory pressure at the node level leads to System OOMs which affects the entire
node and all pods running on it. Nodes can go offline temporarily until memory
has been reclaimed. To avoid (or reduce the probability of) system OOMs kubelet
provides
out of resource
management. Evictions are
supported for
memory
and
only. By reserving some memory via
--eviction-hard
flag, the
kubelet
attempts to evict pods whenever memory
availability on the node drops below the reserved value. Hypothetically, if
system daemons did not exist on a node, pods cannot use more than
capacity - eviction-hard
. For this reason, resources reserved for evictions are not
available for pods.
Enforcing Node Allocatable
:
--enforce-node-allocatable=pods[,][system-reserved][,][kube-reserved]
The scheduler treats 'Allocatable' as the available
capacity
for pods.
kubelet
enforce 'Allocatable' across pods by default. Enforcement is performed
by evicting pods whenever the overall usage across all pods exceeds
'Allocatable'. More details on eviction policy can be found
on the
node pressure eviction
page. This enforcement is controlled by
specifying
pods
value to the kubelet flag
--enforce-node-allocatable
.
Optionally,
kubelet
can be made to enforce
and
by specifying
&amp;
values in
the same flag. Note that to enforce
or
,
or
needs to be specified
respectively.
General Guidelines
System daemons are expected to be treated similar to
.
System daemons can burst within their bounding control groups and this behavior needs
to be managed as part of kubernetes deployments. For example,
kubelet
should
have its own control group and share
resources with the
container runtime. However, Kubelet cannot burst and use up all available Node
resources if
is enforced.
Be extra careful while enforcing
reservation since it can lead
to critical system services being CPU starved, OOM killed, or unable
to fork on the node. The
recommendation is to enforce
only if a user has profiled their
nodes exhaustively to come up with precise estimates and is confident in their
ability to recover if any process in that group is oom-killed.
To begin with enforce 'Allocatable' on
pods
.
Once adequate monitoring and alerting is in place to track kube system
daemons, attempt to enforce
based on usage heuristics.
If absolutely necessary, enforce
over time.
The resource requirements of kube system daemons may grow over time as more and
more features are added. Over time, kubernetes project will attempt to bring
down utilization of node system daemons, but that is not a priority

 as of now.
So expect a drop in
Allocatable
capacity in future releases.
Example Scenario
Here is an example to illustrate Node Allocatable computation:
Node has
32Gi
of
memory
,
16 CPUs
and
100Gi
of
Storage
is set to
cpu=1,memory=2Gi,ephemeral-storage=1Gi
is set to
cpu=500m,memory=1Gi,ephemeral-storage=1Gi
is set to
memory.available&lt;500Mi,nodefs.available&lt;10%
Under this scenario, 'Allocatable' will be 14.5 CPUs, 28.5Gi of memory and
88Gi
of local storage.
Scheduler ensures that the total memory
requests
across all pods on this node does
not exceed 28.5Gi and storage doesn't exceed 88Gi.
Kubelet evicts pods whenever the overall memory usage across pods exceeds 28.5Gi,
or if overall disk usage exceeds 88Gi If all processes on the node consume as
much CPU as they can, pods together cannot consume more than 14.5 CPUs.
If
and/or
is not enforced and system daemons
exceed their reservation,
kubelet
evicts pods whenever the overall node memory
usage is higher than 31.5Gi or
storage
is greater than 90Gi.
30 - Running Kubernetes Node Components as a Non-root User
.
.
Note:
.
.
and
Enable the
kind
kind
See
.
minikube
minikube
Note:
sysbox
Sysbox
See
/proc
and
/sys
Note:
K3s
K3s
experimentally supports rootless mode.
See
Usernetes
Usernetes
$HOME
See
Manually deploy a node that runs the kubele

t in a user namespace
Note:
.
with
.
You do
not
chroot()
nor
in
in
outside
/etc
/run
/var/logs
Note:
cgroup v2
Note:
,
VPNKit
, or
.
NodePort
socat(1)
.
See
the
package
of k3s.
CRI-O
version =
2
[plugins.
]
true
true
true
[plugins.
[plugins.
false
/etc/containerd/co

nfig.toml
.
.
to be set.
[crio]
"overlay"
]
.
.
:
kind
:
:
:
true
:
When the
.
/dev/kmsg
.
.
The
Sysbox
:
kind
:
mode
:
conntrack
:
:
0
:
0s
:
0s
Caveats
nfs
and
iscsi
local
,
hostPath
,
emptyDir
,
configMap
,
secret
, and
page
See Also
Usernetes
31 - Safely Drain a Node
This page shows how to safely drain a
node
,
optionally respecting the PodDisruptionBudget you have defined.
Your Kubernetes server must be at or later than version 1.5.
.
This task also assumes that you have met the following prerequisites:
You do not require your applications to be highly available during the
node drain, or
You have read about the
concept,
and have
configured PodDisruptionBudgets
for
applications that need them.
(Optional) Configure a disruption budget
To ensure that your workloads remain available during maintenance, you can
configure a
.
If availability is important for any applications that run or could run on the node(s)
that you are draining,
configure a PodDisruptionBudgets
first and then continue following this guide.
Use
kubectl drain
to remove a node from service
You can use
to safely evict all of your pods from a
node before you perform maintenance on the node (e.g. kernel upgrade

,
hardware maintenance, etc.). Safe evictions allow the pod's containers
to
gracefully terminate
and will respect the PodDisruptionBudgets you have specified.
Note:
By default
ignores certain system pods on the node
that cannot be killed; see
the
documentation for more details.
When
returns successfully, that indicates that all of
the pods (except the ones excluded as described in the previous paragraph)
have been safely evicted (respecting the desired graceful termination period,
and respecting the PodDisruptionBudget you have defined). It is then safe to
bring down the node by powering down its physical machine or, if running on a
cloud platform, deleting its virtual machine.
First, identify the name of the node you wish to drain. You can list all of the nodes in your cluster with
Next, tell Kubernetes to drain the node:
kubectl drain &lt;node name&gt;
Once it returns (without giving an error), you can power down the node
(or equivalently, if on a cloud platform, delete the virtual machine backing the node).
If you leave the node in the cluster during the maintenance operation, you need to run
kubectl uncordon &lt;node name&gt;
afterwards to tell Kubernetes that it can resume scheduling new pods onto the node.
Draining multiple nodes in parallel
The
command should only be issued to a single node at a
time. However, you can run multiple
commands for
different nodes in parallel, in different terminals or in the
background. Multiple drain commands running concurrently will still
respect the PodDisruptionBudget you specify.
For example, if you have a StatefulSet with three replicas and have
set a PodDisruptionBudget for that set specifying
minAvailable: 2
,
only evicts a pod from the StatefulSet if all three
replicas pods are
healthy
;
if then you issue multiple drain commands in parallel,
Kubernetes respects the PodDisruptionBudget and ensures that
only 1 (calculated as
replicas - minAvailable
) Pod is unavailable
at any given time. Any drains that would cause the number of
healthy
replicas to fall below the specified budget are blocked.
The Eviction API
If you prefer not to use
(such as
to avoid calling to an external command, or to get finer control over the pod
eviction process), you can also programmatically cause evictions using the
eviction API.
For more information, see
API-initiated eviction
.
Follow steps to protect your application by
configuring a Pod Disruption Budget
.
32 - Securing a Cluster
This document covers topics related to protecting a cluster from accidental or malicious access
and provides recommendations on overall security.
minikube
.
Controlling access to the Kubernetes API
As Kubernetes is entirely API-driven, controlling and limiting who can access the cluster and what actions
they are allowed to perform is the first line of defense.
Use Transport Layer Security (TLS) for all API traffic
Kubernetes expects that all API communication in the cluster is encrypted by default with TLS, and the
majority of installation methods will allow the necessary certificates to be created and distributed to
the cluster components. Note that some components and installation methods may enable local ports over
HTTP and administrators should familiarize themselves with the settings of each component to identify
potentially unsecured traffic.
API Authentication
Choose an authentication mechanism for the API servers to use that matches the common a

ccess patterns
when you install a cluster. For instance, small, single-user clusters may wish to use a simple certificate
or static Bearer token approach. Larger clusters may wish to integrate an existing OIDC or LDAP server that
allow users to be subdivided into groups.
All API clients must be authenticated, even those that are part of the infrastructure like nodes,
proxies, the scheduler, and volume plugins. These clients are typically
service accounts
or use x509 client certificates, and they are created automatically at cluster startup or are setup as part of the cluster installation.
Consult the
authentication reference document
API Authorization
Once authenticated, every API call is also expected to pass an authorization check. Kubernetes ships
an integrated
Role-Based Access Control (RBAC)
component that matches an incoming user or group to a
set of permissions bundled into roles. These permissions combine verbs (get, create, delete) with
resources (pods, services, nodes) and can be namespace-scoped or cluster-scoped. A set of out-of-the-box
roles are provided that offer reasonable default separation of responsibility depending on what
actions a client might want to perform. It is recommended that you use the
Node
and
RBAC
authorizers together, in combination with the
NodeRestriction
admission plugin.
As with authentication, simple and broad roles may be appropriate for smaller clusters, but as
more users interact with the cluster, it may become necessary to separate teams into separate
with more limited roles.
With authorization, it is important to understand how updates on one object may cause actions in
other places. For instance, a user may not be able to create pods directly, but allowing them to
create a deployment, which creates pods on their behalf, will let them create those pods
indirectly. Likewise, deleting a node from the API will result in the pods scheduled to that node
being terminated and recreated on other nodes. The out-of-the box roles represent a balance
between flexibility and common use cases, but more limited roles should be carefully reviewed
to prevent accidental escalation. You can make roles specific to your use case if the out-of-box ones don't meet your needs.
authorization reference section
Controlling access to the Kubelet
Kubelets expose HTTPS endpoints which grant powerful control over the node and containers.
By default Kubelets allow unauthenticated access to this API.
Production clusters should enable Kubelet authentication and authorization.
Kubelet authentication/authorization reference
Controlling the capabilities of a workload or user at runtime
Authorization in Kubernetes is intentionally high level, focused on coarse actions on resources.
More powerful controls exist as
policies
to limit by use case how those objects act on the
cluster, themselves, and other resources.
Limiting resource usage on a cluster
Resource quota
limits the number or capacity of
resources granted to a namespace. This is most often used to limit the amount of CPU, memory,
or persistent disk a namespace can allocate, but can also control how many pods, services, or
volumes exist in each namespace.
Limit ranges
restrict the maximum or minimum size of some of the
resources above, to prevent users from requesting unreasonably high or low values for commonly
reserved resources like memory, or to provide default limits when none are specified.
Controlling what privileges containers run with
A pod definition contains a
security context
that allows it to request access to run as a specific Linux user on a node (like root),
access to run privileged or access the host network, and other controls that would otherwise
allow it to run unfettered on a hosting node.
You can configure
Pod security admission
to enforce use of a particular
Pod Security Standard
in a
namespace
, or to detect breaches.
Generally, most applica

tion workloads need limited access to host resources so they can
successfully run as a root process (uid 0) without access to host information. However,
considering the privileges associated with the root user, you should write application
containers to run as a non-root user. Similarly, administrators who wish to prevent
client applications from escaping their containers should apply the
Baseline
or
Pod Security Standard.
Preventing containers from loading unwanted kernel modules
The Linux kernel automatically loads kernel modules from disk if needed in certain
circumstances, such as when a piece of hardware is attached or a filesystem is mounted. Of
particular relevance to Kubernetes, even unprivileged processes can cause certain
network-protocol-related kernel modules to be loaded, just by creating a socket of the
appropriate type. This may allow an attacker to exploit a security hole in a kernel module
that the administrator assumed was not in use.
To prevent specific modules from being automatically loaded, you can uninstall them from
the node, or add rules to block them. On most Linux distributions, you can do that by
creating a file such as
/etc/modprobe.d/kubernetes-blacklist.conf
with contents like:
# DCCP is unlikely to be needed, has had multiple serious
# vulnerabilities, and is not well-maintained.
blacklist dccp

# SCTP is not used in most Kubernetes clusters, and has also had
# vulnerabilities in the past.
blacklist sctp
To block module loading more generically, you can use a Linux Security Module (such as
SELinux) to completely deny the
module_request
permission to containers, preventing the
kernel from loading modules for containers under any circumstances. (Pods would still be
able to use modules that had been loaded manually, or modules that were loaded by the
kernel on behalf of some more-privileged process.)
Restricting network access
The
network policies
for a namespace
allows application authors to restrict which pods in other namespaces may access pods and ports
within their namespaces. Many of the supported
Kubernetes networking providers
now respect network policy.
Quota and limit ranges can also be used to control whether users may request node ports or
load-balanced services, which on many clusters can control whether those users applications
are visible outside of the cluster.
Additional protections may be available that control network rules on a per-plugin or per-
environment basis, such as per-node firewalls, physically separating cluster nodes to
prevent cross talk, or advanced networking policy.
Restricting cloud metadata API access
Cloud platforms (AWS, Azure, GCE, etc.) often expose metadata services locally to instances.
By default these APIs are accessible by pods running on an instance and can contain cloud
credentials for that node, or provisioning data such as kubelet credentials. These credentials
can be used to escalate within the cluster or to other cloud services under the same account.
When running Kubernetes on a cloud platform, limit permissions given to instance credentials, use
to restrict pod access
to the metadata API, and avoid using provisioning data to deliver secrets.
Controlling which nodes pods may access
By default, there are no restrictions on which nodes may run a pod. Kubernetes offers a
rich set of policies for controlling placement of pods onto nodes
and the
taint-based pod placement and eviction
that are available to end users. For many clusters use of these policies to separate workloads
can be a convention that authors adopt or enforce via tooling.
As an administrator, a beta admission plugin
PodNodeSelector
can be used to force pods
within a namespace to default or require a specific node selector, and if end users cannot
alter namespaces, this can strongly limit the placement of all of the pods in a specific workload.
Protecting cluster components from compromise
This section describes some common patterns for protecting clusters f

rom compromise.
Restrict access to etcd
Write access to the etcd backend for the API is equivalent to gaining root on the entire cluster,
and read access can be used to escalate fairly quickly. Administrators should always use strong
credentials from the API servers to their etcd server, such as mutual auth via TLS client certificates,
and it is often recommended to isolate the etcd servers behind a firewall that only the API servers
may access.
Caution:
Allowing other components within the cluster to access the master etcd instance with
read or write access to the full keyspace is equivalent to granting cluster-admin access. Using
separate etcd instances for non-master components or using etcd ACLs to restrict read and write
access to a subset of the keyspace is strongly recommended.
Enable audit logging
The
audit logger
is a beta feature that records actions taken by the
API for later analysis in the event of a compromise. It is recommended to enable audit logging
and archive the audit file on a secure server.
Restrict access to alpha or beta features
Alpha and beta Kubernetes features are in active development and may have limitations or bugs
that result in security vulnerabilities. Always assess the value an alpha or beta feature may
provide against the possible risk to your security posture. When in doubt, disable features you
do not use.
Rotate infrastructure credentials frequently
The shorter the lifetime of a secret or credential the harder it is for an attacker to make
use of that credential. Set short lifetimes on certificates and automate their rotation. Use
an authentication provider that can control how long issued tokens are available and use short
lifetimes where possible. If you use service-account tokens in external integrations, plan to
rotate those tokens frequently. For example, once the bootstrap phase is complete, a bootstrap
token used for setting up nodes should be revoked or its authorization removed.
Review third party integrations before enabling them
Many third party integrations to Kubernetes may alter the security profile of your cluster. When
enabling an integration, always review the permissions that an extension requests before granting
it access. For example, many security integrations may request access to view all secrets on
your cluster which is effectively making that component a cluster admin. When in doubt,
restrict the integration to functioning in a single namespace if possible.
Components that create pods may also be unexpectedly powerful if they can do so inside namespaces
like the
namespace, because those pods can gain access to service account secrets
or run with elevated permissions if those service accounts are granted access to permissive
PodSecurityPolicies
.
If you use
Pod Security admission
and allow
any component to create Pods within a namespace that permits privileged Pods, those Pods may
be able to escape their containers and use this widened access to elevate their privileges.
You should not allow untrusted components to create Pods in any system namespace (those with
names that start with
kube-
) nor in any namespace where that access grant allows the possibility
of privilege escalation.
Encrypt secrets at rest
In general, the etcd database will contain any information accessible via the Kubernetes API
and may grant an attacker significant visibility into the state of your cluster. Always encrypt
your backups using a well reviewed backup and encryption solution, and consider using full disk
encryption where possible.
Kubernetes supports optional
encryption at rest
for information in the Kubernetes API.
This lets you ensure that when Kubernetes stores data for objects (for example,
Secret
or
ConfigMap
objects), the API server writes an encrypted representation of the object.
That encryption means that even someone who has access to etcd backup data is unable
to view the content of those objects.
In Kubernetes 1.26 you can also encrypt custom resources;
encryp

tion-at-rest for extension APIs defined in CustomResourceDefinitions was added to
Kubernetes as part of the v1.26 release.
Receiving alerts for security updates and reporting vulnerabilities
Join the
kubernetes-announce
group for emails about security announcements. See the
security reporting
page for more on how to report vulnerabilities.
33 - Set Kubelet parameters via a config file
A subset of the Kubelet's configuration parameters may be
set via an on-disk config file, as a substitute for command-line flags.
Providing parameters via a config file is the recommended approach because
it simplifies node deployment and configuration management.
Create the config file
The subset of the Kubelet's configuration that can be configured via a file
is defined by the
struct.
The configuration file must be a JSON or YAML representation of the parameters
in this struct. Make sure the Kubelet has read permissions on the file.
Here is an example of what this file might look like:
:
kind
:
address
:
"192.168.0.8"
port
:
20250
serializeImagePulls
:
false
evictionHard
:
memory.available
:
"200Mi"
In the example, the Kubelet is configured to serve on IP address 192.168.0.8 and port 20250, pull images in parallel,
and evict Pods when available memory drops below 200Mi. Since only one of the four evictionHard thresholds is configured,
other evictionHard thresholds are reset to 0 from their built-in defaults.
All other Kubelet configuration values are left at their built-in defaults, unless overridden
by flags. Command line flags which target the same value as a config file will override that value.
Note:
In the example, by changing the default value of only one parameter for
evictionHard, the default values of other parameters will not be inherited and
will be set to zero. In order to provide custom values, you should provide all
the threshold values respectively.
Start a Kubelet process configured via the config file
Note:
If you use kubeadm to initialize your cluster, use the kubelet-config while creating your cluster with
.
See
configuring kubelet using kubeadm
Start the Kubelet with the
--config
flag set to the path of the Kubelet's config file.
The Kubelet will then load its config from this file.
Note that command line flags which target the same value as a config file will override that value.
This helps ensure backwards compatibility with the command-line API.
Note that relative file paths in the Kubelet config file are resolved relative to the
location of the Kubelet config file, whereas relative paths in command line flags are resolved
relative to the Kubelet's current working directory.
Note that some default values differ between command-line flags and the Kubelet config file.
If
--config
is provided and the values are not specified via the command line, the
defaults for the
version apply.
In the above example, this version is
.
Learn more about kubelet configuration by checking the
reference.
34 - Share a Cluster with Namespaces
This page shows how to view, work in, and delete
. The page also shows how to use Kubernetes namespaces to subdivide your cluster.
Have an
.
Pods
,
Services
, and
.
Viewing namespaces
List the current namespaces in a cluster using:
NAME          STATUS    AGE
default       Active    11d
kube-system   Active    11d
kube-public   Active    11d
Kubernetes starts with three initial namespaces:
default
The default namespace for objects with no other namespace
The namespace for objects created by the Kubernetes system
kube-public
This namespace is created automatically and is readable by all users (including those not authenticated). This namespace is mostly reserved for cluster

 usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.
You can also get the summary of a specific namespace using:
kubectl get namespaces &lt;name&gt;
Or you can get detailed information with:
kubectl describe namespaces &lt;name&gt;
Name:           default
Labels:         &lt;none&gt;
Annotations:    &lt;none&gt;
Status:         Active

No resource quota.

Resource Limits
 Type       Resource    Min Max Default
 ----               --------    --- --- ---
 Container          cpu         -   -   100m
Note that these details show both resource quota (if present) as well as resource limit ranges.
Resource quota tracks aggregate usage of resources in the
Namespace
and allows cluster operators
to define
Hard
resource usage limits that a
Namespace
may consume.
A limit range defines min/max constraints on the amount of resources a single entity can consume in
a
Namespace
.
See
Admission control: Limit Range
A namespace can be in one of two phases:
Active
the namespace is in use
Terminating
the namespace is being deleted, and can not be used for new objects
For more details, see
Namespace
in the API reference.
Creating a new namespace
Note:
Avoid creating namespace with prefix
kube-
, since it is reserved for Kubernetes system namespaces.
Create a new YAML file called
my-namespace.yaml
with the contents:
:
v1
kind
:
Namespace
metadata
:
name
:
&lt;insert-namespace-name-here&gt;
Then run:
kubectl create -f ./my-namespace.yaml
Alternatively, you can create namespace using below command:
kubectl create namespace &lt;insert-namespace-name-here&gt;
The name of your namespace must be a valid
DNS label
.
There's an optional field
finalizers
, which allows observables to purge resources whenever the namespace is deleted. Keep in mind that if you specify a nonexistent finalizer, the namespace will be created but will get stuck in the
state if the user tries to delete it.
More information on
can be found in the namespace
design doc
.
Deleting a namespace
Delete a namespace with
kubectl delete namespaces &lt;insert-some-namespace-name&gt;
Warning:
This deletes
everything
under the namespace!
This delete is asynchronous, so for a time you will see the namespace in the
state.
Subdividing your cluster using Kubernetes namespaces
Assuming you have a fresh cluster, you can introspect the available namespaces by doing the following:
In a scenario where an organization is using a shared Kubernetes cluster for development and production use cases:
One pattern this organization could follow is to partition the Kubernetes cluster into two namespaces:
and
.
Create the
namespace using kubectl:
kubectl create -f https://k8s.io/examples/admin/namespace-dev.json
kubectl create -f https://k8s.io/examples/admin/namespace-

prod.json
To be sure things are right, list all of the namespaces in our cluster.
kubectl create deployment snowflake --image
=
registry.k8s.io/serve_hostname  -n
=
development --replicas
=
2
snowflake
kubectl get deployment -n
=
app
=
snowflake -n
=
The
=
kubectl get pods -n
=
=
registry.k8s.io/serve_hostname -n
=
kubectl scale deployment cattle --replicas
=
5
-n
=
=
app
=
cattle -n
=
Understanding the motivation for using namespaces
A single cluster should be able to satisfy the needs of multiple users or groups of users (henceforth a 'user community').
A scope for
Names
.
Each user community wants to be able to work in isolation from other communities.
Each user community has its own:
resources (pods, services, replication controllers, etc.)
policies (who can or cannot perform actions in their community)
constraints (this community is allowed this much quota, etc.)
A cluster operator may create a Namespace for each unique user community.
The Namespace provides a unique scope for:
named resources (to avoid basic naming collisions)
delegated management authority to trusted users
ability to limit community resource consumption
Use cases include:
As a cluster operator, I want to support multiple user communities on a single cluster.
As a cluster operator, I want to delegate authority to partitions of the cluster to trusted users
in those communities.
As a cluster operator, I want to limit the amount of re

sources each community can consume in order
to limit the impact to other communities using the cluster.
As a cluster user, I want to interact with resources that are pertinent to my user community in
isolation of what other user communities are doing on the cluster.
Understanding namespaces and DNS
When you create a
Service
, it creates a corresponding
DNS entry
.
This entry is of the form
&lt;service-name&gt;.&lt;namespace-name&gt;.svc.cluster.local
, which means
that if a container uses
&lt;service-name&gt;
it will resolve to the service which
is local to a namespace. This is useful for using the same configuration across
multiple namespaces such as Development, Staging and Production. If you want to reach
across namespaces, you need to use the fully qualified domain name (FQDN).
setting the namespace preference
.
setting the namespace for a request
See
namespaces design
.
35 - Upgrade A Cluster
This page provides an overview of the steps you should follow to upgrade a
Kubernetes cluster.
The way that you upgrade a cluster depends on how you initially deployed it
and on any subsequent changes.
At a high level, the steps you perform are:
Upgrade the
Upgrade the nodes in your cluster
Upgrade clients such as
kubectl
Adjust manifests and other resources based on the API changes that accompany the
new Kubernetes version
You must have an existing cluster. This page is about upgrading from Kubernetes
1.25 to Kubernetes 1.26. If your cluster
is not currently running Kubernetes 1.25 then please check
the documentation for the version of Kubernetes that you plan to upgrade to.
Upgrade approaches
kubeadm
If your cluster was deployed using the
kubeadm
tool, refer to
for detailed information on how to upgrade the cluster.
Once you have upgraded the cluster, remember to
install the latest version of
kubectl
.
Manual deployments
Caution:
These steps do not account for third-party extensions such as network and storage
plugins.
You should manually update the control plane following this sequence:
etcd (all instances)
kube-apiserver (all control plane hosts)
cloud controller manager, if you use one
At this point you should
kubectl
.
For each node in your cluster,
drain
that node and then either replace it with a new node that uses the 1.26
kubelet, or upgrade the kubelet on that node and bring the node back into service.
Other deployments
Refer to the documentation for your cluster deployment tool to learn the recommended set
up steps for maintenance.
Post-upgrade tasks
Switch your cluster's storage API version
The objects that are serialized into etcd for a cluster's internal
representation of the Kubernetes resources active in the cluster are
written using a particular version of the API.
When the supported API changes, these objects may need to be rewritten
in the newer API. Failure to do this will eventually result in resources
that are no longer decodable or usable by the Kubernetes API server.
For each affected object, fetch it using the latest supported API and then
write it back also using the latest supported API.
Update manifests
Upgrading to a new Kubernetes version can provide new APIs.
You can use
kubectl convert
command to convert manifests between different API versions.
kubectl convert -f pod.yaml --output-version v1
The
kubectl
tool replaces the contents of
pod.yaml
with a manifest that sets
kind
to
Pod (unchanged), but with a revised
.
If your cluster is running device plugins and the node needs to be upgraded to a Kubernetes
release with a newer device plugin API version, device plugins must be upgraded to support
both version before the node is upgraded in order to guarantee that device allocations
continue to complete successfully during the upgrade.
Refer to
API compatiblity
and
Kubelet Device Manager API Versio

ns
36 - Use Cascading Deletion in a Cluster
This page shows you how to specify the type of
cascading deletion
to use in your cluster during
garbage collection
.
minikube
You also need to
create a sample Deployment
to experiment with the different types of cascading deletion. You will need to
recreate the Deployment for each type.
Check owner references on your pods
Check that the
ownerReferences
field is present on your pods:
app
=
nginx --output
=
yaml
The output has an
field similar to this:
:
v1
...
:
-
:
apps/v1
blockOwnerDeletion
:
true
:
true
kind
:
name
:
nginx-deployment-6b474476c4
uid
:
4fdcd81c-bd5d-41f7-97af-3a3b759af9a7
...
Use foreground cascading deletion
By default, Kubernetes uses
background cascading deletion
to delete dependents of an object. You can switch to foreground cascading deletion
kubectl
or the Kubernetes API, depending on the Kubernetes
version your cluster runs.
.
You can delete objects using foreground cascading deletion using
kubectl
or the
Kubernetes API.
Using kubectl
Run the following command:
kubectl delete deployment nginx-deployment --cascade
=
foreground
Using the Kubernetes API
Start a local proxy session:
=
8080
Use
curl
to trigger deletion:
curl -X DELETE localhost:8080/apis/apps/v1/namespaces/default/deployments/nginx-deployment
\
-d
'{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Foreground"}'
\
-H
"Content-Type: application/json"
The output contains a
foregroundDeletion
finalizer
like this:
"kind": "Deployment",
"apiVersion": "apps/v1",
"metadata": {
    "name": "nginx-deployment",
    "namespace": "default",
    "uid": "d1ce1b02-cae8-4288-8a53-30e84d8fa505",
    "resourceVersion": "1363097",
    "creationTimestamp": "2021-07-08T20:24:37Z",
    "deletionTimestamp": "2021-07-08T20:27:39Z",
    "finalizers": [
      "foregroundDeletion"
    ]
    ...
Use background cascading deletion
Create a sample Deployment
.
Use either
kubectl
or the Kubernetes API to delete the Deployment,
depending on the Kubernetes version your cluster runs.
.
You can delete objects using background cascading deletion using
kubectl
or the Kubernetes API.
Kubernetes uses background cascading deletion by default, and does so
even if you run the following commands without the
--cascade
flag or the
argument.
=
background
=
8080
Use
curl
\
-d
'{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Background"}'
\
-H
"kind": "Status",
"apiVersion": "v1",
...
"status": "Success",
"details": {
    "group": "apps",
    "kind": "deployments",
    "uid": "cc9eefb9-2d49-4445-b1c1-d261c9396456"
}
Delete owner objects and orphan dependents
By default, when you tell Kubernetes to delete an object, the
also deletes
dependent objects. You can make Kubernetes
orphan
these dependents using
kubectl
or the Kubernetes API, depending on the Kubernetes version your
cluster runs.
.
kubectl dele

te deployment nginx-deployment --cascade
=
orphan
=
8080
Use
curl
\
-d
'{"kind":"DeleteOptions","apiVersion":"v1","propagationPolicy":"Orphan"}'
\
-H
The output contains
orphan
in the
field, similar to this:
"namespace": "default",
"uid": "6f577034-42a0-479d-be21-78018c466f1f",
"creationTimestamp": "2021-07-09T16:46:37Z",
"deletionTimestamp": "2021-07-09T16:47:08Z",
"deletionGracePeriodSeconds": 0,
"finalizers": [
  "orphan"
],
...
You can check that the Pods managed by the Deployment are still running:
app
=
nginx
owners and dependents
Learn about Kubernetes
.
.
37 - Using a KMS provider for data encryption
This page shows how to configure a Key Management Service (KMS) provider and plugin to enable secret data encryption. Currently there are two KMS API versions. KMS v1 will continue to work while v2 develops in maturity. If you are not sure which KMS API version to pick, choose v1.
minikube
The version of Kubernetes that you need depends on which KMS API version
you have selected.
If you selected KMS API v1, any supported Kubernetes version will work fine.
If you selected KMS API v2, you should use Kubernetes v1.26
(if you are running a different version of Kubernetes that also supports the v2 KMS
API, switch to the documentation for that version of Kubernetes).
.
KMS v1
Kubernetes version 1.10.0 or later is required
Your cluster must use etcd v3 or later
Kubernetes v1.12 [beta]
KMS v2
Kubernetes version 1.25.0 or later is required
Set kube-apiserver feature gate:
--feature-gates=KMSv2=true
to configure a KMS v2 provider
The KMS encryption provider uses an envelope encryption scheme to encrypt data in etcd.
The data is encrypted using a data encryption key (DEK); a new DEK is generated for each encryption.
The DEKs are encrypted with a key encryption key (KEK) that is stored and managed in a remote KMS.
The KMS provider uses gRPC to communicate with a specific KMS plugin.
The KMS plugin, which is implemented as a gRPC server and deployed on the same host(s)
as the Kubernetes control plane, is responsible for all communication with the remote KMS.
Configuring the KMS provider
To configure a KMS provider on the API server, include a provider of type
kms
in the
providers
array in the encryption configuration file and set the following properties:
KMS v1
name
: Display name of the KMS plugin. Cannot be changed once set.
endpoint
: Listen address of the gRPC server (KMS plugin). The endpoint is a UNIX domain socket.
cachesize
: Number of data encryption keys (DEKs) to be cached in the clear.
When cached, DEKs can be used without another call to the KMS;
whereas DEKs that are not cached require a call to the KMS to unwrap.
timeout
: How long should
wait for kms-plugin to respond before
returning an error (default is 3 seconds).
KMS v2
: API Version for KMS provider (Allowed values: v2, v1 or empty. Any other value will result in an error.) Must be set to v2 to use the KMS v2 APIs.
name
endpoint
: Listen address of the gRPC server (

KMS plugin). The endpoint is a UNIX domain socket.
cachesize
timeout
See
.
Implementing a KMS plugin
To implement a KMS plugin, you can develop a new plugin gRPC server or enable a KMS plugin
already provided by your cloud provider.
You then integrate the plugin with the remote KMS and deploy it on the Kubernetes master.
Enabling the KMS supported by your cloud provider
Refer to your cloud provider for instructions on enabling the cloud provider-specific KMS plugin.
Developing a KMS plugin gRPC server
You can develop a KMS plugin gRPC server using a stub file available for Go. For other languages,
you use a proto file to create a stub file that you can use to develop the gRPC server code.
KMS v1
Using Go: Use the functions and data structures in the stub file:
api.pb.go
to develop the gRPC server code
Using languages other than Go: Use the protoc compiler with the proto file:
api.proto
to generate a stub file for the specific language
KMS v2
api.pb.go
api.proto
Then use the functions and data structures in the stub file to develop the server code.
Notes
KMS v1
kms plugin version:
v1beta1
In response to procedure call Version, a compatible KMS plugin should return
v1beta1
as
VersionResponse.version
.
message version:
v1beta1
All messages from KMS provider have the version field set to current version v1beta1.
protocol: UNIX domain socket (
unix
)
The plugin is implemented as a gRPC server that listens at UNIX domain socket. The plugin deployment should create a file on the file system to run the gRPC unix domain socket connection. The API server (gRPC client) is configured with the KMS provider (gRPC server) unix domain socket endpoint in order to communicate with it. An abstract Linux socket may be used by starting the endpoint with
/@
, i.e.
unix:///@foo
. Care must be taken when using this type of socket as they do not have concept of ACL (unlike traditional file based sockets). However, they are subject to Linux networking namespace, so will only be accessible to containers within the same pod unless host networking is used.
KMS v2
v2alpha1
In response to procedure call Status, a compatible KMS plugin should return
v2alpha1
as
StatusResponse.Version
, "ok" as
StatusResponse.Healthz
and a keyID (KMS KEK ID) as
StatusResponse.KeyID
unix
)
/@
, i.e.
Integrating a KMS plugin with the remote KMS
The KMS plugin can communicate with the remote KMS using any protocol supported by the KMS.
All configuration data, including authentication credentials the KMS plugin uses to communicate with the remote KMS,
are stored and managed by the KMS plugin independently.
The KMS plugin can encode the ciphertext with additional metadata that may

 be required before sending it to the KMS for decryption.
Deploying the KMS plugin
Ensure that the KMS plugin runs on the same host(s) as the Kubernetes master(s).
Encrypting your data with the KMS provider
To encrypt the data:
Create a new
file using the appropriate properties for the
kms
provider
to encrypt resources like Secrets and ConfigMaps. If you want to encrypt an extension API that is
defined in a CustomResourceDefinition, your cluster must be running Kubernetes v1.26 or newer.
Set the
flag on the kube-apiserver to point to the location of the configuration file.
should be automatically reloaded if the disk contents change. This enables key rotation without API server restarts.
KMS v1
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
kms
:
name
:
myKmsPluginFoo
endpoint
:
unix:///tmp/socketfile.sock
cachesize
:
100
timeout
:
3s
-
kms
:
name
:
myKmsPluginBar
endpoint
:
cachesize
:
100
timeout
:
3s
KMS v2
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
kms
:
:
v2
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
-
kms
:
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
Setting
to
true
collapses all health checks to a single health check endpoint. Individual health checks are only available when KMS v1 providers are in use and the encryption config is not auto-reloaded.
Following table summarizes the health check endpoints for each KMS version:
KMS configurations
Without Automatic Reload
With Automatic Reload
KMS v1 only
Individual Healthchecks
Single Healthcheck
KMS v2 only
Both KMS v1 and v2
No KMS
None
means that the only health check endpoint is
/healthz/kms-providers
.
means that each KMS plugin has an associated health check endpoint based on its location in the encryption config:
/healthz/kms-provider-0
,
/healthz/kms-provider-1
etc.
These healthcheck endpoint paths are hard coded and generated/controlled by the server. The indices for individual healthchecks corresponds to the order in which the KMS encryption config is processed.
Until the steps defined in
Ensuring all secrets are encrypted
are performed, the
providers
list should end with the
identity: {}
provider to allow unencrypted data to be read. Once all resources are encrypted, the
identity
provider should be removed to prevent the API server from honoring unencrypted data.
For details about the
format, please check the
API server encryption API reference
.
Verifying that the data is encrypted
,
any newly created or updated Secret or other resource types configured in
should be encrypted when stored. To verify,
you can use the
etcdctl
command line program to retrieve the contents of your secret data.
Create a new secret called
secret1
in the
default
=
mykey
=
mydata
Using the
etcdctl
command line, read that secret out of etcd:
=
3
etcdctl get /kubernetes.io/secrets/default/secret1
[
...
]
| hexdump -C
where
[...]
contains the additional arguments for connecting to the etcd server.
Verify the stored secret is prefixed with
k8s:enc:kms:v1:
for KMS v1 or prefixed with
k8s:enc:kms:v2:
for KMS v2, which indicates that the
km

s
provider has encrypted the resulting data.
Verify that the secret is correctly decrypted when retrieved via the API:
kubectl describe secret secret1 -n default
The Secret should contain
mykey: mydata
Because secrets are encrypted on write, performing an update on a secret encrypts that content.
The following command reads all secrets and then updates them to apply server side encryption.
Switching from a local encryption provider to the KMS provider
To switch from a local encryption provider to the
kms
provider and re-encrypt all of the secrets:
Add the
kms
provider as the first entry in the configuration file as shown in the following example.
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
kms
:
name
:
myKmsPlugin
endpoint
:
cachesize
:
100
-
aescbc
:
keys
:
-
name
:
key1
secret
:
Run the following command to force all secrets to be re-encrypted using the
kms
provider.
Disabling encryption at rest
To disable encryption at rest:
Place the
identity
provider as the first entry in the configuration file:
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
kms
:
name
:
endpoint
:
cachesize
:
100
Run the following command to force all secrets to be decrypted.
38 - Using CoreDNS for Service Discovery
This page describes the CoreDNS upgrade process and how to install CoreDNS instead of kube-dns.
minikube
Your Kubernetes server must be at or later than version v1.9.
.
About CoreDNS
CoreDNS
is a flexible, extensible DNS server
that can serve as the Kubernetes cluster DNS.
Like Kubernetes, the CoreDNS project is hosted by the
CNCF
.
You can use CoreDNS instead of kube-dns in your cluster by replacing
kube-dns in an existing deployment, or by using tools like kubeadm
that will deploy and upgrade the cluster for you.
Installing CoreDNS
For manual deployment or replacement of kube-dns, see the documentation at the
CoreDNS GitHub project.
Migrating to CoreDNS
Upgrading an existing cluster with kubeadm
In Kubernetes version 1.21, kubeadm removed its support for
kube-dns
as a DNS application.
For
kubeadm
v1.26, the only supported cluster DNS application
is CoreDNS.
You can move to CoreDNS when you use
kubeadm
to upgrade a cluster that is
using
kube-dns
. In this case,
kubeadm
generates the CoreDNS configuration
("Corefile") based upon the
kube-dns
ConfigMap, preserving configurations for
stub domains, and upstream name server.
Upgrading CoreDNS
You can check the version of CoreDNS that kubeadm installs for each version of
Kubernetes in the page
CoreDNS version in Kubernetes
.
CoreDNS can be upgraded manually in case you want to only upgrade CoreDNS
or use your own custom image.
There is a helpful
guideline and walkthrough
available to ensure a smooth upgrade.
Make sure the existing CoreDNS configuration ("Corefile") is retained when
upgrading your cluster.
If you are upgradi

ng your cluster using the
kubeadm
tool,
kubeadm
can take care of retaining the existing CoreDNS configuration automatically.
Tuning CoreDNS
When resource utilisation is a concern, it may be useful to tune the
configuration of CoreDNS. For more details, check out the
documentation on scaling CoreDNS
.
CoreDNS
to support many more use cases than
kube-dns does by modifying the CoreDNS configuration ("Corefile").
For more information, see the
for the
CoreDNS plugin, or read the
Custom DNS Entries for Kubernetes
.
in the CoreDNS blog.
39 - Using NodeLocal DNSCache in Kubernetes Clusters
Kubernetes v1.18 [stable]
This page provides an overview of NodeLocal DNSCache feature in Kubernetes.
minikube
.
NodeLocal DNSCache improves Cluster DNS performance by running a DNS caching agent
on cluster nodes as a DaemonSet. In today's architecture, Pods in 'ClusterFirst' DNS mode
reach out to a kube-dns
serviceIP
for DNS queries. This is translated to a
kube-dns/CoreDNS endpoint via iptables rules added by kube-proxy.
With this new architecture, Pods will reach out to the DNS caching agent
running on the same node, thereby avoiding iptables DNAT rules and connection tracking.
The local caching agent will query kube-dns service for cache misses of cluster
hostnames ("
" suffix by default).
Motivation
With the current DNS architecture, it is possible that Pods with the highest DNS QPS
have to reach out to a different node, if there is no local kube-dns/CoreDNS instance.
Having a local cache will help improve the latency in such scenarios.
Skipping iptables DNAT and connection tracking will help reduce
conntrack races
and avoid UDP DNS entries filling up conntrack table.
Connections from the local caching agent to kube-dns service can be upgraded to TCP.
TCP conntrack entries will be removed on connection close in contrast with
UDP entries that have to timeout
(
default
nf_conntrack_udp_timeout
is 30 seconds)
Upgrading DNS queries from UDP to TCP would reduce tail latency attributed to
dropped UDP packets and DNS timeouts usually up to 30s (3 retries + 10s timeout).
Since the nodelocal cache listens for UDP DNS queries, applications don't need to be changed.
Metrics &amp; visibility into DNS requests at a node level.
Negative caching can be re-enabled, thereby reducing the number of queries for the kube-dns service.
Architecture Diagram
This is the path followed by DNS Queries after NodeLocal DNSCache is enabled:
Nodelocal DNSCache flow
This image shows how NodeLocal DNSCache handles DNS queries.
Note:
The local listen IP address for NodeLocal DNSCache can be any address that
can be guaranteed to not collide with any existing IP in your cluster.
It's recommended to use an address with a local scope, for example,
from the 'link-local' range '169.254.0.0/16' for IPv4 or from the
'Unique Local Address' range in IPv6 'fd00::/8'.
This feature can be enabled using the following steps:
Prepare a manifest similar to the sample
nodelocaldns.yaml
and save it as
nodelocaldns.yaml.
If using IPv6, the CoreDNS configuration file needs to enclose all the IPv6 addresses
into square brackets if used in 'IP:Port' format.
If you are using the sample manifest from the previous point, this will require you to modify
the configuration line L70
like this: "
health [__PILLAR__LOCAL__DNS__]:8080
"
Substitute the variables in the manifest with the right values:
kubedns
=
`
kubectl get svc kube-dns -n kube-system -o

jsonpath
={
.spec.clusterIP
}
`
domain
=
&lt;cluster-domain&gt;
localdns
=
&lt;node-local-address&gt;
is "
" by default.
is the
local listen IP address chosen for NodeLocal DNSCache.
If kube-proxy is running in IPTABLES mode:
sed -i
"s/__PILLAR__LOCAL__DNS__/
$localdns
/g; s/__PILLAR__DNS__DOMAIN__/
$domain
/g; s/__PILLAR__DNS__SERVER__/
$kubedns
/g"
__PILLAR__CLUSTER__DNS__
and
__PILLAR__UPSTREAM__SERVERS__
will be populated by
the
node-local-dns
pods.
In this mode, the
pods listen on both the kube-dns service IP
as well as
, so pods can look up DNS records using either IP address.
If kube-proxy is running in IPVS mode:
sed -i
$localdns
$domain
/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/
$kubedns
/g"
pods listen only on
.
The
interface cannot bind the kube-dns cluster IP since the
interface used for IPVS loadbalancing already uses this address.
will be populated by the node-local-dns pods.
Run
kubectl create -f nodelocaldns.yaml
If using kube-proxy in IPVS mode,
--cluster-dns
flag to kubelet needs to be modified
to use
that NodeLocal DNSCache is listening on.
Otherwise, there is no need to modify the value of the
flag,
since NodeLocal DNSCache listens on both the kube-dns service IP as well as
.
Once enabled, the
Pods will run in the
namespace
on each of the cluster nodes. This Pod runs
CoreDNS
in cache mode, so all CoreDNS metrics exposed by the different plugins will
be available on a per-node basis.
You can disable this feature by removing the DaemonSet, using
kubectl delete -f &lt;manifest&gt;
.
You should also revert any changes you made to the kubelet configuration.
StubDomains and Upstream server Configuration
StubDomains and upstream servers specified in the
kube-dns
namespace
are automatically picked up by
pods. The ConfigMap contents need to follow the format
shown in
the example
.
The
ConfigMap can also be modified directly with the stubDomain configuration
in the Corefile format. Some cloud providers might not allow modifying
ConfigMap directly.
In those cases, the
kube-dns
ConfigMap can be updated.
Setting memory limits
The
Pods use memory for storing cache entries and processing queries.
Since they do not watch Kubernetes objects, the cluster size or the number of Services / EndpointSlices do not directly affect memory usage. Memory usage is influenced by the DNS query pattern.
From
CoreDNS docs
,
The default cache size is 10000 entries, which uses about 30 MB when completely filled.
This would be the memory usage for each server block (if the cache gets completely filled).
Memory usage can be reduced by specifying smaller cache sizes.
The number of concurrent queries is linked to the memory demand, because each extra
goroutine used for handling a query requires an amount of memory. You can set an upper limit
using the
max_concurrent
option in the forward plugin.
If a
Pod attempts to use more memory than is available (because of total system
resources, or because of a configured
resource limit
), the operating system
may shut down that pod's container.
If this happens, the container that is terminated (“OOMKilled”) does not clean up the custom
packet filtering rules that it previously added during startup.
The
container should get restarted (since managed as part of a DaemonSet), but this
will lead to a brief DNS downtime each time that the container fails: the packet filtering rules direct
DNS queries to a local Pod that is unhealthy.
You can determine a suitable memory limit by runn

ing node-local-dns pods without a limit and
measuring the peak usage. You can also set up and use a
VerticalPodAutoscaler
in
recommender mode
, and then check its recommendations.
40 - Using sysctls in a Kubernetes Cluster
This document describes how to configure and use kernel parameters within a
Kubernetes cluster using the
sysctl
interface.
Note:
Starting from Kubernetes version 1.23, the kubelet supports the use of either
/
or
.
as separators for sysctl names.
Starting from Kubernetes version 1.25, setting Sysctls for a Pod supports setting sysctls with slashes.
For example, you can represent the same sysctl name as
kernel.shm_rmid_forced
using a
period as the separator, or as
kernel/shm_rmid_forced
using a slash as a separator.
For more sysctl parameter conversion method details, please refer to
the page
sysctl.d(5)
from
the Linux man-pages project.
minikube
For some steps, you also need to be able to reconfigure the command line
options for the kubelets running on your cluster.
Listing all Sysctl Parameters
In Linux, the sysctl interface allows an administrator to modify kernel
parameters at runtime. Parameters are available via the
/proc/sys/
virtual
process file system. The parameters cover various subsystems such as:
kernel (common prefix:
kernel.
)
networking (common prefix:
net.
)
virtual memory (common prefix:
vm.
)
MDADM (common prefix:
dev.
)
More subsystems are described in
Kernel docs
.
To get a list of all parameters, you can run
sudo sysctl -a
Enabling Unsafe Sysctls
Sysctls are grouped into
safe
and
unsafe
sysctls. In addition to proper
namespacing, a
safe
sysctl must be properly
isolated
between pods on the
same node. This means that setting a
safe
sysctl for one pod
must not have any influence on any other pod on the node
must not allow to harm the node's health
must not allow to gain CPU or memory resources outside of the resource limits
of a pod.
By far, most of the
namespaced
sysctls are not necessarily considered
safe
.
The following sysctls are supported in the
safe
set:
,
net.ipv4.ip_local_port_range
,
net.ipv4.tcp_syncookies
,
net.ipv4.ping_group_range
(since Kubernetes 1.18),
net.ipv4.ip_unprivileged_port_start
(since Kubernetes 1.22).
Note:
The example
is not namespaced on Linux kernel version 4.4 or lower.
This list will be extended in future Kubernetes versions when the kubelet
supports better isolation mechanisms.
All
safe
sysctls are enabled by default.
All
unsafe
sysctls are disabled by default and must be allowed manually by the
cluster admin on a per-node basis. Pods with disabled unsafe sysctls will be
scheduled, but will fail to launch.
With the warning above in mind, the cluster admin can allow certain
unsafe
sysctls for very special situations such as high-performance or real-time
application tuning.
Unsafe
sysctls are enabled on a node-by-node basis with a
flag of the kubelet; for example:
kubelet --allowed-unsafe-sysctls
\
'kernel.msg*,net.core.somaxconn'
...
For
Minikube
, this can be done via the
extra-config
flag:
minikube start --extra-config
=
"kubelet.allowed-unsafe-sysctls=kernel.msg*,net.core.somaxconn"
...
Only
sysctls can be enabled this way.
Setting Sysctls for a Pod
A number of sysctls are
in today's Linux kernels. This means that
they can be set independently for each pod on a node. Only namespaced sysctls
are configurable via the pod securityContext within Kubernetes.
The following sysctls are known to be namespaced. This list could change
in future versions

 of the Linux kernel.
kernel.shm*
,
kernel.msg*
,
kernel.sem
,
fs.mqueue.*
,
The parameters under
net.*
that can be set in container networking
namespace. However, there are exceptions (e.g.,
net.netfilter.nf_conntrack_max
and
net.netfilter.nf_conntrack_expect_max
can be set in container networking namespace but they are unnamespaced).
Sysctls with no namespace are called
node-level
sysctls. If you need to set
them, you must manually configure them on each node's operating system, or by
using a DaemonSet with privileged containers.
Use the pod securityContext to configure namespaced sysctls. The securityContext
applies to all containers in the same pod.
This example uses the pod securityContext to set a safe sysctl
and two unsafe sysctls
net.core.somaxconn
and
kernel.msgmax
. There is no distinction between
safe
and
unsafe
sysctls in
the specification.
Warning:
Only modify sysctl parameters after you understand their effects, to avoid
destabilizing your operating system.
:
v1
kind
:
Pod
metadata
:
name
:
sysctl-example
spec
:
:
sysctls
:
-
name
:
value
:
"0"
-
name
:
value
:
"1024"
-
name
:
value
:
"65536"
...
Warning:
Due to their nature of being
unsafe
, the use of
unsafe
sysctls
is at-your-own-risk and can lead to severe problems like wrong behavior of
containers, resource shortage or complete breakage of a node.
It is good practice to consider nodes with special sysctl settings as
tainted
within a cluster, and only schedule pods onto them which need those
sysctl settings. It is suggested to use the Kubernetes
taints and toleration
feature
to implement this.
A pod with the
unsafe
sysctls will fail to launch on any node which has not
enabled those two
unsafe
sysctls explicitly. As with
sysctls it
is recommended to use
feature
or
taints on nodes
to schedule those pods onto the right nodes.
41 - Utilizing the NUMA-aware Memory Manager
Kubernetes v1.22 [beta]
enables the feature of guaranteed memory (and hugepages)
allocation for pods in the
QoS class
.
The Memory Manager employs hint generation protocol to yield the most suitable NUMA affinity for a pod.
The Memory Manager feeds the central manager (
) with these affinity hints.
Based on both the hints and Topology Manager policy, the pod is rejected or admitted to the node.
Moreover, the Memory Manager ensures that the memory which a pod requests
is allocated from a minimum number of NUMA nodes.
The Memory Manager is only pertinent to Linux based hosts.
minikube
Your Kubernetes server must be at or later than version v1.21.
.
To align memory resources with other requested resources in a Pod spec:
the CPU Manager should be enabled and proper CPU Manager policy should be configured on a Node.
See
;
the Topology Manager should be enabled and proper Topology Manager policy should be configured on a Node.
See
control Topology Management Policies
.
Starting from v1.22, the Memory Manager is enabled by default through
MemoryManager
.
Preceding v1.22, the
kubelet
must be started with the following flag:
--feature-gates=MemoryManager=true
in order to enable the Memory Manager feature.
How Memory Manager Operates?
The Memory Manager currently offers the guaranteed memory (and hugepages) allocation
for Pods in Guaranteed QoS class.
To immediately put the Memory Manager into operation fo

llow the guidelines in the section
Memory Manager configuration
, and subsequently,
prepare and deploy a
pod as illustrated in the section
Placing a Pod in the Guaranteed QoS class
.
The Memory Manager is a Hint Provider, and it provides topology hints for
the Topology Manager which then aligns the requested resources according to these topology hints.
It also enforces
cgroups
(i.e.
cpuset.mems
) for pods.
The complete flow diagram concerning pod admission and deployment process is illustrated in
Memory Manager KEP: Design Overview
and below:
During this process, the Memory Manager updates its internal counters stored in
Node Map and Memory Maps
to manage guaranteed memory allocation.
The Memory Manager updates the Node Map during the startup and runtime as follows.
Startup
This occurs once a node administrator employs
--reserved-memory
(section
Reserved memory flag
).
In this case, the Node Map becomes updated to reflect this reservation as illustrated in
Memory Manager KEP: Memory Maps at start-up (with examples)
.
The administrator must provide
flag when
Static
policy is configured.
Runtime
Reference
Memory Manager KEP: Memory Maps at runtime (with examples)
illustrates
how a successful pod deployment affects the Node Map, and it also relates to
how potential Out-of-Memory (OOM) situations are handled further by Kubernetes or operating system.
Important topic in the context of Memory Manager operation is the management of NUMA groups.
Each time pod's memory request is in excess of single NUMA node capacity, the Memory Manager
attempts to create a group that comprises several NUMA nodes and features extend memory capacity.
The problem has been solved as elaborated in
Memory Manager KEP: How to enable the guaranteed memory allocation over many NUMA nodes?
.
Also, reference
Memory Manager KEP: Simulation - how the Memory Manager works? (by examples)
illustrates how the management of groups occurs.
Other Managers should be first pre-configured. Next, the Memory Manager feature should be enabled
and be run with
Static
policy (section
).
Optionally, some amount of memory can be reserved for system or kubelet processes to increase
node stability (section
).
Policies
Memory Manager supports two policies. You can select a policy via a
kubelet
flag
--memory-manager-policy
:
None
(default)
Static
This is the default policy and does not affect the memory allocation in any way.
It acts the same as if the Memory Manager is not present at all.
The
None
policy returns default topology hint. This special hint denotes that Hint Provider
(Memory Manger in this case) has no preference for NUMA affinity with any resource.
pod, the
Static
Memory Manger policy returns topology hints
relating to the set of NUMA nodes where the memory can be guaranteed,
and reserves the memory through updating the internal
NodeMap
object.
or
Burstable
pod, the
Static
Memory Manager policy sends back
the default topology hint as there is no request for the guaranteed memory,
and does not reserve the memory in the internal
NodeMap
object.
The
mechanism
is commonly used by node administrators to reserve K8S node system resources for the kubelet
or operating system processes in order to enhance the node stability.
A dedicated set of flags can be used for this purpose to set the total amount of reserved memory
for a node. This pre-configured value is subsequently utilized to calculate
the real amount of node's "allocatable" memory available to pods.
The Kubernetes scheduler incorporates "allocatable" to optimise pod scheduling process.
The foregoing flags include
,
and
--eviction-threshold
.
The sum of their values will account for the total amount of reserved memory.
A new
flag was added to Memory Manager

 to allow for this total reserved memory
to be split (by a node administrator) and accordingly reserved across many NUMA nodes.
The flag specifies a comma-separated list of memory reservations of different memory types per NUMA node.
Memory reservations across multiple NUMA nodes can be specified using semicolon as separator.
This parameter is only useful in the context of the Memory Manager feature.
The Memory Manager will not use this reserved memory for the allocation of container workloads.
For example, if you have a NUMA node "NUMA0" with
10Gi
of memory available, and
the
was specified to reserve
1Gi
of memory at "NUMA0",
the Memory Manager assumes that only
9Gi
is available for containers.
You can omit this parameter, however, you should be aware that the quantity of reserved memory
from all NUMA nodes should be equal to the quantity of memory specified by the
Node Allocatable feature
.
If at least one node allocatable parameter is non-zero, you will need to specify
for at least one NUMA node.
In fact,
eviction-hard
threshold value is equal to
100Mi
by default, so
if
Static
policy is used,
is obligatory.
Also, avoid the following configurations:
duplicates, i.e. the same NUMA node or memory type, but with a different value;
setting zero limit for any of memory types;
NUMA node IDs that do not exist in the machine hardware;
memory type names different than
memory
or
hugepages-&lt;size&gt;
(hugepages of particular
&lt;size&gt;
should also exist).
Syntax:
--reserved-memory N:memory-type1=value1,memory-type2=value2,...
N
(integer) - NUMA node index, e.g.
0
memory-type
(string) - represents memory type:
memory
- conventional memory
hugepages-2Mi
or
hugepages-1Gi
- hugepages
value
(string) - the quantity of reserved memory, e.g.
1Gi
Example usage:
--reserved-memory 0:memory=1Gi,hugepages-1Gi=2Gi
or
--reserved-memory 0:memory=1Gi --reserved-memory 1:memory=2Gi
or
--reserved-memory '0:memory=1Gi;1:memory=2Gi'
When you specify values for
flag, you must comply with the setting that
you prior provided via Node Allocatable Feature flags.
That is, the following rule must be obeyed for each memory type:
sum(reserved-memory(i)) = kube-reserved + system-reserved + eviction-threshold
,
where
i
is an index of a NUMA node.
If you do not follow the formula above, the Memory Manager will show an error on startup.
In other words, the example above illustrates that for the conventional memory (
type=memory
),
we reserve
3Gi
in total, i.e.:
sum(reserved-memory(i)) = reserved-memory(0) + reserved-memory(1) = 1Gi + 2Gi = 3Gi
An example of kubelet command-line arguments relevant to the node Allocatable configuration:
--kube-reserved=cpu=500m,memory=50Mi
--system-reserved=cpu=123m,memory=333Mi
--eviction-hard=memory.available&lt;500Mi
Note:
The default hard eviction threshold is 100MiB, and
not
zero.
Remember to increase the quantity of memory that you reserve by setting
by that hard eviction threshold. Otherwise, the kubelet will not start Memory Manager and
display an error.
Here is an example of a correct configuration:
=
=
true
=
cpu
=
4,memory
=
4Gi
=
cpu
=
1,memory
=
1Gi
=
Static
'0:memory=3Gi;1:memory=2148Mi'
Let us validate the configuration above:
kube-reserved + system-reserved + eviction-hard(default) = reserved-memory(0) + reserved-memory(1)
4GiB + 1GiB + 100MiB = 3GiB + 2148MiB
5120MiB + 100MiB = 3072MiB + 2148MiB
5220MiB = 5220MiB
(which is correct)
None
, the Memory Manager identifies pods
that are in the
QoS class.
The Memory Manager provides specific topology hints to the Topology Manager for each
pod.
For pods in a QoS class other than
, the Memory Manager provides default topology hints
to the Topology Manager.
The fol

lowing excerpts from pod manifests assign a pod to the
QoS class.
Pod with integer CPU(s) runs in the
QoS class, when
requests
limits
:
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
Also, a pod sharing CPU(s) runs in the
requests
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
Notice that both CPU and memory requests must be specified for a Pod to lend it to Guaranteed QoS class.
Troubleshooting
The following means can be used to troubleshoot the reason why a pod could not be deployed or
became rejected at a node:
pod status - indicates topology affinity errors
system logs - include valuable information for debugging, e.g., about generated hints
state file - the dump of internal state of the Memory Manager
(includes
)
starting from v1.22, the
device plugin resource API
can be used
to retrieve information about the memory reserved for containers
Pod status (TopologyAffinityError)
This error typically occurs in the following situations:
a node has not enough resources available to satisfy the pod's request
the pod's request is rejected due to particular Topology Manager policy constraints
The error appears in the status of a pod:
NAME         READY   STATUS                  RESTARTS   AGE
guaranteed   0/1     TopologyAffinityError   0          113s
Use
kubectl describe pod &lt;id&gt;
or
kubectl get events
to obtain detailed error message:
Warning  TopologyAffinityError  10m   kubelet, dell8  Resources cannot be allocated with Topology locality
System logs
Search system logs with respect to a particular pod.
The set of hints that Memory Manager generated for the pod can be found in the logs.
Also, the set of hints generated by CPU Manager should be present in the logs.
Topology Manager merges these hints to calculate a single best hint.
The best hint should be also present in the logs.
The best hint indicates where to allocate all the resources.
Topology Manager tests this hint against its current policy, and based on the verdict,
it either admits the pod to the node or rejects it.
Also, search the logs for occurrences associated with the Memory Manager,
e.g. to find out information about
cgroups
and
updates.
Examine the memory manager state on a node
Let us first deploy a sample
pod whose specification is as follows:
:
v1
kind
:
Pod
metadata
:
name
:
guaranteed
spec
:
:
-
name
:
image
:
consumer
:
Never
resources
:
limits
:
cpu
:
"2"
memory
:
150Gi
requests
:
cpu
:
"2"
memory
:
150Gi
command
:
[
"sleep"
,
]
Next, let us log into the node where it was deployed and examine the state file in
/var/lib/kubelet/memory_manager_state
:
{
"policyName"
:
"Static"
,
"machineState"
:{
"0"
:{
"numberOfAssignments"
:
1
,
"memoryMap"
:{
"hugepages-1Gi"
:{
"total"
:
0
,
"systemReserved"
:
0
,
"allocatable"
:
0
,
"reserved"
:
0
,
"free"
:
0
},
"memory"
:{
"total"
:
134987354112
,
:
3221225472
,
:
131766128640
,
:
,
"free"
:
0
}
},
"nodes"
:[
0
,
1
]
},
"1"
:{
:
1
,
:{
:{
"total"
:
0
,
:
0
,
:
0
,
:
0
,
"free"
:
0
},
"memory"
:{
"total"
:
135286722560
,
:
2252341248
,
:
133034381312
,
:
29295144960
,
"free"
:
103739236352
}
},
"nodes"
:[
0
,
1
]
}
},
"entries"
:{
"fa9bdd38-6df9-4cf9-aa67-8c4814da37a8"
:{
"guaranteed"
:[
{
"numaAffinity"
:[
0
,
1
],
"type"
:
"memory"
,
"size"
:
161061273600
}
]
}
},
"checksum"
:
4142013182
}
It can be deduced from the state file that the pod was p

inned to both NUMA nodes, i.e.:
:
[
0
,
1
]
,
Pinned term means that pod's memory consumption is constrained (through
cgroups
configuration)
to these NUMA nodes.
This automatically implies that Memory Manager instantiated a new group that
comprises these two NUMA nodes, i.e.
0
and
1
indexed NUMA nodes.
Notice that the management of groups is handled in a relatively complex manner, and
further elaboration is provided in Memory Manager KEP in
this
and
this
sections.
In order to analyse memory resources available in a group,the corresponding entries from
NUMA nodes belonging to the group must be added up.
For example, the total amount of free "conventional" memory in the group can be computed
by adding up the free memory available at every NUMA node in the group,
i.e., in the
"memory"
section of NUMA node
0
(
"free":0
) and NUMA node
1
(
"free":103739236352
).
So, the total amount of free "conventional" memory in this group is equal to
0 + 103739236352
bytes.
The line
"systemReserved":3221225472
indicates that the administrator of this node reserved
bytes (i.e.
3Gi
) to serve kubelet and system processes at NUMA node
0
,
by using
flag.
Device plugin resource API
The kubelet provides a
PodResourceLister
gRPC service to enable discovery of resources and associated metadata.
By using its
List gRPC endpoint
,
information about reserved memory for each container can be retrieved, which is contained
in protobuf
ContainerMemory
message.
This information can be retrieved solely for pods in Guaranteed QoS class.
Memory Manager KEP: The Concept of Node Map and Memory Maps
42 - Verify Signed Kubernetes Artifacts
Kubernetes v1.26 [beta]
These instructions are for Kubernetes 1.26. If you want
to check the integrity of components for a different version of Kubernetes,
check the documentation for that Kubernetes release.
You will need to have the following tools installed:
cosign
(
install guide
)
curl
(often provided by your operating system)
Verifying binary signatures
The Kubernetes release process signs all binary artifacts (tarballs, SPDX files,
standalone binaries) by using cosign's keyless signing. To verify a particular
binary, retrieve it together with its signature and certificate:
URL
=
https://dl.k8s.io/release/v1.26.0/bin/linux/amd64
BINARY
=
kubectl
FILES
=(
"
$BINARY
"
"
$BINARY
.sig"
"
$BINARY
.cert"
)
for
FILE in
"
${
FILES
[@]
}
"
;
do
curl -sSfL --retry
3
--retry-delay
3
"
$URL
/
$FILE
"
-o
"
$FILE
"
done
Then verify the blob by using
cosign
:
cosign verify-blob
"
$BINARY
"
--signature
"
$BINARY
"
.sig --certificate
"
$BINARY
"
.cert
Note:
To learn more about keyless signing, please refer to
Keyless
Signatures
.
Verifying image signatures
For a complete list of images that are signed please refer
to
Releases
.
Let's pick one image from this list and verify its signature using
the
cosign verify
command:
COSIGN_EXPERIMENTAL
=
1
cosign verify registry.k8s.io/kube-apiserver-amd64:v1.26.0
Note:
COSIGN_EXPERIMENTAL=1
is used to allow verification of images signed
in
KEYLESS
mode. To learn more about keyless signing, please refer to
Keyless Signatures
.
Verifying images for all control plane components
To verify all signed control plane images, please run this command:
curl -Ls https://sbom.k8s.io/
$(
curl -Ls https://dl.k8s.io/release/latest.txt
)
/release | grep
'PackageName: registry.k8s.io/'
| awk
'{print $2}'
&gt; images.txt
input
=
images.txt
while
IFS
=
read
-r image
do
=
1
"
$image
"
done
&lt;
"
$input
"
Once you have verified an image, specify that image by its digest in your Pod
manifests

 as per this
example:
registry-url/image-name@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2
.
For more information, please refer
to
Image Pull Policy
section.
Verifying Image Signatures with Admission Controller
For non-control plane images (
e.g.
conformance image
)
, signatures can also be verified at deploy time using
sigstore policy-controller
admission controller. To get started with
policy-controller
here are a few helpful
Installation
Configuration Options
Home
Blog
Training
Partners
Community
CC BY 4.0

Debugging Kubernetes nodes with crictl | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install To

ols
Debug Pods
Debug Init

 Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Securit

y Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configura

tion (v1alpha1)
Tasks
Kubernetes v1.11 [stable]
crictl
is a command-line interface for CRI-compatible container runtimes.
You can use it to inspect and debug container runtimes and applications on a
Kubernetes node.
crictl
and its source are hosted in the
cri-tools
repository.
crictl
requires a Linux operating system with a CRI runtime.
Installing crictl
You can download a compressed archive
crictl
from the cri-tools
release page
, for several
different architectures. Download the version that corresponds to your version
of Kubernetes. Extract it and move it to a location on your system path, such as
/usr/local/bin/
.
General usage
The
crictl
command has several subcommands and runtime flags. Use
crictl help
or
crictl &lt;subcommand&gt; help
You can set the endpoint for
crictl
by doing one of the following:
Set the
--runtime-endpoint
and
--image-endpoint
flags.
Set the
CONTAINER_RUNTIME_ENDPOINT
and
IMAGE_SERVICE_ENDPOINT
environment
variables.
Set the endpoint in the configuration file
/etc/crictl.yaml
. To specify a
different file, use the
--config=PATH_TO_FILE
flag when you run
crictl
.
Note:
If you don't set an endpoint,
crictl
attempts to connect to a list of known
endpoints, which might result in an impact to performance.
You can also specify timeout values when connecting to the server and enable or
disable debugging, by specifying
timeout
or
debug
values in the configuration
file or using the
--timeout
and
--debug
command-line flags.
To view or edit the current configuration, view or edit the contents of
. For example, the configuration when using the
container runtime would be similar to this:
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: true
To learn more about
crictl
, refer to the
crictl
.
Example crictl commands
The following examples show some
crictl
commands and example output.
Warning:
If you use
crictl
to create pod sandboxes or containers on a running
Kubernetes cluster, the Kubelet will eventually delete them.
crictl
is not a
general purpose workflow tool, but a tool that is useful for debugging.
List pods
List all pods:
crictl pods
POD ID              CREATED              STATE               NAME                         NAMESPACE           ATTEMPT
926f1b5a1d

33a       About a minute ago   Ready               sh-84d7dcf559-4r2gq          default             0
4dccb216c4adb       About a minute ago   Ready               nginx-65899c769f-wv2gp       default             0
a86316e96fa89       17 hours ago         Ready               kube-proxy-gblk4             kube-system         0
919630b8f81f1       17 hours ago         Ready               nvidia-device-plugin-zgbbv   kube-system         0
List pods by name:
crictl pods --name nginx-65899c769f-wv2gp
POD ID              CREATED             STATE               NAME                     NAMESPACE           ATTEMPT
4dccb216c4adb       2 minutes ago       Ready               nginx-65899c769f-wv2gp   default             0
List pods by label:
crictl pods --label
run
=
nginx
List images
List all images:
crictl images
IMAGE                                     TAG                 IMAGE ID            SIZE
busybox                                   latest              8c811b4aec35f       1.15MB
k8s-gcrio.azureedge.net/hyperkube-amd64   v1.10.3             e179bbfe5d238       665MB
k8s-gcrio.azureedge.net/pause-amd64       3.1                 da86e6ba6ca19       742kB
nginx                                     latest              cd5239a0906a6       109MB
List images by repository:
crictl images nginx
IMAGE               TAG                 IMAGE ID            SIZE
nginx               latest              cd5239a0906a6       109MB
Only list image IDs:
crictl images -q
sha256:8c811b4aec35f259572d0f79207bc0678df4c736eeec50bc9fec37ed936a472a
sha256:e179bbfe5d238de6069f3b03fccbecc3fb4f2019af741bfff1233c4d7b2970c5
sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e
sha256:cd5239a0906a6ccf0562354852fae04bc5b52d72a2aff9a871ddb6bd57553569
List containers
List all containers:
crictl ps -a
CONTAINER ID        IMAGE                                                                                                             CREATED             STATE               NAME                       ATTEMPT
1f73f2d81bf98       busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47                                   7 minutes ago       Running             sh                         1
9c5951df22c78       busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47                                   8 minutes ago       Exited              sh                         0
87d3992f84f74       nginx@sha256:d0a8828cccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f                                     8 minutes ago       Running             nginx                      0
1941fb4da154f       k8s-gcrio.azureedge.net/hyperkube-amd64@sha256:00d814b1f7763f4ab5be80c58e98140dfc69df107f253d7fdd714b30a714260a   18 hours ago        Running             kube-proxy                 0
List running containers:
crictl ps
1f73f2d81bf98       busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47                                   6 minutes ago       Running             sh                         1
87d3992f84f74       nginx@sha256:d0a8828cccb73397acb0073bf34f4d7d8aa315263f1e7806bf8c55d8ac139d5f                                     7 minutes ago       Running             nginx                      0
1941fb4da154f       k8s-gcrio.azureedge.net/hyperkube-amd64@sha256:00d814b1f7763f4ab

5be80c58e98140dfc69df107f253d7fdd714b30a714260a   17 hours ago        Running             kube-proxy                 0
Execute a command in a running container
crictl
exec
-i -t 1f73f2d81bf98 ls
bin   dev   etc   home  proc  root  sys   tmp   usr   var
Get a container's logs
Get all container logs:
crictl logs 87d3992f84f74
10.240.0.96 - - [06/Jun/2018:02:45:49 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
10.240.0.96 - - [06/Jun/2018:02:45:50 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
10.240.0.96 - - [06/Jun/2018:02:45:51 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.47.0" "-"
Get only the latest
N
lines of logs:
crictl logs --tail
=
1
87d3992f84f74
Run a pod sandbox
Using
crictl
to run a pod sandbox is useful for debugging container runtimes.
On a running Kubernetes cluster, the sandbox will eventually be stopped and
deleted by the Kubelet.
Create a JSON file like the following:
{
"metadata"
: {
"name"
:
"nginx-sandbox"
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
"hdishd83djaidwnduwk28bcsb"
},
"log_directory"
:
"/tmp"
,
"linux"
: {
}
}
Use the
crictl runp
command to apply the JSON and run the sandbox.
crictl runp pod-config.json
The ID of the sandbox is returned.
Create a container
Using
crictl
to create a container is useful for debugging container runtimes.
Pull a busybox image
crictl pull busybox
Image is up to date for busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
Create configs for the pod and the container:
Pod config
:
{
: {
"name"
:
"busybox-sandbox"
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
"aewi4aeThua7ooShohbo1phoj"
},
:
"/tmp"
,
"linux"
: {
}
}
Container config
:
{
: {
"name"
:
"busybox"
},
"image"
:{
"image"
:
"busybox"
},
"command"
: [
"top"
],
"log_path"
:
"busybox.log"
,
"linux"
: {
}
}
Create the container, passing the ID of the previously-created pod, the
container config file, and the pod config file. The ID of the container is
returned.
crictl create f84dd361f8dc51518ed291fbadd6db537b0496536c1d2d6c05ff943ce8c9a54f container-config.json pod-config.json
List all containers and verify that the newly-created container has its
state set to
Created
.
CONTAINER ID        IMAGE               CREATED             STATE               NAME                ATTEMPT
3e025dd50a72d       busybox             32 seconds ago      Created             busybox             0
Start a container
To start a container, pass its ID to
crictl start
:
crictl start 3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60
3e025dd50a72d956c4f14881fbb5b1080c9275674e95fb67f965f6478a957d60
Check the container has its state set to
Running
.
crictl ps
CONTAINER ID   IMAGE    CREATED              STATE    NAME     ATTEMPT
3e025dd50a72d  busybox  About a minute ago   Running  busybox  0
crictl
.
Map
docker
CLI commands to
crictl
.
Feedback
Yes
No
.
or
.
Last modified August 11, 2022 at 12:42 PM PST:
Fix Pod spec in debugging crictl (a6d6a5bb04)
List pods
Home
Blog
Training
Partners
Community
Case 

Studies
CC BY 4.0

Configure GMSA for Windows Pods and containers | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

ools
Debug Pods
Debug Ini

t Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Securi

ty Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configur

ation (v1alpha1)
Tasks
This page shows how to configure
Group Managed Service Accounts
(GMSA) for Pods and containers that will run on Windows nodes. Group Managed Service Accounts are a specific type of Active Directory account that provides automatic password management, simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.
In Kubernetes, GMSA credential specs are configured at a Kubernetes cluster-wide scope as Custom Resources. Windows Pods, as well as individual containers within a Pod, can be configured to use a GMSA for domain based functions (e.g. Kerberos authentication) when interacting with other Windows services.
You need to have a Kubernetes cluster and the
kubectl
command-line tool must be configured to communicate with your cluster. The cluster is expected to have Windows worker nodes. This section covers a set of initial steps required once for each cluster:
Install the GMSACredentialSpec CRD
A
(CRD) for GMSA credential spec resources needs to be configured on the cluster to define the custom resource type
GMSACredentialSpec
. Download the GMSA CRD
YAML
and save it as gmsa-crd.yaml.
Next, install the CRD with
kubectl apply -f gmsa-crd.yaml
Install webhooks to validate GMSA users
Two webhooks need to be configured on the Kubernetes cluster to populate and validate GMSA credential spec references at the Pod or container level:
A mutating webhook that expands references to GMSAs (by name from a Pod specification) into the full credential spec in JSON form within the Pod spec.
A validating webhook ensures all references to GMSAs are authorized to be used by the Pod service account.
Installing the above webhooks and associated objects require the steps below:
Create a certificate key pair (that will be used to allow the webhook container to communicate to the cluster)
Install a secret with the certificate from above.
Create a deployment for the core webhook logic.
Create the validating and mutating webhook configurations referring to the deployment.
A
script
can be used to deploy and configure the GMSA webhooks and associated objects mentioned above. The script can be run with a
option to allow you to review the changes that would be made to your cluster.
The
YAML template
used by the script may also be use

d to deploy the webhooks and associated objects manually (with appropriate substitutions for the parameters)
Configure GMSAs and Windows nodes in Active Directory
Before Pods in Kubernetes can be configured to use GMSAs, the desired GMSAs need to be provisioned in Active Directory as described in the
Windows GMSA documentation
. Windows worker nodes (that are part of the Kubernetes cluster) need to be configured in Active Directory to access the secret credentials associated with the desired GMSA as described in the
Create GMSA credential spec resources
With the GMSACredentialSpec CRD installed (as described earlier), custom resources containing GMSA credential specs can be configured. The GMSA credential spec does not contain secret or sensitive data. It is information that a container runtime can use to describe the desired GMSA of a container to Windows. GMSA credential specs can be generated in YAML format with a utility
PowerShell script
.
Following are the steps for generating a GMSA credential spec YAML manually in JSON format and then converting it:
Import the CredentialSpec
module
:
ipmo CredentialSpec.psm1
Create a credential spec in JSON format using
New-CredentialSpec
. To create a GMSA credential spec named WebApp1, invoke
New-CredentialSpec -Name WebApp1 -AccountName WebApp1 -Domain $(Get-ADDomain -Current LocalComputer)
Use
Get-CredentialSpec
to show the path of the JSON file.
Convert the credspec file from JSON to YAML format and apply the necessary header fields
,
kind
,
metadata
and
credspec
to make it a GMSACredentialSpec custom resource that can be configured in Kubernetes.
The following YAML configuration describes a GMSA credential spec named
gmsa-WebApp1
:
:
windows.k8s.io/v1
kind
:
metadata
:
name
:
#This is an arbitrary name but it will be used as a reference
credspec
:
ActiveDirectoryConfig
:
GroupManagedServiceAccounts
:
-
Name
:
WebApp1
#Username of the GMSA account
Scope
:
CONTOSO
#NETBIOS Domain Name
-
Name
:
WebApp1
Scope
:
contoso.com
#DNS Domain Name
CmsPlugins
:
- ActiveDirectory
DomainJoinConfig
:
DnsName
:
DnsTreeName
:
#DNS Domain Name Root
Guid
:
244818ae-87ac-4fcd-92ec-e79e5252348a
#GUID
MachineAccountName
:
WebApp1
NetBiosName
:
CONTOSO
Sid
:
S-1-5-21-2126449477-2524075714-3094792973
#SID of GMSA
The above credential spec resource may be saved as
gmsa-Webapp1-credspec.yaml
and applied to the cluster using:
kubectl apply -f gmsa-Webapp1-credspec.yml
Configure cluster role to enable RBAC on specific GMSA credential specs
A cluster role needs to be defined for each GMSA credential spec resource. This authorizes the
use
verb on a specific GMSA resource by a subject which is typically a service account. The following example shows a cluster role that authorizes usage of the
credential spec from above. Save the file as gmsa-webapp1-role.yaml and apply using
kubectl apply -f gmsa-webapp1-role.yaml
#Create the Role to read the credspec
:
kind
:
metadata
:
name
:
webapp1-role
rules
:
-
apiGroups
:
[
"windows.k8s.io"
]
resources
:
[
"gmsacredentialspecs"
]
verbs
:
[
"use"
]
resourceNames
:
[
"gmsa-WebApp1"
]
Assign role to service accounts to use specific GMSA credspecs
A service account (that Pods will be configured with) needs to be bound to the cluster role create above. This authorizes the service account to use the desired GMSA credential spec resource. The following shows the default service account being bound to a cluster role
to use
credential spec resource created above.
:
kind
:
metadata
:
name
:
allow-default-svc-account-read-on-gmsa-WebApp1
namespace
:
default
subjects
:
-
kind
:
name
:
default
namespace
:
default
roleRef
:
kind
:

name
:
apiGroup
:
Configure GMSA credential spec reference in Pod spec
The Pod spec field
securityContext.windowsOptions.gmsaCredentialSpecName
is used to specify references to desired GMSA credential spec custom resources in Pod specs. This configures all containers in the Pod spec to use the specified GMSA. A sample Pod spec with the annotation populated to refer to
:
:
apps/v1
kind
:
metadata
:
labels
:
run
:
with-creds
name
:
namespace
:
default
spec
:
replicas
:
1
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
windowsOptions
:
gmsaCredentialSpecName
:
gmsa-webapp1
:
-
image
:
mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019
:
Always
name
:
iis
:
:
windows
Individual containers in a Pod spec can also specify the desired GMSA credspec using a per-container
field. For example:
:
apps/v1
kind
:
metadata
:
labels
:
run
:
name
:
namespace
:
default
spec
:
replicas
:
1
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
-
image
:
:
Always
name
:
iis
:
:
:
gmsa-Webapp1
:
:
windows
As Pod specs with GMSA fields populated (as described above) are applied in a cluster, the following sequence of events take place:
The mutating webhook resolves and expands all references to GMSA credential spec resources to the contents of the GMSA credential spec.
The validating webhook ensures the service account associated with the Pod is authorized for the
use
verb on the specified GMSA credential spec.
The container runtime configures each Windows container with the specified GMSA credential spec so that the container can assume the identity of the GMSA in Active Directory and access services in the domain using that identity.
Authenticating to network shares using hostname or FQDN
If you are experiencing issues connecting to SMB shares from Pods using hostname or FQDN, but are able to access the shares via their IPv4 address then make sure the following registry key is set on the Windows nodes.
reg add
"HKLM\SYSTEM\CurrentControlSet\Services\hns\State"
/v EnableCompartmentNamespace /t REG_DWORD /d 1
Running Pods will then need to be recreated to pick up the behavior changes.
More information on how this registry key is used can be found
here
If you are having difficulties getting GMSA to work in your environment, there are a few troubleshooting steps you can take.
First, make sure the credspec has been passed to the Pod. To do this you will need to
exec
into one of your Pods and check the output of the
nltest.exe /parentdomain
command.
In the example below the Pod did not get the credspec correctly:
kubectl exec -it
iis-auth
-7776966999-n5nzr powershell.exe
results in the following error:
Getting parent domain failed: Status = 1722 0x6ba RPC_S_SERVER_UNAVAILABLE
If your Pod did get the credspec correctly, then next check communication with the domain. First, from inside of your Pod, quickly do an nslookup to find the root of your domain.
This will tell us 3 things:
The Pod can reach the DC
The DC can reach the Pod
DNS is working correctly.
If the DNS and communication test passes, next you will need to check if the Pod has established secure channel communication with the domain. To do this, again,
exec
into your Pod and run the
nltest.exe /query
command.
Results in the following output:
I_NetLogonControl failed: Status = 1722 0x6ba RPC_S_SERVER_UNAVAILABLE
This tells us that for some reason, the Pod was unable to logon to the domai

n using the account specified in the credspec. You can try to repair the secure channel by running the following:
nltest /sc_reset
:
domain.example
If the command is successful you will see and output similar to this:
Flags: 30 HAS_IP  HAS_TIMESERV
Trusted DC Name \\dc10.domain.example
Trusted DC Connection Status Status = 0 0x0 NERR_Success
The command completed successfully
If the above corrects the error, you can automate the step by adding the following lifecycle hook to your Pod spec. If it did not correct the error, you will need to examine your credspec again and confirm that it is correct and complete.
image
:
registry.domain.example/iis-auth:1809v1
lifecycle
:
postStart
:
exec
:
command
:
[
"powershell.exe"
,
"-command"
,
"do { Restart-Service -Name netlogon } while ( $($Result = (nltest.exe /query); if ($Result -like '*0x0 NERR_Success*') {return $true} else {return $false}) -eq $false)"
]
:
If you add the
lifecycle
section show above to your Pod spec, the Pod will execute the commands listed to restart the
netlogon
service until the
command exits without error.
Feedback
Yes
No
.
or
.
Last modified February 23, 2022 at 6:23 PM PST:
Solved some typos (038256a86b)
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-apiserver | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Debug Pods
Troublesho

oting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resource

s with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Conf

iguration (v1alpha1)
Reference
Synopsis
The Kubernetes API server validates and configures data
for the api objects which include pods, services, replicationcontrollers, and
others. The API Server services REST operations and provides the frontend to the
cluster's shared state through which all other components interact.
kube-apiserver [flags]
Options
--admission-control-config-file string
File with admission control configuration.
--advertise-address string
The IP address on which to advertise the apiserver to members of the cluster. This address must be reachable by the rest of the cluster. If blank, the --bind-address will be used. If --bind-address is unspecified, the host's default interface will be used.
--aggregator-reject-forwarding-redirect     Default: true
Aggregator reject forwarding redirect response back to client.
--allow-metric-labels stringToString     Default: []
The map from metric-label to value allow-list of this label. The key's format is &lt;MetricName&gt;,&lt;LabelName&gt;. The value's format is &lt;allowed_value&gt;,&lt;allowed_value&gt;...e.g. metric1,label1='v1,v2,v3', metric1,label2='v1,v2,v3' metric2,label1='v1,v2,v3'.
--allow-privileged
If true, allow privileged containers. [default=false]
--anonymous-auth     Default: true
Enables anonymous requests to the secure port of the API server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of system:anonymous, and a group name of system:unauthenticated.
--api-audiences strings
Identifiers of the API. The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences. If the --service-account-issuer flag is configured and this flag is not, this field defaults to a single element list containing the issuer URL.
--audit-log-batch-buffer-size int     Default: 10000
The size of the buffer to store events before batching and writing. Only used in batch mode.
--audit-log-batch-max-size int     Default: 1
The maximum size of a batch. Only used in batch mode.
--audit-log-batch-max-wait duration
The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.
--audit-log-batch-throttle-burst int
Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.
--audit-log-batch-throttle-enable
Whether batching throttling is enabled. Only used in batch mode.
--audit-log-batch-throttle-qps float
Maximum average number of batches per second. Only use

d in batch mode.
--audit-log-compress
If set, the rotated log files will be compressed using gzip.
--audit-log-format string     Default: "json"
Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Known formats are legacy,json.
--audit-log-maxage int
The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
--audit-log-maxbackup int
The maximum number of old audit log files to retain. Setting a value of 0 will mean there's no restriction on the number of files.
--audit-log-maxsize int
The maximum size in megabytes of the audit log file before it gets rotated.
--audit-log-mode string     Default: "blocking"
Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict.
--audit-log-path string
If set, all requests coming to the apiserver will be logged to this file. '-' means standard out.
--audit-log-truncate-enabled
Whether event and batch truncating is enabled.
--audit-log-truncate-max-batch-size int     Default: 10485760
Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size.
--audit-log-truncate-max-event-size int     Default: 102400
Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded.
--audit-log-version string     Default: "audit.k8s.io/v1"
API group and version used for serializing audit events written to log.
--audit-policy-file string
Path to the file that defines the audit policy configuration.
--audit-webhook-batch-buffer-size int     Default: 10000
--audit-webhook-batch-max-size int     Default: 400
--audit-webhook-batch-max-wait duration     Default: 30s
--audit-webhook-batch-throttle-burst int     Default: 15
--audit-webhook-batch-throttle-enable     Default: true
--audit-webhook-batch-throttle-qps float     Default: 10
Maximum average number of batches per second. Only used in batch mode.
--audit-webhook-config-file string
Path to a kubeconfig formatted file that defines the audit webhook configuration.
--audit-webhook-initial-backoff duration     Default: 10s
The amount of time to wait before retrying the first failed request.
--audit-webhook-mode string     Default: "batch"
--audit-webhook-truncate-enabled
--audit-webhook-truncate-max-batch-size int     Default: 10485760
--audit-webhook-truncate-max-event-size int     Default: 102400
--audit-webhook-version string     Default: "audit.k8s.io/v1"
API group and version used for s

erializing audit events written to webhook.
--authentication-token-webhook-cache-ttl duration     Default: 2m0s
The duration to cache responses from the webhook token authenticator.
--authentication-token-webhook-config-file string
File with webhook configuration for token authentication in kubeconfig format. The API server will query the remote service to determine authentication for bearer tokens.
--authentication-token-webhook-version string     Default: "v1beta1"
The API version of the authentication.k8s.io TokenReview to send to and expect from the webhook.
--authorization-mode strings     Default: "AlwaysAllow"
Ordered list of plug-ins to do authorization on secure port. Comma-delimited list of: AlwaysAllow,AlwaysDeny,ABAC,Webhook,RBAC,Node.
--authorization-policy-file string
File with authorization policy in json line by line format, used with --authorization-mode=ABAC, on the secure port.
--authorization-webhook-cache-authorized-ttl duration     Default: 5m0s
The duration to cache 'authorized' responses from the webhook authorizer.
--authorization-webhook-cache-unauthorized-ttl duration     Default: 30s
The duration to cache 'unauthorized' responses from the webhook authorizer.
--authorization-webhook-config-file string
File with webhook configuration in kubeconfig format, used with --authorization-mode=Webhook. The API server will query the remote service to determine access on the API server's secure port.
--authorization-webhook-version string     Default: "v1beta1"
The API version of the authorization.k8s.io SubjectAccessReview to send to and expect from the webhook.
--azure-container-registry-config string
Path to the file containing Azure container registry configuration information.
--bind-address string     Default: 0.0.0.0
The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used.
--cert-dir string     Default: "/var/run/kubernetes"
The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored.
--client-ca-file string
If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
--cloud-config string
The path to the cloud provider configuration file. Empty string for no configuration file.
--cloud-provider string
The provider for cloud services. Empty string for no provider.
--cloud-provider-gce-l7lb-src-cidrs cidrs     Default: 130.211.0.0/22,35.191.0.0/16
CIDRs opened in GCE firewall for L7 LB traffic proxy &amp; health checks
--contention-profiling
Enable lock contention profiling, if profiling is enabled
--cors-allowed-origins strings
List of allowed origins for CORS, comma separated. An allowed origin can be a regular expression to support subdomain matching. If this list is empty CORS will not be enabled.
--default-not-ready-toleration-seconds int     Default: 300
Indicates the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration.
--default-unreachable-toleration-seconds int     Default: 300
Indicates the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration.
--delete-collection-workers int     Default: 1
Number of workers spawned for DeleteCollection call. These are used to speed up namespace cleanup.
--disable-admission-plugins strings
admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, 

CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--disabled-metrics strings
This flag provides an escape hatch for misbehaving metrics. You must provide the fully qualified metric name in order to disable it. Disclaimer: disabling metrics is higher in precedence than showing hidden metrics.
--egress-selector-config-file string
File with apiserver egress selector configuration.
--enable-admission-plugins strings
admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--enable-aggregator-routing
Turns on aggregator routing requests to endpoints IP rather than cluster IP.
--enable-bootstrap-token-auth
Enable to allow secrets of type 'bootstrap.kubernetes.io/token' in the 'kube-system' namespace to be used for TLS bootstrapping authentication.
--enable-garbage-collector     Default: true
Enables the generic garbage collector. MUST be synced with the corresponding flag of the kube-controller-manager.
--enable-priority-and-fairness     Default: true
If true and the APIPriorityAndFairness feature gate is enabled, replace the max-in-flight handler with an enhanced one that queues and dispatches with priority and fairness
--encryption-provider-config string
The file containing configuration for encryption providers to be used for storing secrets in etcd
Determines if the file set by --encryption-provider-config should be automatically reloaded if the disk contents change. Setting this to true disables the ability to uniquely identify distinct KMS plugins via the API server healthz endpoints.
--endpoint-reconciler-type string     Default: "lease"
Use an endpoint reconciler (master-count, lease, none) master-count is deprecated, and will be removed in a future version.
--etcd-

cafile string
SSL Certificate Authority file used to secure etcd communication.
--etcd-certfile string
SSL certification file used to secure etcd communication.
--etcd-compaction-interval duration     Default: 5m0s
The interval of compaction requests. If 0, the compaction request from apiserver is disabled.
--etcd-count-metric-poll-period duration     Default: 1m0s
Frequency of polling etcd for number of resources per type. 0 disables the metric collection.
--etcd-db-metric-poll-interval duration     Default: 30s
The interval of requests to poll etcd and update metric. 0 disables the metric collection
--etcd-healthcheck-timeout duration     Default: 2s
The timeout to use when checking etcd health.
--etcd-keyfile string
SSL key file used to secure etcd communication.
--etcd-prefix string     Default: "/registry"
The prefix to prepend to all resource paths in etcd.
--etcd-readycheck-timeout duration     Default: 2s
The timeout to use when checking etcd readiness
--etcd-servers strings
List of etcd servers to connect with (scheme://ip:port), comma separated.
--etcd-servers-overrides strings
Per-resource etcd servers overrides, comma separated. The individual override format: group/resource#servers, where servers are URLs, semicolon separated. Note that this applies only to resources compiled into this server binary.
--event-ttl duration     Default: 1h0m0s
Amount of time to retain events.
--external-hostname string
The hostname to use when generating externalized URLs for this master (e.g. Swagger API Docs or OpenID Discovery).
--feature-gates &lt;comma-separated 'key=True|False' pairs&gt;
A set of key=value pairs that describe feature gates for alpha/experimental features. Options are:
APIListChunking=true|false (BETA - default=true)
APIPriorityAndFairness=true|false (BETA - default=true)
APIResponseCompression=true|false (BETA - default=true)
APISelfSubjectReview=true|false (ALPHA - default=false)
APIServerIdentity=true|false (BETA - default=true)
APIServerTracing=true|false (ALPHA - default=false)
AggregatedDiscoveryEndpoint=true|false (ALPHA - default=false)
AnyVolumeDataSource=true|false (BETA - default=true)
AppArmor=true|false (BETA - default=true)
CPUManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
CPUManagerPolicyBetaOptions=true|false (BETA - default=true)
CPUManagerPolicyOptions=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (BETA - default=false)
CSIMigrationRBD=true|false (ALPHA - default=false)
CSINodeExpandSecret=true|false (ALPHA - default=false)
CSIVolumeHealth=true|false (ALPHA - default=false)
ComponentSLIs=true|false (ALPHA - default=false)
ContainerCheckpoint=true|false (ALPHA - default=false)
CronJobTimeZone=true|false (BETA - default=true)
CrossNamespaceVolumeDataSource=true|false (ALPHA - default=false)
CustomCPUCFSQuotaPeriod=true|false (ALPHA - default=false)
CustomResourceValidationExpressions=true|false (BETA - default=true)
DisableCloudProviders=true|false (ALPHA - default=false)
DisableKubeletCloudCredentialProviders=true|false (ALPHA - default=false)
DownwardAPIHugePages=true|false (BETA - default=true)
DynamicResourceAllocation=true|false (ALPHA - default=false)
EventedPLEG=true|false (ALPHA - default=false)
ExpandedDNSConfig=true|false (BETA - default=true)
ExperimentalHostUserNamespaceDefaulting=true|false (BETA - default=false)
GRPCContainerProbe=true|false (BETA - default=true)
GracefulNodeShutdown=true|false (BETA - default=true)
GracefulNodeShutdownBasedOnPodPriority=true|false (BETA - default=true)
HPAContainerMetrics=true|false (ALPHA - default=false)
HPAScaleToZero=true|false (ALPHA - default=false)
HonorPVReclaimPolicy=true|false (ALPHA - default=false)
IPTablesOwnershipCleanup=true|false (ALPHA - default=false)
InTreePluginAWSUnregister=true|false (ALPHA - default=false)
InTreePluginAzureDiskUnregister=true|false (ALPHA -

 default=false)
InTreePluginAzureFileUnregister=true|false (ALPHA - default=false)
InTreePluginGCEUnregister=true|false (ALPHA - default=false)
InTreePluginOpenStackUnregister=true|false (ALPHA - default=false)
InTreePluginPortworxUnregister=true|false (ALPHA - default=false)
InTreePluginRBDUnregister=true|false (ALPHA - default=false)
InTreePluginvSphereUnregister=true|false (ALPHA - default=false)
JobMutableNodeSchedulingDirectives=true|false (BETA - default=true)
JobPodFailurePolicy=true|false (BETA - default=true)
JobReadyPods=true|false (BETA - default=true)
KMSv2=true|false (ALPHA - default=false)
KubeletInUserNamespace=true|false (ALPHA - default=false)
KubeletPodResources=true|false (BETA - default=true)
KubeletPodResourcesGetAllocatable=true|false (BETA - default=true)
KubeletTracing=true|false (ALPHA - default=false)
LegacyServiceAccountTokenTracking=true|false (ALPHA - default=false)
LocalStorageCapacityIsolationFSQuotaMonitoring=true|false (ALPHA - default=false)
LogarithmicScaleDown=true|false (BETA - default=true)
LoggingAlphaOptions=true|false (ALPHA - default=false)
LoggingBetaOptions=true|false (BETA - default=true)
MatchLabelKeysInPodTopologySpread=true|false (ALPHA - default=false)
MaxUnavailableStatefulSet=true|false (ALPHA - default=false)
MemoryManager=true|false (BETA - default=true)
MemoryQoS=true|false (ALPHA - default=false)
MinDomainsInPodTopologySpread=true|false (BETA - default=false)
MinimizeIPTablesRestore=true|false (ALPHA - default=false)
MultiCIDRRangeAllocator=true|false (ALPHA - default=false)
NetworkPolicyStatus=true|false (ALPHA - default=false)
NodeInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
NodeOutOfServiceVolumeDetach=true|false (BETA - default=true)
NodeSwap=true|false (ALPHA - default=false)
OpenAPIEnums=true|false (BETA - default=true)
OpenAPIV3=true|false (BETA - default=true)
PDBUnhealthyPodEvictionPolicy=true|false (ALPHA - default=false)
PodAndContainerStatsFromCRI=true|false (ALPHA - default=false)
PodDeletionCost=true|false (BETA - default=true)
PodDisruptionConditions=true|false (BETA - default=true)
PodHasNetworkCondition=true|false (ALPHA - default=false)
PodSchedulingReadiness=true|false (ALPHA - default=false)
ProbeTerminationGracePeriod=true|false (BETA - default=true)
ProcMountType=true|false (ALPHA - default=false)
ProxyTerminatingEndpoints=true|false (BETA - default=true)
QOSReserved=true|false (ALPHA - default=false)
ReadWriteOncePod=true|false (ALPHA - default=false)
RecoverVolumeExpansionFailure=true|false (ALPHA - default=false)
RemainingItemCount=true|false (BETA - default=true)
RetroactiveDefaultStorageClass=true|false (BETA - default=true)
RotateKubeletServerCertificate=true|false (BETA - default=true)
SELinuxMountReadWriteOncePod=true|false (ALPHA - default=false)
SeccompDefault=true|false (BETA - default=true)
ServerSideFieldValidation=true|false (BETA - default=true)
SizeMemoryBackedVolumes=true|false (BETA - default=true)
StatefulSetAutoDeletePVC=true|false (ALPHA - default=false)
StatefulSetStartOrdinal=true|false (ALPHA - default=false)
StorageVersionAPI=true|false (ALPHA - default=false)
StorageVersionHash=true|false (BETA - default=true)
TopologyAwareHints=true|false (BETA - default=true)
TopologyManager=true|false (BETA - default=true)
TopologyManagerPolicyAlphaOptions=true|false (ALPHA - default=false)
TopologyManagerPolicyBetaOptions=true|false (BETA - default=false)
TopologyManagerPolicyOptions=true|false (ALPHA - default=false)
UserNamespacesStatelessPodsSupport=true|false (ALPHA - default=false)
ValidatingAdmissionPolicy=true|false (ALPHA - default=false)
VolumeCapacityPriority=true|false (ALPHA - default=false)
WinDSR=true|false (ALPHA - default=false)
WinOverlay=true|false (BETA - default=true)
WindowsHostNetwork=true|false (ALPHA - default=true)
--goaway-chance float
To prevent HTTP/2 clients from getting stuck on a single apiserver, randomly close a connection (GOAWAY). The client's other in-flight requests won't be affe

cted, and the client will reconnect, likely landing on a different apiserver after going through the load balancer again. This argument sets the fraction of requests that will be sent a GOAWAY. Clusters with single apiservers, or which don't use a load balancer, should NOT enable this. Min is 0 (off), Max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.
-h, --help
help for kube-apiserver
--http2-max-streams-per-connection int
The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.
--kubelet-certificate-authority string
Path to a cert file for the certificate authority.
--kubelet-client-certificate string
Path to a client cert file for TLS.
--kubelet-client-key string
Path to a client key file for TLS.
--kubelet-preferred-address-types strings     Default: "Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP"
List of the preferred NodeAddressTypes to use for kubelet connections.
--kubelet-timeout duration     Default: 5s
Timeout for kubelet operations.
--kubernetes-service-node-port int
If non-zero, the Kubernetes master service (which apiserver creates/maintains) will be of type NodePort, using this as the value of the port. If zero, the Kubernetes master service will be of type ClusterIP.
--lease-reuse-duration-seconds int     Default: 60
The time in seconds that each lease is reused. A lower value could avoid large number of objects reusing the same lease. Notice that a too small value may cause performance problems at storage layer.
--livez-grace-period duration
This option represents the maximum amount of time it should take for apiserver to complete its startup sequence and become live. From apiserver's start time to when this amount of time has elapsed, /livez will assume that unfinished post-start hooks will complete successfully and therefore return true.
--log-flush-frequency duration     Default: 5s
Maximum number of seconds between log flushes
--logging-format string     Default: "text"
Sets the log format. Permitted formats: "text".
--max-connection-bytes-per-sec int
If non-zero, throttle each user connection to this number of bytes/sec. Currently only applies to long-running requests.
--max-mutating-requests-inflight int     Default: 200
This and --max-requests-inflight are summed to determine the server's total concurrency limit (which must be positive) if --enable-priority-and-fairness is true. Otherwise, this flag limits the maximum number of mutating requests in flight, or a zero value disables the limit completely.
--max-requests-inflight int     Default: 400
This and --max-mutating-requests-inflight are summed to determine the server's total concurrency limit (which must be positive) if --enable-priority-and-fairness is true. Otherwise, this flag limits the maximum number of non-mutating requests in flight, or a zero value disables the limit completely.
--min-request-timeout int     Default: 1800
An optional field indicating the minimum number of seconds a handler must keep a request open before timing it out. Currently only honored by the watch request handler, which picks a randomized value above this number as the connection timeout, to spread out load.
--oidc-ca-file string
If set, the OpenID server's certificate will be verified by one of the authorities in the oidc-ca-file, otherwise the host's root CA set will be used.
--oidc-client-id string
The client ID for the OpenID Connect client, must be set if oidc-issuer-url is set.
--oidc-groups-claim string
If provided, the name of a custom OpenID Connect claim for specifying user groups. The claim value is expected to be a string or array of strings. This flag is experimental, please see the authentication documentation for further details.
--oidc-groups-prefix string
If provided, all groups will be prefixed with this value to prevent conflicts with other authentication strategies.
--oidc-issuer-url string
The URL of the OpenID issuer, only HTTPS sche

me will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT).
--oidc-required-claim &lt;comma-separated 'key=value' pairs&gt;
A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. Repeat this flag to specify multiple claims.
--oidc-signing-algs strings     Default: "RS256"
Comma-separated list of allowed JOSE asymmetric signing algorithms. JWTs with a supported 'alg' header values are: RS256, RS384, RS512, ES256, ES384, ES512, PS256, PS384, PS512. Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1.
--oidc-username-claim string     Default: "sub"
The OpenID claim to use as the user name. Note that claims other than the default ('sub') is not guaranteed to be unique and immutable. This flag is experimental, please see the authentication documentation for further details.
--oidc-username-prefix string
If provided, all usernames will be prefixed with this value. If not provided, username claims other than 'email' are prefixed by the issuer URL to avoid clashes. To skip any prefixing, provide the value '-'.
--permit-address-sharing
If true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]
--permit-port-sharing
If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]
--profiling     Default: true
Enable profiling via web interface host:port/debug/pprof/
--proxy-client-cert-file string
Client certificate used to prove the identity of the aggregator or kube-apiserver when it must call out during a request. This includes proxying requests to a user api-server and calling out to webhook admission plugins. It is expected that this cert includes a signature from the CA in the --requestheader-client-ca-file flag. That CA is published in the 'extension-apiserver-authentication' configmap in the kube-system namespace. Components receiving calls from kube-aggregator should use that CA to perform their half of the mutual TLS verification.
--proxy-client-key-file string
Private key for the client certificate used to prove the identity of the aggregator or kube-apiserver when it must call out during a request. This includes proxying requests to a user api-server and calling out to webhook admission plugins.
--request-timeout duration     Default: 1m0s
An optional field indicating the duration a handler must keep a request open before timing it out. This is the default request timeout for requests but may be overridden by flags such as --min-request-timeout for specific types of requests.
--requestheader-allowed-names strings
List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
--requestheader-client-ca-file string
Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
--requestheader-extra-headers-prefix strings
List of request header prefixes to inspect. X-Remote-Extra- is suggested.
--requestheader-group-headers strings
List of request headers to inspect for groups. X-Remote-Group is suggested.
--requestheader-username-headers strings
List of request headers to inspect for usernames. X-Remote-User is common.
--runtime-config &lt;comma-separated 'key=value' pairs&gt;
A set of key=value pairs that enable or disable built-in APIs. Supported options are:
v1=true|false for the core API group
&lt;group&gt;/&lt;version&gt;=true|false for a specific API group and version 

(e.g. apps/v1=true)
api/all=true|false controls all API versions
api/ga=true|false controls all API versions of the form v[0-9]+
api/beta=true|false controls all API versions of the form v[0-9]+beta[0-9]+
api/alpha=true|false controls all API versions of the form v[0-9]+alpha[0-9]+
api/legacy is deprecated, and will be removed in a future version
--secure-port int     Default: 6443
The port on which to serve HTTPS with authentication and authorization. It cannot be switched off with 0.
--service-account-extend-token-expiration     Default: true
Turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ignoring value of service-account-max-token-expiration.
--service-account-issuer strings
Identifier of the service account token issuer. The issuer will assert this identifier in "iss" claim of issued tokens. This value is a string or URI. If this option is not a valid URI per the OpenID Discovery 1.0 spec, the ServiceAccountIssuerDiscovery feature will remain disabled, even if the feature gate is set to true. It is highly recommended that this value comply with the OpenID spec: https://openid.net/specs/openid-connect-discovery-1_0.html. In practice, this means that service-account-issuer must be an https URL. It is also highly recommended that this URL be capable of serving OpenID discovery documents at {service-account-issuer}/.well-known/openid-configuration. When this flag is specified multiple times, the first is used to generate tokens and all are used to determine which issuers are accepted.
--service-account-jwks-uri string
Overrides the URI for the JSON Web Key Set in the discovery doc served at /.well-known/openid-configuration. This flag is useful if the discovery docand key set are served to relying parties from a URL other than the API server's external (as auto-detected or overridden with external-hostname).
--service-account-key-file strings
File containing PEM-encoded x509 RSA or ECDSA private or public keys, used to verify ServiceAccount tokens. The specified file can contain multiple keys, and the flag can be specified multiple times with different files. If unspecified, --tls-private-key-file is used. Must be specified when --service-account-signing-key-file is provided
--service-account-lookup     Default: true
If true, validate ServiceAccount tokens exist in etcd as part of authentication.
--service-account-max-token-expiration duration
The maximum validity duration of a token created by the service account token issuer. If an otherwise valid TokenRequest with a validity duration larger than this value is requested, a token will be issued with a validity duration of this value.
--service-account-signing-key-file string
Path to the file that contains the current private key of the service account token issuer. The issuer will sign issued ID tokens with this private key.
--service-cluster-ip-range string
A CIDR notation IP range from which to assign service cluster IPs. This must not overlap with any IP ranges assigned to nodes or pods. Max of two dual-stack CIDRs is allowed.
--service-node-port-range &lt;a string in the form 'N1-N2'&gt;     Default: 30000-32767
A port range to reserve for services with NodePort visibility. This must not overlap with the ephemeral port range on nodes. Example: '30000-32767'. Inclusive at both ends of the range.
--show-hidden-metrics-for-version string
The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is &lt;major&gt;.&lt;minor&gt;, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the releas

e after that.
--shutdown-delay-duration duration
Time to delay the termination. During that time the server keeps serving requests normally. The endpoints /healthz and /livez will return success, but /readyz immediately returns failure. Graceful termination starts after this delay has elapsed. This can be used to allow load balancer to stop sending traffic to this server.
--shutdown-send-retry-after
If true the HTTP Server will continue listening until all non long running request(s) in flight have been drained, during this window all incoming requests will be rejected with a status code 429 and a 'Retry-After' response header, in addition 'Connection: close' response header is set in order to tear down the TCP connection when idle.
--storage-backend string
The storage backend for persistence. Options: 'etcd3' (default).
--storage-media-type string     Default: "application/vnd.kubernetes.protobuf"
The media type to use to store objects in storage. Some resources or storage backends may only support a specific media type and will ignore this setting. Supported media types: [application/json, application/yaml, application/vnd.kubernetes.protobuf]
--strict-transport-security-directives strings
List of directives for HSTS, comma separated. If this list is empty, then HSTS directives will not be added. Example: 'max-age=31536000,includeSubDomains,preload'
--tls-cert-file string
File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
--tls-cipher-suites strings
Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.
--tls-min-version string
Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
--tls-private-key-file string
File containing the default x509 private key matching --tls-cert-file.
--tls-sni-cert-key string
A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".
--token-auth-file string
If set, the file that will be used to secure the secure port of the API server via token authentication.
--tracing-config-file string
File with apiserver tracing configuration.
-v, --v int
number for 

the log level verbosity
--version version[=true]
Print version information and quit
--vmodule pattern=N,...
comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
--watch-cache     Default: true
Enable watch caching in the apiserver
--watch-cache-sizes strings
Watch cache size settings for some resources (pods, nodes, etc.), comma separated. The individual setting format: resource[.group]#size, where resource is lowercase plural (no version), group is omitted for resources of apiVersion v1 (the legacy core API) and included for others, and size is a number. This option is only meaningful for resources built into the apiserver, not ones defined by CRDs or aggregated from external servers, and is only consulted if the watch-cache is enabled. The only meaningful size setting to supply here is zero, which means to disable watch caching for the associated resource; all non-zero values are equivalent and mean to not disable watch caching for that resource
Feedback
Yes
No
.
or
.
Last modified December 10, 2022 at 11:42 PM PST:
Component reference for 1.26 (f2768f6c40)
Synopsis
Options
Home
Blog
Training
Partners
Community
CC BY 4.0

Object Names and IDs | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggregation

 Layer
Tasks
Debug Pods
Determi

ne the Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standard

s at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Con

figuration (v1)
Concepts
Overview
Each object in your cluster has a
Name
that is unique for that type of resource.
Every Kubernetes object also has a
UID
that is unique across your whole cluster.
For example, you can only have one Pod named
myapp-1234
within the same
namespace
, but you can have one Pod and one Deployment that are each named
.
For non-unique user-provided attributes, Kubernetes provides
labels
and
.
Names
A client-provided string that refers to an object in a resource URL, such as
/api/v1/pods/some-name
.
Only one object of a given kind can have a given name at a time. However, if you delete the object, you can make a new object with the same name.
Note:
In cases when objects represent a physical entity, like a Node representing a physical host, when the host is re-created under the same name without deleting and re-creating the Node, Kubernetes treats the new host as the old one, which may lead to inconsistencies.
Below are four types of commonly used name constraints for resources.
DNS Subdomain Names
Most resource types require a name that can be used as a DNS subdomain name
as defined in
RFC 1123
.
This means the name must:
contain no more than 253 characters
contain only lowercase alphanumeric characters, '-' or '.'
start with an alphanumeric character
end with an alphanumeric character
RFC 1123 Label Names
Some resource types require their names to follow the DNS
label standard as defined in
RFC 1123
.
contain at most 63 characters
contain only lowercase alphanumeric characters or '-'
RFC 1035 Label Names
RFC 1035
.
start with an alphabetic character
Path Segment Names
Some resource types require their names to be able to be safely encoded as a
path segment. In other words, the name may not be "." or ".." and the name may
not contain "/" or "%".
Here's an example manifest for a Pod named
nginx-demo
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Note:
Some resource types have additional restrictions on their names.
UIDs
A Kubernetes systems-generated s

tring to uniquely identify objects.
Every object created over the whole lifetime of a Kubernetes cluster has a distinct UID. It is intended to distinguish between historical occurrences of similar entities.
Kubernetes UIDs are universally unique identifiers (also known as UUIDs).
UUIDs are standardized as ISO/IEC 9834-8 and as ITU-T X.667.
Read about
labels
and
See the
Identifiers and Names in Kubernetes
design document.
Feedback
Yes
No
.
or
.
Last modified November 11, 2022 at 6:22 PM PST:
added annotations in what's next sections (419d3b0d8e)
Names
UIDs
Home
Blog
Training
Partners
Community
CC BY 4.0

ReplicaSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Reference
ReplicaSet ensures that a specified number of pod replicas are running at any given time.
: apps/v1
kind
: ReplicaSet
metadata
(
)
If the Labels of a ReplicaSet are empty, they are defaulted to be the same as the Pod(s) that the ReplicaSet manages. Standard object's metadata. More info:
spec
(
ReplicaSetSpec
)
Spec defines the specification of the desired behavior of the ReplicaSet. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
(
ReplicaSetStatus
)
Status is the most recently observed status of the ReplicaSet. This data may be out of date by some window of time. Populated by the system. Read-only. More info:
ReplicaSetSpec is the specification of a ReplicaSet.
selector
(
Selector is a label query over pods that should match the replica count. Label keys and values that must match in order to be controlled by this replica set. It must match the pod template's labels. More info:
template
(
)
Template is the object that describes the pod that will be created if insufficient replicas are detected. More info:
https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#pod-template
replicas
(int32)
Replicas is the number of desired replicas. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info:
https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#what-is-a-replicationcontroller
(int32)
Minimum number of seconds for which a newly created pod should be ready without any of its container crashing, for it to be considered available. Defaults to 0 (pod will be considered available as soon as it is ready)
ReplicaSetStatus represents the current status of a ReplicaSet.
replicas
Replicas is the most recently observed number of replicas. More info:
(int32)
The number of available replicas (ready for at

 least minReadySeconds) for this replica set.
(int32)
readyReplicas is the number of pods targeted by this ReplicaSet with a Ready Condition.
fullyLabeledReplicas
(int32)
The number of pods that have labels matching the labels of the pod template of the replicaset.
([]ReplicaSetCondition)
type
Represents the latest available observations of a replica set's current state.
ReplicaSetCondition describes the state of a replica set at a certain point.
Type of replica set condition.
(Time)
The last time the condition transitioned from one status to another.
(string)
(string)
(int64)
ObservedGeneration reflects the generation of the most recently observed ReplicaSet.
ReplicaSetList
ReplicaSetList is a collection of ReplicaSets.
: apps/v1
kind
: ReplicaSetList
metadata
(
ListMeta
)
Standard list metadata. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
items
([]
List of ReplicaSets. More info:
https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller
get
read the specified ReplicaSet
GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}
name
(
in path
name of the ReplicaSet
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified ReplicaSet
GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ReplicaSet
GET /apis/apps/v1/namespaces/{namespace}/replicasets
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/apps/v1/replicasets
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ReplicaSet
POST /apis/apps/v1/namespaces/{namespace}/replicasets
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): 

Created
202 (
update
replace the specified ReplicaSet
PUT /apis/apps/v1/namespaces/{namespace}/replicasets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified ReplicaSet
PUT /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ReplicaSet
PATCH /apis/apps/v1/namespaces/{namespace}/replicasets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified ReplicaSet
PATCH /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ReplicaSet
DELETE /apis/apps/v1/namespaces/{namespace}/replicasets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ReplicaSet
DELETE /apis/apps/v1/namespaces/{namespace}/replicasets
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
API

 Reference multi-pages for v1.26 (366ba7f62c)
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Node Reference Information | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Debug Pods
Get a Sh

ell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Rest

rict a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encrypti

on Configuration (v1)
Reference
This section contains the following reference topics about nodes:
the kubelet's
checkpoint API
a list of
You can also read node reference details from elsewhere in the
Kubernetes documentation, including:
Node Metrics Data
.
Feedback
Yes
No
.
or
.
Last modified November 19, 2022 at 2:52 PM PST:
Split out node metrics into reference section (27550bce48)
Home
Blog
Training
Partners
Community
CC BY 4.0

Stateless Applications | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Exposing an External IP Address to Access an Application in a Cluster
This page shows how to create a Kubernetes Service object that exposes an
external IP address.
Install
kubectl
.
Use a cloud provider like Google Kubernetes Engine or Amazon Web Services to
create a Kubernetes cluster. This tutorial creates an
external load balancer
,
which requires a cloud provider.
Configure
kubectl
to communicate with your Kubernetes API server. For instructions, see the
documentation for your cloud provider.
Run five instances of a Hello World application.
Create a Service object that exposes an external IP address.
Use the Service object to access the running application.
Creating a service for an application running in five pods
Run a Hello World application in your cluster:
service/load-balancer-example.yaml
:
apps/v1
kind
:
metadata
:
labels
:
app.kubernetes.io/name
:
load-balancer-example
name
:
hello-world
spec
:
replicas
:
5
selector
:
:
:
template
:
metadata
:
labels
:
:
spec
:
:
-
image
:
gcr.io/google-samples/node-hello:1.0
name
:
ports
:
-
:
8080
kubectl apply -f https://k8s.io/examples/service/load-balancer-example.yaml
The preceding command creates a
and an associated
.
The ReplicaSet has five
Pods
each of which runs the Hello World application.
Display information about the Deployment:
kubectl get deployments hello-world
kubectl describe deployments hello-world
Display information about your ReplicaSet objects:
kubectl get replicasets
kubectl describe replicasets
Create a Service object that exposes the deployment:
kubectl expose deployment hello-world --type
=
LoadBalancer --name
=
my-service
Display information about the Service:
kubectl get services my-service
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)    AGE
my-service   LoadBalancer   10.3.245.137   104.198.205.71   8080/TCP   54s
Note:
The
type=LoadBalancer
service is backed by external cloud providers, which is not covered in this example, please refer to
this page
for the details.
Note:
If the external IP address is shown as &lt;pending&gt;, wait for a minute and enter the same command again.
Display detailed information about the Service:
kubectl describe services my-service
Name:           my-service
Namespace:      default
Labels:         app.kubernetes.io/name=load-balancer-example
Selector:       app.kubernetes.io/name=load-balancer-example
Type:           LoadBalancer
IP:             10.3.245.137
LoadBalancer Ingress:   104.198.205.71
Port:           &lt;unset&gt; 8080/TCP
NodePort:       &lt;unset&gt; 32377/TCP
Endpoints:      10.0.0.6:8080,10.0.1.6:8080,10.0.1.7:8080 + 2 more...
Session Affinity:   None
Events:         &lt;none&gt;
Make a note of the external IP address (
LoadBalancer Ingress
) exposed by
your service. In this example, the external IP address is 104.198.205.71.
Also note the value of
Port
and
NodePort
. In this example, the
Port
is 8080 and the
NodePort
is 32377.
In the preceding output, you can see that the service has several endpoints:
10.0.0.6:8080,10.0.1.6:8080,10.0.1.7:8080 + 2 more. These are internal
addresses of the pods that are running the Hello World application. To
v

erify these are pod addresses, enter this command:
kubectl get pods --output
=
wide
NAME                         ...  IP         NODE
hello-world-2895499144-1jaz9 ...  10.0.1.6   gke-cluster-1-default-pool-e0b8d269-1afc
hello-world-2895499144-2e5uh ...  10.0.1.8   gke-cluster-1-default-pool-e0b8d269-1afc
hello-world-2895499144-9m4h1 ...  10.0.0.6   gke-cluster-1-default-pool-e0b8d269-5v7a
hello-world-2895499144-o4z13 ...  10.0.1.7   gke-cluster-1-default-pool-e0b8d269-1afc
hello-world-2895499144-segjf ...  10.0.2.5   gke-cluster-1-default-pool-e0b8d269-cpuc
Use the external IP address (
) to access the Hello
World application:
curl http://&lt;external-ip&gt;:&lt;port&gt;
where
&lt;external-ip&gt;
is the external IP address (
)
of your Service, and
&lt;port&gt;
is the value of
Port
in your Service
description.
If you are using minikube, typing
minikube service my-service
will
automatically open the Hello World application in a browser.
The response to a successful request is a hello message:
Hello Kubernetes!
Cleaning up
To delete the Service, enter this command:
kubectl delete services my-service
To delete the Deployment, the ReplicaSet, and the Pods that are running
the Hello World application, enter this command:
kubectl delete deployment hello-world
connecting applications with services
.
2 - Example: Deploying PHP Guestbook application with Redis
This tutorial shows you how to build and deploy a simple
(not production
ready)
, multi-tier web application using Kubernetes and
Docker
. This example consists of the following
components:
A single-instance
Redis
to store guestbook entries
Multiple web frontend instances
Start up a Redis leader.
Start up two Redis followers.
Start up the guestbook frontend.
Expose and view the Frontend Service.
Clean up.
minikube
Your Kubernetes server must be at or later than version v1.14.
.
Start up the Redis Database
The guestbook application uses Redis to store its data.
Creating the Redis Deployment
The manifest file, included below, specifies a Deployment controller that runs a single replica Redis Pod.
application/guestbook/redis-leader-deployment.yaml
# SOURCE: https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
:
apps/v1
kind
:
metadata
:
name
:
redis-leader
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
replicas
:
1
selector
:
:
app
:
redis
template
:
metadata
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
:
-
name
:
leader
image
:
"docker.io/redis:6.0.5"
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
6379
Launch a terminal window in the directory you downloaded the manifest files.
Apply the Redis Deployment from the
redis-leader-deployment.yaml
file:
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-leader-deployment.yaml
Query the list of Pods to verify that the Redis Pod is running:
The response should be similar to this:
NAME                           READY   STATUS    RESTARTS   AGE
redis-leader-fb76b4755-xjr2n   1/1     Running   0          13s
Run the following command to view the logs from the Redis leader Pod:
kubectl logs -f deployment/redis-leader
Creating the Redis leader Service
The guestbook application needs to communicate to the Redis to write its data.
You need to apply a
Service
to
proxy the traffic to the Redis Pod. A Service defines a policy to access the
Pods.
applic

ation/guestbook/redis-leader-service.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
ports
:
-
port
:
6379
targetPort
:
6379
selector
:
app
:
redis
role
:
leader
tier
:
backend
Apply the Redis Service from the following
redis-leader-service.yaml
file:
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-leader-service.yaml
Query the list of Services to verify that the Redis Service is running:
NAME           TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes     ClusterIP   10.0.0.1     &lt;none&gt;        443/TCP    1m
redis-leader   ClusterIP   10.103.78.24 &lt;none&gt;        6379/TCP   16s
Note:
This manifest file creates a Service named
with a set of labels
that match the labels previously defined, so the Service routes network
traffic to the Redis Pod.
Set up Redis followers
Although the Redis leader is a single Pod, you can make it highly available
and meet traffic demands by adding a few Redis followers, or replicas.
application/guestbook/redis-follower-deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
redis-follower
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
replicas
:
2
selector
:
:
app
:
redis
template
:
metadata
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
:
-
name
:
follower
image
:
gcr.io/google_samples/gb-redis-follower:v2
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
6379
Apply the Redis Deployment from the following
redis-follower-deployment.yaml
file:
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-follower-deployment.yaml
Verify that the two Redis follower replicas are running by querying the list of Pods:
NAME                             READY   STATUS    RESTARTS   AGE
redis-follower-dddfbdcc9-82sfr   1/1     Running   0          37s
redis-follower-dddfbdcc9-qrt5k   1/1     Running   0          38s
redis-leader-fb76b4755-xjr2n     1/1     Running   0          11m
Creating the Redis follower service
The guestbook application needs to communicate with the Redis followers to
read data. To make the Redis followers discoverable, you must set up another
Service
.
application/guestbook/redis-follower-service.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
ports
:
# the port that this service should serve on
-
port
:
6379
selector
:
app
:
redis
role
:
follower
tier
:
backend
redis-follower-service.yaml
file:
kubectl apply -f https://k8s.io/examples/application/guestbook/redis-follower-service.yaml
NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
kubernetes       ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP    3d19h
redis-follower   ClusterIP   10.110.162.42   &lt;none&gt;        6379/TCP   9s
redis-leader     ClusterIP   10.103.78.24    &lt;none&gt;        6379/TCP   6m10s
Note:
with a set of
labels that match the labels previously defined, so the Service routes network
Set up and Expose the Guestbook Frontend
Now that you have the Redis storage of your guestbook up and running, start
the guestbook web servers. Like the Redis followers, the frontend is deployed
using a Kubernetes Deployment.
The guestbook app uses a PHP frontend. It is configured to communicat

e with
either the Redis follower or leader Services, depending on whether the request
is a read or a write. The frontend exposes a JSON interface, and serves a
jQuery-Ajax-based UX.
Creating the Guestbook Frontend Deployment
application/guestbook/frontend-deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
frontend
spec
:
replicas
:
3
selector
:
:
app
:
guestbook
tier
:
frontend
template
:
metadata
:
labels
:
app
:
guestbook
tier
:
frontend
spec
:
:
-
name
:
php-redis
image
:
gcr.io/google_samples/gb-frontend:v5
env
:
-
name
:
GET_HOSTS_FROM
value
:
"dns"
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
80
Apply the frontend Deployment from the
frontend-deployment.yaml
file:
kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-deployment.yaml
Query the list of Pods to verify that the three frontend replicas are running:
app
=
guestbook -l
tier
=
frontend
NAME                        READY   STATUS    RESTARTS   AGE
frontend-85595f5bf9-5tqhb   1/1     Running   0          47s
frontend-85595f5bf9-qbzwm   1/1     Running   0          47s
frontend-85595f5bf9-zchwc   1/1     Running   0          47s
Creating the Frontend Service
The
Redis
Services you applied is only accessible within the Kubernetes
cluster because the default type for a Service is
ClusterIP
.
ClusterIP
provides a single IP address for the set of Pods the Service is
pointing to. This IP address is accessible only within the cluster.
If you want guests to be able to access your guestbook, you must configure the
frontend Service to be externally visible, so a client can request the Service
from outside the Kubernetes cluster. However a Kubernetes user can use
kubectl port-forward
to access the service even though it uses a
ClusterIP
.
Note:
Some cloud providers, like Google Compute Engine or Google Kubernetes Engine,
support external load balancers. If your cloud provider supports load
balancers and you want to use it, uncomment
type: LoadBalancer
.
application/guestbook/frontend-service.yaml
:
v1
kind
:
Service
metadata
:
name
:
frontend
labels
:
app
:
guestbook
tier
:
frontend
spec
:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
# type: LoadBalancer
#type: LoadBalancer
ports
:
-
port
:
80
selector
:
app
:
guestbook
tier
:
frontend
Apply the frontend Service from the
frontend-service.yaml
file:
kubectl apply -f https://k8s.io/examples/application/guestbook/frontend-service.yaml
Query the list of Services to verify that the frontend Service is running:
kubectl get services
frontend         ClusterIP   10.97.28.230    &lt;none&gt;        80/TCP     19s
redis-follower   ClusterIP   10.110.162.42   &lt;none&gt;        6379/TCP   5m48s
redis-leader     ClusterIP   10.103.78.24    &lt;none&gt;        6379/TCP   11m
Viewing the Frontend Service via
Run the following command to forward port
8080
on your local machine to port
80
on the service.
kubectl port-forward svc/frontend 8080:80
Forwarding from 127.0.0.1:8080 -&gt; 80
Forwarding from [::1]:8080 -&gt; 80
load the page
http://localhost:8080
in your browser to view your guestbook.
If you deployed the
manifest with type:
you need to find the IP address to view your Guestbook.
Run the following command to get the IP address for the frontend Service.

kubectl get service frontend
NAME       TYPE           CLUSTER-IP      EXTERNAL-IP        PORT(S)        AGE
frontend   LoadBalancer   10.51.242.136   109.197.92.229     80:32372/TCP   1m
Copy the external IP address, and load the page in your browser to view your guestbook.
Note:
Try adding some guestbook entries by typing in a message, and clicking Submit.
The message you typed appears in the frontend. This message indicates that
data is successfully added to Redis through the Services you created earlier.
Scale the Web Frontend
You can scale up or down as needed because your servers are defined as a
Service that uses a Deployment controller.
Run the following command to scale up the number of frontend Pods:
kubectl scale deployment frontend --replicas
=
5
Query the list of Pods to verify the number of frontend Pods running:
The response should look similar to this:
frontend-85595f5bf9-5df5m        1/1     Running   0          83s
frontend-85595f5bf9-7zmg5        1/1     Running   0          83s
frontend-85595f5bf9-cpskg        1/1     Running   0          15m
frontend-85595f5bf9-l2l54        1/1     Running   0          14m
frontend-85595f5bf9-l9c8z        1/1     Running   0          14m
redis-follower-dddfbdcc9-82sfr   1/1     Running   0          97m
redis-follower-dddfbdcc9-qrt5k   1/1     Running   0          97m
redis-leader-fb76b4755-xjr2n     1/1     Running   0          108m
Run the following command to scale down the number of frontend Pods:
=
2
frontend-85595f5bf9-cpskg        1/1     Running   0          16m
frontend-85595f5bf9-l9c8z        1/1     Running   0          15m
redis-follower-dddfbdcc9-82sfr   1/1     Running   0          98m
redis-follower-dddfbdcc9-qrt5k   1/1     Running   0          98m
redis-leader-fb76b4755-xjr2n     1/1     Running   0          109m
Deleting the Deployments and Services also deletes any running Pods. Use
labels to delete multiple resources with one command.
Run the following commands to delete all Pods, Deployments, and Services.
kubectl delete deployment -l
app
=
redis
kubectl delete service -l
app
=
redis
kubectl delete deployment frontend
kubectl delete service frontend
deployment.apps "redis-follower" deleted
deployment.apps "redis-leader" deleted
deployment.apps "frontend" deleted
service "frontend" deleted
Query the list of Pods to verify that no Pods are running:
No resources found in default namespace.
Complete the
Kubernetes Basics
Interactive Tutorials
Use Kubernetes to create a blog using
Persistent Volumes for MySQL and Wordpress
Read more about
Home
Blog
Training
Partners
Community
CC BY 4.0

Endpoints | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

d Set Up kubectl on Windows
Debug Pods
Troubleshooting Clus

ters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resources with App

Armor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configuration 

(v1alpha1)
Reference
Endpoints
Endpoints
Endpoints is a collection of endpoints that implement the actual service.
import "k8s.io/api/core/v1"
Endpoints
Endpoints is a collection of endpoints that implement the actual service. Example:
Name: "mysvc",
 Subsets: [
   {
     Addresses: [{"ip": "10.10.1.1"}, {"ip": "10.10.2.2"}],
     Ports: [{"name": "a", "port": 8675}, {"name": "b", "port": 309}]
   },
   {
     Addresses: [{"ip": "10.10.3.3"}],
     Ports: [{"name": "a", "port": 93}, {"name": "b", "port": 76}]
   },
]
: v1
kind
: Endpoints
metadata
(
)
subsets
([]EndpointSubset)
The set of all endpoints is the union of all subsets. Addresses are placed into subsets according to the IPs they share. A single address with multiple ports, some of which are ready and some of which are not (because they come from different containers) will result in the address being displayed in different subsets for the different ports. No address will appear in both Addresses and NotReadyAddresses in the same subset. Sets of addresses and ports that comprise a service.
*EndpointSubset is a group of addresses with a common set of ports. The expanded set of endpoints is the Cartesian product of Addresses x Ports. For example, given:
{
Addresses: [{"ip": "10.10.1.1"}, {"ip": "10.10.2.2"}],
Ports: [{"name": "a", "port": 8675}, {"name": "b", "port": 309}]
}
The resulting set of endpoints can be viewed as:
a: [ 10.10.1.1:8675, 10.10.2.2:8675 ],
b: [ 10.10.1.1:309, 10.10.2.2:309 ]*
subsets.addresses
([]EndpointAddress)
IP addresses which offer the related ports that are marked as ready. These endpoints should be considered safe for load balancers and clients to utilize.
EndpointAddress is a tuple that describes single IP address.
subsets.addresses.ip
The IP of this endpoint. May not be loopback (127.0.0.0/8), link-local (169.254.0.0/16), or link-local multicast ((224.0.0.0/24). IPv6 is also accepted but not fully supported on all platforms. Also, certain kubernetes components, like kube-proxy, are not IPv6 ready.
subsets.addresses.hostname
(string)
The Hostname of this endpoint
subsets.addresses.nodeName
(string)
Optional: Node hosting this endpoint. This can be used to determine endpoints local to a node.
subsets.addresses.targetRef
(
)
Reference to object providing the endpoint.
subsets.notReadyAddresses
IP addresses which offer the related ports but are not currently marked as 

ready because they have not yet finished starting, have recently failed a readiness check, or have recently failed a liveness check.
subsets.notReadyAddresses.ip
subsets.notReadyAddresses.hostname
(string)
subsets.notReadyAddresses.nodeName
(string)
subsets.notReadyAddresses.targetRef
(
)
subsets.ports
([]EndpointPort)
Port numbers available on the related IP addresses.
EndpointPort is a tuple that describes a single port.
subsets.ports.port
The port number of the endpoint.
subsets.ports.protocol
(string)
The IP protocol for this port. Must be UDP, TCP, or SCTP. Default is TCP.
subsets.ports.name
(string)
The name of this port. This must match the 'name' field in the corresponding ServicePort. Must be a DNS_LABEL. Optional only if one port is defined.
subsets.ports.appProtocol
(string)
The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and
https://www.iana.org/assignments/service-names)
. Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.
EndpointsList
EndpointsList is a list of endpoints.
: v1
kind
: EndpointsList
metadata
(
ListMeta
)
items
([]
Endpoints
List of endpoints.
get
read the specified Endpoints
GET /api/v1/namespaces/{namespace}/endpoints/{name}
name
(
in path
name of the Endpoints
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
list
list or watch objects of kind Endpoints
GET /api/v1/namespaces/{namespace}/endpoints
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/endpoints
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create Endpoints
POST /api/v1/namespaces/{namespace}/endpoints
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
202 (
Endpoints
4

01: Unauthorized
update
replace the specified Endpoints
PUT /api/v1/namespaces/{namespace}/endpoints/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
patch
partially update the specified Endpoints
PATCH /api/v1/namespaces/{namespace}/endpoints/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
delete
delete Endpoints
DELETE /api/v1/namespaces/{namespace}/endpoints/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Endpoints
DELETE /api/v1/namespaces/{namespace}/endpoints
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Last modified August 03, 2022 at 6:01 PM PST:
API Reference multi-pages for v1.25 (939b45bb27)
Endpoints
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubelet Configuration (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Com

pute, Storage, and Networking Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodSec

urity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfile, Config

Maps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate Lim

it Configuration (v1alpha1)
Reference
CredentialProviderConfig is the configuration containing information about
each exec credential provider. Kubelet reads this configuration from disk and enables
each provider as specified by the CredentialProvider type.
Field
string
kind
string
providers
[]CredentialProvider
providers is a list of credential provider plugins that will be enabled by the kubelet.
Multiple providers may match against a single image, in which case credentials
from all providers will be returned to the kubelet. If multiple providers are called
for a single image, the results are combined. If providers return overlapping
auth keys, the value from the provider earlier in this list is used.
CredentialProvider
CredentialProvider represents an exec plugin to be invoked by the kubelet. The plugin is only
invoked when an image being pulled matches the images handled by the plugin (see matchImages).
Field
name
string
name is the required name of the credential provider. It must match the name of the
provider executable as seen by the kubelet. The executable must be in the kubelet's
bin directory (set by the --image-credential-provider-bin-dir flag).
[]string
matchImages is a required list of strings used to match against images in order to
determine if this provider should be invoked. If one of the strings matches the
requested image from the kubelet, the plugin will be invoked and given a chance
to provide credentials. Images are expected to contain the registry domain
and URL path.
Each entry in matchImages is a pattern which can optionally contain a port and a path.
'.
.k8s.io' is also supported. Each glob can only match
A match exists between an image and a matchImage when all of the below are true:
The URL path of an 

imageMatch must be a prefix of the target image URL path.
Example values of matchImages:
gcr.io
.
defaultCacheDuration is the default duration the plugin will cache credentials in-memory
if a cache duration is not provided in the plugin response. This field is required.
string
Required input version of the exec CredentialProviderRequest. The returned CredentialProviderResponse
MUST use the same encoding version as the input. Current supported values are:
args
[]string
env
Field
name
string
value
string
FormatOptions
LoggingConfiguration
FormatOptions contains options for the different logging formats.
Field
json
JSONOptions
[Alpha] JSON contains options for logging format "json".
Only available when the LoggingAlphaOptions feature gate is enabled.
JSONOptions contains options for logging format "json".
Field
splitStream
bool
[Alpha] SplitStream redirects error messages to stderr while
info messages go to stdout, with buffering. The default is to write
both to stdout, without buffering. Only available when
the LoggingAlphaOptions feature gate is enabled.
infoBufferSize
k8s.io/apimachinery/pkg/api/resource.QuantityValue
[Alpha] InfoBufferSize sets the size of the info stream when
using split streams. The default is zero, which disables buffering.
LogFormatFactory
LogFormatFactory provides support for a certain additional,
non-default log format.
LoggingConfiguration contains logging options.
Field
format
string
Format Flag specifies the structure of log messages.
default value of format is
text
flushFrequency
time.Duration
Maximum number of nanoseconds (i.e. 1s = 1000000000) between log
flushes. Ignored if the selected logging backend writes log
messages without buffering.
verbosity
VerbosityLevel
Verbosity is the threshold that determines which log messages are
logged. Default is zero which logs only the most important
messages. Higher values enable additional messages. Error messages
are always logged.
vmodule
VModuleConfiguration
VModule overrides the verbosity threshold for individual files.
Only supported for "text" log format.
options
[Alpha] Options holds additional parameters that are specific
to the different logging formats. Only the options for the selected
format get used, but all of them get validated.
TracingConfiguration provides versioned configuration for OpenTelemetry tracing clients.
Field
endpoint
string
Endpoint of the collector this component will report traces to.
The connection is insecure, and does not currently support TLS.
Recommended is unset, and endpoint is the otlp grpc default, localhost:4317.
int32
Recommended is unset. If unset, sampler respects its parent span's sampling
rate, but otherwise never samples.
VModuleCo

nfiguration
(Alias of
[]k8s.io/component-base/logs/api/v1.VModuleItem
)
VModuleConfiguration is a collection of individual file names or patterns
and the corresponding verbosity threshold.
(Alias of
uint32
)
VerbosityLevel represents a klog or logr verbosity threshold.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Reviewing changes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

stall and Set Up kubectl on macOS
Debug Pods
Get a Shell

 to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restric

t a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption 

Configuration (v1)
Reviewing for approvers and reviewers
Feedback
Yes
No
.
or
.
Last modified May 30, 2020 at 3:10 PM PST:
add en pages (ecc27bbbe7)
Home
Blog
Training
Partners
Community
CC BY 4.0

Cluster Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

macOS
Debug Pods
Trou

bleshooting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Re

sources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-prox

y Configuration (v1alpha1)
Reference
Node
Node is a worker node in Kubernetes.
Namespace
Namespace provides a scope for Names.
Event
Event is a report of an event somewhere in the cluster.
Lease
Lease defines a lease concept.
RuntimeClass defines a class of container runtime supported in the cluster.
FlowSchema defines the schema of a group of flows.
PriorityLevelConfiguration represents the configuration of a priority level.
Binding
Binding ties one object to another; for example, a pod is bound to a node by a scheduler.
ComponentStatus (and ComponentStatusList) holds the cluster validation info.
ClusterCIDR represents a single configuration for per-Node Pod CIDR allocations when the MultiCIDRRangeAllocator is enabled (see the config for kube-controller-manager).
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Viewing Pods and Nodes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Netw

ork Plugins
Tasks
Monitoring, Loggin

g, and Debugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Confi

guring a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Image Policy API 

Tutorials
Learn about Kubernetes Pods.
Learn about Kubernetes Nodes.
Troubleshoot deployed applications.
Kubernetes Pods
When you created a Deployment in Module
2
, Kubernetes created a
Pod
to host your application instance. A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. Those resources include:
Shared storage, as Volumes
Networking, as a unique cluster IP address
Information about how to run each container, such as the container image version or specific ports to use
A Pod models an application-specific "logical host" and can contain different application containers which are relatively tightly coupled. For example, a Pod might include both the container with your Node.js app as well as a different container that feeds the data to be published by the Node.js webserver. The containers in a Pod share an IP Address and port space, are always co-located and co-scheduled, and run in a shared context on the same Node.
Pods are the atomic unit on the Kubernetes platform. When we create a Deployment on Kubernetes, that Deployment creates Pods with containers inside them (as opposed to creating containers directly). Each Pod is tied to the Node where it is scheduled, and remains there until termination (according to restart policy) or deletion. In case of a Node failure, identical Pods are scheduled on other available Nodes in the cluster.
Summary:
Pods
Nodes
Kubectl main commands
A Pod is a group of one or more application containers (such as Docker) and includes shared storage (volumes), IP address and information about how to run them.
Pods overview
Nodes
A Pod always runs on a
Node
. A Node is a worker machine in Kubernetes and may be either a virtual or a physical machine, depending on the cluster. Each Node is managed by the control plane. A Node can have multiple pods, and the Kubernetes control plane automatically handles scheduling the pods across the Nodes in the cluster. The control plane's automatic scheduling takes into account the available resources on each Node.
Every Kubernetes Node runs at least:
Kubelet, a process responsible for communication between the Kubernetes control plane and the Node; it manages the Pods and the containers runnin

g on a machine.
A container runtime (like Docker) responsible for pulling the container image from a registry, unpacking the container, and running the application.
Containers should only be scheduled together in a single Pod if they are tightly coupled and need to share resources such as disk.
Node overview
Troubleshooting with kubectl
In Module
2
, you used Kubectl command-line interface. You'll continue to use it in Module 3 to get information about deployed applications and their environments. The most common operations can be done with the following kubectl commands:
- list resources
- show detailed information about a resource
- print the logs from a container in a pod
- execute a command on a container in a pod
You can use these commands to see when applications were deployed, what their current statuses are, where they are running and what their configurations are.
Now that we know more about our cluster components and the command line, let's explore our application.
A node is a worker machine in Kubernetes and may be a VM or physical machine, depending on the cluster. Multiple Pods can run on one Node.
Start Interactive Tutorial
›
Feedback
Yes
No
.
or
.
Last modified October 02, 2022 at 10:10 PM PST:
fix: CSS inconsistencies between docs/tutorials/kubernetes-basics and… (#34188) (d75f302c1f)
Home
Blog
Training
Partners
Community
CC BY 4.0

Documentation style overview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Ag

gregation Layer
Tasks
Debug Pods
Debug a StatefulSe

t
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security

 Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apis

erver Configuration (v1)
The topics in this section provide guidance on writing style, content formatting
and organization, and using Hugo customizations specific to Kubernetes
Documentation Content Guide
Documentation Style Guide
Diagram Guide
Feedback
Yes
No
.
or
.
Last modified August 06, 2018 at 4:15 PM PST:
Reorganize docs contrib guide (#9510) (5ae0d0dd8d)
Home
Blog
Training
Partners
Community
CC BY 4.0

EndpointSlice | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

stall and Set Up kubectl on Windows
Debug Pods
Troubleshoot

ing Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resources 

with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Config

uration (v1alpha1)
Reference
EndpointSlice represents a subset of the endpoints that implement a service.
apiVersion: discovery.k8s.io/v1
import "k8s.io/api/discovery/v1"
EndpointSlice represents a subset of the endpoints that implement a service. For a given service there may be multiple EndpointSlice objects, selected by labels, which must be joined to produce the full set of endpoints.
: discovery.k8s.io/v1
kind
: EndpointSlice
metadata
(
)
Standard object's metadata.
addressType
addressType specifies the type of address carried by this EndpointSlice. All addresses in this slice must be the same type. This field is immutable after creation. The following address types are currently supported: * IPv4: Represents an IPv4 Address. * IPv6: Represents an IPv6 Address. * FQDN: Represents a Fully Qualified Domain Name.
endpoints
([]Endpoint), required
endpoints is a list of unique endpoints in this slice. Each slice may include a maximum of 1000 endpoints.
Endpoint represents a single logical "backend" implementing a service.
endpoints.addresses
([]string), required
Set: unique values will be kept during a merge
addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to:
https://issue.k8s.io/106267
endpoints.conditions
(EndpointConditions)
conditions contains information about the current status of the endpoint.
EndpointConditions represents the current condition of an endpoint.
endpoints.conditions.ready
(boolean)
ready indicates that this endpoint is prepared to receive traffic, according to whatever system is managing the endpoint. A nil value indicates an unknown state. In most cases consumers should interpret this unknown state as ready. For compatibility reasons, ready should never be "true" for terminating endpoints.
endpoints.conditions.serving
(boolean)
serving is identical to ready except that it is set regardless of the terminating state of endpoints. This condition should be set to true for a ready endpoint that is terminating. If nil, consumers should defer to the ready condition.
endpoints.conditions.terminating
(boolean)
terminating indicates that this endpoint is terminating. A nil value indicates an unknown state. C

onsumers should interpret this unknown state to mean that the endpoint is not terminating.
endpoints.deprecatedTopology
(map[string]string)
deprecatedTopology contains topology information part of the v1beta1 API. This field is deprecated, and will be removed when the v1beta1 API is removed (no sooner than kubernetes v1.24). While this field can hold values, it is not writable through the v1 API, and any attempts to write to it will be silently ignored. Topology information can be found in the zone and nodeName fields instead.
endpoints.hints
(EndpointHints)
hints contains information associated with how an endpoint should be consumed.
EndpointHints provides hints describing how an endpoint should be consumed.
endpoints.hints.forZones
([]ForZone)
forZones indicates the zone(s) this endpoint should be consumed by to enable topology aware routing.
ForZone provides information about which zones should consume this endpoint.
endpoints.hints.forZones.name
name represents the name of the zone.
endpoints.hostname
(string)
hostname of this endpoint. This field may be used by consumers of endpoints to distinguish endpoints from each other (e.g. in DNS names). Multiple endpoints which use the same hostname should be considered fungible (e.g. multiple A values in DNS). Must be lowercase and pass DNS Label (RFC 1123) validation.
endpoints.nodeName
(string)
nodeName represents the name of the Node hosting this endpoint. This can be used to determine endpoints local to a Node.
endpoints.targetRef
(
)
targetRef is a reference to a Kubernetes object that represents this endpoint.
endpoints.zone
(string)
zone is the name of the Zone this endpoint exists in.
ports
ports specifies the list of network ports exposed by each endpoint in this slice. Each port must have a unique name. When ports is empty, it indicates that there are no defined ports. When a port is defined with a nil port value, it indicates "all ports". Each slice may include a maximum of 100 ports.
EndpointPort represents a Port used by an EndpointSlice
ports.port
(int32)
The port number of the endpoint. If this is not specified, ports are not restricted and must be interpreted in the context of the specific consumer.
ports.protocol
(string)
ports.name
(string)
The name of this port. All ports in an EndpointSlice must have a unique name. If the EndpointSlice is dervied from a Kubernetes service, this corresponds to the Service.ports[].name. Name must either be an empty string or pass DNS_LABEL validation: * must be no more than 63 characters long. * must consist of lower case alphanumeric characters or '-'. * must start and end with an alphanumeric character. Default is empty string.
ports.appProtocol
(string)
EndpointSliceList
EndpointSliceList represents a list of endpoint slices
kind
: EndpointSliceList
metadata
(
ListMeta
)
Standard list metadata.
items
([]
List of endpoint slices
get
read the specified EndpointSlice
GET /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}
name
(
in path
name of the EndpointSlice
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind EndpointSlice
GET /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices
namespace
(
in path
namespac

e
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/discovery.k8s.io/v1/endpointslices
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create an EndpointSlice
POST /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified EndpointSlice
PUT /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified EndpointSlice
PATCH /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete an EndpointSlice
DELETE /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of EndpointSlice
DELETE /apis/discovery.k8s.io/v1/namespaces/{namespace}/endpointslices
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
timeo

utSeconds
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Manage Kubernetes Objects | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operat

or pattern
Tasks
Debug Pods
Determine the Reason

 for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Clus

ter Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v

1)
Tasks
Declarative and imperative paradigms for interacting with the Kubernetes API.
Use kubectl patch to update Kubernetes API objects in place. Do a strategic merge patch or a JSON merge patch.
Feedback
Yes
No
.
or
.
Last modified June 14, 2020 at 8:35 PM PST:
Add descriptions for tasks (74360fa8ee)
Home
Blog
Training
Partners
Community
CC BY 4.0

Monitoring, Logging, and Debugging | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Set up monitoring and logging to troubleshoot a cluster, or debug a containerized application.
1:
1.1:
Debug Pods
1.2:
1.3:
1.4:
1.5:
1.6:
1.7:
2:
2.1:
2.2:
2.3:
2.4:
2.5:
Auditing
2.6:
2.7:
2.8:
Sometimes things go wrong. This guide is aimed at making them right. It has
two sections:
Debugging your application
- Useful
for users who are deploying code into Kubernetes and wondering why it is not working.
Debugging your cluster
- Useful
for cluster administrators and people whose Kubernetes cluster is unhappy.
You should also check the known issues for the
release
you're using.
Getting help
If your problem isn't answered by any of the guides above, there are variety of
ways for you to get help from the Kubernetes community.
Questions
The documentation on this site has been structured to provide answers to a wide
range of questions.
Concepts
explain the Kubernetes
architecture and how each component works, while
Setup
provides
practical instructions for getting started.
Tasks
show how to
accomplish commonly used tasks, and
Tutorials
are more
comprehensive walkthroughs of real-world, industry-specific, or end-to-end
development scenarios. The
Reference
section provides
detailed documentation on the
and command-line interfaces (CLIs), such as
kubectl
.
Help! My question isn't covered! I need help now!
Someone else from the community may have already asked a similar question or may
be able to help with your problem. The Kubernetes team will also monitor
posts tagged Kubernetes
.
If there aren't any existing questions that help,
please
ensure that your question is on-topic on Stack Overflow
and that you read through the guidance on
how to ask a new question
,
before
asking a new one
!
Slack
Many people from the Kubernetes community hang out on Kubernetes Slack in the
#kubernetes-users
channel.
Slack requires registration; you can
request an invitation
,
and registration is open to everyone). Feel free to come and ask any and all questions.
Once registered, access the
Kubernetes organisation in Slack
via your web browser or via Slack's own dedicated app.
Once you are registered, browse the growing list of channels for various subjects of
interest. For example, people new to Kubernetes may also want to join the
#kubernetes-novice
channel. As another example, developers should join the
#kubernetes-dev
channel.
There are also many country specific / local language channels. Feel free to join
these channels for localized support and info:
Country / language specific Slack channels
Country
Channels
China
#cn-users
,
#cn-events
Finland
#fi-users
France
#fr-users
,
#fr-events
Germany
#de-users
,
#de-events
India
#in-users
,
#in-events
Italy
#it-users
,
#it-events
Japan
#jp-users
,
#jp-events
Korea
#kr-users
Netherlands
#nl-users
Norway
#norw-users
Poland
#pl-users
Russia
#ru-users
Spain
#es-users
Sweden
#se-users
Turkey
#tr-users
,
#tr-events
Forum
You're welcome to join the official Kubernetes Forum:
discuss.kubernetes.io
.
Bugs and feature requests
If you have what looks like a bug, or you would like to make a feature request,
please use the
GitHub issue tracking system
.
Before you fil

e an issue, please search existing issues to see if your issue is
already covered.
If filing a bug, please include detailed information about how to reproduce the
problem, such as:
Kubernetes version:
Cloud provider, OS distro, network configuration, and container runtime version
Steps to reproduce the problem
1 - Troubleshooting Applications
Debugging common containerized application issues.
This doc contains a set of resources for fixing issues with containerized applications. It covers things like common issues with Kubernetes resources (like Pods, Services, or StatefulSets), advice on making sense of container termination messages, and ways to debug running containers.
1.1 - Debug Pods
This guide is to help users debug applications that are deployed into Kubernetes and not behaving correctly.
This is
not
a guide for people who want to debug their cluster. For that you should check out
this guide
.
Diagnosing the problem
The first step in troubleshooting is triage. What is the problem? Is it your Pods, your Replication Controller or
your Service?
Debugging Pods
Debugging Replication Controllers
Debugging Services
The first step in debugging a Pod is taking a look at it. Check the current state of the Pod and recent events with the following command:
kubectl describe pods
${
POD_NAME
}
Look at the state of the containers in the pod. Are they all
Running
? Have there been recent restarts?
Continue debugging depending on the state of the pods.
My pod stays pending
If a Pod is stuck in
Pending
it means that it can not be scheduled onto a node. Generally this is because
there are insufficient resources of one type or another that prevent scheduling. Look at the output of the
kubectl describe ...
command above. There should be messages from the scheduler about why it can not schedule
your pod. Reasons include:
You don't have enough resources
: You may have exhausted the supply of CPU or Memory in your cluster, in this case
you need to delete Pods, adjust resource requests, or add new nodes to your cluster. See
Compute Resources document
You are using
hostPort
: When you bind a Pod to a
hostPort
there are a limited number of places that pod can be
scheduled. In most cases,
hostPort
is unnecessary, try using a Service object to expose your Pod. If you do require
hostPort
then you can only schedule as many Pods as there are nodes in your Kubernetes cluster.
My pod stays waiting
If a Pod is stuck in the
Waiting
state, then it has been scheduled to a worker node, but it can't run on that machine.
Again, the information from
should be informative. The most common cause of
Waiting
pods is a failure to pull the image. There are three things to check:
Make sure that you have the name of the image correct.
Have you pushed the image to the registry?
Try to manually pull the image to see if the image can be pulled. For example,
if you use Docker on your PC, run
docker pull &lt;image&gt;
.
My pod is crashing or otherwise unhealthy
Once your pod has been scheduled, the methods described in
are available for debugging.
My pod is running but not doing what I told it to do
If your pod is not behaving as you expected, it may be that there was an error in your
pod description (e.g.
mypod.yaml
file on your local machine), and that the error
was silently ignored when you created the pod. Often a section of the pod description
is nested incorrectly, or a key name is typed incorrectly, and so the key is ignored.
For example, if you misspelled
command
as
commnd
then the pod will be created but
will not use the command line you intended it to use.
The first thing to do is to delete your pod and try creating it again with the
--validate
option.
For example, run
kubectl apply --validate -f mypod.yaml
.
If you misspelled
command
as
commnd
then will give an error like this:
I0805 10:43:25.129850
46757
schema.go:126
]
unknown field: commnd
I0805 10:43:25.129973
46757

schema.go:129
]
this may be a
false
alarm, see https://github.com/kubernetes/kubernetes/issues/6842
pods/mypod
The next thing to check is whether the pod on the apiserver
matches the pod you meant to create (e.g. in a yaml file on your local machine).
kubectl get pods/mypod -o yaml &gt; mypod-on-apiserver.yaml
and then
manually compare the original pod description,
with the one you got
back from apiserver,
mypod-on-apiserver.yaml
. There will typically be some
lines on the "apiserver" version that are not on the original version. This is
expected. However, if there are lines on the original that are not on the apiserver
version, then this may indicate a problem with your pod spec.
Replication controllers are fairly straightforward. They can either create Pods or they can't. If they can't
create pods, then please refer to the
instructions above
to debug your pods.
You can also use
kubectl describe rc ${CONTROLLER_NAME}
to introspect events related to the replication
controller.
Services provide load balancing across a set of pods. There are several common problems that can make Services
not work properly. The following instructions should help debug Service problems.
First, verify that there are endpoints for the service. For every Service object, the apiserver makes an
endpoints
resource available.
You can view this resource with:
${
SERVICE_NAME
}
Make sure that the endpoints match up with the number of pods that you expect to be members of your service.
For example, if your Service is for an nginx container with 3 replicas, you would expect to see three different
IP addresses in the Service's endpoints.
My service is missing endpoints
If you are missing endpoints, try listing pods using the labels that Service uses. Imagine that you have
a Service where the labels are:
...
spec
:
-
selector
:
name
:
nginx
type
:
frontend
You can use:
kubectl get pods --selector
=
name
=
nginx,type
=
frontend
to list pods that match this selector. Verify that the list matches the Pods that you expect to provide your Service.
Verify that the pod's
matches up with the Service's
Network traffic is not forwarded
Please see
debugging service
If none of the above solves your problem, follow the instructions in
Debugging Service document
to make sure that your
Service
is running, has
Endpoints
, and your
Pods
are
actually serving; you have DNS working, iptables rules installed, and kube-proxy
does not seem to be misbehaving.
You may also visit
troubleshooting document
1.2 - Debug Services
An issue that comes up rather frequently for new installations of Kubernetes is
that a Service is not working properly. You've run your Pods through a
Deployment (or other workload controller) and created a Service, but you
get no response when you try to access it. This document will hopefully help
you to figure out what's going wrong.
Running commands in a Pod
For many steps here you will want to see what a Pod running in the cluster
sees. The simplest way to do this is to run an interactive busybox Pod:
kubectl run -it --rm --restart=Never busybox --image=gcr.io/google-containers/busybox sh
Note:
If you don't see a command prompt, try pressing enter.
If you already have a running Pod that you prefer to use, you can run a
command in it using:
kubectl
exec
&lt;POD-NAME&gt; -c &lt;CONTAINER-NAME&gt; -- &lt;COMMAND&gt;
Setup
For the purposes of this walk-through, let's run some Pods. Since you're
probably debugging your own Service you can substitute your own details, or you
can follow along and get a second data point.
kubectl create deployment hostnames --image
=
deployment.apps/hostnames created
kubectl
commands will print the type and name of the resource created or mutated, which can then be used in subsequent commands.
Let's scale the deployment to 

3 replicas.
kubectl scale deployment hostnames --replicas
=
3
deployment.apps/hostnames scaled
Note that this is the same as if you had started the Deployment with the following
YAML:
:
apps/v1
kind
:
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
:
app
:
hostnames
replicas
:
3
template
:
metadata
:
labels
:
app
:
hostnames
spec
:
:
-
name
:
hostnames
image
:
The label "app" is automatically set by
kubectl create deployment
to the name of the
You can confirm your Pods are running:
app
=
hostnames
hostnames-632524106-bbpiw   1/1       Running   0          2m
hostnames-632524106-ly40y   1/1       Running   0          2m
hostnames-632524106-tlaok   1/1       Running   0          2m
You can also confirm that your Pods are serving. You can get the list of
Pod IP addresses and test them directly.
app
=
hostnames
\
-o go-template
=
'{{range .items}}{{.status.podIP}}{{"\n"}}{{end}}'
10.244.0.5
10.244.0.6
10.244.0.7
The example container used for this walk-through serves its own hostname
via HTTP on port 9376, but if you are debugging your own app, you'll want to
use whatever port number your Pods are listening on.
From within a pod:
for
ep in 10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376;
do
wget -qO-
$ep
done
This should produce something like:
hostnames-632524106-bbpiw
hostnames-632524106-ly40y
hostnames-632524106-tlaok
If you are not getting the responses you expect at this point, your Pods
might not be healthy or might not be listening on the port you think they are.
You might find
to be useful for seeing what is happening, or
perhaps you need to
directly into your Pods and debug from
there.
Assuming everything has gone to plan so far, you can start to investigate why
your Service doesn't work.
Does the Service exist?
The astute reader will have noticed that you did not actually create a Service
yet - that is intentional. This is a step that sometimes gets forgotten, and
is the first thing to check.
What would happen if you tried to access a non-existent Service? If
you have another Pod that consumes this Service by name you would get
something like:
wget -O- hostnames
Resolving hostnames (hostnames)... failed: Name or service not known.
wget: unable to resolve host address 'hostnames'
The first thing to check is whether that Service actually exists:
kubectl get svc hostnames
No resources found.
Error from server (NotFound): services "hostnames" not found
Let's create the Service. As before, this is for the walk-through - you can
use your own Service's details here.
kubectl expose deployment hostnames --port
=
80
--target-port
=
9376
service/hostnames exposed
And read it back:
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
hostnames   ClusterIP   10.0.1.175   &lt;none&gt;        80/TCP    5s
Now you know that the Service exists.
As before, this is the same as if you had started the Service with YAML:
:
v1
kind
:
Service
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
app
:
hostnames
ports
:
-
name
:
default
protocol
:
TCP
port
:
80
:
9376
In order to highlight the full range of configuration, the Service you created
here uses a different port number than the Pods. For many real-world
Services, these values might be the same.
Any Network Policy Ingress rules affecting the target Pods?
If you have deployed any Network Policy Ingress rules which may affect incoming
traffic to
hostnames-*
Pods, these need to be reviewed.
Does the Service work by DNS name?
One of the most common ways that clients consume a Service is through a DNS
name.
From a Pod in the same Namespace:
nslookup hostnames

Name:      

hostnames
Address 1: 10.0.1.175 hostnames.default.svc.cluster.local
If this fails, perhaps your Pod and Service are in different
Namespaces, try a namespace-qualified name (again, from within a Pod):
nslookup hostnames.default

Name:      hostnames.default
If this works, you'll need to adjust your app to use a cross-namespace name, or
run your app and Service in the same Namespace. If this still fails, try a
fully-qualified name:
nslookup hostnames.default.svc.cluster.local

Name:      hostnames.default.svc.cluster.local
Note the suffix here: "default.svc.cluster.local". The "default" is the
Namespace you're operating in. The "svc" denotes that this is a Service.
The "cluster.local" is your cluster domain, which COULD be different in your
own cluster.
You can also try this from a Node in the cluster:
Note:
10.0.0.10 is the cluster's DNS Service IP, yours might be different.
nslookup hostnames.default.svc.cluster.local 10.0.0.10
Server:         10.0.0.10
Address:        10.0.0.10#53

Name:   hostnames.default.svc.cluster.local
Address: 10.0.1.175
If you are able to do a fully-qualified name lookup but not a relative one, you
need to check that your
file in your Pod is correct. From
within a Pod:
cat /etc/resolv.conf
You should see something like:
search default.svc.cluster.local svc.cluster.local cluster.local example.com
The
line must indicate your cluster's DNS Service. This is
passed into
kubelet
with the
flag.
The
search
line must include an appropriate suffix for you to find the
Service name. In this case it is looking for Services in the local
Namespace ("default.svc.cluster.local"), Services in all Namespaces
("svc.cluster.local"), and lastly for names in the cluster ("cluster.local").
Depending on your own install you might have additional records after that (up
to 6 total). The cluster suffix is passed into
kubelet
with the
--cluster-domain
flag. Throughout this document, the cluster suffix is
assumed to be "cluster.local". Your own clusters might be configured
differently, in which case you should change that in all of the previous
commands.
The
options
line must set
ndots
high enough that your DNS client library
considers search paths at all. Kubernetes sets this to 5 by default, which is
high enough to cover all of the DNS names it generates.
Does any Service work by DNS name?
If the above still fails, DNS lookups are not working for your Service. You
can take a step back and see what else is not working. The Kubernetes master
Service should always work. From within a Pod:
nslookup kubernetes.default

Address 1: 10.0.0.1 kubernetes.default.svc.cluster.local
If this fails, please see the
section
of this document, or even go back to the top of this document and start over,
but instead of debugging your own Service, debug the DNS Service.
Does the Service work by IP?
Assuming you have confirmed that DNS works, the next thing to test is whether your
Service works by its IP address. From a Pod in your cluster, access the
Service's IP (from
above).
for
i in
$(
seq
1
3
)
;
do
wget -qO- 10.0.1.175:80
done
If your Service is working, you should get correct responses. If not, there
are a number of things that could be going wrong. Read on.
Is the Service defined correctly?
It might sound silly, but you should really double and triple check that your
Service is correct and matches your Pod's port. Read back your Service
and verify it:
kubectl get service hostnames -o json
{

"kind"
:
"Service"
,
"apiVersion"
:
"v1"
,
: {
"name"
:
"hostnames"
,
:
"default"
,
"uid"
:
"428c8b6c-24bc-11e5-936d-42010af0a9bc"
,
"resourceVersion"
:
"347189"
,
"creationTimestamp"
:
"2015-07-07T15:24:29Z"
,
"labels"
: {
"app"
:
}
},
"spec"
: {
"ports"
: [
{
"name"
:
"default"
,
"protocol"
:
"TCP"
,
"port"
:
80
,
"targetPort"
:
9376
,
"nodePort"
:
0
}
],
"selector"
: {
"app"
:
},
"clusterIP"
:
"10.0.1.175"
,
"type"
:
"ClusterIP"
,
"sessionAffinity"
:
"None"
},
"status"
: {
"loadBalancer"
: {}
}
}
Is the Service port you are trying to access listed in
spec.ports[]
?
Is the
correct for your Pods (some Pods use a different port than the Service)?
If you meant to use a numeric port, is it a number (9376) or a string "9376"?
If you meant to use a named port, do your Pods expose a port with the same name?
Is the port's
protocol
correct for your Pods?
Does the Service have any Endpoints?
If you got this far, you have confirmed that your Service is correctly
defined and is resolved by DNS. Now let's check that the Pods you ran are
actually being selected by the Service.
Earlier you saw that the Pods were running. You can re-check that:
app
=
hostnames
hostnames-632524106-bbpiw   1/1       Running   0          1h
hostnames-632524106-ly40y   1/1       Running   0          1h
hostnames-632524106-tlaok   1/1       Running   0          1h
The
-l app=hostnames
argument is a label selector configured on the Service.
The "AGE" column says that these Pods are about an hour old, which implies that
they are running fine and not crashing.
The "RESTARTS" column says that these pods are not crashing frequently or being
restarted. Frequent restarts could lead to intermittent connectivity issues.
If the restart count is high, read more about how to
debug pods
.
Inside the Kubernetes system is a control loop which evaluates the selector of
every Service and saves the results into a corresponding Endpoints object.
kubectl get endpoints hostnames
NAME        ENDPOINTS
hostnames   10.244.0.5:9376,10.244.0.6:9376,10.244.0.7:9376
This confirms that the endpoints controller has found the correct Pods for
your Service. If the
ENDPOINTS
column is
&lt;none&gt;
, you should check that
the
spec.selector
field of your Service actually selects for
metadata.labels
values on your Pods. A common mistake is to have a typo or
other error, such as the Service selecting for
app=hostnames
, but the
Deployment specifying
run=hostnames
, as in versions previous to 1.18, where
the
kubectl run
command could have been also used to create a Deployment.
Are the Pods working?
At this point, you know that your Service exists and has selected your Pods.
At the beginning of this walk-through, you verified the Pods themselves.
Let's check again that the Pods are actually working - you can bypass the
Service mechanism and go straight to the Pods, as listed by the Endpoints
above.
Note:
These commands use the Pod port (9376), rather than the Service port (80).
From within a Pod:
for
do
wget -qO-
$ep
done
You expect each Pod in the Endpoints list to return its own hostname. If
this is not what happens (or whatever the correct behavior is for your own
Pods), you should investigate what's happening there.
Is the kube-proxy working?
If you get here, your Service is running, has Endpoints, and your Pods
are actually serving. At this point, the whole Service proxy mechanism is
suspect. Let's confirm it, piece by piece.
The default implementation of Services, and the one used on most clusters, is
kube-proxy. This is a program that runs on every node and configures one of a
small set of mechanisms for providing the Service abstraction. If your
cluster does not use kube-proxy, the follow

ing sections will not apply, and you
will have to investigate whatever implementation of Services you are using.
Is kube-proxy running?
Confirm that
is running on your Nodes. Running directly on a
Node, you should get something like the below:
ps auxw | grep kube-proxy
root  4194  0.4  0.1 101864 17696 ?    Sl Jul04  25:43 /usr/local/bin/kube-proxy --master=https://kubernetes-master --kubeconfig=/var/lib/kube-proxy/kubeconfig --v=2
Next, confirm that it is not failing something obvious, like contacting the
master. To do this, you'll have to look at the logs. Accessing the logs
depends on your Node OS. On some OSes it is a file, such as
/var/log/kube-proxy.log, while other OSes use
to access logs. You
should see something like:
I1027 22:14:53.995134    5063 server.go:200] Running in resource-only container "/kube-proxy"
I1027 22:14:53.998163    5063 server.go:247] Using iptables Proxier.
I1027 22:14:54.038140    5063 proxier.go:352] Setting endpoints for "kube-system/kube-dns:dns-tcp" to [10.244.1.3:53]
I1027 22:14:54.038164    5063 proxier.go:352] Setting endpoints for "kube-system/kube-dns:dns" to [10.244.1.3:53]
I1027 22:14:54.038209    5063 proxier.go:352] Setting endpoints for "default/kubernetes:https" to [10.240.0.2:443]
I1027 22:14:54.038238    5063 proxier.go:429] Not syncing iptables until Services and Endpoints have been received from master
I1027 22:14:54.040048    5063 proxier.go:294] Adding new service "default/kubernetes:https" at 10.0.0.1:443/TCP
I1027 22:14:54.040154    5063 proxier.go:294] Adding new service "kube-system/kube-dns:dns" at 10.0.0.10:53/UDP
I1027 22:14:54.040223    5063 proxier.go:294] Adding new service "kube-system/kube-dns:dns-tcp" at 10.0.0.10:53/TCP
If you see error messages about not being able to contact the master, you
should double-check your Node configuration and installation steps.
One of the possible reasons that
cannot run correctly is that the
required
conntrack
binary cannot be found. This may happen on some Linux
systems, depending on how you are installing the cluster, for example, you are
installing Kubernetes from scratch. If this is the case, you need to manually
install the
conntrack
package (e.g.
sudo apt install conntrack
on Ubuntu)
and then retry.
Kube-proxy can run in one of a few modes. In the log listed above, the
line
Using iptables Proxier
indicates that kube-proxy is running in
"iptables" mode. The most common other mode is "ipvs".
Iptables mode
In "iptables" mode, you should see something like the following on a Node:
iptables-save | grep hostnames
-A KUBE-SEP-57KPRZ3JQVENLNBR -s 10.244.3.6/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000
-A KUBE-SEP-57KPRZ3JQVENLNBR -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.3.6:9376
-A KUBE-SEP-WNBA2IHDGP2BOBGZ -s 10.244.1.7/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000
-A KUBE-SEP-WNBA2IHDGP2BOBGZ -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.1.7:9376
-A KUBE-SEP-X3P2623AGDH6CDF3 -s 10.244.2.3/32 -m comment --comment "default/hostnames:" -j MARK --set-xmark 0x00004000/0x00004000
-A KUBE-SEP-X3P2623AGDH6CDF3 -p tcp -m comment --comment "default/hostnames:" -m tcp -j DNAT --to-destination 10.244.2.3:9376
-A KUBE-SERVICES -d 10.0.1.175/32 -p tcp -m comment --comment "default/hostnames: cluster IP" -m tcp --dport 80 -j KUBE-SVC-NWV5X2332I4OT4T3
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-WNBA2IHDGP2BOBGZ
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-X3P2623AGDH6CDF3
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -j KUBE-SEP-57KPRZ3JQVENLNBR
For each port of each Service, there should be 1 rule in
KUBE-SERVICES
and
one
KUBE-SVC-&lt;

hash&gt;
chain. For each Pod endpoint, there should be a small
number of rules in that
KUBE-SVC-&lt;hash&gt;
and one
KUBE-SEP-&lt;hash&gt;
chain with
a small number of rules in it. The exact rules will vary based on your exact
config (including node-ports and load-balancers).
IPVS mode
In "ipvs" mode, you should see something like the following on a Node:
ipvsadm -ln
Prot LocalAddress:Port Scheduler Flags
  -&gt; RemoteAddress:Port           Forward Weight ActiveConn InActConn
...
TCP  10.0.1.175:80 rr
  -&gt; 10.244.0.5:9376               Masq    1      0          0
  -&gt; 10.244.0.6:9376               Masq    1      0          0
  -&gt; 10.244.0.7:9376               Masq    1      0          0
...
For each port of each Service, plus any NodePorts, external IPs, and
load-balancer IPs, kube-proxy will create a virtual server. For each Pod
endpoint, it will create corresponding real servers. In this example, service
hostnames(
10.0.1.175:80
) has 3 endpoints(
10.244.0.5:9376
,
10.244.0.6:9376
,
10.244.0.7:9376
).
Is kube-proxy proxying?
Assuming you do see one the above cases, try again to access your Service by
IP from one of your Nodes:
curl 10.0.1.175:80
If this still fails, look at the
logs for specific lines like:
Setting endpoints for default/hostnames:default to [10.244.0.5:9376 10.244.0.6:9376 10.244.0.7:9376]
If you don't see those, try restarting
with the
-v
flag set to 4, and
then look at the logs again.
Edge case: A Pod fails to reach itself via the Service IP
This might sound unlikely, but it does happen and it is supposed to work.
This can happen when the network is not properly configured for "hairpin"
traffic, usually when
is running in
iptables
mode and Pods
are connected with bridge network. The
Kubelet
exposes a
hairpin-mode
flag
that allows endpoints of a Service to loadbalance
back to themselves if they try to access their own Service VIP. The
flag must either be set to
hairpin-veth
or
promiscuous-bridge
.
The common steps to trouble shoot this are as follows:
Confirm
is set to
or
.
You should see something like the below.
is set to
in the following example.
ps auxw | grep kubelet
root      3392  1.1  0.8 186804 65208 ?        Sl   00:51  11:11 /usr/local/bin/kubelet --enable-debugging-handlers=true --config=/etc/kubernetes/manifests --allow-privileged=True --v=4 --cluster-dns=10.0.0.10 --cluster-domain=cluster.local --configure-cbr0=true --cgroup-root=/ --system-cgroups=/system --hairpin-mode=promiscuous-bridge --runtime-cgroups=/docker-daemon --kubelet-cgroups=/kubelet --babysit-daemons=true --max-pods=110 --serialize-image-pulls=false --outofdisk-transition-frequency=0
Confirm the effective
. To do this, you'll have to look at
kubelet log. Accessing the logs depends on your Node OS. On some OSes it
is a file, such as /var/log/kubelet.log, while other OSes use
to access logs. Please be noted that the effective hairpin mode may not
match
--hairpin-mode
flag due to compatibility. Check if there is any log
lines with key word
hairpin
in kubelet.log. There should be log lines
indicating the effective hairpin mode, like something below.
I0629 00:51:43.648698    3252 kubelet.go:380] Hairpin mode set to "promiscuous-bridge"
If the effective hairpin mode is
, ensure the
Kubelet
has
the permission to operate in
/sys
on node. If everything works properly,
you should see something like:
for
intf in /sys/devices/virtual/net/cbr0/brif/*;
do
cat
$intf
/hairpin_mode;
done
1
1
1
1
, ensure
Kubelet
has the permission to manipulate linux bridge on node. If
cbr0
bridge is
used and configured properly, you should see:
ifconfig cbr0 |grep PROMISC
UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1460  Metric:1
Seek help if none of above works out.
Seek help
If you get this far, something very stra

nge is happening. Your Service is
running, has Endpoints, and your Pods are actually serving. You have DNS
working, and
does not seem to be misbehaving. And yet your
Service is not working. Please let us know what is going on, so we can help
investigate!
Contact us on
Slack
or
Forum
or
GitHub
.
Visit the
troubleshooting overview document
1.3 - Debug a StatefulSet
This task shows you how to debug a StatefulSet.
You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster.
You should have a StatefulSet running that you want to investigate.
Debugging a StatefulSet
In order to list all the pods which belong to a StatefulSet, which have a label
app.kubernetes.io/name=MyApp
set on them,
you can use the following:
kubectl get pods -l app.kubernetes.io/name
=
MyApp
If you find that any Pods listed are in
Unknown
or
state for an extended period of time,
refer to the
Deleting StatefulSet Pods
task for
instructions on how to deal with them.
You can debug individual Pods in a StatefulSet using the
guide.
debugging an init-container
.
1.4 - Determine the Reason for Pod Failure
This page shows how to write and read a Container termination message.
Termination messages provide a way for containers to write
information about fatal events to a location where it can
be easily retrieved and surfaced by tools like dashboards
and monitoring software. In most cases, information that you
put in a termination message should also be written to
the general
Kubernetes logs
.
minikube
Writing and reading a termination message
In this exercise, you create a Pod that runs one container.
The manifest for that Pod specifies a command that runs when the container starts:
debug/termination.yaml
:
v1
kind
:
Pod
metadata
:
name
:
termination-demo
spec
:
:
-
name
:
termination-demo-container
image
:
debian
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
"sleep 10 &amp;&amp; echo Sleep expired &gt; /dev/termination-log"
]
Create a Pod based on the YAML configuration file:
kubectl apply -f https://k8s.io/examples/debug/termination.yaml
In the YAML file, in the
command
and
args
fields, you can see that the
container sleeps for 10 seconds and then writes "Sleep expired" to
the
/dev/termination-log
file. After the container writes
the "Sleep expired" message, it terminates.
Display information about the Pod:
kubectl get pod termination-demo
Repeat the preceding command until the Pod is no longer running.
Display detailed information about the Pod:
kubectl get pod termination-demo --output
=
yaml
The output includes the "Sleep expired" message:
:
v1
kind
:
Pod
...
lastState
:
terminated
:
containerID
:
...
exitCode
:
0
finishedAt
:
...
message
:
|
Sleep expired
...
Use a Go template to filter the output so that it includes only the termination message:
kubectl get pod termination-demo -o go-template
=
"{{range .status.containerStatuses}}{{.lastState.terminated.message}}{{end}}"
If you are running a multi-container Pod, you can use a Go template to include the container's name.
By doing so, you can discover which of the containers is failing:
kubectl get pod multi-container-pod -o go-template
=
'{{range .status.containerStatuses}}{{printf "%s:\n%s\n\n" .name .lastState.terminated.message}}{{end}}'
Customizing the termination message
Kubernetes retrieves termination messages from the termination message file
specified in the
terminationMessagePath
field of a Cont

ainer, which has a default
value of
. By customizing this field, you can tell Kubernetes
to use a different file. Kubernetes use the contents from the specified file to
populate the Container's status message on both success and failure.
The termination message is intended to be brief final status, such as an assertion failure message.
The kubelet truncates messages that are longer than 4096 bytes.
The total message length across all containers is limited to 12KiB, divided equally among each container.
For example, if there are 12 containers (
initContainers
or
), each has 1024 bytes of available termination message space.
The default termination message path is
.
You cannot set the termination message path after a Pod is launched.
In the following example, the container writes termination messages to
/tmp/my-log
for Kubernetes to retrieve:
:
v1
kind
:
Pod
metadata
:
name
:
msg-path-demo
spec
:
:
-
name
:
msg-path-demo-container
image
:
debian
:
"/tmp/my-log"
Moreover, users can set the
terminationMessagePolicy
field of a Container for
further customization. This field defaults to "
File
" which means the termination
messages are retrieved only from the termination message file. By setting the
to "
FallbackToLogsOnError
", you can tell Kubernetes
to use the last chunk of container log output if the termination message file
is empty and the container exited with an error. The log output is limited to
2048 bytes or 80 lines, whichever is smaller.
See the
field in
Container
.
retrieving logs
.
Go templates
.
1.5 - Debug Init Containers
This page shows how to investigate problems related to the execution of
Init Containers. The example command lines below refer to the Pod as
&lt;pod-name&gt;
and the Init Containers as
&lt;init-container-1&gt;
and
&lt;init-container-2&gt;
.
minikube
.
You should be familiar with the basics of
.
You should have
Configured an Init Container
.
Checking the status of Init Containers
Display the status of your pod:
kubectl get pod &lt;pod-name&gt;
For example, a status of
Init:1/2
indicates that one of two Init Containers
has completed successfully:
NAME         READY     STATUS     RESTARTS   AGE
&lt;pod-name&gt;   0/1       Init:1/2   0          7s
See
Understanding Pod status
for more examples of
status values and their meanings.
Getting details about Init Containers
View more detailed information about Init Container execution:
kubectl describe pod &lt;pod-name&gt;
For example, a Pod with two Init Containers might show the following:
Init Containers:
  &lt;init-container-1&gt;:
    Container ID:    ...
    ...
    State:           Terminated
      Reason:        Completed
      Exit Code:     0
      Started:       ...
      Finished:      ...
    Ready:           True
    Restart Count:   0
    ...
  &lt;init-container-2&gt;:
    ...
    State:           Waiting
      Reason:        CrashLoopBackOff
    Last State:      Terminated
      Reason:        Error
      Exit Code:     1
    Ready:           False
    Restart Count:   3
    ...
You can also access the Init Container statuses programmatically by reading the
status.initContainerStatuses
field on the Pod Spec:
kubectl get pod nginx --template
'{{.status.initContainerStatuses}}'
This command will return the same information as 

above in raw JSON.
Accessing logs from Init Containers
Pass the Init Container name along with the Pod name
to access its logs.
kubectl logs &lt;pod-name&gt; -c &lt;init-container-2&gt;
Init Containers that run a shell script print
commands as they're executed. For example, you can do this in Bash by running
set -x
at the beginning of the script.
A Pod status beginning with
Init:
summarizes the status of Init Container
execution. The table below describes some example status values that you might
see while debugging Init Containers.
Status
Meaning
Init:N/M
The Pod has
M
Init Containers, and
N
have completed so far.
Init:Error
An Init Container has failed to execute.
Init:CrashLoopBackOff
An Init Container has failed repeatedly.
Pending
The Pod has not yet begun executing Init Containers.
PodInitializing
or
Running
The Pod has already finished executing Init Containers.
1.6 - Debug Running Pods
This page explains how to debug Pods running (or crashing) on a Node.
Your
Pod
should already be
scheduled and running. If your Pod is not yet running, start with
Debugging
Pods
.
For some of the advanced debugging steps you need to know on which Node the
Pod is running and have shell access to run commands on that Node. You don't
need that access to run the standard debug steps that use
kubectl
.
Using
kubectl describe pod
to fetch details about pods
For this example we'll use a Deployment to create two pods, similar to the earlier example.
application/nginx-with-request.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"128Mi"
cpu
:
"500m"
ports
:
-
:
80
Create deployment by running following command:
kubectl apply -f https://k8s.io/examples/application/nginx-with-request.yaml
Check pod status by following command:
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-67d4bdd6f5-cx2nz   1/1     Running   0          13s
nginx-deployment-67d4bdd6f5-w6kd7   1/1     Running   0          13s
We can retrieve a lot more information about each of these pods using
kubectl describe pod nginx-deployment-67d4bdd6f5-w6kd7
Name:         nginx-deployment-67d4bdd6f5-w6kd7
Priority:     0
Node:         kube-worker-1/192.168.0.113
Start Time:   Thu, 17 Feb 2022 16:51:01 -0500
Labels:       app=nginx
              pod-template-hash=67d4bdd6f5
Status:       Running
IP:           10.88.0.3
IPs:
  IP:           10.88.0.3
  IP:           2001:db8::1
Controlled By:  ReplicaSet/nginx-deployment-67d4bdd6f5
  nginx:
    Container ID:   containerd://5403af59a2b46ee5a23fb0ae4b1e077f7ca5c5fb7af16e1ab21c00e0e616462a
    Image:          nginx
    Image ID:       docker.io/library/nginx@sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 17 Feb 2022 16:51:05 -0500
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:        500m
      memory:     128Mi
    Environment:  &lt;none&gt;
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-bgsgp (ro)
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-bgsgp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       &lt;nil&gt;
    DownwardAPI:             true
QoS Class:                   Gua

ranteed
Node-Selectors:              &lt;none&gt;
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  34s   default-scheduler  Successfully assigned default/nginx-deployment-67d4bdd6f5-w6kd7 to kube-worker-1
  Normal  Pulling    31s   kubelet            Pulling image "nginx"
  Normal  Pulled     30s   kubelet            Successfully pulled image "nginx" in 1.146417389s
  Normal  Created    30s   kubelet            Created container nginx
  Normal  Started    30s   kubelet            Started container nginx
Here you can see configuration information about the container(s) and Pod (labels, resource requirements, etc.), as well as status information about the container(s) and Pod (state, readiness, restart count, events, etc.).
The container state is one of Waiting, Running, or Terminated. Depending on the state, additional information will be provided -- here you can see that for a container in Running state, the system tells you when the container started.
Ready tells you whether the container passed its last readiness probe. (In this case, the container does not have a readiness probe configured; the container is assumed to be ready if no readiness probe is configured.)
Restart Count tells you how many times the container has been restarted; this information can be useful for detecting crash loops in containers that are configured with a restart policy of 'always.'
Currently the only Condition associated with a Pod is the binary Ready condition, which indicates that the pod is able to service requests and should be added to the load balancing pools of all matching services.
Lastly, you see a log of recent events related to your Pod. The system compresses multiple identical events by indicating the first and last time it was seen and the number of times it was seen. "From" indicates the component that is logging the event, "SubobjectPath" tells you which object (e.g. container within the pod) is being referred to, and "Reason" and "Message" tell you what happened.
Example: debugging Pending Pods
A common scenario that you can detect using events is when you've created a Pod that won't fit on any node. For example, the Pod might request more resources than are free on any node, or it might specify a label selector that doesn't match any nodes. Let's say we created the previous Deployment with 5 replicas (instead of 2) and requesting 600 millicores instead of 500, on a four-node cluster where each (virtual) machine has 1 CPU. In that case one of the Pods will not be able to schedule. (Note that because of the cluster addon pods such as fluentd, skydns, etc., that run on each node, if we requested 1000 millicores then none of the Pods would be able to schedule.)
NAME                                READY     STATUS    RESTARTS   AGE
nginx-deployment-1006230814-6winp   1/1       Running   0          7m
nginx-deployment-1006230814-fmgu3   1/1       Running   0          7m
nginx-deployment-1370807587-6ekbw   1/1       Running   0          1m
nginx-deployment-1370807587-fg172   0/1       Pending   0          1m
nginx-deployment-1370807587-fz9sd   0/1       Pending   0          1m
To find out why the nginx-deployment-1370807587-fz9sd pod is not running, we can use
on the pending Pod and look at its events:
kubectl describe pod nginx-deployment-1370807587-fz9sd
Name:		nginx-deployment-1370807587-fz9sd
  Namespace:	default
  Node:		/
  Labels:		app=nginx,pod-template-hash=1370807587
  Status:		Pending
  IP:
  Controllers:	ReplicaSet/nginx-deployment-1370807587
  Containers:
    nginx:
      Image:	nginx
      Port:	80/TCP
      QoS Tier:
        memory:	Guaranteed
        cpu:	Guaranteed
      Limits:
        cpu:	1
        memory:	128Mi
      R

equests:
        cpu:	1
      Environment Variables:
  Volumes:
    default-token-4bcbi:
      Type:	Secret (a volume populated by a Secret)
      SecretName:	default-token-4bcbi
  Events:
    FirstSeen	LastSeen	Count	From			        SubobjectPath	Type		Reason			    Message
    ---------	--------	-----	----			        -------------	--------	------			    -------
    1m		    48s		    7	    {default-scheduler }			        Warning		FailedScheduling	pod (nginx-deployment-1370807587-fz9sd) failed to fit in any node
  fit failure on node (kubernetes-node-6ta5): Node didn't have enough resource: CPU, requested: 1000, used: 1420, capacity: 2000
  fit failure on node (kubernetes-node-wul5): Node didn't have enough resource: CPU, requested: 1000, used: 1100, capacity: 2000
Here you can see the event generated by the scheduler saying that the Pod failed to schedule for reason
FailedScheduling
(and possibly others). The message tells us that there were not enough resources for the Pod on any of the nodes.
To correct this situation, you can use
to update your Deployment to specify four or fewer replicas. (Or you could leave the one Pod pending, which is harmless.)
Events such as the ones you saw at the end of
are persisted in etcd and provide high-level information on what is happening in the cluster. To list all events you can use
but you have to remember that events are namespaced. This means that if you're interested in events for some namespaced object (e.g. what happened with Pods in namespace
my-namespace
) you need to explicitly provide a namespace to the command:
kubectl get events --namespace
=
To see events from all namespaces, you can use the
--all-namespaces
argument.
, another way to get extra information about a pod (beyond what is provided by
kubectl get pod
) is to pass the
-o yaml
output format flag to
. This will give you, in YAML format, even more information than
--essentially all of the information the system has about the Pod. Here you will see things like annotations (which are key-value metadata without the label restrictions, that is used internally by Kubernetes system components), restart policy, ports, and volumes.
kubectl get pod nginx-deployment-1006230814-6winp -o yaml
:
v1
kind
:
Pod
metadata
:
:
"2022-02-17T21:51:01Z"
generateName
:
nginx-deployment-67d4bdd6f5-
labels
:
app
:
nginx
pod-template-hash
:
67d4bdd6f5
name
:
nginx-deployment-67d4bdd6f5-w6kd7
namespace
:
default
:
-
:
apps/v1
:
true
:
true
kind
:
name
:
nginx-deployment-67d4bdd6f5
uid
:
7d41dfd4-84c0-4be4-88ab-cedbe626ad82
:
"1364"
uid
:
a6501da1-0447-4262-98eb-c03d4002222e
spec
:
:
-
image
:
nginx
:
Always
name
:
nginx
ports
:
-
:
80
protocol
:
TCP
resources
:
limits
:
cpu
:
500m
memory
:
128Mi
requests
:
cpu
:
500m
memory
:
128Mi
:
:
File
:
-
mountPath
:
/var/run/secrets/kubernetes.io/serviceaccount
name
:
kube-api-access-bgsgp
readOnly
:
true
dnsPolicy
:
ClusterFirst
enableServiceLinks
:
true
nodeName
:
kube-worker-1
preemptionPolicy
:
PreemptLowerPriority
priority
:
0
:
Always
schedulerName
:
default-scheduler
:
{}
serviceAccount
:
default
:
default
terminationGracePeriodSeconds
:
30
:
-
effect
:
NoExecute
key
:
node.kubernetes.io/not-ready
operator
:
Exists
tolerationSeconds
:
300
-
effect
:
NoExecute
key
:
node.kubernetes.io/unreachable
operator
:
Exists
:
300
volumes
:
-
name
:
projected
:
defaultMode
:
420
sources
:
-
serviceAccountToken
:
expirationSeconds
:
3607
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
kube-root-ca.crt
-
d

ownwardAPI
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
metadata.namespace
path
:
namespace
status
:
:
-
lastProbeTime
:
null
lastTransitionTime
:
status
:
"True"
type
:
Initialized
-
:
null
:
"2022-02-17T21:51:06Z"
status
:
"True"
type
:
Ready
-
:
null
:
status
:
"True"
type
:
ContainersReady
-
:
null
:
status
:
"True"
type
:
PodScheduled
containerStatuses
:
-
:
containerd://5403af59a2b46ee5a23fb0ae4b1e077f7ca5c5fb7af16e1ab21c00e0e616462a
image
:
docker.io/library/nginx:latest
imageID
:
docker.io/library/nginx@sha256:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
lastState
:
{}
name
:
nginx
ready
:
true
restartCount
:
0
started
:
true
state
:
running
:
startedAt
:
"2022-02-17T21:51:05Z"
hostIP
:
192.168.0.113
phase
:
Running
podIP
:
10.88.0.3
podIPs
:
-
ip
:
10.88.0.3
-
ip
:
2001
:db8::1
qosClass
:
startTime
:
Examining pod logs
First, look at the logs of the affected container:
${
POD_NAME
}
${
CONTAINER_NAME
}
If your container has previously crashed, you can access the previous container's crash log with:
${
POD_NAME
}
${
}
Debugging with container exec
If the
container image
includes
debugging utilities, as is the case with images built from Linux and Windows OS
base images, you can run commands inside a specific container with
:
kubectl
exec
${
POD_NAME
}
-c
${
}
--
${
CMD
}
${
ARG1
}
${
ARG2
}
...
${
ARGN
}
Note:
-c ${CONTAINER_NAME}
is optional. You can omit it for Pods that only contain a single container.
As an example, to look at the logs from a running Cassandra pod, you might run
kubectl
exec
cassandra -- cat /var/log/cassandra/system.log
You can run a shell that's connected to your terminal using the
-i
and
-t
arguments to
, for example:
kubectl
exec
-it cassandra -- sh
.
Debugging with an ephemeral debug container
Kubernetes v1.25 [stable]
Ephemeral containers
are useful for interactive troubleshooting when
is insufficient
because a container has crashed or a container image doesn't include debugging
utilities, such as with
distroless images
.
Example debugging using ephemeral containers
kubectl debug
command to add ephemeral containers to a
running Pod. First, create a pod for the example:
kubectl run ephemeral-demo --image
=
registry.k8s.io/pause:3.1 --restart
=
Never
The examples in this section use the
pause
container image because it does not
contain debugging utilities, but this method works with all container
images.
If you attempt to use
to create a shell you will see an error
because there is no shell in this container image.
kubectl
exec
-it ephemeral-demo -- sh
OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"sh\": executable file not found in $PATH": unknown
You can instead add a debugging container using
. If you
specify the
-i
/
--interactive
argument,
kubectl
will automatically attach
to the console of the Ephemeral Container.
kubectl debug -it ephemeral-demo --image
=
busybox:1.28 --target
=
ephemeral-demo
Defaulting debug container name to debugger-8xzrl.
/ #
This command adds a new busybox container and attaches to it. The
--target
parameter targets the process namespace of another container. It's necessary
here because
does not enable
process namespace sharing
in the pod it
creates.
Note:
The
--target
parameter must be supported by the
Container Runtime
. When not supported,
the Ephemeral Container may not be started, or it may be started with an
isolated process namespace so that
ps
does not reveal processes in

 other
containers.
You can view the state of the newly created ephemeral container using
:
kubectl describe pod ephemeral-demo
...
Ephemeral Containers:
  debugger-8xzrl:
    Container ID:   docker://b888f9adfd15bd5739fefaa39e1df4dd3c617b9902082b1cfdc29c4028ffb2eb
    Image:          busybox
    Image ID:       docker-pullable://busybox@sha256:1828edd60c5efd34b2bf5dd3282ec0cc04d47b2ff9caa0b6d4f07a21d1c08084
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
      Started:      Wed, 12 Feb 2020 14:25:42 +0100
    Ready:          False
    Environment:    &lt;none&gt;
    Mounts:         &lt;none&gt;
...
Use
kubectl delete
to remove the Pod when you're finished:
kubectl delete pod ephemeral-demo
Debugging using a copy of the Pod
Sometimes Pod configuration options make it difficult to troubleshoot in certain
situations. For example, you can't run
to troubleshoot your
container if your container image does not include a shell or if your application
crashes on startup. In these situations you can use
to create a
copy of the Pod with configuration values changed to aid debugging.
Copying a Pod while adding a new container
Adding a new container can be useful when your application is running but not
behaving as you expect and you'd like to add additional troubleshooting
utilities to the Pod.
For example, maybe your application's container images are built on
busybox
but you need debugging utilities not included in
busybox
. You can simulate
this scenario using
:
kubectl run myapp --image
=
busybox:1.28 --restart
=
Never -- sleep 1d
Run this command to create a copy of
myapp
named
myapp-debug
that adds a
new Ubuntu container for debugging:
kubectl debug myapp -it --image
=
ubuntu --share-processes --copy-to
=
Defaulting debug container name to debugger-w7xmf.
root@myapp-debug:/#
Note:
automatically generates a container name if you don't choose
one using the
flag.
The
-i
flag causes
to attach to the new container by
default. You can prevent this by specifying
--attach=false
. If your session
becomes disconnected you can reattach using
kubectl attach
.
The
--share-processes
allows the containers in this Pod to see processes
from the other containers in the Pod. For more information about how this
works, see
.
Don't forget to clean up the debugging Pod when you're finished with it:
kubectl delete pod myapp myapp-debug
Copying a Pod while changing its command
Sometimes it's useful to change the command for a container, for example to
add a debugging flag or because the application is crashing.
To simulate a crashing application, use
to create a container
that immediately exits:
kubectl run --image=busybox:1.28 myapp -- false
You can see using
kubectl describe pod myapp
that this container is crashing:
  myapp:
    Image:         busybox
    ...
    Args:
      false
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
You can use
to create a copy of this Pod with the command
changed to an interactive shell:
kubectl debug myapp -it --copy-to=myapp-debug --container=myapp -- sh
/ #
Now you have an interactive shell that you can use to perform tasks like
checking filesystem paths or running the container command manually.
Note:
To change the command of a specific container you must
specify its name using
or
will instead
create a new container to run the command you specified.
The
-i
to attach to the container by default.
You can prevent this by specifying
. If your session becomes
disconnected you can reattach using
ku

bectl attach
.
Copying a Pod while changing container images
In some situations you may want to change a misbehaving Pod from its normal
production container images to an image containing a debugging build or
additional utilities.
As an example, create a Pod using
:
kubectl run myapp --image=busybox:1.28 --restart=Never -- sleep 1d
Now use
to make a copy and change its container image
to
ubuntu
:
kubectl debug myapp --copy-to=myapp-debug --set-image=*=ubuntu
The syntax of
--set-image
uses the same
container_name=image
syntax as
kubectl set image
.
*=ubuntu
means change the image of all containers
to
ubuntu
.
Debugging via a shell on the node
If none of these approaches work, you can find the Node on which the Pod is
running and create a Pod running on the Node. To create
an interactive shell on a Node using
, run:
kubectl debug node/mynode -it --image
=
ubuntu
Creating debugging pod node-debugger-mynode-pdx84 with container debugger on node mynode.
root@ek8s:/#
When creating a debugging session on a node, keep in mind that:
automatically generates the name of the new Pod based on
the name of the Node.
The root filesystem of the Node will be mounted at
/host
.
The container runs in the host IPC, Network, and PID namespaces, although
the pod isn't privileged, so reading some process information may fail,
and
chroot /host
will fail.
If you need a privileged pod, create it manually.
kubectl delete pod node-debugger-mynode-pdx84
1.7 - Get a Shell to a Running Container
minikube
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
nginx
image
:
nginx
:
-
name
:
mountPath
:
:
true
dnsPolicy
:
Default
kubectl
exec
Note:
--
ls /
ls /
tcpdump
lsof
ps aux
has an
emptyDir
at
.
file in the
echo
# Run this in 

the shell inside your container
file:
exit
.
exit
In an ordinary command window, not your shell, list the environment
kubectl
exec
kubectl
exec
kubectl
exec
kubectl
exec
or
-c
to
named
main-app
and
main-app
kubectl
exec
Note:
-i
and
-t
--stdin
and
--tty
Read about
2 - Troubleshooting Clusters
Debugging common cluster issues.
This doc is about cluster troubleshooting; we assume you have already ruled out your application as the root cause of the
problem you are experiencing. See
the
application troubleshooting guide
for tips on application debugging.
You may also visit the
Listing your cluster
The first thing to debug in your cluster is if your nodes are all registered correctly.
And verify that all of the nodes you expect to see are present and that they are all in the
Ready
state.
To get detailed information about the overall health of your cluster, you can run:
kubectl cluster-info dump
Example: debugging a down/unreachable node
Sometimes when debugging it can be useful to look at the status of a node -- for example, because
you've noticed strange behavior of a Pod that's running on the node, or to find out why a Pod
won't schedule onto the node. As with Pods, you can use
and
kubectl get node -o yaml
to retrieve detailed information about nodes. For example, here's what you'll see if
a node is down (disconnected from the network, or kubelet dies and won't restart, etc.). Notice
the events that show the node is NotReady, and also notice that the pods are no longer running
(they are evicted after five minutes of NotReady status).
NAME                     STATUS       ROLES     AGE     VERSION
kube-worker-1            NotReady     &lt;none&gt;    1h      v1.23.3
kubernetes-node-bols     Ready        &lt;none&gt;    1h      v1.23.3
kubernetes-node-st6x     Ready        &lt;none&gt;    1h      v1.23.3
kubernetes-node-unaj     Ready        &lt;none&gt;    1h      v1.23.3
kubectl describe node kube-worker-1
Name:               kube-worker-1
Roles:              &lt;none&gt;
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kube-worker-1
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 17 Feb 2022 16:46:30 -0500
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  kube-worker-1
  AcquireTime:     &lt;unset&gt;
  RenewTime:       Thu, 17 Feb 2022 17:13:09 -0500
  Type                 Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----                 ------    -------

----------                 ------------------                ------              -------
  NetworkUnavailable   False     Thu, 17 Feb 2022 17:09:13 -0500   Thu, 17 Feb 2022 17:09:13 -0500   WeaveIsUp           Weave pod has set this
  MemoryPressure       Unknown   Thu, 17 Feb 2022 17:12:40 -0500   Thu, 17 Feb 2022 17:13:52 -0500   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure         Unknown   Thu, 17 Feb 2022 17:12:40 -0500   Thu, 17 Feb 2022 17:13:52 -0500   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure          Unknown   Thu, 17 Feb 2022 17:12:40 -0500   Thu, 17 Feb 2022 17:13:52 -0500   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready                Unknown   Thu, 17 Feb 2022 17:12:40 -0500   Thu, 17 Feb 2022 17:13:52 -0500   NodeStatusUnknown   Kubelet stopped posting node status.
Addresses:
  InternalIP:  192.168.0.113
  Hostname:    kube-worker-1
Capacity:
  cpu:                2
  ephemeral-storage:  15372232Ki
  hugepages-2Mi:      0
  memory:             2025188Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  14167048988
  memory:             1922788Ki
  pods:               110
System Info:
  Machine ID:                 9384e2927f544209b5d7b67474bbf92b
  System UUID:                aa829ca9-73d7-064d-9019-df07404ad448
  Boot ID:                    5a295a03-aaca-4340-af20-1327fa5dab5c
  Kernel Version:             5.13.0-28-generic
  OS Image:                   Ubuntu 21.10
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.5.9
  Kubelet Version:            v1.23.3
  Kube-Proxy Version:         v1.23.3
Non-terminated Pods:          (4 in total)
  Namespace                   Name                                 CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                 ------------  ----------  ---------------  -------------  ---
  default                     nginx-deployment-67d4bdd6f5-cx2nz    500m (25%)    500m (25%)  128Mi (6%)       128Mi (6%)     23m
  default                     nginx-deployment-67d4bdd6f5-w6kd7    500m (25%)    500m (25%)  128Mi (6%)       128Mi (6%)     23m
  kube-system                 kube-proxy-dnxbz                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         28m
  kube-system                 weave-net-gjxxp                      100m (5%)     0 (0%)      200Mi (10%)      0 (0%)         28m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                1100m (55%)  1 (50%)
  memory             456Mi (24%)  256Mi (13%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
Events:
...
kubectl get node kube-worker-1 -o yaml
:
v1
kind
:
Node
metadata
:
:
:
/run/containerd/containerd.sock
node.alpha.kubernetes.io/ttl
:
"0"
volumes.kubernetes.io/controller-managed-attach-detach
:
"true"
:
"2022-02-17T21:46:30Z"
labels
:
beta.kubernetes.io/arch
:
amd64
beta.kubernetes.io/os
:
linux
kubernetes.io/arch
:
amd64
kubernetes.io/hostname
:
:
linux
name
:
:
"4026"
uid
:
98efe7cb-2978-4a0b-842a-1a7bf12c05f8
spec
:
{}
status
:
addresses
:
-
address
:
type
:
InternalIP
-
address
:
type
:
Hostname
allocatable
:
cpu
:
"2"
:
"14167048988"
:
"0"
memory
:
1922788Ki
pods
:
"110"
capacity
:
cpu
:
"2"
:
15372232Ki
:
"0"
memory
:
2025188Ki
pods
:
"110"
:
-
lastHeartbeatTime
:
"2022-02-17T22:20:32Z"
:
message
:
Weave pod has set this
reason
:
WeaveIsUp
status
:
"False"
type
:
NetworkUnavailable
-
lastHeartbeat

Time
:
"2022-02-17T22:20:15Z"
:
"2022-02-17T22:13:25Z"
message
:
kubelet has sufficient memory available
reason
:
KubeletHasSufficientMemory
status
:
"False"
type
:
MemoryPressure
-
:
:
message
:
kubelet has no disk pressure
reason
:
KubeletHasNoDiskPressure
status
:
"False"
type
:
DiskPressure
-
:
:
message
:
kubelet has sufficient PID available
reason
:
KubeletHasSufficientPID
status
:
"False"
type
:
PIDPressure
-
:
:
"2022-02-17T22:15:15Z"
message
:
kubelet is posting ready status. AppArmor enabled
reason
:
KubeletReady
status
:
"True"
type
:
Ready
daemonEndpoints
:
kubeletEndpoint
:
Port
:
10250
nodeInfo
:
architecture
:
amd64
bootID
:
22333234
-7a6b-44d4-9ce1-67e31dc7e369
containerRuntimeVersion
:
containerd://1.5.9
kernelVersion
:
5.13.0-28
-generic
kubeProxyVersion
:
v1.23.3
kubeletVersion
:
v1.23.3
machineID
:
9384e2927f544209b5d7b67474bbf92b
operatingSystem
:
linux
osImage
:
Ubuntu 21.10
systemUUID
:
aa829ca9-73d7-064d-9019-df07404ad448
Looking at logs
For now, digging deeper into the cluster requires logging into the relevant machines. Here are the locations
of the relevant log files. On systemd-based systems, you may need to use
instead of examining log files.
Control Plane nodes
/var/log/kube-apiserver.log
- API Server, responsible for serving the API
/var/log/kube-scheduler.log
- Scheduler, responsible for making scheduling decisions
/var/log/kube-controller-manager.log
- a component that runs most Kubernetes built-in
controllers
, with the notable exception of scheduling
(the kube-scheduler handles scheduling).
Worker Nodes
/var/log/kubelet.log
- logs from the kubelet, responsible for running containers on the node
/var/log/kube-proxy.log
- logs from
, which is responsible for directing traffic to Service endpoints
Cluster failure modes
This is an incomplete list of things that could go wrong, and how to adjust your cluster setup to mitigate the problems.
Contributing causes
VM(s) shutdown
Network partition within cluster, or between cluster and users
Crashes in Kubernetes software
Data loss or unavailability of persistent storage (e.g. GCE PD or AWS EBS volume)
Operator error, for example misconfigured Kubernetes software or application software
Specific scenarios
API server VM shutdown or apiserver crashing
Results
unable to stop, update, or start new pods, services, replication controller
existing pods and services should continue to work normally, unless they depend on the Kubernetes API
API server backing storage lost
Results
the kube-apiserver component fails to start successfully and become healthy
kubelets will not be able to reach it but will continue to run the same pods and provide the same service proxying
manual recovery or recreation of apiserver state necessary before apiserver is restarted
Supporting services (node controller, replication controller manager, scheduler, etc) VM shutdown or crashes
currently those are colocated with the apiserver, and their unavailability has similar consequences as apiserver
in future, these will be replicated as well and may not be co-located
they do not have their own persistent state
Individual node (VM or physical machine) shuts down
Results
pods on that Node stop running
Network partition
Results
partition A thinks the nodes in partition B are down; partition B thinks the apiserver is down.
(Assuming the master VM ends up in partition A.)
Kubelet software fault
Results
crashing kubelet cannot start new pods on the node
kubelet might delete the pods or not
node marked unhealthy
replication controllers start new pods elsewhere
Cluster operator error
Results
loss of pods, services, etc
lost of apiserver backing store
users unable to read API
etc.
Mitigations
Action: Use IaaS provider's automatic VM re

starting feature for IaaS VMs
Mitigates: Apiserver VM shutdown or apiserver crashing
Mitigates: Supporting services VM shutdown or crashes
Action: Use IaaS providers reliable storage (e.g. GCE PD or AWS EBS volume) for VMs with apiserver+etcd
Mitigates: Apiserver backing storage lost
Action: Use
high-availability
Mitigates: Control plane node shutdown or control plane components (scheduler, API server, controller-manager) crashing
Will tolerate one or more simultaneous node or component failures
Mitigates: API server backing storage (i.e., etcd's data directory) lost
Assumes HA (highly-available) etcd configuration
Action: Snapshot apiserver PDs/EBS-volumes periodically
Mitigates: Some cases of operator error
Mitigates: Some cases of Kubernetes software fault
Action: use replication controller and services in front of pods
Mitigates: Node shutdown
Mitigates: Kubelet software fault
Action: applications (containers) designed to tolerate unexpected restarts
Learn about the metrics available in the
Resource Metrics Pipeline
Discover additional tools for
monitoring resource usage
Use Node Problem Detector to
monitor node health
Use
kubectl debug node
to
debug Kubernetes nodes
Use
crictl
to
Get more information about
Kubernetes auditing
Use
telepresence
to
develop and debug services locally
2.1 - Resource metrics pipeline
For Kubernetes, the
Metrics API
offers a basic set of metrics to support automatic scaling and
similar use cases. This API makes information available about resource usage for node and pod,
including metrics for CPU and memory. If you deploy the Metrics API into your cluster, clients of
the Kubernetes API can then query for this information, and you can use Kubernetes' access control
mechanisms to manage permissions to do so.
The
(HPA) and
(VPA)
use data from the metrics API to adjust workload replicas and resources to meet customer demand.
You can also view the resource metrics using the
kubectl top
command.
Note:
The Metrics API, and the metrics pipeline that it enables, only offers the minimum
CPU and memory metrics to enable automatic scaling using HPA and / or VPA.
If you would like to provide a more complete set of metrics, you can complement
the simpler Metrics API by deploying a second
metrics pipeline
that uses the
Custom Metrics API
.
Figure 1 illustrates the architecture of the resource metrics pipeline.
flowchart RL
subgraph cluster[Cluster]
direction RL
S[
]
A[Metrics-
Server]
subgraph B[Nodes]
direction TB
D[cAdvisor] --&gt; C[kubelet]
E[Container
runtime] --&gt; D
E1[Container
P[pod data] -.- C
end
L[API
server]
W[HPA]
C ----&gt;|Summary
API| A --&gt;|metrics
API| L --&gt; W
end
L ---&gt; K[kubectl
top]
classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000;
class W,B,P,K,cluster,D,E,E1 box
classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000
class S spacewhite
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff;
class A,L,C k8s
JavaScript must be
enabled
to view this content
Figure 1. Resource Metrics Pipeline
The architecture components, from right to left in the figure, consist of the following:
cAdvisor
: Daemon for collecting, aggregating and exposing
container metrics included in Kubelet.
kubelet
: Node agent for managing container
resources. Resource metrics are accessible using the
and
/stats
kubelet
API endpoints.
Summary API
: API provided by the kubelet for discovering and retrieving
per-node summarized stats available through the
/stats
endpoint.
: Cluster addon component that collects and aggregates resource
metrics pulled from each kubelet. The API server serves Metrics API for use by HPA, VPA, and by
the
command. Metrics Server is a reference implementation of the Metrics API.
:

 Kubernetes API supporting access to CPU and memory used for
workload autoscaling. To make this work in your cluster, you need an API extension server that
provides the Metrics API.
Note:
cAdvisor supports reading metrics from cgroups, which works with typical container runtimes on Linux.
If you use a container runtime that uses another resource isolation mechanism, for example
virtualization, then that container runtime must support
CRI Container Metrics
in order for metrics to be available to the kubelet.
Kubernetes 1.8 [beta]
The metrics-server implements the Metrics API. This API allows you to access CPU and memory usage
for the nodes and pods in your cluster. Its primary role is to feed resource usage metrics to K8s
autoscaler components.
Here is an example of the Metrics API request for a
minikube
node piped through
jq
for easier
reading:
"/apis/metrics.k8s.io/v1beta1/nodes/minikube"
| jq
'.'
Here is the same API call using
curl
:
curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube
Sample response:
{
"kind"
:
"NodeMetrics"
,
:
"metrics.k8s.io/v1beta1"
,
: {
"name"
:
"minikube"
,
"selfLink"
:
,
:
"2022-01-27T18:48:43Z"
},
"timestamp"
:
"2022-01-27T18:48:33Z"
,
"window"
:
"30s"
,
"usage"
: {
"cpu"
:
"487558164n"
,
"memory"
:
"732212Ki"
}
}
kube-scheduler-minikube
pod contained in the
namespace and piped through
jq
for easier reading:
"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube"
| jq
'.'
curl
:
curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube
{
"kind"
:
"PodMetrics"
,
:
,
: {
"name"
:
"kube-scheduler-minikube"
,
:
"kube-system"
,
:
,
:
"2022-01-27T19:25:00Z"
},
:
"2022-01-27T19:24:31Z"
,
"window"
:
"30s"
,
"containers"
: [
{
"name"
:
"kube-scheduler"
,
"usage"
: {
"cpu"
:
"9559630n"
,
"memory"
:
"22244Ki"
}
}
]
}
The Metrics API is defined in the
k8s.io/metrics
repository. You must enable the
API aggregation layer
and register an
for the
metrics.k8s.io
API.
To learn more about the Metrics API, see
resource metrics API design
,
the
metrics-server repository
and the
resource metrics API
.
Note:
You must deploy the metrics-server or alternative adapter that serves the Metrics API to be able
to access it.
Measuring resource usage
CPU
CPU is reported as the average core usage measured in cpu units. One cpu, in Kubernetes, is
equivalent to 1 vCPU/Core for cloud providers, and 1 hyper-thread on bare-metal Intel processors.
This value is derived by taking a rate over a cumulative CPU counter provided by the kernel (in
both Linux and Windows kernels). The time window used to calculate CPU is shown under window field
in Metrics API.
To learn more about how Kubernetes allocates and measures CPU resources, see
.
Memory
Memory is reported as the working set, measured in bytes, at the instant the metric was collected.
In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under
memory pressure. However, calculation of the working set varies by host OS, and generally makes
heavy use of heuristics to produce an estimate.
The Kubernetes model for a container's working set expects that the container runtime counts
anonymous memory associated with the container in question. The working set metric typically also
includes some cached (file-backed) memory, because the host OS cannot always reclaim pages.
To learn more about how Kubernetes allocates and measures memory resources, see
meaning of memory
.
Metrics Server
The metrics-s

erver fetches resource metrics from the kubelets and exposes them in the Kubernetes
API server through the Metrics API for use by the HPA and VPA. You can also view these metrics
using the
command.
The metrics-server uses the Kubernetes API to track nodes and pods in your cluster. The
metrics-server queries each node over HTTP to fetch metrics. The metrics-server also builds an
internal view of pod metadata, and keeps a cache of pod health. That cached pod health information
is available via the extension API that the metrics-server makes available.
For example with an HPA query, the metrics-server needs to identify which pods fulfill the label
selectors in the deployment.
The metrics-server calls the
kubelet
API
to collect metrics from each node. Depending on the metrics-server version it uses:
Metrics resource endpoint
in version v0.6.0+ or
Summary API endpoint
/stats/summary
in older versions
To learn more about the metrics-server, see the
.
You can also check out the following:
metrics-server design
metrics-server FAQ
metrics-server known issues
metrics-server releases
To learn about how the kubelet serves node metrics, and how you can access those via
the Kubernetes API, read
.
2.2 - Tools for Monitoring Resources
To scale an application and provide a reliable service, you need to
understand how the application behaves when it is deployed. You can examine
application performance in a Kubernetes cluster by examining the containers,
pods
,
services
, and
the characteristics of the overall cluster. Kubernetes provides detailed
information about an application's resource usage at each of these levels.
This information allows you to evaluate your application's performance and
where bottlenecks can be removed to improve overall performance.
In Kubernetes, application monitoring does not depend on a single monitoring solution.
On new clusters, you can use
resource metrics
or
full metrics
pipelines to collect monitoring statistics.
The resource metrics pipeline provides a limited set of metrics related to
cluster components such as the
Horizontal Pod Autoscaler
controller, as well as the
utility.
These metrics are collected by the lightweight, short-term, in-memory
and
are exposed via the
API.
metrics-server discovers all nodes on the cluster and
queries each node's
kubelet
for CPU and
memory usage. The kubelet acts as a bridge between the Kubernetes master and
the nodes, managing the pods and containers running on a machine. The kubelet
translates each pod into its constituent containers and fetches individual
container usage statistics from the container runtime through the container
runtime interface. If you use a container runtime that uses Linux cgroups and
namespaces to implement containers, and the container runtime does not publish
usage statistics, then the kubelet can look up those statistics directly
(using code from
cAdvisor
).
No matter how those statistics arrive, the kubelet then exposes the aggregated pod
resource usage statistics through the metrics-server Resource Metrics API.
This API is served at
/metrics/resource/v1beta1
on the kubelet's authenticated and
read-only ports.
Full metrics pipeline
A full metrics pipeline gives you access to richer metrics. Kubernetes can
respond to these metrics by automatically scaling or adapting the cluster
based on its current state, using mechanisms such as the Horizontal Pod
Autoscaler. The monitoring pipeline fetches metrics from the kubelet and
then exposes them to Kubernetes via an adapter by implementing either the
custom.metrics.k8s.io
or
external.metrics.k8s.io
API.
, a CNCF project, can natively monitor Kubernetes, nodes, and Prometheus itself.
Full metrics pipeline projects that are not part of the CNCF are outside the scope of Kubernetes documentation.
Learn about 

additional debugging tools, including:
Logging
Monitoring
Getting into containers via
exec
Connecting to containers via proxies
Connecting to containers via port forwarding
Inspect Kubernetes node with crictl
2.3 - Monitor Node Health
Node Problem Detector
is a daemon for monitoring and reporting about a node's health.
You can run Node Problem Detector as a
DaemonSet
or as a standalone daemon.
Node Problem Detector collects information about node problems from various daemons
and reports these conditions to the API server as Node
Condition
s
or as
Event
s.
To learn how to install and use Node Problem Detector, see
Node Problem Detector project documentation
.
minikube
Node Problem Detector uses the kernel log format for reporting kernel issues.
To learn how to extend the kernel log format, see
Add support for another log format
.
Enabling Node Problem Detector
Some cloud providers enable Node Problem Detector as an
Addon
.
You can also enable Node Problem Detector with
kubectl
or by creating an Addon DaemonSet.
Using kubectl to enable Node Problem Detector
kubectl
provides the most flexible management of Node Problem Detector.
You can overwrite the default configuration to fit it into your environment or
to detect customized node problems. For example:
Create a Node Problem Detector configuration similar to
node-problem-detector.yaml
:
debug/node-problem-detector.yaml
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
node-problem-detector-v0.1
namespace
:
labels
:
k8s-app
:
node-problem-detector
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
registry.k8s.io/node-problem-detector:v0.1
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
Note:
You should verify that the system log directory is right for your operating system distribution.
Start node problem detector with
kubectl
:
kubectl apply -f https://k8s.io/examples/debug/node-problem-detector.yaml
Using an Addon pod to enable Node Problem Detector
If you are using a custom cluster bootstrap solution and don't need
to overwrite the default configuration, you can leverage the Addon pod to
further automate the deployment.
Create
, and save the configuration in the Addon pod's
directory
/etc/kubernetes/addons/node-problem-detector
on a control plane node.
Overwrite the configuration
The
default configuration
is embedded when building the Docker image of Node Problem Detector.
However, you can use a
ConfigMap
to overwrite the configuration:
Change the configuration files in
config/
Create the
ConfigMap
node-problem-detector-config
:
kubectl create configmap node-problem-detector-config --from-file
=
config/
Change the
to use the
ConfigMap
:
debug/node-problem-detector-configmap.yaml
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true

"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
-
name
:
config
# Overwrite the config/ directory with ConfigMap volume
mountPath
:
/config
readOnly
:
true
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
-
name
:
config
# Define ConfigMap volume
configMap
:
name
:
Recreate the Node Problem Detector with the new configuration file:
# If you have a node-problem-detector running, delete before recreating
kubectl delete -f https://k8s.io/examples/debug/node-problem-detector.yaml
kubectl apply -f https://k8s.io/examples/debug/node-problem-detector-configmap.yaml
Note:
This approach only applies to a Node Problem Detector started with
kubectl
.
Overwriting a configuration is not supported if a Node Problem Detector runs as a cluster Addon.
The Addon manager does not support
ConfigMap
.
Problem Daemons
A problem daemon is a sub-daemon of the Node Problem Detector. It monitors specific kinds of node
problems and reports them to the Node Problem Detector.
There are several types of supported problem daemons.
A
SystemLogMonitor
type of daemon monitors the system logs and reports problems and metrics
according to predefined rules. You can customize the configurations for different log sources
such as
filelog
,
kmsg
,
kernel
,
abrt
,
and
systemd
.
A
SystemStatsMonitor
type of daemon collects various health-related system stats as metrics.
You can customize its behavior by updating its
.
A
CustomPluginMonitor
type of daemon invokes and checks various node problems by running
user-defined scripts. You can use different custom plugin monitors to monitor different
problems and customize the daemon behavior by updating the
.
A
HealthChecker
type of daemon checks the health of the kubelet and container runtime on a node.
Adding support for other log format
The system log monitor currently supports file-based logs, journald, and kmsg.
Additional sources can be added by implementing a new
log watcher
.
Adding custom plugin monitors
You can extend the Node Problem Detector to execute any monitor scripts written in any language by
developing a custom plugin. The monitor scripts must conform to the plugin protocol in exit code
and standard output. For more information, please refer to the
plugin interface proposal
.
Exporter
An exporter reports the node problems and/or metrics to certain backends.
The following exporters are supported:
Kubernetes exporter
: this exporter reports node problems to the Kubernetes API server.
Temporary problems are reported as Events and permanent problems are reported as Node Conditions.
Prometheus exporter
: this exporter reports node problems and metrics locally as Prometheus
(or OpenMetrics) metrics. You can specify the IP address and port for the exporter using command
line arguments.
Stackdriver exporter
: this exporter reports node problems and metrics to the Stackdriver
Monitoring API. The exporting behavior can be customized using a
.
Recommendations and restrictions
It is recommended to run the Node Problem Detector in your cluster to monitor node health.
When running the Node Problem Detector, you can expect extra resource overhead on each node.
Usually this is fine, because:
The kernel log grows relatively slowly.
A resource limit is set for the Node Problem Detector.
Even under high load, the resource usage is acceptable. For more information, see the Node Problem Detector
benchmark result
.
2.4 - Debugging Kubernetes nodes with crictl
crictl

crictl
cri-tools
crictl
crictl
.
The
crictl
or
crictl
Set the
and
flags.
Set the
and
crictl
.
Note:
crictl
timeout
or
debug
--timeout
and
--debug
crictl
crictl
.
crictl
Warning:
If you use
crictl
crictl
is not a
List pods
926f1b5a1d33a       About a minute ago   Ready               sh-84d7dcf559-4r2gq          default             0
run
=
nginx
IMAGE               TAG                 IMAGE ID          

  SIZE
crictl ps
1941fb4da154f       k8s-gcrio.azureedge.net/hyperkube-amd64@sha256:00d814b1f7763f4ab5be80c58e98140dfc69df107f253d7fdd714b30a714260a   17 hours ago        Running             kube-proxy                 0
crictl
exec
N
=
1
Using
crictl
{
: {
"name"
:
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
},
:
"/tmp"
,
"linux"
: {
}
}
Use the
Using
crictl
Image is up to date for busybox@sha256:141c253bc4c3fd0a201d32dc1f493bc

f3fff003b6df416dea4f41046e0f37d47
Pod config
:
{
: {
"name"
:
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
},
:
"/tmp"
,
"linux"
: {
}
}
:
{
: {
"name"
:
"busybox"
},
"image"
:{
"image"
:
"busybox"
},
"command"
: [
"top"
],
:
,
"linux"
: {
}
}
returned.
Created
.
:
Running
.
crictl ps
crictl
.
Map
docker
crictl
.
2.5 - Auditing
auditing
provides a security-relevant, chronological set of records documenting
the sequence of actions in a cluster. The cluster audits the activities generated by users,
by applications that use the Kubernetes API, and by the control plane itself.
Auditing allows cluster administrators to answer the following questions:
what happened?
when did it happen?
who initiated it?
on what did it happen?
where was it observed?
from where was it initiated?
to where was it going?
Audit records begin their lifecycle inside the
component. Each request on each stage
of its execution generates an audit event, which is then pre-processed according to
a certain policy and written to a backend. The policy determines what's recorded
and the backends persist the records. The current backend implementations
include logs files and webhooks.
Each request can be recorded with an associated
stage
. The defined stages are:
RequestReceived
- The stage for events generated as soon as the audit
handler receives the request, and before it is delegated down the handler
chain.
ResponseStarted
- Once the response headers are sent, but before the
response body is sent. This stage is only generated for long-running requests
(e.g. watch).
ResponseComplete
- The response body has been completed and no more bytes
will be sent.
Panic
- Events generated when a panic occurred.
Note:
The configuration of an
Audit Event configuration
is different from the
Event
API object.
The audit logging feature increases the memory consumption of the API server
because some context required for auditing is stored for each request.
Memory consumption depends on the audit logging configuration.
Audit policy
Audit policy defines rules about what events should be recorded and what data
they should include. The audit policy object structure is defined in the
audit.k8s.io
API group
.
When an event is processed, it's
compared against the list of rules in order. The first matching rule sets the
audit level
of the event. The defined audit levels are:
None
- don't log events that match this rule.
Metadata
- log request metadata (requesting user, timestamp, resource,
verb, etc.) but not request or response body.
Request
- log event metadata and request body but not response body.
This does not apply for non-resource requests.
RequestResponse
- log event metadata, request and response bodies

.
You can pass a file with the policy to
using the
--audit-policy-file
flag. If the flag is omitted, no events are logged.
Note that the
rules
field
must
be provided in the audit policy file.
A policy with no (0) rules is treated as illegal.
Below is an example audit policy file:
audit/audit-policy.yaml
:
audit.k8s.io/v1
# This is required.
kind
:
Policy
# Don't generate audit events for all requests in RequestReceived stage.
omitStages
:
-
"RequestReceived"
rules
:
# Log pod changes at RequestResponse level
-
level
:
resources
:
-
group
:
""
# Resource "pods" doesn't match requests to any subresource of pods,
# which is consistent with the RBAC policy.
resources
:
[
"pods"
]
# Log "pods/log", "pods/status" at Metadata level
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
"pods/log"
,
"pods/status"
]
# Don't log requests to a configmap called "controller-leader"
-
level
:
None
resources
:
-
group
:
""
resources
:
[
]
:
[
"controller-leader"
]
# Don't log watch requests by the "system:kube-proxy" on endpoints or services
-
level
:
None
users
:
[
"system:kube-proxy"
]
verbs
:
[
"watch"
]
resources
:
-
group
:
""
# core API group
resources
:
[
"endpoints"
,
]
# Don't log authenticated requests to certain non-resource URL paths.
-
level
:
None
userGroups
:
[
"system:authenticated"
]
:
-
"/api*"
# Wildcard matching.
-
"/version"
# Log the request body of configmap changes in kube-system.
-
level
:
Request
resources
:
-
group
:
""
resources
:
[
]
# This rule only applies to resources in the "kube-system" namespace.
# The empty string "" can be used to select non-namespaced resources.
:
[
]
# Log configmap and secret changes in all other namespaces at the Metadata level.
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
"secrets"
,
]
# Log all other resources in core and extensions at the Request level.
-
level
:
Request
resources
:
-
group
:
""
-
group
:
"extensions"
# Version of group should NOT be included.
# A catch-all rule to log all other requests at the Metadata level.
-
level
:
Metadata
# Long-running requests like watches that fall under this rule will not
# generate an audit event in RequestReceived.
:
-
You can use a minimal audit policy file to log all requests at the
Metadata
level:
# Log all requests at the Metadata level.
:
kind
:
Policy
rules
:
-
level
:
Metadata
If you're crafting your own audit profile, you can use the audit profile for Google Container-Optimized OS as a starting point. You can check the
configure-helper.sh
script, which generates an audit policy file. You can see most of the audit policy file by looking directly at the script.
You can also refer to the
Policy
configuration reference
for details about the fields defined.
Audit backends
Audit backends persist audit events to an external storage.
Out of the box, the kube-apiserver provides two backends:
Log backend, which writes events into the filesystem
Webhook backend, which sends events to an external HTTP API
In all cases, audit events follow a structure defined by the Kubernetes API in the
API group
.
Note:
In case of patches, request body is a JSON array with patch operations, not a JSON object
with an appropriate Kubernetes API object. For example, the following request body is a valid patch
request to
/apis/batch/v1/namespaces/some-namespace/jobs/some-job-name
:
[
{
"op"
:
"replace"
,
"path"
:
"/spec/parallelism"
,
"value"
:
0
},
{
"op"
:
"remove"
,
"path"
:
"/spec/template/spec/containers/0/terminationMessagePolicy"
}
]
Log backend
The log backend writes audit events to a file in
JSONlines
format.
You can configure the log audit backend using the following
flags:
--audit-log-path
specifies the log fi

le path that log backend uses to write
audit events. Not specifying this flag disables log backend.
-
means standard out
--audit-log-maxage
defined the maximum number of days to retain old audit log files
--audit-log-maxbackup
defines the maximum number of audit log files to retain
--audit-log-maxsize
defines the maximum size in megabytes of the audit log file before it gets rotated
If your cluster's control plane runs the kube-apiserver as a Pod, remember to mount the
hostPath
to the location of the policy file and log file, so that audit records are persisted. For example:
=
/etc/kubernetes/audit-policy.yaml
\
=
/var/log/kubernetes/audit/audit.log
then mount the volumes:
...
:
-
mountPath
:
name
:
audit
readOnly
:
true
-
mountPath
:
/var/log/kubernetes/audit/
name
:
audit-log
readOnly
:
false
and finally configure the
hostPath
:
...
volumes
:
-
name
:
audit
hostPath
:
path
:
type
:
File
-
name
:
audit-log
hostPath
:
path
:
type
:
Webhook backend
The webhook audit backend sends audit events to a remote web API, which is assumed to
be a form of the Kubernetes API, including means of authentication. You can configure
a webhook audit backend using the following kube-apiserver flags:
--audit-webhook-config-file
specifies the path to a file with a webhook
configuration. The webhook configuration is effectively a specialized
.
--audit-webhook-initial-backoff
specifies the amount of time to wait after the first failed
request before retrying. Subsequent requests are retried with exponential backoff.
The webhook config file uses the kubeconfig format to specify the remote address of
the service and credentials used to connect to it.
Event batching
Both log and webhook backends support batching. Using webhook as an example, here's the list of
available flags. To get the same flag for log backend, replace
webhook
with
log
in the flag
name. By default, batching is enabled in
webhook
and disabled in
log
. Similarly, by default
throttling is enabled in
webhook
log
.
--audit-webhook-mode
defines the buffering strategy. One of the following:
batch
- buffer events and asynchronously process them in batches. This is the default.
blocking
- block API server responses on processing each individual event.
blocking-strict
- Same as blocking, but when there is a failure during audit logging at the
RequestReceived stage, the whole request to the kube-apiserver fails.
The following flags are used only in the
batch
mode:
--audit-webhook-batch-buffer-size
defines the number of events to buffer before batching.
If the rate of incoming events overflows the buffer, events are dropped.
--audit-webhook-batch-max-size
defines the maximum number of events in one batch.
--audit-webhook-batch-max-wait
defines the maximum amount of time to wait before unconditionally
batching events in the queue.
--audit-webhook-batch-throttle-qps
defines the maximum average number of batches generated
per second.
--audit-webhook-batch-throttle-burst
defines the maximum number of batches generated at the same
moment if the allowed QPS was underutilized previously.
Parameter tuning
Parameters should be set to accommodate the load on the API server.
For example, if kube-apiserver receives 100 requests each second, and each request is audited only
on
and
stages, you should account for ≅200 audit
events being generated each second. Assuming that there are up to 100 events in a batch,
you should set throttling level at least 2 queries per second. Assuming that the backend can take up to
5 seconds to write events, you should set the buffer size to hold up to 5 seconds of events;
that is: 10 batches, or 1000 events.
In most cases however, the default parameters should be sufficient and you don't have to worry about
setting them manually. You can look at the followin

g Prometheus metrics exposed by kube-apiserver
and in the logs to monitor the state of the auditing subsystem.
apiserver_audit_event_total
metric contains the total number of audit events exported.
apiserver_audit_error_total
metric contains the total number of events dropped due to an error
during exporting.
Log entry truncation
Both log and webhook backends support limiting the size of events that are logged.
As an example, the following is the list of flags available for the log backend:
audit-log-truncate-enabled
whether event and batch truncating is enabled.
audit-log-truncate-max-batch-size
maximum size in bytes of the batch sent to the underlying backend.
audit-log-truncate-max-event-size
maximum size in bytes of the audit event sent to the underlying backend.
By default truncate is disabled in both
webhook
and
log
, a cluster administrator should set
or
audit-webhook-truncate-enabled
to enable the feature.
Mutating webhook auditing annotations
.
Event
and the
Policy
resource types by reading the Audit configuration reference.
2.6 - Debugging Kubernetes Nodes With Kubectl
This page shows how to debug a
node
running on the Kubernetes cluster using
command.
minikube
Your Kubernetes server must be at or later than version 1.2.
.
You need to have permission to create Pods and to assign those new Pods to arbitrary nodes.
You also need to be authorized to create Pods that access filesystems from the host.
Debugging a Node using
Use the
command to deploy a Pod to a Node that you want to troubleshoot.
This command is helpful in scenarios where you can't access your Node by using an SSH connection.
When the Pod is created, the Pod opens an interactive shell on the Node.
To create an interactive shell on a Node named “mynode”, run:
=
ubuntu
root@mynode:/#
The debug command helps to gather information and troubleshoot issues. Commands
that you might use include
ip
,
ifconfig
,
nc
,
ping
, and
ps
and so on. You can also
install other tools, such as
mtr
,
tcpdump
, and
curl
, from the respective package manager.
Note:
The debug commands may differ based on the image the debugging pod is using and
these commands might need to be installed.
The debugging Pod can access the root filesystem of the Node, mounted at
/host
in the Pod.
If you run your kubelet in a filesystem namespace,
the debugging Pod sees the root for that namespace, not for the entire node. For a typical Linux node,
you can look at the following paths to find relevant logs:
/host/var/log/kubelet.log
Logs from the
kubelet
, responsible for running containers on the node.
/host/var/log/kube-proxy.log
Logs from
, which is responsible for directing traffic to Service endpoints.
/host/var/log/containerd.log
process running on the node.
/host/var/log/syslog
Shows general messages and information regarding the system.
/host/var/log/kern.log
Shows kernel logs.
When creating a debugging session on a Node, keep in mind that:
automatically generates the name of the new pod, based on
the name of the node.
/host
.
Although the container runs in the host IPC, Network, and PID namespaces,
the pod isn't privileged. This means that reading so

me process information might fail
because access to that information is restricted to superusers. For example,
will fail.
When you finish using the debugging Pod, delete it:
NAME                          READY   STATUS       RESTARTS   AGE
node-debugger-mynode-pdx84    0/1     Completed    0          8m1s
# Change the pod name accordingly
kubectl delete pod node-debugger-mynode-pdx84 --now
pod "node-debugger-mynode-pdx84" deleted
Note:
The
command won't work if the Node is down (disconnected
from the network, or kubelet dies and won't restart, etc.).
Check
debugging a down/unreachable node
in that case.
2.7 - Developing and debugging services locally using telepresence
Note:
Kubernetes applications usually consist of multiple, separate services, each running in its own container. Developing and debugging these services on a remote Kubernetes cluster can be cumbersome, requiring you to
get a shell on a running container
in order to run debugging tools.
is a tool to ease the process of developing and debugging services locally while proxying the service to a remote Kubernetes cluster. Using
allows you to use custom tools, such as a debugger and IDE, for a local service and provides the service full access to ConfigMap, secrets, and the services running on the remote cluster.
This document describes using
to develop and debug services running on a remote cluster locally.
Kubernetes cluster is installed
kubectl
is configured to communicate with the cluster
Telepresence
is installed
Connecting your local machine to a remote Kubernetes cluster
After installing
, run
telepresence connect
to launch its Daemon and connect your local workstation to the cluster.
$ telepresence connect
 
Launching Telepresence Daemon
...
Connected to context default (https://&lt;cluster public IP&gt;)
You can curl services using the Kubernetes syntax e.g.
curl -ik https://kubernetes.default
Developing or debugging an existing service
When developing an application on Kubernetes, you typically program or debug a single service. The service might require access to other services for testing and debugging. One option is to use the continuous deployment pipeline, but even the fastest deployment pipeline introduces a delay in the program or debug cycle.
Use the
telepresence intercept $SERVICE_NAME --port $LOCAL_PORT:$REMOTE_PORT
command to create an "intercept" for rerouting remote service traffic.
Where:
$SERVICE_NAME
is the name of your local service
$LOCAL_PORT
is the port that your service is running on your local workstation
And
$REMOTE_PORT
is the port your service listens to in the cluster
Running this command tells Telepresence to send remote traffic to your local service instead of the service in the remote Kubernetes cluster. Make edits to your service source code locally, save, and see the corresponding changes when accessing your remote application take effect immediately. You can also run your local service using a debugger or any other local development tool.
How does Telepresence work?
Telepresence installs a traffic-agent sidecar next to your existing application's container running in the remote cluster. It then captures all traffic requests going into the Pod, and instead of forwarding this to the application in the remote cluster, it routes all traffic (when you create a
global intercept
) or a subset of the traffic (when you create a
personal intercept
) to your local development environment.
If you're interested in a hands-on tutorial, check out
this tutorial

that walks through locally developing the Guestbook application on Google Kubernetes Engine.
For further reading, visit the
Telepresence website
.
2.8 - Windows debugging tips
Node-level troubleshooting
My Pods are stuck at "Container Creating" or restarting over and over
Ensure that your pause image is compatible with your Windows OS version.
See
Pause container
to see the latest / recommended pause image and/or get more information.
Note:
If using containerd as your container runtime the pause image is specified in the
plugins.plugins.cri.sandbox_image
field of the of config.toml configration file.
My pods show status as
ErrImgPull
or
ImagePullBackOff
Ensure that your Pod is getting scheduled to a
compatible
Windows Node.
More information on how to specify a compatible node for your Pod can be found in
this guide
.
Network troubleshooting
My Windows Pods do not have network connectivity
If you are using virtual machines, ensure that MAC spoofing is
enabled
on all
the VM network adapter(s).
My Windows Pods cannot ping external resources
Windows Pods do not have outbound rules programmed for the ICMP protocol. However,
TCP/UDP is supported. When trying to demonstrate connectivity to resources
outside of the cluster, substitute
ping &lt;IP&gt;
with corresponding
curl &lt;IP&gt;
commands.
If you are still facing problems, most likely your network configuration in
cni.conf
deserves some extra attention. You can always edit this static file. The
configuration update will apply to any new Kubernetes resources.
One of the Kubernetes networking requirements
(see
Kubernetes model
) is
for cluster communication to occur without
NAT internally. To honor this requirement, there is an
ExceptionList
for all the communication where you do not want outbound NAT to occur. However,
this also means that you need to exclude the external IP you are trying to query
from the
. Only then will the traffic originating from your Windows
pods be SNAT'ed correctly to receive a response from the outside world. In this
regard, your
in
cni.conf
should look as follows:
"ExceptionList": [
                "10.244.0.0/16",  # Cluster subnet
                "10.96.0.0/12",   # Service subnet
                "10.127.130.0/24" # Management (host) subnet
            ]
My Windows node cannot access
NodePort
type Services
Local NodePort access from the node itself fails. This is a known
limitation. NodePort access works from other nodes or external clients.
vNICs and HNS endpoints of containers are being deleted
This issue can be caused when the
hostname-override
parameter is not passed to
. To resolve
it, users need to pass the hostname to kube-proxy as follows:
C:\k\
.exe --hostname-override=$(hostname)
My Windows node cannot access my services using the service IP
This is a known limitation of the networking stack on Windows. However, Windows Pods can access the Service IP.
No network adapter is found when starting the kubelet
The Windows networking stack needs a virtual adapter for Kubernetes networking to work.
If the following commands return no results (in an admin shell),
virtual network creation — a necessary prerequisite for the kubelet to work — has failed:
Get-HnsNetwork
| ? Name
-ieq
"cbr0"
Get-NetAdapter
| ? Name
-Like
"vEthernet (Ethernet*"
Often it is worthwhile to modify the
InterfaceName
parameter of the
start.ps1
script, in cases where the host's network adapter isn't "Ethernet".
Otherwise, consult the output of the
start-kubelet.ps1
script to see if there are errors during virtual network creation.
DNS resolution is not properly working
Check the DNS limitations for Windows in this
section
.
fails with "unable to do port forwarding: wincat not found"
This was implemented in Kubernetes 1.15 by including
wincat.exe
in the pause infrastructure container
mcr.microsoft.com/oss/kubernetes/pause:3.6
.
Be sure to use a supported version of Kubernetes.
If you would like to build your ow

n pause infrastructure container be sure to include
wincat
.
My Kubernetes installation is failing because my Windows Server node is behind a proxy
If you are behind a proxy, the following PowerShell environment variables must be defined:
[Environment]
::SetEnvironmentVariable(
"HTTP_PROXY"
,
"http://proxy.example.com:80/"
,
[EnvironmentVariableTarget]
::Machine)
"HTTPS_PROXY"
,
"http://proxy.example.com:443/"
,
Flannel troubleshooting
With Flannel, my nodes are having issues after rejoining a cluster
Whenever a previously deleted node is being re-joined to the cluster, flannelD
tries to assign a new pod subnet to the node. Users should remove the old pod
subnet configuration files in the following paths:
Remove-Item
C:\k\SourceVip.json
C:\k\SourceVipRequest.json
Flanneld is stuck in "Waiting for the Network to be created"
There are numerous reports of this
issue
;
most likely it is a timing issue for when the management IP of the flannel network is set.
A workaround is to relaunch
start.ps1
or relaunch it manually as follows:
"NODE_NAME"
,
"&lt;Windows_Worker_Hostname&gt;"
)
C:\flannel\flanneld.exe --kubeconfig
-file
=c:\k\config --iface=&lt;Windows_Worker_Node_IP&gt; --ip-masq=1 --kube-subnet-mgr=1
My Windows Pods cannot launch because of missing
/run/flannel/subnet.env
This indicates that Flannel didn't launch correctly. You can either try
to restart
flanneld.exe
or you can copy the files over manually from
on the Kubernetes master to
C:\run\flannel\subnet.env
on the Windows worker node and modify the
FLANNEL_SUBNET
row to a different
number. For example, if node subnet 10.244.4.1/24 is desired:
FLANNEL_NETWORK
=
10.244.0.0/16
=
10.244.4.1/24
FLANNEL_MTU
=
1500
FLANNEL_IPMASQ
=
true
Further investigation
If these steps don't resolve your problem, you can get help running Windows containers on Windows nodes in Kubernetes through:
StackOverflow
Windows Server Container
topic
Kubernetes Official Forum
Kubernetes Slack
#SIG-Windows Channel
Home
Blog
Training
Partners
Community
CC BY 4.0

Managing Service Accounts | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

Reference
A
provides an identity for processes that run in a Pod.
A process inside a Pod can use the identity of its associated service account to
authenticate to the cluster's API server.
For an introduction to service accounts, read
configure service accounts
.
This task guide explains some of the concepts behind ServiceAccounts. The
guide also explains how to obtain or revoke tokens that represent
ServiceAccounts.
minikube
To be able to follow these steps exactly, ensure you have a namespace named
examplens
.
If you don't, create one by running:
kubectl create namespace examplens
User accounts versus service accounts
Kubernetes distinguishes between the concept of a user account and a service account
for a number of reasons:
User accounts are for humans. Service accounts are for application processes,
which (for Kubernetes) run in containers that are part of pods.
User accounts are intended to be global: names must be unique across all
namespaces of a cluster. No matter what namespace you look at, a particular
username that represents a user represents the same user.
In Kubernetes, service accounts are namespaced: two different namespaces can
contain ServiceAccounts that have identical names.
Typically, a cluster's user accounts might be synchronised from a corporate
database, where new user account creation requires special privileges and is
tied to complex business processes. By contrast, service account creation is
intended to be more lightweight, allowing cluster users to create service accounts
for specific tasks on demand. Separating ServiceAccount creation from the steps to
onboard human users makes it easier for workloads to following the principle of
least privilege.
Auditing considerations for humans and service accounts may differ; the separation
makes that easier to achieve.
A configuration bundle for a complex system may include definition of various service
accounts for components of that system. Because service accounts can be created
without many constraints and have namespaced names, such configuration is
usually portable.
Bound service acco

unt token volume mechanism
Kubernetes v1.22 [stable]
By default, the Kubernetes control plane (specifically, the
ServiceAccount admission controller
)
adds a
projected volume
to Pods,
and this volume includes a token for Kubernetes API access.
Here's an example of how that looks for a launched Pod:
...
-
name
:
kube-api-access-&lt;random-suffix&gt;
projected
:
sources
:
-
:
path
:
token
# must match the path the app expects
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
That manifest snippet defines a projected volume that consists of three sources. In this case,
each source also represents a single path within that volume. The three sources are:
A
source, that contains a token that the kubelet acquires from kube-apiserver.
The kubelet fetches time-bound tokens using the TokenRequest API. A token served for a TokenRequest expires
either when the pod is deleted or after a defined lifespan (by default, that is 1 hour).
The token is bound to the specific Pod and has the kube-apiserver as its audience.
This mechanism superseded an earlier mechanism that added a volume based on a Secret,
where the Secret represented the ServiceAccount for the Pod, but did not expire.
A
configMap
source. The ConfigMap contains a bundle of certificate authority data. Pods can use these
certificates to make sure that they are connecting to your cluster's kube-apiserver (and not to middlebox
or an accidentally misconfigured peer).
A
source that looks up the name of the namespace containing the Pod, and makes
that name information available to application code running inside the Pod.
Any container within the Pod that mounts this particular volume can access the above information.
Note:
There is no specific mechanism to invalidate a token issued via TokenRequest. If you no longer
trust a bound service account token for a Pod, you can delete that Pod. Deleting a Pod expires
its bound service account tokens.
Manual Secret management for ServiceAccounts
Versions of Kubernetes before v1.22 automatically created credentials for accessing
the Kubernetes API. This older mechanism was based on creating token Secrets that
could then be mounted into running Pods.
In more recent versions, including Kubernetes v1.26, API credentials
are
obtained directly
using the
API,
and are mounted into Pods using a projected volume.
The tokens obtained using this method have bounded lifetimes, and are automatically
invalidated when the Pod they are mounted into is deleted.
You can still
manually create
a Secret to hold a service account token; for example, if you need a token that never expires.
Once you manually create a Secret and link it to a ServiceAccount, the Kubernetes control plane automatically populates the token into that Secret.
Note:
Although the manual mechanism for creating a long-lived ServiceAccount token exists,
using
to obtain short-lived API access tokens is recommended instead.
Control plane details
Token controller
The service account token controller runs as part of
.
This controller acts asynchronously. It:
watches for ServiceAccount deletion and deletes all corresponding ServiceAccount
token Secrets.
watches for ServiceAccount token Secret addition, and ensures the referenced
ServiceAccount exists, and adds a token to the Secret if needed.
watches for Secret deletion and removes a reference from the corresponding
ServiceAccount if needed.
You must pass a service account private key file to the token controller in
the
using the
--service-account-private-key-file
flag. The private key is used to sign generated service account tokens.
Similarly, you must pass the corresponding public key to the
using the
--service-account-key-file
flag. The public key will be used to
verify the

 tokens during authentication.
The modification of pods is implemented via a plugin
called an
Admission Controller
.
It is part of the API server.
This admission controller acts synchronously to modify pods as they are created.
When this plugin is active (and it is by default on most distributions), then
it does the following when a Pod is created:
If the pod does not have a
.spec.serviceAccountName
set, the admission controller sets the name of the
ServiceAccount for this incoming Pod to
default
.
The admission controller ensures that the ServiceAccount referenced by the incoming Pod exists. If there
is no ServiceAccount with a matching name, the admission controller rejects the incoming Pod. That check
applies even for the
default
ServiceAccount.
Provided that neither the ServiceAccount's
automountServiceAccountToken
field nor the
Pod's
field is set to
false
:
the admission controller mutates the incoming Pod, adding an extra
volume
that contains
a token for API access.
the admission controller adds a
volumeMount
to each container in the Pod,
skipping any containers that already have a volume mount defined for the path
.
For Linux containers, that volume is mounted at
;
on Windows nodes, the mount is at the equivalent path.
If the spec of the incoming Pod doesn't already contain any
imagePullSecrets
, then the
admission controller adds
, copying them from the
.
TokenRequest API
You use the
subresource of a ServiceAccount to obtain a time-bound token for that ServiceAccount.
You don't need to call this to obtain an API token for use within a container, since
the kubelet sets this up for you using a
.
If you want to use the TokenRequest API from
kubectl
, see
Manually create an API token for a ServiceAccount
.
The Kubernetes control plane (specifically, the ServiceAccount admission controller)
adds a projected volume to Pods, and the kubelet ensures that this volume contains a token
that lets containers authenticate as the right ServiceAccount.
(This mechanism superseded an earlier mechanism that added a volume based on a Secret,
where the Secret represented the ServiceAccount for the Pod but did not expire.)
...
-
name
:
projected
:
:
420
# decimal equivalent of octal 0644
sources
:
-
:
:
3607
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
That manifest snippet defines a projected volume that combines information from three sources:
A
A
configMap
A
source. This
volume makes the name of the namespace containing the Pod available
to application code running inside the Pod.
Any container within the Pod that mounts this volume can access the above information.
Create additional API tokens
Caution:
Only create long-lived API tokens if the
token request
mechanism
is not suitable. The token request mechanism provides time-limited tokens; because these
expire, they represen

t a lower risk to information security.
To create a non-expiring, persisted API token for a ServiceAccount, create a
Secret of type
kubernetes.io/service-account-token
with an annotation
referencing the ServiceAccount. The control plane then generates a long-lived token and
updates that Secret with that generated token data.
Here is a sample manifest for such a Secret:
secret/serviceaccount/mysecretname.yaml
:
v1
kind
:
Secret
type
:
metadata
:
name
:
mysecretname
:
kubernetes.io/service-account.name
:
myserviceaccount
To create a Secret based on this example, run:
kubectl -n examplens create -f https://k8s.io/examples/secret/serviceaccount/mysecretname.yaml
To see the details for that Secret, run:
kubectl -n examplens describe secret mysecretname
Name:           mysecretname
Namespace:      examplens
Annotations:    kubernetes.io/service-account.name=myserviceaccount
                kubernetes.io/service-account.uid=8a85c4c4-8483-11e9-bc42-526af7764f64

Type:   kubernetes.io/service-account-token

Data
====
ca.crt:         1362 bytes
namespace:      9 bytes
token:          ...
If you launch a new Pod into the
examplens
namespace, it can use the
service-account-token Secret that you just created.
Delete/invalidate a ServiceAccount token
If you know the name of the Secret that contains the token you want to remove:
kubectl delete secret name-of-secret
Otherwise, first find the Secret for the ServiceAccount.
# This assumes that you already have a namespace named 'examplens'
kubectl -n examplens get serviceaccount/example-automated-thing -o yaml
:
v1
kind
:
metadata
:
:
kubectl.kubernetes.io/last-applied-configuration
:
|
{
:
"v1"
,
"kind"
:
"ServiceAccount"
,
:{
"annotations"
:{},
"name"
:
"example-automated-thing"
,
:
"examplens"
}}
:
"2019-07-21T07:07:07Z"
name
:
example-automated-thing
namespace
:
examplens
:
"777"
selfLink
:
/api/v1/namespaces/examplens/serviceaccounts/example-automated-thing
uid
:
f23fd170-66f2-4697-b049-e1e266b7f835
secrets
:
-
name
:
example-automated-thing-token-zyxwv
Then, delete the Secret you now know the name of:
kubectl -n examplens delete secret/example-automated-thing-token-zyxwv
The control plane spots that the ServiceAccount is missing its Secret,
and creates a replacement:
:
v1
kind
:
metadata
:
:
:
|
{
:
"v1"
,
"kind"
:
,
:{
:{},
"name"
:
,
:
}}
:
name
:
namespace
:
examplens
:
"1026"
selfLink
:
uid
:
secrets
:
-
name
:
example-automated-thing-token-4rdrh
Clean up
If you created a namespace
examplens
to experiment with, you can remove it:
kubectl delete namespace examplens
ServiceAccount controller
A ServiceAccount controller manages the ServiceAccounts inside namespaces, and
ensures a ServiceAccount named "default" exists in every active namespace.
.
watches for ServiceAccount creation and creates a corresponding
ServiceAccount token Secret to allow API access.
watches for Secret deletion and removes a 

reference from the corresponding
the
using the
using the
verify the tokens during authentication.
Read more details about
projected volumes
.
Feedback
Yes
No
.
or
.
Last modified December 12, 2022 at 3:11 PM PST:
Fix typo in SA admission controller steps (2b5dab08f1)
Bound service account token volume mechanism
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Setup tools | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install Too

ls
Debug Pods
Debug Init 

Auditing
Run Jobs
TLS
Tutorials
Security

 Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configurat

Reference
Kubeadm
Feedback
Yes
No
.
or
.
Last modified November 04, 2022 at 11:37 AM PST:
Updates page weights in reference docs section (98f310ab58)
Home
Blog
Training
Partners
Community
CC BY 4.0

Using Minikube to Create a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, Storage, and Ne

tworking Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Contr

oller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Inte

ractive Tutorial - Configuring a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate Limit Configuration (v1a

lpha1)
Tutorials
Learn what a Kubernetes cluster is.
Learn what Minikube is.
Start a Kubernetes cluster using an online terminal.
Kubernetes Clusters
Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit.
The abstractions in Kubernetes allow you to deploy containerized applications to a cluster without tying them specifically to individual machines. To make use of this new model of deployment, applications need to be packaged in a way that decouples them from individual hosts: they need to be containerized. Containerized applications are more flexible and available than in past deployment models, where applications were installed directly onto specific machines as packages deeply integrated into the host.
Kubernetes automates the distribution and scheduling of application containers across a cluster in a more efficient way.
Kubernetes is an open-source platform and is production-ready.
A Kubernetes cluster consists of two types of resources:
The
Control Plane
coordinates the cluster
Nodes
are the workers that run applications
Summary:
Kubernetes cluster
Minikube
Kubernetes is a production-grade, open-source platform that orchestrates the placement (scheduling) and execution of application containers within and across computer clusters.
Cluster Diagram
The Control Plane is responsible for managing the cluster.
The Control Plane coordinates all activities in your cluster, such as scheduling applications, maintaining applications' desired state, scaling applications, and rolling out new updates.
A node is a VM or a physical computer that serves as a worker machine in a Kubernetes cluster.
Each node has a Kubelet, which is an agent for managing the node and communicating with the Kubernetes control plane. The node should also have tools for handling container operations, such as containerd or Docker. A Kubernetes cluster that handles production traffic should have a minimum of three nodes because if one node goes down, both an
etcd
member and a control plane instance are lost, and redundancy is compromised. You can mitigate this risk by adding more control plane nodes.
Control Planes manage the cluster and the nodes that are used to host the running applications.
When you d

eploy applications on Kubernetes, you tell the control plane to start the application containers. The control plane schedules the containers to run on the cluster's nodes.
The nodes communicate with the control plane using the
, which the control plane exposes. End users can also use the Kubernetes API directly to interact with the cluster.
A Kubernetes cluster can be deployed on either physical or virtual machines. To get started with Kubernetes development, you can use Minikube. Minikube is a lightweight Kubernetes implementation that creates a VM on your local machine and deploys a simple cluster containing only one node. Minikube is available for Linux, macOS, and Windows systems. The Minikube CLI provides basic bootstrapping operations for working with your cluster, including start, stop, status, and delete. For this tutorial, however, you'll use a provided online terminal with Minikube pre-installed.
Now that you know what Kubernetes is, let's go to the online tutorial and start our first cluster!
›
Feedback
Yes
No
.
or
.
Last modified January 03, 2023 at 1:07 PM PST:
Add link to Etcd information in cluster-intro page (#37042) (fbd7819eca)
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm Configuration (v1beta2) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
Overview
Package v1beta2 has been DEPRECATED by v1beta3.
Package v1beta2 defines the v1beta2 version of the kubeadm configuration file format.
This version improves on the v1beta1 format by fixing some minor issues and adding a few new fields.
A list of changes since v1beta1:
"certificateKey" field is added to InitConfiguration and JoinConfiguration.
"ignorePreflightErrors" field is added to the NodeRegistrationOptions.
The JSON "omitempty" tag is used in a more places where appropriate.
The JSON "omitempty" tag of the "taints" field (inside NodeRegistrationOptions) is removed.
See the Kubernetes 1.15 changelog for further details.
Migration from old kubeadm config versions
Please convert your v1beta1 configuration files to v1beta2 using the "kubeadm config migrate" command of kubeadm v1.15.x
(conversion from older releases of kubeadm config files requires older release of kubeadm as well e.g.
kubeadm v1.11 should be used to migrate v1alpha1 to v1alpha2; kubeadm v1.12 should be used to translate v1alpha2 to v1alpha3;
kubeadm v1.13 or v1.14 should be used to translate v1alpha3 to v1beta1)
Nevertheless, kubeadm v1.15.x will support reading from v1beta1 version of the kubeadm config file format.
Basics
The preferred way to configure kubeadm is to pass an YAML configuration file with the
--config
option. Some of the
configuration options defined in the kubeadm config file are also available as command line flags, but only
the most common/simple use case are supported with this approach.
A kubeadm config file could contain multiple configuration types separated using three dashes (
---
).
kubeadm supports the following configuration types:
:
kubeadm.k8s.io/v1beta2
kind
:
InitConfiguration
:
kind
:
:
kind
:
:
kind
:
:
kind
:
JoinConfiguration
To print the defaults for "init" and "join" actions use the following commands:
kubeadm config print init-defaults
kubeadm config print join-defaults
The list of configuration types that must be included in a configuration file depends by the action you are
performing (
init
or
join
) and by the configuration options you are going to use (defaults or advanced customization).
If some configuration types are not provided, or provided only partially, kubeadm will use default values; defaults
provided by ku

beadm includes also enforcing consistency of values across components when required (e.g.
--cluster-cidr
flag on controller manager and
on kube-proxy).
Users are always allowed to override default values, with the only exception of a small subset of setting with
relevance for security (e.g. enforce authorization-mode Node and RBAC on API server)
If the user provides a configuration types that is not expected for the action you are performing, kubeadm will
ignore those types and print a warning.
Kubeadm init configuration types
When executing kubeadm init with the
--config
option, the following configuration types could be used:
InitConfiguration, ClusterConfiguration, KubeProxyConfiguration, KubeletConfiguration, but only one
between InitConfiguration and ClusterConfiguration is mandatory.
:
kind
:
bootstrapTokens
:
...
nodeRegistration
:
...
The InitConfiguration type should be used to configure runtime settings, that in case of
are the configuration of the bootstrap token and all the setting which are specific to the node where kubeadm
is executed, including:
, that holds fields that relate to registering the new node to the cluster;
use it to customize the node name, the CRI socket to use or any other settings that should apply to this
node only (e.g. the node ip).
localAPIEndpoint
, that represents the endpoint of the instance of the API server to be deployed on this node;
use it e.g. to customize the API server advertise address.
:
kind
:
networking
:
...
etcd
:
...
apiServer
:
extraArgs
:
...
extraVolumes
:
...
...
The ClusterConfiguration type should be used to configure cluster-wide settings,
including settings for:
Networking, that holds configuration for the networking topology of the cluster; use it e.g. to customize
pod subnet or services subnet.
Etcd configurations; use it e.g. to customize the local etcd or to configure the API server
for using an external etcd cluster.
kube-apiserver, kube-scheduler, kube-controller-manager configurations; use it to customize control-plane
components by adding customized setting or overriding kubeadm default settings.
:
kind
:
...
The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances deployed
in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.
See https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/ or
https://pkg.go.dev/k8s.io/kube-proxy/config/v1alpha1#KubeProxyConfiguration
for kube proxy official documentation.
:
kind
:
...
The KubeletConfiguration type should be used to change the configurations that will be passed to all kubelet instances
deployed in the cluster. If this object is not provided or provided only partially, kubeadm applies defaults.
See https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ or
https://pkg.go.dev/k8s.io/kubelet/config/v1beta1#KubeletConfiguration
for kubelet official documentation.
Here is a fully populated example of a single YAML file containing multiple
configuration types to be used during a
run.
:
kind
:
:
-
token
:
"9a08jv.c0izixklcxtmnze7"
:
"kubeadm bootstrap token"
ttl
:
"24h"
-
token
:
"783bde.3f89s0fje9f38fhf"
:
"another bootstrap token"
usages
:
- authentication
- signing
groups
:
- system:bootstrappers:kubeadm:default-node-token
:
name
:
"ec2-10-100-0-1"
criSocket
:
"/var/run/dockershim.sock"
taints
:
-
key
:
"kubeadmNode"
value
:
"someValue"
effect
:
"NoSchedule"
kubeletExtraArgs
:
v
:
4
ignorePreflightErrors
:
- IsPrivilegedUser
:
advertiseAddress
:
"10.100.0.1"
bindPort
:
644

3
certificateKey
:
"e6a2eb8581237ab72a4f494f30285ec12a9694d750b9785706a83bfcbbbd2204"
---
:
kind
:
etcd
:
# one of local or external
local
:
imageRepository
:
"registry.k8s.io"
imageTag
:
"3.2.24"
dataDir
:
"/var/lib/etcd"
extraArgs
:
listen-client-urls
:
"http://10.100.0.1:2379"
serverCertSANs
:
-
"ec2-10-100-0-1.compute-1.amazonaws.com"
peerCertSANs
:
-
# external:
#   endpoints:
#     - "10.100.0.1:2379"
#     - "10.100.0.2:2379"
#   caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
#   certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
#   keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
:
serviceSubnet
:
"10.96.0.0/16"
podSubnet
:
"10.244.0.0/24"
dnsDomain
:
"cluster.local"
:
"v1.12.0"
:
"10.100.0.1:6443"
apiServer
:
extraArgs
:
authorization-mode
:
"Node,RBAC"
:
-
name
:
"some-volume"
hostPath
:
"/etc/some-path"
mountPath
:
"/etc/some-pod-path"
readOnly
:
false
pathType
:
File
certSANs
:
-
"10.100.1.1"
-
timeoutForControlPlane
:
4m0s
:
extraArgs
:
"node-cidr-mask-size":
"20"
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
scheduler
:
extraArgs
:
address
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
:
:
useHyperKubeImage
:
false
:
"example-cluster"
---
:
kind
:
# kubelet specific options here
---
:
kind
:
# kube-proxy specific options here
Kubeadm join configuration types
When executing kubeadm join with the
--config
option, the JoinConfiguration type should be provided.
:
kind
:
...
The JoinConfiguration type should be used to configure runtime settings, that in case of
are the discovery method used for accessing the cluster info and all the setting which are specific
to the node where kubeadm is executed, including:
node only (e.g. the node IP).
apiEndpoint
, that represents the endpoint of the instance of the API server to be eventually deployed on this node.
ClusterStatus
ClusterConfiguration contains cluster-wide configuration for a kubeadm cluster
Field
string
kind
string
etcd
Etcd
etcd
holds configuration for etcd.
holds configuration for the networking topology of the cluster.
string
is the target version of the control plane.
string
sets a stable IP address or DNS name for the control plane; it
can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port.
In case the
is not specified, the
+
bindPort
are used; in case the
is specified but without a TCP port,
the
bindPort
is used.
Possible usages are:
In a cluster with more than one control plane instances, this field should be
assigned the address of the external load balancer in front of the
control plane instances.
In environments with enforced node recycling, the
could be used for assigning a stable DNS to the control plane.
apiServer
APIServer
apiServer
contains extra settings for the API server.

ControlPlaneComponent
contains extra settings for the controller manager.
scheduler
scheduler
contains extra settings for the scheduler.
dns
DNS
dns
defines the options for the DNS add-on installed in the cluster.
string
specifies where to store or look for all required certificates.
string
sets the container registry to pull images from.
If empty,
registry.k8s.io
will be used by default; in case of kubernetes version is
a CI build (kubernetes version starts with
ci/
)
gcr.io/k8s-staging-ci-images
is used as a default for control plane components and for kube-proxy, while
will be used for all the other images.
bool
controls if hyperkube should be used for Kubernetes components
instead of their respective separate images.
DEPRECATED: As
hyperkube
is itself deprecated, this fields is too. It will be
removed in future kubeadm config versions, kubeadm will print multiple warnings
when this set to true, and at some point it may become ignored.
contains the feature gates enabled by the user.
string
The cluster name.
ClusterStatus contains the cluster status. The ClusterStatus will be stored in
the kubeadm-config ConfigMap in the cluster, and then updated by kubeadm when
additional control plane instance joins or leaves the cluster.
Field
string
kind
string
apiEndpoints
map[string]github.com/tengqm/kubeconfig/config/kubeadm/v1beta2.APIEndpoint
currently available in the cluster, one for each control
plane/API server instance.
The key of the map is the IP of the host's default interface.
InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime
Field
string
kind
string
[]BootstrapToken
is respected at
time and describes a set of bootstrap tokens to create.
This information IS NOT uploaded to the kubeadm cluster ConfigMap, partly because of its sensitive nature.
NodeRegistrationOptions
holds fields that relate to registering the new control-plane node to the cluster.
APIEndpoint
represents the endpoint of the API server instance that's deployed on this control plane node.
In HA setups, this differs from
ClusterConfiguration.controlPlaneEndpoint
in the sense that ControlPlaneEndpoint
is the global endpoint for the cluster, which then load-balances the requests to each individual API server. This
configuration object lets you customize what IP/DNS name and port the local API server advertises it's accessible
on. By default, kubeadm tries to auto-detect the IP of the default interface and use that, but in case that process
fails you may set the desired value here.
string
sets the key with which certificates and keys are encrypted prior to being uploaded in
a secret in the cluster during the
uploadcerts init
phase.
JoinConfiguration contains elements describing a particular node.
Field
string
kind
string
holds fields that relate to registering the new
control-plane node to the cluster
caCertPath
string
is the path to the SSL certificate authority used to
secure comunications between a node and the control-plane.
Defaults to "/etc/kubernetes/pki/ca.crt".
discovery
Discovery
discovery
specifies the options for the kubelet to use during the TLS
bootstrap process.
c

ontrolPlane
JoinControlPlane
controlPlane
defines the additional control plane instance to be deployed
on the joining node. If nil, no additional control plane instance will be deployed.
APIEndpoint struct contains elements of API server instance deployed on a node.
Field
string
sets the IP address for the API server to advertise.
bindPort
int32
bindPort
sets the secure port for the API Server to bind to.
Defaults to 6443.
APIServer
APIServer holds settings necessary for API server deployments in the cluster.
Field
(Members of
are embedded into this type.)
certSANs
[]string
certSANs
sets extra Subject Alternative Names (SANs) for the API Server
signing certificate.
controls the timeout that we wait for the API server
to appear.
BootstrapToken
BootstrapToken describes one bootstrap token, stored as a Secret in the cluster
Field
token
BootstrapTokenString
token
is used for establishing bidirectional trust between nodes and control-planes.
Used for joining nodes in the cluster.
string
sets a human-friendly message why this token exists and what it's used
for, so other administrators can know its purpose.
ttl
ttl
defines the time to live for this token. Defaults to '24h'.
expires
and
ttl
are mutually exclusive.
expires
meta/v1.Time
expires
specifies the timestamp when this token expires. Defaults to being set
dynamically at runtime based on the
ttl
.
expires
and
ttl
usages
[]string
usages
describes the ways in which this token can be used. Can by default be used
for establishing bidirectional trust, but that can be changed here.
groups
[]string
groups
specifies the extra groups that this token will authenticate as when/if
used for authentication.
BootstrapTokenDiscovery
Discovery
BootstrapTokenDiscovery is used to set the options for bootstrap token based discovery
Field
token
string
token
is a token used to validate cluster information fetched from
the control-plane.
apiServerEndpoint
string
is an IP or domain name to the API server from which information
will be fetched.
caCertHashes
[]string
specifies a set of public key pins to verify when token-based discovery
is used. The root CA found during discovery must match one of these values.
Specifying an empty set disables root CA pinning, which can be unsafe.
Each hash is specified as ":", where the only currently supported type is "sha256".
This is a hex-encoded SHA-256 hash of the Subject Public Key Info (SPKI) object in
DER-encoded ASN.1. These hashes can be calculated using, for example, OpenSSL.
unsafeSkipCAVerification
bool
allows token-based discovery without CA verification via
. This can weaken the security of kubeadm since other nodes can
impersonate the control-plane.
BootstrapTokenString is a token of the format abcdef.abcdef0123456789 that is used
for both validation of the practically of the API server from a joining node's point
of view and as an authentication method for the node in the bootstrap phase of
"kubeadm join". This token is and should be short-lived
Field
-
string
-
string
APIServer
ControlPlaneComponent holds settings common to control plane component of the cluster
Field
extra

Args
extraArgs
is an extra set of flags to pass to a control plane component.
A key in this map is the flag name as it appears on the command line except
without leading dash(es).
[]HostPathMount
is an extra set of host volumes mounted to the control plane
DNS
DNS defines the DNS addon that should be used in the cluster
Field
type
DNSAddOnType
type
defines the DNS add-on to be used.
ImageMeta
ImageMeta
ImageMeta
ImageMeta allows to customize the image used for the DNS component
(Alias of
string
)
DNS
DNSAddOnType defines string identifying DNS add-on types.
Discovery
Discovery specifies the options for the kubelet to use during the TLS Bootstrap process
Field
bootstrapToken
is used to set the options for bootstrap token based discovery.
and
file
file
FileDiscovery
file
is used to specify a file or URL to a kubeconfig file from which to load
cluster information.
and
file
tlsBootstrapToken
string
is a token used for TLS bootstrapping.
If
is set, this field is defaulted to
.bootstrapToken.token
,
but can be overridden.
If
file
is set, this field
must be set
in case the KubeConfigFile does not
contain any other authentication information.
timeout
timeout
modifies the discovery timeout.
Etcd
Etcd contains elements describing Etcd configuration.
Field
local
LocalEtcd
local
provides configuration knobs for configuring the local etcd instance.
local
and
external
external
ExternalEtcd
external
describes how to connect to an external etcd cluster.
local
and
external
Etcd
ExternalEtcd describes an external etcd cluster.
Kubeadm has no knowledge of where certificate files live and they must be supplied.
Field
endpoints
[]string
endpoints
of etcd members. Required for external etcd.
caFile
string
caFile
is an SSL Certificate Authority (CA) file used to secure etcd communication.
Required if using a TLS connection.
certFile
string
certFile
is an SSL certification file used to secure etcd communication.
keyFile
string
keyFile
is an SSL key file used to secure etcd communication.
Discovery
FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load cluster information
Field
string
is used to specify the actual file path or URL to the kubeconfig file
from which to load cluster information.
HostPathMount
HostPathMount contains elements describing volumes that are mounted from the host.
Field
name
string
name
of the volume inside the Pod template.
hostPath
string
hostPath
is the path in the host that will be mounted inside the Pod.
mountPath
string
mountPath
is the path inside the Pod where hostPath volume will be mounted.
readOnly
bool
readOnly
controls write access to the volume.
pathType
core/v1.HostPathType
pathType
is the type of the HostPath.
ImageMeta
DNS
LocalEtcd
ImageMeta allows to customize the image used for components that are not
originated from the Kubernetes/Kubernetes release process
Field
string
If not set, the
defined in ClusterConfiguration will be used.
imageTag
s

tring
imageTag
allows for specifying a tag for the image.
In case this value is set, kubeadm does not change automatically the
version of the above components during upgrades.
JoinControlPlane contains elements describing an additional control plane instance
to be deployed on the joining node.
Field
represents the endpoint of the API server instance
to be deployed on this node.
string
is the key that is used for decryption of certificates after
they are downloaded from the secret upon joining a new control plane node.
The corresponding encryption key is in the InitConfiguration.
LocalEtcd
Etcd
LocalEtcd describes that kubeadm should run an etcd cluster locally.
Field
ImageMeta
ImageMeta
ImageMeta
ImageMeta allows to customize the container used for etcd.
dataDir
string
dataDir
is the directory etcd will place its data.
Defaults to "/var/lib/etcd".
extraArgs
extraArgs
are extra arguments provided to the etcd binary when run
inside a static pod.
A key in this map is the flag name as it appears on the
command line except without leading dash(es).
[]string
sets extra Subject Alternative Names (SANs) for the
etcd server signing certificate.
[]string
etcd peer signing certificate.
Networking contains elements describing cluster's networking configuration
Field
string
is the subnet used by kubernetes Services. Defaults to "10.96.0.0/12".
podSubnet
string
podSubnet
is the subnet used by Pods.
dnsDomain
string
dnsDomain
is the DNS domain used by kubernetes Services. Defaults to "cluster.local".
NodeRegistrationOptions holds fields that relate to registering a new control-plane
or node to the cluster, either via "kubeadm init" or "kubeadm join".
Field
name
string
name
is the
.Metadata.Name
field of the Node API object that will be created
in this
or
operation.
This field is also used in the
CommonName
field of the kubelet's client certificate
to the API server.
Defaults to the hostname of the node if not provided.
criSocket
string
criSocket
is used to retrieve container runtime information. This information will
be annotated to the Node API object, for later re-use.
taints
[]core/v1.Taint
taints
specifies the taints the Node API object should be registered with.
If this field is unset, i.e. nil, in the
process it will be defaulted with
a control-plane taint for control-plane nodes. If you don't want to taint your control-plane
node, set this field to an empty list, i.e.
taints: []
, in the YAML file. This field is
solely used for Node registration.
passes through extra arguments to the kubelet. The arguments here are
passed to the kubelet command line via the environment file kubeadm writes at runtime for
the kubelet to source. This overrides the generic base-level configuration in the
'kubelet-config-1.X' ConfigMap.
Flags have higher priority when parsing. These values are local and specific to the node
kubeadm is executing on.
A key in this map is the flag name as it appears on the command line except without leading dash(es).
[]string
provides a list of pre-flight errors to be ignored when the
current node is registered.
If you plan to report an issue with this page, mention that the page is auto-generated in your is

sue description. The fix may need to happen elsewhere in the Kubernetes project.
Feedback
Yes
No
.
or
.
APIServer
DNS
Discovery
Etcd
ImageMeta
LocalEtcd
Home
Blog
Training
Partners
Community
CC BY 4.0

Versions in CustomResourceDefinitions | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
This page explains how to add versioning information to
CustomResourceDefinitions
, to indicate the stability
level of your CustomResourceDefinitions or advance your API to a new version with conversion between API representations. It also describes how to upgrade an object from one version to another.
minikube
You should have an initial understanding of
custom resources
.
Your Kubernetes server must be at or later than version v1.16.
.
Overview
The CustomResourceDefinition API provides a workflow for introducing and upgrading
to new versions of a CustomResourceDefinition.
When a CustomResourceDefinition is created, the first version is set in the
spec.versions
list to an appropriate stability level
and a version number. For example
v1beta1
would indicate that the first
version is not yet stable. All custom resource objects will initially be stored
at this version.
Once the CustomResourceDefinition is created, clients may begin using the
v1beta1
API.
Later it might be necessary to add new version such as
v1
.
Adding a new version:
Pick a conversion strategy. Since custom resource objects need the ability to
be served at both versions, that means they will sometimes be served in a
different version than the one stored. To make this possible, the custom resource objects must sometimes be converted between the
version they are stored at and the version they are served at. If the
conversion involves schema changes and requires custom logic, a conversion
webhook should be used. If there are no schema changes, the default
None
conversion strategy may be used and only the
field will be
modified when serving different versions.
If using conversion webhooks, create and deploy the conversion webhook. See
the
Webhook conversion
Update the CustomResourceDefinition to include the new version in the
list with
served:true
. Also, set
spec.conversion
field
to the selected conversion strategy. If using a conversion webhook

, configure
spec.conversion.webhookClientConfig
field to call the webhook.
Once the new version is added, clients may incrementally migrate to the new
version. It is perfectly safe for some clients to use the old version while
others use the new version.
Migrate stored objects to the new version:
See the
upgrade existing objects to a new stored version
section.
It is safe for clients to use both the old and new version before, during and
after upgrading the objects to a new stored version.
Removing an old version:
Ensure all clients are fully migrated to the new version. The kube-apiserver
logs can be reviewed to help identify any clients that are still accessing via
the old version.
Set
served
to
false
for the old version in the
list. If
any clients are still unexpectedly using the old version they may begin reporting
errors attempting to access the custom resource objects at the old version.
If this occurs, switch back to using
on the old version, migrate the
remaining clients to the new version and repeat this step.
Ensure the
upgrade of existing objects to the new stored version
step has been completed.
Verify that the
storage
is set to
true
for the new version in the
list in the CustomResourceDefinition.
Verify that the old version is no longer listed in the CustomResourceDefinition
status.storedVersions
.
Remove the old version from the CustomResourceDefinition
list.
Drop conversion support for the old version in conversion webhooks.
Specify multiple versions
The CustomResourceDefinition API
versions
field can be used to support multiple versions of custom resources that you
have developed. Versions can have different schemas, and conversion webhooks can convert custom resources between versions.
Webhook conversions should follow the
Kubernetes API conventions
wherever applicable.
Specifically, See the
API change documentation
for a set of useful gotchas and suggestions.
Note:
In
apiextensions.k8s.io/v1beta1
, there was a
version
field instead of
versions
. The
version
field is deprecated and optional, but if it is not empty, it must
match the first item in the
versions
field.
This example shows a CustomResourceDefinition with two versions. For the first
example, the assumption is all versions share the same schema with no conversion
between them. The comments in the YAML provide more context.
apiextensions.k8s.io/v1
:
kind
:
metadata
:
# name must match the spec fields below, and be in the form: &lt;plural&gt;.&lt;group&gt;
name
:
crontabs.example.com
spec
:
# group name to use for REST API: /apis/&lt;group&gt;/&lt;version&gt;
group
:
example.com
# list of versions supported by this CustomResourceDefinition
versions
:
-
name
:
v1beta1
# Each version can be enabled/disabled by Served flag.
served
:
true
# One and only one version must be marked as the storage version.
storage
:
true
# A schema is required
schema
:
openAPIV3Schema
:
type
:
object
properties
:
host
:
type
:
string
port
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
# The conversion section is introduced in Kubernetes 1.13+ with a default value of
# None conversion (strategy sub-field set to None).
conversion
:
# None conversion assumes the same schema for all versions and only sets the apiVersion
# field of custom resources to the proper value
strategy
:
None
# either Namespaced or Cluster
scope
:
Namespaced
names
:
# plural name to be used in the URL: /apis/&lt;group&gt;/&lt;version&gt;/&lt;plural&gt;
plural
:
crontabs
# singular name to be used as an alias on the CLI and for display
singular
:
crontab
# kind is normally the CamelCased singular type. Your resource manifests use this.
kind
:
CronTab
# shortNames allow shorter string to match your resource on the CLI
shortNames
:
- ct
# Deprecated in v1.16 in fa

vor of apiextensions.k8s.io/v1
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
-
name
:
v1
served
:
true
storage
:
false
validation
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
None
scope
:
names
:
plural
:
crontabs
singular
:
crontab
# kind is normally the PascalCased singular type. Your resource manifests use this.
kind
:
CronTab
:
- ct
You can save the CustomResourceDefinition in a YAML file, then use
to create it.
kubectl apply -f my-versioned-crontab.yaml
After creation, the API server starts to serve each enabled version at an HTTP
REST endpoint. In the above example, the API versions are available at
/apis/example.com/v1beta1
and
/apis/example.com/v1
.
Version priority
Regardless of the order in which versions are defined in a
CustomResourceDefinition, the version with the highest priority is used by
kubectl as the default version to access objects. The priority is determined
by parsing the
name
field to determine the version number, the stability
(GA, Beta, or Alpha), and the sequence within that stability level.
The algorithm used for sorting the versions is designed to sort versions in the
same way that the Kubernetes project sorts Kubernetes versions. Versions start with a
v
followed by a number, an optional
beta
or
alpha
designation, and
optional additional numeric versioning information. Broadly, a version string might look
like
v2
or
v2beta1
. Versions are sorted using the following algorithm:
Entries that follow Kubernetes version patterns are sorted before those that
do not.
For entries that follow Kubernetes version patterns, the numeric portions of
the version string is sorted largest to smallest.
If the strings
beta
or
alpha
follow the first numeric portion, they sorted
in that order, after the equivalent string without the
beta
or
alpha
suffix (which is presumed to be the GA version).
If another number follows the
beta
, or
alpha
, those numbers are also
sorted from largest to smallest.
Strings that don't fit the above format are sorted alphabetically and the
numeric portions are not treated specially. Notice that in the example below,
foo1
is sorted above
foo10
. This is different from the sorting of the
numeric portion of entries that do follow the Kubernetes version patterns.
This might make sense if you look at the following sorted version list:
- v10
- v2
- v1
- v11beta2
- v10beta3
- v3beta1
- v12alpha1
- v11alpha2
- foo1
- foo10
For the example in
, the
version sort order is
v1
, followed by
v1beta1
. This causes the kubectl
command to use
v1
as the default version unless the provided object specifies
the version.
Version deprecation
Starting in v1.19, a CustomResourceDefinition can indicate a particular version of the resource it defines is deprecated.
When API requests to a deprecated version of that resource are made, a warning messag

e is returned in the API response as a header.
The warning message for each deprecated version of the resource can be customized if desired.
A customized warning message should indicate the deprecated API group, version, and kind,
and should indicate what API group, version, and kind should be used instead, if applicable.
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
# This indicates the v1alpha1 version of the custom resource is deprecated.
# API requests to this version receive a warning header in the server response.
:
true
# This overrides the default warning returned to API clients making v1alpha1 API requests.
deprecationWarning
:
"example.com/v1alpha1 CronTab is deprecated; see http://example.com/v1alpha1-v1 for instructions to migrate to example.com/v1 CronTab"
schema
:
...
-
name
:
v1beta1
served
:
true
# This indicates the v1beta1 version of the custom resource is deprecated.
# A default warning message is returned for this version.
:
true
schema
:
...
-
name
:
v1
served
:
true
storage
:
true
schema
:
...
# Deprecated in v1.16 in favor of apiextensions.k8s.io/v1
:
kind
:
metadata
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
:
...
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
:
true
:
-
name
:
v1beta1
served
:
true
:
true
-
name
:
v1
served
:
true
storage
:
true
Version removal
An older API version cannot be dropped from a CustomResourceDefinition manifest until existing stored data has been migrated to the newer API version for all clusters that served the older version of the custom resource, and the old version is removed from the
of the CustomResourceDefinition.
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1beta1
# This indicates the v1beta1 version of the custom resource is no longer served.
# API requests to this version receive a not found error in the server response.
served
:
false
schema
:
...
-
name
:
v1
served
:
true
# The new served version should be set as the storage version
storage
:
true
schema
:
...
Kubernetes v1.16 [stable]
Note:
Webhook conversion is available as beta since 1.15, and as alpha since Kubernetes 1.13. The
CustomResourceWebhookConversion
feature must be enabled, which is the case automatically for many clusters for beta features. Please refer to the
documentation for more information.
The above example has a None conversion between versions which only sets the
field
on conversion and does not change the rest of the object. The API server also supports webhook
conversions that call an external service in case a conversion is required. For example when:
custom resource 

is requested in a different version than stored version.
Watch is created in one version but the changed object is stored in another version.
custom resource PUT request is in a different version than storage version.
To cover all of these cases and to optimize conversion by the API server,
the conversion requests may contain multiple objects in order to minimize the external calls.
The webhook should perform these conversions independently.
Write a conversion webhook server
Please refer to the implementation of the
custom resource conversion webhook
server
that is validated in a Kubernetes e2e test. The webhook handles the
ConversionReview
requests sent by the API servers, and sends back conversion
results wrapped in
ConversionResponse
. Note that the request
contains a list of custom resources that need to be converted independently without
changing the order of objects.
The example server is organized in a way to be reused for other conversions.
Most of the common code are located in the
framework file
that leaves only
one function
to be implemented for different conversions.
Note:
The example conversion webhook server leaves the
ClientAuth
field
empty
,
which defaults to
NoClientCert
. This means that the webhook server does not
authenticate the identity of the clients, supposedly API servers. If you need
mutual TLS or other ways to authenticate the clients, see
how to
authenticate API servers
.
Permissible mutations
A conversion webhook must not mutate anything inside of
metadata
of the converted object
other than
labels
and
.
Attempted changes to
name
,
UID
and
namespace
are rejected and fail the request
which caused the conversion. All other changes are ignored.
Deploy the conversion webhook service
Documentation for deploying the conversion webhook is the same as for the
admission webhook example service
.
The assumption for next sections is that the conversion webhook server is deployed to a service
named
example-conversion-webhook-server
in
default
namespace and serving traffic on path
/crdconvert
.
Note:
When the webhook server is deployed into the Kubernetes cluster as a
service, it has to be exposed via a service on port 443 (The server
itself can have an arbitrary port but the service object should map it to port 443).
The communication between the API server and the webhook service may fail
if a different port is used for the service.
Configure CustomResourceDefinition to use conversion webhooks
The
None
conversion example can be extended to use the conversion webhook by modifying
section of the
spec
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
# Each version can define its own schema when there is no top-level
# schema is defined.
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
# a Webhook strategy instruct API server to call an external webhook for any conversion between custom resources.
strategy
:
Webhook
# webhook is required when strategy is `Webhook` and it configures the webhook endpoint to be called by API server.
webhook
:
# conversionReviewVersions indicates what ConversionReview versions are understood/preferred by the webhook.
# The first version in the list understood by the API server is sent to the webhook.
# The webhook must respond with a ConversionReview obj

ect in the same version it received.
conversionReviewVersions
:
[
"v1"
,
"v1beta1"
]
clientConfig
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
"Ci0tLS0tQk...&lt;base64-encoded PEM bundle&gt;...tLS0K"
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
kind
:
metadata
:
name
:
spec
:
group
:
# prunes object fields that are not specified in OpenAPI schemas below.
preserveUnknownFields
:
false
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
Webhook
# webhookClientConfig is required when strategy is `Webhook` and it configures the webhook endpoint to be called by API server.
webhookClientConfig
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
to apply it.
kubectl apply -f my-versioned-crontab-with-conversion.yaml
Make sure the conversion service is up and running before applying new changes.
Contacting the webhook
Once the API server has determined a request should be sent to a conversion webhook,
it needs to know how to contact the webhook. This is specified in the
stanza of the webhook configuration.
Conversion webhooks can either be called via a URL or a service reference,
and can optionally include a custom CA bundle to use to verify the TLS connection.
URL
url
gives the location of the webhook, in standard URL form
(
scheme://host:port/path
).
The
host
should not refer to a service running in the cluster; use
a service reference by specifying the
service
field instead.
The host might be resolved via external DNS in some apiservers
(i.e.,
cannot resolve in-cluster DNS as that would
be a layering violation).
host
may also be an IP address.
Please note that using
localhost
or
127.0.0.1
as a
host
is
risky unless you take great care to run this webhook on all hosts
which run an apiserver which might need to make calls to this
webhook. Such installations are likely to be non-portable or not readily run in a new cluster.
The scheme must be "https"; the URL must begin with "https://".
Attempting to use a user or basic au

th (for example "user:password@") is not allowed.
Fragments ("#...") and query parameters ("?...") are also not allowed.
Here is an example of a conversion webhook configured to call a URL
(and expects the TLS certificate to be verified using system trust roots, so does not specify a caBundle):
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
url
:
"https://my-webhook.example.com:9443/my-webhook-path"
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
url
:
...
Service Reference
The
service
stanza inside
is a reference to the service for a conversion webhook.
If the webhook is running within the cluster, then you should use
service
instead of
url
.
The service namespace and name are required. The port is optional and defaults to 443.
The path is optional and defaults to "/".
Here is an example of a webhook that is configured to call a service on port "1234"
at the subpath "/my-path", and to verify the TLS connection against the ServerName
my-service-name.my-service-namespace.svc
using a custom CA bundle.
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
service
:
namespace
:
my-service-namespace
name
:
my-service-name
path
:
/my-path
port
:
1234
caBundle
:
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
caBundle
:
...
Webhook request and response
Request
Webhooks are sent a POST request, with
Content-Type: application/json
,
with a
API object in the
apiextensions.k8s.io
API group
serialized to JSON as the body.
Webhooks can specify what versions of
objects they accept
with the
field in their CustomResourceDefinition:
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
[
"v1"
,
"v1beta1"
]
...
is a required field when creating
custom resource definitions.
Webhooks are required to support at least one
version understood by the current and previous API server.
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
[
"v1"
,
"v1beta1"
]
...
If no
are specified, the default when creating
custom resource definitions is
v1beta1
.
API servers send the first
version in the
list they support.
If none of the versions in the list are supported by the API server, the custom resource definition will not be allowed to be created.
If an API server encounters a conversion webhook configuration that was previously created and does not support any of the
versions the API server knows how to send, attempts to call to the webhook will fail.
This example shows the data contained in an
object
for a request to convert
CronTab
objects to
example.com/v1
:
{
"apiVersion":
"apiextensions.k

8s.io/v1"
,
"kind":
"ConversionReview"
,
"request":
{
# Random uid uniquely identifying this conversion call
"uid":
"705ab4f5-6393-11e8-b7cc-42010a800002"
,
# The API group and version the objects should be converted to
"desiredAPIVersion":
"example.com/v1"
,
# The list of objects to convert.
# May contain one or more objects, in one or more versions.
"objects":
[
{
"kind":
"CronTab"
,
"example.com/v1beta1"
,
"metadata":
{
"creationTimestamp":
"2019-09-04T14:03:02Z"
,
"name":
"local-crontab"
,
"namespace":
"default"
,
"resourceVersion":
"143"
,
"uid":
"3415a7fc-162b-4300-b5da-fd6083580d66"
},
"hostPort":
"localhost:1234"
},
{
"kind":
"CronTab"
,
,
{
"2019-09-03T13:02:01Z"
,
"name":
"remote-crontab"
,
"12893"
,
"uid":
"359a83ec-b575-460d-b553-d859cedde8a0"
},
"example.com:2345"
}
]
}
}
{
"apiextensions.k8s.io/v1beta1"
,
"kind":
,
{
"uid":
,
,
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
}
]
}
}
Response
Webhooks respond with a 200 HTTP status code,
,
and a body containing a
object (in the same version they were sent),
with the
response
stanza populated, serialized to JSON.
If conversion succeeds, a webhook should return a
response
stanza containing the following fields:
uid
, copied from the
request.uid
sent to the webhook
result
, set to
{"status":"Success"}
convertedObjects
, containing all of the objects from
request.objects
, converted to
request.desiredVersion
Example of a minimal successful response from a webhook:
{
"apiextensions.k8s.io/v1"
,
"kind":
,
"response":
{
# must match &lt;request.uid&gt;
"uid":
,
"result":
{
"status":
"Success"
},
# Objects must match the order of request.objects, and have apiVersion set to &lt;request.desiredAPIVersion&gt;.
# kind, metadata.uid, metadata.name, and metadata.namespace fields must not be changed by the webhook.
# metadata.labels and metadata.annotations fields may be changed by the webhook.
# All other changes to metadata fields by the webhook are ignored.
"convertedObjects":
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
"host":
"localhost"
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
"example.com"
,
"port":
"2345"
}
]
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
},
# Objects must match the order of request.ob

jects, and have apiVersion set to &lt;request.desiredAPIVersion&gt;.
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
"host":
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
,
"port":
"2345"
}
]
}
}
If conversion fails, a webhook should return a
response
uid
result
, set to
{"status":"Failed"}
Warning:
Failing conversion can disrupt read and write access to the custom resources,
including the ability to update or delete the resources. Conversion failures
should be avoided whenever possible, and should not be used to enforce validation
constraints (use validation schemas or webhook admission instead).
Example of a response from a webhook indicating a conversion request failed, with an optional message:
{
,
"kind":
,
{
"uid":
"&lt;value from request.uid&gt;"
,
"result":
{
"status":
"Failed"
,
"message":
"hostPort could not be parsed into a separate host and port"
}
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
,
}
}
}
Writing, reading, and updating versioned CustomResourceDefinition objects
When an object is written, it is stored at the version designated as the
storage version at the time of the write. If the storage version changes,
existing objects are never converted automatically. However, newly-created
or updated objects are written at the new storage version. It is possible for an
object to have been written at a version that is no longer served.
When you read an object, you specify the version as part of the path.
You can request an object at any version that is currently served.
If you specify a version that is different from the object's stored version,
Kubernetes returns the object to you at the version you requested, but the
stored object is not changed on disk.
What happens to the object that is being returned while serving the read
request depends on what is specified in the CRD's
:
if the default
strategy
value
None
is specified, the only modifications
to the object are changing the
string and perhaps
pruning
unknown fields
(depending on the configuration). Note that this is unlikely to lead to good
results if the schemas differ between the storage and requested version.
In particular, you should not use this strategy if the same data is
represented in different fields between versions.
if
webhook conversion
is specified, then this
mechanism controls the conversion.
If you update an existing object, it is rewritten at the version that is
currently the storage version. This is the only way that objects can change from
one version to another.
To illustrate this, consider the following hypothetical series of events:
The storage version is
v1beta1
. You create an object. It is stored at version
v1beta1
You add version
v1
to your CustomResourceDefinition and designate it as
the storage version. Here the schemas for
v1
and
v1beta1
are identical,
which is ty

pically the case when promoting an API to stable in the
Kubernetes ecosystem.
You read your object at version
v1beta1
, then you read the object again at
version
v1
. Both returned objects are identical except for the apiVersion
field.
You create a new object. It is stored at version
v1
. You now
have two objects, one of which is at
v1beta1
, and the other of which is at
v1
.
You update the first object. It is now stored at version
v1
since that
is the current storage version.
Previous storage versions
The API server records each version which has ever been marked as the storage
version in the status field
storedVersions
. Objects may have been stored
at any version that has ever been designated as a storage version. No objects
can exist in storage at a version that has never been a storage version.
Upgrade existing objects to a new stored version
When deprecating versions and dropping support, select a storage upgrade
procedure.
Option 1:
Use the Storage Version Migrator
Run the
storage Version migrator
field.
Option 2:
Manually upgrade the existing objects to a new stored version
The following is an example procedure to upgrade from
v1beta1
to
v1
.
Set
v1
as the storage in the CustomResourceDefinition file and apply it
using kubectl. The
is now
v1beta1, v1
.
Write an upgrade procedure to list all existing objects and write them with
the same content. This forces the backend to write objects in the current
storage version, which is
v1
.
Remove
v1beta1
from the CustomResourceDefinition
field.
Note:
The flag
--subresource
is used with the kubectl get, patch, edit, and replace commands to
fetch and update the subresources,
status
and
scale
, for all the API resources that
support them. This flag is available starting from kubectl version v1.24. Previously, reading
subresources (like
status
) via kubectl involved using
kubectl --raw
, and updating
subresources using kubectl was not possible at all. Starting from v1.24, the
kubectl
tool
can be used to edit or patch the
status
subresource on a CRD object. See
How to patch a Deployment using the subresource flag
.
This page is part of the documentation for Kubernetes v1.26.
If you are running a different version of Kubernetes, consult the documentation for that release.
Here is an example of how to patch the
status
subresource for a CRD object using
kubectl
:
kubectl patch customresourcedefinitions &lt;CRD_Name&gt; --subresource
=
'status'
--type
=
'merge'
-p
'{"status":{"storedVersions":["v1"]}}'
Feedback
Yes
No
.
or
.
Last modified December 22, 2022 at 8:58 AM PST:
Fix link and bullets alignment in custom-resource-definition-versioning.md (0cfca02a33)
Overview
URL
Request
Response
Home
Blog
Training
Partners
Community
CC BY 4.0

Use Custom Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubec

tl on macOS
Debug Pods
Get a Shell to a Running Containe

r
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access

 to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kub

e-proxy Configuration (v1alpha1)
Tasks
Feedback
Yes
No
.
or
.
Last modified June 16, 2021 at 5:57 PM PST:
Remove exec permission on markdown files (e9703497a1)
Home
Blog
Training
Partners
Community
CC BY 4.0

CSIStorageCapacity | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl o

n macOS
Debug Pods
Tr

oubleshooting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to 

Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-pr

oxy Configuration (v1alpha1)
Reference
CSIStorageCapacity stores the result of one CSI GetCapacity call.
apiVersion: storage.k8s.io/v1
import "k8s.io/api/storage/v1"
CSIStorageCapacity stores the result of one CSI GetCapacity call. For a given StorageClass, this describes the available capacity in a particular topology segment. This can be used when considering where to instantiate new PersistentVolumes.
For example this can express things like: - StorageClass "standard" has "1234 GiB" available in "topology.kubernetes.io/zone=us-east1" - StorageClass "localssd" has "10 GiB" available in "kubernetes.io/hostname=knode-abc123"
The following three cases all imply that no capacity is available for a certain combination: - no object exists with suitable topology and storage class name - such an object exists, but the capacity is unset - such an object exists, but the capacity is zero
The producer of these objects can decide which approach is more suitable.
They are consumed by the kube-scheduler when a CSI driver opts into capacity-aware scheduling with CSIDriverSpec.StorageCapacity. The scheduler compares the MaximumVolumeSize against the requested size of pending volumes to filter out unsuitable nodes. If MaximumVolumeSize is unset, it falls back to a comparison against the less precise Capacity. If that is also unset, the scheduler assumes that capacity is insufficient and tries some other node.
: storage.k8s.io/v1
kind
: CSIStorageCapacity
metadata
(
)
Standard object's metadata. The name has no particular meaning. It must be be a DNS subdomain (dots allowed, 253 characters). To ensure that there are no conflicts with other CSI drivers on the cluster, the recommendation is to use csisc-&lt;uuid&gt;, a generated name, or a reverse-domain name which ends with the unique CSI driver name.
Objects are namespaced.
More info:
The name of the StorageClass that the reported capacity applies to. It must meet the same requirements as the name of a StorageClass object (non-empty, DNS subdomain). If that object no longer exists, the CSIStorageCapacity object is obsolete and should be removed by its creator. This field is immutable.
capacity
(
Quantity
)
Capacity is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.
The semantic is currently (CSI spec 1.2) defined as: The ava

ilable capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable.
maximumVolumeSize
(
Quantity
)
MaximumVolumeSize is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.
This is defined since CSI spec 1.4.0 as the largest size that may be used in a CreateVolumeRequest.capacity_range.required_bytes field to create a volume with the same parameters as those in GetCapacityRequest. The corresponding value in the Kubernetes API is ResourceRequirements.Requests in a volume claim.
nodeTopology
(
)
NodeTopology defines which nodes have access to the storage for which capacity was reported. If not set, the storage is not accessible from any node in the cluster. If empty, the storage is accessible from all nodes. This field is immutable.
CSIStorageCapacityList
CSIStorageCapacityList is a collection of CSIStorageCapacity objects.
kind
: CSIStorageCapacityList
metadata
(
ListMeta
)
items
([]
Map: unique values on key name will be kept during a merge
Items is the list of CSIStorageCapacity objects.
get
read the specified CSIStorageCapacity
GET /apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities/{name}
name
(
in path
name of the CSIStorageCapacity
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind CSIStorageCapacity
GET /apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/storage.k8s.io/v1/csistoragecapacities
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a CSIStorageCapacity
POST /apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified CSIStorageCapacity
PUT /apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun

(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified CSIStorageCapacity
PATCH /apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a CSIStorageCapacity
DELETE /apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of CSIStorageCapacity
DELETE /apis/storage.k8s.io/v1/namespaces/{namespace}/csistoragecapacities
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Last modified April 09, 2022 at 9:47 AM PST:
Update API Reference multi-pages for v1.24 (363eea4e22)
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Control Topology Management Policies on a node | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

 Tools
Debug Pods
Debug I

nit Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Secu

rity Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Config

Tasks
minikube
.
giv

e the optimal result, if an undesirable hint is stored the preferred field for the hint will be
Aligns the requested resources that Hint Provider provides topology hints for.
scope
and
policy
.
The
scope
pod
or
container
policy
,
,
, etc.).
scopes
and
policies
Note:
See
.
Note:
container
(default)
pod
flag.
The
container
pod
scope.
pod scope
pod
.
Using the
pod
policy, a pod is accepted only if a suitable set of NUMA n

odes
none
(default)
Note:
pod
scope, the container, which is considered by
the same
The
not
attempt to
have the
error.
not
attempt to
error.
TopologyManage

rPolicyOptions
and
If the
and
the
spec
:
:
-
name
:
nginx
image
:
nginx
requests
or
limits
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
Burstable
none
static
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
are equal
to
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
are equal
to
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
:
"1"
:
"1"
requests
:
:
"1"
:
"1"
static
static
pod, the
none
pod, the
static
there will be a state explosion when trying to enumerate the possible NUMA affinit

ies and
Feedback
Yes
No
.
or
.
Last modified November 08, 2022 at 10:35 AM PST:
topologymanager: document topology manager policy options (701ed987c5)
pod scope
Home
Blog
Training
Partners
Community
CC BY 4.0

Interactive Tutorial - Exploring Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, Storage, and Networ

king Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controlle

r
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interact

ive Tutorial - Configuring a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate Limit Configuration (v1alpha

1)
Tutorials
To interact with the Terminal, please use the desktop/tablet version
&lt; Return to Module 2
Home
Continue to Module 4 &gt;
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Cloud Controller Manager Administration | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

tall Tools
Debug Pods
Deb

ug Init Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod 

Security Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Co

nfiguration (v1alpha1)
Tasks
The
and
kubelet
during initialization. This marks the node as needing a second initialization from an external controller before it can be scheduled work. Note that in the event that cloud controller manager is not available, new nodes in the cluster will be left unschedulable. T

he taint is important since the scheduler may require cloud specific information about nodes such as their region or type (high cpu, gpu, high memory, spot instance, etc).
Examples
.
# This is an example of how to set up cloud-controller-manager as a Daemonset in your cluster.
---
:
v1
kind
:
metadata
:
name
:
namespace
:
---
:
kind
:
metadata
:
name
:
roleRef
:
apiGroup
:
kind
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
---
:
apps/v1
kind
:
DaemonSet
metadata
:
labels
:
k8s-app
:
name
:
namespace
:
spec
:
selector
:
:
k8s-app
:
template
:
metadata
:
labels
:
k8s-app
:
spec
:
:
:
-
name
:
image
:
command
:
:
-
key
:
value
:
"true"
effect
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
:
""
Running cloud controller manager comes with a few possible limitations. Although these limitations are being addressed in upcoming 

releases, it's important that you are aware of these limitations for production workloads.
here
.
A good example of this is the TLS bootstrapping feature in the Kubelet. TLS bootstrapping assumes that the Kubelet has the ability to ask the cloud provider (or a local metadata service) for all its address types (private, public, etc) but cloud controller manager cannot set a node's address types without being initialized in the first place which requires that the kubelet has TLS certificates to communicate with the apiserver.
.
Feedback
Yes
No
.
or
.
Last modified October 12, 2020 at 4:42 PM PST:
Fix a markup error. (9b55db0286)
Examples
Home
Blog
Training
Partners
Community
CC BY 4.0

WebhookAdmission Configuration (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
WebhookAdmission
WebhookAdmission provides configuration for the webhook admission controller.
Field
string
kind
string
kubeConfigFile
string
KubeConfigFile is the path to the kubeconfig file.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Command line tool (kubectl) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extending the Kubernete

s API
Tasks
Debu

g Pods
Auditing
Run Jobs
TLS
Tutorials
Configuring Redis using

 a ConfigMap
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apis

erver Audit Configuration (v1)
Reference
Kubernetes provides a command line tool for communicating with a Kubernetes cluster's
,
using the Kubernetes API.
This tool is named
kubectl
.
For configuration,
kubectl
looks for a file named
config
in the
$HOME/.kube
You can specify other
files by setting the
environment variable or by setting the
--kubeconfig
flag.
This overview covers
kubectl
syntax, describes the command operations, and provides common examples.
For details about each command, including all the supported flags and subcommands, see the
kubectl
reference documentation.
For installation instructions, see
Installing kubectl
;
for a quick guide, see the
cheat sheet
.
If you're used to using the
docker
command-line tool,
kubectl
for Docker Users
explains some equivalent commands for Kubernetes.
Syntax
Use the following syntax to run
kubectl
commands from your terminal window:
kubectl
[
command
]
[
TYPE
]
[
NAME
]
[
flags
]
where
command
,
TYPE
,
NAME
, and
flags
are:
command
: Specifies the operation that you want to perform on one or more resources,
for example
create
,
get
,
describe
,
delete
.
TYPE
: Specifies the
resource type
. Resource types are case-insensitive and
you can specify the singular, plural, or abbreviated forms.
For example, the following commands produce the same output:
kubectl get pod pod1
kubectl get pods pod1
kubectl get po pod1
NAME
: Specifies the name of the resource. Names are case-sensitive. If the name is omitted, details for all resources are displayed, for example
.
When performing an operation on multiple resources, you can specify each resource by type and name or specify one or more files:
To specify resources by type and name:
To group resources if they are all the same type:
TYPE1 name1 name2 name&lt;#&gt;
.
Example:
kubectl get pod example-pod1 example-pod2
To specify multiple resource types individually:
TYPE1/name1 TYPE1/name2 TYPE2/name3 TYPE&lt;#&gt;/name&lt;#&gt;
.
Example:
kubectl get pod/example-pod1 replicationcontroller/example-rc1
To specify resources with one or more files:
-f file1 -f file2 -f file&lt;#&gt;
Use YAML rather than JSON
since YAML tends to be more user-friendly, especially for configuration files.
Example:
kubectl get -f ./pod.yaml
flags
: Specifies optional flags. For example, you can use the
-s
or
--server
flags to specify the address and port of the Kubernetes API 

server.
Caution:
Flags that you specify from the command line override default values and any corresponding environment variables.
If you need help, run
kubectl help
from the terminal window.
In-cluster authentication and namespace overrides
By default
kubectl
will first determine if it is running within a pod, and thus in a cluster. It starts by checking for the
KUBERNETES_SERVICE_HOST
and
KUBERNETES_SERVICE_PORT
environment variables and the existence of a service account token file at
/var/run/secrets/kubernetes.io/serviceaccount/token
. If all three are found in-cluster authentication is assumed.
To maintain backwards compatibility, if the
POD_NAMESPACE
environment variable is set during in-cluster authentication it will override the default namespace from the service account token. Any manifests or tools relying on namespace defaulting will be affected by this.
environment variable
If the
environment variable is set, cli operations on namespaced resources will default to the variable value. For example, if the variable is set to
seattle
,
would return pods in the
seattle
namespace. This is because pods are a namespaced resource, and no namespace was provided in the command. Review the output of
kubectl api-resources
to determine if a resource is namespaced.
Explicit use of
--namespace &lt;value&gt;
overrides this behavior.
How kubectl handles ServiceAccount tokens
If:
there is Kubernetes service account token file mounted at
, and
the
environment variable is set, and
the
you don't explicitly specify a namespace on the kubectl command line
then kubectl assumes it is running in your cluster. The kubectl tool looks up the
namespace of that ServiceAccount (this is the same as the namespace of the Pod)
and acts against that namespace. This is different from what happens outside of a
cluster; when kubectl runs outside a cluster and you don't specify a namespace,
the kubectl command acts against the namespace set for the current context in your
client configuration. To change the default namespace for your kubectl you can use the
following command:
kubectl config set-context --current --namespace
=
&lt;namespace-name&gt;
The following table includes short descriptions and the general syntax for all of the
kubectl
operations:
Operation
Syntax
alpha
kubectl alpha SUBCOMMAND [flags]
List the available commands that correspond to alpha features, which are not enabled in Kubernetes clusters by default.
annotate
kubectl annotate (-f FILENAME | TYPE NAME | TYPE/NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--overwrite] [--all] [--resource-version=version] [flags]
Add or update the annotations of one or more resources.
api-resources
kubectl api-resources [flags]
List the API resources that are available.
api-versions
kubectl api-versions [flags]
List the API versions that are available.
apply
kubectl apply -f FILENAME [flags]
Apply a configuration change to a resource from a file or stdin.
attach
kubectl attach POD -c CONTAINER [-i] [-t] [flags]
Attach to a running container either to view the output stream or interact with the container (stdin).
auth
kubectl auth [flags] [options]
Inspect authorization.
autoscale
kubectl autoscale (-f FILENAME | TYPE NAME | TYPE/NAME) [--min=MINPODS] --max=MAXPODS [--cpu-percent=CPU] [flags]
Automatically scale the set of pods that are managed by a replication controller.
certificate
kubectl certificate SUBCOMMAND [options]
Modify certificate resources.
cluster-info
kubectl cluster-info [flags]
Display endpoint information about the master and services in the cluster.
completion
kubectl completion SHELL [options]
Output shell completion code for the specified shell (bash or zsh).
config
kubectl config SUBCOMMAND [flags]
Modifies kubeconfig files. See the individual subcommands for details.
convert
kubectl convert -f

 FILENAME [options]
Convert config files between different API versions. Both YAML and JSON formats are accepted. Note - requires
kubectl-convert
plugin to be installed.
cordon
kubectl cordon NODE [options]
Mark node as unschedulable.
cp
kubectl cp &lt;file-spec-src&gt; &lt;file-spec-dest&gt; [options]
Copy files and directories to and from containers.
create
kubectl create -f FILENAME [flags]
Create one or more resources from a file or stdin.
delete
kubectl delete (-f FILENAME | TYPE [NAME | /NAME | -l label | --all]) [flags]
Delete resources either from a file, stdin, or specifying label selectors, names, resource selectors, or resources.
describe
kubectl describe (-f FILENAME | TYPE [NAME_PREFIX | /NAME | -l label]) [flags]
Display the detailed state of one or more resources.
diff
kubectl diff -f FILENAME [flags]
Diff file or stdin against live configuration.
drain
kubectl drain NODE [options]
Drain node in preparation for maintenance.
edit
kubectl edit (-f FILENAME | TYPE NAME | TYPE/NAME) [flags]
Edit and update the definition of one or more resources on the server by using the default editor.
events
kubectl events
List events
exec
kubectl exec POD [-c CONTAINER] [-i] [-t] [flags] [-- COMMAND [args...]]
Execute a command against a container in a pod.
explain
kubectl explain [--recursive=false] [flags]
Get documentation of various resources. For instance pods, nodes, services, etc.
expose
kubectl expose (-f FILENAME | TYPE NAME | TYPE/NAME) [--port=port] [--protocol=TCP|UDP] [--target-port=number-or-name] [--name=name] [--external-ip=external-ip-of-service] [--type=type] [flags]
Expose a replication controller, service, or pod as a new Kubernetes service.
get
kubectl get (-f FILENAME | TYPE [NAME | /NAME | -l label]) [--watch] [--sort-by=FIELD] [[-o | --output]=OUTPUT_FORMAT] [flags]
List one or more resources.
kustomize
kubectl kustomize &lt;dir&gt; [flags] [options]
List a set of API resources generated from instructions in a kustomization.yaml file. The argument must be the path to the directory containing the file, or a git repository URL with a path suffix specifying same with respect to the repository root.
label
kubectl label (-f FILENAME | TYPE NAME | TYPE/NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--overwrite] [--all] [--resource-version=version] [flags]
Add or update the labels of one or more resources.
logs
kubectl logs POD [-c CONTAINER] [--follow] [flags]
Print the logs for a container in a pod.
options
kubectl options
List of global command-line options, which apply to all commands.
patch
kubectl patch (-f FILENAME | TYPE NAME | TYPE/NAME) --patch PATCH [flags]
Update one or more fields of a resource by using the strategic merge patch process.
plugin
kubectl plugin [flags] [options]
Provides utilities for interacting with plugins.
port-forward
kubectl port-forward POD [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N] [flags]
Forward one or more local ports to a pod.
proxy
kubectl proxy [--port=PORT] [--www=static-dir] [--www-prefix=prefix] [--api-prefix=prefix] [flags]
Run a proxy to the Kubernetes API server.
replace
kubectl replace -f FILENAME
Replace a resource from a file or stdin.
rollout
kubectl rollout SUBCOMMAND [options]
Manage the rollout of a resource. Valid resource types include: deployments, daemonsets and statefulsets.
run
kubectl run NAME --image=image [--env="key=value"] [--port=port] [--dry-run=server|client|none] [--overrides=inline-json] [flags]
Run a specified image on the cluster.
scale
kubectl scale (-f FILENAME | TYPE NAME | TYPE/NAME) --replicas=COUNT [--resource-version=version] [--current-replicas=count] [flags]
Update the size of the specified replication controller.
set
kubectl set SUBCOMMAND [options]
Configure application resources.
taint
kubectl taint NODE NAME KEY_1=VAL_1:TAINT_EFFECT_1 ... KEY_N=VAL_N:TAINT_EFFECT_N [options]
Update the taints on one or more nodes.
top
kubectl top [flags] [options]
Display Resource (CPU/Memory/Storage) usage.
uncordon
kubectl uncordon NODE [

options]
Mark node as schedulable.
version
kubectl version [--client] [flags]
Display the Kubernetes version running on the client and server.
wait
kubectl wait ([-f FILENAME] | resource.group/resource.name | resource.group [(-l label | --all)]) [--for=delete|--for condition=available] [options]
Experimental: Wait for a specific condition on one or many resources.
To learn more about command operations, see the
kubectl
Resource types
The following table includes a list of all the supported resource types and their abbreviated aliases.
(This output can be retrieved from
, and was accurate as of Kubernetes 1.25.0)
NAME
SHORTNAMES
APIVERSION
NAMESPACED
KIND
bindings
v1
true
Binding
componentstatuses
cs
v1
false
cm
v1
true
ConfigMap
endpoints
ep
v1
true
Endpoints
events
ev
v1
true
Event
limitranges
limits
v1
true
ns
v1
false
Namespace
nodes
no
v1
false
Node
pvc
v1
true
persistentvolumes
pv
v1
false
pods
po
v1
true
Pod
podtemplates
v1
true
replicationcontrollers
rc
v1
true
resourcequotas
quota
v1
true
secrets
v1
true
Secret
serviceaccounts
sa
v1
true
services
svc
v1
true
Service
mutatingwebhookconfigurations
admissionregistration.k8s.io/v1
false
validatingwebhookconfigurations
false
customresourcedefinitions
crd,crds
false
apiservices
apiregistration.k8s.io/v1
false
controllerrevisions
apps/v1
true
daemonsets
ds
apps/v1
true
DaemonSet
deployments
deploy
apps/v1
true
replicasets
rs
apps/v1
true
statefulsets
sts
apps/v1
true
tokenreviews
authentication.k8s.io/v1
false
localsubjectaccessreviews
authorization.k8s.io/v1
true
selfsubjectaccessreviews
false
selfsubjectrulesreviews
false
subjectaccessreviews
false
horizontalpodautoscalers
hpa
autoscaling/v2
true
cronjobs
cj
batch/v1
true
CronJob
jobs
batch/v1
true
Job
certificatesigningrequests
csr
certificates.k8s.io/v1
false
leases
coordination.k8s.io/v1
true
Lease
endpointslices
discovery.k8s.io/v1
true
events
ev
events.k8s.io/v1
true
Event
flowschemas
false
prioritylevelconfigurations
false
ingressclasses
false
ingresses
ing
true
Ingress
networkpolicies
netpol
true
runtimeclasses
node.k8s.io/v1
false
poddisruptionbudgets
pdb
policy/v1
true
podsecuritypolicies
psp
policy/v1beta1
false
PodSecurityPolicy
clusterrolebindings
false
clusterroles
false
rolebindings
true
roles
true
Role
priorityclasses
pc
scheduling.k8s.io/v1
false
csidrivers
storage.k8s.io/v1
false
CSIDriver
csinodes
false
CSINode
csistoragecapacities
true
storageclasses
sc
false
volumeattachments
false
Output options
Use the following sections for information about how you can format or sort the output of certain commands. For details about which commands support the various output options, see the
kubectl
Formatting output
The default output format for all
kubectl
commands is the human readable plain-text format. To output deta

ils to your terminal window in a specific format, you can add either the
-o
or
--output
flags to a supported
kubectl
command.
Syntax
kubectl
[
command
]
[
TYPE
]
[
NAME
]
-o &lt;output_format&gt;
Depending on the
kubectl
operation, the following output formats are supported:
Output format
-o custom-columns=&lt;spec&gt;
Print a table using a comma separated list of
custom columns
.
-o custom-columns-file=&lt;filename&gt;
Print a table using the
template in the
&lt;filename&gt;
file.
-o json
Output a JSON formatted API object.
-o jsonpath=&lt;template&gt;
Print the fields defined in a
jsonpath
expression.
-o jsonpath-file=&lt;filename&gt;
Print the fields defined by the
jsonpath
expression in the
file.
-o name
Print only the resource name and nothing else.
-o wide
Output in the plain-text format with any additional information. For pods, the node name is included.
-o yaml
Output a YAML formatted API object.
Example
In this example, the following command outputs the details for a single pod as a YAML formatted object:
kubectl get pod web-pod-13je7 -o yaml
Remember: See the
kubectl
reference documentation
for details about which output format is supported by each command.
Custom columns
To define custom columns and output only the details that you want into a table, you can use the
custom-columns
option.
You can choose to define the custom columns inline or use a template file:
or
.
Examples
Inline:
kubectl get pods &lt;pod-name&gt; -o custom-columns
=
NAME:.metadata.name,RSRC:.metadata.resourceVersion
Template file:
kubectl get pods &lt;pod-name&gt; -o custom-columns-file
=
template.txt
where the
file contains:
NAME          RSRC
metadata.name metadata.resourceVersion
The result of running either command is similar to:
NAME           RSRC
submit-queue   610995
Server-side columns
kubectl
supports receiving specific column information from the server about objects.
This means that for any given resource, the server will return columns and rows relevant to that resource, for the client to print.
This allows for consistent human-readable output across clients used against the same cluster, by having the server encapsulate the details of printing.
This feature is enabled by default. To disable it, add the
--server-print=false
flag to the
command.
Examples
To print information about the status of a pod, use a command like the following:
kubectl get pods &lt;pod-name&gt; --server-print
=
false
NAME       AGE
pod-name   1m
Sorting list objects
To output objects to a sorted list in your terminal window, you can add the
--sort-by
flag to a supported
kubectl
command. Sort your objects by specifying any numeric or string field with the
--sort-by
flag. To specify a field, use a
jsonpath
Syntax
kubectl
[
command
]
[
TYPE
]
[
NAME
]
--sort-by
=
&lt;jsonpath_exp&gt;
Example
To print a list of pods sorted by name, you run:
kubectl get pods --sort-by
=
.metadata.name
Examples: Common operations
Use the following set of examples to help you familiarize yourself with running the commonly used
kubectl
- Apply or Update a resource from a file or stdin.
# Create a service using the definition in example-service.yaml.
kubectl apply -f example-service.yaml
# Create a replication controller using the definition in example-controller.yaml.
kubectl apply -f example-controller.yaml
# Create the objects that are defined in any .yaml, .yml, or .json file within the &lt;directory&gt; directory.
kubectl apply -f &lt;directory&gt;
- List one or more resources.
# List all pods in plain-text output format.
# List all pods in plain-text output format and include additional information (such as node name).
kubectl get pods -o wide
# List the replication controller with the specified name in plain-text output format. Tip: You can shorten 

and replace the 'replicationcontroller' resource type with the alias 'rc'.
kubectl get replicationcontroller &lt;rc-name&gt;
# List all replication controllers and services together in plain-text output format.
kubectl get rc,services
# List all daemon sets in plain-text output format.
kubectl get ds
# List all pods running on node server01
kubectl get pods --field-selector
=
spec.nodeName
=
server01
- Display detailed state of one or more resources, including the uninitialized ones by default.
# Display the details of the node with name &lt;node-name&gt;.
kubectl describe nodes &lt;node-name&gt;
# Display the details of the pod with name &lt;pod-name&gt;.
kubectl describe pods/&lt;pod-name&gt;
# Display the details of all the pods that are managed by the replication controller named &lt;rc-name&gt;.
# Remember: Any pods that are created by the replication controller get prefixed with the name of the replication controller.
kubectl describe pods &lt;rc-name&gt;
# Describe all pods
Note:
The
command is usually used for retrieving one or more
resources of the same resource type. It features a rich set of flags that allows
you to customize the output format using the
-o
or
--output
flag, for example.
You can specify the
-w
or
--watch
flag to start watching updates to a particular
object. The
command is more focused on describing the many
related aspects of a specified resource. It may invoke several API calls to the
API server to build a view for the user. For example, the
command retrieves not only the information about the node, but also a summary of
the pods running on it, the events generated for the node etc.
- Delete resources either from a file, stdin, or specifying label selectors, names, resource selectors, or resources.
# Delete a pod using the type and name specified in the pod.yaml file.
kubectl delete -f pod.yaml
# Delete all the pods and services that have the label '&lt;label-key&gt;=&lt;label-value&gt;'.
kubectl delete pods,services -l &lt;label-key&gt;
=
&lt;label-value&gt;
# Delete all pods, including uninitialized ones.
kubectl delete pods --all
- Execute a command against a container in a pod.
# Get output from running 'date' from pod &lt;pod-name&gt;. By default, output is from the first container.
kubectl
exec
&lt;pod-name&gt; -- date
# Get output from running 'date' in container &lt;container-name&gt; of pod &lt;pod-name&gt;.
kubectl
exec
&lt;pod-name&gt; -c &lt;container-name&gt; -- date
# Get an interactive TTY and run /bin/bash from pod &lt;pod-name&gt;. By default, output is from the first container.
kubectl
exec
-ti &lt;pod-name&gt; -- /bin/bash
- Print the logs for a container in a pod.
# Return a snapshot of the logs from pod &lt;pod-name&gt;.
kubectl logs &lt;pod-name&gt;
# Start streaming the logs from pod &lt;pod-name&gt;. This is similar to the 'tail -f' Linux command.
kubectl logs -f &lt;pod-name&gt;
kubectl diff
- View a diff of the proposed updates to a cluster.
# Diff resources included in "pod.json".
kubectl diff -f pod.json
# Diff file read from stdin.
cat service.yaml | kubectl diff -f -
Examples: Creating and using plugins
Use the following set of examples to help you familiarize yourself with writing and using
kubectl
plugins:
# create a simple plugin in any language and name the resulting executable file
# so that it begins with the prefix "kubectl-"
cat ./kubectl-hello
#!/bin/sh
# this plugin prints the words "hello world"
echo
"hello world"
With a plugin written, let's make it executable:
chmod a+x ./kubectl-hello
# and move it to a location in our PATH
sudo mv ./kubectl-hello /usr/local/bin
sudo chown root:root /usr/local/bin
# You have now created and "installed" a kubectl plugin.
# You can begin using this plugin by invoking it from kubectl as if it were a regular command
kubectl hello
hello world
# You can "uninstall" a plugin, by removing it from the 

folder in your
# $PATH where you placed it
sudo rm /usr/local/bin/kubectl-hello
In order to view all of the plugins that are available to
kubectl
, use
the
kubectl plugin list
subcommand:
The following kubectl-compatible plugins are available:

/usr/local/bin/kubectl-hello
/usr/local/bin/kubectl-foo
/usr/local/bin/kubectl-bar
also warns you about plugins that are not
executable, or that are shadowed by other plugins; for example:
sudo chmod -x /usr/local/bin/kubectl-foo
# remove execute permission

  - warning: /usr/local/bin/kubectl-foo identified as a plugin, but it is not executable

error: one plugin warning was found
You can think of plugins as a means to build more complex functionality on top
of the existing kubectl commands:
cat ./kubectl-whoami
The next few examples assume that you already made
kubectl-whoami
have
the following contents:
#!/bin/bash
# this plugin makes use of the `kubectl config` command in order to output
# information about the current user, based on the currently selected context
kubectl config view --template
=
'{{ range .contexts }}{{ if eq .name "'
$(
)
'" }}Current user: {{ printf "%s\n" .context.user }}{{ end }}{{ end }}'
Running the above command gives you an output containing the user for the
current context in your KUBECONFIG file:
# make the file executable
sudo chmod +x ./kubectl-whoami
# and move it into your PATH
sudo mv ./kubectl-whoami /usr/local/bin
kubectl whoami
Current user: plugins-user
Read the
kubectl
reference documentation:
the kubectl
command reference
the
command line arguments
reference
kubectl
usage conventions
Read about
JSONPath support
in kubectl
Read about how to
extend kubectl with plugins
To find out more about plugins, take a look at the
example CLI plugin
.
Feedback
Yes
No
.
or
.
Last modified November 28, 2022 at 8:22 AM PST:
move events to correct place (1c26c7be36)
Syntax
Home
Blog
Training
Partners
Community
CC BY 4.0

Networking Reference | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
This section of the Kubernetes documentation provides reference details
of Kubernetes networking.
Feedback
Yes
No
.
or
.
Last modified November 24, 2022 at 6:11 PM PST:
Set page weights (b581bd417a)
Home
Blog
Training
Partners
Community
CC BY 4.0

Update Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Performing a Rolling Update
Perform a rolling update using kubectl.
Updating an application
Users expect applications to be available all the time and developers are expected to deploy new versions of them several times a day. In Kubernetes this is done with rolling updates.
Rolling updates
allow Deployments' update to take place with zero downtime by incrementally updating Pods instances with new ones. The new Pods will be scheduled on Nodes with available resources.
In the previous module we scaled our application to run multiple instances. This is a requirement for performing updates without affecting application availability. By default, the maximum number of Pods that can be unavailable during the update and the maximum number of new Pods that can be created, is one. Both options can be configured to either numbers or percentages (of Pods).
In Kubernetes, updates are versioned and any Deployment update can be reverted to a previous (stable) version.
Summary:
Updating an app
Rolling updates allow Deployments' update to take place with zero downtime by incrementally updating Pods instances with new ones.
Rolling updates overview
Previous
Next
Similar to application Scaling, if a Deployment is exposed publicly, the Service will load-balance the traffic only to available Pods during the update. An available Pod is an instance that is available to the users of the application.
Rolling updates allow the following actions:
Promote an application from one environment to another (via container image updates)
Rollback to previous versions
Continuous Integration and Continuous Delivery of applications with zero downtime
If a Deployment is exposed publicly, the Service will load-balance the traffic only to available Pods during the update.
In the following interactive tutorial, we'll update our application to a new version, and also perform a rollback.
›
2 - Interactive Tutorial - Updating Your App
&lt; Return to Module 5
Return to Kubernetes Basics
Home
Blog
Training
Partners
Community
CC BY 4.0

Lease | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 kubectl on Windows
Debug Pods
Res

ource metrics pipeline
Auditing
Run Jobs
TLS
Tutorials
Security
Re

strict a Container's Syscalls with seccomp
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configuration (v1alpha

1)
Reference
Lease
Lease
apiVersion: coordination.k8s.io/v1
import "k8s.io/api/coordination/v1"
Lease
: coordination.k8s.io/v1
kind
: Lease
metadata
(
)
More info:
spec
(
LeaseSpec
)
Specification of the Lease. More info:
LeaseSpec
LeaseSpec is a specification of a Lease.
acquireTime
(MicroTime)
acquireTime is a time when the current lease was acquired.
MicroTime is version of Time with microsecond level precision.
holderIdentity
(string)
holderIdentity contains the identity of the holder of a current lease.
leaseDurationSeconds
(int32)
leaseDurationSeconds is a duration that candidates for a lease need to wait to force acquire it. This is measure against time of last observed RenewTime.
leaseTransitions
(int32)
leaseTransitions is the number of transitions of a lease between holders.
renewTime
renewTime is a time when the current holder of a lease has last updated the lease.
LeaseList
LeaseList is a list of Lease objects.
kind
: LeaseList
metadata
(
ListMeta
)
items
([]
Lease
Items is a list of schema objects.
get
read the specified Lease
GET /apis/coordination.k8s.io/v1/namespaces/{namespace}/leases/{name}
name
(
in path
name of the Lease
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
list
list or watch objects of kind Lease
GET /apis/coordination.k8s.io/v1/namespaces/{namespace}/leases
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
list
HTTP Requ

est
GET /apis/coordination.k8s.io/v1/leases
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
create
create a Lease
POST /apis/coordination.k8s.io/v1/namespaces/{namespace}/leases
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
202 (
Lease
update
replace the specified Lease
PUT /apis/coordination.k8s.io/v1/namespaces/{namespace}/leases/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
patch
partially update the specified Lease
PATCH /apis/coordination.k8s.io/v1/namespaces/{namespace}/leases/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
delete
delete a Lease
DELETE /apis/coordination.k8s.io/v1/namespaces/{namespace}/leases/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Lease
DELETE /apis/coordination.k8s.io/v1/namespaces/{namespace}/leases
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Lease
LeaseSpec
LeaseList
get
r

ead the specified Lease
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

The Kubernetes API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Agg

regation Layer
Tasks
Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security 

Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

rver Configuration (v1)
Concepts
Overview
The Kubernetes API lets you query and manipulate the state of objects in Kubernetes. The core of Kubernetes' control plane is the API server and the HTTP API that it exposes. Users, the different parts of your cluster, and external components all communicate with one another through the API server.
The core of Kubernetes'
is the
API server
. The API server
exposes an HTTP API that lets end users, different parts of your cluster, and
external components communicate with one another.
The Kubernetes API lets you query and manipulate the state of API objects in Kubernetes
(for example: Pods, Namespaces, ConfigMaps, and Events).
Most operations can be performed through the
kubectl
command-line interface or other
command-line tools, such as
kubeadm
, which in turn use the
API. However, you can also access the API directly using REST calls.
Consider using one of the
if you are writing an application using the Kubernetes API.
OpenAPI specification
Complete API details are documented using
OpenAPI
.
OpenAPI V2
The Kubernetes API server serves an aggregated OpenAPI v2 spec via the
/openapi/v2
endpoint. You can request the response format using
request headers as follows:
Valid request header values for OpenAPI v2 queries
Header
Possible values
Notes
Accept-Encoding
gzip
not supplying this header is also acceptable
Accept
application/com.github.proto-openapi.spec.v2@v1.0+protobuf
mainly for intra-cluster use
application/json
default
*
serves
Kubernetes implements an alternative Protobuf based serialization format that
is primarily intended for intra-cluster communication. For more information
about this format, see the
Kubernetes Protobuf serialization
design proposal and the
Interface Definition Language (IDL) files for each schema located in the Go
packages that define the API objects.
OpenAPI V3
Kubernetes v1.24 [beta]
Kubernetes v1.26 offers beta support for publishing its APIs as OpenAPI v3; this is a
beta feature that is enabled by default.
You can disable the beta feature by turning off the
named
OpenAPIV3
for the kube-apiserver component.
A discovery endpoint
/openapi/v3
is provided to see a list of all
group/versions available. This endpoint only returns JSON. These group/versions
are provided in the following format:
{
"paths":
{
...,
"api/v1":
{
"serverRelativeURL":
"/openapi/v3/api/v1?hash=CC0E9BFD992D8C59AEC98A1E2336F899E8318D3CF4C

68944C3DEC640AF5AB52D864AC50DAA8D145B3494F75FA3CFF939FCBDDA431DAD3CA79738B297795818CF"
},
"apis/admissionregistration.k8s.io/v1":
{
"/openapi/v3/apis/admissionregistration.k8s.io/v1?hash=E19CC93A116982CE5422FC42B590A8AFAD92CDE9AE4D59B5CAAD568F083AD07946E6CB5817531680BCE6E215C16973CD39003B0425F3477CFD854E89A9DB6597"
},
....
}
}
The relative URLs are pointing to immutable OpenAPI descriptions, in
order to improve client-side caching. The proper HTTP caching headers
are also set by the API server for that purpose (
Expires
to 1 year in
the future, and
Cache-Control
to
immutable
). When an obsolete URL is
used, the API server returns a redirect to the newest URL.
The Kubernetes API server publishes an OpenAPI v3 spec per Kubernetes
group version at the
/openapi/v3/apis/&lt;group&gt;/&lt;version&gt;?hash=&lt;hash&gt;
endpoint.
Refer to the table below for accepted request headers.
Valid request header values for OpenAPI v3 queries
Header
Notes
gzip
Accept
application/com.github.proto-openapi.spec.v3@v1.0+protobuf
default
*
serves
Persistence
Kubernetes stores the serialized state of objects by writing them into
etcd
.
API groups and versioning
To make it easier to eliminate fields or restructure resource representations,
Kubernetes supports multiple API versions, each at a different API path, such
as
/api/v1
or
/apis/rbac.authorization.k8s.io/v1alpha1
.
Versioning is done at the API level rather than at the resource or field level
to ensure that the API presents a clear, consistent view of system resources
and behavior, and to enable controlling access to end-of-life and/or
experimental APIs.
To make it easier to evolve and to extend its API, Kubernetes implements
API groups
that can be
enabled or disabled
.
API resources are distinguished by their API group, resource type, namespace
(for namespaced resources), and name. The API server handles the conversion between
API versions transparently: all the different versions are actually representations
of the same persisted data. The API server may serve the same underlying data
through multiple API versions.
For example, suppose there are two API versions,
v1
and
v1beta1
, for the same
resource. If you originally created an object using the
v1beta1
version of its
API, you can later read, update, or delete that object using either the
v1beta1
or the
v1
API version, until the
v1beta1
version is deprecated and removed.
At that point you can continue accessing and modifying the object using the
v1
API.
API changes
Any system that is successful needs to grow and change as new use cases emerge or existing ones change.
Therefore, Kubernetes has designed the Kubernetes API to continuously change and grow.
The Kubernetes project aims to
not
break compatibility with existing clients, and to maintain that
compatibility for a length of time so that other projects have an opportunity to adapt.
In general, new API resources and new resource fields can be added often and frequently.
Elimination of resources or fields requires following the
API deprecation policy
.
Kubernetes makes a strong commitment to maintain compatibility for official Kubernetes APIs
once they reach general availability (GA), typically at API version
v1
. Additionally,
Kubernetes maintains compatibility with data persisted via
beta
API versions of official Kubernetes APIs,
and ensures that data can be converted and accessed via GA API versions when the feature goes stable.
If you adopt a beta API version, you will need to transition to a subsequent beta or stable API version
once the API graduates. The best time to do this is while the beta API is in its deprecation period,
since objects are simultaneously accessible via both API versions. Once the beta API completes its
deprecation period and is no longer served, the replacement API version must be used.
Note:
Although 

Kubernetes also aims to maintain compatibility for
alpha
APIs versions, in some
circumstances this is not possible. If you use any alpha API versions, check the release notes
for Kubernetes when upgrading your cluster, in case the API did change in incompatible
ways that require deleting all existing alpha objects prior to upgrade.
Refer to
API versions reference
for more details on the API version level definitions.
API Extension
The Kubernetes API can be extended in one of two ways:
Custom resources
let you declaratively define how the API server should provide your chosen resource API.
You can also extend the Kubernetes API by implementing an
aggregation layer
.
Learn how to extend the Kubernetes API by adding your own
.
Controlling Access To The Kubernetes API
describes
how the cluster manages authentication and authorization for API access.
Learn about API endpoints, resource types and samples by reading
.
Learn about what constitutes a compatible change, and how to change the API, from
.
Feedback
Yes
No
.
or
.
Last modified October 24, 2022 at 12:03 PM PST:
Change weight for localization correctness (95683e0b2e)
OpenAPI V2
OpenAPI V3
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubeadm | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Kubeadm
1:
Kubeadm Generated
1.1:
1.2:
1.3:
1.4:
1.5:
1.6:
1.7:
1.8:
1.9:
1.10:
1.11:
1.12:
1.13:
1.14:
1.15:
1.16:
1.17:
1.18:
1.19:
1.20:
1.21:
1.22:
1.23:
1.24:
1.25:
1.26:
1.27:
1.28:
1.29:
1.30:
1.31:
1.32:
1.33:
1.34:
1.35:
1.36:
1.37:
1.38:
1.39:
1.40:
1.41:
1.42:
1.43:
1.44:
1.45:
1.46:
1.47:
1.48:
1.49:
1.50:
1.51:
1.52:
1.53:
1.54:
1.55:
1.56:
1.57:
1.58:
1.59:
1.60:
1.61:
1.62:
1.63:
1.64:
1.65:
1.66:
1.67:
1.68:
1.69:
1.70:
1.71:
1.72:
1.73:
1.74:
1.75:
1.76:
1.77:
1.78:
1.79:
1.80:
1.81:
1.82:
1.83:
1.84:
1.85:
1.86:
1.87:
1.88:
1.89:
1.90:
1.91:
1.92:
1.93:
1.94:
1.95:
1.96:
1.97:
1.98:
1.99:
1.100:
1.101:
1.102:
1.103:
1.104:
1.105:
1.106:
1.107:
1.108:
1.109:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
Kubeadm is a tool built to provide
and
as best-practice "fast paths" for creating Kubernetes clusters.
kubeadm performs the actions necessary to get a minimum viable cluster up and running. By design, it cares only about bootstrapping, not about provisioning machines. Likewise, installing various nice-to-have addons, like the Kubernetes Dashboard, monitoring solutions, and cloud-specific addons, is not in scope.
Instead, we expect higher-level and more tailored tooling to be built on top of kubeadm, and ideally, using kubeadm as the basis of all deployments will make it easier to create conformant clusters.
How to install
To install kubeadm, see the
installation guide
.
to bootstrap a Kubernetes control-plane node
to bootstrap a Kubernetes worker node and join it to the cluster
to upgrade a Kubernetes cluster to a newer version
if you initialized your cluster using kubeadm v1.7.x or lower, to configure your cluster for
to manage tokens for
to revert any changes made to this host by
or
to manage Kubernetes certificates
to manage kubeconfig files
to print the kubeadm version
to preview a set of features made available for gathering feedback from the community
1 - Kubeadm Generated
1.1 -
kubeadm: easily bootstrap a secure Kubernetes cluster
Synopsis
┌──────────────────────────────────────────────────────────┐
│ KUBEADM                                                  │
│ Easily bootstrap a secure Kubernetes cluster             │
│                                                          │
│ Please give us feedback at:                              │
│ https://github.com/kubernetes/kubeadm/issues             │
└──────────────────────────────────────────────────────────┘
Create a two-machine cluster with one control-plane node
(which controls the cluster), and one worker node
(where your workloads, like Pods and Deployments run).

│ On the first machine:                                    │
├──────────────────────────────────────────────────────────┤
│ control-plane# kubeadm init                              │

│ On the second machine:                                   │
│ worker# kubeadm join 

&amp;lt;arguments-returned-from-init&amp;gt;      │

You can then repeat the second step on as many other machines as you like.
Options
-h, --help
help for kubeadm
--rootfs string
[EXPERIMENTAL] The path to the 'real' host root filesystem.
1.2 -
Commands related to handling kubernetes certificates
Synopsis
kubeadm certs [flags]
Options
-h, --help
help for certs
Options inherited from parent commands
1.3 -
Generate certificate keys
Synopsis
This command will print out a secure randomly-generated certificate key that can be used with
the "init" command.
You can also use "kubeadm init --upload-certs" without specifying a certificate key and it will
generate and print one for you.
kubeadm certs certificate-key [flags]
Options
-h, --help
help for certificate-key
1.4 -
Check certificates expiration for a Kubernetes cluster
Synopsis
Checks expiration for the certificates in the local PKI managed by kubeadm.
kubeadm certs check-expiration [flags]
Options
--cert-dir string     Default: "/etc/kubernetes/pki"
The path where to save the certificates
--config string
Path to a kubeadm configuration file.
-h, --help
help for check-expiration
--kubeconfig string     Default: "/etc/kubernetes/admin.conf"
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.
1.5 -
Generate keys and certificate signing requests
Synopsis
Generates keys and certificate signing requests (CSRs) for all the certificates required to run the control plane. This command also generates partial kubeconfig files with private key data in the "users &gt; user &gt; client-key-data" field, and for each kubeconfig file an accompanying ".csr" file is created.
This command is designed for use in
Kubeadm External CA Mode
. It generates CSRs which you can then submit to your external certificate authority for signing.
The PEM encoded signed certificates should then be saved alongside the key files, using ".crt" as the file extension, or in the case of kubeconfig files, the PEM encoded signed certificate should be base64 encoded and added to the kubeconfig file in the "users &gt; user &gt; client-certificate-data" field.
kubeadm certs generate-csr [flags]
Examples
# The following command will generate keys and CSRs for all control-plane certificates and kubeconfig files:
  kubeadm certs generate-csr --kubeconfig-dir /tmp/etc-k8s --cert-dir /tmp/etc-k8s/pki
Options
--cert-dir string
-h, --help
help for generate-csr
--kubeconfig-dir string     Default: "/etc/kubernetes"
The path where to save the kubeconfig file.
1.6 -
Renew certificates for a Kubernetes cluster
Synopsis
This command is not meant to be run on its own. See list of available subcommands.
kubeadm certs renew [flags]
Options
-h, --help
help for renew
1.7 -
Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself
Synopsis
Renew the certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself.
Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them.
Renewal by default tries to use the certificate au

thority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request.
After renewal, in order to make changes effective, is required to restart control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere.
kubeadm certs renew admin.conf [flags]
Options
-h, --help
help for admin.conf
1.8 -
Renew all available certificates
Synopsis
Renew all known certificates necessary to run the control plane. Renewals are run unconditionally, regardless of expiration date. Renewals can also be run individually for more control.
kubeadm certs renew all [flags]
Options
-h, --help
help for all
1.9 -
Renew the certificate the apiserver uses to access etcd
Synopsis
Renew the certificate the apiserver uses to access etcd.
Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request.
kubeadm certs renew apiserver-etcd-client [flags]
Options
-h, --help
help for apiserver-etcd-client
1.10 -
Renew the certificate for the API server to connect to kubelet
Synopsis
Renew the certificate for the API server to connect to kubelet.
kubeadm certs renew apiserver-kubelet-client [flags]
Options
-h, --help
help for apiserver-kubelet-c

lient
1.11 -
Renew the certificate for serving the Kubernetes API
Synopsis
Renew the certificate for serving the Kubernetes API.
kubeadm certs renew apiserver [flags]
Options
-h, --help
help for apiserver
1.12 -
Renew the certificate embedded in the kubeconfig file for the controller manager to use
Synopsis
Renew the certificate embedded in the kubeconfig file for the controller manager to use.
kubeadm certs renew controller-manager.conf [flags]
Options
-h, --help
help for controller-manager.conf
1.13 -
Renew the certificate for liveness probes to healthcheck etcd
Synopsis
Renew the certificate for liveness probes to healthcheck etcd.
kubeadm certs renew etcd-healthcheck-client [flags]
Options
-h, --help
help for etcd-healthcheck-client
--kubeconf

ig string     Default: "/etc/kubernetes/admin.conf"
1.14 -
Renew the certificate for etcd nodes to communicate with each other
Synopsis
Renew the certificate for etcd nodes to communicate with each other.
kubeadm certs renew etcd-peer [flags]
Options
-h, --help
help for etcd-peer
1.15 -
Renew the certificate for serving etcd
Synopsis
Renew the certificate for serving etcd.
kubeadm certs renew etcd-server [flags]
Options
-h, --help
help for etcd-server
1.16 -
Renew the certificate for the front proxy client
Synopsis
Renew the certificate for the front proxy client.
kubeadm certs renew front-proxy-client [flags]
Options
-h, --help
help for front-proxy-client
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of sta

ndard locations can be searched for an existing kubeconfig file.
1.17 -
Renew the certificate embedded in the kubeconfig file for the scheduler manager to use
Synopsis
Renew the certificate embedded in the kubeconfig file for the scheduler manager to use.
kubeadm certs renew scheduler.conf [flags]
Options
-h, --help
help for scheduler.conf
1.18 -
Output shell completion code for the specified shell (bash or zsh)
Synopsis
The shell code must be evaluated to provide interactive
completion of kubeadm commands. This can be done by sourcing it from
the .bash_profile.
Note: this requires the bash-completion framework.
To install it on Mac use homebrew:
$ brew install bash-completion
Once installed, bash_completion must be evaluated. This can be done by adding the
following line to the .bash_profile
$ source $(brew --prefix)/etc/bash_completion
If bash-completion is not installed on Linux, please install the 'bash-completion' package
via your distribution's package manager.
Note for zsh users: [1] zsh completions are only supported in versions of zsh &gt;= 5.2
kubeadm completion SHELL [flags]
Examples
# Install bash completion on a Mac using homebrew
brew install bash-completion
printf "\n# Bash completion support\nsource $(brew --prefix)/etc/bash_completion\n" &gt;&gt; $HOME/.bash_profile
source $HOME/.bash_profile

# Load the kubeadm completion code for bash into the current shell
source &lt;(kubeadm completion bash)

# Write bash completion code to a file and source it from .bash_profile
kubeadm completion bash &gt; ~/.kube/kubeadm_completion.bash.inc
printf "\n# Kubeadm shell completion\nsource '$HOME/.kube/kubeadm_completion.bash.inc'\n" &gt;&gt; $HOME/.bash_profile

# Load the kubeadm completion code for zsh[1] into the current shell
source &lt;(kubeadm completion zsh)
Options
-h, --help
help for completion
1.19 -
Manage configuration for a kubeadm cluster persisted in a ConfigMap in the cluster
Synopsis
There is a ConfigMap in the kube-system namespace called "kubeadm-config" that kubeadm uses to store internal configuration about the
cluster. kubeadm CLI v1.8.0+ automatically creates this ConfigMap with the config used with 'kubeadm init', but if you
initialized your cluster using kubeadm v1.7.x or lower, you must use the 'config upload' command to create this
ConfigMap. This is required so that 'kubeadm upgrade' can configure your upgraded cluster correctly.
kubeadm config [flags]
Options
-h, --help
help for config
The kubeconfig file to use when talking to the cluster. If the flag is 

not set, a set of standard locations can be searched for an existing kubeconfig file.
1.20 -
Interact with container images used by kubeadm
Synopsis
kubeadm config images [flags]
Options
-h, --help
help for images
1.21 -
Print a list of images kubeadm will use. The configuration file is used in case any images or image repositories are customized
Synopsis
kubeadm config images list [flags]
Options
--allow-missing-template-keys     Default: true
If true, ignore any errors in templates when a field or map key is missing in the template. Only applies to golang and jsonpath output formats.
-o, --experimental-output string     Default: "text"
Output format. One of: text|json|yaml|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file.
--feature-gates string
A set of key=value pairs that describe feature gates for various features. Options are:
PublicKeysECDSA=true|false (ALPHA - default=false)
RootlessControlPlane=true|false (ALPHA - default=false)
-h, --help
help for list
--image-repository string     Default: "registry.k8s.io"
Choose a container registry to pull control plane images from
--kubernetes-version string     Default: "stable-1"
Choose a specific Kubernetes version for the control plane.
--show-managed-fields
If true, keep the managedFields when printing objects in JSON or YAML format.
1.22 -
Pull images used by kubeadm
Synopsis
kubeadm config images pull [flags]
Options
--cri-socket string
Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.
-h, --help
help for pull
1.23 -
Read an older version of the kubeadm configuration API types from a file, and output the similar config object for the newer version
Synopsis
This command lets you convert configuration objects of older versions to the latest supported version,
locally in the CLI tool without ever touching anything in the cluster.
In this version of kubeadm, the following API versions are supported:
Further, kubeadm can only write out 

config of version "kubeadm.k8s.io/v1beta3", but read both types.
So regardless of what version you pass to the --old-config parameter here, the API object will be
read, deserialized, defaulted, converted, validated, and re-serialized when written to stdout or
--new-config if specified.
In other words, the output of this command is what kubeadm actually would read internally if you
submitted this file to "kubeadm init"
kubeadm config migrate [flags]
Options
-h, --help
help for migrate
--new-config string
Path to the resulting equivalent kubeadm config file using the new API version. Optional, if not specified output will be sent to STDOUT.
--old-config string
Path to the kubeadm config file that is using an old API version and should be converted. This flag is mandatory.
1.24 -
Print configuration
Synopsis
This command prints configurations for subcommands provided.
For details, see:
https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#section-directories
kubeadm config print [flags]
Options
-h, --help
help for print
1.25 -
Print default init configuration, that can be used for 'kubeadm init'
Synopsis
This command prints objects such as the default init configuration that is used for 'kubeadm init'.
Note that sensitive values like the Bootstrap Token fields are replaced with placeholder values like "abcdef.0123456789abcdef" in order to pass validation but
not perform the real computation for creating a token.
kubeadm config print init-defaults [flags]
Options
--component-configs strings
A comma-separated list for component config API objects to print the default values for. Available values: [KubeProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed.
-h, --help
help for init-defaults
1.26 -
Print default join configuration, that can be used for 'kubeadm join'
Synopsis
This command prints objects such as the default join configuration that is used for 'kubeadm join'.
kubeadm config print join-defaults [flags]
Options
-h, --help
help for join-defaults
1.27 -
Run this command in order to set up the Kubernetes control plane
Synopsis
The "init" co

mmand executes the following phases:
preflight                    Run pre-flight checks
certs                        Certificate generation
  /ca                          Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components
  /apiserver                   Generate the certificate for serving the Kubernetes API
  /apiserver-kubelet-client    Generate the certificate for the API server to connect to kubelet
  /front-proxy-ca              Generate the self-signed CA to provision identities for front proxy
  /front-proxy-client          Generate the certificate for the front proxy client
  /etcd-ca                     Generate the self-signed CA to provision identities for etcd
  /etcd-server                 Generate the certificate for serving etcd
  /etcd-peer                   Generate the certificate for etcd nodes to communicate with each other
  /etcd-healthcheck-client     Generate the certificate for liveness probes to healthcheck etcd
  /apiserver-etcd-client       Generate the certificate the apiserver uses to access etcd
  /sa                          Generate a private key for signing service account tokens along with its public key
kubeconfig                   Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
  /admin                       Generate a kubeconfig file for the admin to use and for kubeadm itself
  /kubelet                     Generate a kubeconfig file for the kubelet to use *only* for cluster bootstrapping purposes
  /controller-manager          Generate a kubeconfig file for the controller manager to use
  /scheduler                   Generate a kubeconfig file for the scheduler to use
kubelet-start                Write kubelet settings and (re)start the kubelet
control-plane                Generate all static Pod manifest files necessary to establish the control plane
  /apiserver                   Generates the kube-apiserver static Pod manifest
  /controller-manager          Generates the kube-controller-manager static Pod manifest
  /scheduler                   Generates the kube-scheduler static Pod manifest
etcd                         Generate static Pod manifest file for local etcd
  /local                       Generate the static Pod manifest file for a local, single-node local etcd instance
upload-config                Upload the kubeadm and kubelet configuration to a ConfigMap
  /kubeadm                     Upload the kubeadm ClusterConfiguration to a ConfigMap
  /kubelet                     Upload the kubelet component config to a ConfigMap
upload-certs                 Upload certificates to kubeadm-certs
mark-control-plane           Mark a node as a control-plane
bootstrap-token              Generates bootstrap tokens used to join a node to a cluster
kubelet-finalize             Updates settings relevant to the kubelet after TLS bootstrap
  /experimental-cert-rotation  Enable kubelet client certificate rotation
addon                        Install required addons for passing conformance tests
  /coredns                     Install the CoreDNS addon to a Kubernetes cluster
  /kube-proxy                  Install the kube-proxy addon to a Kubernetes cluster
show-join-command            Show the join command for control-plane and worker node
kubeadm init [flags]
Options
--apiserver-advertise-address string
The IP address the API Server will advertise it's listening on. If not set the default network interface will be used.
--apiserver-bind-port int32     Default: 6443
Port for the API Server to bind to.
--apiserver-cert-extra-sans strings
Optional extra Subject Alternative Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
The path where to save and store the certificates.
--certificate-key string
Key used to encrypt the control-plane certificates in the kubeadm-certs Secret.
Path to a 

kubeadm configuration file.
--control-plane-endpoint string
Specify a stable IP address or DNS name for the control plane.
--dry-run
Don't apply any changes; just output what would be done.
-h, --help
help for init
--ignore-preflight-errors strings
A list of checks whose errors will be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
--node-name string
Specify the node name.
--patches string
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
--pod-network-cidr string
Specify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
--service-cidr string     Default: "10.96.0.0/12"
Use alternative range of IP address for service VIPs.
--service-dns-domain string     Default: "cluster.local"
Use alternative domain for services, e.g. "myorg.internal".
--skip-certificate-key-print
Don't print the key used to encrypt the control-plane certificates.
--skip-phases strings
List of phases to be skipped
--skip-token-print
Skip printing of the default bootstrap token generated by 'kubeadm init'.
--token string
The token to use for establishing bidirectional trust between nodes and control-plane nodes. The format is [a-z0-9]{6}.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
--token-ttl duration     Default: 24h0m0s
The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire
--upload-certs
Upload control-plane certificates to the kubeadm-certs Secret.
1.28 -
Use this command to invoke single phase of the init workflow
Synopsis
Options
-h, --help
help for phase
1.29 -
Install required addons for passing conformance tests
Synopsis
kubeadm init phase addon [flags]
Options
-h, --help
help for addon
1.30 -
Install all the addons
Synopsis
kubeadm init phase addon all [flags]
Options
--dry-run
Don't apply any changes; just output what would be d

one.
-h, --help
1.31 -
Install the CoreDNS addon to a Kubernetes cluster
Synopsis
Install the CoreDNS addon components via the API server. Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed.
kubeadm init phase addon coredns [flags]
Options
--dry-run
-h, --help
help for coredns
--print-manifest
Print the addon manifests to STDOUT instead of installing them
1.32 -
Install the kube-proxy addon to a Kubernetes cluster
Synopsis
Install the kube-proxy addon components via the API server.
kubeadm init phase addon kube-proxy [flags]
Options
--dry-run
-h, --help
help for kube-proxy
Print the addon mani

fests to STDOUT instead of installing them
1.33 -
Generates bootstrap tokens used to join a node to a cluster
Synopsis
Bootstrap tokens are used for establishing bidirectional trust between a node joining the cluster and a control-plane node.
This command makes all the configurations required to make bootstrap tokens works and then creates an initial token.
kubeadm init phase bootstrap-token [flags]
Examples
# Make all the bootstrap token configurations and create an initial token, functionally
  # equivalent to what generated by kubeadm init.
  kubeadm init phase bootstrap-token
Options
--dry-run
-h, --help
help for bootstrap-token
1.34 -
Certificate generation
Synopsis
kubeadm init phase certs [flags]
Options
-h, --help
1.35 -
Generate all certificates
Synopsis
kubeadm init phase certs all [flags]
Options
--dry-run
-h, --help
1.36 -
Generate the certificate the apiserver uses to access etcd
Synopsis
Generate the certificate the apiserver uses to access etcd, and save them into apiserver-etcd-client.crt and apiserver-etcd-client.key files.
If both files already exist, kubeadm skips the generation step and existing files will be used.
Alpha Disclaimer: this command is currently alpha.
kubeadm init phase certs apiserver-etcd-client [flags]
Options
--dry-run
-h, --help
1.37 -
Generate the certificate for the API server to connect to kubelet
Synopsis
Generate the certificate for the API server to connect to kubelet, and save them into apiserver-kubelet-client.crt and apiserver-kubelet-client.key files.
If both file

s already exist, kubeadm skips the generation step and existing files will be used.
kubeadm init phase certs apiserver-kubelet-client [flags]
Options
--dry-run
-h, --help
help for apiserver-kubelet-client
1.38 -
Generate the certificate for serving the Kubernetes API
Synopsis
Generate the certificate for serving the Kubernetes API, and save them into apiserver.crt and apiserver.key files.
kubeadm init phase certs apiserver [flags]
Options
--dry-run
-h, --help
1.39 -
Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components
Synopsis
Generate the self-signed Kubernetes CA to provision identities for other Kubernetes components, and save them into ca.crt and ca.key files.
kubeadm init phase certs ca [flags]
Options
--dry-run
-h, --help
help for ca
1.40 -
Generate the self-signed CA to provision identities for etcd
Synopsis
Generate the self-signed CA to provision identities for etcd, and save them into etcd/ca.crt and etcd/ca.key files.
kubeadm init phase certs etcd-ca [flags]
Options
--dry-run
-h, --help
help for etcd-ca
1.41 -

Generate the certificate for liveness probes to healthcheck etcd
Synopsis
Generate the certificate for liveness probes to healthcheck etcd, and save them into etcd/healthcheck-client.crt and etcd/healthcheck-client.key files.
kubeadm init phase certs etcd-healthcheck-client [flags]
Options
--dry-run
-h, --help
1.42 -
Generate the certificate for etcd nodes to communicate with each other
Synopsis
Generate the certificate for etcd nodes to communicate with each other, and save them into etcd/peer.crt and etcd/peer.key files.
Default SANs are localhost, 127.0.0.1, 127.0.0.1, ::1
kubeadm init phase certs etcd-peer [flags]
Options
--dry-run
-h, --help
1.43 -
Generate the certificate for serving etcd
Synopsis
Generate the certificate for serving etcd, and save them into etcd/server.crt and etcd/server.key files.
kubeadm init phase certs etcd-server [flags]
Options
--dry-run
-h, --help
1.44 -
Generate the self-signed CA to provision identities for front proxy
Synopsis
Generate the self-signed CA to provision identities for front proxy, and save them into front-proxy-ca.crt and front-proxy-ca.key files.
kubeadm init phase certs front-proxy-ca [flags]
Options
--dry-run
-h, --help
help for front-proxy-ca
1.45 -
Generate the certificate for the front proxy client
Synopsis
Generate the certificate for the front proxy client, and save them into front-proxy-client.crt and front-proxy-client.key files.
Alpha Dis

claimer: this command is currently alpha.
kubeadm init phase certs front-proxy-client [flags]
Options
--dry-run
-h, --help
1.46 -
Generate a private key for signing service account tokens along with its public key
Synopsis
Generate the private key for signing service account tokens along with its public key, and save them into sa.key and sa.pub files. If both files already exist, kubeadm skips the generation step and existing files will be used.
kubeadm init phase certs sa [flags]
Options
-h, --help
help for sa
1.47 -
Generate all static Pod manifest files necessary to establish the control plane
Synopsis
kubeadm init phase control-plane [flags]
Options
-h, --help
help for control-plane
1.48 -
Generate all static Pod manifest files
Synopsis
kubeadm init phase control-plane all [flags]
Examples
# Generates all static Pod manifest files for control plane components,
  # functionally equivalent to what is generated by kubeadm init.
  kubeadm init phase control-plane all
  
  # Generates all static Pod manifest files using options read from a configuration file.
  kubeadm init phase control-plane all --config config.yaml
Options
--apiserver-extra-args &lt;comma-separated 'key=value' pairs&gt;
A set of extra flags to pass to the API Server or override default ones in form of &lt;flagname&gt;=&lt;value&gt;
--controller-manager-extra-args &lt;comma-separated 'key=value' pairs&gt;
A set of extra flags to pass to the Controller Manager or override default ones in form of &lt;flagname&gt;=&lt;value&gt;
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "

yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
--scheduler-extra-args &lt;comma-separated 'key=value' pairs&gt;
A set of extra flags to pass to the Scheduler or override default ones in form of &lt;flagname&gt;=&lt;value&gt;
1.49 -
Generates the kube-apiserver static Pod manifest
Synopsis
kubeadm init phase control-plane apiserver [flags]
Options
--dry-run
-h, --help
1.50 -
Generates the kube-controller-manager static Pod manifest
Synopsis
kubeadm init phase control-plane controller-manager [flags]
Options
--dry-run
-h, --help
help for controller-manager
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube

-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.51 -
Generates the kube-scheduler static Pod manifest
Synopsis
kubeadm init phase control-plane scheduler [flags]
Options
--dry-run
-h, --help
help for scheduler
1.52 -
Generate static Pod manifest file for local etcd
Synopsis
kubeadm init phase etcd [flags]
Options
-h, --help
help for etcd
1.53 -
Generate the static Pod manifest file for a local, single-node local etcd instance
Synopsis
kubeadm init phase etcd local [flags]
Examples
# Generates the static Pod manifest file for etcd, functionally
  # equivalent to what is generated by kubeadm init.
  kubeadm init phase etcd local
  
  # Generates the static Pod manifest file for etcd using options
  # read from a configuration file.
  kubeadm init phase etcd local --config config.yaml
Options
--dry-run
-h, --help
help for local
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically

.
1.54 -
Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
Synopsis
kubeadm init phase kubeconfig [flags]
Options
-h, --help
help for kubeconfig
1.55 -
Generate a kubeconfig file for the admin to use and for kubeadm itself
Synopsis
Generate the kubeconfig file for the admin and for kubeadm itself, and save it to admin.conf file.
kubeadm init phase kubeconfig admin [flags]
Options
--dry-run
-h, --help
help for admin
1.56 -
Generate all kubeconfig files
Synopsis
kubeadm init phase kubeconfig all [flags]
Options
--dry-run
-h, --help
1.57 -
Generate a kubeconfig file for the controller manager to use
Synopsis
Generate the kubeconfig file for the controller manager to use and save it to controller-manager.conf file
kubeadm init phase kubeconfig controller-manager [flags]
Options
--dry-run
-h, --help
1.58 -
Generate a kubeconfig file for the kubelet to use
only
for cluster bootstrapping purpose

s
Synopsis
Generate the kubeconfig file for the kubelet to use and save it to kubelet.conf file.
Please note that this should
only
be used for cluster bootstrapping purposes. After your control plane is up, you should request all kubelet credentials from the CSR API.
kubeadm init phase kubeconfig kubelet [flags]
Options
--dry-run
-h, --help
help for kubelet
1.59 -
Generate a kubeconfig file for the scheduler to use
Synopsis
Generate the kubeconfig file for the scheduler to use and save it to scheduler.conf file.
kubeadm init phase kubeconfig scheduler [flags]
Options
--dry-run
-h, --help
1.60 -
Updates settings relevant to the kubelet after TLS bootstrap
Synopsis
kubeadm init phase kubelet-finalize [flags]
Examples
# Updates settings relevant to the kubelet after TLS bootstrap"
  kubeadm init phase kubelet-finalize all --config
Options
-h, --help
help for kubelet-finalize
1.61 -
Run all kubelet-finalize phases
Synopsis
kubeadm init phase kubelet-finalize all [flags]
Examples
Options
--dry-run
-h, --help
1.62 -
Enable kubelet client certificate rotation
Synopsis
kubeadm init phase kubelet-finalize experimental-cert-rotation [flags]
Options
--dry-run
-h, --help
help for experimental-cert-rotation
--rootfs strin

g
1.63 -
Write kubelet settings and (re)start the kubelet
Synopsis
Write a file with KubeletConfiguration and an environment file with node specific kubelet settings, and then (re)start kubelet.
kubeadm init phase kubelet-start [flags]
Examples
# Writes a dynamic environment file with kubelet flags from a InitConfiguration file.
  kubeadm init phase kubelet-start --config config.yaml
Options
--dry-run
-h, --help
help for kubelet-start
1.64 -
Mark a node as a control-plane
Synopsis
kubeadm init phase mark-control-plane [flags]
Examples
# Applies control-plane label and taint to the current node, functionally equivalent to what executed by kubeadm init.
  kubeadm init phase mark-control-plane --config config.yaml
  
  # Applies control-plane label and taint to a specific node
  kubeadm init phase mark-control-plane --node-name myNode
Options
--dry-run
-h, --help
help for mark-control-plane
1.65 -
Run pre-flight checks
Synopsis
Run pre-flight checks for kubeadm init.
kubeadm init phase preflight [flags]
Examples
# Run pre-flight checks for kubeadm init using a config file.
  kubeadm init phase preflight --config kubeadm-config.yaml
Options
--dry-run
-h, --help
help for preflight
1.66 -
Show the join command for control-plane and worker node
Synopsis
kubeadm init phase show-join-command [flags]
Options
-h, --help
help for show-join-command
1.67 -
Upload certificates to kubeadm-certs
Synopsis
Upload control plane certificates to the kubeadm-certs Secret
kubeadm init phase upload-certs [flags]
Options
--dry-run
-h, --help
help for upload-certs
--skip

-certificate-key-print
1.68 -
Upload the kubeadm and kubelet configuration to a ConfigMap
Synopsis
kubeadm init phase upload-config [flags]
Options
-h, --help
help for upload-config
1.69 -
Upload all configuration to a config map
Synopsis
kubeadm init phase upload-config all [flags]
Options
--dry-run
-h, --help
1.70 -
Upload the kubeadm ClusterConfiguration to a ConfigMap
Synopsis
Upload the kubeadm ClusterConfiguration to a ConfigMap called kubeadm-config in the kube-system namespace. This enables correct configuration of system components and a seamless user experience when upgrading.
Alternatively, you can use kubeadm config.
kubeadm init phase upload-config kubeadm [flags]
Examples
# upload the configuration of your cluster
  kubeadm init phase upload-config --config=myConfig.yaml
Options
--dry-run
-h, --help
1.71 -
Upload the kubelet component config to a ConfigMap
Synopsis
Upload the kubelet configuration extracted from the kubeadm InitConfiguration object to a kubelet-config ConfigMap in the cluster
kubeadm init phase upload-config kubelet [flags]
Examples
# Upload the kubelet configuration from the kubeadm Config file to a ConfigMap in the cluster.
  kubeadm init phase upload-config kubelet --config kubeadm.yaml
Options
--dry-run
-h, --help
1.72 -
Run this on any machine you wish to join an existing cluster
Synopsis
When joining a kubeadm initialized cluster, we need to establish
bidirectional trust. This is split into discovery (having the Node
trust the Kubernetes Control Plane) and TLS bootstrap (having the
Kubernetes Control Plane trust the Node).
There are 2 main schemes for discovery. The first is to use a shared
token along with the IP address of the API server. The second is to
provide a file - a subset of the standard kubeconfig file. The
discovery/kubeconfig file supports token, client-go authentication
plugins ("exec"), "tokenFile", and "authProvider". This file can be a
local file or downloaded via an HTTPS URL. The forms are
kubeadm join --discovery-token abcdef.1234567890abcdef 1.2.3.4:6443,
kubeadm join --discovery-file path/to/file.conf, or kubeadm join
--di

scovery-file https://url/file.conf. Only one form can be used. If
the discovery information is loaded from a URL, HTTPS must be used.
Also, in that case the host installed CA bundle is used to verify
the connection.
If you use a shared token for discovery, you should also pass the
--discovery-token-ca-cert-hash flag to validate the public key of the
root certificate authority (CA) presented by the Kubernetes Control Plane.
The value of this flag is specified as "&lt;hash-type&gt;:&lt;hex-encoded-value&gt;",
where the supported hash type is "sha256". The hash is calculated over
the bytes of the Subject Public Key Info (SPKI) object (as in RFC7469).
This value is available in the output of "kubeadm init" or can be
calculated using standard tools. The --discovery-token-ca-cert-hash flag
may be repeated multiple times to allow more than one public key.
If you cannot know the CA public key hash ahead of time, you can pass
the --discovery-token-unsafe-skip-ca-verification flag to disable this
verification. This weakens the kubeadm security model since other nodes
can potentially impersonate the Kubernetes Control Plane.
The TLS bootstrap mechanism is also driven via a shared token. This is
used to temporarily authenticate with the Kubernetes Control Plane to submit a
certificate signing request (CSR) for a locally created key pair. By
default, kubeadm will set up the Kubernetes Control Plane to automatically
approve these signing requests. This token is passed in with the
--tls-bootstrap-token abcdef.1234567890abcdef flag.
Often times the same token is used for both parts. In this case, the
--token flag can be used instead of specifying each token individually.
The "join [api-server-endpoint]" command executes the following phases:
preflight              Run join pre-flight checks
control-plane-prepare  Prepare the machine for serving a control plane
  /download-certs        [EXPERIMENTAL] Download certificates shared among control-plane nodes from the kubeadm-certs Secret
  /certs                 Generate the certificates for the new control plane components
  /kubeconfig            Generate the kubeconfig for the new control plane components
  /control-plane         Generate the manifests for the new control plane components
kubelet-start          Write kubelet settings, certificates and (re)start the kubelet
control-plane-join     Join a machine as a control plane instance
  /etcd                  Add a new local etcd member
  /update-status         Register the new control-plane node into the ClusterStatus maintained in the kubeadm-config ConfigMap (DEPRECATED)
  /mark-control-plane    Mark a node as a control-plane
kubeadm join [api-server-endpoint] [flags]
Options
If the node should host a new control plane instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.
If the node should host a new control plane instance, the port for the API Server to bind to.
Use this key to decrypt the certificate secrets uploaded by init.
--control-plane
Create a new control plane instance on this node
--discovery-file string
For file-based discovery, a file or URL from which to load cluster information.
--discovery-token string
For token-based discovery, the token used to validate cluster information fetched from the API server.
--discovery-token-ca-cert-hash strings
For token-based discovery, validate that the root CA public key matches this hash (format: "&lt;type&gt;:&lt;value&gt;").
--discovery-token-unsafe-skip-ca-verification
For token-based discovery, allow joining without --discovery-token-ca-cert-hash

 pinning.
--dry-run
-h, --help
help for join
--tls-bootstrap-token string
Specify the token used to temporarily authenticate with the Kubernetes Control Plane while joining the node.
Use this token for both discovery-token and tls-bootstrap-token when those values are not provided.
1.73 -
Use this command to invoke single phase of the join workflow
Synopsis
Options
-h, --help
1.74 -
Join a machine as a control plane instance
Synopsis
kubeadm join phase control-plane-join [flags]
Examples
# Joins a machine as a control plane instance
  kubeadm join phase control-plane-join all
Options
-h, --help
help for control-plane-join
1.75 -
Synopsis
kubeadm join phase control-plane-join all [flags]
Options
--dry-run
-h, --help
1.76 -
Add a new local etcd member
Synopsis
kubeadm join phase control-plane-join etcd [flags]
Options
--dry-run
-h, --help
Path to a directory that contains file

s named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.77 -
Synopsis
kubeadm join phase control-plane-join mark-control-plane [flags]
Options
--dry-run
-h, --help
1.78 -
Register the new control-plane node into the ClusterStatus maintained in the kubeadm-config ConfigMap (DEPRECATED)
Synopsis
kubeadm join phase control-plane-join update-status [flags]
Options
-h, --help
help for update-status
1.79 -
Prepare the machine for serving a control plane
Synopsis
kubeadm join phase control-plane-prepare [flags]
Examples
# Prepares the machine for serving a control plane
  kubeadm join phase control-plane-prepare all
Options
-h, --help
help for control-plane-prepare
1.80 -
Synopsis
kubeadm join phase control-plane-prepare all [api-server-endpoint] [flags]
Options
For token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver

0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.81 -
Generate the certificates for the new control plane components
Synopsis
kubeadm join phase control-plane-prepare certs [api-server-endpoint] [flags]
Options
--dry-run
-h, --help
1.82 -
Generate the manifests for the new control plane components
Synopsis
kubeadm join phase control-plane-prepare control-plane [flags]
Options
--dry-run
-h, --help
1.83 -
[EXPERIMENTAL] Download certificates shared among control-plane nodes from the kubeadm-cert

s Secret
Synopsis
[EXPERIMENTAL] Download certificates shared among control-plane nodes from the kubeadm-certs Secret
kubeadm join phase control-plane-prepare download-certs [api-server-endpoint] [flags]
Options
--dry-run
-h, --help
help for download-certs
1.84 -
Generate the kubeconfig for the new control plane components
Synopsis
kubeadm join phase control-plane-prepare kubeconfig [api-server-endpoint] [flags]
Options
--dry-run
-h, --help
1.85 -
Write kubelet settings, certificates and (re)start the kubelet
Synopsis
kubeadm join phase kubelet-start [api-server-endpoint] [flags]
Options
--dry-run
-h, --help
h

elp for kubelet-start
1.86 -
Run join pre-flight checks
Synopsis
Run pre-flight checks for kubeadm join.
kubeadm join phase preflight [api-server-endpoint] [flags]
Examples
# Run join pre-flight checks using a config file.
  kubeadm join phase preflight --config kubeadm-config.yaml
Options
--dry-run
-h, --help
1.87 -
Kubeconfig file utilities
Synopsis
Kubeconfig file utilities.
Options
-h, --help
1.88 -
Output a kubeconfig file for an additional user
Synopsis
Output a kubeconfig file for an additional user.
kubeadm kubeconfig user [flags]
Examples
# Output a kubeconfig file for an additional user named foo using a kubeadm config file bar
  kubeadm kubeconfig user --client-name=foo --config=bar
Options
--client-name string
The name of user. It will be used as the CN if client certificates are created
-h, --help
help for user
--org strings
The or

ganizations of the client certificate. It will be used as the O if client certificates are created
The token that should be used as the authentication mechanism for this kubeconfig, instead of client certificates
--validity-period duration     Default: 8760h0m0s
The validity period of the client certificate. It is an offset from the current time.
1.89 -
Performs a best effort revert of changes made to this host by 'kubeadm init' or 'kubeadm join'
Synopsis
The "reset" command executes the following phases:
preflight           Run reset pre-flight checks
remove-etcd-member  Remove a local etcd member.
cleanup-node        Run cleanup node.
kubeadm reset [flags]
Options
The path to the directory where the certificates are stored. If specified, clean this directory.
--cleanup-tmp-dir
Cleanup the "/etc/kubernetes/tmp" directory
--dry-run
-f, --force
Reset the node without prompting for confirmation.
-h, --help
help for reset
1.90 -
Use this command to invoke single phase of the reset workflow
Synopsis
Options
-h, --help
1.91 -
Run cleanup node.
Synopsis
kubeadm reset phase cleanup-node [flags]
Options
--dry-run
-h, --help
help for cleanup-node
1.92 -
Run reset pre-flight checks
Synopsis
Run pre-flight checks for kubeadm reset.
kubeadm reset phase preflight [flags]
Options
--dry-run
-h, --help
1.93 -
Remove a local etcd member.
Synopsis
Remove a local etcd member for a control plane node.
kubeadm reset phase remove-etcd-member [flags]
Options
--dry-run
-h, --help
help for remove-etcd-member
The kubeconfig file to use when talking to the c

luster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.
1.94 -
Manage bootstrap tokens
Synopsis
This command manages bootstrap tokens. It is optional and needed only for advanced use cases.
In short, bootstrap tokens are used for establishing bidirectional trust between a client and a server.
A bootstrap token can be used when a client (for example a node that is about to join the cluster) needs
to trust the server it is talking to. Then a bootstrap token with the "signing" usage can be used.
bootstrap tokens can also function as a way to allow short-lived authentication to the API Server
(the token serves as a way for the API Server to trust the client), for example for doing the TLS Bootstrap.
What is a bootstrap token more exactly?
It is a Secret in the kube-system namespace of type "bootstrap.kubernetes.io/token".
A bootstrap token must be of the form "[a-z0-9]{6}.[a-z0-9]{16}". The former part is the public token ID,
while the latter is the Token Secret and it must be kept private at all circumstances!
The name of the Secret must be named "bootstrap-token-(token-id)".
You can read more about bootstrap tokens here:
https://kubernetes.io/docs/admin/bootstrap-tokens/
kubeadm token [flags]
Options
--dry-run
Whether to enable dry-run mode or not
-h, --help
help for token
1.95 -
Create bootstrap tokens on the server
Synopsis
This command will create a bootstrap token for you.
You can specify the usages for this token, the "time to live" and an optional human friendly description.
The [token] is the actual token to write.
This should be a securely generated random token of the form "[a-z0-9]{6}.[a-z0-9]{16}".
If no [token] is given, kubeadm will generate a random token instead.
kubeadm token create [token]
Options
When used together with '--print-join-command', print the full 'kubeadm join' flag needed to join the cluster as a control-plane. To create a new certificate key you must use 'kubeadm init phase upload-certs --upload-certs'.
--description string
A human friendly description of how this token is used.
--groups strings     Default: "system:bootstrappers:kubeadm:default-node-token"
Extra groups that this token will authenticate as when used for authentication. Must match "\Asystem:bootstrappers:[a-z0-9:-]{0,255}[a-z0-9]\z"
-h, --help
help for create
--print-join-command
Instead of printing only the token, print the full 'kubeadm join' flag needed to join the cluster using the token.
--ttl duration     Default: 24h0m0s
--usages strings     Default: "signing,authentication"
Describes the ways in which this token can be used. You can pass --usages multiple times or provide a comma separated list of options. Valid options: [signing,authentication]
--dry-run
1.96 -
Delete bootstrap tokens on the server
Synopsis
This command will delete a list of bootstrap tokens for you.
The [token-value] is the full Token of the form "[a-z0-9]{6}.[a-z0-9]{16}" or the
Token ID of the form "[a-z

0-9]{6}" to delete.
kubeadm token delete [token-value] ...
Options
-h, --help
help for delete
--dry-run
1.97 -
Generate and print a bootstrap token, but do not create it on the server
Synopsis
This command will print out a randomly-generated bootstrap token that can be used with
the "init" and "join" commands.
You don't have to use this command in order to generate a token. You can do so
yourself as long as it is in the format "[a-z0-9]{6}.[a-z0-9]{16}". This
command is provided for convenience to generate tokens in the given format.
You can also use "kubeadm init" without specifying a token and it will
kubeadm token generate [flags]
Options
-h, --help
help for generate
--dry-run
1.98 -
List bootstrap tokens on the server
Synopsis
This command will list all bootstrap tokens for you.
kubeadm token list [flags]
Options
-h, --help
--dry-run
1.99 -
Upgrade your cluster smoothly to a newer version with this command
Synopsis
kubeadm upgrade [flags]
Options
-h, --help
help for upgrade
1.100 -
Upgrade your Kubernetes cluster to the specified version
Synopsis
kubeadm upgrade apply [version]
Options
--allow-experimental-upgrades
Show unstable versions of Kubernetes as an upgrade alternative and allow upgrading to an alpha/beta/release candidate versions of Kubernetes.
--allow-release-candidate-upgrades
Show release candidate versions of Kubernetes as an upgrade alternative and allow upgrading to a release candidate versions of Kubernetes.
--certificate-renewal     Default: true
Perform the renewal of certificates used by component changed during upgrades.
--dry-run
Do not change any state, just output what actions would be performed.
--etcd-upgrade     Default: true
Perform the upgrade of etcd.
Force upgrading although some requirements might not be met. This also implies non-interactive mode.
-h, --help
help for apply

--print-config
Specifies whether the configuration file that will be used in the upgrade should be printed or not.
-y, --yes
Perform the upgrade and do not prompt for confirmation (non-interactive mode).
1.101 -
Show what differences would be applied to existing static pod manifests. See also: kubeadm upgrade apply --dry-run
Synopsis
kubeadm upgrade diff [version] [flags]
Options
--api-server-manifest string     Default: "/etc/kubernetes/manifests/kube-apiserver.yaml"
path to API server manifest
-c, --context-lines int     Default: 3
How many lines of context in the diff
--controller-manager-manifest string     Default: "/etc/kubernetes/manifests/kube-controller-manager.yaml"
path to controller manifest
-h, --help
help for diff
--scheduler-manifest string     Default: "/etc/kubernetes/manifests/kube-scheduler.yaml"
path to scheduler manifest
1.102 -
Upgrade commands for a node in the cluster
Synopsis
The "node" command executes the following phases:
preflight       Run upgrade node pre-flight checks
control-plane   Upgrade the control plane instance deployed on this node, if any
kubelet-config  Upgrade the kubelet configuration for this node
kubeadm upgrade node [flags]
Options
--dry-run
Do not change any state, just output the actions that would be performed.
-h, --help
help for node
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "

suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.103 -
Use this command to invoke single phase of the node workflow
Synopsis
Options
-h, --help
1.104 -
Upgrade the control plane instance deployed on this node, if any
Synopsis
kubeadm upgrade node phase control-plane [flags]
Options
--dry-run
-h, --help
1.105 -
Upgrade the kubelet configuration for this node
Synopsis
Download the kubelet configuration from the kubelet-config ConfigMap stored in the cluster
kubeadm upgrade node phase kubelet-config [flags]
Options
--dry-run
-h, --help
help for kubelet-config
1.106 -
Run upgrade node pre-flight checks
Synopsis
Run pre-flight checks for kubeadm upgrade node.
kubeadm upgrade node phase preflight [flags]
Options
-h, --help
1.107 -
Check which versions are available to upgrade to and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter
Synopsis
Check which versions are available to upgrade to and validate whether your current cl

uster is upgradeable. To skip the internet check, pass in the optional [version] parameter
kubeadm upgrade plan [version] [flags]
Options
-h, --help
help for plan
-o, --output string     Default: "text"
EXPERIMENTAL: Output format. One of: text|json|yaml.
1.108 -
Print the version of kubeadm
Synopsis
kubeadm version [flags]
Options
-h, --help
help for version
-o, --output string
Output format; available options are 'yaml', 'json' and 'short'
1.109 -
All files in this directory are auto-generated from other repos.
Do not edit them manually. You must edit them in their upstream repo.
2 - kubeadm init
This command initializes a Kubernetes control-plane node.
Synopsis
The "init" command executes the following phases:
  /scheduler                   Generate a kubeconfig file

 for the scheduler to use
Options
--dry-run
-h, --help
S

pecify range of IP addresses for the pod network. If set, the control plane will automatically allocate CIDRs for every node.
Init workflow
bootstraps a Kubernetes control-plane node by executing the
following steps:
Runs a series of pre-flight checks to validate the system state
before making changes. Some checks only trigger warnings, others are
considered errors and will exit kubeadm until the problem is corrected or the
user specifies
--ignore-preflight-errors=&lt;list-of-errors&gt;
.
Generates a self-signed CA to set up identities for each component in the cluster. The user can provide their
own CA cert and/or key by dropping it in the cert directory configured via
(
by default).
The APIServer certs will have additional SAN entries for any
--apiserver-cert-extra-sans
arguments, lowercased if necessary.
Writes kubeconfig files in
for the kubelet, the controller-manager and the
scheduler to use to connect to the API server, each with its own identity, as well as an
additional kubeconfig file for administration named
.
Generates static Pod manifests for the API server,
controller-manager and scheduler. In case an external etcd is not provided,
an additional static Pod manifest is generated for etcd.
Static Pod manifests are written to
; the kubelet
watches this directory for Pods to create on startup.
Once control plane Pods are up and running, the
sequence can continue.
Apply labels and taints to the control-plane node so that no additional workloads will
run there.
Generates the token that additional nodes can use to register
themselves with a control-plane in the future. Optionally, the user can provide a
token via
--token
, as described in the
docs.
Makes all the necessary configurations for allowing node joining with the
Bootstrap Tokens
and
TLS Bootstrap
mechanism:
Write a ConfigMap for making available all the information required
for joining, and set up related RBAC access rules.
Let Bootstrap Tokens access the CSR signing API.
Configure auto-approval for new CSR requests.
See
for additional info.
Installs a DNS server (CoreDNS) and the kube-proxy addon components via the API server.
In Kubernetes version 1.11 and later CoreDNS is the default DNS server.
Please note that although the DNS server is deployed, it will not be scheduled until CNI is installed.
Warning:
kube-dns usage with kubeadm is deprecated as of v1.18 and is removed in v1.21.
Using init phases with kubeadm
Kubeadm allows you to create a control-plane node in phases using the
command.
To view the ordered list of phases and sub-phases you can call
kubeadm init --help
. The list
will be located at the top of the help screen and each phase will have a description next to it.
Note that by calling
all of the phases and sub-phases will be executed in this exact order.
Some phases have unique flags, so if you want to ha

ve a look at the list of available options add
--help
sudo kubeadm init phase control-plane controller-manager --help
--help
to see the list of sub-phases for a certain parent phase:
sudo kubeadm init phase control-plane --help
also exposes a flag called
--skip-phases
that can be used to skip certain phases.
The flag accepts a list of phase names and the names can be taken from the above ordered list.
An example:
sudo kubeadm init phase control-plane all --config
=
configfile.yaml
sudo kubeadm init phase etcd
local
--config
=
# you can now modify the control plane and etcd manifest files
sudo kubeadm init --skip-phases
=
control-plane,etcd --config
=
What this example would do is write the manifest files for the control plane and etcd in
based on the configuration in
. This allows you to
modify the files and then skip these phases using
. By calling the last command you
will create a control plane node with the custom manifest files.
Alternatively, you can use the
skipPhases
.
Using kubeadm init with a configuration file
Caution:
The config file is still considered beta and may change in future versions.
It's possible to configure
with a configuration file instead of command
line flags, and some more advanced features may only be available as
configuration file options. This file is passed using the
--config
flag and it must
contain a
structure and optionally more structures separated by
---\n
Mixing
--config
with others flags may not be allowed in some cases.
The default configuration can be printed out using the
kubeadm config print
command.
If your configuration is not using the latest version it is
recommended
that you migrate using
the
kubeadm config migrate
command.
For more information on the fields and usage of the configuration you can navigate to our
API reference page
.
Using kubeadm init with feature gates
Kubeadm supports a set of feature gates that are unique to kubeadm and can only be applied
during cluster creation with
. These features can control the behavior
of the cluster. Feature gates are removed after a feature graduates to GA.
To pass a feature gate you can either use the
flag for
, or you can add items into the
field when you pass
a
using
--config
.
Passing
feature gates for core Kubernetes components
directly to kubeadm is not supported. Instead, it is possible to pass them by
.
List of feature gates:
kubeadm feature gates
Feature
Default
Alpha
Beta
GA
PublicKeysECDSA
false
1.19
-
-
RootlessControlPlane
false
1.22
-
-
UnversionedKubeletConfigMap
true
1.22
1.23
1.25
Note:
Once a feature gate goes GA its value becomes locked to
true
by default.
Feature gate descriptions:
Can be used to create a cluster that uses ECDSA certificates instead of the default RSA algorithm.
Renewal of existing ECDSA certificates is also supported using
, but you cannot
switch between the RSA and ECDSA algorithms on the fly or during upgrades.
Setting this flag configures the kubeadm deployed control plane component static Pod containers
for
,
,
and
etcd
to run as non-root users.
If the flag is not set, those components run as root. You can change the value of this feature gate before
you upgrade to a newer version of Kubernetes.
This flag controls the name of the
ConfigMap
where kubeadm stores
kubelet configuration data. With this flag not specified or set to
true
, the ConfigMap is named
.
If you set this flag to
false
, the name of the ConfigMap includes the major and minor version for Kubernetes
(for example:
kubelet-config-

1.26
). Kubeadm ensures that RBAC rules for reading and writing
that ConfigMap are appropriate for the value you set. When kubeadm writes this ConfigMap (during
or
), kubeadm respects the value of
. When reading that ConfigMap
(during
,
,
kubeadm upgrade ...
), kubeadm attempts to use unversioned ConfigMap name first;
if that does not succeed, kubeadm falls back to using the legacy (versioned) name for that ConfigMap.
Adding kube-proxy parameters
For information about kube-proxy parameters in the kubeadm configuration see:
kube-proxy reference
For information about enabling IPVS mode with kubeadm see:
IPVS
Passing custom flags to control plane components
For information about passing flags to control plane components see:
control-plane-flags
Running kubeadm without an Internet connection
For running kubeadm without an Internet connection you have to pre-pull the required control-plane images.
You can list and pull the images using the
kubeadm config images
sub-command:
kubeadm config images list
kubeadm config images pull
You can pass
--config
to the above commands with a
kubeadm configuration file
to control the
and
fields.
All default
images that kubeadm requires support multiple architectures.
Using custom images
By default, kubeadm pulls images from
. If the
requested Kubernetes version is a CI label (such as
ci/latest
)
is used.
You can override this behavior by using
kubeadm with a configuration file
.
Allowed customization are:
To provide
which affects the version of the images.
To provide an alternative
to be used instead of
.
To provide a specific
and
imageTag
for etcd or CoreDNS.
Image paths between the default
and a custom repository specified using
may differ for backwards compatibility reasons. For example,
one image might have a subpath at
registry.k8s.io/subpath/image
, but be defaulted
to
my.customrepository.io/image
when using a custom repository.
To ensure you push the images to your custom repository in paths that kubeadm
can consume, you must:
Pull images from the defaults paths at
using
kubeadm config images {list|pull}
.
Push images to the paths from
kubeadm config images list --config=config.yaml
,
where
config.yaml
contains the custom
, and/or
imageTag
for etcd and CoreDNS.
Pass the same
to
.
Custom sandbox (pause) images
To set a custom image for these you need to configure this in your
to use the image.
Consult the documentation for your container runtime to find out how to change this setting;
for selected container runtimes, you can also find advice within the
topic.
Uploading control-plane certificates to the cluster
By adding the flag
to
you can temporary upload
the control-plane certificates to a Secret in the cluster. Please note that this Secret
will expire automatically after 2 hours. The certificates are encrypted using
a 32byte key that can be specified using
--certificate-key
. The same key can be used
to download the certificates when additional control-plane nodes are joining, by passing
and
to
.
The following phase command can be used to re-upload the certificates after expiration:
kubeadm init phase upload-certs --upload-certs --config
=
SOME_YAML_FILE
Note:
A predefined
can be provided in
when passing the
with
--config
.
If a predefined certificate key is not passed to
and
kubeadm init phase upload-certs
a new key will be generated automatically.
The following command can be used to generate a new key on demand:
kubeadm certs certificate-key
F

or detailed information on certificate management with kubeadm see
.
The document includes information about using external CA, custom certificates
and certificate renewal.
Managing the kubeadm drop-in file for the kubelet
The
kubeadm
package ships with a configuration file for running the
kubelet
by
systemd
.
Note that the kubeadm CLI never touches this drop-in file. This drop-in file is part of the kubeadm
DEB/RPM package.
For further information, see
Managing the kubeadm drop-in file for systemd
.
Use kubeadm with CRI runtimes
By default kubeadm attempts to detect your container runtime. For more details on this detection,
see the
kubeadm CRI installation guide
.
Setting the node name
By default,
kubeadm
assigns a node name based on a machine's host address.
You can override this setting with the
--node-name
flag.
The flag passes the appropriate
--hostname-override
value to the kubelet.
Be aware that overriding the hostname can
interfere with cloud providers
.
Automating kubeadm
Rather than copying the token you obtained from
to each node, as
in the
basic kubeadm tutorial
,
you can parallelize the token distribution for easier automation. To implement this automation,
you must know the IP address that the control-plane node will have after it is started, or use a
DNS name or an address of a load balancer.
Generate a token. This token must have the form
&lt;6 character string&gt;.&lt;16 character string&gt;
. More formally, it must match the regex:
[a-z0-9]{6}\.[a-z0-9]{16}
.
kubeadm can generate a token for you:
kubeadm token generate
Start both the control-plane node and the worker nodes concurrently with this token.
As they come up they should find each other and form the cluster. The same
--token
argument can be used on both
and
.
Similar can be done for
when joining additional control-plane
nodes. The key can be generated using:
Once the cluster is up, you can grab the admin credentials from the control-plane node
at
and use that to talk to the cluster.
Note that this style of bootstrap has some relaxed security guarantees because
it does not allow the root CA hash to be validated with
--discovery-token-ca-cert-hash
(since it's not generated when the nodes are
provisioned). For details, see the
.
to understand more about
phases
to bootstrap a Kubernetes
worker node and join it to the cluster
to upgrade a Kubernetes
cluster to a newer version
to revert any changes made
to this host by
or
3 - kubeadm join
This command initializes a Kubernetes worker node and joins it to the cluster.
Synopsis
--discovery-file https://url/file.conf. Only one form can be used. If
--discovery-token-ca-cert-hash flag to validate the public key of t

he
Options
--dry-run
-h, --help
Path to a directory

 that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
The join workflow
bootstraps a Kubernetes worker node or a control-plane node and adds it to the cluster.
This action consists of the following steps for worker nodes:
kubeadm downloads necessary cluster information from the API server.
By default, it uses the bootstrap token and the CA key hash to verify the
authenticity of that data. The root CA can also be discovered directly via a
file or URL.
Once the cluster information is known, kubelet can start the TLS bootstrapping
process.
The TLS bootstrap uses the shared token to temporarily authenticate
with the Kubernetes API server to submit a certificate signing request (CSR); by
default the control plane signs this CSR request automatically.
Finally, kubeadm configures the local kubelet to connect to the API
server with the definitive identity assigned to the node.
For control-plane nodes additional steps are performed:
Downloading certificates shared among control-plane nodes from the cluster
(if explicitly requested by the user).
Generating control-plane component manifests, certificates and kubeconfig.
Adding new local etcd member.
Using join phases with kubeadm
Kubeadm allows you join a node to the cluster in phases using
.
kubeadm join --help
. The list will be located
at the top of the help screen and each phase will have a description next to it.
Some phases have unique flags, so if you want to have a look at the list of available options add
--help
kubeadm join phase kubelet-start --help
Similar to the
command,
allows you to skip a list of phases using the
flag.
sudo kubeadm join --skip-phases
=
preflight --config
=
field in
.
Discovering what cluster CA to trust
The kubeadm discovery has several options, each with security tradeoffs.
The right method for your environment depends on how you provision nodes and the
security expectations you have about your network and node lifecycles.
Token-based discovery with CA pinning
This is the default mode in kubeadm. In this mode, kubeadm downloads
the cluster configuration (including root CA) and validates it using the token
as well as validating that the root CA public key matches the provided hash and
that the API server certificate is valid under the root CA.
The CA key hash has the format
sha256:&lt;hex_encoded_hash&gt;
. By default, the hash value is returned in the
command printed at the end of
or in the output of
kubeadm token create --print-join-command
. It is in a standard format (see
RFC7469
) and can also be calculated by 3rd party tools or provisioning systems. For example, using the OpenSSL CLI:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt 

| openssl rsa -pubin -outform der 2&gt;/dev/null | openssl dgst -sha256 -hex | sed
's/^.* //'
Example
commands:
For worker nodes:
kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef 1.2.3.4:6443
For control-plane nodes:
kubeadm join --discovery-token abcdef.1234567890abcdef --discovery-token-ca-cert-hash sha256:1234..cdef --control-plane 1.2.3.4:6443
You can also call
join
for a control-plane node with
to copy certificates to this node,
if the
command was called with
.
Advantages:
Allows bootstrapping nodes to securely discover a root of trust for the
control-plane node even if other worker nodes or the network are compromised.
Convenient to execute manually since all of the information required fits
into a single
command.
Disadvantages:
The CA hash is not normally known until the control-plane node has been provisioned,
which can make it more difficult to build automated provisioning tools that
use kubeadm. By generating your CA in beforehand, you may workaround this
limitation.
Token-based discovery without CA pinning
This mode relies only on the symmetric token to sign
(HMAC-SHA256) the discovery information that establishes the root of trust for
the control-plane. To use the mode the joining nodes must skip the hash validation of the
CA public key, using
. You should consider
using one of the other modes if possible.
Example
command:
kubeadm join --token abcdef.1234567890abcdef --discovery-token-unsafe-skip-ca-verification 1.2.3.4:6443
Still protects against many network-level attacks.
The token can be generated ahead of time and shared with the control-plane node and
worker nodes, which can then bootstrap in parallel without coordination. This
allows it to be used in many provisioning scenarios.
If an attacker is able to steal a bootstrap token via some vulnerability,
they can use that token (along with network-level access) to impersonate the
control-plane node to other bootstrapping nodes. This may or may not be an appropriate
tradeoff in your environment.
File or HTTPS-based discovery
This provides an out-of-band way to establish a root of trust between the control-plane node
and bootstrapping nodes. Consider using this mode if you are building automated provisioning
using kubeadm. The format of the discovery file is a regular Kubernetes
file.
In case the discovery file does not contain credentials, the TLS discovery token will be used.
Example
commands:
kubeadm join --discovery-file path/to/file.conf
(local file)
kubeadm join --discovery-file https://url/file.conf
(remote HTTPS URL)
control-plane node even if the network or other worker nodes are compromised.
Requires that you have some way to carry the discovery information from
the control-plane node to the bootstrapping nodes. If the discovery file contains credentials
you must keep it secret and transfer it over a secure channel. This might be possible with your
cloud provider or provisioning tool.
Securing your installation even more
The defaults for kubeadm may not work for everyone. This section documents how to tighten up a kubeadm installation
at the cost of some usability.
Turning off auto-approval of node client certificates
By default, there is a CSR auto-approver enabled that basically approves any client certificate request
for a kubelet when a Bootstrap Token was used when authenticating. If you don't want the cluster to
automatically approve kubelet client certs, you can turn it off by executing this command:
kubectl delete clusterrolebinding kubeadm:node-autoapprove-bootstrap
After that,
will block until the admin has manually approved the CSR in flight:
Using
, you can see that the original CSR 

is in the Pending state.
NAME                                                   AGE       REQUESTOR                 CONDITION
node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ   18s       system:bootstrap:878f07   Pending
kubectl certificate approve
allows the admin to approve CSR.This action tells a certificate signing
controller to issue a certificate to the requestor with the attributes requested in the CSR.
kubectl certificate approve node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ
certificatesigningrequest "node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ" approved
This would change the CRS resource to Active state.
node-csr-c69HXe7aYcqkS1bKmH4faEnHAWxn6i2bHZ2mD04jZyQ   1m        system:bootstrap:878f07   Approved,Issued
This forces the workflow that
will only succeed if
has been run.
Turning off public access to the
ConfigMap
In order to achieve the joining flow using the token as the only piece of validation information, a
ConfigMap with some data needed for validation of the control-plane node's identity is exposed publicly by
default. While there is no private data in this ConfigMap, some users might wish to turn
it off regardless. Doing so will disable the ability to use the
--discovery-token
flag of the
flow. Here are the steps to do so:
Fetch the
file from the API Server:
kubectl -n kube-public get cm cluster-info -o
jsonpath
=
'{.data.kubeconfig}'
| tee cluster-info.yaml
:
v1
kind
:
Config
clusters
:
-
cluster
:
:
&lt;ca-cert&gt;
server
:
https://&lt;ip&gt;:&lt;port&gt;
name
:
""
contexts
:
[]
:
""
:
{}
users
:
[]
Use the
cluster-info.yaml
file as an argument to
kubeadm join --discovery-file
.
Turn off public access to the
kubectl -n kube-public delete rolebinding kubeadm:bootstrap-signer-clusterinfo
These commands should be run after
but before
.
Using kubeadm join with a configuration file
Caution:
--config
contain a
structure. Mixing
--config
with others flags may not be
allowed in some cases.
command.
the
command.
API reference
.
to bootstrap a Kubernetes control-plane node.
.
or
.
4 - kubeadm upgrade
is a user-friendly command that wraps complex upgrading logic
behind one command, with support for both planning an upgrade and actually performing it.
kubeadm upgrade guidance
The steps for performing an upgrade using kubeadm are outlined in
this document
.
For older versions of kubeadm, please refer to older documentation sets of the Kubernetes website.
You can use
kubeadm upgrade diff
to see the changes that would be applied to static pod manifests.
In Kubernetes v1.15.0 and later,
and
will also
automatically renew the kubeadm managed certificates on this node, including those stored in kubeconfig files.
To opt-out, it is possible to pass the flag

. For more details about certificate
renewal see the
certificate management documentation
.
Note:
The commands
and
have a legacy
--config
flag which makes it possible to reconfigure the cluster, while performing planning or upgrade of that particular
control-plane node. Please be aware that the upgrade workflow was not designed for this scenario and there are
reports of unexpected results.
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named

 "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
-y, --yes
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
5 - kubeadm 

config
During
, kubeadm uploads the
object to your cluster
in the
namespace. This configuration is then read during
,
and
.
You can use
to print the default static configuration that kubeadm
uses for
and
.
Note:
The output of the command is meant to serve as an example. You must manually edit the output
of this command to adapt to your setup. Remove the fields that you are not certain about and kubeadm
will try to default them on runtime by examining the host.
init
and
join
navigate to
or
.
For more information on using the kubeadm configuration API navigate to
.
You can use
to convert your old configuration files that contain a deprecated
API version to a newer, supported API version.
and
can be used to list and pull the images
that kubeadm requires.
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Read an old

er version of the kubeadm configuration API types from a file, and output the similar config object for the newer version
Synopsis
Further, kubeadm can only write out config of version "kubeadm.k8s.io/v1beta3", but read both types.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
--kubernetes-versio

n string     Default: "stable-1"
6 - kubeadm reset
Performs a best effort revert of changes made by
or
.
Synopsis
Options
--dry-run
-h, --help
Reset workflow
is responsible for cleaning up a node local file system from files that were created using
the
or
commands. For control-plane nodes
reset
also removes the local stacked
etcd member of this node from the etcd cluster.
can be used to execute the separate phases of the above workflow.
To skip a list of phases you can use the
flag, which works in a similar way to
the
and
phase runners.
External etcd clean up
will not delete any etcd data if external etcd is used. This means that if you run
again using the same etcd endpoints, you will see state from previous clusters.
To wipe etcd data it is recommended you use a client like etcdctl, such as:
etcdctl del
""
--prefix
See the
7 - kubeadm token
Bootstrap tokens are used for establishing bidirectional trust between a node joining
the cluster and a control-plane node, as described in
authenticating with bootstrap tokens
.
creates an initial token with a 24-hour TTL. The following commands allow you to manage
such a token and also to create and manage new ones.
kubeadm token create
Synopsis
Options
W

hen used together with '--print-join-command', print the full 'kubeadm join' flag needed to join the cluster as a control-plane. To create a new certificate key you must use 'kubeadm init phase upload-certs --upload-certs'.
-h, --help
--dry-run
kubeadm token delete
Synopsis
Token ID of the form "[a-z0-9]{6}" to delete.
Options
-h, --help
--dry-run
Synopsis
Options
-h, --help
--dry-run
kubeadm token list
Synopsis
Options
-h, --help
--dry-run
--kubeconfig string     Default: "/e

tc/kubernetes/admin.conf"
8 - kubeadm version
This command prints the version of kubeadm.
Synopsis
Options
-h, --help
9 - kubeadm alpha
Caution:
provides a preview of a set of features made available for gathering feedback
from the community. Please try it out and give us feedback!
Currently there are no experimental commands under
.
to connect a node to the cluster
or
10 - kubeadm certs
provides utilities for managing certificates.
For more details on how these commands can be used, see
.
A collection of operations for operating Kubernetes certificates.
overview
Synopsis
Options
-h, --help
You can renew all Kubernetes certificates using the
all
subcommand or renew them selectively.
For more details see
.
renew
all
apiserver-etcd-client
apiserver-kubelet-client
apiserver
etcd-healthcheck-client
etcd-peer
etcd-server
front-proxy-client
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
kubeadm certs renew admin.c

onf [flags]
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
The path where to save the 

certificates
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
The path where to save the certifi

cates
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
--kubeconfig string     Def

ault: "/etc/kubernetes/admin.conf"
This command can be used to generate a new control-plane certificate key.
The key can be passed as
to
and
to enable the automatic copy of certificates when joining additional control-plane nodes.
certificate-key
Synopsis
Options
-h, --help
This command checks expiration for the certificates in the local PKI managed by kubeadm.
.
Synopsis
Options
-h, --help
kubeadm certs generate-csr
This command can be used to generate keys and CSRs for all control-plane certificates and kubeconfig files.
The user can then sign the CSRs with a CA of their choice.
generate-csr
Synopsis
Examples
Options
-h, --help
or
11 - kubeadm init phase
enables you to invoke atomic steps of the bootstrap process.
Hence, you can let kubeadm do some of the work and you can fill in the gaps
if you w

ish to apply customization.
is consistent with the
kubeadm init workflow
,
and behind the scene both use the same code.
kubeadm init phase preflight
Using this command you can execute preflight checks on a control-plane node.
preflight
Synopsis
Examples
Options
--dry-run
-h, --help
kubeadm init phase kubelet-start
This phase will write the kubelet configuration file and environment file and then start the kubelet.
kubelet-start
Synopsis
Examples
Options
--dry-run
-h, --help
kubeadm init phase certs
Can be used to create all required certificates by kubeadm.
certs
all
ca
apiserver
front-proxy-ca
etcd-ca
etcd-peer
healthcheck-client
sa
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Choose a specific Kubernetes version for the co

ntrol plane.
Synopsis
Options
--dry-run
-h, --help
help for ca
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
If both files already exist, kubeadm skips the generation step and existing files wi

ll be used.
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
--kubernetes-version string     D

efault: "stable-1"
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
-h, --help
help for sa
kubeadm init phase kubeconfig
You can create all required kubeconfig files by calling the
all
subcommand or call them individually.
all
admin
kubelet
controller-manager
scheduler
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
--kubeconfig-dir string  

   Default: "/etc/kubernetes"
Synopsis
Options
--dry-run
-h, --help
only
for cluster bootstrapping purposes
Synopsis
only
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
[EXPERIMENTAL] The path to the 'real' host root

 filesystem.
Synopsis
Options
--dry-run
-h, --help
kubeadm init phase control-plane
Using this phase you can create all required static Pod files for the control plane components.
all
apiserver
scheduler
Synopsis
Options
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "s

trategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver

", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
Synopsis
Options
--dry-run
-h, --help
kubeadm init phase etcd
Use the following phase to create a local etcd instance based on a static Pod file.
etcd
local
Synopsis
Options
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be ei

ther "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
kubeadm init phase upload-config
You can use this command to upload the kubeadm configuration to your cluster.
Alternatively, you can use
.
upload-config
all
kubeadm
kubelet
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Use the following phase to upload control-plane certificates to the cluster.
By default the certs and encryption key expire after two hours.
upload-certs
Synopsis
Options
--dry-run
-h, --help
The kubeconfig file to use when talking to the cluster. If 

the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.
kubeadm init phase mark-control-plane
Use the following phase to label and taint the node as a control plane node.
mark-control-plane
Synopsis
Examples
  
Options
--dry-run
-h, --help
kubeadm init phase bootstrap-token
Use the following phase to configure bootstrap tokens.
bootstrap-token
Synopsis
Examples
Options
--dry-run
-h, --help
kubeadm init phase kubelet-finalize
Use the following phase to update settings relevant to the kubelet after TLS
bootstrap. You can use the
all
subcommand to run all
kubelet-finalize
phases.
kubelet-finalize-all
kubelet-finalize-cert-rotation
Synopsis
Examples
Options
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
kubeadm init phase kubelet-finalize experimental-cert-rota

tion [flags]
Options
--dry-run
-h, --help
kubeadm init phase addon
You can install all the available addons with the
all
subcommand, or
install them selectively.
addon
all
coredns
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Use alternative domain for services, e.g. "myorg.inter

nal".
Synopsis
Options
--dry-run
-h, --help
For more details on each field in the
v1beta3
configuration you can navigate to our
API reference pages.
or
to try experimental functionality
12 - kubeadm join phase
enables you to invoke atomic steps of the join process.
if you wish to apply customization.
kubeadm join workflow
,
phase
Synopsis
Options
-h, --help
kubeadm join phase preflight
Using this phase you can execute preflight checks on a joining node.
preflight
Synopsis
Examples
Options
For token-based discovery, val

idate that the root CA public key matches this hash (format: "&lt;type&gt;:&lt;value&gt;").
--dry-run
-h, --help
kubeadm join phase control-plane-prepare
Using this phase you can prepare a node for serving a control-plane.
control-plane-prepare
all
download-certs
certs
Synopsis
Examples
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
[EXPERIMEN

TAL] Download certificates shared among control-plane nodes from the kubeadm-certs Secret
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h

, --help
Synopsis
Options
--dry-run
-h, --help
kubeadm join phase kubelet-start
Using this phase you can write the kubelet settings, certificates and (re)start the kubelet.
Synopsis
Options
--dry-run
-h, --help
Use this token for both discovery-

token and tls-bootstrap-token when those values are not provided.
kubeadm join phase control-plane-join
Using this phase you can join a node as a control-plane instance.
control-plane-join
all
etcd
update-status
Synopsis
Examples
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
Create a new control plane instance on this nod

e
-h, --help
Synopsis
Options
--dry-run
-h, --help
or
13 - kubeadm kubeconfig
provides utilities for managing kubeconfig files.
For examples on how to use
see
.
overview
Synopsis
Options
-h, --help
This command can be used to output a kubeconfig file for an additional user.
user
Synopsis
Examples
Options
-h, --help
The organizations of the client certificate. It will be used as the O if client certificates are created
14 - kubeadm reset phase
enables you to invoke atomic steps of the node reset process.
kubeadm reset workflow
,
phase
Synopsis
Options
-h, --help
kubeadm reset phase preflight
Using this phase you can execute preflight checks on a node that is being reset.
preflight
Synopsis
Options
--dry-run
-h, --help
kubeadm reset phase remove-etcd-member
Using this phase you can remove this control-plane node's etcd member from the etcd cluster.
remove-etcd-member
Remove a local etcd 

member.
Synopsis
Options
--dry-run
-h, --help
kubeadm reset phase cleanup-node
Using this phase you can perform cleanup on this node.
cleanup-node
Synopsis
Options
--dry-run
-h, --help
or
15 - kubeadm upgrade phase
In v1.15.0, kubeadm introduced preliminary support for
phases.
Phases for other
sub-commands such as
apply
, could be added in the
following releases.
kubeadm upgrade node phase
Using this phase you can choose to execute the separate steps of the upgrade of
secondary control-plane or worker nodes. Please note that
still has to
be called on a primary control-plane node.
phase
preflight
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. Th

e default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
Synopsis
Options
--dry-run
-h, --help
or
to upgrade a kubeadm node
16 - Implementation details
Kubernetes v1.10 [stable]
and
together provides a nice user experience for creating a
best-practice but bare Kubernetes cluster from scratch.
However, it might not be obvious
how
kubeadm does that.
This document provides additional details on what happen under the hood, with the aim of sharing
knowledge on Kubernetes cluster best practices.
Core design principles
The cluster that
and
set up should be:
Secure
: It should adopt latest best-practices like:
enforcing RBAC
using the Node Authorizer
using secure communication between the control plane components
using secure communication between the API server and the kubelets
lock-down the kubelet API
locking down access to the API for system components like the kube-proxy and CoreDNS
locking down what a Bootstrap Token can access
User-friendly
: The user should not have to run anything more than a couple of commands:
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl apply -f &lt;network-of-choice.yaml&gt;
kubeadm join --token &lt;token&gt; &lt;endpoint&gt;:&lt;port&gt;
Extendable
:
It should
not
favor any particular network provider. Configuring the cluster network is out-of-scope
It should provide the possibility to use a config file for customizing various parameters
Constants and well-known values and paths
In order to reduce complexity and to simplify development of higher level tools that build on top of kubeadm, it uses a
limited set of constant values for well-known paths and file names.
The Kubernetes directory
/etc/kubernetes
is a constant in the application, since it is clearly the given path
in a majority of cases, and the most intuitive location; other constants paths and file names are:
as the path where kubelet should look for static Pod manifests.
Names of static Pod manifests are:
etcd.yaml
kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml
as the path where kubeconfig files with identities for control plane
components are stored. Names of kubeconfig files are:
(
b

ootstrap-kubelet.conf
during TLS bootstrap)
for the cluster admin and kubeadm itself
Names of certificates and key files :
ca.crt
,
ca.key
for the Kubernetes certificate authority
apiserver.crt
,
apiserver.key
for the API server certificate
apiserver-kubelet-client.crt
,
apiserver-kubelet-client.key
for the client certificate used
by the API server to connect to the kubelets securely
sa.pub
,
sa.key
for the key used by the controller manager when signing ServiceAccount
front-proxy-ca.crt
,
front-proxy-ca.key
for the front proxy certificate authority
front-proxy-client.crt
,
front-proxy-client.key
for the front proxy client
kubeadm init workflow internal design
The
internal workflow
consists of a sequence of atomic work tasks to perform,
as described in
.
The
command allows
users to invoke each task individually, and ultimately offers a reusable and composable
API/toolbox that can be used by other Kubernetes bootstrap tools, by any IT automation tool or by
an advanced user for creating custom clusters.
Preflight checks
Kubeadm executes a set of preflight checks before starting the init, with the aim to verify
preconditions and avoid common cluster startup problems.
The user can skip specific preflight checks or all of them with the
--ignore-preflight-errors
option.
[warning] If the Kubernetes version to use (specified with the
--kubernetes-version
flag) is
at least one minor version higher than the kubeadm CLI version.
Kubernetes system requirements:
if running on linux:
[error] if Kernel is older than the minimum required version
[error] if required cgroups subsystem aren't set up
[error] if the CRI endpoint does not answer
[error] if user is not root
[error] if the machine hostname is not a valid DNS subdomain
[warning] if the host name cannot be reached via network lookup
[error] if kubelet version is lower that the minimum kubelet version supported by kubeadm (current minor -1)
[error] if kubelet version is at least one minor higher than the required controlplane version (unsupported version skew)
[warning] if kubelet service does not exist or if it is disabled
[warning] if firewalld is active
[error] if API server bindPort or ports 10250/10251/10252 are used
[Error] if
/etc/kubernetes/manifest
folder already exists and it is not empty
[Error] if
/proc/sys/net/bridge/bridge-nf-call-iptables
file does not exist/does not contain 1
[Error] if advertise address is ipv6 and
/proc/sys/net/bridge/bridge-nf-call-ip6tables
does not exist/does not contain 1.
[Error] if swap is on
[Error] if
conntrack
,
ip
,
iptables
,
mount
,
nsenter
commands are not present in the command path
[warning] if
ebtables
,
ethtool
,
socat
,
tc
,
touch
,
crictl
[warning] if extra arg flags for API server, controller manager, scheduler contains some invalid options
[warning] if connection to https://API.AdvertiseAddress:API.BindPort goes through proxy
[warning] if connection to services subnet goes through proxy (only first address checked)
[warning] if connection to Pods subnet goes through proxy (only first address checked)
If external etcd is provided:
[Error] if etcd version is older than the minimum required version
[Error] if etcd certificates or keys are specified, but not provided
If external etcd is NOT provided (and thus local etcd will be installed):
[Error] if ports 2379 is used
[Error] if Etcd.DataDir folder already exists and it is not empty
If authorization mode is ABAC:
[Error] if abac_policy.json does not exist
If authorization mode is WebHook
[Error] if webhook_authz.conf does not exist
Please note that:
Preflight checks can be invoked individually with the
command
Generate the necessary certificates
Kubeadm generates certificate and private key pairs for different purposes:
A self signed certificate authority for the Kubernetes cluster saved into
ca.crt
file and
ca.key
pri

vate key file
A serving certificate for the API server, generated using
ca.crt
as the CA, and saved into
file with its private key
. This certificate should contain
following alternative names:
The Kubernetes service's internal clusterIP (the first address in the services CIDR, e.g.
10.96.0.1
if service subnet is
10.96.0.0/12
)
Kubernetes DNS names, e.g.
if
--service-dns-domain
flag value is
, plus default DNS names
,
,
The node-name
The
--apiserver-advertise-address
Additional alternative names specified by the user
A client certificate for the API server to connect to the kubelets securely, generated using
ca.crt
as the CA and saved into
.
This certificate should be in the
organization
A private key for signing ServiceAccount Tokens saved into
sa.key
file along with its public key
sa.pub
A certificate authority for the front proxy saved into
file with its key
A client cert for the front proxy client, generate using
as the CA and
saved into
Certificates are stored by default in
, but this directory is configurable
using the
flag.
If a given certificate and private key pair both exist, and its content is evaluated compliant with the above specs, the existing files will
be used and the generation phase for the given certificate skipped. This means the user can, for example, copy an existing CA to
/etc/kubernetes/pki/ca.{crt,key}
, and then kubeadm will use those files for signing the rest of the certs.
See also
using custom certificates
Only for the CA, it is possible to provide the
ca.crt
file but not the
ca.key
file, if all other certificates and kubeconfig files
already are in place kubeadm recognize this condition and activates the ExternalCA , which also implies the
csrsigner
controller in
controller-manager won't be started
If kubeadm is running in
external CA mode
;
all the certificates must be provided by the user, because kubeadm cannot generate them by itself
In case of kubeadm is executed in the
--dry-run
mode, certificates files are written in a temporary folder
Certificate generation can be invoked individually with the
kubeadm init phase certs all
command
Generate kubeconfig files for control plane components
Kubeadm generates kubeconfig files with identities for control plane components:
A kubeconfig file for the kubelet to use during TLS bootstrap -
/etc/kubernetes/bootstrap-kubelet.conf. Inside this file there is a bootstrap-token or embedded
client certificates for authenticating this node with the cluster.
This client cert should:
Be in the
organization, as required by the
Node Authorization
module
Have the Common Name (CN)
system:node:&lt;hostname-lowercased&gt;
A kubeconfig file for controller-manager,
/etc/kubernetes/controller-manager.conf
; inside this
file is embedded a client certificate with controller-manager identity. This client cert should
have the CN
, as defined by default
RBAC core components roles
A kubeconfig file for scheduler,
/etc/kubernetes/scheduler.conf
; inside this file is embedded
a client certificate with scheduler identity.
This client cert should have the CN
Additionally, a kubeconfig file for kubeadm itself and the admin is generated and saved into the
file. The "admin" here is defined as the actual person(s) that is
administering the cluster and wants to have full control (
root
) over the cluster. The
embedded client certificate for admin should be in the
organization, as defined
by default
RBAC user facing role bindings

.
It should also include a CN. Kubeadm uses the
kubernetes-admin
CN.
ca.crt
certificate is embedded in all the kubeconfig files.
If a given kubeconfig file exists, and its content is evaluated compliant with the above specs,
the existing file will be used and the generation phase for the given kubeconfig skipped
ExternalCA mode
,
all the required kubeconfig must be provided by the user as well, because kubeadm cannot
generate any of them by itself
--dry-run
mode, kubeconfig files are written in a temporary folder
Kubeconfig files generation can be invoked individually with the
kubeadm init phase kubeconfig all
command
Generate static Pod manifests for control plane components
Kubeadm writes static Pod manifest files for control plane components to
. The kubelet watches this directory for Pods to create on startup.
Static Pod manifest share a set of common properties:
All static Pods are deployed on
namespace
All static Pods get
tier:control-plane
and
component:{component-name}
labels
All static Pods use the
priority class
hostNetwork: true
is set on all static Pods to allow control plane startup before a network is
configured; as a consequence:
The
address
that the controller-manager and the scheduler use to refer the API server is
127.0.0.1
If using a local etcd server,
etcd-servers
address will be set to
127.0.0.1:2379
Leader election is enabled for both the controller-manager and the scheduler
Controller-manager and the scheduler will reference kubeconfig files with their respective, unique identities
All static Pods get any extra flags specified by the user as described in
passing custom arguments to control plane components
All static Pods get any extra Volumes specified by the user (Host path)
All images will be pulled from registry.k8s.io by default.
See
using custom images
for customizing the image repository
--dry-run
mode, static Pods files are written in a
temporary folder
Static Pod manifest generation for control plane components can be invoked individually with
the
kubeadm init phase control-plane all
command
API server
The static Pod manifest for the API server is affected by following parameters provided by the users:
The
apiserver-advertise-address
and
apiserver-bind-port
to bind to; if not provided, those
value defaults to the IP address of the default network interface on the machine and port 6443
The
service-cluster-ip-range
to use for services
If an external etcd server is specified, the
address and related TLS settings
(
etcd-cafile
,
etcd-certfile
,
etcd-keyfile
);
if an external etcd server is not be provided, a local etcd will be used (via host network)
If a cloud provider is specified, the corresponding
is configured, together
with the
--cloud-config
path if such file exists (this is experimental, alpha and will be
removed in a future version)
Other API server flags that are set unconditionally are:
--insecure-port=0
to avoid insecure connections to the api server
--enable-bootstrap-token-auth=true
to enable the
BootstrapTokenAuthenticator
authentication module.
See
TLS Bootstrapping
for more details
to
true
(required e.g. by kube proxy)
--requestheader-client-ca-file
to
--enable-admission-plugins
to:
NamespaceLifecycle
e.g. to avoid deletion of system reserved namespaces
LimitRanger
and
to enforce limits on namespaces
to enforce service account automation
PersistentVolumeLabel
attaches region or zone labels to PersistentVolumes as defined by the cloud provider (This
admission controller is deprecated and will be removed in a future version.
It is not deployed by kubeadm by default with v1.9 onwards when not explicitly opting into
using
gce
or
aws
as cloud providers)
DefaultStorageClass
to enforce default storage

 class on
objects
DefaultTolerationSeconds
to limit what a kubelet can modify (e.g. only pods on this node)
--kubelet-preferred-address-types
to
InternalIP,ExternalIP,Hostname;
this makes
and other API server-kubelet communication work in environments where the hostnames of the
nodes aren't resolvable
Flags for using certificates generated in previous steps:
to
ca.crt
to
to
--kubelet-client-certificate
to
--kubelet-client-key
to
to
sa.pub
to
--proxy-client-cert-file
to
--proxy-client-key-file
to
Other flags for securing the front proxy
(
API Aggregation
)
communications:
--requestheader-username-headers=X-Remote-User
--requestheader-group-headers=X-Remote-Group
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-allowed-names=front-proxy-client
Controller manager
The static Pod manifest for the controller manager is affected by following parameters provided by
the users:
If kubeadm is invoked specifying a
--pod-network-cidr
, the subnet manager feature required for
some CNI network plugins is enabled by setting:
--allocate-node-cidrs=true
and
--node-cidr-mask-size
flags according to the given CIDR
is specified, together
with the
path if such configuration file exists (this is experimental, alpha
and will be removed in a future version)
Other flags that are set unconditionally are:
--controllers
enabling all the default controllers plus
BootstrapSigner
and
TokenCleaner
controllers for TLS bootstrap. See
--use-service-account-credentials
to
true
--root-ca-file
to
ca.crt
to
ca.crt
, if External CA mode is disabled, otherwise to
""
to
ca.key
""
to
sa.key
Scheduler
The static Pod manifest for the scheduler is not affected by parameters provided by the users.
Generate static Pod manifest for local etcd
If you specified an external etcd this step will be skipped, otherwise kubeadm generates a
static Pod manifest file for creating a local etcd instance running in a Pod with following attributes:
listen on
localhost:2379
and use
HostNetwork=true
make a
hostPath
mount out from the
dataDir
to the host's filesystem
Any extra flags specified by the user
The etcd container image will be pulled from
registry.gcr.io
by default. See
for customizing the image repository.
If you run kubeadm in
--dry-run
mode, the etcd static Pod manifest is written
into a temporary folder.
You can directly invoke static Pod manifest generation for local etcd, using the
kubeadm init phase etcd local
command.
Wait for the control plane to come up
kubeadm waits (upto 4m0s) until
localhost:6443/healthz
(kube-apiserver liveness) returns
ok
.
However in order to detect deadlock conditions, kubeadm fails fast if
localhost:10255/healthz
(kubelet liveness) or
localhost:10255/healthz/syncloop
(kubelet readiness) don't return
ok
within 40s and 60s respectively.
kubeadm relies on the kubelet to pull the control plane images and run them properly as static Pods.
After the control plane is up, kubeadm completes the tasks described in following paragraphs.
Save the kubeadm ClusterConfiguration in a ConfigMap for later reference
kubeadm saves the configuration passed to
in a ConfigMap named
under
This will ensure that kubeadm actions executed in future (e.g
) will be able to
determine the actual/current cluster state and mak

e new decisions based on that data.
Before saving the ClusterConfiguration, sensitive information like the token is stripped from the configuration
Upload of control plane node configuration can be invoked individually with the command
.
Mark the node as control-plane
As soon as the control plane is available, kubeadm executes following actions:
Labels the node as control-plane with
node-role.kubernetes.io/control-plane=""
Taints the node with
node-role.kubernetes.io/control-plane:NoSchedule
Please note that the phase to mark the control-plane phase can be invoked
individually with the
command.
node-role.kubernetes.io/master:NoSchedule
and
The
taint is deprecated and will be removed in kubeadm version 1.25
Mark control-plane phase phase can be invoked individually with the command
Configure TLS-Bootstrapping for node joining
Kubeadm uses
for joining new nodes to an existing cluster; for more details see also
design proposal
.
ensures that everything is properly configured for this process, and this includes
following steps as well as setting API server and controller flags as already described in
previous paragraphs.
TLS bootstrapping for nodes can be configured with the command
,
executing all the configuration steps described in following paragraphs;
alternatively, each step can be invoked individually
Create a bootstrap token
create a first bootstrap token, either generated automatically or provided by the
user with the
--token
flag; as documented in bootstrap token specification, token should be
saved as secrets with name
bootstrap-token-&lt;token-id&gt;
under
The default token created by
will be used to validate temporary user during TLS
bootstrap process; those users will be member of
system:bootstrappers:kubeadm:default-node-token
group
The token has a limited validity, default 24 hours (the interval may be changed with the
—token-ttl
flag)
Additional tokens can be created with the
command, that provide as well other useful functions for token management.
Allow joining nodes to call CSR API
Kubeadm ensures that users in
group are able to
access the certificate signing API.
This is implemented by creating a ClusterRoleBinding named
kubeadm:kubelet-bootstrap
between the
group above and the default RBAC role
system:node-bootstrapper
.
Set up auto approval for new bootstrap tokens
Kubeadm ensures that the Bootstrap Token will get its CSR request automatically approved by the
csrapprover controller.
This is implemented by creating ClusterRoleBinding named
kubeadm:node-autoapprove-bootstrap
group and the default role
system:certificates.k8s.io:certificatesigningrequests:nodeclient
.
The role
should be created as
well, granting POST permission to
/apis/certificates.k8s.io/certificatesigningrequests/nodeclient
.
Set up nodes certificate rotation with auto approval
Kubeadm ensures that certificate rotation is enabled for nodes, and that new certificate request
for nodes will get its CSR request automatically approved by the csrapprover controller.
kubeadm:node-autoapprove-certificate-rotation
group and the default
role
system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
.
Create the public cluster-info ConfigMap
This phase creates the
Additionally it creates a Role and a RoleBinding grant

ing access to the ConfigMap for
unauthenticated users (i.e. users in RBAC group
).
The access to the
ConfigMap
is not
rate-limited. This may or may not be a
problem if you expose your cluster's API server to the internet; worst-case scenario here is a
DoS attack where an attacker uses all the in-flight requests the kube-apiserver can handle to
serving the
Install addons
Kubeadm installs the internal DNS server and the kube-proxy addon components via the API server.
This phase can be invoked individually with the command
kubeadm init phase addon all
.
proxy
A ServiceAccount for
is created in the
namespace; then kube-proxy is
deployed as a DaemonSet:
The credentials (
ca.crt
and
token
) to the control plane come from the ServiceAccount
The location (URL) of the API server comes from a ConfigMap
The
ServiceAccount is bound to the privileges in the
system:node-proxier
DNS
The CoreDNS service is named
kube-dns
. This is done to prevent any interruption
in service when the user is switching the cluster DNS from kube-dns to CoreDNS
the
--config
method described
here
.
A ServiceAccount for CoreDNS is created in the
The
coredns
In Kubernetes version 1.21, support for using
kube-dns
with kubeadm was removed.
You can use CoreDNS with kubeadm even when the related Service is named
kube-dns
.
kubeadm join phases internal design
Similarly to
, also
internal workflow consists of a sequence of
atomic work tasks to perform.
This is split into discovery (having the Node trust the Kubernetes Master) and TLS bootstrap
(having the Kubernetes Master trust the Node).
see
or the corresponding
.
kubeadm
executes a set of preflight checks before starting the join, with the aim to verify
preflight checks are basically a subset
preflight checks
Starting from 1.24, kubeadm uses crictl to communicate to all known CRI endpoints.
Starting from 1.9, kubeadm provides support for joining nodes running on Windows; in that case,
linux specific controls are skipped.
In any case the user can skip specific preflight checks (or eventually all preflight checks)
with the
option.
Discovery cluster-info
There are 2 main schemes for discovery. The first is to use a shared token along with the IP
address of the API server.
The second is to provide a file (that is a subset of the standard kubeconfig file).
Shared token discovery
If
is invoked with
, token discovery is used; in this case the
node basically retrieves the cluster CA certificates from the
In order to prevent "man in the middle" attacks, several steps are taken:
First, the CA certificate is retrieved via insecure connection (this is possible because
granted access to
users for
)
Then the CA certificate goes trough following validation steps:
Basic validation: using the token ID against a JWT signature
Pub key validation: using provided
. This value is available
in the output of
or can be calculated using standard tools (the hash is
calculated over the bytes of the Subject Public Key Info (SPKI) object as in RFC7469). The
--discovery-token-ca-cert-hash flag
As a additional validation, the CA certificate is retrieved via secure connection and then
compared with the CA retrieved initially
Pub key validation can be skipped passing
flag;
This weakens the kubead

m security model since others can potentially impersonate the Kubernetes Master.
File/https discovery
If
--discovery-file
, file discovery is used; this file can be a
local file or downloaded via an HTTPS URL; in case of HTTPS, the host installed CA bundle is used
to verify the connection.
With file discovery, the cluster CA certificates is provided into the file itself; in fact, the
discovery file is a kubeconfig file with only
server
and
attributes
set, as described in
reference doc; when the connection with the cluster is established, kubeadm try to access the
ConfigMap, and if available, uses it.
Once the cluster info are known, the file
bootstrap-kubelet.conf
is written, thus allowing
kubelet to do TLS Bootstrapping.
The TLS bootstrap mechanism uses the shared token to temporarily authenticate with the Kubernetes
API server to submit a certificate signing request (CSR) for a locally created key pair.
The request is then automatically approved and the operation completes saving
ca.crt
file and
file to be used by kubelet for joining the cluster, while
is deleted.
The temporary authentication is validated against the token saved during the
process (or with additional tokens created with
)
The temporary authentication resolve to a user member of
group which was granted access to CSR api
during the
process
The automatic CSR approval is managed by the csrapprover controller, according with
configuration done the
process
Home
Blog
Training
Partners
Community
CC BY 4.0

Distribute Credentials Securely Using Secrets | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Op

erator pattern
Tasks
Debug Pods
Determine the Re

ason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the 

Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuratio

n (v1)
Tasks
This page shows how to securely inject sensitive data, such as passwords and
encryption keys, into Pods.
minikube
Convert your secret data to a base-64 representation
Suppose you want to have two pieces of secret data: a username
my-app
and a password
39528$vdg7Jb
. First, use a base64 encoding tool to convert your username and password to a base64 representation. Here's an example using the commonly available base64 program:
echo
-n
'my-app'
| base64
echo
-n
'39528$vdg7Jb'
| base64
The output shows that the base-64 representation of your username is
bXktYXBw
,
and the base-64 representation of your password is
Mzk1MjgkdmRnN0pi
.
Caution:
Use a local tool trusted by your OS to decrease the security risks of external tools.
Create a Secret
Here is a configuration file you can use to create a Secret that holds your
username and password:
pods/inject/secret.yaml
:
v1
kind
:
Secret
metadata
:
name
:
test-secret
data
:
username
:
bXktYXBw
password
:
Create the Secret
kubectl apply -f https://k8s.io/examples/pods/inject/secret.yaml
View information about the Secret:
kubectl get secret test-secret
Output:
NAME          TYPE      DATA      AGE
test-secret   Opaque    2         1m
View more detailed information about the Secret:
kubectl describe secret test-secret
Output:
Name:       test-secret
Namespace:  default
Labels:     &lt;none&gt;

Type:   Opaque

Data
====
password:   13 bytes
username:   7 bytes
Create a Secret directly with kubectl
If you want to skip the Base64 encoding step, you can create the
same Secret using the
kubectl create secret
command. For example:
kubectl create secret generic test-secret --from-literal
=
'username=my-app'
--from-literal
=
'password=39528$vdg7Jb'
This is more convenient. The detailed approach shown earlier runs
through each step explicitly to demonstrate what is happening.
Create a Pod that has access to the secret data through a Volume
Here is a configuration 

file you can use to create a Pod:
pods/inject/secret-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
secret-test-pod
spec
:
:
-
name
:
image
:
nginx
:
# name must match the volume name below
-
name
:
secret-volume
mountPath
:
/etc/secret-volume
# The secret data is exposed to Containers in the Pod through a Volume.
volumes
:
-
name
:
secret
:
secretName
:
kubectl apply -f https://k8s.io/examples/pods/inject/secret-pod.yaml
Verify that your Pod is running:
kubectl get pod secret-test-pod
Output:
NAME              READY     STATUS    RESTARTS   AGE
secret-test-pod   1/1       Running   0          42m
Get a shell into the Container that is running in your Pod:
kubectl
exec
-i -t secret-test-pod -- /bin/bash
The secret data is exposed to the Container through a Volume mounted under
.
In your shell, list the files in the
# Run this in the shell inside the container
ls /etc/secret-volume
The output shows two files, one for each piece of secret data:
password username
In your shell, display the contents of the
username
and
password
files:
echo
"
$(
cat /etc/secret-volume/username
)
"
echo
"
$(
cat /etc/secret-volume/password
)
"
The output is your username and password:
my-app
Define container environment variables using Secret data
Define a container environment variable with data from a single Secret
Define an environment variable as a key-value pair in a Secret:
kubectl create secret generic backend-user --from-literal
=
backend-username
=
'backend-admin'
Assign the
value defined in the Secret to the
SECRET_USERNAME
environment variable in the Pod specification.
pods/inject/pod-single-secret-env-variable.yaml
:
v1
kind
:
Pod
metadata
:
name
:
env-single-secret
spec
:
:
-
name
:
envars-test-container
image
:
nginx
env
:
-
name
:
valueFrom
:
secretKeyRef
:
name
:
backend-user
key
:
kubectl create -f https://k8s.io/examples/pods/inject/pod-single-secret-env-variable.yaml
In your shell, display the content of
container environment variable
kubectl
exec
-i -t env-single-secret -- /bin/sh -c
'echo $SECRET_USERNAME'
The output is
backend-admin
Define container environment variables with data from multiple Secrets
As with the previous example, create the Secrets first.
=
=
kubectl create secret generic db-user --from-literal
=
db-username
=
'db-admin'
Define the environment variables in the Pod specification.
pods/inject/pod-multiple-secret-env-variable.yaml
:
v1
kind
:
Pod
metadata
:
name
:
envvars-multiple-secrets
spec
:
:
-
name
:
image
:
nginx
env
:
-
name
:
BACKEND_USERNAME
valueFrom
:
:
name
:
key
:
-
name
:
DB_USERNAME
valueFrom
:
:
name
:
db-user
key
:
kubectl create -f https://k8s.io/examples/pods/inject/pod-multiple-secret-env-variable.yaml
In your shell, display the container environment variables
kubectl
exec
-i -t envvars-multiple-secrets -- /bin/sh -c
'env | grep _USERNAME'
DB_USERNAME=db-admin
BACKEND_USERNAME=backend-admin
Configure all key-value pairs in a Secret as container environment variables
Note:
This functionality is available in Kubernetes v1.6 and later.
Create a Secret containing multiple key-value pairs
=
username
=
'my-app'
=
password
=
Use envFrom to define all of the Secret's data as container environment variables. The key from the Secret becomes the environment variable name in the Pod.
pods/inject/pod-secret-envFrom.yaml
:
v1
kind
:
Pod
metadata
:
name
:
envfrom-secret
spec
:

:
-
name
:
image
:
nginx
envFrom
:
-
secretRef
:
name
:
kubectl create -f https://k8s.io/examples/pods/inject/pod-secret-envFrom.yaml
In your shell, display
username
and
password
container environment variables
kubectl
exec
-i -t envfrom-secret -- /bin/sh -c
'echo "username: $username\npassword: $password\n"'
username: my-app
password: 39528$vdg7Jb
Secret
Volume
Pod
Secrets
.
Volumes
.
Feedback
Yes
No
.
or
.
Last modified September 09, 2022 at 10:54 PM PST:
fix some layout (f6fa07357d)
Home
Blog
Training
Partners
Community
CC BY 4.0

Reserve Compute Resources for System Daemons | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

Tasks
Capacity
node.
The
kubelet
minikube
.
to set an
.
v1.Node
of
in the CLI.
Resources can be reserved for two categories of system daemons in the
kubelet
.
kubelet
kubelet
.
The
kubelet
flag.
cgroupfs
systemd
is an alternative dr

iver that manages cgroup sandboxes using
systemd
kubelet
must
systemd
:
:
kubelet
,
,
, etc.
cpu
,
memory
, and
,
pid
may be
kubelet
flag.
does not
create
systemd
,
with
.slice
appended.
:
:
like
sshd
,
udev
, etc.
memory
for the
kernel
too since
kernel
in
cpu
,
memory
, and
,
pid
may be
daemons.
Note that
kubelet
does not
create
exist.
kubelet
systemd
,
with
.slice
appended.
:
:
does not
have
and
,
defined by
and
options.
interrupts/timers

, so the rest CPUs on the system can be used exclusively for
:
provides
memory
and
flag, the
kubelet
:
capacity
for pods.
kubelet
on the
pods
.
kubelet
and
&amp;
values in
or
,
or
.
kubelet
should
pods
.
over time.
down utilization of node system daemons, but that is not a priority as of now.
Node has
32Gi
of
memory
,
16 CPUs
and
100Gi
of
Storage
is set to
is set to
is set to
88Gi
requests
or if overa

ll disk usage exceeds 88Gi If all processes on the node consume as
If
and/or
kubelet
storage
Feedback
Yes
No
.
or
.
Last modified December 23, 2022 at 7:23 PM PST:
Add reservedSystemCpus in KubeletConfiguration as the flag is not the same (3662a15bec)
Home
Blog
Training
Partners
Community
CC BY 4.0

HorizontalPodAutoscaler | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
configuration of a horizontal pod autoscaler.
apiVersion: autoscaling/v1
import "k8s.io/api/autoscaling/v1"
: autoscaling/v1
kind
: HorizontalPodAutoscaler
metadata
(
)
Standard object metadata. More info:
spec
(
HorizontalPodAutoscalerSpec
)
behaviour of autoscaler. More info:
.
status
(
HorizontalPodAutoscalerStatus
)
current information about the autoscaler.
specification of a horizontal pod autoscaler.
maxReplicas
upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than MinReplicas.
scaleTargetRef
(CrossVersionObjectReference), required
reference to scaled resource; horizontal pod autoscaler will learn the current resource consumption and will set the desired number of pods by using its Scale subresource.
CrossVersionObjectReference contains enough information to let you identify the referred resource.
scaleTargetRef.kind
Kind of the referent; More info:
scaleTargetRef.name
Name of the referent; More info:
http://kubernetes.io/docs/user-guide/identifiers#names
scaleTargetRef.apiVersion
(string)
API version of the referent
minReplicas
(int32)
minReplicas is the lower limit for the number of replicas to which the autoscaler can scale down. It defaults to 1 pod. minReplicas is allowed to be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one Object or External metric is configured. Scaling is active as long as at least one metric value is available.
targetCPUUtilizationPercentage
(int32)
target average CPU utilization (represented as a percentage of requested CPU) over all the pods; if not specified the default autoscaling policy will be used.
current status of a horizontal pod autoscaler
current number of replicas of pods managed by this autoscaler.
desiredReplicas
desired number of replicas of pods managed by this autoscaler.
currentCPUUtilizationPercentage
(int32)
current average CPU utilization over all pods, represented as a percentage of requested CPU, e.g. 70 means that an avera

ge pod is using now 70% of its requested CPU.
lastScaleTime
(Time)
last time the HorizontalPodAutoscaler scaled the number of pods; used by the autoscaler to control how often the number of pods is changed.
(int64)
most recent generation observed by this autoscaler.
HorizontalPodAutoscalerList
list of horizontal pod autoscaler objects.
kind
: HorizontalPodAutoscalerList
metadata
(
ListMeta
)
items
([]
get
read the specified HorizontalPodAutoscaler
GET /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
name of the HorizontalPodAutoscaler
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified HorizontalPodAutoscaler
GET /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind HorizontalPodAutoscaler
GET /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/autoscaling/v1/horizontalpodautoscalers
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a HorizontalPodAutoscaler
POST /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified HorizontalPodAutoscaler
PUT /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
Horizo

ntalPodAutoscaler
): Created
update
replace status of the specified HorizontalPodAutoscaler
PUT /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified HorizontalPodAutoscaler
PATCH /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified HorizontalPodAutoscaler
PATCH /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a HorizontalPodAutoscaler
DELETE /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of HorizontalPodAutoscaler
DELETE /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
HorizontalPodAutoscale

rList
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

ObjectFieldSelector | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 on macOS
Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access t

o Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-

proxy Configuration (v1alpha1)
Reference
ObjectFieldSelector selects an APIVersioned field of an object.
fieldPath
Path of the field to select in the specified API version.
(string)
Version of the schema the FieldPath is written in terms of, defaults to "v1".
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

ClusterRole | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

Reference
apiVersion: rbac.authorization.k8s.io/v1
import "k8s.io/api/rbac/v1"
: rbac.authorization.k8s.io/v1
kind
: ClusterRole
metadata
(
)
aggregationRule
(AggregationRule)
AggregationRule is an optional field that describes how to build the Rules for this ClusterRole. If AggregationRule is set, then the Rules are controller managed and direct changes to Rules will be stomped by the controller.
AggregationRule describes how to locate ClusterRoles to aggregate into the ClusterRole
aggregationRule.clusterRoleSelectors
([]
)
ClusterRoleSelectors holds a list of selectors which will be used to find ClusterRoles and create the rules. If any of the selectors match, then the ClusterRole's permissions will be added
rules
([]PolicyRule)
Rules holds all the PolicyRules for this ClusterRole
PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace the rule applies to.
rules.apiGroups
([]string)
APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed. "" represents the core API group and "*" represents all API groups.
rules.resources
Resources is a list of resources this rule applies to. '*' represents all resources.
rules.verbs
Verbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '*' represents all verbs.
rules.resourceNames
ResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed.
rules.nonResourceURLs
NonResourceURLs is a set of partial urls that a user should have access to. *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"), but not both.
ClusterRoleList
ClusterRoleList is a collection of ClusterRoles
kind
: ClusterRoleList
metadata

(
ListMeta
)
items
([]
Items is a list of ClusterRoles
get
read the specified ClusterRole
GET /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
name
(
in path
name of the ClusterRole
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ClusterRole
GET /apis/rbac.authorization.k8s.io/v1/clusterroles
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ClusterRole
POST /apis/rbac.authorization.k8s.io/v1/clusterroles
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ClusterRole
PUT /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ClusterRole
PATCH /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ClusterRole
DELETE /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ClusterRole
DELETE /apis/rbac.authorization.k8s.io/v1/clusterroles
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
Open an issue in the GitHub repo if y

ou want to
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Verify Signed Kubernetes Artifacts | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on Li

nux
Debug Pods
Get a

 Shell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
R

estrict a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encry

ption Configuration (v1)
Tasks
cosign
(
)
curl
URL
=
BINARY
=
kubectl
FILES
=(
"
$BINARY
"
"
$BINARY
.sig"
"
$BINARY
.cert"
)
for
FILE in
"
${
FILES
[@]
}
"
;
do
3
3
"
$URL
/
$FILE
"
-o
"
$FILE
"
done
cosign
:
"
$BINARY
"
"
$BINARY
"
"
$BINARY
"
.cert
Note:
Keyless
.
to
Releases
.
the
command:
=
1
Note:
in
KEYLESS
.
$(
)
| awk
input
=
while
IFS
=
read
-r image
do
=
1
"
$image
"
done
&lt;
"
$input
"
manifests as per this
example:
.
to
section.
e.g.
)
Configuration 

Options
Feedback
Yes
No
.
or
.
Last modified November 11, 2022 at 9:22 AM PST:
Add documentation about signed Kubernetes artifacts (f8a2d2319a)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

alpha1)
Tasks
- Useful
- Useful
release
Questions
Concepts
Setup
provides
Tasks
show how to
Tutorials
are more
Reference
kubectl
.
.
please
,
before
!
Slack
channel.
,
channel.
Country
Channels
China
#cn-users
,
#cn-even

ts
Finland
#fi-users
France
#fr-users
,
Germany
#de-users
,
India
#in-users
,
Italy
#it-users
,
Japan
#jp-users
,
Korea
#kr-users
#nl-users
Norway
Poland
#pl-users
Russia
#ru-users
Spain
#es-users
Sweden
#se-users
Turkey
#tr-users
,
Forum
.
.
Before you file an issue, please search existing issues to see if your issue is
Feedback
Yes
No
.
or
.
Questions
Slack
Forum
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubelet Device Manager API Versions | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
This page provides details of version compatibility between the Kubernetes
device plugin API
,
and different versions of Kubernetes itself.
Compatibility matrix
v1alpha1
v1beta1
Kubernetes 1.21
-
✓
Kubernetes 1.22
-
✓
Kubernetes 1.23
-
✓
Kubernetes 1.24
-
✓
Kubernetes 1.25
-
✓
Kubernetes 1.26
-
✓
Key:
✓
Exactly the same features / API objects in both device plugin API and
the Kubernetes version.
+
The device plugin API has features or API objects that may not be present in the
Kubernetes cluster, either because the device plugin API has added additional new API
calls, or that the server has removed an old API call. However, everything they have in
common (most other APIs) will work. Note that alpha APIs may vanish or
change significantly between one minor release and the next.
-
The Kubernetes cluster has features the device plugin API can't use,
either because server has added additional API calls, or that device plugin API has
removed an old API call. However, everything they share in common (most APIs) will work.
Feedback
Yes
No
.
or
.
Last modified November 04, 2022 at 10:19 AM PST:
node: devicemgr: docs: Updates based on review comments (Nov 4, 2022) (b353fbe22f)
Home
Blog
Training
Partners
Community
CC BY 4.0

Reviewing pull requests | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

tall and Set Up kubectl on Linux
Debug Pods
Debug Init Contai

ners
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Stand

ards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

1alpha1)
Anyone can review a documentation pull request. Visit the
pull requests
section in the Kubernetes website repository to see open pull requests.
Reviewing documentation pull requests is a great way to introduce yourself to the Kubernetes
community. It helps you learn the code base and build trust with other contributors.
Before reviewing, it's a good idea to:
Read the
and
style guide
so you can leave informed comments.
Understand the different
roles and responsibilities
in the Kubernetes documentation community.
Before you start a review:
Read the
CNCF Code of Conduct
and ensure that you abide by it at all times.
Be polite, considerate, and helpful.
Comment on positive aspects of PRs as well as changes.
Be empathetic and mindful of how your review may be received.
Assume good intent and ask clarifying questions.
Experienced contributors, consider pairing with new contributors whose work requires extensive changes.
Review process
In general, review pull requests for content and style in English. Figure 1 outlines the steps for
the review process. The details for each step follow.
flowchart LR
subgraph fourth[Start review]
S[ ] -.-
M[add comments] --&gt; N[review changes]
N --&gt; O[new contributors should
choose Comment]
end
subgraph third[Select PR]
T[ ] -.-
J[read description
and comments]--&gt; K[preview changes in
Netlify preview build]
end
A[Review open PR list]--&gt; B[Filter open PRs
by label]
B --&gt; third --&gt; fourth
classDef grey fill:#dddddd,stroke:#ffffff,stroke-width:px,color:#000000, font-size:15px;
classDef white fill:#ffffff,stroke:#000,stroke-width:px,color:#000,font-weight:bold
class A,B,J,K,M,N,O grey
class S,T spacewhite
class third,fourth white
enabled
Figure 1. Review process steps.
Go to
https://github.com/kubernetes/website/pulls
.
You see a list of every open pull request against the Kubernetes website and docs.
Filter the open PRs using one or all of the following labels:
cncf-cla: yes
(Recommended): PRs submitted by contributors who have not signed the CLA
cannot be merged. See
Sign the CLA
language/en
(Recommended): Filters for english language PRs only.
size/&lt;size&gt;
: filters for PRs of a certain size. If you're new, start with smaller PRs.
Additionally, ensure the PR isn't marked as a work in progress. PRs using the
w

ork in progress
label are not ready for review yet.
Once you've selected a PR to review, understand the change by:
Reading the PR description to understand the changes made, and read any linked issues
Reading any comments by other reviewers
Clicking the
Files changed
tab to see the files and lines changed
Previewing the changes in the Netlify preview build by scrolling to the PR's build check
section at the bottom of the
Conversation
tab.
Here's a screenshot (this shows GitHub's desktop site; if you're reviewing
on a tablet or smartphone device, the GitHub web UI is slightly different):
To open the preview, click on the
Details
link of the
deploy/netlify
line in the
list of checks.
Go to the
tab to start your review.
Click on the
+
symbol beside the line you want to comment on.
Fill in any comments you have about the line and click either
Add single comment
(if you have only one comment to make) or
Start a review
(if you have multiple comments to make).
When finished, click
Review changes
at the top of the page. Here, you can add
a summary of your review (and leave some positive comments for the contributor!).
Please always use the "Comment"
Avoid clicking the "Request changes" button when finishing your review.
If you want to block a PR from being merged before some further changes are made,
you can leave a "/hold" comment.
Mention why you are setting a hold, and optionally specify the conditions under
which the hold can be removed by you or other reviewers.
Avoid clicking the "Approve" button when finishing your review.
Leaving a "/approve" comment is recommended most of the time.
Reviewing checklist
When reviewing, use the following as a starting point.
Language and grammar
Are there any obvious errors in language or grammar? Is there a better way to phrase something?
Focus on the language and grammar of the parts of the page that the author is changing.
Unless the author is clearly aiming to update the entire page, they have no obligation to
fix every issue on the page.
When a PR updates an existing page, you should focus on reviewing the parts of
the page that are being updated. That changed content should be reviewed for technical
and editorial correctness.
If you find errors on the page that don't directly relate to what the PR author
is attempting to address, then it should be treated as a separate issue (check
that there isn't an existing issue about this first).
Watch out for pull requests that
move
content. If an author renames a page
or combines two pages, we (Kubernetes SIG Docs) usually avoid asking that author to fix every grammar or spelling nit
that we could spot within that moved content.
Are there any complicated or archaic words which could be replaced with a simpler word?
Are there any words, terms or phrases in use which could be replaced with a non-discriminatory alternative?
Does the word choice and its capitalization follow the
?
Are there long sentences which could be shorter or less complex?
Are there any long paragraphs which might work better as a list or table?
Content
Does similar content exist elsewhere on the Kubernetes site?
Does the content excessively link to off-site, individual vendor or non-open source documentation?
Website
Did this PR change or remove a page title, slug/alias or anchor link? If so, are there broken
links as a result of this PR? Is there another option, like changing the page title without
changing the slug?
Does the PR introduce a new page? If so:
Is the page using the right
page content type
and associated Hugo shortcodes?
Does the page appear correctly in the section's side navigation (or at all)?
Should the page appear on the
Docs Home
listing?
Do the changes show up in the Netlify preview? Be particularly vigilant about lists, code
blocks, tables, notes and images.
Other
Watch out for
trivial edits
;
if you see a change that you think is a trivial edit, please point out that policy
(it's still OK to accept the change if it is genuinely an i

mprovement).
Encourage authors who are making whitespace fixes to do
so in the first commit of their PR, and then add other changes on top of that. This
makes both merges and reviews easier. Watch out especially for a trivial change that
happens in a single commit along with a large amount of whitespace cleanup
(and if you see that, encourage the author to fix it).
As a reviewer, if you identify small issues with a PR that aren't essential to the meaning,
such as typos or incorrect whitespace, prefix your comments with
nit:
.
This lets the author know that this part of your feedback is non-critical.
If you are considering a pull request for approval and all the remaining feedback is
marked as a nit, you can merge the PR anyway. In that case, it's often useful to open
an issue about the remaining nits. Consider whether you're able to meet the requirements
for marking that new issue as a
Good First Issue
;
if you can, these are a good source.
Feedback
Yes
No
.
or
.
Last modified September 28, 2022 at 1:39 PM PST:
Revise PR review guidelines (336cd5f23e)
Content
Website
Other
Home
Blog
Training
Partners
Community
CC BY 4.0

Debug a StatefulSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on Linu

x
Debug Pods
Get a S

hell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Res

trict a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encrypt

ion Configuration (v1)
Tasks
=
MyApp
Unknown
or
task for
guide.
.
Feedback
Yes
No
.
or
.
Last modified July 11, 2022 at 8:41 AM PST:
[en] update en docs to use recommended labels (f9ebc90ff7)
Home
Blog
Training
Partners
Community
CC BY 4.0

kubectl Cheat Sheet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Cust

om Resources
Tasks
Debug Pods
Deb

ug Services
Auditing
Run Jobs
TLS
Tutorials
Configuring Redis using a ConfigM

ap
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Audi

t Configuration (v1)
Reference
This page contains a list of commonly used
kubectl
commands and flags.
Kubectl autocomplete
BASH
source
&lt;
(
kubectl completion bash
)
# set up autocomplete in bash into the current shell, bash-completion package should be installed first.
echo
"source &lt;(kubectl completion bash)"
&gt;&gt; ~/.bashrc
# add autocomplete permanently to your bash shell.
You can also use a shorthand alias for
kubectl
that also works with completion:
alias
k
=
kubectl
complete
-o default -F __start_kubectl k
ZSH
source
&lt;
(
kubectl completion zsh
)
# set up autocomplete in zsh into the current shell
echo
'[[ $commands[kubectl] ]] &amp;&amp; source &lt;(kubectl completion zsh)'
&gt;&gt; ~/.zshrc
# add autocomplete permanently to your zsh shell
A note on
Appending
happens frequently enough where you should be aware of the shorthand for
:
kubectl -A
Kubectl context and configuration
Set which Kubernetes cluster
kubectl
communicates with and modifies configuration
information. See
Authenticating Across Clusters with kubeconfig
documentation for
detailed config file information.
# Show Merged kubeconfig settings.
# use multiple kubeconfig files at the same time and view merged config
=
~/.kube/config:~/.kube/kubconfig2
# get the password for the e2e user
jsonpath
=
'{.users[?(@.name == "e2e")].user.password}'
jsonpath
=
'{.users[].name}'
# display the first user
jsonpath
=
'{.users[*].name}'
# get a list of users
kubectl config get-contexts
# display list of contexts
# display the current-context
kubectl config use-context my-cluster-name
# set the default context to my-cluster-name
kubectl config set-cluster my-cluster-name
# set a cluster entry in the kubeconfig
# configure the URL to a proxy server to use for requests made by this client in the kubeconfig
kubectl config set-cluster my-cluster-name --proxy-url
=
my-proxy-url
# add a new user to your kubeconf that supports basic auth
kubectl config set-credentials kubeuser/foo.kubernetes.com --username
=
kubeuser --password
=
kubepassword
# permanently save the namespace for all subsequent kubectl commands in that context.
=
ggckad-s2
# set a context utilizing a specific username 

and namespace.
kubectl config set-context gce --user
=
cluster-admin --namespace
=
foo
\
kubectl config use-context gce
kubectl config
unset
users.foo
# delete user foo
# short alias to set/show context/namespace (only works for bash and bash-compatible shells, current context to be set before using kn to set namespace)
alias
kx
=
'f() { [ "$1" ] &amp;&amp; kubectl config use-context $1 || kubectl config current-context ; } ; f'
alias
kn
=
'f() { [ "$1" ] &amp;&amp; kubectl config set-context --current --namespace $1 || kubectl config view --minify | grep namespace | cut -d" " -f6 ; } ; f'
Kubectl apply
apply
manages applications through files defining Kubernetes resources. It creates and updates resources in a cluster through running
. This is the recommended way of managing Kubernetes applications on production. See
Kubectl Book
.
Creating objects
Kubernetes manifests can be defined in YAML or JSON. The file extension
.yaml
,
.yml
, and
.json
kubectl apply -f ./my-manifest.yaml
# create resource(s)
kubectl apply -f ./my1.yaml -f ./my2.yaml
# create from multiple files
kubectl apply -f ./dir
# create resource(s) in all manifest files in dir
kubectl apply -f https://git.io/vPieo
# create resource(s) from url
kubectl create deployment nginx --image
=
nginx
# start a single instance of nginx
# create a Job which prints "Hello World"
kubectl create job hello --image
=
busybox:1.28 --
echo
"Hello World"
# create a CronJob that prints "Hello World" every minute
kubectl create cronjob hello --image
=
busybox:1.28   --schedule
=
"*/1 * * * *"
--
echo
kubectl explain pods
# get the documentation for pod manifests
# Create multiple YAML objects from stdin
cat
&lt;&lt;EOF | kubectl apply -f -
kind: Pod
metadata:
name: busybox-sleep
spec:
- name: busybox
image: busybox:1.28
args:
- sleep
- "1000000"
---
kind: Pod
metadata:
name: busybox-sleep-less
spec:
args:
- sleep
- "1000"
EOF
# Create a secret with several keys
cat
metadata:
name: mysecret
type: Opaque
data:
password: $(echo -n "s33msi4" | base64 -w0)
username: $(echo -n "jane" | base64 -w0)
EOF
Viewing and finding resources
# Get commands with basic output
# List all services in the namespace
# List all pods in all namespaces
# List all pods in the current namespace, with more details
kubectl get deployment my-dep
# List a particular deployment
# List all pods in the namespace
kubectl get pod my-pod -o yaml
# Get a pod's YAML
# Describe commands with verbose output
kubectl describe nodes my-node
kubectl describe pods my-pod
# List Services Sorted by Name
kubectl get services --sort-by
=
# List pods Sorted by Restart Count
=
'.status.containerStatuses[0].restartCount'
# List PersistentVolumes sorted by capacity
kubectl get pv --sort-by
=
.spec.capacity.storage
# Get the version label of all pods with label app=cassandra
=
app
=
cassandra -o
\
jsonpath
=
'{.items[*].metadata.labels.version}'
# Retrieve the value of a key with dots, e.g. 'ca.crt'
kubectl get configmap myconfig
\
-o
jsonpath
=
'{.data.ca\.crt}'
# Retrieve a base64 encoded value with dashes instead of underscores.
kubectl get secret my-secret --template
=
'{{index .data "key-name-with-dashes"}}'
# Get all worker nodes (use a selector to exclude results that have a label
# named 'node-role.kubernetes.io/control-plane')
kubectl get node --selector
=
'!node-role.kubernetes.io/control-plane'
# Get all running pods in the namespace
=
status.phase
=
Running
# Get ExternalIPs of all nodes
jsonpath
=
'{.items[*].status.addresses[?(@.type=="ExternalIP")].address}'
# List Names of Pods that belong to Particular RC

# "jq" command useful for transformations that are too complex for jsonpath, it can be found at https://stedolan.github.io/jq/
sel
=
${
$(
kubectl get rc my-rc --output
=
json | jq -j
'.spec.selector | to_entries | .[] | "\(.key)=\(.value),"'
)
%?
}
echo
$(
=
$sel
--output
=
jsonpath
={
.items..metadata.name
}
)
# Show labels for all pods (or any other Kubernetes object that supports labelling)
kubectl get pods --show-labels
# Check which nodes are ready
JSONPATH
=
'{range .items[*]}{@.metadata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}'
\
jsonpath
=
"
$JSONPATH
"
| grep
"Ready=True"
# Output decoded secrets without external tools
kubectl get secret my-secret -o go-template
=
'{{range $k,$v := .data}}{{"### "}}{{$k}}{{"\n"}}{{$v|base64decode}}{{"\n\n"}}{{end}}'
# List all Secrets currently in use by a pod
kubectl get pods -o json | jq
'.items[].spec.containers[].env[]?.valueFrom.secretKeyRef.name'
| grep -v null | sort | uniq
# List all containerIDs of initContainer of all pods
# Helpful when cleaning up stopped containers, while avoiding removal of initContainers.
kubectl get pods --all-namespaces -o
jsonpath
=
'{range .items[*].status.initContainerStatuses[*]}{.containerID}{"\n"}{end}'
| cut -d/ -f3
# List Events sorted by timestamp
kubectl get events --sort-by
=
.metadata.creationTimestamp
# List all warning events
kubectl events --types
=
Warning
# Compares the current state of the cluster against the state that the cluster would be in if the manifest was applied.
kubectl diff -f ./my-manifest.yaml
# Produce a period-delimited tree of all keys returned for nodes
# Helpful when locating a key within a complex nested JSON structure
kubectl get nodes -o json | jq -c
'paths|join(".")'
# Produce a period-delimited tree of all keys returned for pods, etc
kubectl get pods -o json | jq -c
# Produce ENV for all pods, assuming you have a default container for the pods, default namespace and the `env` command is supported.
# Helpful when running any supported command across all pods, not just `env`
for
pod in
$(
kubectl get po --output
=
jsonpath
={
}
)
;
do
echo
$pod
kubectl
exec
-it
$pod
-- env;
done
# Get a deployment's status subresource
kubectl get deployment nginx-deployment --subresource
=
status
Updating resources
kubectl
set
image deployment/frontend
www
=
image:v2
# Rolling update "www" containers of "frontend" deployment, updating the image
kubectl rollout
history
deployment/frontend
# Check the history of deployments including the revision
kubectl rollout undo deployment/frontend
# Rollback to the previous deployment
kubectl rollout undo deployment/frontend --to-revision
=
2
# Rollback to a specific revision
kubectl rollout status -w deployment/frontend
# Watch rolling update status of "frontend" deployment until completion
kubectl rollout restart deployment/frontend
# Rolling restart of the "frontend" deployment
cat pod.json | kubectl replace -f -
# Replace a pod based on the JSON passed into stdin
# Force replace, delete and then re-create the resource. Will cause a service outage.
kubectl replace --force -f ./pod.json
# Create a service for a replicated nginx, which serves on port 80 and connects to the containers on port 8000
kubectl expose rc nginx --port
=
80
=
8000
# Update a single-container pod's image version (tag) to v4
kubectl get pod mypod -o yaml | sed
's/\(image: myimage\):.*$/\1:v4/'
| kubectl replace -f -
kubectl label pods my-pod new-label
=
awesome
# Add a Label
kubectl label pods my-pod new-label-
# Remove a label
kubectl annotate pods my-pod icon-url
=
http://goo.gl/XXBTWq
# Add an annotation
kubectl autoscale deployment foo --min
=
2
--max
=
10
# Auto scale a deployment "foo"
Patching resources
# Partially update a node
kubectl patch node k8s-node-1 -p
'{"spec":{"unschedulable":true}}'
# Update a container's image; spec.containers[*].name is required be

cause it's a merge key
kubectl patch pod valid-pod -p
'{"spec":{"containers":[{"name":"kubernetes-serve-hostname","image":"new image"}]}}'
# Update a container's image using a json patch with positional arrays
kubectl patch pod valid-pod --type
=
'json'
-p
=
'[{"op": "replace", "path": "/spec/containers/0/image", "value":"new image"}]'
# Disable a deployment livenessProbe using a json patch with positional arrays
kubectl patch deployment valid-deployment  --type json   -p
=
'[{"op": "remove", "path": "/spec/template/spec/containers/0/livenessProbe"}]'
# Add a new element to a positional array
kubectl patch sa default --type
=
'json'
-p
=
'[{"op": "add", "path": "/secrets/1", "value": {"name": "whatever" } }]'
# Update a deployment's replica count by patching its scale subresource
kubectl patch deployment nginx-deployment --subresource
=
'scale'
--type
=
'merge'
-p
'{"spec":{"replicas":2}}'
Editing resources
Edit any API resource in your preferred editor.
kubectl edit svc/docker-registry
# Edit the service named docker-registry
=
"nano"
# Use an alternative editor
Scaling resources
kubectl scale --replicas
=
3
rs/foo
# Scale a replicaset named 'foo' to 3
=
3
-f foo.yaml
# Scale a resource specified in "foo.yaml" to 3
kubectl scale --current-replicas
=
2
=
3
deployment/mysql
# If the deployment named mysql's current size is 2, scale mysql to 3
=
5
rc/foo rc/bar rc/baz
# Scale multiple replication controllers
Deleting resources
kubectl delete -f ./pod.json
# Delete a pod using the type and name specified in pod.json
kubectl delete pod unwanted --now
# Delete a pod with no grace period
kubectl delete pod,service baz foo
# Delete pods and services with same names "baz" and "foo"
kubectl delete pods,services -l
name
=
myLabel
# Delete pods and services with label name=myLabel
kubectl -n my-ns delete pod,svc --all
# Delete all pods and services in namespace my-ns,
# Delete all pods matching the awk pattern1 or pattern2
kubectl get pods  -n mynamespace --no-headers
=
true
| awk
'/pattern1|pattern2/{print $1}'
| xargs  kubectl delete -n mynamespace pod
Interacting with running Pods
kubectl logs my-pod
# dump pod logs (stdout)
kubectl logs -l
name
=
myLabel
# dump pod logs, with label name=myLabel (stdout)
kubectl logs my-pod --previous
# dump pod logs (stdout) for a previous instantiation of a container
kubectl logs my-pod -c my-container
# dump pod container logs (stdout, multi-container case)
name
=
myLabel -c my-container
kubectl logs my-pod -c my-container --previous
# dump pod container logs (stdout, multi-container case) for a previous instantiation of a container
kubectl logs -f my-pod
# stream pod logs (stdout)
kubectl logs -f my-pod -c my-container
# stream pod container logs (stdout, multi-container case)
kubectl logs -f -l
name
=
myLabel --all-containers
# stream all pods logs with label name=myLabel (stdout)
kubectl run -i --tty busybox --image
=
busybox:1.28 -- sh
# Run pod as interactive shell
kubectl run nginx --image
=
nginx -n mynamespace
# Start a single instance of nginx pod in the namespace of mynamespace
=
nginx --dry-run
=
client -o yaml &gt; pod.yaml
# Generate spec for running pod nginx and write it into a file called pod.yaml
kubectl attach my-pod -i
# Attach to Running Container
kubectl port-forward my-pod 5000:6000
# Listen on port 5000 on the local machine and forward to port 6000 on my-pod
kubectl
exec
my-pod -- ls /
# Run command in existing pod (1 container case)
kubectl
exec
--stdin --tty my-pod -- /bin/sh
# Interactive shell access to a running pod (1 container case)
kubectl
exec
my-pod -c my-container -- ls /
# Run command in existing pod (multi-container case)
kubectl top pod POD_NAME --containers
# Show metrics for a given pod and its containers
kubectl top pod POD_NAME --sort-by
=
cpu
# Show metrics for a

 given pod and sort it by 'cpu' or 'memory'
Copying files and directories to and from containers
kubectl cp /tmp/foo_dir my-pod:/tmp/bar_dir
# Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the current namespace
kubectl cp /tmp/foo my-pod:/tmp/bar -c my-container
# Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container
kubectl cp /tmp/foo my-namespace/my-pod:/tmp/bar
# Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace my-namespace
kubectl cp my-namespace/my-pod:/tmp/foo /tmp/bar
# Copy /tmp/foo from a remote pod to /tmp/bar locally
Note:
kubectl cp
requires that the 'tar' binary is present in your container image. If 'tar' is not present,
kubectl cp
will fail.
For advanced use cases, such as symlinks, wildcard expansion or file mode preservation consider using
.
tar cf - /tmp/foo | kubectl
exec
-i -n my-namespace my-pod -- tar xf - -C /tmp/bar
kubectl
exec
-n my-namespace my-pod -- tar cf - /tmp/foo | tar xf - -C /tmp/bar
Interacting with Deployments and Services
kubectl logs deploy/my-deployment
# dump Pod logs for a Deployment (single-container case)
kubectl logs deploy/my-deployment -c my-container
# dump Pod logs for a Deployment (multi-container case)
kubectl port-forward svc/my-service
5000
# listen on local port 5000 and forward to port 5000 on Service backend
kubectl port-forward svc/my-service 5000:my-service-port
# listen on local port 5000 and forward to Service target port with name &lt;my-service-port&gt;
kubectl port-forward deploy/my-deployment 5000:6000
# listen on local port 5000 and forward to port 6000 on a Pod created by &lt;my-deployment&gt;
kubectl
exec
deploy/my-deployment -- ls
# run command in first Pod and first container in Deployment (single- or multi-container cases)
Interacting with Nodes and cluster
kubectl cordon my-node
# Mark my-node as unschedulable
kubectl drain my-node
# Drain my-node in preparation for maintenance
kubectl uncordon my-node
# Mark my-node as schedulable
kubectl top node my-node
# Show metrics for a given node
kubectl cluster-info
# Display addresses of the master and services
# Dump current cluster state to stdout
kubectl cluster-info dump --output-directory
=
/path/to/cluster-state
# Dump current cluster state to /path/to/cluster-state
# View existing taints on which exist on current nodes.
=
'custom-columns=NodeName:.metadata.name,TaintKey:.spec.taints[*].key,TaintValue:.spec.taints[*].value,TaintEffect:.spec.taints[*].effect'
# If a taint with that key and effect already exists, its value is replaced as specified.
kubectl taint nodes foo
dedicated
=
special-user:NoSchedule
List all supported resource types along with their shortnames,
API group
, whether they are
, and
Kind
:
Other operations for exploring API resources:
kubectl api-resources --namespaced
=
true
# All namespaced resources
=
false
# All non-namespaced resources
kubectl api-resources -o name
# All resources with simple output (only the resource name)
kubectl api-resources -o wide
# All resources with expanded (aka "wide") output
kubectl api-resources --verbs
=
list,get
# All resources that support the "list" and "get" request verbs
kubectl api-resources --api-group
=
# All resources in the "extensions" API group
To output details to your terminal window in a specific format, add the
-o
(or
--output
) flag to a supported
kubectl
command.
-o=custom-columns=&lt;spec&gt;
Print a table using a comma separated list of custom columns
-o=custom-columns-file=&lt;filename&gt;
Print a table using the custom columns template in the
file
-o=json
Output a JSON formatted API object
-o=jsonpath=&lt;template&gt;
Print the

 fields defined in a
jsonpath
expression
-o=jsonpath-file=&lt;filename&gt;
jsonpath
file
-o=name
Print only the resource name and nothing else
-o=wide
Output in the plain-text format with any additional information, and for pods, the node name is included
-o=yaml
Output a YAML formatted API object
Examples using
-o=custom-columns
:
# All images running in a cluster
kubectl get pods -A -o
=
=
'DATA:spec.containers[*].image'
# All images running in namespace: default, grouped by Pod
kubectl get pods --namespace default --output
=
=
"NAME:.metadata.name,IMAGE:.spec.containers[*].image"
# All images excluding "registry.k8s.io/coredns:1.6.2"
=
=
'DATA:spec.containers[?(@.image!="registry.k8s.io/coredns:1.6.2")].image'
# All fields under metadata regardless of name
=
=
'DATA:metadata.*'
More examples in the kubectl
.
Kubectl output verbosity and debugging
Kubectl verbosity is controlled with the
-v
or
--v
flags followed by an integer representing the log level. General Kubernetes logging conventions and the associated log levels are described
here
.
Verbosity
--v=0
Generally useful for this to
always
be visible to a cluster operator.
--v=1
A reasonable default log level if you don't want verbosity.
--v=2
Useful steady state information about the service and important log messages that may correlate to significant changes in the system. This is the recommended default log level for most systems.
--v=3
Extended information about changes.
--v=4
Debug level verbosity.
--v=5
Trace level verbosity.
--v=6
Display requested resources.
--v=7
Display HTTP request headers.
--v=8
Display HTTP request contents.
--v=9
Display HTTP request contents without truncation of contents.
Read the
kubectl overview
and learn about
JsonPath
.
See
kubectl
options.
Also read
to understand how to use kubectl in reusable scripts.
See more community
kubectl cheatsheets
.
Feedback
Yes
No
.
or
.
Last modified November 22, 2022 at 9:21 PM PST:
Fix typos in cheatsheet.md (3e20970eb3)
BASH
ZSH
A note on
Home
Blog
Training
Partners
Community
CC BY 4.0

CustomResourceDefinition | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set 

Up kubectl on macOS
Debug Pods
Get a Shell to a Running 

Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container'

s Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configuration 

(v1)
Reference
CustomResourceDefinition represents a resource that should be exposed on the API server.
apiVersion: apiextensions.k8s.io/v1
import "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
CustomResourceDefinition represents a resource that should be exposed on the API server. Its name MUST be in the format &lt;.spec.name&gt;.&lt;.spec.group&gt;.
: apiextensions.k8s.io/v1
kind
: CustomResourceDefinition
metadata
(
)
Standard object's metadata More info:
spec
(
CustomResourceDefinitionSpec
spec describes how the user wants the resources to appear
status
(
CustomResourceDefinitionStatus
)
status indicates the actual state of the CustomResourceDefinition
CustomResourceDefinitionSpec describes how a user wants their resource to appear
group
group is the API group of the defined custom resource. The custom resources are served under
/apis/\&lt;group&gt;/...
. Must match the name of the CustomResourceDefinition (in the form
\&lt;names.plural&gt;.\&lt;group&gt;
).
names
(CustomResourceDefinitionNames), required
names specify the resource and kind names for the custom resource.
CustomResourceDefinitionNames indicates the names to serve this CustomResourceDefinition
names.kind
kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as the
kind
attribute in API calls.
names.plural
plural is the plural name of the resource to serve. The custom resources are served under
/apis/\&lt;group&gt;/\&lt;version&gt;/.../\&lt;plural&gt;
). Must be all lowercase.
names.categories
categories is a list of grouped resources this custom resource belongs to (e.g. 'all'). This is published in API discovery documents, and used by clients to support invocations like
kubectl get all
.
names.listKind
(string)
listKind is the serialized kind of the list for this resource. Defaults to "
kind
List".
names.shortNames
shortNames are short names for the resource, exposed in API discovery documents, and used by clients to support invocations like
kubectl get \&lt;shortname&gt;
. It must be all lowercase.
names.singular
(string)
singular is the singular name of the resource. It must

 be all lowercase. Defaults to lowercased
kind
.
scope
scope indicates whether the defined custom resource is cluster- or namespace-scoped. Allowed values are
Cluster
and
.
versions
([]CustomResourceDefinitionVersion), required
versions is the list of all API versions of the defined custom resource. Version names are used to compute the order in which served versions are listed in API discovery. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10.
CustomResourceDefinitionVersion describes a version for CRD.
versions.name
name is the version name, e.g. “v1”, “v2beta1”, etc. The custom resources are served under this version at
/apis/\&lt;group&gt;/\&lt;version&gt;/...
if
served
is true.
versions.served
(boolean), required
served is a flag enabling/disabling this version from being served via REST APIs
versions.storage
storage indicates this version should be used when persisting custom resources to storage. There must be exactly one version with storage=true.
versions.additionalPrinterColumns
([]CustomResourceColumnDefinition)
additionalPrinterColumns specifies additional columns returned in Table output. See
https://kubernetes.io/docs/reference/using-api/api-concepts/#receiving-resources-as-tables
for details. If no columns are specified, a single column displaying the age of the custom resource is used.
CustomResourceColumnDefinition specifies a column for server side printing.
versions.additionalPrinterColumns.jsonPath
jsonPath is a simple JSON path (i.e. with array notation) which is evaluated against each custom resource to produce the value for this column.
versions.additionalPrinterColumns.name
name is a human readable name for the column.
versions.additionalPrinterColumns.type
type is an OpenAPI type definition for this column. See
https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types
versions.additionalPrinterColumns.description
(string)
description is a human readable description of this column.
versions.additionalPrinterColumns.format
(string)
format is an optional OpenAPI type definition for this column. The 'name' format is applied to the primary identifier column to assist in clients identifying column is the resource name. See
versions.additionalPrinterColumns.priority
(int32)
priority is an integer defining the relative importance of this column compared to others. Lower numbers are considered higher priority. Columns that may be omitted in limited space scenarios should be given a priority greater than 0.
versions.deprecated
(boolean)
deprecated indicates this version of the custom resource API is deprecated. When set to true, API requests to this version receive a warning header in the server response. Defaults to false.
versions.deprecationWarning
(string)
deprecationWarning overrides the default warning returned to API clients. May only be set when
is true. The default warning indicates this version is deprecated and recommends use of the newest served version of equal or greater stability, if one exists.
versions.schema
(CustomResourceValidation)
schema describes the schema used for validation, pruning, and defaulting of this version of the custom resource.
CustomResourceValidation is a list of validation methods f

or CustomResources.
versions.schema.openAPIV3Schema
(
JSONSchemaProps
)
openAPIV3Schema is the OpenAPI v3 schema to use for validation and pruning.
versions.subresources
(CustomResourceSubresources)
subresources specify what subresources this version of the defined custom resource have.
CustomResourceSubresources defines the status and scale subresources for CustomResources.
versions.subresources.scale
(CustomResourceSubresourceScale)
scale indicates the custom resource should serve a
/scale
subresource that returns an
autoscaling/v1
Scale object.
CustomResourceSubresourceScale defines how to serve the scale subresource for CustomResources.
versions.subresources.scale.specReplicasPath
specReplicasPath defines the JSON path inside of a custom resource that corresponds to Scale
spec.replicas
. Only JSON paths without the array notation are allowed. Must be a JSON Path under
.spec
. If there is no value under the given path in the custom resource, the
/scale
subresource will return an error on GET.
versions.subresources.scale.statusReplicasPath
statusReplicasPath defines the JSON path inside of a custom resource that corresponds to Scale
status.replicas
.status
value in the
/scale
subresource will default to 0.
versions.subresources.scale.labelSelectorPath
(string)
labelSelectorPath defines the JSON path inside of a custom resource that corresponds to Scale
status.selector
.status
or
.spec
. Must be set to work with HorizontalPodAutoscaler. The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form. More info:
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions#scale-subresource
If there is no value under the given path in the custom resource, the
/scale
subresource will default to the empty string.
versions.subresources.status
(CustomResourceSubresourceStatus)
status indicates the custom resource should serve a
/status
subresource. When enabled: 1. requests to the custom resource primary endpoint ignore changes to the
status
stanza of the object. 2. requests to the custom resource
/status
subresource ignore changes to anything other than the
status
stanza of the object.
CustomResourceSubresourceStatus defines how to serve the status subresource for CustomResources. Status is represented by the
.status
JSON path inside of a CustomResource. When set, * exposes a /status subresource for the custom resource * PUT requests to the /status subresource take a custom resource object, and ignore changes to anything except the status stanza * PUT/POST/PATCH requests to the custom resource ignore changes to the status stanza
(CustomResourceConversion)
conversion defines conversion settings for the CRD.
CustomResourceConversion describes how to convert different versions of a CR.
conversion.strategy
strategy specifies how custom resources are converted between versions. Allowed values are: -
None
: The converter only change the apiVersion and would not touch any other field in the custom resource. -
Webhook
: API Server will call to an external webhook to do the conversion. Additional information
is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhook to be set.
conversion.webhook
(WebhookConversion)
webhook describes how to call the conversion webhook. Required when
strategy
is set to
Webhook
.
WebhookConversion describes how to call a conversion webhook
conversion.webhook.conversionReviewVersions
conversionReviewVersions is an ordered list of preferred
versions the Web

hook expects. The API server will use the first version in the list which it supports. If none of the versions specified in this list are supported by API server, conversion will fail for the custom resource. If a persisted Webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail.
conversion.webhook.clientConfig
(WebhookClientConfig)
clientConfig is the instructions for how to call the webhook if strategy is
Webhook
.
WebhookClientConfig contains the information to make a TLS connection with the webhook.
conversion.webhook.clientConfig.caBundle
([]byte)
caBundle is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used.
conversion.webhook.clientConfig.service
service is a reference to the service for this webhook. Either service or url must be specified.
service
.
conversion.webhook.clientConfig.service.name
name is the name of the service. Required
conversion.webhook.clientConfig.service.namespace
namespace is the namespace of the service. Required
conversion.webhook.clientConfig.service.path
(string)
path is an optional URL path at which the webhook will be contacted.
conversion.webhook.clientConfig.service.port
(int32)
port is an optional service port at which the webhook will be contacted.
port
should be a valid port number (1-65535, inclusive). Defaults to 443 for backward compatibility.
conversion.webhook.clientConfig.url
(string)
url gives the location of the webhook, in standard URL form (
). Exactly one of
url
or
service
must be specified.
The
host
should not refer to a service running in the cluster; use the
service
field instead. The host might be resolved via external DNS in some apiservers (e.g.,
cannot resolve in-cluster DNS as that would be a layering violation).
host
localhost
or
127.0.0.1
as a
host
is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster.
A path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier.
Attempting to use a user or basic auth e.g. "user:password@" is not allowed. Fragments ("#...") and query parameters ("?...") are not allowed, either.
(boolean)
preserveUnknownFields indicates that object fields which are not specified in the OpenAPI schema should be preserved when persisting to storage. apiVersion, kind, metadata and known fields inside metadata are always preserved. This field is deprecated in favor of setting
x-preserve-unknown-fields
to true in
spec.versions[*].schema.openAPIV3Schema
. See
https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#field-pruning
JSONSchemaProps is a JSON-Schema following Specification Draft 4 (
http://json-schema.org/)
.
$ref
(string)
$schema
(string)
additionalItems
(JSONSchemaPropsOrBool)
JSONSchemaPropsOrBool represents JSONSchemaProps or a boolean value. Defaults to true for the boolean property.
additionalProperties
allOf
([]
)
anyOf
([]
)
default
(JSON)
default is a default value for undefined object fields. Defaulting is a beta feature under the CustomResourceDefaulting feature gate. Defaulting requires s

pec.preserveUnknownFields to be false.
JSON represents any valid JSON value. These types are supported: bool, int64, float64, string, []interface{}, map[string]interface{} and nil.
definitions
(map[string]
)
dependencies
(map[string]JSONSchemaPropsOrStringArray)
JSONSchemaPropsOrStringArray represents a JSONSchemaProps or a string array.
(string)
enum
([]JSON)
example
(JSON)
exclusiveMaximum
(boolean)
exclusiveMinimum
(boolean)
externalDocs
(ExternalDocumentation)
ExternalDocumentation allows referencing an external resource for extended documentation.
externalDocs.description
(string)
externalDocs.url
(string)
format
(string)
format is an OpenAPI v3 format string. Unknown formats are ignored. The following formats are validated:
bsonobjectid: a bson object ID, i.e. a 24 characters hex string - uri: an URI as parsed by Golang net/url.ParseRequestURI - email: an email address as parsed by Golang net/mail.ParseAddress - hostname: a valid representation for an Internet host name, as defined by RFC 1034, section 3.1 [RFC1034]. - ipv4: an IPv4 IP as parsed by Golang net.ParseIP - ipv6: an IPv6 IP as parsed by Golang net.ParseIP - cidr: a CIDR as parsed by Golang net.ParseCIDR - mac: a MAC address as parsed by Golang net.ParseMAC - uuid: an UUID that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?[0-9a-f]{4}-?[0-9a-f]{4}-?[0-9a-f]{12}$ - uuid3: an UUID3 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?3[0-9a-f]{3}-?[0-9a-f]{4}-?[0-9a-f]{12}$ - uuid4: an UUID4 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?4[0-9a-f]{3}-?[89ab][0-9a-f]{3}-?[0-9a-f]{12}$ - uuid5: an UUID5 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?5[0-9a-f]{3}-?[89ab][0-9a-f]{3}-?[0-9a-f]{12}$ - isbn: an ISBN10 or ISBN13 number string like "0321751043" or "978-0321751041" - isbn10: an ISBN10 number string like "0321751043" - isbn13: an ISBN13 number string like "978-0321751041" - creditcard: a credit card number defined by the regex ^(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35\d{3})\d{11})$ with any non digit characters mixed in - ssn: a U.S. social security number following the regex ^\d{3}[- ]?\d{2}[- ]?\d{4}$ - hexcolor: an hexadecimal color code like "#FFFFFF: following the regex ^#?([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$ - rgbcolor: an RGB color code like rgb like "rgb(255,255,2559" - byte: base64 encoded binary data - password: any kind of string - date: a date string like "2006-01-02" as defined by full-date in RFC3339 - duration: a duration string like "22 ns" as parsed by Golang time.ParseDuration or compatible with Scala duration format - datetime: a date time string like "2014-12-15T19:30:20.000Z" as defined by date-time in RFC3339.
id
(string)
items
(JSONSchemaPropsOrArray)
JSONSchemaPropsOrArray represents a value that can either be a JSONSchemaProps or an array of JSONSchemaProps. Mainly here for serialization purposes.
maxItems
(int64)
maxLength
(int64)
maxProperties
(int64)
maximum
(double)
minItems
(int64)
minLength
(int64)
minProperties
(int64)
minimum
(double)
multipleOf
(double)
not
(
)
nullable
(boolean)
oneOf
([]
)
pattern
(string)
patternProperties
)
)
required
title
(string)
type
(string)
uniqueItems
(boolean)
x-kubernetes-embedded-resource
(boolean)
x-kubernetes-embedded-resource defines that the value is an embedded Kubernetes runtime.Object, with TypeMeta and ObjectMeta. The type must be object. It is allowed to further restrict the embedded

 object. kind, apiVersion and metadata are validated automatically. x-kubernetes-preserve-unknown-fields is allowed to be true, but does not have to be if the object is fully specified (up to kind, apiVersion, metadata).
x-kubernetes-int-or-string
(boolean)
x-kubernetes-int-or-string specifies that this value is either an integer or a string. If this is true, an empty type is allowed and type as child of anyOf is permitted if following one of the following patterns:
anyOf:
type: integer
type: string
allOf:
anyOf:
... zero or more
x-kubernetes-list-map-keys
x-kubernetes-list-map-keys annotates an array with the x-kubernetes-list-type
map
by specifying the keys used as the index of the map.
This tag MUST only be used on lists that have the "x-kubernetes-list-type" extension set to "map". Also, the values specified for this attribute must be a scalar typed field of the child structure (no nesting is supported).
The properties specified must either be required or have a default value, to ensure those properties are present for all list items.
x-kubernetes-list-type
(string)
x-kubernetes-list-type annotates an array to further describe its topology. This extension must only be used on lists and may have 3 possible values:
atomic
: the list is treated as a single entity, like a scalar.
Atomic lists will be entirely replaced when updated. This extension
may be used on any type of list (struct, scalar, ...).
set
:
Sets are lists that must not have multiple items with the same value. Each
value must be a scalar, an object with x-kubernetes-map-type
atomic
or an
array with x-kubernetes-list-type
atomic
.
map
:
These lists are like maps in that their elements have a non-index key
used to identify them. Order is preserved upon merge. The map tag
must only be used on a list with elements of type object.
Defaults to atomic for arrays.
x-kubernetes-map-type
(string)
x-kubernetes-map-type annotates an object to further describe its topology. This extension must only be used when type is object and may have 2 possible values:
granular
:
These maps are actual maps (key-value pairs) and each fields are independent
from each other (they can each be manipulated by separate actors). This is
the default behaviour for all maps.
atomic
Atomic maps will be entirely replaced when updated.
x-kubernetes-preserve-unknown-fields
(boolean)
x-kubernetes-preserve-unknown-fields stops the API server decoding step from pruning fields which are not specified in the validation schema. This affects fields recursively, but switches back to normal pruning behaviour if nested properties or additionalProperties are specified in the schema. This can either be true or undefined. False is forbidden.
x-kubernetes-validations
([]ValidationRule)
rule
Map: unique values on key rule will be kept during a merge
x-kubernetes-validations describes a list of validation rules written in the CEL expression language. This field is an alpha-level. Using this field requires the feature gate
CustomResourceValidationExpressions
ValidationRule describes a validation rule written in the CEL expression language.
x-kubernetes-validations.rule
Rule represents the expression which will be evaluated by CEL. ref:
https://github.com/google/cel-spec
The Rule is scoped to the location of the x-kubernetes-validations extension in the schema. The
self
variable in the CEL expression is bound to the scoped value. Example: - Rule scoped to the root of a resource with a status subresource: {"rule": "self.status.actual &lt;= self.spec.maxDesired"}
If the Rule is scoped to an object with properties, the accessible properties of the object are field selectable via
self.field
and field presence can be checked via
has(self.field)
. Null valued fields are treated as absent fields in CEL expressions. If the Rule is scoped to an object with additionalProper

ties (i.e. a map) the value of the map are accessible via
self[mapKey]
, map containment can be checked via
mapKey in self
and all entries of the map are accessible via CEL macros and functions such as
self.all(...)
. If the Rule is scoped to an array, the elements of the array are accessible via
self[i]
and also by macros and functions. If the Rule is scoped to a scalar,
self
is bound to the scalar value. Examples: - Rule scoped to a map of objects: {"rule": "self.components['Widget'].priority &lt; 10"} - Rule scoped to a list of integers: {"rule": "self.values.all(value, value &gt;= 0 &amp;&amp; value &lt; 100)"} - Rule scoped to a string value: {"rule": "self.startsWith('kube')"}
The
,
kind
,
and
metadata.generateName
are always accessible from the root of the object and from any x-kubernetes-embedded-resource annotated objects. No other metadata properties are accessible.
Unknown data preserved in custom resources via x-kubernetes-preserve-unknown-fields is not accessible in CEL expressions. This includes: - Unknown field values that are preserved by object schemas with x-kubernetes-preserve-unknown-fields. - Object properties where the property schema is of an "unknown type". An "unknown type" is recursively defined as:
A schema with no type and x-kubernetes-preserve-unknown-fields set to true
An array where the items schema is of an "unknown type"
An object where the additionalProperties schema is of an "unknown type"
Only property names of the form
[a-zA-Z_.-/][a-zA-Z0-9_.-/]*
are accessible. Accessible property names are escaped according to the following rules when accessed in the expression: - '
' escapes to '
underscores
' - '.' escapes to '
dot
' - '-' escapes to '
dash
' - '/' escapes to '
slash
' - Property names that exactly match a CEL RESERVED keyword escape to '
{keyword}__'. The keywords are:
"true", "false", "null", "in", "as", "break", "const", "continue", "else", "for", "function", "if",
"import", "let", "loop", "package", "namespace", "return".
Examples:
Rule accessing a property named "namespace": {"rule": "self.
namespace
&gt; 0"}
Rule accessing a property named "x-prop": {"rule": "self.x__dash__prop &gt; 0"}
Rule accessing a property named "redact__d": {"rule": "self.redact__underscores__d &gt; 0"}
Equality on arrays with x-kubernetes-list-type of 'set' or 'map' ignores element order, i.e. [1, 2] == [2, 1]. Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:
'set':
X + Y
performs a union where the array positions of all elements in
X
are preserved and
non-intersecting elements in
Y
are appended, retaining their partial order.
'map':
X + Y
performs a merge where the array positions of all keys in
X
are preserved but the values
are overwritten by values in
Y
when the key sets of
X
and
Y
intersect. Elements in
Y
with
non-intersecting keys are appended, retaining their partial order.
x-kubernetes-validations.message
(string)
Message represents the message displayed when validation fails. The message is required if the Rule contains line breaks. The message must not contain line breaks. If unset, the message is "failed rule: {Rule}". e.g. "must be a URL with the host matching spec.host"
CustomResourceDefinitionStatus indicates the state of the CustomResourceDefinition
acceptedNames
(CustomResourceDefinitionNames)
acceptedNames are the names that are actually being used to serve discovery. They may be different than the names in spec.
acceptedNames.kind
kind
acceptedNames.plural
. Must mat

ch the name of the CustomResourceDefinition (in the form
acceptedNames.categories
.
acceptedNames.listKind
(string)
kind
List".
acceptedNames.shortNames
acceptedNames.singular
(string)
singular is the singular name of the resource. It must be all lowercase. Defaults to lowercased
kind
.
([]CustomResourceDefinitionCondition)
conditions indicate state for particular aspects of a CustomResourceDefinition
CustomResourceDefinitionCondition contains details for the current condition of this pod.
status is the status of the condition. Can be True, False, Unknown.
type is the type of the condition. Types include Established, NamesAccepted and Terminating.
(Time)
lastTransitionTime last time the condition transitioned from one status to another.
(string)
message is a human-readable message indicating details about last transition.
(string)
reason is a unique, one-word, CamelCase reason for the condition's last transition.
storedVersions lists all versions of CustomResources that were ever persisted. Tracking these versions allows a migration path for stored versions in etcd. The field is mutable so a migration controller can finish a migration to another version (ensuring no old objects are left in storage), and then remove the rest of the versions from this list. Versions may not be removed from
while they exist in this list.
CustomResourceDefinitionList
CustomResourceDefinitionList is a list of CustomResourceDefinition objects.
items
([]
items list individual CustomResourceDefinition objects
(string)
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
(string)
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info:
metadata
(
ListMeta
)
get
read the specified CustomResourceDefinition
GET /apis/apiextensions.k8s.io/v1/customresourcedefinitions/{name}
name
(
in path
name of the CustomResourceDefinition
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified CustomResourceDefinition
GET /apis/apiextensions.k8s.io/v1/customresourcedefinitions/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind CustomResourceDefinition
HTTP Requ

est
GET /apis/apiextensions.k8s.io/v1/customresourcedefinitions
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a CustomResourceDefinition
POST /apis/apiextensions.k8s.io/v1/customresourcedefinitions
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified CustomResourceDefinition
PUT /apis/apiextensions.k8s.io/v1/customresourcedefinitions/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified CustomResourceDefinition
PUT /apis/apiextensions.k8s.io/v1/customresourcedefinitions/{name}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified CustomResourceDefinition
PATCH /apis/apiextensions.k8s.io/v1/customresourcedefinitions/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified CustomResourceDefinition
PATCH /apis/apiextensions.k8s.io/v1/customresourcedefinitions/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a CustomResourceDefinition
DELETE /apis/apiextensions.k8s.io/v1/customresourcedefinitions/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of CustomResourceDefinition
DELETE /apis/apiextensions.k8s.io/v1/customresourcedefinitions
body
:
continue
(
in 

query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Last modified September 20, 2022 at 4:22 PM PST:
Fix the link failure in the English custom-resource-definition-v1.md (caa9d018bd)
get
get
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm alpha | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set

 Up kubectl on macOS
Debug Pods
Get a Shell to a Running

 Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container

's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configuration

 (v1)
Reference
Kubeadm
Caution:
.
or
Feedback
Yes
No
.
or
.
Last modified January 04, 2021 at 3:39 PM PST:
deprecate kubeadm alpha kubeconfig in 1.21 (e4ac283321)
Home
Blog
Training
Partners
Community
CC BY 4.0

Interactive Tutorial - Creating a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extending Kub

ernetes
Tasks
Migrate from PodSecurityPolicy to the Buil

t-In PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProf

ile, ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Eve

nt Rate Limit Configuration (v1alpha1)
Tutorials
The screen is too narrow to interact with the Terminal, please use a desktop/tablet.
Home
Continue to Module 2 &gt;
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Managing Kubernetes Objects Using Imperative Commands | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
O

perator pattern
Tasks
Debug Pods
Determine the R

eason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the

 Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configurati

on (v1)
Tasks
Kubernetes objects can quickly be created, updated, and deleted directly using
imperative commands built into the
kubectl
command-line tool. This document
explains how those commands are organized and how to use them to manage live objects.
Install
kubectl
.
minikube
.
Trade-offs
The
kubectl
tool supports three kinds of object management:
Imperative commands
Imperative object configuration
Declarative object configuration
See
for a discussion of the advantages and disadvantage of each kind of object management.
How to create objects
The
kubectl
tool supports verb-driven commands for creating some of the most common
object types. The commands are named to be recognizable to users unfamiliar with
the Kubernetes object types.
run
: Create a new Pod to run a Container.
expose
: Create a new Service object to load balance traffic across Pods.
autoscale
: Create a new Autoscaler object to automatically horizontally scale a controller, such as a Deployment.
The
kubectl
tool also supports creation commands driven by object type.
These commands support more object types and are more explicit about
their intent, but require users to know the type of objects they intend
to create.
create &lt;objecttype&gt; [&lt;subtype&gt;] &lt;instancename&gt;
Some objects types have subtypes that you can specify in the
create
command.
For example, the Service object has several subtypes including ClusterIP,
LoadBalancer, and NodePort. Here's an example that creates a Service with
subtype NodePort:
kubectl create service nodeport &lt;myservicename&gt;
In the preceding example, the
create service nodeport
command is called
a subcommand of the
create service
command.
-h
flag to find the arguments and flags supported by
a subcommand:
kubectl create service nodeport -h
How to update objects
The
kubectl
command supports verb-driven commands for some common update operations.
These

 commands are named to enable users unfamiliar with Kubernetes
objects to perform updates without knowing the specific fields
that must be set:
scale
: Horizontally scale a controller to add or remove Pods by updating the replica count of the controller.
annotate
: Add or remove an annotation from an object.
label
: Add or remove a label from an object.
The
kubectl
command also supports update commands driven by an aspect of the object.
Setting this aspect may set different fields for different object types:
set
&lt;field&gt;
: Set an aspect of an object.
Note:
In Kubernetes version 1.5, not every verb-driven command has an associated aspect-driven command.
The
kubectl
tool supports these additional ways to update a live object directly,
however they require a better understanding of the Kubernetes object schema.
edit
: Directly edit the raw configuration of a live object by opening its configuration in an editor.
patch
: Directly modify specific fields of a live object by using a patch string.
For more details on patch strings, see the patch section in
API Conventions
.
How to delete objects
delete
command to delete an object from a cluster:
delete &lt;type&gt;/&lt;name&gt;
Note:
You can use
for both imperative commands and imperative object
configuration. The difference is in the arguments passed to the command. To use
as an imperative command, pass the object to be deleted as
an argument. Here's an example that passes a Deployment object named nginx:
kubectl delete deployment/nginx
How to view an object
There are several commands for printing information about an object:
get
: Prints basic information about matching objects. Use
get -h
to see a list of options.
describe
: Prints aggregated detailed information about matching objects.
logs
: Prints the stdout and stderr for a container running in a Pod.
Using
set
commands to modify objects before creation
There are some object fields that don't have a flag you can use
in a
create
command. In some of those cases, you can use a combination of
set
and
create
to specify a value for the field before object
creation. This is done by piping the output of the
create
command to the
set
command, and then back to the
create
command. Here's an example:
kubectl create service clusterip my-svc --clusterip
=
"None"
-o yaml --dry-run
=
client | kubectl
set
selector --local -f -
'environment=qa'
-o yaml | kubectl create -f -
The
kubectl create service -o yaml --dry-run=client
command creates the configuration for the Service, but prints it to stdout as YAML instead of sending it to the Kubernetes API server.
The
kubectl set selector --local -f - -o yaml
command reads the configuration from stdin, and writes the updated configuration to stdout as YAML.
The
kubectl create -f -
command creates the object using the configuration provided via stdin.
Using
--edit
to modify objects before creation
You can use
kubectl create --edit
to make arbitrary changes to an object
before it is created. Here's an example:
=
"None"
=
client &gt; /tmp/srv.yaml
kubectl create --edit -f /tmp/srv.yaml
The
kubectl create service
command creates the configuration for the Service and saves it to
/tmp/srv.yaml
.
The
command opens the configuration file for editing before it creates the object.
Kubectl Command Reference
Feedback
Yes
No
.
or
.
Last modified May 19, 2022 at 5:00 PM PST:
Updated references in different pages with the correct titles of the documents (f559518520)
Edit t

his page
Using
set
Using
--edit
Home
Blog
Training
Partners
Community
CC BY 4.0

Example: Configuring a Java Microservice | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Externalizing config using MicroProfile, ConfigMaps and Secrets
In this tutorial you will learn how and why to externalize your microservice’s configuration.
Specifically, you will learn how to use Kubernetes ConfigMaps and Secrets to set environment
variables and then consume them using MicroProfile Config.
Creating Kubernetes ConfigMaps &amp; Secrets
There are several ways to set environment variables for a Docker container in Kubernetes,
including: Dockerfile, kubernetes.yml, Kubernetes ConfigMaps, and Kubernetes Secrets. In the
tutorial, you will learn how to use the latter two for setting your environment variables whose
values will be injected into your microservices. One of the benefits for using ConfigMaps and
Secrets is that they can be re-used across multiple containers, including being assigned to
different environment variables for the different containers.
ConfigMaps are API Objects that store non-confidential key-value pairs. In the Interactive
Tutorial you will learn how to use a ConfigMap to store the application's name. For more
information regarding ConfigMaps, you can find the documentation
here
.
Although Secrets are also used to store key-value pairs, they differ from ConfigMaps in that
they're intended for confidential/sensitive information and are stored using Base64 encoding.
This makes secrets the appropriate choice for storing such things as credentials, keys, and
tokens, the former of which you'll do in the Interactive Tutorial. For more information on
Secrets, you can find the documentation
here
.
Externalizing Config from Code
Externalized application configuration is useful because configuration usually changes depending
on your environment. In order to accomplish this, we'll use Java's Contexts and Dependency
Injection (CDI) and MicroProfile Config. MicroProfile Config is a feature of MicroProfile, a set
of open Java technologies for developing and deploying cloud-native microservices.
CDI provides a standard dependency injection capability enabling an application to be assembled
from collaborating, loosely-coupled beans. MicroProfile Config provides apps and microservices a
standard way to obtain config properties from various sources, including the application, runtime,
and environment. Based on the source's defined priority, the properties are automatically
combined into a single set of properties that the application can access via an API. Together,
CDI &amp; MicroProfile will be used in the Interactive Tutorial to retrieve the externally provided
properties from the Kubernetes ConfigMaps and Secrets and get injected into your application code.
Many open source frameworks and runtimes implement and support MicroProfile Config. Throughout
the interactive tutorial, you'll be using Open Liberty, a flexible open-source Java runtime for
building and running cloud-native apps and microservices. However, any MicroProfile compatible
runtime could be used instead.
Create a Kubernetes ConfigMap and Secret
Inject microservice configuration using MicroProfile Config
Example: Externalizing config using MicroProfile, ConfigMaps and Secrets
2 - Interactive Tutorial - Configuring a Java Microservice
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundatio

n ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Kubernetes Object Management | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
T

asks
Debug Pods
Determine the Reason for Pod Fai

lure
Auditing
Run Jobs
TLS
Tutorials
Security
Ap

ply Pod Security Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apis

erver Configuration (v1alpha1)
Concepts
Overview
The
kubectl
command-line tool supports several different ways to create and manage
Kubernetes objects. This document provides an overview of the different
approaches. Read the
Kubectl book
for
details of managing objects by Kubectl.
Management techniques
Warning:
A Kubernetes object should be managed using only one technique. Mixing
and matching techniques for the same object results in undefined behavior.
Management technique
Operates on
Recommended environment
Supported writers
Learning curve
Live objects
Development projects
1+
Lowest
Individual files
Production projects
1
Moderate
Directories of files
1+
Highest
When using imperative commands, a user operates directly on live objects
in a cluster. The user provides operations to
the
kubectl
command as arguments or flags.
This is the recommended way to get started or to run a one-off task in
a cluster. Because this technique operates directly on live
objects, it provides no history of previous configurations.
Examples
Run an instance of the nginx container by creating a Deployment object:
kubectl create deployment nginx --image nginx
Advantages compared to object configuration:
Commands are expressed as a single action word.
Commands require only a single step to make changes to the cluster.
Disadvantages compared to object configuration:
Commands do not integrate with change review processes.
Commands do not provide an audit trail associated with changes.
Commands do not provide a source of records except for what is live.
Commands do not provide a template for creating new objects.
In imperative object configuration, the kubectl command specifies the
operation (create, replace, etc.), optional flags and at least one file
name. The file specified must contain a full definition of the object
in YAML or JSON format.
See the
for more details on object definitions.
Warning:
The imperative
replace
command replaces the existing
spec with the newly provided one, dropping all changes to the object missing from
the configuration file. This approach should not be used with resource
types whose specs are updated independently of the configuration file.
Services of type
, for example, have their
externalIPs
field updated
independently from the configuratio

n by the cluster.
Examples
Create the objects defined in a configuration file:
kubectl create -f nginx.yaml
Delete the objects defined in two configuration files:
kubectl delete -f nginx.yaml -f redis.yaml
Update the objects defined in a configuration file by overwriting
the live configuration:
kubectl replace -f nginx.yaml
Advantages compared to imperative commands:
Object configuration can be stored in a source control system such as Git.
Object configuration can integrate with processes such as reviewing changes before push and audit trails.
Object configuration provides a template for creating new objects.
Disadvantages compared to imperative commands:
Object configuration requires basic understanding of the object schema.
Object configuration requires the additional step of writing a YAML file.
Advantages compared to declarative object configuration:
Imperative object configuration behavior is simpler and easier to understand.
As of Kubernetes version 1.5, imperative object configuration is more mature.
Disadvantages compared to declarative object configuration:
Imperative object configuration works best on files, not directories.
Updates to live objects must be reflected in configuration files, or they will be lost during the next replacement.
When using declarative object configuration, a user operates on object
configuration files stored locally, however the user does not define the
operations to be taken on the files. Create, update, and delete operations
are automatically detected per-object by
kubectl
. This enables working on
directories, where different operations might be needed for different objects.
Note:
Declarative object configuration retains changes made by other
writers, even if the changes are not merged back to the object configuration file.
This is possible by using the
patch
API operation to write only
observed differences, instead of using the
replace
API operation to replace the entire object configuration.
Examples
Process all object configuration files in the
configs
directory, and create or
patch the live objects. You can first
diff
to see what changes are going to be
made, and then apply:
kubectl diff -f configs/
kubectl apply -f configs/
Recursively process directories:
kubectl diff -R -f configs/
kubectl apply -R -f configs/
Advantages compared to imperative object configuration:
Changes made directly to live objects are retained, even if they are not merged back into the configuration files.
Declarative object configuration has better support for operating on directories and automatically detecting operation types (create, patch, delete) per-object.
Disadvantages compared to imperative object configuration:
Declarative object configuration is harder to debug and understand results when they are unexpected.
Partial updates using diffs create complex merge and patch operations.
Feedback
Yes
No
.
or
.
Last modified October 24, 2022 at 11:52 AM PST:
update page weights for concepts overview working with objects (6bfe72e2b0)
Examples
Examples
Examples
Home
Blog
Training
Partners
Community
CC BY 4

.0

Change the default StorageClass | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
minikube
.
standard
(
default
)
(default)
.
set to
true
false
.
false
:
where
standard
.
kubectl p

atch storageclass gold -p
without
gold
(
default
)
.
Feedback
Yes
No
.
or
.
Last modified February 04, 2021 at 4:41 PM PST:
clean up use of word: simply (3fd65482e8)
Home
Blog
Training
Partners
Community
CC BY 4.0

Developing Cloud Controller Manager | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
component
Out of tree
.
Use
main.go
main.go
main.go
init
.
In tree
DaemonSet
Feedback
Yes
No
.
or
.
Last 

modified January 02, 2021 at 7:16 PM PST:
Cloud Controller Manager: fix a broken link (89578d9146)
Out of tree
In tree
Home
Blog
Training
Partners
Community
CC BY 4.0

Extend kubectl with plugins | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on Lin

ux
Debug Pods
Get a 

Shell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Re

strict a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryp

tion Configuration (v1)
Tasks
Extend kubectl by creating and installing kubectl plugins.
This guide demonstrates how to install and write extensions for
kubectl
. By thinking of core
kubectl
commands as essential building blocks for interacting with a Kubernetes cluster, a cluster administrator can think
of plugins as a means of utilizing these building blocks to create more complex behavior. Plugins extend
kubectl
with new sub-commands, allowing for new and custom features not included in the main distribution of
kubectl
.
You need to have a working
kubectl
binary installed.
Installing kubectl plugins
A plugin is a standalone executable file, whose name begins with
kubectl-
. To install a plugin, move its executable file to anywhere on your
PATH
.
You can also discover and install kubectl plugins available in the open source
using
Krew
. Krew is a plugin manager maintained by
the Kubernetes SIG CLI community.
Caution:
Kubectl plugins available via the Krew
plugin index
are not audited for security. You should install and run third-party plugins at your
own risk, since they are arbitrary programs running on your machine.
Discovering plugins
kubectl
provides a command
that searches your
PATH
for valid plugin executables.
Executing this command causes a traversal of all files in your
PATH
. Any files that are executable, and begin with
kubectl-
will show up
in the order in which they are present in your
PATH
in this command's output.
A warning will be included for any files beginning with
kubectl-
that are
not
executable.
A warning will also be included for any valid plugin files that overlap each other's name.
You can use
Krew
to discover and install
kubectl
plugins from a community-curated
.
It is currently not possible to create plugins that overwrite existing
kubectl
commands. For example, creating a plugin
kubectl-version
will cause that plugin to never be executed, as the existing
command will always take precedence over it. Due to this limitation, it is also
not
possible to use plugins to add new subcommands to existing
kubectl
commands. For example, adding a subcommand
kubectl create foo
by naming your plugin
kubectl-create-foo
will cause that plugin to be ignored.
shows warnings for any valid plugins that attempt to do this.
Writing kubectl plugins
You can write a plugin in any programming language or script that allows you to write command-line commands.
There is no plugin installation or pre-loading required. 

Plugin executables receive
the inherited environment from the
kubectl
binary.
A plugin determines which command path it wishes to implement based on its name.
For example, a plugin named
kubectl-foo
kubectl foo
. You must
install the plugin executable somewhere in your
PATH
.
Example plugin
# optional argument handling
if
[[
"
$1
"
==
"version"
]]
then
echo
"1.0.0"
exit
0
fi
if
[[
"
$1
"
==
"config"
]]
then
echo
"
$KUBECONFIG
"
exit
0
fi
echo
"I am a plugin named kubectl-foo"
Using a plugin
To use a plugin, make the plugin executable:
sudo chmod +x ./kubectl-foo
and place it anywhere in your
PATH
:
sudo mv ./kubectl-foo /usr/local/bin
You may now invoke your plugin as a
kubectl
command:
I am a plugin named kubectl-foo
All args and flags are passed as-is to the executable:
kubectl foo version
1.0.0
All environment variables are also passed as-is to the executable:
export
=
~/.kube/config
kubectl foo config
/home/&lt;user&gt;/.kube/config
=
/etc/kube/config kubectl foo config
/etc/kube/config
Additionally, the first argument that is passed to a plugin will always be the full path to the location where it was invoked (
$0
would equal
in the example above).
Naming a plugin
As seen in the example above, a plugin determines the command path that it will implement based on its filename. Every sub-command in the command path that a plugin targets, is separated by a dash (
-
).
For example, a plugin that wishes to be invoked whenever the command
kubectl foo bar baz
is invoked by the user, would have the filename of
kubectl-foo-bar-baz
.
Flags and argument handling
Note:
The plugin mechanism does
not
create any custom, plugin-specific values or environment variables for a plugin process.
An older kubectl plugin mechanism provided environment variables such as
KUBECTL_PLUGINS_CURRENT_NAMESPACE
; that no longer happens.
kubectl plugins must parse and validate all of the arguments passed to them.
See
using the command line runtime package
for details of a Go library aimed at plugin authors.
Here are some additional cases where users invoke your plugin while providing additional flags and arguments. This builds upon the
plugin from the scenario above.
If you run
kubectl foo bar baz arg1 --flag=value arg2
, kubectl's plugin mechanism will first try to find the plugin with the longest possible name, which in this case
would be
kubectl-foo-bar-baz-arg1
. Upon not finding that plugin, kubectl then treats the last dash-separated value as an argument (
arg1
in this case), and attempts to find the next longest possible name,
.
Upon having found a plugin with this name, kubectl then invokes that plugin, passing all args and flags after the plugin's name as arguments to the plugin process.
Example:
# create a plugin
echo
-e
'#!/bin/bash\n\necho "My first command-line argument was $1"'
&gt; kubectl-foo-bar-baz
sudo chmod +x ./kubectl-foo-bar-baz
# "install" your plugin by moving it to a directory in your $PATH
sudo mv ./kubectl-foo-bar-baz /usr/local/bin
# check that kubectl recognizes your plugin

/usr/local/bin/kubectl-foo-bar-baz
# test that calling your plugin via a "kubectl" command works
# even when additional arguments and flags are passed to your
# plugin executable by the user.
kubectl foo bar baz arg1 --meaningless-flag=true
My first command-line argument was arg1
As you can see, your plugin was found based on the
kubectl
command specified by a user, and all extra arguments and flags were passed as-is to the plugin executable once it was found.
Names with dashes and underscores
Although the
kubectl
plugin mechanism uses the dash (
-
) in plugin filenames to separate the sequence of sub-commands processed by the plugin, it is still possible to create a plugin
command containing dashes in its commandline invoc

ation by using underscores (
_
) in its filename.
Example:
# create a plugin containing an underscore in its filename
echo
-e
'#!/bin/bash\n\necho "I am a plugin with a dash in my name"'
&gt; ./kubectl-foo_bar
sudo chmod +x ./kubectl-foo_bar
# move the plugin into your $PATH
sudo mv ./kubectl-foo_bar /usr/local/bin
# You can now invoke your plugin via kubectl:
kubectl foo-bar
I am a plugin with a dash in my name
Note that the introduction of underscores to a plugin filename does not prevent you from having commands such as
kubectl foo_bar
.
The command from the above example, can be invoked using either a dash (
-
) or an underscore (
_
):
# You can invoke your custom command with a dash
# You can also invoke your custom command with an underscore
Name conflicts and overshadowing
It is possible to have multiple plugins with the same filename in different locations throughout your
PATH
.
For example, given a
PATH
with the following value:
PATH=/usr/local/bin/plugins:/usr/local/bin/moreplugins
, a copy of plugin
could exist in
/usr/local/bin/plugins
and
/usr/local/bin/moreplugins
,
such that the output of the
command is:
PATH
=
/usr/local/bin/plugins:/usr/local/bin/moreplugins kubectl plugin list

/usr/local/bin/plugins/kubectl-foo
/usr/local/bin/moreplugins/kubectl-foo
  - warning: /usr/local/bin/moreplugins/kubectl-foo is overshadowed by a similarly named plugin: /usr/local/bin/plugins/kubectl-foo

In the above scenario, the warning under
tells you that this plugin will never be executed. Instead, the executable that appears first in your
PATH
,
, will always be found and executed first by the
kubectl
plugin mechanism.
A way to resolve this issue is to ensure that the location of the plugin that you wish to use with
kubectl
always comes first in your
PATH
. For example, if you want to always use
anytime that the
kubectl
command
was invoked, change the value of your
PATH
to be
/usr/local/bin/moreplugins:/usr/local/bin/plugins
.
Invocation of the longest executable filename
There is another kind of overshadowing that can occur with plugin filenames. Given two plugins present in a user's
PATH
:
kubectl-foo-bar
and
, the
kubectl
plugin mechanism will always choose the longest possible plugin name for a given user command. Some examples below, clarify this further:
# for a given kubectl command, the plugin with the longest possible filename will always be preferred
Plugin kubectl-foo-bar-baz is executed
kubectl foo bar
Plugin kubectl-foo-bar is executed
kubectl foo bar baz buz
Plugin kubectl-foo-bar-baz is executed, with "buz" as its first argument
kubectl foo bar buz
Plugin kubectl-foo-bar is executed, with "buz" as its first argument
This design choice ensures that plugin sub-commands can be implemented across multiple files, if needed, and that these sub-commands can be nested under a "parent" plugin command:
ls ./plugin_command_tree
kubectl-parent
kubectl-parent-subcommand
kubectl-parent-subcommand-subsubcommand
Checking for plugin warnings
You can use the aforementioned
command to ensure that your plugin is visible by
kubectl
, and verify that there are no warnings preventing it from being called as a
kubectl
command.

test/fixtures/pkg/kubectl/plugins/kubectl-foo
  - warning: /usr/local/bin/kubectl-foo is overshadowed by a similarly named plugin: test/fixtures/pkg/kubectl/plugins/kubectl-foo
plugins/kubectl-invalid
  - warning: plugins/kubectl-invalid identified as a kubectl plugin, but it is not executable

error: 2 

plugin warnings were found
Using the command line runtime package
If you're writing a plugin for kubectl and you're using Go, you can make use
of the
cli-runtime
utility libraries.
These libraries provide helpers for parsing or updating a user's
file, for making REST-style requests to the API server, or to bind flags
associated with configuration and printing.
See the
Sample CLI Plugin
for
an example usage of the tools provided in the CLI Runtime repo.
Distributing kubectl plugins
If you have developed a plugin for others to use, you should consider how you
package it, distribute it and deliver updates to your users.
Krew
Krew
offers a cross-platform way to package and
distribute your plugins. This way, you use a single packaging format for all
target platforms (Linux, Windows, macOS etc) and deliver updates to your users.
Krew also maintains a
plugin
index
so that other people can
discover your plugin and install it.
Native / platform specific package management
Alternatively, you can use traditional package managers such as,
apt
or
yum
on Linux, Chocolatey on Windows, and Homebrew on macOS. Any package
manager will be suitable if it can place new executables placed somewhere
in the user's
PATH
.
As a plugin author, if you pick this option then you also have the burden
of updating your kubectl plugin's distribution package across multiple
platforms for each release.
Source code
You can publish the source code; for example, as a Git repository. If you
choose this option, someone who wants to use that plugin must fetch the code,
set up a build environment (if it needs compiling), and deploy the plugin.
If you also make compiled packages available, or use Krew, that will make
installs easier.
Check the Sample CLI Plugin repository for a
detailed example
of a
plugin written in Go.
In case of any questions, feel free to reach out to the
SIG CLI team
.
Read about
Krew
, a package manager for kubectl plugins.
Feedback
Yes
No
.
or
.
Krew
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Debug Pods
Troublesh

ooting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resourc

es with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Con

figuration (v1alpha1)
Tutorials
Security
Security
Feedback
Yes
No
.
or
.
Last modified November 08, 2021 at 5:19 PM PST:
Tutorial for pod security admission (d29e93acbb)
Home
Blog
Training
Partners
Community
CC BY 4.0

Access Services Running on Clusters | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set U

p kubectl on Linux
Debug Pods
Debug Run

ning Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Na

mespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-

apiserver Encryption Configuration (v1)
Tasks
This page shows how to connect to services running on the Kubernetes cluster.
minikube
.
Accessing services running on the cluster
In Kubernetes,
nodes
,
pods
and
services
all have
their own IPs. In many cases, the node IPs, pod IPs, and some service IPs on a cluster will not be
routable, so they will not be reachable from a machine outside the cluster,
such as your desktop machine.
Ways to connect
You have several options for connecting to nodes, pods and services from outside the cluster:
Access services through public IPs.
Use a service with type
NodePort
or
to make the service reachable outside
the cluster. See the
services
and
kubectl expose
Depending on your cluster environment, this may only expose the service to your corporate network,
or it may expose it to the internet. Think about whether the service being exposed is secure.
Does it do its own authentication?
Place pods behind services. To access one specific pod from a set of replicas, such as for debugging,
place a unique label on the pod and create a new service which selects this label.
In most cases, it should not be necessary for application developer to directly access
nodes via their nodeIPs.
Access services, nodes, or pods using the Proxy Verb.
Does apiserver authentication and authorization prior to accessing the remote service.
Use this if the services are not secure enough to expose to the internet, or to gain
access to ports on the node IP, or for debugging.
Proxies may cause problems for some web applications.
Only works for HTTP/HTTPS.
Described
here
.
Access from a node or pod in the cluster.
Run a pod, and then connect to a shell in it using
.
Connect to other nodes, pods, and services from that shell.
Some clusters may allow you to ssh to a node in the cluster. From there you may be able to
access cluster services. This is a non-standard method, and will work on some clusters but
not others. Browsers and other tools may or may not be installed. Cluster DN

S may not work.
Discovering builtin services
Typically, there are several services which are started on a cluster by kube-system. Get a list of these
with the
command:
Kubernetes master is running at https://192.0.2.1
elasticsearch-logging is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy
kibana-logging is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/kibana-logging/proxy
kube-dns is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/kube-dns/proxy
grafana is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
heapster is running at https://192.0.2.1/api/v1/namespaces/kube-system/services/monitoring-heapster/proxy
This shows the proxy-verb URL for accessing each service.
For example, this cluster has cluster-level logging enabled (using Elasticsearch), which can be reached
at
https://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/
if suitable credentials are passed, or through a kubectl proxy at, for example:
http://localhost:8080/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/
.
Note:
See
for how to pass credentials or use kubectl proxy.
Manually constructing apiserver proxy URLs
As mentioned above, you use the
command to retrieve the service's proxy URL. To create proxy URLs that include service endpoints, suffixes, and parameters, you append to the service's proxy URL:
http://
kubernetes_master_address
/api/v1/namespaces/
namespace_name
/services/
[https:]service_name[:port_name]
/proxy
If you haven't specified a name for your port, you don't have to specify
port_name
in the URL. You can also use the port number in place of the
port_name
for both named and unnamed ports.
By default, the API server proxies to your service using HTTP. To use HTTPS, prefix the service name with
https:
:
http://&lt;kubernetes_master_address&gt;/api/v1/namespaces/&lt;namespace_name&gt;/services/&lt;service_name&gt;/proxy
The supported formats for the
&lt;service_name&gt;
segment of the URL are:
- proxies to the default or unnamed port using http
&lt;service_name&gt;:&lt;port_name&gt;
- proxies to the specified port name or port number using http
https:&lt;service_name&gt;:
- proxies to the default or unnamed port using https (note the trailing colon)
https:&lt;service_name&gt;:&lt;port_name&gt;
- proxies to the specified port name or port number using https
Examples
To access the Elasticsearch service endpoint
_search?q=user:kimchy
, you would use:
http://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_search?q=user:kimchy
To access the Elasticsearch cluster health information
_cluster/health?pretty=true
https://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy/_cluster/health?pretty=true
The health information is similar to this:
{
"cluster_name"
:
"kubernetes_logging"
,
"status"
:
"yellow"
,
"timed_out"
:
false
,
"number_of_nodes"
:
1
,
"number_of_data_nodes"
:
1
,
"active_primary_shards"
:
5
,
"active_shards"
:
5
,
"relocating_shards"
:
0
,
"initializing_shards"
:
0
,
"unassigned_shards"
:
5
}
To access the
https
Elasticsearch service health information
https://192.0.2.1/api/v1/namespaces/kube-system/services/https:elasticsearch-logging:/proxy/_cluster/health?pretty=true
Using web browsers to access services running on the cluster
You may be able to put an apiserver proxy URL into the address bar of a browser. However:
Web browsers cannot usually pass tokens, so you may need to use basic (password) auth. Apiserver can be configured to accept basic auth,
but your cluster may not be configured to accept basic auth.
Some web apps may not work, particularly those with client side javascript that const

ruct URLs in a
way that is unaware of the proxy path prefix.
Feedback
Yes
No
.
or
.
Last modified May 18, 2022 at 11:59 AM PST:
Fix links about apiserver proxy (56e597ced4)
Home
Blog
Training
Partners
Community
CC BY 4.0

Define Dependent Environment Variables | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and

 Set Up kubectl on Linux
Debug Pods
Deb

ug Running Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at 

the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
This page shows how to define dependent environment variables for a container
in a Kubernetes Pod.
minikube
Define an environment dependent variable for a container
When you create a Pod, you can set dependent environment variables for the containers that run in the Pod. To set dependent environment variables, you can use $(VAR_NAME) in the
value
of
env
in the configuration file.
In this exercise, you create a Pod that runs one container. The configuration
file for the Pod defines a dependent environment variable with common usage defined. Here is the configuration manifest for the
Pod:
pods/inject/dependent-envars.yaml
:
v1
kind
:
Pod
metadata
:
name
:
dependent-envars-demo
spec
:
:
-
name
:
args
:
- while true; do echo -en '\n'; printf UNCHANGED_REFERENCE=$UNCHANGED_REFERENCE'\n'; printf SERVICE_ADDRESS=$SERVICE_ADDRESS'\n';printf ESCAPED_REFERENCE=$ESCAPED_REFERENCE'\n'; sleep 30; done;
command
:
- sh
- -c
image
:
env
:
-
name
:
SERVICE_PORT
value
:
"80"
-
name
:
SERVICE_IP
value
:
"172.17.0.1"
-
name
:
UNCHANGED_REFERENCE
value
:
"$(PROTOCOL)://$(SERVICE_IP):$(SERVICE_PORT)"
-
name
:
PROTOCOL
value
:
"https"
-
name
:
SERVICE_ADDRESS
value
:
-
name
:
ESCAPED_REFERENCE
value
:
"$$(PROTOCOL)://$(SERVICE_IP):$(SERVICE_PORT)"
Create a Pod based on that manifest:
kubectl apply -f https://k8s.io/examples/pods/inject/dependent-envars.yaml
pod/dependent-envars-demo created
List the running Pods:
kubectl get pods dependent-envars-demo
dependent-envars-demo     1/1       Running   0          9s
Check the logs for the container running in your Pod:
kubectl logs pod/dependent-envars-demo
UNCHANGED_REFERENCE=$(PROTOCOL)://172.17.0.1:80
SERVICE_ADDRESS=https://172.17.0.1:80
ESCAPED_REFERENCE=$(PROTOCOL)://172.17.0.1:80
As shown above, you have defined the correct dependency reference of
, bad dependency reference of
and sk

ip dependent references of
.
When an environment variable is already defined when being referenced,
the reference can be correctly resolved, such as in the
case.
Note that order matters in the
env
list. An environment variable is not considered
"defined" if it is specified further down the list. That is why
fails to resolve
$(PROTOCOL)
in the example above.
When the environment variable is undefined or only includes some variables, the undefined environment variable is treated as a normal string, such as
. Note that incorrectly parsed environment variables, in general, will not block the container from starting.
The
$(VAR_NAME)
syntax can be escaped with a double
$
, ie:
$$(VAR_NAME)
.
Escaped references are never expanded, regardless of whether the referenced variable
is defined or not. This can be seen from the
case above.
environment variables
.
See
EnvVarSource
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

ConfigMap | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

Reference
ConfigMap
ConfigMap
ConfigMap holds configuration data for pods to consume.
ConfigMap
: v1
kind
: ConfigMap
metadata
(
)
binaryData
(map[string][]byte)
BinaryData contains the binary data. Each key must consist of alphanumeric characters, '-', '_' or '.'. BinaryData can contain byte sequences that are not in the UTF-8 range. The keys stored in BinaryData must not overlap with the ones in the Data field, this is enforced during validation process. Using this field will require 1.10+ apiserver and kubelet.
data
Data contains the configuration data. Each key must consist of alphanumeric characters, '-', '_' or '.'. Values with non-UTF-8 byte sequences must use the BinaryData field. The keys stored in Data must not overlap with the keys in the BinaryData field, this is enforced during validation process.
immutable
(boolean)
Immutable, if set to true, ensures that data stored in the ConfigMap cannot be updated (only object metadata can be modified). If not set to true, the field can be modified at any time. Defaulted to nil.
ConfigMapList
ConfigMapList is a resource containing a list of ConfigMap objects.
: v1
kind
: ConfigMapList
metadata
(
ListMeta
)
More info:
items
([]
ConfigMap
Items is the list of ConfigMaps.
get
read the specified ConfigMap
GET /api/v1/namespaces/{namespace}/configmaps/{name}
name
(
in path
name of the ConfigMap
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
list
list or watch objects of kind ConfigMap
GET /api/v1/namespaces/{namespace}/configmaps
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
Config

MapList
): OK
list
GET /api/v1/configmaps
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ConfigMap
POST /api/v1/namespaces/{namespace}/configmaps
namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
202 (
ConfigMap
update
replace the specified ConfigMap
PUT /api/v1/namespaces/{namespace}/configmaps/{name}
name
(
in path
namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
patch
partially update the specified ConfigMap
PATCH /api/v1/namespaces/{namespace}/configmaps/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
delete
delete a ConfigMap
DELETE /api/v1/namespaces/{namespace}/configmaps/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ConfigMap
DELETE /api/v1/namespaces/{namespace}/configmaps
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Print e

ntire section
ConfigMap
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Admission Controllers Reference | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
Admission Controllers Reference
This page provides an overview of Admission Controllers.
What are they?
An
admission controller
is a piece of code that intercepts requests to the
Kubernetes API server prior to persistence of the object, but after the request
is authenticated and authorized.
Admission controllers may be
validating
,
mutating
, or both. Mutating
controllers may modify related objects to the requests they admit; validating controllers may not.
Admission controllers limit requests to create, delete, modify objects. Admission
controllers can also block custom verbs, such as a request connect to a Pod via
an API server proxy. Admission controllers do
not
(and cannot) block requests
to read (
get
,
watch
or
list
) objects.
The admission controllers in Kubernetes 1.26 consist of the
list
below, are compiled into the
binary, and may only be configured by the cluster
administrator. In that list, there are two special controllers:
MutatingAdmissionWebhook and ValidatingAdmissionWebhook. These execute the
mutating and validating (respectively)
admission control webhooks
which are configured in the API.
Admission control phases
The admission control process proceeds in two phases. In the first phase,
mutating admission controllers are run. In the second phase, validating
admission controllers are run. Note again that some of the controllers are
both.
If any of the controllers in either phase reject the request, the entire
request is rejected immediately and an error is returned to the end-user.
Finally, in addition to sometimes mutating the object in question, admission
controllers may sometimes have side effects, that is, mutate related
resources as part of request processing. Incrementing quota usage is the
canonical example of why this is necessary. Any such side-effect needs a
corresponding reclamation or reconciliation process, as a given admission
controller does not know for sure that a given request will pass all of the
other admission controllers.
Why do I need them?
Several important features of Kubernetes require an admission controller to be enabled in order
to properly support the feature. As a result, a Kubernetes API server that is not properly
configured with the right set of admission controllers is an incomplete server and will not
support all the features you expect.
How do I turn on an admission controller?
The Kubernetes API server flag
enable-admission-plugins
takes a comma-delimited list of admission control plugins to invoke prior to modifying objects in the 

cluster.
For example, the following command line enables the
and the
admission control plugins:
kube-apiserver --enable-admission-plugins
=
NamespaceLifecycle,LimitRanger ...
Note:
Depending on the way your Kubernetes cluster is deployed and how the API server is
started, you may need to apply the settings in different ways. For example, you may
have to modify the systemd unit file if the API server is deployed as a systemd
service, you may modify the manifest file for the API server if Kubernetes is deployed
in a self-hosted way.
How do I turn off an admission controller?
disable-admission-plugins
takes a comma-delimited list of admission control plugins to be disabled, even if they are in the list of plugins enabled by default.
kube-apiserver --disable-admission-plugins
=
PodNodeSelector,AlwaysDeny ...
Which plugins are enabled by default?
To see which admission plugins are enabled:
kube-apiserver -h | grep enable-admission-plugins
In Kubernetes 1.26, the default ones are:
CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, LimitRanger, MutatingAdmissionWebhook, NamespaceLifecycle, PersistentVolumeClaimResize, PodSecurity, Priority, ResourceQuota, RuntimeClass, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook
Note:
The
ValidatingAdmissionPolicy
admission plugin is enabled
by default, but is only active if you enable the the
and
the
admissionregistration.k8s.io/v1alpha1
API.
What does each admission controller do?
AlwaysAdmit
Kubernetes v1.13 [deprecated]
This admission controller allows all pods into the cluster. It is
because
its behavior is the same as if there were no admission controller at all.
AlwaysDeny
Rejects all requests. AlwaysDeny is
as it has no real meaning.
AlwaysPullImages
This admission controller modifies every new Pod to force the image pull policy to
Always
. This is useful in a
multitenant cluster so that users can be assured that their private images can only be used by those
who have the credentials to pull them. Without this admission controller, once an image has been pulled to a
node, any pod from any user can use it by knowing the image's name (assuming the Pod is
scheduled onto the right node), without any authorization check against the image. When this admission controller
is enabled, images are always pulled prior to starting containers, which means valid credentials are
required.
CertificateApproval
This admission controller observes requests to approve CertificateSigningRequest resources and performs additional
authorization checks to ensure the approving user has permission to
approve
certificate requests with the
spec.signerName
requested on the CertificateSigningRequest resource.
See
for more
information on the permissions required to perform different actions on CertificateSigningRequest resources.
CertificateSigning
This admission controller observes updates to the
status.certificate
field of CertificateSigningRequest resources
and performs an additional authorization checks to ensure the signing user has permission to
sign
requests with the
See
for more
CertificateSubjectRestriction
This admission controller observes creation of CertificateSigningRequest resources that have a
of
kubernetes.io/kube-apiserver-client
. It rejects any request that specifies a 'group' (or 'organization attribute')
of
.
DefaultIngressClass
This admission controller observes creation of
Ingress
obj

ects that do not request any specific
ingress class and automatically adds a default ingress class to them. This way, users that do not
request any special ingress class do not need to care about them at all and they will get the
default one.
This admission controller does not do anything when no default ingress class is configured. When more than one ingress
class is marked as default, it rejects any creation of
Ingress
with an error and an administrator
must revisit their
objects and mark only one as default (with the annotation
"ingressclass.kubernetes.io/is-default-class"). This admission controller ignores any
Ingress
updates; it acts only on creation.
See the
Ingress
documentation for more about ingress
classes and how to mark one as default.
objects that do not request any specific storage class
and automatically adds a default storage class to them.
This way, users that do not request any special storage class do not need to care about them at all and they
will get the default one.
This admission controller does not do anything when no default storage class is configured. When more than one storage
objects and mark only one as default.
This admission controller ignores any
See
persistent volume
documentation about persistent volume claims and
storage classes and how to mark a storage class as default.
This admission controller sets the default forgiveness toleration for pods to tolerate
the taints
notready:NoExecute
and
unreachable:NoExecute
based on the k8s-apiserver input parameters
default-not-ready-toleration-seconds
and
default-unreachable-toleration-seconds
if the pods don't already
have toleration for taints
node.kubernetes.io/not-ready:NoExecute
or
node.kubernetes.io/unreachable:NoExecute
.
The default value for
and
is 5 minutes.
DenyServiceExternalIPs
This admission controller rejects all net-new usage of the
Service
field
. This
feature is very powerful (allows network traffic interception) and not well
controlled by policy. When enabled, users of the cluster may not create new
Services which use
and may not add new values to
on
existing
Service
objects. Existing uses of
are not affected,
and users may remove values from
on existing
Service
objects.
Most users do not need this feature at all, and cluster admins should consider disabling it.
Clusters that do need to use this feature should consider using some custom policy to manage usage
of it.
This admission controller is disabled by default.
EventRateLimit
Kubernetes v1.13 [alpha]
This admission controller mitigates the problem where the API server gets flooded by
requests to store new Events. The cluster admin can specify event rate limits by:
Enabling the
admission controller;
Referencing an
configuration file from the file provided to the API
server's command line flag
--admission-control-config-file
:
:
kind
:
plugins
:
-
name
:
path
:
eventconfig.yaml
...
There are four types of limits that can be specified in the configuration:
Server
: All Event requests (creation or modifications) received by the API server share a single bucket.
Namespace
: Each namespace has a dedicated bucket.
User
: Each user is allocated a bucket.
SourceAndObject
: A bucket is assigned by each combination of source and
involved object of the event.
Below is a sample
for such a configuration:
:
eventratelimit.admission.k8s.io/v1alpha1
kind
:
limits
:
-
type
:
Name

space
qps
:
50
burst
:
100
cacheSize
:
2000
-
type
:
User
qps
:
10
burst
:
50
See the
EventRateLimit Config API (v1alpha1)
ExtendedResourceToleration
This plug-in facilitates creation of dedicated nodes with extended resources.
If operators want to create dedicated nodes with extended resources (like GPUs, FPGAs etc.), they are expected to
taint the node
with the extended resource
name as the key. This admission controller, if enabled, automatically
adds tolerations for such taints to pods requesting extended resources, so users don't have to manually
add these tolerations.
ImagePolicyWebhook
The ImagePolicyWebhook admission controller allows a backend webhook to make admission decisions.
Configuration file format
ImagePolicyWebhook uses a configuration file to set options for the behavior of the backend.
This file may be json or yaml and has the following format:
imagePolicy
:
:
/path/to/kubeconfig/for/backend
# time in s to cache approval
allowTTL
:
50
# time in s to cache denial
denyTTL
:
50
# time in ms to wait between retries
retryBackoff
:
500
# determines behavior if the webhook backend fails
defaultAllow
:
true
Reference the ImagePolicyWebhook configuration file from the file provided to the API server's command line flag
:
:
kind
:
plugins
:
-
name
:
path
:
imagepolicyconfig.yaml
...
Alternatively, you can embed the configuration directly in the file:
:
kind
:
plugins
:
-
name
:
:
:
:
&lt;path-to-kubeconfig-file&gt;
allowTTL
:
50
denyTTL
:
50
:
500
:
true
The ImagePolicyWebhook config file must reference a
formatted file which sets up the connection to the backend.
It is required that the backend communicate over TLS.
The kubeconfig file's
cluster
field must point to the remote service, and the
user
field
must contain the returned authorizer.
# clusters refers to the remote service.
clusters
:
-
name
:
name-of-remote-imagepolicy-service
cluster
:
:
/path/to/ca.pem
# CA for verifying the remote service.
server
:
https://images.example.com/policy
# URL of remote service to query. Must use 'https'.
# users refers to the API server's webhook configuration.
users
:
-
name
:
name-of-api-server
user
:
:
/path/to/cert.pem
# cert for the webhook admission controller to use
:
/path/to/key.pem
# key matching the cert
For additional HTTP configuration, refer to the
Request payloads
When faced with an admission decision, the API Server POSTs a JSON serialized
imagepolicy.k8s.io/v1alpha1
ImageReview
object describing the action.
This object contains fields describing the containers being admitted, as well as
any pod annotations that match
*.image-policy.k8s.io/*
.
Note:
The webhook API objects are subject to the same versioning compatibility rules
as other Kubernetes API objects. Implementers should be aware of looser compatibility
promises for alpha objects and check the
field of the request to
ensure correct deserialization.
Additionally, the API Server must enable the
API extensions
group (
--runtime-config=imagepolicy.k8s.io/v1alpha1=true
).
An example request body:
{
:
"imagepolicy.k8s.io/v1alpha1"
,
"kind"
:
"ImageReview"
,
"spec"
:{
:[
{
"image"
:
"myrepo/myimage:v1"
},
{
"image"
:
"myrepo/myimage@sha256:beb6bd6a68f114c1dc2ea4b28db81bdf91de202a9014972bec5e4d9171d90ed"
}
],
:{
"mycluster.image-policy.k8s.io/ticket-1234"
:
"break-glass"
},
:
"mynamespace"
}
}
The remote service is expected to fill the
status
field of

 the request and
respond to either allow or disallow access. The response body's
spec
field is ignored, and
may be omitted. A permissive response would return:
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
true
}
}
To disallow access, the service would return:
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"reason"
:
"image currently blacklisted"
}
}
For further documentation refer to the
imagepolicy.v1alpha1
API
.
Extending with Annotations
All annotations on a Pod that match
are sent to the webhook.
Sending annotations allows users who are aware of the image policy backend to
send extra information to it, and for different backends implementations to
accept different information.
Examples of information you might put here are:
request to "break glass" to override a policy, in case of emergency.
a ticket number from a ticket system that documents the break-glass request
provide a hint to the policy server as to the imageID of the image being provided, to save it a lookup
In any case, the annotations are provided by the user and are not validated by Kubernetes in any way.
LimitPodHardAntiAffinityTopology
This admission controller denies any pod that defines
AntiAffinity
topology key other than
in
requiredDuringSchedulingRequiredDuringExecution
.
This admission controller will observe the incoming request and ensure that it does not violate
any of the constraints enumerated in the
object in a
Namespace
. If you are using
objects in your Kubernetes deployment, you MUST use this admission controller to
enforce those constraints. LimitRanger can also be used to apply default resource requests to Pods
that don't specify any; currently, the default LimitRanger applies a 0.1 CPU requirement to all
Pods in the
default
See the
LimitRange API reference
and the
example of LimitRange
MutatingAdmissionWebhook
This admission controller calls any mutating webhooks which match the request. Matching
webhooks are called in serial; each one may modify the object if it desires.
This admission controller (as implied by the name) only runs in the mutating phase.
If a webhook called by this has side effects (for example, decrementing quota) it
must
have a reconciliation system, as it is not guaranteed that subsequent
webhooks or validating admission controllers will permit the request to finish.
If you disable the MutatingAdmissionWebhook, you must also disable the
object in the
group/version via the
flag, both are on by default.
Use caution when authoring and installing mutating webhooks
Users may be confused when the objects they try to create are different from
what they get back.
Built in control loops may break when the objects they try to create are
different when read back.
Setting originally unset fields is less likely to cause problems than
overwriting fields set in the original request. Avoid doing the latter.
Future changes to control loops for built-in resources or third-party resources
may break webhooks that work well today. Even when the webhook installation API
is finalized, not all possible webhook behaviors will be guaranteed to be supported
indefinitely.
NamespaceAutoProvision
This admission controller examines all incoming requests on namespaced resources and checks
if the referenced namespace does exist.
It creates a namespace if it cannot be found.
This admission controller is useful in deployments that do not want to restrict creation of
a namespace prior to its usage.
NamespaceExists
This admission controller checks all requests on namespaced resources other than
Namespace
itself.
If the namespace referenced from a request doesn't exist, the request is rejected.

This admission controller enforces that a
Namespace
that is undergoing termination cannot have
new objects created in it, and ensures that requests in a non-existent
Namespace
are rejected.
This admission controller also prevents deletion of three system reserved namespaces
default
,
,
.
A
Namespace
deletion kicks off a sequence of operations that remove all objects (pods, services,
etc.) in that namespace. In order to enforce integrity of that process, we strongly recommend
running this admission controller.
This admission controller limits the
Node
and
Pod
objects a kubelet can modify. In order to be limited by this admission controller,
kubelets must use credentials in the
group, with a username in the form
system:node:&lt;nodeName&gt;
.
Such kubelets will only be allowed to modify their own
Node
API object, and only modify
Pod
API objects that are bound to their node.
kubelets are not allowed to update or remove taints from their
Node
The
admission plugin prevents kubelets from deleting their
Node
API object,
and enforces kubelet modification of labels under the
kubernetes.io/
or
k8s.io/
prefixes as follows:
Prevents
kubelets from adding/removing/updating labels with a
node-restriction.kubernetes.io/
prefix.
This label prefix is reserved for administrators to label their
Node
objects for workload isolation purposes,
and kubelets will not be allowed to modify labels with that prefix.
Allows
kubelets to add/remove/update these labels and label prefixes:
beta.kubernetes.io/instance-type
node.kubernetes.io/instance-type
failure-domain.beta.kubernetes.io/region
(deprecated)
failure-domain.beta.kubernetes.io/zone
topology.kubernetes.io/region
topology.kubernetes.io/zone
kubelet.kubernetes.io/
-prefixed labels
node.kubernetes.io/
Use of any other labels under the
kubernetes.io
or
k8s.io
prefixes by kubelets is reserved,
and may be disallowed or allowed by the
admission plugin in the future.
Future versions may add additional restrictions to ensure kubelets have the minimal set of
permissions required to operate correctly.
OwnerReferencesPermissionEnforcement
This admission controller protects the access to the
metadata.ownerReferences
of an object
so that only users with
delete
permission to the object can change it.
This admission controller also protects the access to
metadata.ownerReferences[x].blockOwnerDeletion
of an object, so that only users with
update
permission to the
subresource of the referenced
owner
can change it.
PersistentVolumeClaimResize
This admission controller implements additional validations for checking incoming
resize requests.
admission controller is recommended.
This admission controller prevents resizing of all claims by default unless a claim's
explicitly enables resizing by setting
allowVolumeExpansion
to
true
.
For example: all
s created from the following
support volume expansion:
:
kind
:
metadata
:
name
:
gluster-vol-default
provisioner
:
kubernetes.io/glusterfs
parameters
:
resturl
:
"http://192.168.10.100:8080"
restuser
:
""
secretNamespace
:
""
:
""
:
true
For more information about persistent volume claims, see
.
This admission controller automatically attaches region or zone labels to PersistentVolumes
as defined by the cloud provider (for example, Azure or GCP).
It helps ensure the Pods and the PersistentVolumes mounted are in the same
region and/or zone.
If the admission controller doesn't support automatic labelling your PersistentVolumes, you
may need to add the labels manually to pre

vent pods from mounting volumes from
a different zone. PersistentVolumeLabel is
as labeling for persistent volumes has been taken over by
the
.
Kubernetes v1.5 [alpha]
This admission controller defaults and limits what node selectors may be used within a namespace
by reading a namespace annotation and a global configuration.
uses a configuration file to set options for the behavior of the backend.
Note that the configuration file format will move to a versioned file in a future release.
podNodeSelectorPluginConfig
:
clusterDefaultNodeSelector
:
name-of-node-selector
namespace1
:
namespace2
:
Reference the
configuration file from the file provided to the API server's
:
:
kind
:
plugins
:
-
name
:
path
:
podnodeselector.yaml
...
Configuration Annotation Format
uses the annotation key
scheduler.alpha.kubernetes.io/node-selector
to assign
node selectors to namespaces.
:
v1
kind
:
Namespace
metadata
:
:
:
name
:
namespace3
Internal Behavior
This admission controller has the following behavior:
If the
Namespace
has an annotation with a key
,
use its value as the node selector.
If the namespace lacks such an annotation, use the
defined in the
plugin configuration file as the node selector.
Evaluate the pod's node selector against the namespace node selector for conflicts. Conflicts
result in rejection.
Evaluate the pod's node selector against the namespace-specific allowed selector defined the
plugin configuration file. Conflicts result in rejection.
Note:
PodNodeSelector allows forcing pods to run on specifically labeled nodes. Also see the PodTolerationRestriction
admission plugin, which allows preventing pods from running on specifically tainted nodes.
PodSecurity
This is the replacement for the deprecated
defined in the next section. This admission controller acts on creation and modification of the pod and
determines if it should be admitted based on the requested security context and the
.
See the
Pod Security Admission documentation
Kubernetes v1.21 [deprecated]
This admission controller acts on creation and modification of the pod and determines if it should be admitted
based on the requested security context and the available Pod Security Policies.
See also the
PodTolerationRestriction
Kubernetes v1.7 [alpha]
The PodTolerationRestriction admission controller verifies any conflict between tolerations of a
pod and the tolerations of its namespace.
It rejects the pod request if there is a conflict.
It then merges the tolerations annotated on the namespace into the tolerations of the pod.
The resulting tolerations are checked against a list of allowed tolerations annotated to the namespace.
If the check succeeds, the pod request is admitted otherwise it is rejected.
If the namespace of the pod does not have any associated default tolerations or allowed
tolerations annotated, the cluster-level default tolerations or cluster-level list of allowed tolerations are used
instead if they are specified.
Tolerations to a namespace are assigned via the
scheduler.alpha.kubernetes.io/defaultTolerations
annotation key.
The list of allowed tolerations can be added via the
scheduler.al

pha.kubernetes.io/tolerationsWhitelist
Example for namespace annotations:
:
v1
kind
:
Namespace
metadata
:
name
:
apps-that-need-nodes-exclusively
:
:
'[{"operator": "Exists", "effect": "NoSchedule", "key": "dedicated-node"}]'
scheduler.alpha.kubernetes.io/tolerationsWhitelist
:
Priority
The priority admission controller uses the
field and populates the integer
value of the priority.
If the priority class is not found, the Pod is rejected.
object in a
Namespace
. If you are
using
objects in your Kubernetes deployment, you MUST use this admission
controller to enforce quota constraints.
See the
ResourceQuota API reference
and the
example of Resource Quota
If you define a RuntimeClass with
Pod overhead
configured, this admission controller checks incoming Pods.
When enabled, this admission controller rejects any Pod create requests
that have the overhead already set.
For Pods that have a RuntimeClass configured and selected in their
.spec
,
this admission controller sets
.spec.overhead
in the Pod based on the value
defined in the corresponding RuntimeClass.
See also
SecurityContextDeny
This admission controller will deny any Pod that attempts to set certain escalating
fields, as shown in the
task.
If you don't use
,
, nor any external enforcement mechanism,
then you could use this admission controller to restrict the set of values a security context can take.
See
for more context on restricting
pod privileges.
This admission controller implements automation for
serviceAccounts
.
The Kubernetes project strongly recommends enabling this admission controller.
You should enable this admission controller if you intend to make any use of Kubernetes
objects.
StorageObjectInUseProtection
The
plugin adds the
kubernetes.io/pvc-protection
or
kubernetes.io/pv-protection
finalizers to newly created Persistent Volume Claims (PVCs) or Persistent Volumes (PV).
In case a user deletes a PVC or PV the PVC or PV is not removed until the finalizer is removed
from the PVC or PV by PVC or PV Protection Controller.
Refer to the
Storage Object in Use Protection
for more detailed information.
TaintNodesByCondition
This admission controller
taints
newly created
Nodes as
NotReady
and
. That tainting avoids a race condition that could cause Pods
to be scheduled on new Nodes before their taints were updated to accurately reflect their reported
conditions.
implements the CEL validation for incoming matched requests.
It is enabled when both feature gate
validatingadmissionpolicy
and
group/version are enabled.
If any of the ValidatingAdmissionPolicy fails, the request fails.
ValidatingAdmissionWebhook
This admission controller calls any validating webhooks which match the request. Matching
webhooks are called in parallel; if any of them rejects the request, the request
fails. This admission controller only runs in the validation phase; the webhooks it calls may not
mutate the object, as opposed to the webhooks called by the
admission controller.
must
webhooks or other validating admission controllers will permit the request to fini

sh.
If you disable the ValidatingAdmissionWebhook, you must also disable the
flag.
Is there a recommended set of admission controllers to use?
Yes. The recommended admission controllers are enabled by default
(shown
here
),
so you do not need to explicitly specify them.
You can enable additional admission controllers beyond the default set using the
flag (
order doesn't matter
).
Feedback
Yes
No
.
or
.
Last modified December 20, 2022 at 1:24 AM PST:
Updated the wrong format (b590431f4e)
Priority
Home
Blog
Training
Partners
Community
CC BY 4.0

Updating Reference Documentation | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operat

1)
The topics in this section document how to generate the Kubernetes
reference guides.
To build the reference documentation, see the following guide:
Generating Reference Documentation Quickstart
Feedback
Yes
No
.
or
.
Last modified October 27, 2022 at 12:34 PM PST:
Revise page title (e071c5f206)
Home
Blog
Training
Partners
Community
CC BY 4.0

Volume | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Up kubectl on Windows
Debug Pods
R

esource metrics pipeline
Auditing
Run Jobs
TLS
Tutorials
Security

Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configuration (v1alp

ha1)
Reference
Volume
Volume
Volume represents a named volume in a pod that may be accessed by any container in the pod.
Volume
name
name of the volume. Must be a DNS_LABEL and unique within the pod. More info:
https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
Exposed Persistent volumes
persistentVolumeClaim
(PersistentVolumeClaimVolumeSource)
persistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
PersistentVolumeClaimVolumeSource references the user's PVC in the same namespace. This volume finds the bound PV and mounts that volume for the pod. A PersistentVolumeClaimVolumeSource is, essentially, a wrapper around another type of volume that is owned by someone else (the system).
persistentVolumeClaim.claimName
claimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info:
persistentVolumeClaim.readOnly
(boolean)
readOnly Will force the ReadOnly setting in VolumeMounts. Default false.
Projections
configMap
(ConfigMapVolumeSource)
configMap represents a configMap that should populate this volume
*Adapts a ConfigMap into a volume.
The contents of the target ConfigMap's Data field will be presented in a volume as files using the keys in the Data field as the file names, unless the items element is populated with specific mappings of keys to paths. ConfigMap volumes support ownership management and SELinux relabeling.*
configMap.name
(string)
Name of the referent. More info:
configMap.optional
(boolean)
optional specify whether the ConfigMap or its keys must be defined
configMap.defaultMode
(int32)
defaultMode is optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
configMap.items
([]
KeyToPath
)
items if unspecified, each key-value pa

ir in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
secret
(SecretVolumeSource)
secret represents a secret that should populate this volume. More info:
https://kubernetes.io/docs/concepts/storage/volumes#secret
*Adapts a Secret into a volume.
The contents of the target Secret's Data field will be presented in a volume as files using the keys in the Data field as the file names. Secret volumes support ownership management and SELinux relabeling.*
secret.secretName
(string)
secretName is the name of the secret in the pod's namespace to use. More info:
secret.optional
(boolean)
optional field specify whether the Secret or its keys must be defined
secret.defaultMode
(int32)
defaultMode is Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
secret.items
([]
KeyToPath
)
items If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
(DownwardAPIVolumeSource)
downwardAPI represents downward API about the pod that should populate this volume
DownwardAPIVolumeSource represents a volume containing downward API info. Downward API volumes support ownership management and SELinux relabeling.
downwardAPI.defaultMode
(int32)
Optional: mode bits to use on created files by default. Must be a Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
downwardAPI.items
([]
DownwardAPIVolumeFile
)
Items is a list of downward API volume file
projected
(ProjectedVolumeSource)
projected items for all in one resources secrets, configmaps, and downward API
Represents a projected volume source
projected.defaultMode
(int32)
defaultMode are the mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
projected.sources
([]VolumeProjection)
sources is the list of volume projections
Projection that may be projected along with other supported volume types
projected.sources.configMap
(ConfigMapProjection)
configMap information about the configMap data to project
*Adapts a ConfigMap into a projected volume.
The contents of the target ConfigMap's Dat

a field will be presented in a projected volume as files using the keys in the Data field as the file names, unless the items element is populated with specific mappings of keys to paths. Note that this is identical to a configmap volume source without the default mode.*
projected.sources.configMap.name
(string)
projected.sources.configMap.optional
(boolean)
projected.sources.configMap.items
([]
KeyToPath
)
items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
projected.sources.downwardAPI
(DownwardAPIProjection)
downwardAPI information about the downwardAPI data to project
Represents downward API info for projecting into a projected volume. Note that this is identical to a downwardAPI volume source without the default mode.
projected.sources.downwardAPI.items
([]
)
Items is a list of DownwardAPIVolume file
projected.sources.secret
(SecretProjection)
secret information about the secret data to project
*Adapts a secret into a projected volume.
The contents of the target Secret's Data field will be presented in a projected volume as files using the keys in the Data field as the file names. Note that this is identical to a secret volume source without the default mode.*
projected.sources.secret.name
(string)
projected.sources.secret.optional
(boolean)
optional field specify whether the Secret or its key must be defined
projected.sources.secret.items
([]
KeyToPath
)
items if unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
projected.sources.serviceAccountToken
(ServiceAccountTokenProjection)
serviceAccountToken is information about the serviceAccountToken data to project
ServiceAccountTokenProjection represents a projected service account token volume. This projection can be used to insert a service account token into the pods runtime filesystem for use against APIs (Kubernetes API Server or otherwise).
projected.sources.serviceAccountToken.path
path is the path relative to the mount point of the file to project the token into.
projected.sources.serviceAccountToken.audience
(string)
audience is the intended audience of the token. A recipient of a token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. The audience defaults to the identifier of the apiserver.
projected.sources.serviceAccountToken.expirationSeconds
(int64)
expirationSeconds is the requested duration of validity of the service account token. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.Defaults to 1 hour and must be at least 10 minutes.
Local / Temporary Directory
emptyDir
(EmptyDirVolumeSource)
emptyDir represents a te

mporary directory that shares a pod's lifetime. More info:
https://kubernetes.io/docs/concepts/storage/volumes#emptydir
Represents an empty directory for a pod. Empty directory volumes support ownership management and SELinux relabeling.
emptyDir.medium
(string)
medium represents what type of storage medium should back this directory. The default is "" which means to use the node's default medium. Must be an empty string (default) or Memory. More info:
emptyDir.sizeLimit
(
Quantity
)
sizeLimit is the total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info:
http://kubernetes.io/docs/user-guide/volumes#emptydir
hostPath
(HostPathVolumeSource)
hostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info:
https://kubernetes.io/docs/concepts/storage/volumes#hostpath
Represents a host path mapped into a pod. Host path volumes do not support ownership management or SELinux relabeling.
hostPath.path
path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info:
hostPath.type
(string)
type for HostPath Volume Defaults to "" More info:
Persistent volumes
awsElasticBlockStore
(AWSElasticBlockStoreVolumeSource)
awsElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
*Represents a Persistent Disk resource in AWS.
An AWS EBS disk must exist before mounting to a container. The disk must also be in the same AWS zone as the kubelet. An AWS EBS disk can only be mounted as read/write once. AWS EBS volumes support ownership management and SELinux relabeling.*
awsElasticBlockStore.volumeID
volumeID is unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info:
awsElasticBlockStore.fsType
(string)
fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info:
awsElasticBlockStore.partition
(int32)
partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty).
awsElasticBlockStore.readOnly
(boolean)
readOnly value true will force the readOnly setting in VolumeMounts. More info:
azureDisk
(AzureDiskVolumeSource)
azureDisk represents an Azure Data Disk mount on the host and bind mount to the pod.
AzureDisk represents an Azure Data Disk mount on the host and bind mount to the pod.
azureDisk.diskName
diskName is the Name of the data disk in the blob storage
azureDisk.diskURI
diskURI is the URI of data disk in the blob storage
azureDisk.cachingMode
(string)
cachingMode is the Host Caching mode: None, Read Only, Read Write.
azureDisk.fsType
(string)
fsType is Filesystem type to mount.

 Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified.
azureDisk.kind
(string)
kind expected values are Shared: multiple blob disks per storage account Dedicated: single blob disk per storage account Managed: azure managed data disk (only in managed availability set). defaults to shared
azureDisk.readOnly
(boolean)
readOnly Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.
azureFile
(AzureFileVolumeSource)
azureFile represents an Azure File Service mount on the host and bind mount to the pod.
AzureFile represents an Azure File Service mount on the host and bind mount to the pod.
azureFile.secretName
secretName is the name of secret that contains Azure Storage Account Name and Key
azureFile.shareName
shareName is the azure share Name
azureFile.readOnly
(boolean)
readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.
cephfs
(CephFSVolumeSource)
cephFS represents a Ceph FS mount on the host that shares a pod's lifetime
Represents a Ceph Filesystem mount that lasts the lifetime of a pod Cephfs volumes do not support ownership management or SELinux relabeling.
cephfs.monitors
monitors is Required: Monitors is a collection of Ceph monitors More info:
https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
cephfs.path
(string)
path is Optional: Used as the mounted root, rather than the full Ceph tree, default is /
cephfs.readOnly
(boolean)
readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info:
cephfs.secretFile
(string)
secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info:
cephfs.secretRef
(
)
secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty. More info:
cephfs.user
(string)
user is optional: User is the rados user name, default is admin More info:
cinder
(CinderVolumeSource)
cinder represents a cinder volume attached and mounted on kubelets host machine. More info:
https://examples.k8s.io/mysql-cinder-pd/README.md
Represents a cinder volume resource in Openstack. A Cinder volume must exist before mounting to a container. The volume must also be in the same region as the kubelet. Cinder volumes support ownership management and SELinux relabeling.
cinder.volumeID
volumeID used to identify the volume in cinder. More info:
cinder.fsType
(string)
fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info:
cinder.readOnly
(boolean)
readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info:
cinder.secretRef
(
)
secretRef is optional: points to a secret object containing parameters used to connect to OpenStack.
csi
(CSIVolumeSource)
csi (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature).
Represents a source location of a volume to mount, managed by an external CSI driver
csi.driver
driver is the name of the CSI driver that handles this volume. Consult with your admin for the correct name as registered in the cluster.
csi.fsType
(string)
fsType to mount. Ex. "ext4", "xfs", "ntfs". If 

not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply.
csi.nodePublishSecretRef
(
)
nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed.
csi.readOnly
(boolean)
readOnly specifies a read-only configuration for the volume. Defaults to false (read/write).
csi.volumeAttributes
volumeAttributes stores driver-specific properties that are passed to the CSI driver. Consult your driver's documentation for supported values.
ephemeral
(EphemeralVolumeSource)
ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed.
Use this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity
tracking are needed,
c) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through
a PersistentVolumeClaim (see EphemeralVolumeSource for more
information on the connection between this volume type
and PersistentVolumeClaim).
Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod.
Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information.
A pod can use both types of ephemeral volumes and persistent volumes at the same time.
Represents an ephemeral volume that is handled by a normal storage driver.
ephemeral.volumeClaimTemplate
(PersistentVolumeClaimTemplate)
Will be used to create a stand-alone PVC to provision the volume. The pod in which this EphemeralVolumeSource is embedded will be the owner of the PVC, i.e. the PVC will be deleted together with the pod. The name of the PVC will be
\&lt;pod name&gt;-\&lt;volume name&gt;
where
\&lt;volume name&gt;
is the name from the
PodSpec.Volumes
array entry. Pod validation will reject the pod if the concatenated name is not valid for a PVC (for example, too long).
An existing PVC with that name that is not owned by the pod will
not
be used for the pod to avoid using an unrelated volume by mistake. Starting the pod is then blocked until the unrelated PVC is removed. If such a pre-created PVC is meant to be used by the pod, the PVC has to updated with an owner reference to the pod once the pod exists. Normally this should not be necessary, but it may be useful when manually reconstructing a broken cluster.
This field is read-only and no changes will be made by Kubernetes to the PVC after it has been created.
Required, must not be nil.
PersistentVolumeClaimTemplate is used to produce PersistentVolumeClaim objects as part of an EphemeralVolumeSource.
ephemeral.volumeClaimTemplate.spec
(
PersistentVolumeClaimSpec
The specification for the PersistentVolumeClaim. The entire content is copied unchanged into the PVC that gets created from this template. The same fields as in a PersistentVolumeClaim are also valid here.
ephemeral.volumeClaimTemplate.metadata
(
)
May contain labels and annotations that will be copied into the PVC when creating it. No other fields are allowed and will be rejected during validation.
fc
(FCVolumeSource)
fc represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod.
Represents a Fibre Channel volume. Fibre Channel volumes can only be mounted as read/write once. Fibre Channel volumes support ownership management and SELinux relabeling.
fc.fsType
(string)
fsType is the filesystem type

 to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified.
fc.lun
(int32)
lun is Optional: FC target lun number
fc.readOnly
(boolean)
readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.
fc.targetWWNs
targetWWNs is Optional: FC target worldwide names (WWNs)
fc.wwids
wwids Optional: FC volume world wide identifiers (wwids) Either wwids or combination of targetWWNs and lun must be set, but not both simultaneously.
flexVolume
(FlexVolumeSource)
flexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.
FlexVolume represents a generic volume resource that is provisioned/attached using an exec based plugin.
flexVolume.driver
driver is the name of the driver to use for this volume.
flexVolume.fsType
(string)
fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default filesystem depends on FlexVolume script.
flexVolume.options
options is Optional: this field holds extra command options if any.
flexVolume.readOnly
(boolean)
readOnly is Optional: defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.
flexVolume.secretRef
(
)
secretRef is Optional: secretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts.
flocker
(FlockerVolumeSource)
flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running
Represents a Flocker volume mounted by the Flocker agent. One and only one of datasetName and datasetUUID should be set. Flocker volumes do not support ownership management or SELinux relabeling.
flocker.datasetName
(string)
datasetName is Name of the dataset stored as metadata -&gt; name on the dataset for Flocker should be considered as deprecated
flocker.datasetUUID
(string)
datasetUUID is the UUID of the dataset. This is unique identifier of a Flocker dataset
gcePersistentDisk
(GCEPersistentDiskVolumeSource)
gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
*Represents a Persistent Disk resource in Google Compute Engine.
A GCE PD must exist before mounting to a container. The disk must also be in the same GCE project and zone as the kubelet. A GCE PD can only be mounted as read/write once or read-only many times. GCE PDs support ownership management and SELinux relabeling.*
gcePersistentDisk.pdName
pdName is unique name of the PD resource in GCE. Used to identify the disk in GCE. More info:
gcePersistentDisk.fsType
(string)
fsType is filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info:
gcePersistentDisk.partition
(int32)
partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). More info:
gcePersistentDisk.readOnly
(boolean)
readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info:

iscsi
(ISCSIVolumeSource)
iscsi represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info:
https://examples.k8s.io/volumes/iscsi/README.md
Represents an ISCSI disk. ISCSI volumes can only be mounted as read/write once. ISCSI volumes support ownership management and SELinux relabeling.
iscsi.iqn
iqn is the target iSCSI Qualified Name.
iscsi.lun
lun represents iSCSI Target Lun number.
iscsi.targetPortal
targetPortal is iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).
iscsi.chapAuthDiscovery
(boolean)
chapAuthDiscovery defines whether support iSCSI Discovery CHAP authentication
iscsi.chapAuthSession
(boolean)
chapAuthSession defines whether support iSCSI Session CHAP authentication
iscsi.fsType
(string)
https://kubernetes.io/docs/concepts/storage/volumes#iscsi
iscsi.initiatorName
(string)
initiatorName is the custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface &lt;target portal&gt;:&lt;volume name&gt; will be created for the connection.
iscsi.iscsiInterface
(string)
iscsiInterface is the interface Name that uses an iSCSI transport. Defaults to 'default' (tcp).
iscsi.portals
portals is the iSCSI Target Portal List. The portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).
iscsi.readOnly
(boolean)
readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false.
iscsi.secretRef
(
)
secretRef is the CHAP Secret for iSCSI target and initiator authentication
nfs
(NFSVolumeSource)
nfs represents an NFS mount on the host that shares a pod's lifetime More info:
https://kubernetes.io/docs/concepts/storage/volumes#nfs
Represents an NFS mount that lasts the lifetime of a pod. NFS volumes do not support ownership management or SELinux relabeling.
nfs.path
path that is exported by the NFS server. More info:
nfs.server
server is the hostname or IP address of the NFS server. More info:
nfs.readOnly
(boolean)
readOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info:
photonPersistentDisk
(PhotonPersistentDiskVolumeSource)
photonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine
Represents a Photon Controller persistent disk resource.
photonPersistentDisk.pdID
pdID is the ID that identifies Photon Controller persistent disk
photonPersistentDisk.fsType
(string)
fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified.
portworxVolume
(PortworxVolumeSource)
portworxVolume represents a portworx volume attached and mounted on kubelets host machine
PortworxVolumeSource represents a Portworx volume resource.
portworxVolume.volumeID
volumeID uniquely identifies a Portworx volume
portworxVolume.fsType
(string)
fSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs". Implicitly inferred to be "ext4" if unspecified.
portworxVolume.readOnly
(boolean)
readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts

.
quobyte
(QuobyteVolumeSource)
quobyte represents a Quobyte mount on the host that shares a pod's lifetime
Represents a Quobyte mount that lasts the lifetime of a pod. Quobyte volumes do not support ownership management or SELinux relabeling.
quobyte.registry
registry represents a single or multiple Quobyte Registry services specified as a string as host:port pair (multiple entries are separated with commas) which acts as the central registry for volumes
quobyte.volume
volume is a string that references an already created Quobyte volume by name.
quobyte.group
(string)
group to map volume access to Default is no group
quobyte.readOnly
(boolean)
readOnly here will force the Quobyte volume to be mounted with read-only permissions. Defaults to false.
quobyte.tenant
(string)
tenant owning the given Quobyte volume in the Backend Used with dynamically provisioned Quobyte volumes, value is set by the plugin
quobyte.user
(string)
user to map volume access to Defaults to serivceaccount user
rbd
(RBDVolumeSource)
rbd represents a Rados Block Device mount on the host that shares a pod's lifetime. More info:
https://examples.k8s.io/volumes/rbd/README.md
Represents a Rados Block Device mount that lasts the lifetime of a pod. RBD volumes support ownership management and SELinux relabeling.
rbd.image
image is the rados image name. More info:
https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
rbd.monitors
monitors is a collection of Ceph monitors. More info:
rbd.fsType
(string)
https://kubernetes.io/docs/concepts/storage/volumes#rbd
rbd.keyring
(string)
keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info:
rbd.pool
(string)
pool is the rados pool name. Default is rbd. More info:
rbd.readOnly
(boolean)
rbd.secretRef
(
)
secretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info:
rbd.user
(string)
user is the rados user name. Default is admin. More info:
scaleIO
(ScaleIOVolumeSource)
scaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes.
ScaleIOVolumeSource represents a persistent ScaleIO volume
scaleIO.gateway
gateway is the host address of the ScaleIO API Gateway.
scaleIO.secretRef
(
secretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail.
scaleIO.system
system is the name of the storage system as configured in ScaleIO.
scaleIO.fsType
(string)
fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Default is "xfs".
scaleIO.protectionDomain
(string)
protectionDomain is the name of the ScaleIO Protection Domain for the configured storage.
scaleIO.readOnly
(boolean)
scaleIO.sslEnabled
(boolean)
sslEnabled Flag enable/disable SSL communication with Gateway, default false
scaleIO.storageMode
(string)
storageMode indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is T

hinProvisioned.
scaleIO.storagePool
(string)
storagePool is the ScaleIO Storage Pool associated with the protection domain.
scaleIO.volumeName
(string)
volumeName is the name of a volume already created in the ScaleIO system that is associated with this volume source.
storageos
(StorageOSVolumeSource)
storageOS represents a StorageOS volume attached and mounted on Kubernetes nodes.
Represents a StorageOS persistent volume resource.
storageos.fsType
(string)
storageos.readOnly
(boolean)
storageos.secretRef
(
)
secretRef specifies the secret to use for obtaining the StorageOS API credentials. If not specified, default values will be attempted.
storageos.volumeName
(string)
volumeName is the human-readable name of the StorageOS volume. Volume names are only unique within a namespace.
storageos.volumeNamespace
(string)
volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.
vsphereVolume
(VsphereVirtualDiskVolumeSource)
vsphereVolume represents a vSphere volume attached and mounted on kubelets host machine
Represents a vSphere volume resource.
vsphereVolume.volumePath
volumePath is the path that identifies vSphere volume vmdk
vsphereVolume.fsType
(string)
fsType is filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified.
vsphereVolume.storagePolicyID
(string)
storagePolicyID is the storage Policy Based Management (SPBM) profile ID associated with the StoragePolicyName.
vsphereVolume.storagePolicyName
(string)
storagePolicyName is the storage Policy Based Management (SPBM) profile name.
Deprecated
gitRepo
(GitRepoVolumeSource)
gitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container.
*Represents a volume that is populated with the contents of a git repository. Git repo volumes do not support ownership management. Git repo volumes support SELinux relabeling.
DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container.*
gitRepo.repository
repository is the URL
gitRepo.directory
(string)
directory is the target directory name. Must not contain or start with '..'. If '.' is supplied, the volume directory will be the git repository. Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name.
gitRepo.revision
(string)
revision is the commit hash for the specified revision.
DownwardAPIVolumeFile represents information to create the file containing the pod field
path
Required: Path is the relative path name of the file to be created. Must not be absolute or contain the '..' path. Must be utf-8 encoded. The first item of the relative path must not start with '..'
fieldRef
(
)
Required: Selects a field of the pod: only annotations, labels, name and namespace are supported.
mode
(int32)
Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777

 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
resourceFieldRef
(
)
Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, requests.cpu and requests.memory) are currently supported.
KeyToPath
Maps a string key to a path within a volume.
key
key is the key to project.
path
path is the relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'.
mode
(int32)
mode is Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
Feedback
Yes
No
.
or
.
Last modified December 02, 2022 at 6:19 AM PST:
remove glusterfs references from the docs (#37697) (34c152a433)
Volume
KeyToPath
Home
Blog
Training
Partners
Community
CC BY 4.0

Indexed Job for Parallel Processing with Static Work Assignment | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API A

ggregation Layer
Tasks
Debug Pods
Debug a StatefulS

et
Auditing
Run Jobs
TLS
Tutorials
Security

y Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-api

server Configuration (v1)
Tasks
Run Jobs
In this example, you will run a Kubernetes Job that uses multiple parallel
worker processes.
Each worker is a different container running in its own Pod. The Pods have an
index number
that the control plane sets automatically, which allows each Pod
to identify which part of the overall task to work on.
The pod index is available in the
batch.kubernetes.io/job-completion-index
as a string representing its
decimal value. In order for the containerized task process to obtain this index,
you can publish the value of the annotation using the
downward API
mechanism.
For convenience, the control plane automatically sets the downward API to
expose the index in the
JOB_COMPLETION_INDEX
Here is an overview of the steps in this example:
Define a Job manifest using indexed completion
.
The downward API allows you to pass the pod index annotation as an
environment variable or file to the container.
Start an
Indexed
Job based on that manifest
.
You should already be familiar with the basic,
non-parallel, use of
Job
.
minikube
.
Choose an approach
To access the work item from the worker program, you have a few options:
Read the
environment variable. The Job
automatically links this variable to the annotation containing the completion
index.
Read a file that contains the completion index.
Assuming that you can't modify the program, you can wrap it with a script
that reads the index using any of the methods above and converts it into
something that the program can use as input.
For this example, imagine that you chose option 3 and you want to run the
rev
utility. This
program accepts a file as an argument and prints its content reversed.
rev data.txt
You'll use the
rev
tool from the
busybox
container image.
As this is only 

an example, each Pod only does a tiny piece of work (reversing a short
string). In a real workload you might, for example, create a Job that represents
the
task of producing 60 seconds of video based on scene data.
Each work item in the video rendering Job would be to render a particular
frame of that video clip. Indexed completion would mean that each Pod in
the Job knows which frame to render and publish, by counting frames from
the start of the clip.
Define an Indexed Job
Here is a sample Job manifest that uses
Indexed
completion mode:
application/job/indexed-job.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
'indexed-job'
spec
:
completions
:
5
parallelism
:
3
completionMode
:
Indexed
template
:
spec
:
:
Never
:
-
name
:
'input'
image
:
'docker.io/library/bash'
command
:
-
"bash"
-
"-c"
- |
items=(foo bar baz qux xyz)
echo ${items[$JOB_COMPLETION_INDEX]} &gt; /input/data.txt
:
-
mountPath
:
/input
name
:
input
:
-
name
:
'worker'
image
:
'docker.io/library/busybox'
command
:
-
"rev"
-
"/input/data.txt"
:
-
mountPath
:
/input
name
:
input
volumes
:
-
name
:
input
emptyDir
:
{}
In the example above, you use the builtin
variable set by the Job controller for all containers. An
init container
maps the index to a static value and writes it to a file that is shared with the
container running the worker through an
emptyDir volume
.
Optionally, you can
define your own environment variable through the downward
API
to publish the index to containers. You can also choose to load a list of values
from a
ConfigMap as an environment variable or file
.
Alternatively, you can directly
use the downward API to pass the annotation
value as a volume file
,
like shown in the following example:
application/job/indexed-job-vol.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
5
:
3
:
Indexed
template
:
spec
:
:
Never
:
-
name
:
'worker'
image
:
command
:
-
"rev"
-
:
-
mountPath
:
/input
name
:
input
volumes
:
-
name
:
input
:
items
:
-
path
:
"data.txt"
fieldRef
:
fieldPath
:
metadata.annotations['batch.kubernetes.io/job-completion-index']
Running the Job
Now run the Job:
# This uses the first approach (relying on $JOB_COMPLETION_INDEX)
kubectl apply -f https://kubernetes.io/examples/application/job/indexed-job.yaml
When you create this Job, the control plane creates a series of Pods, one for each index you specified. The value of
.spec.parallelism
determines how many can run at once whereas
.spec.completions
determines how many Pods the Job creates in total.
Because
is less than
, the control plane waits for some of the first Pods to complete before starting more of them.
You can wait for the Job to succeed, with a timeout:
# The check for condition name is case insensitive
kubectl
wait
--for
=
condition
=
complete
--timeout
=
300s job/indexed-job
Now, describe the Job and check that it was successful.
kubectl describe jobs/indexed-job
Name:              indexed-job
Namespace:         default
Selector:          controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756
Labels:            controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756
                   job-name=indexed-job
Annotations:       &lt;none&gt;
Parallelism:       3
Completions:       5
Start Time:        Thu, 11 Mar 2021 15:47:34 +0000
Pods Statuses:     2 Running / 3 Succeeded / 0 Failed
Completed Indexes: 0-2
Pod Template:
  Labels:  controller-uid=bf865e04-0b67-483b-9a90-74cfc4c3e756
           job-name=indexed-job
  Init Containers:
   input:
    Image:      docker.io/library/bash
    Port:       &lt;none&gt;
    Host Port:  &lt;none&gt;
    Command:
      bash
      -c
      items=(foo bar baz qux xyz)
      echo ${items[$JOB_COMPLETION_INDEX]} &gt; /inp

ut/data.txt

    Mounts:
      /input from input (rw)
   worker:
    Image:      docker.io/library/busybox
    Command:
      rev
      /input/data.txt
    Mounts:
  Volumes:
   input:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  &lt;unset&gt;
Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  4s    job-controller  Created pod: indexed-job-njkjj
  Normal  SuccessfulCreate  4s    job-controller  Created pod: indexed-job-9kd4h
  Normal  SuccessfulCreate  4s    job-controller  Created pod: indexed-job-qjwsz
  Normal  SuccessfulCreate  1s    job-controller  Created pod: indexed-job-fdhq5
  Normal  SuccessfulCreate  1s    job-controller  Created pod: indexed-job-ncslj
In this example, you run the Job with custom values for each index. You can
inspect the output of one of the pods:
kubectl logs indexed-job-fdhq5
# Change this to match the name of a Pod from that Job
xuq
Feedback
Yes
No
.
or
.
Last modified October 29, 2022 at 7:49 PM PST:
Use kubectl to wait for Jobs to succeed (5ed062fb1f)
Home
Blog
Training
Partners
Community
CC BY 4.0

CSINode | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Se

t Up kubectl on Windows
Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resources with AppArmo

r
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configuration (v1a

lpha1)
Reference
CSINode
CSINode
CSINode holds information about all CSI drivers installed on a node.
CSINode
CSINode holds information about all CSI drivers installed on a node. CSI drivers do not need to create the CSINode object directly. As long as they use the node-driver-registrar sidecar container, the kubelet will automatically populate the CSINode object for the CSI driver as part of kubelet plugin registration. CSINode has the same name as a node. If the object is missing, it means either there are no CSI Drivers available on the node, or the Kubelet version is low enough that it doesn't create this object. CSINode has an OwnerReference that points to the corresponding node object.
kind
: CSINode
metadata
(
)
metadata.name must be the Kubernetes node name.
spec
(
CSINodeSpec
spec is the specification of CSINode
CSINodeSpec holds information about the specification of all CSI drivers installed on a node
drivers
([]CSINodeDriver), required
name
drivers is a list of information of all CSI Drivers existing on a node. If all drivers in the list are uninstalled, this can become empty.
CSINodeDriver holds information about the specification of one CSI driver installed on a node
drivers.name
This is the name of the CSI driver that this object refers to. This MUST be the same name returned by the CSI GetPluginName() call for that driver.
drivers.nodeID
nodeID of the node from the driver point of view. This field enables Kubernetes to communicate with storage systems that do not share the same nomenclature for nodes. For example, Kubernetes may refer to a given node as "node1", but the storage system may refer to the same node as "nodeA". When Kubernetes issues a command to the storage system to attach a volume to a specific node, it can use this field to refer to the node name using the ID that the storage system will understand, e.g. "nodeA" instead of "node1". This field is required.
drivers.allocatable
(VolumeNodeResources)
allocatable represents the volume resources of a node that are available for scheduling. This field is beta.
VolumeNodeResources is a set of resource limits for scheduling of volumes.
drivers.allocatable.count
(int32)
Maximum number of unique volumes managed by the CSI driver that can be used on a node. A volume that is both attached and mounted on a node is considered to be used once, not twice. The same rule applies for a unique volume that is sh

ared among multiple pods on the same node. If this field is not specified, then the supported number of volumes on this node is unbounded.
drivers.topologyKeys
topologyKeys is the list of keys supported by the driver. When a driver is initialized on a cluster, it provides a set of topology keys that it understands (e.g. "company.com/zone", "company.com/region"). When a driver is initialized on a node, it provides the same topology keys along with values. Kubelet will expose these topology keys as labels on its own node object. When Kubernetes does topology aware provisioning, it can use this list to determine which labels it should retrieve from the node object and pass back to the driver. It is possible for different nodes to use different topology keys. This can be empty if driver does not support topology.
CSINodeList
CSINodeList is a collection of CSINode objects.
kind
: CSINodeList
metadata
(
ListMeta
)
items
([]
CSINode
items is the list of CSINode
get
read the specified CSINode
GET /apis/storage.k8s.io/v1/csinodes/{name}
name
(
in path
name of the CSINode
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
list
list or watch objects of kind CSINode
GET /apis/storage.k8s.io/v1/csinodes
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a CSINode
POST /apis/storage.k8s.io/v1/csinodes
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
202 (
CSINode
update
replace the specified CSINode
PUT /apis/storage.k8s.io/v1/csinodes/{name}
name
(
in path
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
patch
partially update the specified CSINode
PATCH /apis/storage.k8s.io/v1/csinodes/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
delete
delete a CSINode
DELETE /apis/storage.k8s.io/v1/csinodes/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
CSINode
): OK
202 (
CSINode
delete collection of CSINode
DELETE /apis/storage.k8s.io/v1/csinodes
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
gracePeriodSeco

nds
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
CSINode
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

LabelSelector | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
A label selector is a label query over a set of resources.
import "k8s.io/apimachinery/pkg/apis/meta/v1"
A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.
matchExpressions
([]LabelSelectorRequirement)
matchExpressions is a list of label selector requirements. The requirements are ANDed.
A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
matchExpressions.key
key
key is the label key that the selector applies to.
matchExpressions.operator
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
matchExpressions.values
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Managing Secrets | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Managing confidential settings data using Secrets.
Creating Secret objects using kubectl command line.
Creating Secret objects using resource configuration file.
Creating Secret objects using kustomization.yaml file.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Using Node Authorization | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

(v1)
Reference
Node authorization is a special-purpose authorization mode that specifically authorizes API requests made by kubelets.
Overview
The Node authorizer allows a kubelet to perform API operations. This includes:
Read operations:
services
endpoints
nodes
pods
secrets, configmaps, persistent volume claims and persistent volumes related to pods bound to the kubelet's node
Write operations:
nodes and node status (enable the
admission plugin to limit a kubelet to modify its own node)
pods and pod status (enable the
admission plugin to limit a kubelet to modify pods bound to itself)
events
Auth-related operations:
read/write access to the
CertificateSigningRequests API
for TLS bootstrapping
the ability to create TokenReviews and SubjectAccessReviews for delegated authentication/authorization checks
In future releases, the node authorizer may add or remove permissions to ensure kubelets
have the minimal set of permissions required to operate correctly.
In order to be authorized by the Node authorizer, kubelets must use a credential that identifies them as
being in the
group, with a username of
.
This group and user name format match the identity created for each kubelet as part of
kubelet TLS bootstrapping
.
The value of
&lt;nodeName&gt;
must
match precisely the name of the node as registered by the kubelet. By default, this is the host name as provided by
hostname
, or overridden via the
kubelet option
. However, when using the
kubelet option, the specific hostname may be determined by the cloud provider, ignoring the local
hostname
and the
option.
For specifics about how the kubelet determines the hostname, see the
kubelet options reference
.
To enable the Node authorizer, start the apiserver with
--authorization-mode=Node
.
To limit the API objects kubelets are able to write, enable the
admission plugin by starting the apiserver with
--enable-admission-plugins=...,NodeRestriction,...
Migration considerations
Kubelets outside the
group
group would not be authorized by the
Node
authorization mode,
and would need to continue to be authorized via whatever mechanism currently authorizes them.
The node admission plugin would not restrict requests from these kubelets.
Kubelets with undifferentiated usernames
In some deployments, kubelets have credentials that place them in the
group,
but do not ident

ify the particular node they are associated with,
because they do not have a username in the
system:node:...
format.
These kubelets would not be authorized by the
Node
The
admission plugin would ignore requests from these kubelets,
since the default node identifier implementation would not consider that a node identity.
Upgrades from previous versions using RBAC
Upgraded pre-1.7 clusters using
RBAC
will continue functioning as-is because the
group binding will already exist.
If a cluster admin wishes to start using the
Node
authorizer and
admission plugin
to limit node access to the API, that can be done non-disruptively:
Enable the
Node
authorization mode (
--authorization-mode=Node,RBAC
) and the
Ensure all kubelets' credentials conform to the group/username requirements
Audit apiserver logs to ensure the
Node
authorizer is not rejecting requests from kubelets (no persistent
NODE DENY
messages logged)
Delete the
system:node
cluster role binding
RBAC Node Permissions
In 1.6, the
cluster role was automatically bound to the
group when using the
RBAC Authorization mode
.
In 1.7, the automatic binding of the
group to the
role is deprecated
because the node authorizer accomplishes the same purpose with the benefit of additional restrictions
on secret and configmap access. If the
Node
and
RBAC
authorization modes are both enabled,
the automatic binding of the
role is not created in 1.7.
In 1.8, the binding will not be created at all.
When using RBAC, the
cluster role will continue to be created,
for compatibility with deployment methods that bind other users or groups to that role.
Feedback
Yes
No
.
or
.
Last modified April 23, 2022 at 2:32 PM PST:
Update references to the kubelet security files (a3ea9f4caf)
Overview
group
Home
Blog
Training
Partners
Community
CC BY 4.0

DaemonSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

Reference
DaemonSet
DaemonSet
DaemonSet represents the configuration of a daemon set.
DaemonSet
: apps/v1
kind
: DaemonSet
metadata
(
)
spec
(
DaemonSetSpec
)
The desired behavior of this daemon set. More info:
status
(
DaemonSetStatus
)
The current status of this daemon set. This data may be out of date by some window of time. Populated by the system. Read-only. More info:
DaemonSetSpec is the specification of a daemon set.
selector
(
A label query over pods that are managed by the daemon set. Must match in order to be controlled. It must match the pod template's labels. More info:
template
(
An object that describes the pod that will be created. The DaemonSet will create exactly one copy of this pod on every node that matches the template's node selector (or on every node if no node selector is specified). More info:
(int32)
The minimum number of seconds for which a newly created DaemonSet pod should be ready without any of its container crashing, for it to be considered available. Defaults to 0 (pod will be considered available as soon as it is ready).
(DaemonSetUpdateStrategy)
An update strategy to replace existing DaemonSet pods with new pods.
DaemonSetUpdateStrategy is a struct used to control the update strategy for a DaemonSet.
(string)
Type of daemon set update. Can be "RollingUpdate" or "OnDelete". Default is RollingUpdate.
(RollingUpdateDaemonSet)
Rolling update config params. Present only if type = "RollingUpdate".
Spec to control the desired behavior of daemon set rolling update.
updateStrategy.rollingUpdate.maxSurge
The maximum number of nodes with an existing available DaemonSet pod that can have an updated DaemonSet pod during during an update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavaila

ble is 0. Absolute number is calculated from percentage by rounding up to a minimum of 1. Default value is 0. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their a new pod created before the old pod is marked as deleted. The update starts by launching new pods on 30% of nodes. Once an updated pod is available (Ready for at least minReadySeconds) the old DaemonSet pod on that node is marked deleted. If the old pod becomes unavailable for any reason (Ready transitions to false, is evicted, or is drained) an updated pod is immediatedly created on that node without considering surge limits. Allowing surge implies the possibility that the resources consumed by the daemonset on any given node can double if the readiness check fails, and so resource intensive daemonsets should take into account that they may cause evictions during disruption.
The maximum number of DaemonSet pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of total number of DaemonSet pods at the start of the update (ex: 10%). Absolute number is calculated from percentage by rounding up. This cannot be 0 if MaxSurge is 0 Default value is 1. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their pods stopped for an update at any given time. The update starts by stopping at most 30% of those DaemonSet pods and then brings up new DaemonSet pods in their place. Once the new pods are available, it then proceeds onto other DaemonSet pods, thus ensuring that at least 70% of original number of DaemonSet pods are available at all times during the update.
(int32)
The number of old history to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10.
DaemonSetStatus represents the current status of a daemon set.
numberReady
numberReady is the number of nodes that should be running the daemon pod and have one or more of the daemon pod running with a Ready Condition.
numberAvailable
(int32)
The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and available (ready for at least spec.minReadySeconds)
numberUnavailable
(int32)
The number of nodes that should be running the daemon pod and have none of the daemon pod running and available (ready for at least spec.minReadySeconds)
numberMisscheduled
The number of nodes that are running the daemon pod, but are not supposed to run the daemon pod. More info:
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
desiredNumberScheduled
The total number of nodes that should be running the daemon pod (including nodes correctly running the daemon pod). More info:
currentNumberScheduled
The number of nodes that are running at least 1 daemon pod and are supposed to run the daemon pod. More info:
updatedNumberScheduled
(int32)
The total number of nodes that are running updated daemon pod
(int32)
Count of hash collisions for the DaemonSet. The DaemonSet controller uses t

his field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.
([]DaemonSetCondition)
type
Represents the latest available observations of a DaemonSet's current state.
DaemonSetCondition describes the state of a DaemonSet at a certain point.
Type of DaemonSet condition.
(Time)
(string)
(string)
(int64)
The most recent generation observed by the daemon set controller.
DaemonSetList
DaemonSetList is a collection of daemon sets.
: apps/v1
kind
: DaemonSetList
metadata
(
ListMeta
)
items
([]
DaemonSet
A list of daemon sets.
get
read the specified DaemonSet
GET /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}
name
(
in path
name of the DaemonSet
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
get
read status of the specified DaemonSet
GET /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
list
list or watch objects of kind DaemonSet
GET /apis/apps/v1/namespaces/{namespace}/daemonsets
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/apps/v1/daemonsets
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a DaemonSet
POST /apis/apps/v1/namespaces/{namespace}/daemonsets
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
202 (
DaemonSet
update
replace the specified DaemonSet
PUT /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
DaemonSet
, required
dr

yRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
update
replace status of the specified DaemonSet
PUT /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
partially update the specified DaemonSet
PATCH /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
partially update status of the specified DaemonSet
PATCH /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
delete
delete a DaemonSet
DELETE /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of DaemonSet
DELETE /apis/apps/v1/namespaces/{namespace}/daemonsets
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
DaemonSet
get
get
list
list
create

update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Service | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

lpha1)
Reference
Service
Service
Service is a named abstraction of software service (for example, mysql) consisting of local port (for example 3306) that the proxy listens on, and the selector that determines which pods will answer requests sent through the proxy.
Service
: v1
kind
: Service
metadata
(
)
spec
(
ServiceSpec
)
Spec defines the behavior of a service.
status
(
ServiceStatus
)
Most recently observed status of the service. Populated by the system. Read-only. More info:
ServiceSpec describes the attributes that a user creates on a service.
selector
Route service traffic to pods with label keys and values matching this selector. If empty or not present, the service is assumed to have an external process managing its endpoints, which Kubernetes will not modify. Only applies to types ClusterIP, NodePort, and LoadBalancer. Ignored if type is ExternalName. More info:
https://kubernetes.io/docs/concepts/services-networking/service/
ports
([]ServicePort)
port
Map: unique values on keys
port, protocol
will be kept during a merge
The list of ports that are exposed by this service. More info:
https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
ServicePort contains information on service's port.
The port that will be exposed by this service.
ports.targetPort
Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod's container ports. If this is not specified, the value of the 'port' field is used (an identity map). This field is ignored for services with clusterIP=None, and should be omitted or set equal to the 'port' field. More info:
https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
IntOrString is a type that can ho

ld an int32 or a string. When used in JSON or YAML marshalling and unmarshalling, it produces or consumes the inner type. This allows you to have, for example, a JSON field that can accept a name or number.
(string)
The IP protocol for this port. Supports "TCP", "UDP", and "SCTP". Default is TCP.
(string)
The name of this port within the service. This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. When considering the endpoints for a Service, this must match the 'name' field in the EndpointPort. Optional if only one ServicePort is defined on this service.
ports.nodePort
(int32)
The port on each node on which this service is exposed when type is NodePort or LoadBalancer. Usually assigned by the system. If a value is specified, in-range, and not in use it will be used, otherwise the operation will fail. If not specified, a port will be allocated if this Service requires one. If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type from NodePort to ClusterIP). More info:
https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
(string)
type
(string)
type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. "ClusterIP" allocates a cluster-internal IP address for load-balancing to endpoints. Endpoints are determined by the selector or if that is not specified, by manual construction of an Endpoints object or EndpointSlice objects. If clusterIP is "None", no virtual IP is allocated and the endpoints are published as a set of endpoints rather than a virtual IP. "NodePort" builds on ClusterIP and allocates a port on every node which routes to the same endpoints as the clusterIP. "LoadBalancer" builds on NodePort and creates an external load-balancer (if supported in the current cloud) which routes to the same endpoints as the clusterIP. "ExternalName" aliases this service to the specified externalName. Several other fields do not apply to ExternalName services. More info:
https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
ipFamilies
IPFamilies is a list of IP families (e.g. IPv4, IPv6) assigned to this service. This field is usually assigned automatically based on cluster configuration and the ipFamilyPolicy field. If this field is specified manually, the requested family is available in the cluster, and ipFamilyPolicy allows it, it will be used; otherwise creation of the service will fail. This field is conditionally mutable: it allows for adding or removing a secondary IP family, but it does not allow changing the primary IP family of the Service. Valid values are "IPv4" and "IPv6". This field only applies to Services of types ClusterIP, NodePort, and LoadBalancer, and does apply to "headless" services. This field will be wiped when updating a Service to type ExternalName.
This field may hold a maximum of two entries (dual-stack families, in either order). These families must correspond to the values of the clusterIPs field, if specified. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field.
ipFamilyPolicy
(string)
IPFamilyPolicy represents the dual-stack-ness requested or required by this Service. If there is no value provided, then this field will be set to SingleStack. Services can be "SingleStack" (a single IP family), "PreferDualStack" (two IP families on dual-stack configured clusters or a single IP fa

mily on single-stack clusters), or "RequireDualStack" (two IP families on dual-stack configured clusters, otherwise fail). The ipFamilies and clusterIPs fields depend on the value of this field. This field will be wiped when updating a service to type ExternalName.
clusterIP
(string)
clusterIP is the IP address of the service and is usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be blank) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above). Valid values are "None", empty string (""), or a valid IP address. Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. More info:
clusterIPs
ClusterIPs is a list of IP addresses assigned to this service, and are usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be empty) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above). Valid values are "None", empty string (""), or a valid IP address. Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. If this field is not specified, it will be initialized from the clusterIP field. If this field is specified, clients must ensure that clusterIPs[0] and clusterIP have the same value.
This field may hold a maximum of two entries (dual-stack IPs, in either order). These IPs must correspond to the values of the ipFamilies field. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field. More info:
externalIPs is a list of IP addresses for which nodes in the cluster will also accept traffic for this service. These IPs are not managed by Kubernetes. The user is responsible for ensuring that traffic arrives at a node with this IP. A common example is external load-balancers that are not part of the Kubernetes system.
sessionAffinity
(string)
Supports "ClientIP" and "None". Used to maintain session affinity. Enable client IP based session affinity. Must be ClientIP or None. Defaults to None. More info:
loadBalancerIP
(string)
Only applies to Service Type: LoadBalancer. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature. Deprecated: This field was under-specified and its meaning varies across implementations, and it cannot support dual-stack. As of Kuber

netes v1.24, users are encouraged to use implementation-specific annotations when available. This field may be removed in a future API version.
loadBalancerSourceRanges
If specified and supported by the platform, this will restrict traffic through the cloud-provider load-balancer will be restricted to the specified client IPs. This field will be ignored if the cloud-provider does not support the feature." More info:
https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/
loadBalancerClass
(string)
loadBalancerClass is the class of the load balancer implementation this Service belongs to. If specified, the value of this field must be a label-style identifier, with an optional prefix, e.g. "internal-vip" or "example.com/internal-vip". Unprefixed names are reserved for end-users. This field can only be set when the Service type is 'LoadBalancer'. If not set, the default load balancer implementation is used, today this is typically done through the cloud provider integration, but should apply for any default implementation. If set, it is assumed that a load balancer implementation is watching for Services with a matching class. Any default load balancer implementation (e.g. cloud providers) should ignore Services that set this field. This field can only be set when creating or updating a Service to type 'LoadBalancer'. Once set, it can not be changed. This field will be wiped when a service is updated to a non 'LoadBalancer' type.
externalName
(string)
externalName is the external reference that discovery mechanisms will return as an alias for this service (e.g. a DNS CNAME record). No proxying will be involved. Must be a lowercase RFC-1123 hostname (
https://tools.ietf.org/html/rfc1123
) and requires
type
to be "ExternalName".
externalTrafficPolicy
(string)
externalTrafficPolicy describes how nodes distribute service traffic they receive on one of the Service's "externally-facing" addresses (NodePorts, ExternalIPs, and LoadBalancer IPs). If set to "Local", the proxy will configure the service in a way that assumes that external load balancers will take care of balancing the service traffic between nodes, and so each node will deliver traffic only to the node-local endpoints of the service, without masquerading the client source IP. (Traffic mistakenly sent to a node with no endpoints will be dropped.) The default value, "Cluster", uses the standard behavior of routing to all endpoints evenly (possibly modified by topology and other features). Note that traffic sent to an External IP or LoadBalancer IP from within the cluster will always get "Cluster" semantics, but clients sending to a NodePort from within the cluster may need to take traffic policy into account when picking a node.
internalTrafficPolicy
(string)
InternalTrafficPolicy describes how nodes distribute service traffic they receive on the ClusterIP. If set to "Local", the proxy will assume that pods only want to talk to endpoints of the service on the same node as the pod, dropping the traffic if there are no local endpoints. The default value, "Cluster", uses the standard behavior of routing to all endpoints evenly (possibly modified by topology and other features).
healthCheckNodePort
(int32)
healthCheckNodePort specifies the healthcheck nodePort for the service. This only applies when type is set to LoadBalancer and externalTrafficPolicy is set to Local. If a value is specified, is in-range, and is not in use, it will be used. If not specified, a value will be automatically allocated. External systems (e.g. load-balancers) can use this port to determine if a given node holds endpoints for this service or not. If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type). This field cannot be updated once set.
publishNotReadyAddresses
(boolean)
publishNotReadyAddresses indicates that any agent which

 deals with endpoints for this Service should disregard any indications of ready/not-ready. The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery. The Kubernetes controllers that generate Endpoints and EndpointSlice resources for Services interpret this to mean that all endpoints are considered "ready" even if the Pods themselves are not. Agents which consume only Kubernetes generated endpoints through the Endpoints or EndpointSlice resources can safely assume this behavior.
sessionAffinityConfig
(SessionAffinityConfig)
sessionAffinityConfig contains the configurations of session affinity.
SessionAffinityConfig represents the configurations of session affinity.
sessionAffinityConfig.clientIP
(ClientIPConfig)
clientIP contains the configurations of Client IP based session affinity.
ClientIPConfig represents the configurations of Client IP based session affinity.
sessionAffinityConfig.clientIP.timeoutSeconds
(int32)
timeoutSeconds specifies the seconds of ClientIP type session sticky time. The value must be &gt;0 &amp;&amp; &lt;=86400(for 1 day) if ServiceAffinity == "ClientIP". Default value is 10800(for 3 hours).
allocateLoadBalancerNodePorts
(boolean)
allocateLoadBalancerNodePorts defines if NodePorts will be automatically allocated for services with type LoadBalancer. Default is "true". It may be set to "false" if the cluster load-balancer does not rely on NodePorts. If the caller requests specific NodePorts (by specifying a value), those requests will be respected, regardless of this field. This field may only be set for services with type LoadBalancer and will be cleared if the type is changed to any other type.
ServiceStatus represents the current status of a service.
([]Condition)
type
Current service state
Condition contains details for one aspect of the current state of this API Resource.
(Time), required
lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
message is a human readable message indicating details about the transition. This may be an empty string.
reason contains a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
status of the condition, one of True, False, Unknown.
type of condition in CamelCase or in foo.example.com/CamelCase.
conditions.observedGeneration
(int64)
observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
loadBalancer
(LoadBalancerStatus)
LoadBalancer contains the current status of the load-balancer, if one is present.
LoadBalancerStatus represents the status of a load-balancer.
loadBalancer.ingress
([]LoadBalancerIngress)
Ingress is a list containing ingress points for the load-balancer. Traffic intended for the service should be sent to these ingress points.
LoadBalancerIngress represents the status of a load-balancer ingress point: traffic intended for the service should be sent to an i

ngress point.
loadBalancer.ingress.hostname
(string)
Hostname is set for load-balancer ingress points that are DNS based (typically AWS load-balancers)
loadBalancer.ingress.ip
(string)
IP is set for load-balancer ingress points that are IP based (typically GCE or OpenStack load-balancers)
loadBalancer.ingress.ports
([]PortStatus)
Ports is a list of records of service ports If used, every port defined in the service should have an entry in it
**
loadBalancer.ingress.ports.port
Port is the port number of the service port of which status is recorded here
loadBalancer.ingress.ports.protocol
Protocol is the protocol of the service port of which status is recorded here The supported values are: "TCP", "UDP", "SCTP"
loadBalancer.ingress.ports.error
(string)
Error is to record the problem with the service port The format of the error shall comply with the following rules: - built-in error values shall be specified in this file and those shall use
CamelCase names
cloud provider specific error values must have names that comply with the
format foo.example.com/CamelCase.
ServiceList
ServiceList holds a list of services.
: v1
kind
: ServiceList
metadata
(
ListMeta
)
items
([]
Service
List of services
get
read the specified Service
GET /api/v1/namespaces/{namespace}/services/{name}
name
(
in path
name of the Service
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
get
read status of the specified Service
GET /api/v1/namespaces/{namespace}/services/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
list
list or watch objects of kind Service
GET /api/v1/namespaces/{namespace}/services
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/services
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a Service
POST /api/v1/namespaces/{namespace}/services
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
202 (
Service
update
replace the specified Service
PUT /api/v1/namespaces/{namespace}/services/{name}
name
(
in path
namespace
(
in path
namespace
bo

dy
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
update
replace status of the specified Service
PUT /api/v1/namespaces/{namespace}/services/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
patch
partially update the specified Service
PATCH /api/v1/namespaces/{namespace}/services/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
patch
partially update status of the specified Service
PATCH /api/v1/namespaces/{namespace}/services/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
delete
delete a Service
DELETE /api/v1/namespaces/{namespace}/services/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Service
): OK
202 (
Service
delete collection of Service
DELETE /api/v1/namespaces/{namespace}/services
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Last modified August 17, 2022 at 10:01 AM PST:
Update after rc.1 (d330f97f82)
Service
get
get
list
list
create
update
update
replace status of th

e specified Service
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Pods and Containers | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Perform common configuration tasks for Pods and containers.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
1 - Assign Memory Resources to Containers and Pods
This page shows how to assign a memory
request
and a memory
limit
to a
Container. A Container is guaranteed to have as much memory as it requests,
but is not allowed to use more memory than its limit.
minikube
.
Each node in your cluster must have at least 300 MiB of memory.
A few of the steps on this page require you to run the
service in your cluster. If you have the metrics-server
running, you can skip those steps.
If you are running Minikube, run the following command to enable the
metrics-server:
minikube addons
enable
To see whether the metrics-server is running, or another provider of the resource metrics
API (
), run the following command:
kubectl get apiservices
If the resource metrics API is available, the output includes a
reference to
.
NAME
v1beta1.metrics.k8s.io
kubectl create namespace mem-example
Specify a memory request and a memory limit
To specify a memory request for a Container, include the
resources:requests
field
in the Container's resource manifest. To specify a memory limit, include
resources:limits
.
In this exercise, you create a Pod that has one Container. The Container has a memory
request of 100 MiB and a memory limit of 200 MiB. Here's the configuration file
for the Pod:
pods/resource/memory-request-limit.yaml
:
v1
kind
:
Pod
metadata
:
name
:
memory-demo
namespace
:
mem-example
spec
:
:
-
name
:
memory-demo-ctr
image
:
polinux/stress
resources
:
requests
:
memory
:
"100Mi"
limits
:
memory
:
"200Mi"
command
:
[
"stre

ss"
]
args
:
[
"--vm"
,
"1"
,
"--vm-bytes"
,
"150M"
,
"--vm-hang"
,
"1"
]
The
args
section in the configuration file provides arguments for the Container when it starts.
The
"--vm-bytes", "150M"
arguments tell the Container to attempt to allocate 150 MiB of memory.
kubectl apply -f https://k8s.io/examples/pods/resource/memory-request-limit.yaml --namespace
=
Verify that the Pod Container is running:
kubectl get pod memory-demo --namespace
=
kubectl get pod memory-demo --output
=
=
The output shows that the one Container in the Pod has a memory request of 100 MiB
and a memory limit of 200 MiB.
...
resources
:
requests
:
memory
:
100Mi
limits
:
memory
:
200Mi
...
Run
to fetch the metrics for the pod:
kubectl top pod memory-demo --namespace
=
The output shows that the Pod is using about 162,900,000 bytes of memory, which
is about 150 MiB. This is greater than the Pod's 100 MiB request, but within the
Pod's 200 MiB limit.
NAME                        CPU(cores)   MEMORY(bytes)
memory-demo                 &lt;something&gt;  162856960
kubectl delete pod memory-demo --namespace
=
Exceed a Container's memory limit
A Container can exceed its memory request if the Node has memory available. But a Container
is not allowed to use more than its memory limit. If a Container allocates more memory than
its limit, the Container becomes a candidate for termination. If the Container continues to
consume memory beyond its limit, the Container is terminated. If a terminated Container can be
restarted, the kubelet restarts it, as with any other type of runtime failure.
In this exercise, you create a Pod that attempts to allocate more memory than its limit.
Here is the configuration file for a Pod that has one Container with a
memory request of 50 MiB and a memory limit of 100 MiB:
pods/resource/memory-request-limit-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
memory-demo-2
namespace
:
spec
:
:
-
name
:
memory-demo-2-ctr
image
:
resources
:
requests
:
memory
:
"50Mi"
limits
:
memory
:
"100Mi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"250M"
,
,
"1"
]
In the
args
section of the configuration file, you can see that the Container
will attempt to allocate 250 MiB of memory, which is well above the 100 MiB limit.
kubectl apply -f https://k8s.io/examples/pods/resource/memory-request-limit-2.yaml --namespace
=
kubectl get pod memory-demo-2 --namespace
=
At this point, the Container might be running or killed. Repeat the preceding command until the Container is killed:
NAME            READY     STATUS      RESTARTS   AGE
memory-demo-2   0/1       OOMKilled
1
24s
Get a more detailed view of the Container status:
kubectl get pod memory-demo-2 --output
=
=
The output shows that the Container was killed because it is out of memory (OOM):
lastState
:
:
:
65183c1877aaec2e8427bc95609cc52677a454b56fcb24340dbd22917c23b10f
exitCode
:
137
:
2017-06-20T20:52:19Z
reason
:
OOMKilled
startedAt
:
null
The Container in this exercise can be restarted, so the kubelet restarts it. Repeat
this command several times to see that the Container is repeatedly killed and restarted:
=
The output shows that the Container is killed, restarted, killed again, restarted again, and so on:
kubectl get pod memory-demo-2 --namespace=mem-example
memory-demo-2   0/1       OOMKilled   1          37s
NAME            READY     STATUS    RESTARTS   AGE
memory-demo-2   1/1       Running   2          40s
View detailed information about the Pod history:
kubectl describ

e pod memory-demo-2 --namespace=mem-example
The output shows that the Container starts and fails repeatedly:
... Normal  Created   Created container with id 66a3a20aa7980e61be4922780bf9d24d1a1d8b7395c09861225b0eba1b1f8511
... Warning BackOff   Back-off restarting failed container
View detailed information about your cluster's Nodes:
kubectl describe nodes
The output includes a record of the Container being killed because of an out-of-memory condition:
Warning OOMKilling Memory cgroup out of memory: Kill process 4481 (stress) score 1994 or sacrifice child
kubectl delete pod memory-demo-2 --namespace
=
Specify a memory request that is too big for your Nodes
Memory requests and limits are associated with Containers, but it is useful to think
of a Pod as having a memory request and limit. The memory request for the Pod is the
sum of the memory requests for all the Containers in the Pod. Likewise, the memory
limit for the Pod is the sum of the limits of all the Containers in the Pod.
Pod scheduling is based on requests. A Pod is scheduled to run on a Node only if the Node
has enough available memory to satisfy the Pod's memory request.
In this exercise, you create a Pod that has a memory request so big that it exceeds the
capacity of any Node in your cluster. Here is the configuration file for a Pod that has one
Container with a request for 1000 GiB of memory, which likely exceeds the capacity
of any Node in your cluster.
pods/resource/memory-request-limit-3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
memory-demo-3
namespace
:
spec
:
:
-
name
:
memory-demo-3-ctr
image
:
resources
:
requests
:
memory
:
"1000Gi"
limits
:
memory
:
"1000Gi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"150M"
,
,
"1"
]
kubectl apply -f https://k8s.io/examples/pods/resource/memory-request-limit-3.yaml --namespace
=
View the Pod status:
kubectl get pod memory-demo-3 --namespace
=
The output shows that the Pod status is PENDING. That is, the Pod is not scheduled to run on any Node, and it will remain in the PENDING state indefinitely:
kubectl get pod memory-demo-3 --namespace=mem-example
memory-demo-3   0/1       Pending   0          25s
View detailed information about the Pod, including events:
kubectl describe pod memory-demo-3 --namespace
=
The output shows that the Container cannot be scheduled because of insufficient memory on the Nodes:
Events:
  ...  Reason            Message
       ------            -------
  ...  FailedScheduling  No nodes are available that match all of the following predicates:: Insufficient memory (3).
Memory units
The memory resource is measured in bytes. You can express memory as a plain integer or a
fixed-point integer with one of these suffixes: E, P, T, G, M, K, Ei, Pi, Ti, Gi, Mi, Ki.
For example, the following represent approximately the same value:
128974848, 129e6, 129M, 123Mi
kubectl delete pod memory-demo-3 --namespace
=
If you do not specify a memory limit
If you do not specify a memory limit for a Container, one of the following situations applies:
The Container has no upper bound on the amount of memory it uses. The Container
could use all of the memory available on the Node where it is running which in turn could invoke the OOM Killer. Further, in case of an OOM Kill, a container with no resource limits will have a greater chance of being killed.
The Container is running in a namespace that has a default memory limit, and the
Container is automatically assigned the default limit. Cluster administrators can use a
to specify a default value for the memory limit.
Motivation for memory requests and limits
By configuring memory requests and limits for the Containers that run in your
cluster, you can make efficient use of the memory resources available on your cluster's
Nodes. B

y keeping a Pod's memory request low, you give the Pod a good chance of being
scheduled. By having a memory limit that is greater than the memory request, you accomplish two things:
The Pod can have bursts of activity where it makes use of memory that happens to be available.
The amount of memory a Pod can use during a burst is limited to some reasonable amount.
Clean up
Delete your namespace. This deletes all the Pods that you created for this task:
kubectl delete namespace mem-example
2 - Assign CPU Resources to Containers and Pods
This page shows how to assign a CPU
request
and a CPU
limit
to
a container. Containers cannot use more CPU than the configured limit.
Provided the system has CPU time free, a container is guaranteed to be
allocated as much CPU as it requests.
minikube
.
Your cluster must have at least 1 CPU available for use to run the task examples.
If you are running
Minikube
, run the
following command to enable metrics-server:
enable
To see whether metrics-server (or another provider of the resource metrics
API,
) is running, type the following command:
If the resource metrics API is available, the output will include a
.
NAME
Create a
Namespace
so that the resources you
create in this exercise are isolated from the rest of your cluster.
kubectl create namespace cpu-example
Specify a CPU request and a CPU limit
To specify a CPU request for a container, include the
field
in the Container resource manifest. To specify a CPU limit, include
.
In this exercise, you create a Pod that has one container. The container has a request
of 0.5 CPU and a limit of 1 CPU. Here is the configuration file for the Pod:
pods/resource/cpu-request-limit.yaml
:
v1
kind
:
Pod
metadata
:
name
:
cpu-demo
namespace
:
cpu-example
spec
:
:
-
name
:
cpu-demo-ctr
image
:
resources
:
limits
:
cpu
:
"1"
requests
:
cpu
:
"0.5"
args
:
- -cpus
-
"2"
The
args
section of the configuration file provides arguments for the container when it starts.
The
-cpus "2"
argument tells the Container to attempt to use 2 CPUs.
kubectl apply -f https://k8s.io/examples/pods/resource/cpu-request-limit.yaml --namespace
=
Verify that the Pod is running:
kubectl get pod cpu-demo --namespace
=
kubectl get pod cpu-demo --output
=
=
The output shows that the one container in the Pod has a CPU request of 500 milliCPU
and a CPU limit of 1 CPU.
resources
:
limits
:
cpu
:
"1"
requests
:
cpu
:
500m
Use
to fetch the metrics for the Pod:
kubectl top pod cpu-demo --namespace
=
This example output shows that the Pod is using 974 milliCPU, wh

ich is
slightly less than the limit of 1 CPU specified in the Pod configuration.
cpu-demo                    974m         &lt;something&gt;
Recall that by setting
-cpu "2"
, you configured the Container to attempt to use 2 CPUs, but the Container is only being allowed to use about 1 CPU. The container's CPU use is being throttled, because the container is attempting to use more CPU resources than its limit.
Note:
Another possible explanation for the CPU use being below 1.0 is that the Node might not have
enough CPU resources available. Recall that the prerequisites for this exercise require your cluster to have at least 1 CPU available for use. If your Container runs on a Node that has only 1 CPU, the Container cannot use more than 1 CPU regardless of the CPU limit specified for the Container.
CPU units
The CPU resource is measured in
CPU
units. One CPU, in Kubernetes, is equivalent to:
1 AWS vCPU
1 GCP Core
1 Azure vCore
1 Hyperthread on a bare-metal Intel processor with Hyperthreading
Fractional values are allowed. A Container that requests 0.5 CPU is guaranteed half as much
CPU as a Container that requests 1 CPU. You can use the suffix m to mean milli. For example
100m CPU, 100 milliCPU, and 0.1 CPU are all the same. Precision finer than 1m is not allowed.
CPU is always requested as an absolute quantity, never as a relative quantity; 0.1 is the same
amount of CPU on a single-core, dual-core, or 48-core machine.
kubectl delete pod cpu-demo --namespace
=
Specify a CPU request that is too big for your Nodes
CPU requests and limits are associated with Containers, but it is useful to think
of a Pod as having a CPU request and limit. The CPU request for a Pod is the sum
of the CPU requests for all the Containers in the Pod. Likewise, the CPU limit for
a Pod is the sum of the CPU limits for all the Containers in the Pod.
Pod scheduling is based on requests. A Pod is scheduled to run on a Node only if
the Node has enough CPU resources available to satisfy the Pod CPU request.
In this exercise, you create a Pod that has a CPU request so big that it exceeds
the capacity of any Node in your cluster. Here is the configuration file for a Pod
that has one Container. The Container requests 100 CPU, which is likely to exceed the
capacity of any Node in your cluster.
pods/resource/cpu-request-limit-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
cpu-demo-2
namespace
:
spec
:
:
-
name
:
cpu-demo-ctr-2
image
:
resources
:
limits
:
cpu
:
"100"
requests
:
cpu
:
"100"
args
:
- -cpus
-
"2"
kubectl apply -f https://k8s.io/examples/pods/resource/cpu-request-limit-2.yaml --namespace
=
kubectl get pod cpu-demo-2 --namespace
=
The output shows that the Pod status is Pending. That is, the Pod has not been
scheduled to run on any Node, and it will remain in the Pending state indefinitely:
NAME         READY     STATUS    RESTARTS   AGE
cpu-demo-2   0/1       Pending   0          7m
kubectl describe pod cpu-demo-2 --namespace
=
The output shows that the Container cannot be scheduled because of insufficient
CPU resources on the Nodes:
Events:
  Reason                        Message
  ------                        -------
  FailedScheduling      No nodes are available that match all of the following predicates:: Insufficient cpu (3).
kubectl delete pod cpu-demo-2 --namespace
=
If you do not specify a CPU limit
If you do not specify a CPU limit for a Container, then one of these situations applies:
The Container has no upper bound on the CPU resources it can use. The Container
could use all of the CPU resources available on the Node where it is running.
The Container is running in a namespace that has a default CPU limit, and the
Container is automatically assigned the default l

imit. Cluster administrators can use a
to specify a default value for the CPU limit.
If you specify a CPU limit but do not specify a CPU request
If you specify a CPU limit for a Container but do not specify a CPU request, Kubernetes automatically
assigns a CPU request that matches the limit. Similarly, if a Container specifies its own memory limit,
but does not specify a memory request, Kubernetes automatically assigns a memory request that matches
the limit.
Motivation for CPU requests and limits
By configuring the CPU requests and limits of the Containers that run in your
cluster, you can make efficient use of the CPU resources available on your cluster
Nodes. By keeping a Pod CPU request low, you give the Pod a good chance of being
scheduled. By having a CPU limit that is greater than the CPU request, you accomplish two things:
The Pod can have bursts of activity where it makes use of CPU resources that happen to be available.
The amount of CPU resources a Pod can use during a burst is limited to some reasonable amount.
Clean up
kubectl delete namespace cpu-example
3 - Configure GMSA for Windows Pods and containers
kubectl
A
YAML
A
script
option to allow you to revie

w the changes that would be made to your cluster.
The
used by the script may also be used to deploy the webhooks and associated objects manually (with appropriate substitutions for the parameters)
.
module
:
Use
,
kind
,
metadata
and
credspec
:
:
kind
:
metadata
:
name
:
credspec
:
:
:
-
Name
:
WebApp1
Scope
:
CONTOSO
-
Name
:
WebApp1
Scope
:
:
:
DnsName
:
:
Guid
:
#GUID
:
WebApp1
:
CONTOSO
Sid
:
use
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"use"
]
:
[
]
to use
:
kind
:
metadata
:
name
:
name

space
:
default
subjects
:
-
kind
:
name
:
default
namespace
:
default
roleRef
:
kind
:
name
:
apiGroup
:
:
:
apps/v1
kind
:
metadata
:
labels
:
run
:
name
:
namespace
:
default
spec
:
replicas
:
1
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
:
:
:
-
image
:
:
Always
name
:
iis
:
:
windows
:
apps/v1
kind
:
metadata
:
labels
:
run
:
name
:
namespace
:
default
spec
:
replicas
:
1
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
-
image
:
:
Always
name
:
iis
:
:
:
:
:
windows
use
reg add
here
exec
command.
iis-auth
exec
command.
I_NetLogonControl failed: Status = 1722 0x6ba

 RPC_S_SERVER_UNAVAILABLE
This tells us that for some reason, the Pod was unable to logon to the domain using the account specified in the credspec. You can try to repair the secure channel by running the following:
:
image
:
lifecycle
:
postStart
:
exec
:
command
:
[
,
,
]
:
lifecycle
netlogon
4 - Configure RunAsUserName for Windows pods and containers
This page shows how to use the
runAsUserName
setting for Pods and containers that will run on Windows nodes. This is roughly equivalent of the Linux-specific
runAsUser
setting, allowing you to run applications in a container as a different username than the default.
You need to have a Kubernetes cluster and the kubectl command-line tool must be configured to communicate with your cluster. The cluster is expected to have Windows worker nodes where pods with containers running Windows workloads will get scheduled.
Set the Username for a Pod
To specify the username with which to execute the Pod's container processes, include the
field (
PodSecurityContext
) in the Pod specification, and within it, the
(
WindowsSecurityContextOptions
) field containing the
field.
The Windows security context options that you specify for a Pod apply to all Containers and init Containers in the Pod.
Here is a configuration file for a Windows Pod that has the
field set:
windows/run-as-username-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
run-as-username-pod-demo
spec
:
:
:
:
"ContainerUser"
:
-
name
:
run-as-username-demo
image
:
mcr.microsoft.com/windows/servercore:ltsc2019
command
:
[
"ping"
,
"-t"
,
]
:
:
windows
kubectl apply -f https://k8s.io/examples/windows/run-as-username-pod.yaml
Verify that the Pod's Container is running:
kubectl get pod run-as-username-pod-demo
Get a shell to the running Container:
kubectl
exec
-it run-as-username-pod-demo -- powershell
Check that the shell is running user the correct username:
echo
$env:USERNAME
The output should be:
ContainerUser
Set the Username for a Container
To specify the username with which to execute a Container's processes, include the
field (
) in the Container manifest, and within it, the
(
field.
The Windows security context options that you specify for a Container apply only to that individual Container, and they override the settings made at the Pod level.
Here is the configuration file for a Pod that has one Container, and the
field is set at the Pod level and the Container level:
windows/run-as-username-container.yaml
:
v1
kind
:
Pod
metadata
:
name
:
run-as-username-container-demo
spec
:
:
:
:
:
-
name
:
image
:
mcr.microsoft.com/windows/servercore:ltsc

2019
command
:
[
"ping"
,
"-t"
,
]
:
:
:
"ContainerAdministrator"
:
:
windows
kubectl apply -f https://k8s.io/examples/windows/run-as-username-container.yaml
kubectl get pod run-as-username-container-demo
kubectl
exec
-it run-as-username-container-demo -- powershell
Check that the shell is running user the correct username (the one set at the Container level):
echo
ContainerAdministrator
Windows Username limitations
In order to use this feature, the value set in the
field must be a valid username. It must have the following format:
DOMAIN\USER
, where
DOMAIN\
is optional. Windows user names are case insensitive. Additionally, there are some restrictions regarding the
DOMAIN
and
USER
:
The
field cannot be empty, and it cannot contain control characters (ASCII values:
0x00-0x1F
,
0x7F
)
The
DOMAIN
must be either a NetBios name, or a DNS name, each with their own restrictions:
NetBios names: maximum 15 characters, cannot start with
.
(dot), and cannot contain the following characters:
\ / : * ? " &lt; &gt; |
DNS names: maximum 255 characters, contains only alphanumeric characters, dots, and dashes, and it cannot start or end with a
.
(dot) or
-
(dash).
The
USER
must have at most 20 characters, it cannot contain
only
dots or spaces, and it cannot contain the following characters:
" / \ [ ] : ; | = , + * ? &lt; &gt; @
.
Examples of acceptable values for the
field:
,
,
NT AUTHORITY\NETWORK SERVICE
,
NT AUTHORITY\LOCAL SERVICE
.
For more information about these limtations, check
here
and
here
.
Managing Workload Identity with Group Managed Service Accounts (GMSA)
Configure GMSA for Windows pods and containers
5 - Create a Windows HostProcess Pod
Windows HostProcess containers enable you to run containerized
workloads on a Windows host. These containers operate as
normal processes but have access to the host network namespace,
storage, and devices when given the appropriate user privileges.
HostProcess containers can be used to deploy network plugins,
storage configurations, device plugins, kube-proxy, and other
components to Windows nodes without the need for dedicated proxies or
the direct installation of host services.
Administrative tasks such as installation of security patches, event
log collection, and more can be performed without requiring cluster operators to
log onto each Windows node. HostProcess containers can run as any user that is
available on the host or is in the domain of the host machine, allowing administrators
to restrict resource access through user permissions. While neither filesystem or process
isolation are supported, a new volume is created on the host upon starting the container
to give it a clean and consolidated workspace. HostProcess containers can also be built on
top of existing Windows base images and do not inherit the same
compatibility requirements
as Windows server containers, meaning that the version of the base images does not need
to match that of the host. It is, however, recommended that you use the same base image
version as your Windows Server container workloads to ensure you do not have any unused
images taking up space on the node. HostProcess containers also support
volume mounts
within the container volume.
When should I use a Windows HostProcess container?
When you need to perform tasks which require the networking namespace of the host.
HostProcess containers have access to the host's network interfaces and IP addresses.
You need access to resources on the host such as the filesystem, event logs, etc.
Installation of specific device drivers or Windows services.
Consolidation of admin

istrative tasks and security policies. This reduces the degree of
privileges needed by Windows nodes.
This task guide is specific to Kubernetes v1.26.
If you are not running Kubernetes v1.26, check the documentation for
that version of Kubernetes.
In Kubernetes 1.26, the HostProcess container feature is enabled by default. The kubelet will
communicate with containerd directly by passing the hostprocess flag via CRI. You can use the
latest version of containerd (v1.6+) to run HostProcess containers.
How to install containerd.
These limitations are relevant for Kubernetes v1.26:
HostProcess containers require containerd 1.6 or higher
and
containerd 1.7 is recommended.
HostProcess pods can only contain HostProcess containers. This is a current limitation
of the Windows OS; non-privileged Windows containers cannot share a vNIC with the host IP namespace.
HostProcess containers run as a process on the host and do not have any degree of
isolation other than resource constraints imposed on the HostProcess user account. Neither
filesystem or Hyper-V isolation are supported for HostProcess containers.
Volume mounts are supported and are mounted under the container volume. See
Volume Mounts
A limited set of host user accounts are available for HostProcess containers by default.
See
Choosing a User Account
.
Resource limits (disk, memory, cpu count) are supported in the same fashion as processes
on the host.
Both Named pipe mounts and Unix domain sockets are
not
supported and should instead
be accessed via their path on the host (e.g. \\.\pipe\*)
HostProcess Pod configuration requirements
Enabling a Windows HostProcess pod requires setting the right configurations in the pod security
configuration. Of the policies defined in the
HostProcess pods are disallowed by the baseline and restricted policies. It is therefore recommended
that HostProcess pods run in alignment with the privileged profile.
When running under the privileged policy, here are
the configurations which need to be set to enable the creation of a HostProcess pod:
Privileged policy specification
Control
Policy
securityContext.windowsOptions.hostProcess
Windows pods offer the ability to run
HostProcess containers
which enables privileged access to the Windows node.
Allowed Values
true
Pods container HostProcess containers must use the host's network namespace.
true
securityContext.windowsOptions.runAsUserName
Specification of which user the HostProcess container should run as is required for the pod spec.
NT AUTHORITY\SYSTEM
NT AUTHORITY\Local service
NT AUTHORITY\NetworkService
Local usergroup names (see below)
runAsNonRoot
Because HostProcess containers have privileged access to the host, the
field cannot be set to true.
Undefined/Nil
false
Example manifest (excerpt)
spec
:
:
:
hostProcess
:
true
:
"NT AUTHORITY\\Local service"
:
true
:
-
name
:
test
image
:
image1:latest
command
:
- ping
- -t
-
127.0.0.1
:
"kubernetes.io/os":
windows
Volume mounts
HostProcess containers support the ability to mount volumes within the container volume space.
Volume mount behavior differs depending on the version of containerd runtime used by on the node.
Containerd v1.6
Applications running inside the container can access volume mounts directly via relative or
absolute paths. An environment variable
$CONTAINER_SANDBOX_MOUNT_POINT
is set upon container
creation and provides the absolute host path to the container volume. Relative paths are based
upon the
.spec.containers.volumeMounts.mountPath
To access service account tokens (for example) the following path structures are supported within the container:
.\var\run\secrets\kubernetes.io\serviceaccount\
$CONTAINER_SANDBOX_MOUNT_POINT\var\run\secrets\kubernetes.io\serviceaccount\
Containerd v1.7 (and greater)
Ap

plications running inside the container can access volume mounts directly via the volumeMount's
specified
mountPath
(just like Linux and non-HostProcess Windows containers).
For backwards compatibility volumes can also be accessed via using the same relative paths configured
by containerd v1.6.
As an example, to access service account tokens within the container you would use one of the following paths:
c:\var\run\secrets\kubernetes.io\serviceaccount
/var/run/secrets/kubernetes.io/serviceaccount/
Resource limits
Resource limits (disk, memory, cpu count) are applied to the job and are job wide.
For example, with a limit of 10MB set, the memory allocated for any HostProcess job object
will be capped at 10MB. This is the same behavior as other Windows container types.
These limits would be specified the same way they are currently for whatever orchestrator
or runtime is being used. The only difference is in the disk resource usage calculation
used for resource tracking due to the difference in how HostProcess containers are bootstrapped.
Choosing a user account
System accounts
By default, HostProcess containers support the ability to run as one of three supported Windows service accounts:
LocalSystem
LocalService
NetworkService
You should select an appropriate Windows service account for each HostProcess
container, aiming to limit the degree of privileges so as to avoid accidental (or even
malicious) damage to the host. The LocalSystem service account has the highest level
of privilege of the three and should be used only if absolutely necessary. Where possible,
use the LocalService service account as it is the least privileged of the three options.
Local accounts
If configured, HostProcess containers can also run as local user accounts which allows for node operators to give
fine-grained access to workloads.
To run HostProcess containers as a local user; A local usergroup must first be created on the node
and the name of that local usergroup must be specified in the
field in the deployment.
Prior to initializing the HostProcess container, a new
ephemeral
local user account to be created and joined to the specified usergroup, from which the container is run.
This provides a number a benefits including eliminating the need to manage passwords for local user accounts.
An initial HostProcess container running as a service account can be used to
prepare the user groups for later HostProcess containers.
Note:
Running HostProcess containers as local user accounts requires containerd v1.7+
Example:
Create a local user group on the node (this can be done in another HostProcess container).
net localgroup hpc-localgroup /add
Grant access to desired resources on the node to the local usergroup.
This can be done with tools like
icacls
.
Set
to the name of the local usergroup for the pod or individual containers.
:
:
:
true
:
hpc-localgroup
Schedule the pod!
Base Image for HostProcess Containers
HostProcess containers can be built from any of the existing
Windows Container base images
.
Additionally a new base mage has been created just for HostProcess containers!
For more information please check out the
windows-host-process-containers-base-image github project
.
Troubleshooting HostProcess containers
HostProcess containers fail to start with
failed to create user process token: failed to logon user: Access is denied.: unknown
Ensure containerd is running as
or
service accounts. User accounts (even Administrator accounts) do not have permissions to create logon tokens for any of the supported
user accounts
.
6 - Configure Quality of Service for Pods
This page shows how to configure Pods so that they will be assigned particular
Quality of Service (QoS) classes. Kubernetes uses QoS classes to make decisions about
scheduling and evicting Pods.

minikube
.
QoS classes
When Kubernetes creates a Pod it assigns one of these QoS classes to the Pod:
Burstable
kubectl create namespace qos-example
Create a Pod that gets assigned a QoS class of Guaranteed
For a Pod to be given a QoS class of Guaranteed:
Every Container in the Pod must have a memory limit and a memory request.
For every Container in the Pod, the memory limit must equal the memory request.
Every Container in the Pod must have a CPU limit and a CPU request.
For every Container in the Pod, the CPU limit must equal the CPU request.
These restrictions apply to init containers and app containers equally.
Here is the configuration file for a Pod that has one Container. The Container has a memory limit and a
memory request, both equal to 200 MiB. The Container has a CPU limit and a CPU request, both equal to 700 milliCPU:
pods/qos/qos-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
qos-demo
namespace
:
qos-example
spec
:
:
-
name
:
qos-demo-ctr
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"700m"
requests
:
memory
:
"200Mi"
cpu
:
"700m"
kubectl apply -f https://k8s.io/examples/pods/qos/qos-pod.yaml --namespace
=
kubectl get pod qos-demo --namespace
=
qos-example --output
=
yaml
The output shows that Kubernetes gave the Pod a QoS class of Guaranteed. The output also
verifies that the Pod Container has a memory request that matches its memory limit, and it has
a CPU request that matches its CPU limit.
spec
:
:
...
resources
:
limits
:
cpu
:
700m
memory
:
200Mi
requests
:
cpu
:
700m
memory
:
200Mi
...
status
:
qosClass
:
Note:
If a Container specifies its own memory limit, but does not specify a memory request, Kubernetes
automatically assigns a memory request that matches the limit. Similarly, if a Container specifies its own
CPU limit, but does not specify a CPU request, Kubernetes automatically assigns a CPU request that matches
the limit.
kubectl delete pod qos-demo --namespace
=
Create a Pod that gets assigned a QoS class of Burstable
A Pod is given a QoS class of Burstable if:
The Pod does not meet the criteria for QoS class Guaranteed.
At least one Container in the Pod has a memory or CPU request or limit.
Here is the configuration file for a Pod that has one Container. The Container has a memory limit of 200 MiB
and a memory request of 100 MiB.
pods/qos/qos-pod-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
qos-demo-2
namespace
:
spec
:
:
-
name
:
qos-demo-2-ctr
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
kubectl apply -f https://k8s.io/examples/pods/qos/qos-pod-2.yaml --namespace
=
kubectl get pod qos-demo-2 --namespace
=
=
yaml
The output shows that Kubernetes gave the Pod a QoS class of Burstable.
spec
:
:
-
image
:
nginx
:
Always
name
:
resources
:
limits
:
memory
:
200Mi
requests
:
memory
:
100Mi
...
status
:
qosClass
:
Burstable
kubectl delete pod qos-demo-2 --namespace
=
Create a Pod that gets assigned a QoS class of BestEffort
For a Pod to be given a QoS class of BestEffort, the Containers in the Pod must not
have any memory or CPU limits or requests.
Here

 is the configuration file for a Pod that has one Container. The Container has no memory or CPU
limits or requests:
pods/qos/qos-pod-3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
qos-demo-3
namespace
:
spec
:
:
-
name
:
qos-demo-3-ctr
image
:
nginx
kubectl apply -f https://k8s.io/examples/pods/qos/qos-pod-3.yaml --namespace
=
kubectl get pod qos-demo-3 --namespace
=
=
yaml
The output shows that Kubernetes gave the Pod a QoS class of BestEffort.
spec
:
:
...
resources
:
{}
...
status
:
qosClass
:
kubectl delete pod qos-demo-3 --namespace
=
Create a Pod that has two Containers
Here is the configuration file for a Pod that has two Containers. One container specifies a memory
request of 200 MiB. The other Container does not specify any requests or limits.
pods/qos/qos-pod-4.yaml
:
v1
kind
:
Pod
metadata
:
name
:
qos-demo-4
namespace
:
spec
:
:
-
name
:
qos-demo-4-ctr-1
image
:
nginx
resources
:
requests
:
memory
:
"200Mi"
-
name
:
qos-demo-4-ctr-2
image
:
redis
Notice that this Pod meets the criteria for QoS class Burstable. That is, it does not meet the
criteria for QoS class Guaranteed, and one of its Containers has a memory request.
kubectl apply -f https://k8s.io/examples/pods/qos/qos-pod-4.yaml --namespace
=
kubectl get pod qos-demo-4 --namespace
=
=
yaml
The output shows that Kubernetes gave the Pod a QoS class of Burstable:
spec
:
:
...
name
:
resources
:
requests
:
memory
:
200Mi
...
name
:
resources
:
{}
...
status
:
qosClass
:
Burstable
kubectl delete pod qos-demo-4 --namespace
=
Clean up
kubectl delete namespace qos-example
Control Topology Management policies on a node
7 - Assign Extended Resources to a Container
This page shows how to assign extended resources to a Container.
minikube
.
Before you do this exercise, do the exercise in
.
That will configure one of your Nodes to advertise a dongle resource.
Assign an extended resource to a Pod
To request an extended resource, include the
field in your
Container manifest. Extended resources are fully qualified with any domain outside of
*.kubernetes.io/
. Valid extended resource names have the form
example.com/foo
where
is replaced with your organization's domain and
foo
is a
descriptive resource name.
Here is the configuration file for a Pod that has one Container:
pods/resource/extended-resource-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
extended-resource-demo
spec
:
:
-
name
:
extended-resource-demo-ctr
image
:
nginx
resources
:
requests
:
:
3
limits
:
:
3
In the c

onfiguration file, you can see that the Container requests 3 dongles.
Create a Pod:
kubectl apply -f https://k8s.io/examples/pods/resource/extended-resource-pod.yaml
kubectl get pod extended-resource-demo
Describe the Pod:
kubectl describe pod extended-resource-demo
The output shows dongle requests:
Limits
:
:
3
Requests
:
:
3
Here is the configuration file for a Pod that has one Container. The Container requests
two dongles.
pods/resource/extended-resource-pod-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
extended-resource-demo-2
spec
:
:
-
name
:
extended-resource-demo-2-ctr
image
:
nginx
resources
:
requests
:
:
2
limits
:
:
2
Kubernetes will not be able to satisfy the request for two dongles, because the first Pod
used three of the four available dongles.
Attempt to create a Pod:
kubectl apply -f https://k8s.io/examples/pods/resource/extended-resource-pod-2.yaml
Describe the Pod
kubectl describe pod extended-resource-demo-2
The output shows that the Pod cannot be scheduled, because there is no Node that has
2 dongles available:
  Type    Status
  PodScheduled  False
...
Events:
  ...
  ... Warning   FailedScheduling  pod (extended-resource-demo-2) failed to fit in any node
fit failure summary on nodes : Insufficient example.com/dongle (1)
kubectl get pod extended-resource-demo-2
The output shows that the Pod was created, but not scheduled to run on a Node.
It has a status of Pending:
extended-resource-demo-2   0/1       Pending   0          6m
Clean up
Delete the Pods that you created for this exercise:
kubectl delete pod extended-resource-demo
kubectl delete pod extended-resource-demo-2
8 - Configure a Pod to Use a Volume for Storage
This page shows how to configure a Pod to use a Volume for storage.
A Container's file system lives only as long as the Container does. So when a
Container terminates and restarts, filesystem changes are lost. For more
consistent storage that is independent of the Container, you can use a
Volume
. This is especially important for stateful
applications, such as key-value stores (such as Redis) and databases.
minikube
.
Configure a volume for a Pod
In this exercise, you create a Pod that runs one Container. This Pod has a
Volume of type
emptyDir
that lasts for the life of the Pod, even if the Container terminates and
restarts. Here is the configuration file for the Pod:
pods/storage/redis.yaml
:
v1
kind
:
Pod
metadata
:
name
:
redis
spec
:
:
-
name
:
redis
image
:
redis
:
-
name
:
redis-storage
mountPath
:
/data/redis
volumes
:
-
name
:
emptyDir
:
{}
kubectl apply -f https://k8s.io/examples/pods/storage/redis.yaml
Verify that the Pod's Container is running, and then watch for changes to
the Pod:
kubectl get pod redis --watch
The output looks like this:
redis     1/1       Running   0          13s
In another terminal, get a shell to the running Container:
kubectl
exec
-it redis -- /bin/bash
In your shell, go to
, and then create a file:
root@redis:/data#
cd
/data/redis/
root@redis:/data/r

edis#
echo
Hello &gt; test-file
In your shell, list the running processes:
root@redis:/data/redis# apt-get update
root@redis:/data/redis# apt-get install procps
root@redis:/data/redis# ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
redis        1  0.1  0.1  33308  3828 ?        Ssl  00:46   0:00 redis-server *:6379
root        12  0.0  0.0  20228  3020 ?        Ss   00:47   0:00 /bin/bash
root        15  0.0  0.0  17500  2072 ?        R+   00:48   0:00 ps aux
In your shell, kill the Redis process:
root@redis:/data/redis#
kill
&lt;pid&gt;
where
is the Redis process ID (PID).
In your original terminal, watch for changes to the Redis Pod. Eventually,
you will see something like this:
NAME      READY     STATUS     RESTARTS   AGE
redis     1/1       Running    0          13s
redis     0/1       Completed  0         6m
redis     1/1       Running    1         6m
At this point, the Container has terminated and restarted. This is because the
Redis Pod has a
of
Always
.
Get a shell into the restarted Container:
kubectl
exec
, and verify that
test-file
is still there.
cd
root@redis:/data/redis# ls
test-file
Delete the Pod that you created for this exercise:
See
Volume
.
See
Pod
.
In addition to the local disk storage provided by
emptyDir
, Kubernetes
supports many different network-attached storage solutions, including PD on
GCE and EBS on EC2, which are preferred for critical data and will handle
details such as mounting and unmounting the devices on the nodes. See
Volumes
9 - Configure a Pod to Use a PersistentVolume for Storage
This page shows you how to configure a Pod to use a
for storage.
Here is a summary of the process:
You, as cluster administrator, create a PersistentVolume backed by physical
storage. You do not associate the volume with any Pod.
You, now taking the role of a developer / cluster user, create a
PersistentVolumeClaim that is automatically bound to a suitable
You create a Pod that uses the above PersistentVolumeClaim for storage.
You need to have a Kubernetes cluster that has only one Node, and the
kubectl
command-line tool must be configured to communicate with your cluster. If you
do not already have a single-node cluster, you can create one by using
Minikube
.
Familiarize yourself with the material in
.
Create an index.html file on your Node
Open a shell to the single Node in your cluster. How you open a shell depends
on how you set up your cluster. For example, if you are using Minikube, you
can open a shell to your Node by entering
minikube ssh
.
In your shell on that Node, create a
/mnt/data
# This assumes that your Node uses "sudo" to run commands
# as the superuser
sudo mkdir /mnt/data
In the
/mnt/data
directory, create an
file:
# This again assumes that your Node uses "sudo" to run commands
sudo sh -c
"echo 'Hello from Kubernetes storage' &gt; /mnt/data/index.html"
Note:
If your Node uses a tool for superuser access other than
sudo
, you can
usually make this work if you replace
sudo
with the name of the other tool.
Test that the
file exists:
cat /mnt/data/index.html
Hello from Kubernetes storage
You can now close the shell to your Node.
Create a PersistentVolume
In this exercise, you create a
hostPath
PersistentVolume. Kubernetes supports
hostPath for development and testing on a single-node cluster. A hostPath
PersistentVolume uses a file or directory on the Node to emulate network-attached storage.
In a production cluster, you would not use hostPath. Instead a cluster administrator
would provision a network resource like a Google Compute Engine persistent disk,
an NFS share, or an Amazon Elastic Block Store volume. Clus

ter administrators can also
use
StorageClasses
to set up
dynamic provisioning
.
Here is the configuration file for the hostPath PersistentVolume:
pods/storage/pv-volume.yaml
:
v1
kind
:
metadata
:
name
:
task-pv-volume
labels
:
type
:
local
spec
:
:
manual
capacity
:
storage
:
10Gi
:
hostPath
:
path
:
"/mnt/data"
The configuration file specifies that the volume is at
/mnt/data
on the
cluster's Node. The configuration also specifies a size of 10 gibibytes and
an access mode of
ReadWriteOnce
, which means the volume can be mounted as
read-write by a single Node. It defines the
StorageClass name
manual
for the PersistentVolume, which will be used to bind
PersistentVolumeClaim requests to this PersistentVolume.
Create the PersistentVolume:
kubectl apply -f https://k8s.io/examples/pods/storage/pv-volume.yaml
View information about the PersistentVolume:
kubectl get pv task-pv-volume
The output shows that the PersistentVolume has a
STATUS
of
Available
. This
means it has not yet been bound to a PersistentVolumeClaim.
NAME             CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS      CLAIM     STORAGECLASS   REASON    AGE
task-pv-volume   10Gi       RWO           Retain          Available             manual                   4s
The next step is to create a PersistentVolumeClaim. Pods use PersistentVolumeClaims
to request physical storage. In this exercise, you create a PersistentVolumeClaim
that requests a volume of at least three gibibytes that can provide read-write
access for at least one Node.
Here is the configuration file for the PersistentVolumeClaim:
pods/storage/pv-claim.yaml
:
v1
kind
:
metadata
:
name
:
task-pv-claim
spec
:
:
manual
:
resources
:
requests
:
storage
:
3Gi
kubectl apply -f https://k8s.io/examples/pods/storage/pv-claim.yaml
After you create the PersistentVolumeClaim, the Kubernetes control plane looks
for a PersistentVolume that satisfies the claim's requirements. If the control
plane finds a suitable PersistentVolume with the same StorageClass, it binds the
claim to the volume.
Look again at the PersistentVolume:
Now the output shows a
STATUS
of
Bound
.
NAME             CAPACITY   ACCESSMODES   RECLAIMPOLICY   STATUS    CLAIM                   STORAGECLASS   REASON    AGE
task-pv-volume   10Gi       RWO           Retain          Bound     default/task-pv-claim   manual                   2m
Look at the PersistentVolumeClaim:
kubectl get pvc task-pv-claim
The output shows that the PersistentVolumeClaim is bound to your PersistentVolume,
.
NAME            STATUS    VOLUME           CAPACITY   ACCESSMODES   STORAGECLASS   AGE
task-pv-claim   Bound     task-pv-volume   10Gi       RWO           manual         30s
The next step is to create a Pod that uses your PersistentVolumeClaim as a volume.
Here is the configuration file for the Pod:
pods/storage/pv-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
task-pv-pod
spec
:
volumes
:
-
name
:
task-pv-storage
:
claimName
:
:
-
name
:
task-pv-container
image
:
nginx
ports
:
-
:
80
name
:
"http-server"
:
-
mountPath
:
"/usr/share/nginx/html"
name
:
Notice that the Pod's configuration file specifies a PersistentVolumeClaim, but
it does not specify a PersistentVolume. From the Pod's point of view, the claim
is a volume.
kubectl apply -f https://k8s.io/examples/pods/storage/pv-pod.yaml
Verify that the container in the Pod is running;
kubectl get pod task-pv-pod
Get a shell to the container running in your Pod:
kubectl
exec
-it task-pv-pod -- /bin/bash
In your shell, verify that nginx is serving the
file from the
hostPath volume:
# Be sure to run these 3 commands inside the root shell

 that comes from
# running "kubectl exec" in the previous step
apt update
apt install curl
file on the
If you see that message, you have successfully configured a Pod to
use storage from a PersistentVolumeClaim.
Clean up
Delete the Pod, the PersistentVolumeClaim and the PersistentVolume:
kubectl delete pod task-pv-pod
kubectl delete pvc task-pv-claim
kubectl delete pv task-pv-volume
If you don't already have a shell open to the Node in your cluster,
open a new shell the same way that you did earlier.
In the shell on your Node, remove the file and directory that you created:
sudo rm /mnt/data/index.html
sudo rmdir /mnt/data
Mounting the same persistentVolume in two places
pods/storage/pv-duplicate.yaml
:
v1
kind
:
Pod
metadata
:
name
:
test
spec
:
:
-
name
:
test
image
:
nginx
:
# a mount for site-data
-
name
:
config
mountPath
:
subPath
:
html
# another mount for nginx config
-
name
:
config
mountPath
:
/etc/nginx/nginx.conf
subPath
:
nginx.conf
volumes
:
-
name
:
config
:
claimName
:
test-nfs-claim
You can perform 2 volume mounts on your nginx container:
for the static website
for the default config
Access control
Storage configured with a group ID (GID) allows writing only by Pods using the same
GID. Mismatched or missing GIDs cause permission denied errors. To reduce the
need for coordination with users, an administrator can annotate a PersistentVolume
with a GID. Then the GID is automatically added to any Pod that uses the
Use the
pv.beta.kubernetes.io/gid
annotation as follows:
:
v1
kind
:
metadata
:
name
:
pv1
:
:
"1234"
When a Pod consumes a PersistentVolume that has a GID annotation, the annotated GID
is applied to all containers in the Pod in the same way that GIDs specified in the
Pod's security context are. Every GID, whether it originates from a PersistentVolume
annotation or the Pod's specification, is applied to the first process run in
each container.
Note:
When a Pod consumes a PersistentVolume, the GIDs associated with the
PersistentVolume are not present on the Pod resource itself.
.
Read the
Persistent Storage design document
.
Reference
PersistentVolumeSpec
10 - Configure a Pod to Use a Projected Volume for Storage
This page shows how to use a
projected
Volume to mount
several existing volume sources into the same directory. Currently,
secret
,
configMap
,
,
and
volumes can be projected.
Note:
is not a volume type.
minikube
.
Configure a projected volume for a pod
In this exercise, you create username and password
Secrets
from local files. You then create a Pod that runs one container, using a
projected
Volume to mount the Secrets into the same shared directory.
pods/storage/projected.yaml
:
v1
kind
:
Pod
metadata
:
name
:
test-projected-volume
spec
:
:
-
name
:
image
:
args
:
- sleep
-
"86400"
:
-
name
:
all-in-one
mountPath
:
"/projected-

volume"
readOnly
:
true
volumes
:
-
name
:
projected
:
sources
:
-
secret
:
name
:
user
-
secret
:
name
:
pass
Create the Secrets:
# Create files containing the username and password:
echo
-n
"admin"
&gt; ./username.txt
echo
-n
"1f2d1e2e67df"
&gt; ./password.txt
# Package these files into secrets:
kubectl create secret generic user --from-file
=
./username.txt
kubectl create secret generic pass --from-file
=
./password.txt
kubectl apply -f https://k8s.io/examples/pods/storage/projected.yaml
Verify that the Pod's container is running, and then watch for changes to
the Pod:
kubectl get --watch pod test-projected-volume
NAME                    READY     STATUS    RESTARTS   AGE
test-projected-volume   1/1       Running   0          14s
In another terminal, get a shell to the running container:
kubectl
exec
-it test-projected-volume -- /bin/sh
In your shell, verify that the
projected-volume
directory contains your projected sources:
ls /projected-volume/
Clean up
Delete the Pod and the Secrets:
kubectl delete pod test-projected-volume
kubectl delete secret user pass
projected
volumes.
Read the
all-in-one volume
11 - Configure a Security Context for a Pod or Container
A security context defines privilege and access control settings for
a Pod or Container. Security context settings include, but are not limited to:
Discretionary Access Control: Permission to access an object, like a file, is based on
user ID (UID) and group ID (GID)
.
Security Enhanced Linux (SELinux)
:
Objects are assigned security labels.
Running as privileged or unprivileged.
Linux Capabilities
:
Give a process some privileges, but not all the privileges of the root user.
AppArmor
:
Use program profiles to restrict the capabilities of individual programs.
Seccomp
: Filter a process's system calls.
: Controls whether a process can gain more privileges than
its parent process. This bool directly controls whether the
no_new_privs
flag gets set on the container process.
is always true when the container:
is run as privileged, or
has
CAP_SYS_ADMIN
readOnlyRootFilesystem
: Mounts the container's root filesystem as read-only.
The above bullets are not a complete set of security context settings -- please see
for a comprehensive list.
minikube
.
Set the security context for a Pod
To specify security settings for a Pod, include the
field
in the Pod specification. The
field is a
object.
The security settings that you specify for a Pod apply to all Containers in the Pod.
Here is a configuration file for a Pod that has a
and an
emptyDir
volume:
pods/security/security-context.yaml
:
v1
kind
:
Pod
metadata
:
name
:
security-context-demo
spec
:
:
runAsUser
:
1000
runAsGroup
:
3000
fsGroup
:
2000
volumes
:
-
name
:
sec-ctx-vol
emptyDir
:
{}
:
-
name
:
sec-ctx-demo
image
:
command
:
[
"sh"
,
"-c"
,
"sleep 1h"
]
:
-
name
:
mountPath
:
/data/demo
:
:
false
In the configuration file, the
runAsUser
field specifies that for any Containers in
the Pod, all processes run with user ID 1000. The
field specifies the primary group ID of 3000 for
all processes within any containers of the Pod. If this field is omitted, the primary group ID of the containers
will be root(0). Any files create

d will also be owned by user 1000 and group 3000 when
is specified.
Since
fsGroup
field is specified, all processes of the container are also part of the supplementary group ID 2000.
The owner for volume
and any files created in that volume will be Group ID 2000.
kubectl apply -f https://k8s.io/examples/pods/security/security-context.yaml
kubectl get pod security-context-demo
kubectl
exec
-it security-context-demo -- sh
ps
The output shows that the processes are running as user 1000, which is the value of
runAsUser
:
PID   USER     TIME  COMMAND
    1 1000      0:00 sleep 1h
    6 1000      0:00 sh
...
In your shell, navigate to
/data
, and list the one directory:
cd
/data
ls -l
The output shows that the
directory has group ID 2000, which is
the value of
fsGroup
.
drwxrwsrwx 2 root 2000 4096 Jun  6 20:08 demo
, and create a file:
cd
demo
echo
hello &gt; testfile
List the file in the
ls -l
The output shows that
testfile
has group ID 2000, which is the value of
fsGroup
.
-rw-r--r-- 1 1000 2000 6 Jun  6 20:08 testfile
id
uid=1000 gid=3000 groups=2000
From the output, you can see that
gid
is 3000 which is same as the
field.
If the
was omitted, the
gid
would remain as 0 (root) and the process will
be able to interact with files that are owned by the root(0) group and groups that have
the required group permissions for the root (0) group.
Exit your shell:
exit
Configure volume permission and ownership change policy for Pods
Kubernetes v1.23 [stable]
By default, Kubernetes recursively changes ownership and permissions for the contents of each
volume to match the
fsGroup
specified in a Pod's
when that volume is
mounted.
For large volumes, checking and changing ownership and permissions can take a lot of time,
slowing Pod startup. You can use the
fsGroupChangePolicy
field inside a
to control the way that Kubernetes checks and manages ownership and permissions
for a volume.
-
defines behavior for changing ownership
and permission of the volume before being exposed inside a Pod.
This field only applies to volume types that support
fsGroup
controlled ownership and permissions.
This field has two possible values:
OnRootMismatch
: Only change permissions and ownership if the permission and the ownership of
root directory does not match with expected permissions of the volume.
This could help shorten the time it takes to change ownership and permission of a volume.
Always
: Always change permission and ownership of the volume when volume is mounted.
:
runAsUser
:
1000
:
3000
fsGroup
:
2000
:
"OnRootMismatch"
Note:
This field has no effect on ephemeral volume types such as
secret
,
configMap
,
and
emptydir
.
Delegating volume permission and ownership change to CSI driver
If you deploy a
Container Storage Interface (CSI)
driver which supports the
VOLUME_MOUNT_GROUP
NodeServiceCapability
, the
process of setting file ownership and permissions based on the
fsGroup
will be performed by the CSI driver
instead of Kubernetes. In this case, since Kubernetes doesn't perform any
ownership and permission change,
does not take effect, and
as specified by CSI, the driver is expected to mount the volume with the
provided
fsGroup
, resulting in a volume that is readable/writable by the
fsGroup
.
Set the security context for a Container
To specify security settings for a Container, include the
field
in the Container manifest. The
field is a
object.
Security settings 

that you specify for a Container apply only to
the individual Container, and they override settings made at the Pod level when
there is overlap. Container settings do not affect the Pod's Volumes.
Here is the configuration file for a Pod that has one Container. Both the Pod
and the Container have a
field:
pods/security/security-context-2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
security-context-demo-2
spec
:
:
runAsUser
:
1000
:
-
name
:
sec-ctx-demo-2
image
:
:
runAsUser
:
2000
:
false
kubectl apply -f https://k8s.io/examples/pods/security/security-context-2.yaml
kubectl get pod security-context-demo-2
Get a shell into the running Container:
kubectl
exec
-it security-context-demo-2 -- sh
ps aux
The output shows that the processes are running as user 2000. This is the value
of
runAsUser
specified for the Container. It overrides the value 1000 that is
specified for the Pod.
2000         1  0.0  0.0   4336   764 ?        Ss   20:36   0:00 /bin/sh -c node server.js
2000         8  0.1  0.5 772124 22604 ?        Sl   20:36   0:00 node server.js
...
exit
Set capabilities for a Container
With
Linux capabilities
,
you can grant certain privileges to a process without granting all the privileges
of the root user. To add or remove Linux capabilities for a Container, include the
capabilities
field in the
section of the Container manifest.
First, see what happens when you don't include a
field.
Here is configuration file that does not add or remove any Container capabilities:
pods/security/security-context-3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
security-context-demo-3
spec
:
:
-
name
:
sec-ctx-3
image
:
kubectl apply -f https://k8s.io/examples/pods/security/security-context-3.yaml
kubectl get pod security-context-demo-3
kubectl
exec
-it security-context-demo-3 -- sh
ps aux
The output shows the process IDs (PIDs) for the Container:
USER  PID %CPU %MEM    VSZ   RSS TTY   STAT START   TIME COMMAND
root    1  0.0  0.0   4336   796 ?     Ss   18:17   0:00 /bin/sh -c node server.js
root    5  0.1  0.5 772124 22700 ?     Sl   18:17   0:00 node server.js
In your shell, view the status for process 1:
cd
/proc/1
cat status
The output shows the capabilities bitmap for the process:
...
CapPrm:	00000000a80425fb
CapEff:	00000000a80425fb
...
Make a note of the capabilities bitmap, and then exit your shell:
exit
Next, run a Container that is the same as the preceding container, except
that it has additional capabilities set.
Here is the configuration file for a Pod that runs one Container. The configuration
adds the
CAP_NET_ADMIN
and
CAP_SYS_TIME
capabilities:
pods/security/security-context-4.yaml
:
v1
kind
:
Pod
metadata
:
name
:
security-context-demo-4
spec
:
:
-
name
:
sec-ctx-4
image
:
:
:
add
:
[
"NET_ADMIN"
,
"SYS_TIME"
]
kubectl apply -f https://k8s.io/examples/pods/security/security-context-4.yaml
kubectl
exec
-it security-context-demo-4 -- sh
In your shell, view the capabilities for process 1:
cd
/proc/1
cat status
The output shows capabilities bitmap for the process:
...
CapPrm:	00000000aa0435fb
CapEff:	00000000aa0435fb
...
Compare the capabilities of the two Containers:
00000000a80425fb
00000000aa0435fb
In the capability bitmap of the first container, bits 12 and 25 are clear. In the second container,
bits 12 and 25 are set. Bit 12 is
, and b

it 25 is
.
See
capability.h
for definitions of the capability constants.
Note:
Linux capability constants have the form
CAP_XXX
.
But when you list capabilities in your container manifest, you must
omit the
CAP_
portion of the constant.
For example, to add
, include
SYS_TIME
in your list of capabilities.
Set the Seccomp Profile for a Container
To set the Seccomp profile for a Container, include the
field
in the
section of your Pod or Container manifest. The
field is a
SeccompProfile
object consisting of
type
and
.
Valid options for
type
include
,
, and
Localhost
.
must only be set if
type: Localhost
. It
indicates the path of the pre-configured profile on the node, relative to the
kubelet's configured Seccomp profile location (configured with the
--root-dir
flag).
Here is an example that sets the Seccomp profile to the node's container runtime
default profile:
...
:
:
type
:
Here is an example that sets the Seccomp profile to a pre-configured file at
&lt;kubelet-root-dir&gt;/seccomp/my-profiles/profile-allow.json
:
...
:
:
type
:
Localhost
:
my-profiles/profile-allow.json
Assign SELinux labels to a Container
To assign SELinux labels to a Container, include the
seLinuxOptions
field in
the
field is an
SELinuxOptions
object. Here's an example that applies an SELinux level:
...
:
:
level
:
"s0:c123,c456"
Note:
To assign SELinux labels, the SELinux security module must be loaded on the host operating system.
Efficient SELinux volume relabeling
By default, the contrainer runtime recursively assigns SELinux label to all
files on all Pod volumes. To speed up this process, Kubernetes can change the
SELinux label of a volume instantly by using a mount option
-o context=&lt;label&gt;
.
To benefit from this speedup, all these conditions must be met:
Alpha feature gates
ReadWriteOncePod
and
SELinuxMountReadWriteOncePod
must
be enabled.
Pod must use PersistentVolumeClaim with
accessModes: ["ReadWriteOncePod"]
.
Pod (or all its Containers that use the PersistentVolumeClaim) must
have
set.
The corresponding PersistentVolume must be either a volume that uses a
CSI
driver, or a volume that uses the
legacy
iscsi
volume type.
If you use a volume backed by a CSI driver, that CSI driver must announce that it
supports mounting with
-o context
by setting
spec.seLinuxMount: true
in
its CSIDriver instance.
For any other volume types, SELinux relabelling happens another way: the container
runtime recursively changes the SELinux label for all inodes (files and directories)
in the volume.
The more files and directories in the volume, the longer that relabelling takes.
Note:
In Kubernetes 1.25, the kubelet loses track of volume labels after restart. In
other words, then kubelet may refuse to start Pods with errors similar to "conflicting
SELinux labels of volume", while there are no conflicting labels in Pods. Make sure
nodes are
fully drained
before restarting kubelet.
The security context for a Pod applies to the Pod's Containers and also to
the Pod's Volumes when applicable. Specifically
fsGroup
and
are
applied to Volumes as follows:
fsGroup
: Volumes that support ownership management are modified to be owned
and writable by the GID specified in
fsGroup
. See the
Ownership Management design document
: Volumes that support SELinux labeling are relabeled to be accessible
by the label specified under
. Usually you only
need to set the
level
section. This sets the
Multi-Category Security (MCS)
label given to all Containers in the Pod as well as the Volumes.
Warning:
After you specify an MCS label for a

 Pod, all Pods with the same label can access the Volume.
If you need inter-Pod protection, you must assign a unique MCS label to each Pod.
Clean up
Delete the Pod:
kubectl delete pod security-context-demo
kubectl delete pod security-context-demo-2
kubectl delete pod security-context-demo-3
kubectl delete pod security-context-demo-4
Tuning Docker with the newest security enhancements
Security Contexts design document
PodSecurity Admission
AllowPrivilegeEscalation design
document
For more information about security mechanisms in Linux, see
Overview of Linux Kernel Security Features
(Note: Some information is out of date)
12 - Configure Service Accounts for Pods
Kubernetes offers two distinct ways for clients that run within your
cluster, or that otherwise have a relationship to your cluster's
to authenticate to the
API server
.
A
service account
provides an identity for processes that run in a Pod,
and maps to a ServiceAccount object. When you authenticate to the API
server, you identify yourself as a particular
user
. Kubernetes recognises
the concept of a user, however, Kubernetes itself does
not
have a User
API.
This task guide is about ServiceAccounts, which do exist in the Kubernetes
API. The guide shows you some ways to configure ServiceAccounts for Pods.
minikube
Use the default service account to access the API server
When Pods contact the API server, Pods authenticate as a particular
ServiceAccount (for example,
default
). There is always at least one
ServiceAccount in each
namespace
.
Every Kubernetes namespace contains at least one ServiceAccount: the default
ServiceAccount for that namespace, named
default
.
If you do not specify a ServiceAccount when you create a Pod, Kubernetes
automatically assigns the ServiceAccount named
default
in that namespace.
You can fetch the details for a Pod you have created. For example:
kubectl get pods/&lt;podname&gt; -o yaml
In the output, you see a field
spec.serviceAccountName
.
automatically
sets that value if you don't specify it when you create a Pod.
An application running inside a Pod can access the Kubernetes API using
automatically mounted service account credentials.
See
accessing the Cluster
to learn more.
When a Pod authenticates as a ServiceAccount, its level of access depends on the
authorization plugin and policy
in use.
Opt out of API credential automounting
If you don't want the
kubelet
to automatically mount a ServiceAccount's API credentials, you can opt out of
the default behavior.
You can opt out of automounting API credentials on
for a service account by setting
automountServiceAccountToken: false
on the ServiceAccount:
:
v1
kind
:
metadata
:
name
:
build-robot
:
false
...
You can also opt out of automounting API credentials for a particular Pod:
:
v1
kind
:
Pod
metadata
:
name
:
my-pod
spec
:
:
:
false
...
If both the ServiceAccount and the Pod's
.spec
specify a value for
, the Pod spec takes precedence.
Use more than one ServiceAccount
Every namespace has at least one ServiceAccount: the default ServiceAccount
resource, called
default
. You can list all ServiceAccount resources in your
current namespace
with:
kubectl get serviceaccounts
NAME      SECRETS    AGE
default   1          1d
You can create a

dditional ServiceAccount objects like this:
kind: ServiceAccount
metadata:
name: build-robot
EOF
The name of a ServiceAccount object must be a valid
.
If you get a complete dump of the service account object, like this:
kubectl get serviceaccounts/build-robot -o yaml
:
v1
kind
:
metadata
:
:
2019-06-16T00:12:34Z
name
:
namespace
:
default
:
"272500"
uid
:
721ab723-13bc-11e5-aec2-42010af0021e
You can use authorization plugins to
set permissions on service accounts
.
To use a non-default service account, set the
field of a Pod to the name of the ServiceAccount you wish to use.
You can only set the
field when creating a Pod, or in a
template for a new Pod. You cannot update the
field
of a Pod that already exists.
Note:
The
.spec.serviceAccount
field is a deprecated alias for
.
If you want to remove the fields from a workload resource, set both fields to empty explicitly
on the
pod template
.
Cleanup
If you tried creating
ServiceAccount from the example above,
you can clean it up by running:
kubectl delete serviceaccount/build-robot
Suppose you have an existing service account named "build-robot" as mentioned earlier.
You can get a time-limited API token for that ServiceAccount using
kubectl
:
kubectl create token build-robot
The output from that command is a token that you can use to authenticate as that
ServiceAccount. You can request a specific token duration using the
--duration
command line argument to
kubectl create token
(the actual duration of the issued
token might be shorter, or could even be longer).
Note:
Versions of Kubernetes before v1.22 automatically created long term credentials for
accessing the Kubernetes API. This older mechanism was based on creating token Secrets
that could then be mounted into running Pods. In more recent versions, including
Kubernetes v1.26, API credentials are obtained directly by using the
API,
and are mounted into Pods using a
.
You can still manually create a service account token Secret; for example,
if you need a token that never expires. However, using the
subresource to obtain a token to access the API is recommended instead.
Manually create a long-lived API token for a ServiceAccount
If you want to obtain an API token for a ServiceAccount, you create a new Secret
with a special annotation,
.
metadata:
name: build-robot-secret
kubernetes.io/service-account.name: build-robot
EOF
If you view the Secret using:
kubectl get secret/build-robot-secret -o yaml
you can see that the Secret now contains an API token for the "build-robot" ServiceAccount.
Because of the annotation you set, the control plane automatically generates a token for that
ServiceAccounts, and stores them into the associated Secret. The control plane also cleans up
tokens for deleted ServiceAccounts.
kubectl describe secrets/build-robot-secret
Name:           build-robot-secret
Annotations:    kubernetes.io/service-account.name: build-robot
                kubernetes.io/service-account.uid: da68f9c6-9d26-11e7-b84e-002dc52800da

Data
====
ca.crt:         1338 bytes
namespace:      7 bytes
token:          ...
Note:
The content of
token
is elided here.
Take care not to display the contents of a
Secret somewhere that 

your terminal / computer screen could be seen by an onlooker.
When you delete a ServiceAccount that has an associated Secret, the Kubernetes
control plane automatically cleans up the long-lived token from that Secret.
Add ImagePullSecrets to a service account
First,
create an imagePullSecret
.
Next, verify it has been created. For example:
Create an imagePullSecret, as described in
Specifying ImagePullSecrets on a Pod
.
kubectl create secret docker-registry myregistrykey --docker-server
=
DUMMY_SERVER
\
--docker-username
=
DUMMY_USERNAME --docker-password
=
DUMMY_DOCKER_PASSWORD
\
--docker-email
=
DUMMY_DOCKER_EMAIL
Verify it has been created.
kubectl get secrets myregistrykey
NAME             TYPE                              DATA    AGE
myregistrykey    kubernetes.io/.dockerconfigjson   1       1d
Add image pull secret to service account
Next, modify the default service account for the namespace to use this Secret as an imagePullSecret.
kubectl patch serviceaccount default -p
'{"imagePullSecrets": [{"name": "myregistrykey"}]}'
You can achieve the same outcome by editing the object manually:
kubectl edit serviceaccount/default
The output of the
sa.yaml
file is similar to this:
Your selected text editor will open with a configuration looking something like this:
:
v1
kind
:
metadata
:
:
2021-07-07T22:02:39Z
name
:
default
namespace
:
default
:
"243024"
uid
:
052fb0f4-3d50-11e5-b066-42010af0d7b6
Using your editor, delete the line with key
, add lines for
imagePullSecrets:
and save it. Leave the
uid
value set the same as you found it.
After you made those changes, the edited ServiceAccount looks something like this:
:
v1
kind
:
metadata
:
:
name
:
default
namespace
:
default
uid
:
:
-
name
:
myregistrykey
Verify that imagePullSecrets are set for new Pods
Now, when a new Pod is created in the current namespace and using the default
ServiceAccount, the new Pod has its
spec.imagePullSecrets
field set automatically:
=
nginx --restart
=
Never
kubectl get pod nginx -o
=
jsonpath
=
'{.spec.imagePullSecrets[0].name}{"\n"}'
ServiceAccount token volume projection
Kubernetes v1.20 [stable]
Note:
To enable and use token request projection, you must specify each of the following
command line arguments to
:
--service-account-issuer
defines the Identifier of the service account token issuer. You can specify the
argument multiple times, this can be useful to enable
a non-disruptive change of the issuer. When this flag is specified multiple times,
the first is used to generate tokens and all are used to determine which issuers
are accepted. You must be running Kubernetes v1.22 or later to be able to specify
multiple times.
specifies the path to a file containing PEM-encoded X.509 private or public keys
(RSA or ECDSA), used to verify ServiceAccount tokens. The specified file can contain
multiple keys, and the flag can be specified multiple times with different files.
If specified multiple times, tokens signed by any of the specified keys are considered
valid by the Kubernetes API server.
--service-account-signing-key-file
specifies the path to a file that contains the current private key of the service
account token issuer. The issuer signs issued ID tokens with this private key.
--api-audiences
(can be omitted)
defines audiences for ServiceAccount tokens. The service account token authenticator
validates that tokens used against the API are bound to at least one of these audiences.
If
api-audiences
is specified multiple times, tokens for any of the specified audiences
are considered valid by the Kubernetes API server. If you specify the
com

mand line argument but you don't set
, the control plane defaults to
a single element audience list that contains only the issuer URL.
The kubelet can also project a ServiceAccount token into a Pod. You can
specify desired properties of the token, such as the audience and the validity
duration. These properties are
not
configurable on the default ServiceAccount
token. The token will also become invalid against the API when either the Pod
or the ServiceAccount is deleted.
You can configure this behavior for the
spec
of a Pod using a
type called
ServiceAccountToken
.
Launch a Pod using service account token projection
To provide a Pod with a token with an audience of
vault
and a validity duration
of two hours, you could define a Pod manifest that is similar to:
pods/pod-projected-svc-token.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
-
image
:
nginx
name
:
nginx
:
-
mountPath
:
/var/run/secrets/tokens
name
:
vault-token
:
volumes
:
-
name
:
projected
:
sources
:
-
:
path
:
:
7200
audience
:
vault
kubectl create -f https://k8s.io/examples/pods/pod-projected-svc-token.yaml
The kubelet will: request and store the token on behalf of the Pod; make
the token available to the Pod at a configurable file path; and refresh
the token as it approaches expiration. The kubelet proactively requests rotation
for the token if it is older than 80% of its total time-to-live (TTL),
or if the token is older than 24 hours.
The application is responsible for reloading the token when it rotates. It's
often good enough for the application to load the token on a schedule
(for example: once every 5 minutes), without tracking the actual expiry time.
Service account issuer discovery
If you have enabled
token projection
for ServiceAccounts in your cluster, then you can also make use of the discovery
feature. Kubernetes provides a way for clients to federate as an
identity provider
,
so that one or more external systems can act as a
relying party
.
Note:
The issuer URL must comply with the
OIDC Discovery Spec
. In
practice, this means it must use the
https
scheme, and should serve an OpenID
provider configuration at
{service-account-issuer}/.well-known/openid-configuration
.
If the URL does not comply, ServiceAccount issuer discovery endpoints are not
registered or accessible.
When enabled, the Kubernetes API server publishes an OpenID Provider
Configuration document via HTTP. The configuration document is published at
/.well-known/openid-configuration
.
The OpenID Provider Configuration is sometimes referred to as the
discovery document
.
The Kubernetes API server publishes the related
JSON Web Key Set (JWKS), also via HTTP, at
/openid/v1/jwks
.
Note:
The responses served at
and
are designed to be OIDC compatible, but not strictly OIDC
compliant. Those documents contain only the parameters necessary to perform
validation of Kubernetes service account tokens.
Clusters that use
RBAC
include a
default ClusterRole called
system:service-account-issuer-discovery
.
A default ClusterRoleBinding assigns this role to the
system:serviceaccounts
group,
which all ServiceAccounts implicitly belong to.
This allows pods running on the cluster to access the service account discovery document
via their mounted service account token. Administrators may, additionally, choose to
bind the role to
system:authenticated
or
depending on their
security requirements and which external systems they intend to federate with.
The JWKS response contains public keys that a relying party can use to validate
the Kubernetes service account tokens. Relying parties first query for the
OpenID Provider Configuration, and use the
jwks_uri
field in the response to
find the JWKS.
In many cases, Kubernetes A

PI servers are not available on the public internet,
but public endpoints that serve cached responses from the API server can be made
available by users or by service providers. In these cases, it is possible to
override the
jwks_uri
in the OpenID Provider Configuration so that it points
to the public endpoint, rather than the API server's address, by passing the
--service-account-jwks-uri
flag to the API server. Like the issuer URL, the
JWKS URI is required to use the
https
scheme.
See also:
Read the
Cluster Admin Guide to Service Accounts
Read about
Authorization in Kubernetes
Read about
Secrets
or learn to
distribute credentials securely using Secrets
but also bear in mind that using Secrets for authenticating as a ServiceAccount
is deprecated. The recommended alternative is
.
Read about
.
For background on OIDC discovery, read the
ServiceAccount signing key retrieval
Kubernetes Enhancement Proposal
Read the
13 - Pull an Image from a Private Registry
This page shows how to create a Pod that uses a
Secret
to pull an image
from a private container image registry or repository. There are many private
registries in use. This task uses
Docker Hub
as an example registry.
🛇 This item links to a third party project or product that is not part of Kubernetes itself.
More information
minikube
To do this exercise, you need the
docker
command line tool, and a
Docker ID
for which you know the password.
If you are using a different private container registry, you need the command
line tool for that registry and any login information for the registry.
Log in to Docker Hub
On your laptop, you must authenticate with a registry in order to pull a private image.
Use the
docker
tool to log in to Docker Hub. See the
log in
section of
Docker ID accounts
docker login
When prompted, enter your Docker ID, and then the credential you want to use (access token,
or the password for your Docker ID).
The login process creates or updates a
config.json
file that holds an authorization token. Review
how Kubernetes interprets this file
.
View the
file:
cat ~/.docker/config.json
The output contains a section similar to this:
{
"auths"
: {
"https://index.docker.io/v1/"
: {
"auth"
:
"c3R...zE2"
}
}
}
Note:
If you use a Docker credentials store, you won't see that
auth
entry but a
credsStore
entry with the name of the store as value.
In that case, you can create a secret directly. See
Create a Secret by providing credentials on the command line
.
Create a Secret based on existing credentials
A Kubernetes cluster uses the Secret of
kubernetes.io/dockerconfigjson
type to authenticate with
a container registry to pull a private image.
If you already ran
, you can copy
that credential into Kubernetes:
kubectl create secret generic regcred
\
--from-file
=
.dockerconfigjson
=
&lt;path/to/.docker/config.json&gt;
\
--type
=
If you need more control (for example, to set a namespace or a label on the new
secret) then you can customise the Secret before storing it.
Be sure to:
set the name of the data item to
base64 encode the Docker configuration file and then paste that string, unbroken
as the value for field
data[".dockerconfigjson"]
set
type
to
Example:
:
v1
kind
:
Secret
metadata
:
name
:
namespace
:
awesomeapps
data
:
:
UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsb

GxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
type
:
If you get the error message
error: no objects passed to create
, it may mean the base64 encoded string is invalid.
If you get an error message like
Secret "myregistrykey" is invalid: data[.dockerconfigjson]: invalid value ...
, it means
the base64 encoded string in the data was successfully decoded, but could not be parsed as a
.docker/config.json
file.
Create this Secret, naming it
regcred
:
kubectl create secret docker-registry regcred --docker-server
=
&lt;your-registry-server&gt; --docker-username
=
&lt;your-name&gt; --docker-password
=
&lt;your-pword&gt; --docker-email
=
&lt;your-email&gt;
where:
&lt;your-registry-server&gt;
is your Private Docker Registry FQDN.
Use
https://index.docker.io/v1/
for DockerHub.
&lt;your-name&gt;
is your Docker username.
&lt;your-pword&gt;
is your Docker password.
is your Docker email.
You have successfully set your Docker credentials in the cluster as a Secret called
regcred
.
Note:
Typing secrets on the command line may store them in your shell history unprotected, and
those secrets might also be visible to other users on your PC during the time that
kubectl
is running.
Inspecting the Secret
regcred
To understand the contents of the
regcred
Secret you created, start by viewing the Secret in YAML format:
kubectl get secret regcred --output
=
yaml
:
v1
kind
:
Secret
metadata
:
...
name
:
regcred
...
data
:
:
eyJodHRwczovL2luZGV4L ... J0QUl6RTIifX0=
type
:
The value of the
field is a base64 representation of your Docker credentials.
To understand what is in the
field, convert the secret data to a
readable format:
=
"jsonpath={.data.\.dockerconfigjson}"
{
"auths"
:{
"your.private.registry.example.com"
:{
"username"
:
"janedoe"
,
"password"
:
"xxxxxxxxxxx"
,
"email"
:
"jdoe@example.com"
,
"auth"
:
}}}
auth
field, convert the base64-encoded data to a readable format:
echo
The output, username and password concatenated with a
:
, is similar to this:
janedoe:xxxxxxxxxxx
Notice that the Secret data contains the authorization token similar to your local
~/.docker/config.json
file.
You have successfully set your Docker credentials as a Secret called
regcred
in the cluster.
Create a Pod that uses your Secret
Here is a manifest for an example Pod that needs access to your Docker credentials in
regcred
:
pods/private-reg-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
private-reg
spec
:
:
-
name
:
private-reg-container
image
:
&lt;your-private-image&gt;
:
-
name
:
regcred
Download the above file onto your computer:
curl -L -o my-private-reg-pod.yaml https://k8s.io/examples/pods/private-reg-pod.yaml
In file
my-private-reg-pod.yaml
, replace
with the path to an image in a private registry such as:
your.private.registry.example.com/janedoe/jdoe-private:v1
To pull the image from the private registry, Kubernetes needs credentials.
The
field in the configuration file specifies that
Kubernetes should get the credentials from a Secret named
regcred
.
Create a Pod that uses your Secret, and verify that the Pod is running:
kubectl apply -f my-private-reg-pod.yaml
kubectl get pod private-reg
Secrets
or read the API reference for
Secret
using a private registry
.
adding image pull secrets to a service account
.
See
kubectl create secret docker-registry
.
See the
field within the
container d

efinitions
of a Pod
14 - Configure Liveness, Readiness and Startup Probes
This page shows how to configure liveness, readiness and startup probes for containers.
The
kubelet
uses liveness probes to know when to
restart a container. For example, liveness probes could catch a deadlock,
where an application is running, but unable to make progress. Restarting a
container in such a state can help to make the application more available
despite bugs.
The kubelet uses readiness probes to know when a container is ready to start
accepting traffic. A Pod is considered ready when all of its containers are ready.
One use of this signal is to control which Pods are used as backends for Services.
When a Pod is not ready, it is removed from Service load balancers.
The kubelet uses startup probes to know when a container application has started.
If such a probe is configured, it disables liveness and readiness checks until
it succeeds, making sure those probes don't interfere with the application startup.
This can be used to adopt liveness checks on slow starting containers, avoiding them
getting killed by the kubelet before they are up and running.
minikube
Define a liveness command
Many applications running for long periods of time eventually transition to
broken states, and cannot recover except by being restarted. Kubernetes provides
liveness probes to detect and remedy such situations.
In this exercise, you create a Pod that runs a container based on the
registry.k8s.io/busybox
image. Here is the configuration file for the Pod:
pods/probe/exec-liveness.yaml
:
v1
kind
:
Pod
metadata
:
labels
:
test
:
liveness
name
:
liveness-exec
spec
:
:
-
name
:
liveness
image
:
args
:
- /bin/sh
- -c
- touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
livenessProbe
:
exec
:
command
:
- cat
- /tmp/healthy
initialDelaySeconds
:
5
periodSeconds
:
5
In the configuration file, you can see that the Pod has a single
Container
.
The
field specifies that the kubelet should perform a liveness
probe every 5 seconds. The
field tells the kubelet that it
should wait 5 seconds before performing the first probe. To perform a probe, the
kubelet executes the command
cat /tmp/healthy
in the target container. If the
command succeeds, it returns 0, and the kubelet considers the container to be alive and
healthy. If the command returns a non-zero value, the kubelet kills the container
and restarts it.
When the container starts, it executes this command:
/bin/sh -c
"touch /tmp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600"
For the first 30 seconds of the container's life, there is a
/tmp/healthy
file.
So during the first 30 seconds, the command
returns a success
code. After 30 seconds,
returns a failure code.
kubectl apply -f https://k8s.io/examples/pods/probe/exec-liveness.yaml
Within 30 seconds, view the Pod events:
kubectl describe pod liveness-exec
The output indicates that no liveness probes have failed yet:
Type    Reason     Age   From               Message
  Normal  Scheduled  11s   default-scheduler  Successfully assigned default/liveness-exec to node01
  Normal  Pulling    9s    kubelet, node01    Pulling image "registry.k8s.io/busybox"
  Normal  Pulled     7s    kubelet, node01    Successfully pulled image "registry.k8s.io/busybox"
  Normal  Created    7s    kubelet, node01    Created container liveness
  Normal  Started    7s    kubelet, node01    Started container l

iveness
After 35 seconds, view the Pod events again:
At the bottom of the output, there are messages indicating that the liveness
probes have failed, and the failed containers have been killed and recreated.
Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  57s                default-scheduler  Successfully assigned default/liveness-exec to node01
  Normal   Pulling    55s                kubelet, node01    Pulling image "registry.k8s.io/busybox"
  Normal   Pulled     53s                kubelet, node01    Successfully pulled image "registry.k8s.io/busybox"
  Normal   Created    53s                kubelet, node01    Created container liveness
  Normal   Started    53s                kubelet, node01    Started container liveness
  Warning  Unhealthy  10s (x3 over 20s)  kubelet, node01    Liveness probe failed: cat: can't open '/tmp/healthy': No such file or directory
  Normal   Killing    10s                kubelet, node01    Container liveness failed liveness probe, will be restarted
Wait another 30 seconds, and verify that the container has been restarted:
kubectl get pod liveness-exec
RESTARTS
has been incremented. Note that the
RESTARTS
counter increments as soon as a failed container comes back to the running state:
liveness-exec   1/1       Running   1          1m
Define a liveness HTTP request
Another kind of liveness probe uses an HTTP GET request. Here is the configuration
file for a Pod that runs a container based on the
registry.k8s.io/liveness
image.
pods/probe/http-liveness.yaml
:
v1
kind
:
Pod
metadata
:
labels
:
test
:
liveness
name
:
liveness-http
spec
:
:
-
name
:
liveness
image
:
args
:
- /server
:
httpGet
:
path
:
/healthz
port
:
8080
httpHeaders
:
-
name
:
Custom-Header
value
:
Awesome
:
3
:
3
In the configuration file, you can see that the Pod has a single container.
The
probe every 3 seconds. The
should wait 3 seconds before performing the first probe. To perform a probe, the
kubelet sends an HTTP GET request to the server that is running in the container
and listening on port 8080. If the handler for the server's
/healthz
path
returns a success code, the kubelet considers the container to be alive and
healthy. If the handler returns a failure code, the kubelet kills the container
Any code greater than or equal to 200 and less than 400 indicates success. Any
other code indicates failure.
You can see the source code for the server in
server.go
.
For the first 10 seconds that the container is alive, the
/healthz
handler
returns a status of 200. After that, the handler returns a status of 500.
http.
HandleFunc
(
,
func
(w http.ResponseWriter, r
*
http.Request) {
duration
:=
time.
Now
().
Sub
(started)
if
duration.
Seconds
() &gt;
10
{
w.
WriteHeader
(
500
)
w.
Write
([]
byte
(fmt.
Sprintf
(
"error: %v"
, duration.
Seconds
())))
}
else
{
w.
(
200
)
w.
Write
([]
byte
(
"ok"
))
}
})
The kubelet starts performing health checks 3 seconds after the container starts.
So the first couple of health checks will succeed. But after 10 seconds, the health
checks will fail, and the kubelet will kill and restart the container.
To try the HTTP liveness check, create a Pod:
kubectl apply -f https://k8s.io/examples/pods/probe/http-liveness.yaml
After 10 seconds, view Pod events to verify that liveness probes have failed and
the container has been restarted:
kubectl describe pod liveness-http
In releases prior to v1.13 (including v1.13), if the environment variable
http_proxy
(or
HTTP_PROXY
) is set on the node where a Pod is running,
the HTTP liveness probe uses t

hat proxy.
In releases after v1.13, local HTTP proxy environment variable settings do not
affect the HTTP liveness probe.
Define a TCP liveness probe
A third type of liveness probe uses a TCP socket. With this configuration, the
kubelet will attempt to open a socket to your container on the specified port.
If it can establish a connection, the container is considered healthy, if it
can't it is considered a failure.
pods/probe/tcp-liveness-readiness.yaml
:
v1
kind
:
Pod
metadata
:
name
:
goproxy
labels
:
app
:
goproxy
spec
:
:
-
name
:
goproxy
image
:
registry.k8s.io/goproxy:0.1
ports
:
-
:
8080
readinessProbe
:
tcpSocket
:
port
:
8080
:
5
:
10
:
tcpSocket
:
port
:
8080
:
15
:
20
As you can see, configuration for a TCP check is quite similar to an HTTP check.
This example uses both readiness and liveness probes. The kubelet will send the
first readiness probe 5 seconds after the container starts. This will attempt to
connect to the
goproxy
container on port 8080. If the probe succeeds, the Pod
will be marked as ready. The kubelet will continue to run this check every 10
seconds.
In addition to the readiness probe, this configuration includes a liveness probe.
The kubelet will run the first liveness probe 15 seconds after the container
starts. Similar to the readiness probe, this will attempt to connect to the
goproxy
container on port 8080. If the liveness probe fails, the container
will be restarted.
To try the TCP liveness check, create a Pod:
kubectl apply -f https://k8s.io/examples/pods/probe/tcp-liveness-readiness.yaml
After 15 seconds, view Pod events to verify that liveness probes:
kubectl describe pod goproxy
Define a gRPC liveness probe
If your application implements
gRPC Health Checking Protocol
,
kubelet can be configured to use it for application liveness checks.
You must enable the
GRPCContainerProbe
in order to configure checks that rely on gRPC.
Here is an example manifest:
pods/probe/grpc-liveness.yaml
:
v1
kind
:
Pod
metadata
:
name
:
etcd-with-grpc
spec
:
:
-
name
:
etcd
image
:
registry.k8s.io/etcd:3.5.1-0
command
:
[
"/usr/local/bin/etcd"
,
"--data-dir"
,
,
"--listen-client-urls"
,
"http://0.0.0.0:2379"
,
"--advertise-client-urls"
,
"http://127.0.0.1:2379"
,
"--log-level"
,
"debug"
]
ports
:
-
:
2379
:
grpc
:
port
:
2379
:
10
To use a gRPC probe,
port
must be configured. If the health endpoint is configured
on a non-default service, you must also specify the
service
.
Note:
Unlike HTTP and TCP probes, named ports cannot be used and custom host cannot be configured.
Configuration problems (for example: incorrect port and service, unimplemented health checking protocol)
are considered a probe failure, similar to HTTP and TCP probes.
To try the gRPC liveness check, create a Pod using the command below.
In the example below, the etcd pod is configured to use gRPC liveness probe.
kubectl apply -f https://k8s.io/examples/pods/probe/grpc-liveness.yaml
After 15 seconds, view Pod events to verify that the liveness check has not failed:
kubectl describe pod etcd-with-grpc
Before Kubernetes 1.23, gRPC health probes were often implemented using
grpc-health-probe
,
as described in the blog post
Health checking gRPC servers on Kubernetes
.
The built-in gRPC probes behavior is similar to one implemented by grpc-health-probe.
When migrating from grpc-health-probe to built-in probes, remember the following differences:
Built-in probes run against the pod IP address, unlike grpc-health-probe that often runs against
127.0.0.1
.
Be sure to configure your gRPC endpoint to listen on the Pod's IP address.
Built-in probes do not support any authentication parameters (like
-tls
).
There are no error codes for built-in probes. All errors are considered as probe failures.
If
ExecProb

eTimeout
feature gate is set to
false
, grpc-health-probe does
not
respect the
setting (which defaults to 1s),
while built-in probe would fail on timeout.
Use a named port
You can use a named
port
for HTTP and TCP probes. (gRPC probes do not support named ports).
ports
:
-
name
:
liveness-port
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
Protect slow starting containers with startup probes
Sometimes, you have to deal with legacy applications that might require
an additional startup time on their first initialization.
In such cases, it can be tricky to set up liveness probe parameters without
compromising the fast response to deadlocks that motivated such a probe.
The trick is to set up a startup probe with the same command, HTTP or TCP
check, with a
failureThreshold * periodSeconds
long enough to cover the
worse case startup time.
So, the previous example would become:
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
failureThreshold
:
1
:
10
startupProbe
:
httpGet
:
path
:
/healthz
port
:
:
30
:
10
Thanks to the startup probe, the application will have a maximum of 5 minutes
(30 * 10 = 300s) to finish its startup.
Once the startup probe has succeeded once, the liveness probe takes over to
provide a fast response to container deadlocks.
If the startup probe never succeeds, the container is killed after 300s and
subject to the pod's
.
Define readiness probes
Sometimes, applications are temporarily unable to serve traffic.
For example, an application might need to load large data or configuration
files during startup, or depend on external services after startup.
In such cases, you don't want to kill the application,
but you don't want to send it requests either. Kubernetes provides
readiness probes to detect and mitigate these situations. A pod with containers
reporting that they are not ready does not receive traffic through Kubernetes
Services.
Note:
Readiness probes runs on the container during its whole lifecycle.
Caution:
Liveness probes
do not
wait for readiness probes to succeed. If you want to wait before executing a liveness probe you should use initialDelaySeconds or a startupProbe.
Readiness probes are configured similarly to liveness probes. The only difference
is that you use the
field instead of the
field.
:
exec
:
command
:
- cat
:
5
:
5
Configuration for HTTP and TCP readiness probes also remains identical to
liveness probes.
Readiness and liveness probes can be used in parallel for the same container.
Using both can ensure that traffic does not reach a container that is not ready
for it, and that containers are restarted when they fail.
Configure Probes
Probes
have a number of fields that
you can use to more precisely control the behavior of startup, liveness and readiness
checks:
: Number of seconds after the container has started
before startup, liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.
: How often (in seconds) to perform the probe. Default to 10
seconds. Minimum value is 1.
: Number of seconds after which the probe times out. Defaults
to 1 second. Minimum value is 1.
successThreshold
: Minimum consecutive successes for the probe to be
considered successful after having failed. Defaults to 1. Must be 1 for liveness
and startup Probes. Minimum value is 1.
: When a probe fails, Kubernetes will
try
times before giving up. Giving up in case of liveness probe means restarting the container. In case of readiness probe the Pod will be marked Unready.
Defaults to 3. Minimum value is 1.
Note:
Before Kubernetes 1.20, the field
was not respected for exec 

probes:
probes continued running indefinitely, even past their configured deadline,
until a result was returned.
This defect was corrected in Kubernetes v1.20. You may have been relying on the previous behavior,
even without realizing it, as the default timeout is 1 second.
As a cluster administrator, you can disable the
ExecProbeTimeout
(set it to
false
)
on each kubelet to restore the behavior from older versions, then remove that override
once all the exec probes in the cluster have a
value set.
If you have pods that are impacted from the default 1 second timeout,
you should update their probe timeout so that you're ready for the
eventual removal of that feature gate.
With the fix of the defect, for exec probes, on Kubernetes
1.20+
with the
container runtime,
the process inside the container may keep running even after probe returned failure because of the timeout.
Caution:
Incorrect implementation of readiness probes may result in an ever growing number
of processes in the container, and resource starvation if this is left unchecked.
HTTP probes
have additional fields that can be set on
httpGet
:
host
: Host name to connect to, defaults to the pod IP. You probably want to
set "Host" in httpHeaders instead.
scheme
: Scheme to use for connecting to the host (HTTP or HTTPS). Defaults to HTTP.
path
: Path to access on the HTTP server. Defaults to /.
: Custom headers to set in the request. HTTP allows repeated headers.
port
: Name or number of the port to access on the container. Number must be
in the range 1 to 65535.
For an HTTP probe, the kubelet sends an HTTP request to the specified path and
port to perform the check. The kubelet sends the probe to the pod's IP address,
unless the address is overridden by the optional
host
field in
httpGet
. If
scheme
HTTPS
, the kubelet sends an HTTPS request skipping the
certificate verification. In most scenarios, you do not want to set the
host
field.
Here's one scenario where you would set it. Suppose the container listens on 127.0.0.1
and the Pod's
field is true. Then
host
, under
httpGet
, should be set
to 127.0.0.1. If your pod relies on virtual hosts, which is probably the more common
case, you should not use
host
, but rather set the
Host
header in
.
For an HTTP probe, the kubelet sends two request headers in addition to the mandatory
Host
header:
User-Agent
, and
Accept
. The default values for these headers are
kube-probe/1.26
(where
1.26
is the version of the kubelet ), and
*/*
You can override the default headers by defining
.httpHeaders
for the probe; for example
:
httpGet
:
:
-
name
:
Accept
value
:
:
httpGet
:
:
-
name
:
value
:
MyUserAgent
You can also remove these two headers by defining them with an empty value.
:
httpGet
:
:
-
name
:
Accept
value
:
""
:
httpGet
:
:
-
name
:
value
:
""
TCP probes
For a TCP probe, the kubelet makes the probe connection at the node, not in the pod, which
means that you can not use a service name in the
host
parameter since the kubelet is unable
to resolve it.
Probe-level
Prior to release 1.21, the pod-level
was used
for terminating a container that failed its liveness or startup probe. This
coupling was unintended and may have resulted in failed containers taking an
unusually long time to restart when a pod-level
was set.
In 1.25 and beyond, users can specify a probe-level
as part of the probe specification. When both a pod- and probe-level
are set, the kubelet will use the probe-level value.
Note:
Beginning in Kubernetes 1.25, the
ProbeTerminationGracePeriod
feature is enabled
by default. For users 

choosing to disable this feature, please note the following:
The
feature gate is only available on the API Server.
The kubelet always honors the probe-level
field if
it is present on a Pod.
If you have existing Pods where the
field is set and
you no longer wish to use per-probe termination grace periods, you must delete
those existing Pods.
When you (or the control plane, or some other component) create replacement
Pods, and the feature gate
is disabled, then the
API server ignores the Probe-level
field, even if
a Pod or pod template specifies it.
For example,
spec
:
:
3600
# pod-level
:
-
name
:
test
image
:
...
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
:
1
:
60
# Override pod-level terminationGracePeriodSeconds #
:
60
cannot be set for readiness probes.
It will be rejected by the API server.
Container Probes
.
You can also read the API references for:
Pod
, and specifically:
container(s)
probe(s)
15 - Assign Pods to Nodes
This page shows how to assign a Kubernetes Pod to a particular node in a
minikube
.
Add a label to a node
List the
nodes
in your cluster, along with their labels:
kubectl get nodes --show-labels
NAME      STATUS    ROLES    AGE     VERSION        LABELS
worker0   Ready     &lt;none&gt;   1d      v1.13.0        ...,kubernetes.io/hostname
=
worker0
worker1   Ready     &lt;none&gt;   1d      v1.13.0        ...,kubernetes.io/hostname
=
worker1
worker2   Ready     &lt;none&gt;   1d      v1.13.0        ...,kubernetes.io/hostname
=
worker2
Choose one of your nodes, and add a label to it:
kubectl label nodes &lt;your-node-name&gt;
disktype
=
ssd
where
is the name of your chosen node.
Verify that your chosen node has a
disktype=ssd
label:
worker0   Ready     &lt;none&gt;   1d      v1.13.0        ...,disktype
=
ssd,kubernetes.io/hostname
=
worker0
=
worker1
=
worker2
In the preceding output, you can see that the
worker0
node has a
label.
Create a pod that gets scheduled to your chosen node
This pod configuration file describes a pod that has a node selector,
disktype: ssd
. This means that the pod will get scheduled on a node that has
a
label.
pods/pod-nginx.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
labels
:
env
:
test
spec
:
:
-
name
:
nginx
image
:
nginx
:
:
disktype
:
ssd
Use the configuration file to create a pod that will get scheduled on your
chosen node:
kubectl apply -f https://k8s.io/examples/pods/pod-nginx.yaml
Verify that the pod is running on your chosen node:
=
wide
NAME     READY     STATUS    RESTARTS   AGE    IP           NODE
nginx    1/1       Running
0
13s    10.200.0.4   worker0
Create a pod that gets scheduled to specific node
You can also sc

hedule a pod to one specific node via setting
nodeName
.
pods/pod-nginx-specific-node.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
nodeName
:
foo-node
# schedule pod to specific node
:
-
name
:
nginx
image
:
nginx
:
Use the configuration file to create a pod that will get scheduled on
foo-node
only.
labels and selectors
.
nodes
.
16 - Assign Pods to Nodes using Node Affinity
This page shows how to assign a Kubernetes Pod to a particular node using Node Affinity in a
minikube
Your Kubernetes server must be at or later than version v1.10.
.
List the nodes in your cluster, along with their labels:
=
worker0
=
worker1
=
worker2
disktype
=
ssd
where
label:
worker0   Ready     &lt;none&gt;   1d      v1.13.0        ...,disktype=ssd,kubernetes.io/hostname=worker0
worker1   Ready     &lt;none&gt;   1d      v1.13.0        ...,kubernetes.io/hostname=worker1
worker2   Ready     &lt;none&gt;   1d      v1.13.0        ...,kubernetes.io/hostname=worker2
worker0
node has a
label.
Schedule a Pod using required node affinity
This manifest describes a Pod that has a
requiredDuringSchedulingIgnoredDuringExecution
node affinity,
.
This means that the pod will get scheduled only on a node that has a
label.
pods/pod-nginx-required-affinity.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
affinity
:
nodeAffinity
:
:
nodeSelectorTerms
:
-
:
-
key
:
disktype
operator
:
In
values
:
- ssd
:
-
name
:
nginx
image
:
nginx
:
Apply the manifest to create a Pod that is scheduled onto your
kubectl apply -f https://k8s.io/examples/pods/pod-nginx-required-affinity.yaml
=
wide
nginx    1/1       Running   0          13s    10.200.0.4   worker0
Schedule a Pod using preferred node affinity
preferredDuringSchedulingIgnoredDuringExecution
.
This means that the pod will prefer a node that has a
label.
pods/pod-nginx-preferred-affinity.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
affinity
:
:
:
-
weight
:
1
preference
:
:
-
key
:
disktype
operator
:
In
values
:
- ssd
:
-
name
:
nginx
image
:
nginx
:
kubectl apply -f https://k

8s.io/examples/pods/pod-nginx-preferred-affinity.yaml
=
wide
Node Affinity
.
17 - Configure Pod Initialization
This page shows how to use an Init Container to initialize a Pod before an
application Container runs.
minikube
.
Create a Pod that has an Init Container
In this exercise you create a Pod that has one application Container and one
Init Container. The init container runs to completion before the application
container starts.
pods/init-containers.yaml
:
v1
kind
:
Pod
metadata
:
name
:
init-demo
spec
:
:
-
name
:
nginx
image
:
nginx
ports
:
-
:
80
:
-
name
:
workdir
mountPath
:
# These containers are run during pod initialization
:
-
name
:
install
image
:
command
:
- wget
-
"-O"
-
"/work-dir/index.html"
- http://info.cern.ch
:
-
name
:
workdir
mountPath
:
"/work-dir"
dnsPolicy
:
Default
volumes
:
-
name
:
workdir
emptyDir
:
{}
In the configuration file, you can see that the Pod has a Volume that the init
container and the application container share.
The init container mounts the
shared Volume at
/work-dir
, and the application container mounts the shared
Volume at
. The init container runs the following command
and then terminates:
wget -O /work-dir/index.html http://info.cern.ch
Notice that the init container writes the
file in the root directory
of the nginx server.
kubectl apply -f https://k8s.io/examples/pods/init-containers.yaml
Verify that the nginx container is running:
kubectl get pod init-demo
The output shows that the nginx container is running:
NAME        READY     STATUS    RESTARTS   AGE
init-demo   1/1       Running   0          1m
Get a shell into the nginx container running in the init-demo Pod:
kubectl exec -it init-demo -- /bin/bash
root@nginx:~# apt-get update
root@nginx:~# apt-get install curl
root@nginx:~# curl localhost
The output shows that nginx is serving the web page that was written by the init container:
&lt;html&gt;&lt;head&gt;&lt;/head&gt;&lt;body&gt;&lt;header&gt;
&lt;title&gt;http://info.cern.ch&lt;/title&gt;
&lt;/header&gt;

&lt;h1&gt;http://info.cern.ch - home of the first website&lt;/h1&gt;
  ...
  &lt;li&gt;&lt;a href="http://info.cern.ch/hypertext/WWW/TheProject.html"&gt;Browse the first website&lt;/a&gt;&lt;/li&gt;
  ...
communicating between Containers running in the same Pod
.
.
Volumes
.
Debugging Init Containers
18 - Attach Handlers to Container Lifecycle Events
This page shows how to attach handlers to Container lifecycle events. Kubernetes supports
the postStart and preStop events. Kubernetes sends the postStart event immediately
after a Container is started, and it sends the preStop event immediately before the
Container is terminated. A Container may specify one handler per event.
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not a

cting as control plane hosts. If you do not already have a
minikube
.
Define postStart and preStop handlers
In this exercise, you create a Pod that has one Container. The Container has handlers
for the postStart and preStop events.
pods/lifecycle-events.yaml
:
v1
kind
:
Pod
metadata
:
name
:
lifecycle-demo
spec
:
:
-
name
:
lifecycle-demo-container
image
:
nginx
lifecycle
:
postStart
:
exec
:
command
:
[
"/bin/sh"
,
"-c"
,
"echo Hello from the postStart handler &gt; /usr/share/message"
]
preStop
:
exec
:
command
:
[
"/bin/sh"
,
"-c"
,
"nginx -s quit; while killall -0 nginx; do sleep 1; done"
]
In the configuration file, you can see that the postStart command writes a
message
file to the Container's
/usr/share
directory. The preStop command shuts down
nginx gracefully. This is helpful if the Container is being terminated because of a failure.
kubectl apply -f https://k8s.io/examples/pods/lifecycle-events.yaml
Verify that the Container in the Pod is running:
kubectl get pod lifecycle-demo
Get a shell into the Container running in your Pod:
kubectl exec -it lifecycle-demo -- /bin/bash
postStart
handler created the
message
file:
root@lifecycle-demo:/# cat /usr/share/message
The output shows the text written by the postStart handler:
Hello from the postStart handler
Kubernetes sends the postStart event immediately after the Container is created.
There is no guarantee, however, that the postStart handler is called before
the Container's entrypoint is called. The postStart handler runs asynchronously
relative to the Container's code, but Kubernetes' management of the container
blocks until the postStart handler completes. The Container's status is not
set to RUNNING until the postStart handler completes.
Kubernetes sends the preStop event immediately before the Container is terminated.
Kubernetes' management of the Container blocks until the preStop handler completes,
unless the Pod's grace period expires. For more details, see
.
Note:
Kubernetes only sends the preStop event when a Pod or a container in the Pod is
.
This means that the preStop hook is not invoked when the Pod is
completed
.
About this limitation, please see
Container hooks
for the detail.
Container lifecycle hooks
.
lifecycle of a Pod
.
Reference
Lifecycle
Container
See
in
PodSpec
19 - Configure a Pod to Use a ConfigMap
Many applications rely on configuration which is used during either application initialization or runtime.
Most of the times there is a requirement to adjust values assigned to configuration parameters.
ConfigMaps are the Kubernetes way to inject application pods with configuration data.
ConfigMaps allow you to decouple configuration artifacts from image content to keep containerized applications portable. This page provides a series of usage examples demonstrating how to create ConfigMaps and configure Pods using data stored in ConfigMaps.
minikube
.
Create a ConfigMap
You can use either
kubectl create configmap
or a ConfigMap generator in
kustomization.yaml
to create a ConfigMap. Note that
kubectl
starts to support
since 1.14.
Create a ConfigMap Using kubectl create configmap
Use the
kubectl crea

te configmap
command to create ConfigMaps from
directories
,
files
, or
literal values
:
kubectl create configmap &lt;map-name&gt; &lt;data-source&gt;
where &lt;map-name&gt; is the name you want to assign to the ConfigMap and &lt;data-source&gt; is the directory, file, or literal value to draw the data from.
The name of a ConfigMap object must be a valid
.
When you are creating a ConfigMap based on a file, the key in the &lt;data-source&gt; defaults to the basename of the file, and the value defaults to the file content.
You can use
or
to retrieve information
about a ConfigMap.
Create ConfigMaps from directories
You can use
to create a ConfigMap from multiple files in the same directory. When you are creating a ConfigMap based on a directory, kubectl identifies files whose basename is a valid key in the directory and packages each of those files into the new ConfigMap. Any directory entries except regular files are ignored (e.g. subdirectories, symlinks, devices, pipes, etc).
# Create the local directory
mkdir -p configure-pod-container/configmap/
# Download the sample files into `configure-pod-container/configmap/` directory
wget https://kubernetes.io/examples/configmap/game.properties -O configure-pod-container/configmap/game.properties
wget https://kubernetes.io/examples/configmap/ui.properties -O configure-pod-container/configmap/ui.properties
# Create the configmap
kubectl create configmap game-config --from-file
=
configure-pod-container/configmap/
The above command packages each file, in this case,
game.properties
and
ui.properties
in the
directory into the game-config ConfigMap. You can display details of the ConfigMap using the following command:
kubectl describe configmaps game-config
Name:         game-config

Data
====
game.properties:
----
enemies=aliens
lives=3
enemies.cheat=true
enemies.cheat.level=noGoodRotten
secret.code.passphrase=UUDDLRLRBABAS
secret.code.allowed=true
secret.code.lives=30
ui.properties:
----
color.good=purple
color.bad=yellow
allow.textmode=true
how.nice.to.look=fairlyNice
The
and
directory are represented in the
data
kubectl get configmaps game-config -o yaml
:
v1
kind
:
ConfigMap
metadata
:
:
2016-02-18T18:52:05Z
name
:
game-config
namespace
:
default
:
"516"
uid
:
b4952dc3-d670-11e5-8cd0-68f728db1985
data
:
:
|
lives=3
:
|
Create ConfigMaps from files
You can use
to create a ConfigMap from an individual file, or from multiple files.
kubectl create configmap game-config-2 --from-file
=
configure-pod-container/configmap/game.properties
would produce the following ConfigMap:
kubectl describe configmaps game-config-2
where the output is similar to this:
Name:         game-config-2

Data
====
----
lives=3
You can pass in the
argument multiple times to create a ConfigMap from multiple data sources.
=
configure-pod-container/configmap/game.properties --from-file
=
configure-pod-container/configmap/ui.properties
You can display details of the
game-config-2
ConfigMap using the following command:
kubect

l describe configmaps game-config-2

Data
====
----
lives=3
----
When
kubectl
creates a ConfigMap from inputs that are not ASCII or UTF-8, the tool puts these into the
field of the ConfigMap, and not in
data
. Both text and binary data sources can be combined in one ConfigMap.
If you want to view the
keys (and their values) in a ConfigMap, you can run
kubectl get configmap -o jsonpath='{.binaryData}' &lt;name&gt;
.
Use the option
--from-env-file
to create a ConfigMap from an env-file, for example:
# Env-files contain a list of environment variables.
# These syntax rules apply:
#   Each line in an env file has to be in VAR=VAL format.
#   Lines beginning with # (i.e. comments) are ignored.
#   Blank lines are ignored.
#   There is no special handling of quotation marks (i.e. they will be part of the ConfigMap value)).
wget https://kubernetes.io/examples/configmap/game-env-file.properties -O configure-pod-container/configmap/game-env-file.properties
wget https://kubernetes.io/examples/configmap/ui-env-file.properties -O configure-pod-container/configmap/ui-env-file.properties
# The env-file `game-env-file.properties` looks like below
cat configure-pod-container/configmap/game-env-file.properties
enemies
=
aliens
lives
=
3
allowed
=
"true"
# This comment and the empty line above it are ignored
kubectl create configmap game-config-env-file
\
=
configure-pod-container/configmap/game-env-file.properties
kubectl get configmap game-config-env-file -o yaml
:
v1
kind
:
ConfigMap
metadata
:
:
2017-12-27T18:36:28Z
name
:
game-config-env-file
namespace
:
default
:
"809965"
uid
:
d9d1ca5b-eb34-11e7-887b-42010a8002b8
data
:
allowed
:
'"true"'
enemies
:
aliens
lives
:
"3"
Starting with Kubernetes v1.23,
kubectl
supports the
argument to be
specified multiple times to create a ConfigMap from multiple data sources.
kubectl create configmap config-multi-env-files
\
=
\
=
configure-pod-container/configmap/ui-env-file.properties
kubectl get configmap config-multi-env-files -o yaml
:
v1
kind
:
ConfigMap
metadata
:
:
2017-12-27T18:38:34Z
name
:
config-multi-env-files
namespace
:
default
:
"810136"
uid
:
252c4572-eb35-11e7-887b-42010a8002b8
data
:
allowed
:
'"true"'
color
:
purple
enemies
:
aliens
how
:
fairlyNice
lives
:
"3"
textmode
:
"true"
Define the key to use when creating a ConfigMap from a file
You can define a key other than the file name to use in the
data
section of your ConfigMap when using the
argument:
kubectl create configmap game-config-3 --from-file
=
&lt;my-key-name&gt;
=
&lt;path-to-file&gt;
where
is the key you want to use in the ConfigMap and
is the location of the data source file you want the key to represent.
=
game-special-key
=
kubectl get configmaps game-config-3 -o yaml
:
v1
kind
:
ConfigMap
metadata
:
:
2016-02-18T18:54:22Z
name
:
game-config-3
namespace
:
default
:
"530"
uid
:
05f8da22-d67

1-11e5-8cd0-68f728db1985
data
:
:
|
lives=3
Create ConfigMaps from literal values
You can use
with the
argument to define a literal value from the command line:
kubectl create configmap special-config --from-literal
=
special.how
=
very --from-literal
=
special.type
=
charm
You can pass in multiple key-value pairs. Each pair provided on the command line is represented as a separate entry in the
data
kubectl get configmaps special-config -o yaml
:
v1
kind
:
ConfigMap
metadata
:
:
2016-02-18T19:14:38Z
name
:
special-config
namespace
:
default
:
"651"
uid
:
dadce046-d673-11e5-8cd0-68f728db1985
data
:
:
very
:
charm
Create a ConfigMap from generator
kubectl
supports
You can also create a ConfigMap from generators and then apply it to create the object on
the Apiserver. The generators
should be specified in a
inside a directory.
Generate ConfigMaps from files
For example, to generate a ConfigMap from files
# Create a kustomization.yaml file with ConfigMapGenerator
cat
&lt;&lt;EOF &gt;./kustomization.yaml
configMapGenerator:
- name: game-config-4
files:
- configure-pod-container/configmap/game.properties
EOF
Apply the kustomization directory to create the ConfigMap object.
kubectl apply -k .
configmap/game-config-4-m9dm2f92bt created
You can check that the ConfigMap was created like this:
kubectl get configmap
NAME                       DATA   AGE
game-config-4-m9dm2f92bt
1
37s
kubectl describe configmaps/game-config-4-m9dm2f92bt
Name:         game-config-4-m9dm2f92bt
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
{
:
"v1"
,
"data"
:
{
"game.properties"
:
"enemies=aliens\nlives=3\nenemies.cheat=true\nenemies.cheat.level=noGoodRotten\nsecret.code.p...
Data
====
----
lives=3
Events:  &lt;none&gt;
Note that the generated ConfigMap name has a suffix appended by hashing the contents. This ensures that a
new ConfigMap is generated each time the content is modified.
Define the key to use when generating a ConfigMap from a file
You can define a key other than the file name to use in the ConfigMap generator.
with the key
cat
- name: game-config-5
files:
- game-special-key=configure-pod-container/configmap/game.properties
EOF
configmap/game-config-5-m67dt67794 created
Generate ConfigMaps from Literals
To generate a ConfigMap from literals
special.type=charm
and
special.how=very
,
you can specify the ConfigMap generator in
as
cat
- name: special-config-2
literals:
- special.how=very
- special.type=charm
EOF
configmap/special-config-2-c92b5mmcf2 created
Define container environment variables using ConfigMap data
Define a container environment variable with data from a single ConfigMap
Define an environment variable as a key-value pair in a ConfigMap:
=
=
very
Assign the
valu

e defined in the ConfigMap to the
SPECIAL_LEVEL_KEY
pods/pod-single-configmap-env-variable.yaml
:
v1
kind
:
Pod
metadata
:
name
:
dapi-test-pod
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
# Define the environment variable
-
name
:
valueFrom
:
configMapKeyRef
:
# The ConfigMap containing the value you want to assign to SPECIAL_LEVEL_KEY
name
:
# Specify the key associated with the value
key
:
:
Never
kubectl create -f https://kubernetes.io/examples/pods/pod-single-configmap-env-variable.yaml
Now, the Pod's output includes environment variable
SPECIAL_LEVEL_KEY=very
.
Define container environment variables with data from multiple ConfigMaps
As with the previous example, create the ConfigMaps first.
configmap/configmaps.yaml
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
---
:
v1
kind
:
ConfigMap
metadata
:
name
:
env-config
namespace
:
default
data
:
log_level
:
INFO
Create the ConfigMap:
kubectl create -f https://kubernetes.io/examples/configmap/configmaps.yaml
pods/pod-multiple-configmap-env-variable.yaml
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
LOG_LEVEL
valueFrom
:
:
name
:
key
:
log_level
:
Never
kubectl create -f https://kubernetes.io/examples/pods/pod-multiple-configmap-env-variable.yaml
Now, the Pod's output includes environment variables
and
LOG_LEVEL=INFO
.
Configure all key-value pairs in a ConfigMap as container environment variables
Note:
Create a ConfigMap containing multiple key-value pairs.
configmap/configmap-multikeys.yaml
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
SPECIAL_LEVEL
:
very
SPECIAL_TYPE
:
charm
kubectl create -f https://kubernetes.io/examples/configmap/configmap-multikeys.yaml
Use
envFrom
to define all of the ConfigMap's data as container environment variables. The key from the ConfigMap becomes the environment variable name in the Pod.
pods/pod-configmap-envFrom.yaml
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
envFrom
:
-
configMapRef
:
name
:
:
Never
kubectl create -f https://kubernetes.io/examples/pods/pod-configmap-envFrom.yaml
SPECIAL_LEVEL=very
and
SPECIAL_TYPE=charm
.
Use ConfigMap-defined environment variables in Pod commands
You can use ConfigMap-defined environment variables in the
command
and
args
of a container using the
Kubernetes substitution syntax.
For example, the following Pod specification
pods/pod-configmap-env-var-valueFrom.yaml
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/echo"
,
"$(SPECIAL_LEVEL_KEY) $(SPECIAL_TYPE_KEY)"
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
SPECIAL_TYPE_KEY
valueFrom
:
:
name
:
key
:
:
Never
created by running
kubectl create -f https://kubernetes.io/examples/pods/pod-configmap-env-var-valueFrom.yaml
produces the following output in the
container:
very charm
Add ConfigMap data to a 

Volume
As explained in
, when you create a ConfigMap using
, the filename becomes a key stored in the
data
section of the ConfigMap. The file contents become the key's value.
The examples in this section refer to a ConfigMap named special-config, shown below.
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
:
charm
Populate a Volume with data stored in a ConfigMap
Add the ConfigMap name under the
volumes
section of the Pod specification.
This adds the ConfigMap data to the directory specified as
volumeMounts.mountPath
(in this case,
/etc/config
).
The
command
section lists directory files with names that match the keys in ConfigMap.
pods/pod-configmap-volume.yaml
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"ls /etc/config/"
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
# Provide the name of the ConfigMap containing the files you want
# to add to the container
name
:
:
Never
kubectl create -f https://kubernetes.io/examples/pods/pod-configmap-volume.yaml
When the pod runs, the command
ls /etc/config/
produces the output below:
Caution:
If there are some files in the
/etc/config/
directory, they will be deleted.
Note:
Text data is exposed as files using the UTF-8 character encoding. To use some other character encoding, use binaryData.
Add ConfigMap data to a specific path in the Volume
Use the
path
field to specify the desired file path for specific ConfigMap items.
In this case, the
item will be mounted in the
volume at
/etc/config/keys
.
pods/pod-configmap-volume-specific-key.yaml
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"cat /etc/config/keys"
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
items
:
-
key
:
path
:
keys
:
Never
kubectl create -f https://kubernetes.io/examples/pods/pod-configmap-volume-specific-key.yaml
cat /etc/config/keys
very
Caution:
Like before, all previous files in the
directory will be deleted.
Project keys to specific paths and file permissions
You can project keys to specific paths and specific permissions on a per-file
basis. The
Secrets
user guide explains the syntax.
Understanding ConfigMaps and Pods
The ConfigMap API resource stores configuration data as key-value pairs. The data can be consumed in pods or provide the configurations for system components such as controllers. ConfigMap is similar to
Secrets
, but provides a means of working with strings that don't contain sensitive information. Users and system components alike can store configuration data in ConfigMap.
Note:
ConfigMaps should reference properties files, not replace them. Think of the ConfigMap as representing something similar to the Linux
/etc
directory and its contents. For example, if you create a
Kubernetes Volume
from a ConfigMap, each data item in the ConfigMap is represented by an individual file in the volume.
The ConfigMap's
data
field contains the configuration data. As shown in the example below, this can be simple -- like individual properties defined using
-- or complex -- like configuration files or JSON blobs defined using
.
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
example-config
namespace

:
default
data
:
# example of a simple property defined using --from-literal
example.property.1
:
hello
example.property.2
:
world
# example of a complex property defined using --from-file
example.property.file
:
|-
property.1=value-1
property.2=value-2
property.3=value-3
Restrictions
You must create the
ConfigMap
object before you reference it in a Pod specification. Alternatively, mark the ConfigMap reference as
optional
in the Pod spec (see
Optional ConfigMaps
). If you reference a ConfigMap that doesn't exist and you don't mark the reference as
optional
, the Pod won't start. Similarly, references to keys that don't exist in the ConfigMap will also prevent the Pod from starting, unless you mark the key references as
optional
.
If you use
envFrom
to define environment variables from ConfigMaps, keys that are considered invalid will be skipped. The pod will be allowed to start, but the invalid names will be recorded in the event log (
InvalidVariableNames
). The log message lists each skipped key. For example:
LASTSEEN FIRSTSEEN COUNT NAME          KIND  SUBOBJECT  TYPE      REASON                            SOURCE                MESSAGE
0s       0s        1     dapi-test-pod Pod              Warning   InvalidEnvironmentVariableNames   {kubelet, 127.0.0.1}  Keys [1badkey, 2alsobad] from the EnvFrom configMap default/myconfig were skipped since they are considered invalid environment variable names.
ConfigMaps reside in a specific
Namespace
. A ConfigMap can only be referenced by pods residing in the same namespace.
You can't use ConfigMaps for
static pods
, because the Kubelet does not support this.
You can mark a reference to a ConfigMap as
optional
in a Pod specification.
If the ConfigMap doesn't exist, the configuration for which it provides data in the Pod (e.g. environment variable, mounted volume) will be empty.
If the ConfigMap exists, but the referenced key is non-existent the data is also empty.
For example, the following Pod specification marks an environment variable from a ConfigMap as optional:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
gcr.io/google_containers/busybox
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
a-config
key
:
akey
optional
:
true
# mark the variable as optional
:
Never
If you run this pod, and there is no ConfigMap named
a-config
, the output is empty.
If you run this pod, and there is a ConfigMap named
a-config
but that ConfigMap doesn't have
a key named
akey
, the output is also empty. If you do set a value for
akey
in the
a-config
ConfigMap, this pod prints that value and then terminates.
You can also mark the volumes and files provided by a ConfigMap as optional. Kubernetes always creates the mount paths for the volume, even if the referenced ConfigMap or key doesn't exist. For example, the following
Pod specification marks a volume that references a ConfigMap as optional:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"ls /etc/config"
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
no
-config
optional
:
true
# mark the source ConfigMap as optional
:
Never
Mounted ConfigMaps are updated automatically
When a mounted ConfigMap is updated, the projected content is eventually updated too. This applies in the case where an optionally referenced ConfigMap comes into
existence after a pod has started.
The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. However, it uses its local TTL-based cache for getting the current value of the
ConfigMap. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are pr

ojected to the pod can be as long as
kubelet sync period (1 minute by default) + TTL of ConfigMaps cache (1 minute by default) in kubelet.
Note:
A container using a ConfigMap as a
subPath
volume will not receive ConfigMap updates.
Follow a real world example of
.
20 - Share Process Namespace between Containers in a Pod
This page shows how to configure process namespace sharing for a pod. When
process namespace sharing is enabled, processes in a container are visible
to all other containers in the same pod.
You can use this feature to configure cooperating containers, such as a log
handler sidecar container, or to troubleshoot container images that don't
include debugging utilities like a shell.
minikube
Configure a Pod
Process namespace sharing is enabled using the
shareProcessNamespace
field of
.spec
for a Pod. For example:
pods/share-process-namespace.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
true
:
-
name
:
nginx
image
:
nginx
-
name
:
shell
image
:
:
:
add
:
- SYS_PTRACE
stdin
:
true
tty
:
true
Create the pod
nginx
on your cluster:
kubectl apply -f https://k8s.io/examples/pods/share-process-namespace.yaml
Attach to the
shell
container and run
ps
:
kubectl attach -it nginx -c shell
If you don't see a command prompt, try pressing enter. In the container shell:
# run this inside the "shell" container
ps ax
    1 root      0:00 /pause
    8 root      0:00 nginx: master process nginx -g daemon off;
   14 101       0:00 nginx: worker process
   15 root      0:00 sh
   21 root      0:00 ps ax
You can signal processes in other containers. For example, send
SIGHUP
to
nginx
to restart the worker process. This requires the
SYS_PTRACE
capability.
kill
-HUP
8
# change "8" to match the PID of the nginx leader process, if necessary
ps ax
   22 101       0:00 nginx: worker process
   23 root      0:00 ps ax
It's even possible to access the file system of another container using the
/proc/$pid/root
link.
# change "8" to the PID of the Nginx process, if necessary
head /proc/8/root/etc/nginx/nginx.conf
user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
Understanding process namespace sharing
Pods share many resources so it makes sense they would also share a process
namespace. Some containers may expect to be isolated from others, though,
so it's important to understand the differences:
The container process no longer has PID 1.
Some containers refuse
to start without PID 1 (for example, containers using
systemd
) or run
commands like
kill -HUP 1
to signal the container process. In pods with a
shared process namespace,
kill -HUP 1
will signal the pod sandbox
(
/pause
in the above example).
Processes are visible to other containers in the pod.
This includes all
information visible in
/proc
, such as passwords that were passed as arguments
or environment variables. These are protected only by regular Unix permissions.
Container filesystems are visible to other containers in the pod through the
link.
This makes debugging easier, but it also mea

ns
that filesystem secrets are protected only by filesystem permissions.
21 - Use a User Namespace With a Pod
This page shows how to configure a user namespace for stateless pods. This
allows to isolate the user running inside the container from the one in the
host.
A process running as root in a container can run as a different (non-root) user
in the host; in other words, the process has full privileges for operations
inside the user namespace, but is unprivileged for operations outside the
You can use this feature to reduce the damage a compromised container can do to
the host or other pods in the same node. There are
several security
vulnerabilities
rated either
HIGH
or
CRITICAL
that were not
exploitable when user namespaces is active. It is expected user namespace will
mitigate some future vulnerabilities too.
Without using a user namespace a container running as root, in the case of a
container breakout, has root privileges on the node. And if some capability were
granted to the container, the capabilities are valid on the host too. None of
this is true when user namespaces are used.
minikube
Your Kubernetes server must be at or later than version v1.25.
.
The node OS needs to be Linux
You need to exec commands in the host
You need to be able to exec into pods
Feature gate
UserNamespacesStatelessPodsSupport
need to be enabled.
In addition, support is needed in the
to use this feature with Kubernetes stateless pods:
CRI-O: v1.25 has support for user namespaces.
Please note that
if your container runtime doesn't support user namespaces, the
new
pod.spec
field will be silently ignored and the pod will be created without
user namespaces.
Run a Pod that uses a user namespace
A user namespace for a stateless pod is enabled setting the
hostUsers
field of
.spec
to
false
pods/user-namespaces-stateless.yaml
:
v1
kind
:
Pod
metadata
:
name
:
userns
spec
:
hostUsers
:
false
:
-
name
:
shell
command
:
[
"sleep"
,
]
image
:
debian
Create the pod on your cluster:
kubectl apply -f https://k8s.io/examples/pods/user-namespaces-stateless.yaml
Attach to the container and run
readlink /proc/self/ns/user
:
kubectl attach -it userns bash
And run the command. The output is similar to this:
user:[4026531837]
cat /proc/self/uid_map
0          0 4294967295
Then, open a shell in the host and run the same command.
The output must be different. This means the host and the pod are using a
different user namespace. When user namespaces are not enabled, the host and the
pod use the same user namespace.
If you are running the kubelet inside a user namespace, you need to compare the
output from running the command in the pod to the output of running in the host:
readlink /proc/$pid/ns/user
user:[4026534732]
replacing
$pid
with the kubelet PID.
22 - Create static Pods
are managed directly by the kubelet daemon on a specific node,
without the
API server
observing them.
Unlike Pods that are managed by the control plane (for example, a
);
instead, the kubelet watches each static Pod (and restarts it if it fails).
Static Pods are always bound to one
Kubelet
on a specific node.
The kubelet automatically tries to create a
mirror Pod
on the Kubernetes API server for each static Pod.
This means that the Pods running on a node are visible on the

 API server,
but cannot be controlled from there.
The Pod names will be suffixed with the node hostname with a leading hyphen.
Note:
If you are running clustered Kubernetes and are using static
Pods to run a Pod on every node, you should probably be using a
DaemonSet
instead.
Note:
The
spec
of a static Pod cannot refer to other API objects
(e.g.,
,
ConfigMap
,
Secret
, etc).
Note:
Static pods do not support
ephemeral containers
.
minikube
.
This page assumes you're using
CRI-O
to run Pods,
and that your nodes are running the Fedora operating system.
Instructions for other distributions or Kubernetes installations may vary.
Create a static pod
You can configure a static Pod with either a
file system hosted configuration file
or a
web hosted configuration file
.
Filesystem-hosted static Pod manifest
Manifests are standard Pod definitions in JSON or YAML format in a specific directory. Use the
staticPodPath: &lt;the directory&gt;
,
which periodically scans the directory and creates/deletes static Pods as YAML/JSON files appear/disappear there.
Note that the kubelet will ignore files starting with dots when scanning the specified directory.
For example, this is how to start a simple web server as a static Pod:
Choose a node where you want to run the static Pod. In this example, it's
my-node1
.
ssh my-node1
Choose a directory, say
and place a web server Pod definition there, for example
/etc/kubernetes/manifests/static-web.yaml
:
# Run this command on the node where kubelet is running
mkdir -p /etc/kubernetes/manifests/
cat
&lt;&lt;EOF &gt;/etc/kubernetes/manifests/static-web.yaml
kind: Pod
metadata:
name: static-web
labels:
role: myrole
spec:
- name: web
image: nginx
ports:
- name: web
containerPort: 80
protocol: TCP
EOF
Configure your kubelet on the node to use this directory by running it with
--pod-manifest-path=/etc/kubernetes/manifests/
argument. On Fedora edit
/etc/kubernetes/kubelet
to include this line:
KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --pod-manifest-path=/etc/kubernetes/manifests/"
or add the
.
Restart the kubelet. On Fedora, you would run:
# Run this command on the node where the kubelet is running
Web-hosted static pod manifest
Kubelet periodically downloads a file specified by
--manifest-url=&lt;URL&gt;
argument
and interprets it as a JSON/YAML file that contains Pod definitions.
Similar to how
filesystem-hosted manifests
work, the kubelet
refetches the manifest on a schedule. If there are changes to the list of static
Pods, the kubelet applies them.
To use this approach:
Create a YAML file and store it on a web server so that you can pass the URL of that file to the kubelet.
:
v1
kind
:
Pod
metadata
:
name
:
static-web
labels
:
role
:
myrole
spec
:
:
-
name
:
web
image
:
nginx
ports
:
-
name
:
web
:
80
protocol
:
TCP
Configure the kubelet on your selected node to use this web manifest by running it with
--manifest-url=&lt;manifest-url&gt;
. On Fedora, edit
KUBELET_ARGS="--cluster-dns=10.254.0.10 --cluster-domain=kube.local --manifest-url=&lt;manifest-url&gt;"
Observe static pod behavior
When the kubelet starts, it aut

omatically starts all defined static Pods. As you have
defined a static Pod and restarted the kubelet, the new static Pod should
already be running.
You can view running containers (including static Pods) by running (on the node):
crictl ps
The output might be something like:
CONTAINER       IMAGE                                 CREATED           STATE      NAME    ATTEMPT    POD ID
129fd7d382018   docker.io/library/nginx@sha256:...    11 minutes ago    Running    web     0          34533c6729106
Note:
crictl
outputs the image URI and SHA-256 checksum.
NAME
will look more like:
docker.io/library/nginx@sha256:0d17b565c37bcbd895e9d92315a05c1c3c9a29f762b011a10c54a66cd53c9b31
.
You can see the mirror Pod on the API server:
NAME         READY   STATUS    RESTARTS        AGE
static-web   1/1     Running   0               2m
Note:
Make sure the kubelet has permission to create the mirror Pod in the API server. If not, the creation request is rejected by the API server.
Labels
from the static Pod are
propagated into the mirror Pod. You can use those labels as normal via
selectors
, etc.
If you try to use
kubectl
to delete the mirror Pod from the API server,
the kubelet
doesn't
remove the static Pod:
kubectl delete pod static-web
pod "static-web" deleted
You can see that the Pod is still running:
NAME         READY   STATUS    RESTARTS   AGE
static-web   1/1     Running   0          4s
Back on your node where the kubelet is running, you can try to stop the container manually.
You'll see that, after a time, the kubelet will notice and will restart the Pod
automatically:
# Run these commands on the node where the kubelet is running
crictl stop 129fd7d382018
# replace with the ID of your container
sleep
20
crictl ps
89db4553e1eeb   docker.io/library/nginx@sha256:...    19 seconds ago    Running    web     1          34533c6729106
Dynamic addition and removal of static pods
The running kubelet periodically scans the configured directory (
in our example) for changes and adds/removes Pods as files appear/disappear in this directory.
# This assumes you are using filesystem-hosted static Pod configuration
#
mv /etc/kubernetes/manifests/static-web.yaml /tmp
sleep
20
crictl ps
# You see that no nginx container is running
mv /tmp/static-web.yaml  /etc/kubernetes/manifests/
sleep
20
crictl ps
f427638871c35   docker.io/library/nginx@sha256:...    19 seconds ago    Running    web     1          34533c6729106
23 - Translate a Docker Compose File to Kubernetes Resources
What's Kompose? It's a conversion tool for all things compose (namely Docker Compose) to container orchestrators (Kubernetes or OpenShift).
More information can be found on the Kompose website at
http://kompose.io
.
minikube
.
Install Kompose
We have multiple ways to install Kompose. Our preferred method is downloading the binary from the latest GitHub release.
GitHub download
Build from source
CentOS package
Fedora package
Homebrew (macOS)
Kompose is released via GitHub on a three-week cycle, you can see all current releases on the
GitHub release page
.
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-lin

ux-amd64 -o kompose
# macOS
curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-darwin-amd64 -o kompose
# Windows
curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-windows-amd64.exe -o kompose.exe
chmod +x kompose
sudo mv ./kompose /usr/local/bin/kompose
Alternatively, you can download the
tarball
.
Installing using
go get
pulls from the master branch with the latest development changes.
go get -u github.com/kubernetes/kompose
Kompose is in
EPEL
CentOS repository.
EPEL
repository already installed and enabled you can do it by running
sudo yum install epel-release
.
If you have
EPEL
enabled in your system, you can install Kompose like any other package.
sudo yum -y install kompose
Kompose is in Fedora 24, 25 and 26 repositories. You can install it like any other package.
sudo dnf -y install kompose
On macOS you can install the latest release via
Homebrew
:
brew install kompose
Use Kompose
In a few steps, we'll take you from Docker Compose to Kubernetes. All
you need is an existing
docker-compose.yml
file.
Go to the directory containing your
file. If you don't have one, test using this one.
version
:
"2"
services
:
redis-master
:
image
:
registry.k8s.io/redis:e2e
ports
:
-
"6379"
redis-slave
:
image
:
gcr.io/google_samples/gb-redisslave:v3
ports
:
-
"6379"
:
- GET_HOSTS_FROM=dns
frontend
:
image
:
gcr.io/google-samples/gb-frontend:v4
ports
:
-
"80:80"
:
labels
:
kompose.service.type
:
To convert the
file to files that you can use with
kubectl
, run
kompose convert
and then
kubectl apply -f &lt;output file&gt;
.
INFO Kubernetes file "frontend-service.yaml" created
   INFO Kubernetes file "frontend-service.yaml" created
INFO Kubernetes file "redis-master-service.yaml" created
   INFO Kubernetes file "redis-master-service.yaml" created
INFO Kubernetes file "redis-slave-service.yaml" created
   INFO Kubernetes file "redis-slave-service.yaml" created
INFO Kubernetes file "frontend-deployment.yaml" created
   INFO Kubernetes file "frontend-deployment.yaml" created
INFO Kubernetes file "redis-master-deployment.yaml" created
   INFO Kubernetes file "redis-master-deployment.yaml" created
INFO Kubernetes file "redis-slave-deployment.yaml" created
   INFO Kubernetes file "redis-slave-deployment.yaml" created
kubectl apply -f frontend-service.yaml,redis-master-service.yaml,redis-slave-service.yaml,frontend-deployment.yaml,redis-master-deployment.yaml,redis-slave-deployment.yaml
service/frontend created
service/redis-master created
service/redis-slave created
deployment.apps/frontend created
deployment.apps/redis-master created
deployment.apps/redis-slave created
Your deployments are running in Kubernetes.
Access your application.
If you're already using
minikube
for your development process:
minikube service frontend
Otherwise, let's look up what IP your service is using!
kubectl describe svc frontend
Name:                   frontend
Namespace:              default
Labels:                 service=frontend
Selector:               service=frontend
Type:                   LoadBalancer
IP:                     10.0.0.183
LoadBalancer Ingress:   192.0.2.89
Port:                   80      80/TCP
NodePort:               80      31144/TCP
Endpoints:              172.17.0.4:80
Session Affinity:       None
No events.
If you're using a cloud provider, your IP will be listed next to
.
curl http://192.0.2.89
User Guide
CLI
Alternative

 Conversions
Labels
Restart
Docker Compose Versions
Kompose has support for two providers: OpenShift and Kubernetes.
You can choose a targeted provider using global option
--provider
. If no provider is specified, Kubernetes is set by default.
Kompose supports conversion of V1, V2, and V3 Docker Compose files into Kubernetes and OpenShift objects.
example
kompose --file docker-voting.yml convert
WARN Unsupported key networks - ignoring
WARN Unsupported key build - ignoring
INFO Kubernetes file "worker-svc.yaml" created
INFO Kubernetes file "db-svc.yaml" created
INFO Kubernetes file "redis-svc.yaml" created
INFO Kubernetes file "result-svc.yaml" created
INFO Kubernetes file "vote-svc.yaml" created
INFO Kubernetes file "redis-deployment.yaml" created
INFO Kubernetes file "result-deployment.yaml" created
INFO Kubernetes file "vote-deployment.yaml" created
INFO Kubernetes file "worker-deployment.yaml" created
INFO Kubernetes file "db-deployment.yaml" created
ls
db-deployment.yaml  docker-compose.yml         docker-gitlab.yml  redis-deployment.yaml  result-deployment.yaml  vote-deployment.yaml  worker-deployment.yaml
db-svc.yaml         docker-voting.yml          redis-svc.yaml     result-svc.yaml        vote-svc.yaml           worker-svc.yaml
You can also provide multiple docker-compose files at the same time:
kompose -f docker-compose.yml -f docker-guestbook.yml convert
INFO Kubernetes file "frontend-service.yaml" created         
INFO Kubernetes file "mlbparks-service.yaml" created         
INFO Kubernetes file "mongodb-service.yaml" created          
INFO Kubernetes file "redis-master-service.yaml" created     
INFO Kubernetes file "redis-slave-service.yaml" created      
INFO Kubernetes file "frontend-deployment.yaml" created      
INFO Kubernetes file "mlbparks-deployment.yaml" created      
INFO Kubernetes file "mongodb-deployment.yaml" created       
INFO Kubernetes file "mongodb-claim0-persistentvolumeclaim.yaml" created
INFO Kubernetes file "redis-master-deployment.yaml" created  
ls
mlbparks-deployment.yaml  mongodb-service.yaml                       redis-slave-service.jsonmlbparks-service.yaml  
frontend-deployment.yaml  mongodb-claim0-persistentvolumeclaim.yaml  redis-master-service.yaml
frontend-service.yaml     mongodb-deployment.yaml                    redis-slave-deployment.yaml
redis-master-deployment.yaml
When multiple docker-compose files are provided the configuration is merged. Any configuration that is common will be overridden by subsequent file.
OpenShift
example
kompose --provider openshift --file docker-voting.yml convert
WARN [worker] Service cannot be created because of missing port.
INFO OpenShift file "vote-service.yaml" created             
INFO OpenShift file "db-service.yaml" created               
INFO OpenShift file "redis-service.yaml" created            
INFO OpenShift file "result-service.yaml" created           
INFO OpenShift file "vote-deploymentconfig.yaml" created    
INFO OpenShift file "vote-imagestream.yaml" created         
INFO OpenShift file "worker-deploymentconfig.yaml" created  
INFO OpenShift file "worker-imagestream.yaml" created       
INFO OpenShift file "db-deploymentconfig.yaml" created      
INFO OpenShift file "db-imagestream.yaml" created           
INFO OpenShift file "redis-deploymentconfig.yaml" created   
INFO OpenShift file "redis-imagestream.yaml" created        
INFO OpenShift file "result-deploymentconfig.yaml" created  
INFO OpenShift file "result-imagestream.yaml" created
It also supports creating buildconfig for build directive in a service. By default, it uses the remote repo for the current git branch as the source repo, and the current branch as the source branch for the build. You can specify a different source repo and branch using
--build-repo
and
--build-branch
options respectively.
kompose --provider openshift --file buildconfig/docker

-compose.yml convert
WARN [foo] Service cannot be created because of missing port.
INFO OpenShift Buildconfig using git@github.com:rtnpro/kompose.git::master as source.
INFO OpenShift file "foo-deploymentconfig.yaml" created     
INFO OpenShift file "foo-imagestream.yaml" created          
INFO OpenShift file "foo-buildconfig.yaml" created
Note:
If you are manually pushing the OpenShift artifacts using
oc create -f
, you need to ensure that you push the imagestream artifact before the buildconfig artifact, to workaround this OpenShift issue:
https://github.com/openshift/origin/issues/4518
.
Alternative Conversions
The default
kompose
transformation will generate Kubernetes
and
Services
, in yaml format. You have alternative option to generate json with
-j
. Also, you can alternatively generate
Replication Controllers
objects,
Daemon Sets
, or
Helm
charts.
kompose convert -j
INFO Kubernetes file
"redis-svc.json"
created
"web-svc.json"
created
"redis-deployment.json"
created
"web-deployment.json"
created
The
*-deployment.json
files contain the Deployment objects.
kompose convert --replication-controller
"redis-svc.yaml"
created
"web-svc.yaml"
created
"redis-replicationcontroller.yaml"
created
"web-replicationcontroller.yaml"
created
The
*-replicationcontroller.yaml
files contain the Replication Controller objects. If you want to specify replicas (default is 1), use
flag:
kompose convert --replication-controller --replicas 3
.
kompose convert --daemon-set
created
created
"redis-daemonset.yaml"
created
"web-daemonset.yaml"
created
The
*-daemonset.yaml
files contain the DaemonSet objects.
If you want to generate a Chart to be used with
Helm
run:
kompose convert -c
INFO Kubernetes file "web-svc.yaml" created
INFO Kubernetes file "web-deployment.yaml" created
chart created in "./docker-compose/"
tree docker-compose/
docker-compose
├── Chart.yaml
├── README.md
└── templates
    ├── redis-deployment.yaml
    ├── redis-svc.yaml
    ├── web-deployment.yaml
    └── web-svc.yaml
The chart structure is aimed at providing a skeleton for building your Helm charts.
Labels
kompose
supports Kompose-specific labels within the
file in order to explicitly define a service's behavior upon conversion.
defines the type of service to be created.
version
:
"2"
services
:
nginx
:
image
:
nginx
dockerfile
:
foobar
build
:
./foobar
cap_add
:
- ALL
container_name
:
foobar
labels
:
:
nodeport
kompose.service.expose
defines if the service needs to be made accessible from outside the cluster or not. If the value is set to "true", the provider sets the endpoint automatically, and for any other value, the value is set as the hostname. If multiple ports are defined in a service, the first one is chosen to be the exposed.
For the Kubernetes provider, an ingress resource is created and it is assumed that an ingress controller has already been configured.
For the OpenShift provider, a route is created.
version
:
"2"
services
:
web
:
image
:
tuna/docker-counter23
ports
:
-
"5000:5000"
links
:
- redis
labels
:
:
"counter.example.com"
redis
:
image
:
redis:3.0
ports
:
-
"6379"
The currently supported options are:
Key
Value
nodeport / clusterip / loadbalancer
true / hostname
Note:
The
label should be defined with
ports
only, otherwise
kompose
will fail.
Restart
If you want to create normal pods without controllers you can use
restart
construct of docker-compose to define that. Follow table below to see what happens on th

e
restart
value.
restart
object created
Pod
""
controller object
Always
always
Always
on-failure
Pod
OnFailure
no
Pod
Never
Note:
The controller object could be
deployment
or
replicationcontroller
.
For example, the
pival
service will become pod down here. This container calculated value of
pi
.
version
:
'2'
services
:
pival
:
image
:
perl
command
:
[
"perl"
,
"-Mbignum=bpi"
,
"-wle"
,
"print bpi(2000)"
]
restart
:
"on-failure"
Warning about Deployment Configurations
If the Docker Compose file has a volume specified for a service, the Deployment (Kubernetes) or DeploymentConfig (OpenShift) strategy is changed to "Recreate" instead of "RollingUpdate" (default). This is done to avoid multiple instances of a service from accessing a volume at the same time.
If the Docker Compose file has service name with
_
in it (for example,
web_service
), then it will be replaced by
-
and the service name will be renamed accordingly (for example,
web-service
). Kompose does this because "Kubernetes" doesn't allow
_
in object name.
Please note that changing service name might break some
files.
Kompose supports Docker Compose versions: 1, 2 and 3. We have limited support on versions 2.1 and 3.2 due to their experimental nature.
A full list on compatibility between all three versions is listed in our
conversion document
including a list of all incompatible Docker Compose keys.
24 - Enforce Pod Security Standards by Configuring the Built-in Admission Controller
As of v1.22, Kubernetes provides a built-in
to enforce the
.
You can configure this admission controller to set cluster-wide defaults and
exemptions
.
Your Kubernetes server must be at or later than version v1.22.
.
Ensure the
Configure the Admission Controller
:
kind
:
plugins
:
-
name
:
:
:
kind
:
PodSecurityConfiguration
# Defaults applied when a mode label is not set.
#
# Level label values must be one of:
# - "privileged" (default)
# - "baseline"
# - "restricted"
#
# Version label values must be one of:
# - "latest" (default)
# - specific version like "v1.26"
defaults
:
enforce
:
"privileged"
enforce-version
:
"latest"
audit
:
audit-version
:
"latest"
warn
:
warn-version
:
"latest"
:
# Array of authenticated usernames to exempt.
usernames
:
[]
# Array of runtime class names to exempt.
runtimeClasses
:
[]
# Array of namespaces to exempt.
:
[]
Note:
v1beta1
.
v1alpha1
.
25 - Enforce Pod Security Standards with Namespace Labels
Namespaces can be labeled to enforce the
. The three policies
,
baseline
and
broadly cover the security spectrum
and are implemented by the
.
.
Ensure the
Requiring the
baseline
Pod Security Standard with namespace labels
This manifest defines a Namespace
my-baseline-namespace
that:
Blocks
any pods that don't satisfy the
baseline
policy requirements.
Generates a user-facing warning and adds an audit annotation to any created pod that does not
meet the
Pins the versions of the
baseline
and
policies to v1.26.
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
:
baseline
:
v1.26
# We are setting these to our _desired_ `enforce` level.

:
:
v1.26
:
:
v1.26
Add labels to existing namespaces with
Note:
When an
enforce
policy (or version) label is added or changed, the admission plugin will test
each pod in the namespace against the new policy. Violations are returned to the user as warnings.
It is helpful to apply the
--dry-run
flag when initially evaluating security profile changes for
namespaces. The Pod Security Standard checks will still be run in
dry run
mode, giving you
information about how the new policy would treat existing pods, without actually updating a policy.
=
\
=
baseline
Applying to all namespaces
If you're just getting started with the Pod Security Standards, a suitable first step would be to
configure all namespaces with audit annotations for a stricter level such as
baseline
:
kubectl label --overwrite ns --all
\
=
baseline
\
=
baseline
Note that this is not setting an enforce level, so that namespaces that haven't been explicitly
evaluated can be distinguished. You can list namespaces without an explicitly set enforce level
using this command:
kubectl get namespaces --selector
=
'!pod-security.kubernetes.io/enforce'
Applying to a single namespace
You can update a specific namespace as well. This command adds the
enforce=restricted
policy to
my-existing-namespace
, pinning the restricted policy version to v1.26.
kubectl label --overwrite ns my-existing-namespace
\
=
\
=
v1.26
26 - Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller
This page describes the process of migrating from PodSecurityPolicies to the built-in PodSecurity
admission controller. This can be done effectively using a combination of dry-run and
audit
and
warn
modes, although this becomes harder if mutating PSPs are used.
.
If you are currently running a version of Kubernetes other than
{{ skew currentVersion }}, you may want to switch to viewing this
page in the documentation for the version of Kubernetes that you
are actually running.
This page assumes you are already familiar with the basic
concepts.
Overall approach
There are multiple strategies you can take for migrating from PodSecurityPolicy to Pod Security
Admission. The following steps are one possible migration path, with a goal of minimizing both the
risks of a production outage and of a security gap.
Decide whether Pod Security Admission is the right fit for your use case.
Review namespace permissions
Simplify &amp; standardize PodSecurityPolicies
Update namespaces
Identify an appropriate Pod Security level
Verify the Pod Security level
Enforce the Pod Security level
Bypass PodSecurityPolicy
Review namespace creation processes
Disable PodSecurityPolicy
0. Decide whether Pod Security Admission is right for you
Pod Security Admission was designed to meet the most common security needs out of the box, and to
provide a standard set of security levels across clusters. However, it is less flexible than
PodSecurityPolicy. Notably, the following features are supported by PodSecurityPolicy but not Pod
Security Admission:
Setting default security constraints
- Pod Security Admission is a non-mutating admission
controller, meaning it won't modify pods before validating them. If you were relying on this
aspect of PSP, you will need to either modify your workloads to meet the Pod Security constraints,
or use a
Mutating Admission Webhook
to make those changes. See
Simplify &amp; Standardize PodSecurityPolicies
below for more detail.
Fine-grained control over policy defini

tion
- Pod Security Admission only supports
3 standard levels
.
If you require more control over specific constraints, then you will need to use a
Validating Admission Webhook
to enforce those policies.
Sub-namespace policy granularity
- PodSecurityPolicy lets you bind different policies to
different Service Accounts or users, even within a single namespace. This approach has many
pitfalls and is not recommended, but if you require this feature anyway you will
need to use a 3rd party webhook instead. The exception to this is if you only need to completely exempt
specific users or
RuntimeClasses
, in which case Pod
Security Admission does expose some
static configuration for exemptions
.
Even if Pod Security Admission does not meet all of your needs it was designed to be
complementary
to other policy enforcement mechanisms, and can provide a useful fallback running alongside other
admission webhooks.
1. Review namespace permissions
Pod Security Admission is controlled by
labels on
.
This means that anyone who can update (or patch or create) a namespace can also modify the Pod
Security level for that namespace, which could be used to bypass a more restrictive policy. Before
proceeding, ensure that only trusted, privileged users have these namespace permissions. It is not
recommended to grant these powerful permissions to users that shouldn't have elevated permissions,
but if you must you will need to use an
admission webhook
to place additional restrictions on setting Pod Security labels on Namespace objects.
2. Simplify &amp; standardize PodSecurityPolicies
In this section, you will reduce mutating PodSecurityPolicies and remove options that are outside
the scope of the Pod Security Standards. You should make the changes recommended here to an offline
copy of the original PodSecurityPolicy being modified. The cloned PSP should have a different
name that is alphabetically before the original (for example, prepend a
0
to it). Do not create the
new policies in Kubernetes yet - that will be covered in the
Rollout the updated
policies
section below.
2.a. Eliminate purely mutating fields
If a PodSecurityPolicy is mutating pods, then you could end up with pods that don't meet the Pod
Security level requirements when you finally turn PodSecurityPolicy off. In order to avoid this, you
should eliminate all PSP mutation prior to switching over. Unfortunately PSP does not cleanly
separate mutating &amp; validating fields, so this is not a straightforward migration.
You can start by eliminating the fields that are purely mutating, and don't have any bearing on the
validating policy. These fields (also listed in the
reference) are:
.spec.defaultAllowPrivilegeEscalation
.spec.runtimeClass.defaultRuntimeClassName
.metadata.annotations['seccomp.security.alpha.kubernetes.io/defaultProfileName']
.metadata.annotations['apparmor.security.beta.kubernetes.io/defaultProfileName']
.spec.defaultAddCapabilities
- Although technically a mutating &amp; validating field, these should
be merged into
.spec.allowedCapabilities
which performs the same validation without mutation.
Caution:
Removing these could result in workloads missing required configuration, and cause problems. See
Rollout the updated policies
below for advice on how to roll these changes
out safely.
2.b. Eliminate options not covered by the Pod Security Standards
There are several fields in PodSecurityPolicy that are not covered by the Pod Security Standards. If
you must enforce these options, you will need to supplement Pod Security Admission with an
,
which is outside the scope of this guide.
First, you can remove the purely validating fields that the Pod Security Standards do not cover.
These fields (also listed in the
reference with "no opinion") are:
.spec.allowedHostPaths
.spec.allowedFlexVolumes
.spec.allowedCSIDrivers
.spec.forbiddenSysctls
.spec.ru

ntimeClass
You can also remove the following fields, that are related to POSIX / UNIX group controls.
Caution:
If any of these use the
MustRunAs
strategy they may be mutating! Removing these could result in
workloads not setting the required groups, and cause problems. See
.spec.runAsGroup
.spec.supplementalGroups
.spec.fsGroup
The remaining mutating fields are required to properly support the Pod Security Standards, and will
need to be handled on a case-by-case basis later:
.spec.requiredDropCapabilities
- Required to drop
ALL
for the Restricted profile.
.spec.seLinux
- (Only mutating with the
MustRunAs
rule) required to enforce the SELinux
requirements of the Baseline &amp; Restricted profiles.
.spec.runAsUser
- (Non-mutating with the
RunAsAny
rule) required to enforce
RunAsNonRoot
for
the Restricted profile.
.spec.allowPrivilegeEscalation
- (Only mutating if set to
false
) required for the Restricted
profile.
2.c. Rollout the updated PSPs
Next, you can rollout the updated policies to your cluster. You should proceed with caution, as
removing the mutating options may result in workloads missing required configuration.
For each updated PodSecurityPolicy:
Identify pods running under the original PSP. This can be done using the
kubernetes.io/psp
annotation. For example, using kubectl:
PSP_NAME
=
"original"
# Set the name of the PSP you're checking for
jsonpath
=
"{range .items[?(@.metadata.annotations.kubernetes\.io\/psp=='
$PSP_NAME
')]}{.metadata.namespace} {.metadata.name}{'\n'}{end}"
Compare these running pods against the original pod spec to determine whether PodSecurityPolicy
has modified the pod. For pods created by a
workload resource
you can compare the pod with the PodTemplate in the controller resource. If any changes are
identified, the original Pod or PodTemplate should be updated with the desired configuration.
The fields to review are:
.metadata.annotations['container.apparmor.security.beta.kubernetes.io/*']
(replace * with each container name)
.spec.runtimeClassName
.spec.securityContext.fsGroup
.spec.securityContext.seccompProfile
.spec.securityContext.seLinuxOptions
.spec.securityContext.supplementalGroups
On containers, under
.spec.containers[*]
and
.spec.initContainers[*]
:
.securityContext.allowPrivilegeEscalation
.securityContext.capabilities.add
.securityContext.capabilities.drop
.securityContext.readOnlyRootFilesystem
.securityContext.runAsGroup
.securityContext.runAsNonRoot
.securityContext.runAsUser
.securityContext.seccompProfile
.securityContext.seLinuxOptions
Create the new PodSecurityPolicies. If any Roles or ClusterRoles are granting
use
on all PSPs
this could cause the new PSPs to be used instead of their mutating counter-parts.
Update your authorization to grant access to the new PSPs. In RBAC this means updating any Roles
or ClusterRoles that grant the
use
permision on the original PSP to also grant it to the
updated PSP.
Verify: after some soak time, rerun the command from step 1 to see if any pods are still using
the original PSPs. Note that pods need to be recreated after the new policies have been rolled
out before they can be fully verified.
(optional) Once you have verified that the original PSPs are no longer in use, you can delete
them.
3. Update Namespaces
The following steps will need to be performed on every namespace in the cluster. Commands referenced
in these steps use the
$NAMESPACE
variable to refer to the namespace being updated.
3.a. Identify an appropriate Pod Security level
Start reviewing the
and
familiarizing yourself with the 3 different levels.
There are several ways to choose a Pod Security level for your namespace:
By security requirements for the namespace
- If you are familiar with the expected access
level for the namespace, you can choose an appropriate level based on those requirements, similar
to how one might approach

 this on a new cluster.
By existing PodSecurityPolicies
- Using the
reference you can map each
PSP to a Pod Security Standard level. If your PSPs aren't based on the Pod Security Standards, you
may need to decide between choosing a level that is at least as permissive as the PSP, and a
level that is at least as restrictive. You can see which PSPs are in use for pods in a given
namespace with this command:
-o
jsonpath
=
"{.items[*].metadata.annotations.kubernetes\.io\/psp}"
| tr
" "
"\n"
| sort -u
By existing pods
- Using the strategies under
,
you can test out both the Baseline and Restricted levels to see
whether they are sufficiently permissive for existing workloads, and chose the least-privileged
valid level.
Caution:
Options 2 &amp; 3 above are based on
existing
pods, and may miss workloads that aren't currently
running, such as CronJobs, scale-to-zero workloads, or other workloads that haven't rolled out.
3.b. Verify the Pod Security level
Once you have selected a Pod Security level for the namespace (or if you're trying several), it's a
good idea to test it out first (you can skip this step if using the Privileged level). Pod Security
includes several tools to help test and safely roll out profiles.
First, you can dry-run the policy, which will evaluate pods currently running in the namespace
against the applied policy, without making the new policy take effect:
# $LEVEL is the level to dry-run, either "baseline" or "restricted".
=
server --overwrite ns
=
$LEVEL
This command will return a warning for any
existing
pods that are not valid under the proposed
level.
The second option is better for catching workloads that are not currently running: audit mode. When
running under audit-mode (as opposed to enforcing), pods that violate the policy level are recorded
in the audit logs, which can be reviewed later after some soak time, but are not forbidden. Warning
mode works similarly, but returns the warning to the user immediately. You can set the audit level
on a namespace with this command:
kubectl label --overwrite ns
=
$LEVEL
If either of these approaches yield unexpected violations, you will need to either update the
violating workloads to meet the policy requirements, or relax the namespace Pod Security level.
3.c. Enforce the Pod Security level
When you are satisfied that the chosen level can safely be enforced on the namespace, you can update
the namespace to enforce the desired level:
=
$LEVEL
3.d. Bypass PodSecurityPolicy
Finally, you can effectively bypass PodSecurityPolicy at the namespace level by binding the fully
privileged PSP
to all service
accounts in the namespace.
# The following cluster-scoped commands are only needed once.
kubectl apply -f privileged-psp.yaml
kubectl create clusterrole privileged-psp --verb use --resource podsecuritypolicies.policy --resource-name privileged
# Per-namespace disable
kubectl create -n
rolebinding disable-psp --clusterrole privileged-psp --group system:serviceaccounts:
Since the privileged PSP is non-mutating, and the PSP admission controller always
prefers non-mutating PSPs, this will ensure that pods in this namespace are no longer being modified
or restricted by PodSecurityPolicy.
The advantage to disabling PodSecurityPolicy on a per-namespace basis like this is if a problem
arises you can easily roll the change back by deleting the RoleBinding. Just make sure the
pre-existing PodSecurityPolicies are still in place!
# Undo PodSecurityPolicy disablement.
kubectl delete -n
rolebinding disable-psp
4. Review namespace creation processes
Now that existing namespaces have been updated to enforce Pod Security Admission, you should ensure
that your

 processes and/or policies for creating new namespaces are updated to ensure that an
appropriate Pod Security profile is applied to new namespaces.
You can also statically configure the Pod Security admission controller to set a default enforce,
audit, and/or warn level for unlabeled namespaces. See
5. Disable PodSecurityPolicy
Finally, you're ready to disable PodSecurityPolicy. To do so, you will need to modify the admission
configuration of the API server:
.
To verify that the PodSecurityPolicy admission controller is no longer enabled, you can manually run
a test by impersonating a user without access to any PodSecurityPolicies (see the
PodSecurityPolicy example
), or by verifying in
the API server logs. At startup, the API server outputs log lines listing the loaded admission
controller plugins:
I0218 00:59:44.903329      13 plugins.go:158] Loaded 16 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,Priority,DefaultTolerationSeconds,ExtendedResourceToleration,PersistentVolumeLabel,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I0218 00:59:44.903350      13 plugins.go:161] Loaded 14 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurity,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,DenyServiceExternalIPs,ValidatingAdmissionWebhook,ResourceQuota.
You should see
(in the validating admission controllers), and neither list should
contain
.
Once you are certain the PSP admission controller is disabled (and after sufficient soak time to be
confident you won't need to roll back), you are free to delete your PodSecurityPolicies and any
associated Roles, ClusterRoles, RoleBindings and ClusterRoleBindings (just make sure they don't
grant any other unrelated permissions).
Home
Blog
Training
Partners
Community
CC BY 4.0

Client Libraries | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up 

kubectl on macOS
Debug Pods
Get a Shell to a Running Con

tainer
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's A

ccess to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configuration (v1

)
Reference
This page contains an overview of the client libraries for using the Kubernetes
API from various programming languages.
To write applications using the
Kubernetes REST API
,
you do not need to implement the API calls and request/response types yourself.
You can use a client library for the programming language you are using.
Client libraries often handle common tasks such as authentication for you.
Most client libraries can discover and use the Kubernetes Service Account to
authenticate if the API client is running inside the Kubernetes cluster, or can
understand the
format to read the credentials and the API Server address.
Officially-supported Kubernetes client libraries
The following client libraries are officially maintained by
Kubernetes SIG API Machinery
.
Language
Client Library
Sample Programs
C
github.com/kubernetes-client/c
browse
dotnet
github.com/kubernetes-client/csharp
browse
Go
github.com/kubernetes/client-go/
browse
Haskell
github.com/kubernetes-client/haskell
browse
Java
github.com/kubernetes-client/java
browse
github.com/kubernetes-client/javascript
browse
Perl
github.com/kubernetes-client/perl/
browse
Python
github.com/kubernetes-client/python/
browse
Ruby
github.com/kubernetes-client/ruby/
browse
Community-maintained client libraries
Note:
The following Kubernetes API client libraries are provided and maintained by
their authors, not the Kubernetes team.
Language
Clojure
github.com/yanatan16/clj-kubernetes-api
DotNet
github.com/tonnyeremin/kubernetes_gen
DotNet (RestSharp)
github.com/masroorhasan/Kubernetes.DotNet
Elixir
github.com/obmarg/kazan
Elixir
github.com/coryodaniel/k8s
Go
github.com/ericchiang/k8s
Java (OSGi)
bitbucket.org/amdatulabs/amdatu-kubernetes
Java (Fabric8, OSGi)
github.com/fabric8io/kubernetes-client
Java
github.com/manusa/yakc
Lisp
github.com/brendandburns/cl-k8s
Lisp
github.com/xh4/cube
Node.js (TypeScript)
github.com/Goyoo/node-k8s-client
Node.js
github.com/ajpauwels/easy-k8s
Node.js
github.com/godaddy/kubernetes-client
Node.js
github.com/tenxcloud/node-kubernetes-client
Perl
metacpan.org/pod/Net::Kubernetes
PHP
github.com/allansun/kubernetes-php-client
PHP
github.com/maclof/kubernetes-client
PHP
github.com/travisghansen/kubernetes-client

-php
PHP
github.com/renoki-co/php-k8s
Python
github.com/fiaas/k8s
Python
github.com/gtsystem/lightkube
Python
github.com/mnubo/kubernetes-py
Python
github.com/tomplus/kubernetes_asyncio
Python
github.com/Frankkkkk/pykorm
Ruby
github.com/abonas/kubeclient
Ruby
github.com/k8s-ruby/k8s-ruby
Ruby
github.com/kontena/k8s-client
Rust
github.com/clux/kube-rs
Rust
github.com/ynqa/kubernetes-rust
Scala
github.com/hagay3/skuber
Scala
github.com/hnaderi/scala-k8s
Scala
github.com/joan38/kubernetes-client
Swift
github.com/swiftkube/client
Feedback
Yes
No
.
or
.
Last modified October 31, 2022 at 6:06 PM PST:
Update link for Java Client Library examples (efe16f42f4)
Home
Blog
Training
Partners
Community
CC BY 4.0

Overview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Overview
Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation. It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
1:
2:
3:
3.1:
3.2:
3.3:
3.4:
3.5:
3.6:
3.7:
3.8:
3.9:
3.10:
This page is an overview of Kubernetes.
Kubernetes is a portable, extensible, open source platform for managing containerized
workloads and services, that facilitates both declarative configuration and automation.
It has a large, rapidly growing ecosystem. Kubernetes services, support, and tools are widely available.
The name Kubernetes originates from Greek, meaning helmsman or pilot. K8s as an abbreviation
results from counting the eight letters between the "K" and the "s". Google open-sourced the
Kubernetes project in 2014. Kubernetes combines
over 15 years of Google's experience
running
production workloads at scale with best-of-breed ideas and practices from the community.
Going back in time
Let's take a look at why Kubernetes is so useful by going back in time.
Traditional deployment era:
Early on, organizations ran applications on physical servers. There was no way to define
resource boundaries for applications in a physical server, and this caused resource
allocation issues. For example, if multiple applications run on a physical server, there
can be instances where one application would take up most of the resources, and as a result,
the other applications would underperform. A solution for this would be to run each application
on a different physical server. But this did not scale as resources were underutilized, and it
was expensive for organizations to maintain many physical servers.
Virtualized deployment era:
As a solution, virtualization was introduced. It allows you
to run multiple Virtual Machines (VMs) on a single physical server's CPU. Virtualization
allows applications to be isolated between VMs and provides a level of security as the
information of one application cannot be freely accessed by another application.
Virtualization allows better utilization of resources in a physical server and allows
better scalability because an application can be added or updated easily, reduces
hardware costs, and much more. With virtualization you can present a set of physical
resources as a cluster of disposable virtual machines.
Each VM is a full machine running all the components, including its own operating
system, on top of the virtualized hardware.
Container deployment era:
Containers are similar to VMs, but they have relaxed
isolation properties to share the Operating System (OS) among the applications.
Therefore, containers are considered lightweight. Similar to a VM, a container
has its own filesystem, share of CPU, memory, process space, and more. As they
are decoupled from the underlying infrastructure, they are portable across clouds
and OS distributions.
Containers have become popular because they provide extra benefits, such as:
Agile application creation and deployment: increased ease and efficiency of
container image creation compared to VM image use.
Continuous development, integrat

ion, and deployment: provides for reliable
and frequent container image build and deployment with quick and efficient
rollbacks (due to image immutability).
Dev and Ops separation of concerns: create application container images at
build/release time rather than deployment time, thereby decoupling
applications from infrastructure.
Observability: not only surfaces OS-level information and metrics, but also
application health and other signals.
Environmental consistency across development, testing, and production: runs
the same on a laptop as it does in the cloud.
Cloud and OS distribution portability: runs on Ubuntu, RHEL, CoreOS, on-premises,
on major public clouds, and anywhere else.
Application-centric management: raises the level of abstraction from running an
OS on virtual hardware to running an application on an OS using logical resources.
Loosely coupled, distributed, elastic, liberated micro-services: applications are
broken into smaller, independent pieces and can be deployed and managed dynamically –
not a monolithic stack running on one big single-purpose machine.
Resource isolation: predictable application performance.
Resource utilization: high efficiency and density.
Why you need Kubernetes and what it can do
Containers are a good way to bundle and run your applications. In a production
environment, you need to manage the containers that run the applications and
ensure that there is no downtime. For example, if a container goes down, another
container needs to start. Wouldn't it be easier if this behavior was handled by a system?
That's how Kubernetes comes to the rescue! Kubernetes provides you with a framework
to run distributed systems resiliently. It takes care of scaling and failover for
your application, provides deployment patterns, and more. For example: Kubernetes
can easily manage a canary deployment for your system.
Kubernetes provides you with:
Service discovery and load balancing
Kubernetes can expose a container using the DNS name or using their own IP address.
If traffic to a container is high, Kubernetes is able to load balance and distribute
the network traffic so that the deployment is stable.
Storage orchestration
Kubernetes allows you to automatically mount a storage system of your choice, such as
local storages, public cloud providers, and more.
Automated rollouts and rollbacks
You can describe the desired state for your deployed containers using Kubernetes,
and it can change the actual state to the desired state at a controlled rate.
For example, you can automate Kubernetes to create new containers for your
deployment, remove existing containers and adopt all their resources to the new container.
Automatic bin packing
You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks.
You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit
containers onto your nodes to make the best use of your resources.
Self-healing
Kubernetes restarts containers that fail, replaces containers, kills containers that don't
respond to your user-defined health check, and doesn't advertise them to clients until they
are ready to serve.
Secret and configuration management
Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens,
and SSH keys. You can deploy and update secrets and application configuration without
rebuilding your container images, and without exposing secrets in your stack configuration.
What Kubernetes is not
Kubernetes is not a traditional, all-inclusive PaaS (Platform as a Service) system.
Since Kubernetes operates at the container level rather than at the hardware level,
it provides some generally applicable features common to PaaS offerings, such as
deployment, scaling, load balancing, and lets users integrate their logging, monitoring,
and alerting solutions. However, Kubernetes is not monolithic, and these default solutions
are optional and pluggable. Kubernetes provides the building blocks for bui

lding developer
platforms, but preserves user choice and flexibility where it is important.
Kubernetes:
Does not limit the types of applications supported. Kubernetes aims to support an
extremely diverse variety of workloads, including stateless, stateful, and data-processing
workloads. If an application can run in a container, it should run great on Kubernetes.
Does not deploy source code and does not build your application. Continuous Integration,
Delivery, and Deployment (CI/CD) workflows are determined by organization cultures and
preferences as well as technical requirements.
Does not provide application-level services, such as middleware (for example, message buses),
data-processing frameworks (for example, Spark), databases (for example, MySQL), caches, nor
cluster storage systems (for example, Ceph) as built-in services. Such components can run on
Kubernetes, and/or can be accessed by applications running on Kubernetes through portable
mechanisms, such as the
Open Service Broker
.
Does not dictate logging, monitoring, or alerting solutions. It provides some integrations
as proof of concept, and mechanisms to collect and export metrics.
Does not provide nor mandate a configuration language/system (for example, Jsonnet). It provides
a declarative API that may be targeted by arbitrary forms of declarative specifications.
Does not provide nor adopt any comprehensive machine configuration, maintenance, management,
or self-healing systems.
Additionally, Kubernetes is not a mere orchestration system. In fact, it eliminates the need
for orchestration. The technical definition of orchestration is execution of a defined workflow:
first do A, then B, then C. In contrast, Kubernetes comprises a set of independent, composable
control processes that continuously drive the current state towards the provided desired state.
It shouldn't matter how you get from A to C. Centralized control is also not required. This
results in a system that is easier to use and more powerful, robust, resilient, and extensible.
Take a look at the
Ready to
Get Started
?
1 - Kubernetes Components
A Kubernetes cluster consists of the components that are a part of the control plane and a set of machines called nodes.
When you deploy Kubernetes, you get a cluster.
A Kubernetes cluster consists of a set of worker machines, called
nodes
,
that run containerized applications. Every cluster has at least one worker node.
The worker node(s) host the
Pods
that are
the components of the application workload. The
manages the worker
nodes and the Pods in the cluster. In production environments, the control plane usually
runs across multiple computers and a cluster usually runs multiple nodes, providing
fault-tolerance and high availability.
This document outlines the various components you need to have for
a complete and working Kubernetes cluster.
The components of a Kubernetes cluster
Control Plane Components
The control plane's components make global decisions about the cluster (for example, scheduling), as well as detecting and responding to cluster events (for example, starting up a new
pod
when a deployment's
replicas
field is unsatisfied).
Control plane components can be run on any machine in the cluster. However,
for simplicity, set up scripts typically start all control plane components on
the same machine, and do not run user containers on this machine. See
Creating Highly Available clusters with kubeadm
for an example control plane setup that runs across multiple machines.
The API server is a component of the Kubernetes
that exposes the Kubernetes API.
The API server is the front end for the Kubernetes control plane.
The main implementation of a Kubernetes API server is
.
kube-apiserver is designed to scale horizontally—that is, it scales by deploying more instances.
You can run several insta

nces of kube-apiserver and balance traffic between those instances.
etcd
Consistent and highly-available key value store used as Kubernetes' backing store for all cluster data.
back up
plan
.
Control plane component that watches for newly created
Pods
with no assigned
node
, and selects a node for them
to run on.
Factors taken into account for scheduling decisions include:
individual and collective resource requirements, hardware/software/policy
constraints, affinity and anti-affinity specifications, data locality,
inter-workload interference, and deadlines.
Control plane component that runs
Logically, each
is a separate process, but to reduce complexity, they are all compiled into a single binary and run in a single process.
Some types of these controllers are:
Node controller: Responsible for noticing and responding when nodes go down.
Job controller: Watches for Job objects that represent one-off tasks, then creates
Pods to run those tasks to completion.
EndpointSlice controller: Populates EndpointSlice objects (to provide a link between Services and Pods).
ServiceAccount controller: Create default ServiceAccounts for new namespaces.
A Kubernetes
component
The cloud-controller-manager only runs controllers that are specific to your cloud provider.
If you are running Kubernetes on your own premises, or in a learning environment inside your
own PC, the cluster does not have a cloud controller manager.
As with the kube-controller-manager, the cloud-controller-manager combines several logically
independent control loops into a single binary that you run as a single process. You can
scale horizontally (run more than one copy) to improve performance or to help tolerate failures.
The following controllers can have cloud provider dependencies:
Node controller: For checking the cloud provider to determine if a node has been deleted in the cloud after it stops responding
Route controller: For setting up routes in the underlying cloud infrastructure
Service controller: For creating, updating and deleting cloud provider load balancers
Node Components
Node components run on every node, maintaining running pods and providing the Kubernetes runtime environment.
kubelet
An agent that runs on each
node
in the cluster. It makes sure that
are running in a
Pod
.
The kubelet takes a set of PodSpecs that are provided through various mechanisms and ensures that the containers described in those PodSpecs are running and healthy. The kubelet doesn't manage containers which were not created by Kubernetes.
kube-proxy is a network proxy that runs on each
node
in your cluster,
implementing part of the Kubernetes
Service
concept.
maintains network rules on nodes. These network rules allow network
communication to your Pods from network sessions inside or outside of
your cluster.
kube-proxy uses the operating system packet filtering layer if there is one
and it's available. Otherwise, kube-proxy forwards the traffic itself.
Container runtime
The container runtime is the software that is responsible for running containers.
Kubernetes supports container runtimes such as
,
CRI-O
,
and any other implementation of the
Kubernetes CRI (Container Runtime
Interface)
.
Addons
Addons use Kubernetes resources (
DaemonSet
,
, etc)
to implement cluster features. Because these are providing cluster-level features, namespaced resources
for addons belong within the
S

elected addons are described below; for an extended list of available addons, please
see
Addons
.
DNS
While the other addons are not strictly required, all Kubernetes clusters should have
cluster DNS
, as many examples rely on it.
Cluster DNS is a DNS server, in addition to the other DNS server(s) in your environment, which serves DNS records for Kubernetes services.
Containers started by Kubernetes automatically include this DNS server in their DNS searches.
Web UI (Dashboard)
Dashboard
is a general purpose, web-based UI for Kubernetes clusters. It allows users to manage and troubleshoot applications running in the cluster, as well as the cluster itself.
Container Resource Monitoring
records generic time-series metrics
about containers in a central database, and provides a UI for browsing that data.
Cluster-level Logging
A
mechanism is responsible for
saving container logs to a central log store with search/browsing interface.
Nodes
Read etcd's official
2 - The Kubernetes API
is the
API server
kubectl
kubeadm
OpenAPI
.
OpenAPI V2
Header
Notes
gzip
Accept
default
*
serves
OpenAPI V3
named
OpenAPIV3
{
"paths":
{
...,
"api/v1":
{
"/openapi/v3/api/v1?hash=CC0E9BFD992D8C59AEC98A1E2336F899E8318D3CF4C68944C3DEC640AF5AB52D864AC50DAA8D145B3494F75FA3CFF939FCBDDA431DAD3CA79738B297795818CF"
},
{
},
....
}
}
The relative URLs are pointing to immutable OpenAPI descriptions, 

in
Expires
to 1 year in
to
immutable
endpoint.
Header
Notes
gzip
Accept
default
*
serves
etcd
.
as
/api/v1
or
.
API groups
that can be
.
v1
and
v1beta1
v1beta1
v1beta1
or the
v1
v1beta1
v1
API.
not
.
v1
beta
Note:
Although Kubernetes also aims to maintain compatibility for
alpha
Refer to
for more details on the API version level definiti

ons.
.
.
describes
.
.
3 - Working with Kubernetes Objects
Kubernetes objects are persistent entities in the Kubernetes system. Kubernetes uses these entities to represent the state of your cluster. Learn about the Kubernetes object model and how to work with these objects.
3.1 - Understanding Kubernetes Objects
.yaml
format.
cluster's
.
kubectl
.
spec
status
.
spec
its
.
The
status
supplied.
spec
a replacement instance.
.
kubectl
kubectl
kubectl
request.
.yaml
file that shows the requir

ed fields and object spec for a Kubernetes Deployment:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.yaml
command
in the
kubectl
.yaml
In the
.yaml
kind
metadata
name
string,
UID
namespace
spec
spec
spec
field
.spec
spec
field
.spec
Within the
.spec
template
.status
.status
Pods
objects.
kubectl
and
.
3.2 - Kubernetes Object Management
The
kubectl
for
Warning:
1+
Lowest
1
Moderate
1+
Highest
the
kubectl
Examples

See the
Warning:
replace
independently from the configuration by the cluster.
Examples
kubectl
Note:
patch
replace
Examples
configs
diff
Declarative Ma

nagement of Kubernetes Objects Using Kustomize
3.3 - Object Names and IDs
Name
UID
namespace
.
labels
and
.
Names
.
Note:
RFC 1123
.
RFC 1123
.
RFC 1035
.
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Note:
UIDs
A Kubernetes systems-generated string to uniquely identify objects.
Read about
labels
and
See the
3.4 - Labels and Selectors
Labels
are key/value pairs that are attached to objects, such as pods.
Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system.
Labels can be used to organize and to select subsets of objects. Labels can be attached to objects at creation time and subsequently added and modified at any time.
Each object can have a set of key/value labels defined. Each Key must be unique for a given object.
:
{
"labels"
: {
"key1"
:
"value1"
,
"key2"
:
"value2"
}
}
Labels allow for efficient queries and watches and are ideal for use in UIs
and CLIs. Non-identifying information should be recorded using
.
Labels enable users to map their own organizational structures onto system objects in a loosely coupled fashion, without requiring clients to store these mappings.
Service deployments and batch processin

g pipelines are often multi-dimensional entities (e.g., multiple partitions or deployments, multiple release tracks, multiple tiers, multiple micro-services per tier). Management often requires cross-cutting operations, which breaks encapsulation of strictly hierarchical representations, especially rigid hierarchies determined by the infrastructure rather than by users.
Example labels:
"release" : "stable"
,
"release" : "canary"
"environment" : "dev"
,
"environment" : "qa"
,
"environment" : "production"
"tier" : "frontend"
,
"tier" : "backend"
,
"tier" : "cache"
"partition" : "customerA"
,
"partition" : "customerB"
"track" : "daily"
,
"track" : "weekly"
These are examples of
commonly used labels
; you are free to develop your own conventions. Keep in mind that label Key must be unique for a given object.
Syntax and character set
Labels
are key/value pairs. Valid label keys have two segments: an optional prefix and name, separated by a slash (
/
). The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character (
[a-z0-9A-Z]
) with dashes (
-
), underscores (
_
), dots (
.
), and alphanumerics between. The prefix is optional. If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (
.
), not longer than 253 characters in total, followed by a slash (
/
).
If the prefix is omitted, the label Key is presumed to be private to the user. Automated system components (e.g.
,
,
,
kubectl
, or other third-party automation) which add labels to end-user objects must specify a prefix.
The
and
k8s.io/
prefixes are
reserved
for Kubernetes core components.
Valid label value:
must be 63 characters or less (can be empty),
unless empty, must begin and end with an alphanumeric character (
),
could contain dashes (
-
_
), dots (
.
), and alphanumerics between.
For example, here's the configuration file for a Pod that has two labels
environment: production
and
app: nginx
:
:
v1
kind
:
Pod
metadata
:
name
:
label-demo
labels
:
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Label selectors
Unlike
names and UIDs
, labels do not provide uniqueness. In general, we expect many objects to carry the same label(s).
Via a
label selector
, the client/user can identify a set of objects. The label selector is the core grouping primitive in Kubernetes.
The API currently supports two types of selectors:
equality-based
and
set-based
.
A label selector can be made of multiple
requirements
which are comma-separated. In the case of multiple requirements, all must be satisfied so the comma separator acts as a logical
AND
(
) operator.
The semantics of empty or non-specified selectors are dependent on the context,
and API types that use selectors should document the validity and meaning of
them.
Note:
For some API types, such as ReplicaSets, the label selectors of two instances must not overlap within a namespace, or the controller can see that as conflicting instructions and fail to determine how many replicas should be present.
Caution:
For both equality-based and set-based conditions there is no logical
OR
(
||
) operator. Ensure your filter statements are structured accordingly.
Equality-based
requirement
Equality-
or
inequality-based
requirements allow filtering by label keys and values. Matching objects must satisfy all of the specified label constraints, though they may have additional labels as well.
Three kinds of operators are admitted
=
,
==
,
!=
. The first two represent
equality
(and are synonyms), while the latter represents
inequality
environment = production
tier != frontend
The former selects all resources with key equal to
and value equal to
.
The latter selects all resources with key equal to
tier
and value distinct from
frontend
, and a

ll resources with no labels with the
tier
key.
One could filter for resources in
excluding
frontend
using the comma operator:
environment=production,tier!=frontend
One usage scenario for equality-based label requirement is for Pods to specify
node selection criteria. For example, the sample Pod below selects nodes with
the label "
accelerator=nvidia-tesla-p100
".
:
v1
kind
:
Pod
metadata
:
name
:
cuda-test
spec
:
:
-
name
:
cuda-test
image
:
"registry.k8s.io/cuda-vector-add:v0.1"
resources
:
limits
:
nvidia.com/gpu
:
1
:
accelerator
:
nvidia-tesla-p100
Set-based
Set-based
label requirements allow filtering keys according to a set of values. Three kinds of operators are supported:
in
,
notin
and
exists
(only the key identifier). For example:
environment in (production, qa)
tier notin (frontend, backend)
partition
!partition
The first example selects all resources with key equal to
or
qa
.
The second example selects all resources with key equal to
tier
and values other than
frontend
and
backend
, and all resources with no labels with the
tier
key.
The third example selects all resources including a label with key
partition
; no values are checked.
The fourth example selects all resources without a label with key
partition
Similarly the comma separator acts as an
AND
operator. So filtering resources with a
partition
key (no matter the value) and with
different than
qa
can be achieved using
partition,environment notin (qa)
.
The
set-based
label selector is a general form of equality since
environment=production
is equivalent to
environment in (production)
; similarly for
!=
and
notin
.
Set-based
requirements can be mixed with
requirements. For example:
partition in (customerA, customerB),environment!=qa
.
API
LIST and WATCH filtering
LIST and WATCH operations may specify label selectors to filter the sets of objects returned using a query parameter. Both requirements are permitted (presented here as they would appear in a URL query string):
requirements:
?labelSelector=environment%3Dproduction,tier%3Dfrontend
set-based
?labelSelector=environment+in+%28production%2Cqa%29%2Ctier+in+%28frontend%29
Both label selector styles can be used to list or watch resources via a REST client. For example, targeting
apiserver
with
kubectl
and using
one may write:
=
production,tier
=
frontend
or using
set-based
'environment in (production),tier in (frontend)'
As already mentioned
set-based
requirements are more expressive.  For instance, they can implement the
OR
operator on values:
'environment in (production, qa)'
or restricting negative matching via
exists
operator:
'environment,environment notin (frontend)'
Set references in API objects
Some Kubernetes objects, such as
services
and
,
also use label selectors to specify sets of other resources, such as
pods
.
Service and ReplicationController
The set of pods that a
service
targets is defined with a label selector. Similarly, the population of pods that a
should manage is also defined with a label selector.
Labels selectors for both objects are defined in
json
or
yaml
files using maps, and only
requirement selectors are supported:
:
{
"component"
:
"redis"
,
}
or
selector
:
component
:
redis
this selector (respectively in
json
or
yaml
format) is equivalent to
component=redis
or
component in (redis)
.
Resources that support set-based requirements
Newer resources, such as
Job
,
,
, and
DaemonSet
,
support
set-based
requirements as well.
selector
:
:
component
:
redis
:
- {
key: tier, operator: In, values
:
[cache]}
- {
key: environment, operator: NotIn, values
:
[dev]}

is a map of
{key,value}
pairs. A single
in the
map is equivalent to an element of
, whose
key
field is "key", the
operator
is "In", and the
values
array contains only "value".
is a list of pod selector requirements. Valid operators include In, NotIn, Exists, and DoesNotExist. The values set must be non-empty in the case of In and NotIn. All of the requirements, from both
and
are ANDed together -- they must all be satisfied in order to match.
Selecting sets of nodes
One use case for selecting over labels is to constrain the set of nodes onto which a pod can schedule.
See the documentation on
node selection
add a label to a node
Find
Well-known labels, Annotations and Taints
See
Recommended labels
Use Labels effectively
to manage deployments.
Read a blog on
Writing a Controller for Pod Labels
3.5 - Namespaces
provides a mechanism for isolating groups of resources within a single cluster. Names of resources need to be unique within a namespace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects
(e.g. Deployments, Services, etc)
and not for cluster-wide objects
(e.g. StorageClass, Nodes, PersistentVolumes, etc)
.
When to Use Multiple Namespaces
Namespaces are intended for use in environments with many users spread across multiple
teams, or projects. For clusters with a few to tens of users, you should not
need to create or think about namespaces at all. Start using namespaces when you
need the features they provide.
Namespaces provide a scope for names. Names of resources need to be unique within a namespace,
but not across namespaces. Namespaces cannot be nested inside one another and each Kubernetes
resource can only be in one namespace.
Namespaces are a way to divide cluster resources between multiple users (via
).
It is not necessary to use multiple namespaces to separate slightly different
resources, such as different versions of the same software: use
labels
to distinguish
resources within the same namespace.
Note:
For a production cluster, consider
not
using the
default
namespace. Instead, make other namespaces and use those.
Initial namespaces
Kubernetes starts with four initial namespaces:
default
Kubernetes includes this namespace so that you can start using your new cluster without first creating a namespace.
kube-node-lease
This namespace holds
Lease
objects associated with each node. Node leases allow the kubelet to send
heartbeats
so that the control plane can detect node failure.
This namespace is readable by
all
clients (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.
The namespace for objects created by the Kubernetes system.
Working with Namespaces
Creation and deletion of namespaces are described in the
Admin Guide documentation for namespaces
.
Note:
Avoid creating namespaces with the prefix
kube-
You can list the current namespaces in a cluster using:
kubectl get namespace
NAME              STATUS   AGE
default           Active   1d
kube-node-lease   Active   1d
kube-public       Active   1d
kube-system       Active   1d
Setting the namespace for a request
To set the namespace for a current request, use the
--namespace
flag.
=
nginx --namespace
=
=
Setting the namespace preference
You can permanently save the namespace for all subsequent kubectl commands in that
context.
kubectl config set-context --

current --namespace
=
# Validate it
kubectl config view --minify | grep namespace:
Namespaces and DNS
Service
,
it creates a corresponding
DNS entry
.
that if a container only uses
, it will resolve to the service which
As a result, all namespace names must be valid
RFC 1123 DNS labels
.
Warning:
By creating namespaces with the same name as
public top-level
domains
, Services in these
namespaces can have short DNS names that overlap with public DNS records.
Workloads from any namespace performing a DNS lookup without a
trailing dot
will
be redirected to those services, taking precedence over public DNS.
To mitigate this, limit privileges for creating namespaces to trusted users. If
required, you could additionally configure third-party security controls, such
as
admission
webhooks
,
to block creating any namespace with the name of
public
TLDs
.
Not all objects are in a namespace
Most Kubernetes resources (e.g. pods, services, replication controllers, and others) are
in some namespaces. However namespace resources are not themselves in a namespace.
And low-level resources, such as
nodes
and
persistentVolumes
, are not in any namespace.
To see which Kubernetes resources are and aren't in a namespace:
# In a namespace
=
true
# Not in a namespace
=
false
Automatic labelling
Kubernetes 1.21 [beta]
The Kubernetes control plane sets an immutable
label
kubernetes.io/metadata.name
on all namespaces, provided that the
NamespaceDefaultLabelName
The value of the label is the namespace name.
creating a new namespace
.
deleting a namespace
.
3.6 - Annotations
You can use Kubernetes annotations to attach arbitrary non-identifying metadata
to objects. Clients such as tools and libraries can retrieve this metadata.
Attaching metadata to objects
You can use either labels or annotations to attach metadata to Kubernetes
objects. Labels can be used to select objects and to find
collections of objects that satisfy certain conditions. In contrast, annotations
are not used to identify and select objects. The metadata
in an annotation can be small or large, structured or unstructured, and can
include characters not permitted by labels.
Annotations, like labels, are key/value maps:
:
{
: {
"key1"
:
"value1"
,
"key2"
:
"value2"
}
}
Note:
The keys and the values in the map must be strings. In other words, you cannot use
numeric, boolean, list or other types for either the keys or the values.
Here are some examples of information that could be recorded in annotations:
Fields managed by a declarative configuration layer. Attaching these fields
as annotations distinguishes them from default values set by clients or
servers, and from auto-generated fields and fields set by
auto-sizing or auto-scaling systems.
Build, release, or image information like timestamps, release IDs, git branch,
PR numbers, image hashes, and registry address.
Pointers to logging, monitoring, analytics, or audit repositories.
Client library or tool information that can be used for debugging purposes:
for example, name, version, and build information.
User or tool/system provenance information, such as URLs of related objects
from other ecosystem components.
Lightweight rollout tool metadata: for example, config or checkpoints.
Phone or pager numbers of persons responsible, or directory entries that
specify where that information can be found, such as a team web site.
Directives from the end-user to

 the implementations to modify behavior or
engage non-standard features.
Instead of using annotations, you could store this type of information in an
external database or directory, but that would make it much harder to produce
shared client libraries and tools for deployment, management, introspection,
and the like.
are key/value pairs. Valid annotation keys have two segments: an optional prefix and name, separated by a slash (
/
-
_
), dots (
.
.
/
).
If the prefix is omitted, the annotation Key is presumed to be private to the user. Automated system components (e.g.
,
,
,
kubectl
, or other third-party automation) which add annotations to end-user objects must specify a prefix.
The
and
k8s.io/
prefixes are reserved for Kubernetes core components.
For example, here's the configuration file for a Pod that has the annotation
imageregistry: https://hub.docker.com/
:
:
v1
kind
:
Pod
metadata
:
name
:
annotations-demo
:
imageregistry
:
"https://hub.docker.com/"
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.
3.7 - Field Selectors
Field selectors
let you
select Kubernetes resources
based on the value of one or more resource fields. Here are some examples of field selector queries:
metadata.name=my-service
metadata.namespace!=default
status.phase=Pending
This
kubectl
command selects all Pods for which the value of the
field is
Running
:
kubectl get pods --field-selector status.phase
=
Running
Note:
Field selectors are essentially resource
filters
. By default, no selectors/filters are applied, meaning that all resources of the specified type are selected. This makes the
kubectl
queries
and
kubectl get pods --field-selector ""
equivalent.
Supported fields
Supported field selectors vary by Kubernetes resource type. All resource types support the
and
fields. Using unsupported field selectors produces an error. For example:
kubectl get ingress --field-selector foo.bar
=
baz
Error from server (BadRequest): Unable to find "ingresses" that match label selector "", field selector "foo.bar=baz": "foo.bar" is not a known field selector: only "metadata.name", "metadata.namespace"
Supported operators
=
,
==
, and
!=
operators with field selectors (
=
and
==
mean the same thing). This
kubectl
command, for example, selects all Kubernetes Services that aren't in the
default
kubectl get services  --all-namespaces --field-selector metadata.namespace!
=
default
Chained selectors
As with
label
and other selectors, field selectors can be chained together as a comma-separated list. This
kubectl
command selects all Pods for which the
does not equal
Running
and the
spec.restartPolicy
field equals
Always
:
=
status.phase!
=
Running,spec.restartPolicy
=
Always
Multiple resource types
You can use field selectors across multiple resource types. This
kubectl
command selects all Statefulsets and Services that are not in the
default
kubectl get statefulsets,services --all-namespaces --field-selector metadata.namespace!
=
default
3.8 - Finalizers
Finalizers are namespaced keys that tell Kubernetes to wait until specific
conditions are met before it fully deletes resources marked for deletion.
Finalizers alert
to clean up resources the deleted object owned.
When you tell Kubernetes to delete an object that has finalizers

 specified for
it, the Kubernetes API marks the object for deletion by populating
.metadata.deletionTimestamp
,
and returns a
202
status code (HTTP "Accepted"). The target object remains in a terminating state while the
control plane, or other components, take the actions defined by the finalizers.
After these actions are complete, the controller removes the relevant finalizers
from the target object. When the
metadata.finalizers
field is empty,
Kubernetes considers the deletion complete and deletes the object.
You can use finalizers to control
of resources. For example, you can define a finalizer to clean up related resources or
infrastructure before the controller deletes the target resource.
of resources by alerting
to perform specific cleanup tasks before
deleting the target resource.
Finalizers don't usually specify the code to execute. Instead, they are
typically lists of keys on a specific resource similar to annotations.
Kubernetes specifies some finalizers automatically, but you can also specify
your own.
How finalizers work
When you create a resource using a manifest file, you can specify finalizers in
the
field. When you attempt to delete the resource, the
API server handling the delete request notices the values in the
field
and does the following:
Modifies the object to add a
metadata.deletionTimestamp
field with the
time you started the deletion.
Prevents the object from being removed until its
field is empty.
Returns a
202
status code (HTTP "Accepted")
The controller managing that finalizer notices the update to the object setting the
, indicating deletion of the object has been requested.
The controller then attempts to satisfy the requirements of the finalizers
specified for that resource. Each time a finalizer condition is satisfied, the
controller removes that key from the resource's
field. When the
field is emptied, an object with a
deletionTimestamp
field set
is automatically deleted. You can also use finalizers to prevent deletion of unmanaged resources.
A common example of a finalizer is
, which prevents
accidental deletion of
objects. When a
object is in use by a Pod, Kubernetes adds the
pv-protection
finalizer. If you
try to delete the
, it enters a
status, but the
controller can't delete it because the finalizer exists. When the Pod stops
using the
, Kubernetes clears the
finalizer,
and the controller deletes the volume.
Owner references, labels, and finalizers
Like
labels
,
owner references
describe the relationships between objects in Kubernetes, but are used for a
different purpose. When a
manages objects
like Pods, it uses labels to track changes to groups of related objects. For
example, when a
Job
creates one or
more Pods, the Job controller applies labels to those pods and tracks changes to
any Pods in the cluster with the same label.
The Job controller also adds
to those Pods, pointing at the
Job that created the Pods. If you delete the Job while these Pods are running,
Kubernetes uses the owner references (not labels) to determine which Pods in the
cluster need cleanup.
Kubernetes also processes finalizers when it identifies owner references on a
resource targeted for deletion.
In some situations, finalizers can block the deletion of dependent objects,
which can cause the targeted owner object to remain for
longer than expected without being fully deleted. In these situations, you
should check finalizers and owner references on the target owner and dependent
objects to troubleshoot the cause.
Note:
In cases where objects are stuck in a deleting state, avoid manually
removing finalizers to allow deletion to continue. Finalizers are usually added
to resources for a r

eason, so forcefully removing them can lead to issues in
your cluster. This should only be done when the purpose of the finalizer is
understood and is accomplished in another way (for example, manually cleaning
up some dependent object).
Read
Using Finalizers to Control Deletion
on the Kubernetes blog.
3.9 - Owners and Dependents
In Kubernetes, some objects are
owners
of other objects. For example, a
is the owner of a set of Pods. These owned objects are
dependents
of their owner.
Ownership is different from the
mechanism that some resources also use. For example, consider a Service that
creates
objects. The Service uses labels to allow the control plane to
determine which
objects are used for that Service. In addition
to the labels, each
that is managed on behalf of a Service has
an owner reference. Owner references help different parts of Kubernetes avoid
interfering with objects they don’t control.
Owner references in object specifications
Dependent objects have a
field that references their
owner object. A valid owner reference consists of the object name and a UID
within the same namespace as the dependent object. Kubernetes sets the value of
this field automatically for objects that are dependents of other objects like
ReplicaSets, DaemonSets, Deployments, Jobs and CronJobs, and ReplicationControllers.
You can also configure these relationships manually by changing the value of
this field. However, you usually don't need to and can allow Kubernetes to
automatically manage the relationships.
Dependent objects also have an
ownerReferences.blockOwnerDeletion
field that
takes a boolean value and controls whether specific dependents can block garbage
collection from deleting their owner object. Kubernetes automatically sets this
field to
true
if a
(for example, the Deployment controller) sets the value of the
field. You can also set the value of the
field manually to control which dependents block garbage
collection.
A Kubernetes admission controller controls user access to change this field for
dependent resources, based on the delete permissions of the owner. This control
prevents unauthorized users from delaying owner object deletion.
Note:
Cross-namespace owner references are disallowed by design.
Namespaced dependents can specify cluster-scoped or namespaced owners.
A namespaced owner
must
exist in the same namespace as the dependent.
If it does not, the owner reference is treated as absent, and the dependent
is subject to deletion once all owners are verified absent.
Cluster-scoped dependents can only specify cluster-scoped owners.
In v1.20+, if a cluster-scoped dependent specifies a namespaced kind as an owner,
it is treated as having an unresolvable owner reference, and is not able to be garbage collected.
In v1.20+, if the garbage collector detects an invalid cross-namespace
ownerReference
,
or a cluster-scoped dependent with an
referencing a namespaced kind, a warning Event
with a reason of
OwnerRefInvalidNamespace
and an
involvedObject
of the invalid dependent is reported.
You can check for that kind of Event by running
kubectl get events -A --field-selector=reason=OwnerRefInvalidNamespace
.
Ownership and finalizers
When you tell Kubernetes to delete a resource, the API server allows the
managing controller to process any
finalizer rules
for the resource.
prevent accidental deletion of resources your cluster may still need to function
correctly. For example, if you try to delete a
that is still
in use by a Pod, the deletion does not happen immediately because the
has the
finalizer on it.
Instead, the volume remains in the
status until Kubernetes clears
the finalizer, which only happens after the
is no longer
bound to a Pod.
Kubernetes al

so adds finalizers to an owner resource when you use either
foreground or orphan cascading deletion
.
In foreground deletion, it adds the
finalizer so that the
controller must delete dependent resources that also have
ownerReferences.blockOwnerDeletion=true
before it deletes the owner. If you
specify an orphan deletion policy, Kubernetes adds the
orphan
finalizer so
that the controller ignores dependent resources after it deletes the owner
object.
Kubernetes finalizers
.
.
Read the API reference for
object metadata
.
3.10 - Recommended Labels
You can visualize and manage Kubernetes objects with more tools than kubectl and
the dashboard. A common set of labels allows tools to work interoperably, describing
objects in a common manner that all tools can understand.
In addition to supporting tooling, the recommended labels describe applications
in a way that can be queried.
The metadata is organized around the concept of an
application
. Kubernetes is not
a platform as a service (PaaS) and doesn't have or enforce a formal notion of an application.
Instead, applications are informal and described with metadata. The definition of
what an application contains is loose.
Note:
These are recommended labels. They make it easier to manage applications
but aren't required for any core tooling.
Shared labels and annotations share a common prefix:
app.kubernetes.io
. Labels
without a prefix are private to users. The shared prefix ensures that shared labels
do not interfere with custom user labels.
Labels
In order to take full advantage of using these labels, they should be applied
on every resource object.
Key
Example
Type
The name of the application
mysql
string
app.kubernetes.io/instance
A unique name identifying the instance of an application
mysql-abcxzy
string
app.kubernetes.io/version
The current version of the application (e.g., a
SemVer 1.0
, revision hash, etc.)
5.7.21
string
app.kubernetes.io/component
The component within the architecture
database
string
app.kubernetes.io/part-of
The name of a higher level application this one is part of
wordpress
string
app.kubernetes.io/managed-by
The tool being used to manage the operation of an application
helm
string
To illustrate these labels in action, consider the following
object:
# This is an excerpt
:
apps/v1
kind
:
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
database
:
wordpress
:
helm
Applications And Instances Of Applications
An application can be installed one or more times into a Kubernetes cluster and,
in some cases, the same namespace. For example, WordPress can be installed more
than once where different websites are different installations of WordPress.
The name of an application and the instance name are recorded separately. For
example, WordPress has a
of
wordpress
while it has
an instance name, represented as
with a value of
wordpress-abcxzy
. This enables the application and instance of the application
to be identifiable. Every instance of an application must have a unique name.
Examples
To illustrate different ways to use these labels the following examples have varying complexity.
A Simple Stateless Service
Consider the case for a simple stateless service deployed using
and
Service
objects. The following two snippets represent how the labels could be used in their simplest form.
The
is used to oversee the pods running the application itself.
:
apps/v1
kind
:
metadata
:
labels
:
:
myservice
:
myservice-abcxzy
...
The
Service
is used to expose the application.
:
v1
kind
:
Service
metadata
:
labels
:
app.kubernetes.i

o/name
:
myservice
:
...
Web Application With A Database
Consider a slightly more complicated application: a web application (WordPress)
using a database (MySQL), installed using Helm. The following snippets illustrate
the start of objects used to deploy this application.
The start to the following
is used for WordPress:
:
apps/v1
kind
:
metadata
:
labels
:
:
wordpress
:
:
"4.9.4"
:
helm
:
server
:
wordpress
...
The
Service
is used to expose WordPress:
:
v1
kind
:
Service
metadata
:
labels
:
:
wordpress
:
:
"4.9.4"
:
helm
:
server
:
wordpress
...
MySQL is exposed as a
with metadata for both it and the larger application it belongs to:
:
apps/v1
kind
:
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
helm
:
database
:
wordpress
...
The
Service
is used to expose MySQL as part of WordPress:
:
v1
kind
:
Service
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
helm
:
database
:
wordpress
...
With the MySQL
and
Service
you'll notice information about both MySQL and WordPress, the broader application, are included.
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubelet authentication/authorization | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 Up kubectl on Linux
Debug Pods
Debug R

unning Pods
Auditing
Run Jobs
TLS
Tutorials
Security

Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kub

e-apiserver Encryption Configuration (v1)
Reference
Overview
A kubelet's HTTPS endpoint exposes APIs which give access to data of varying sensitivity,
and allow you to perform operations with varying levels of power on the node and within containers.
This document describes how to authenticate and authorize access to the kubelet's HTTPS endpoint.
Kubelet authentication
By default, requests to the kubelet's HTTPS endpoint that are not rejected by other configured
authentication methods are treated as anonymous requests, and given a username of
system:anonymous
and a group of
.
To disable anonymous access and send
401 Unauthorized
responses to unauthenticated requests:
start the kubelet with the
--anonymous-auth=false
flag
To enable X509 client certificate authentication to the kubelet's HTTPS endpoint:
flag, providing a CA bundle to verify client certificates with
start the apiserver with
and
flags
see the
apiserver authentication documentation
To enable API bearer tokens (including service account tokens) to be used to authenticate to the kubelet's HTTPS endpoint:
ensure the
authentication.k8s.io/v1beta1
API group is enabled in the API server
--authentication-token-webhook
and
flags
the kubelet calls the
API on the configured API server to determine user information from bearer tokens
Kubelet authorization
Any request that is successfully authenticated (including an anonymous request) is then authorized. The default authorization mode is
AlwaysAllow
, which allows all requests.
There are many possible reasons to subdivide access to the kubelet API:
anonymous auth is enabled, but anonymous users' ability to call the kubelet API should be limited
bearer token auth is enabled, but arbitrary API users' (like service accounts) ability to call the kubelet API should be limited
client certificate auth is enabled, but only some of the client certificates signed by the configured CA should be allowed to use the kubelet API
To subdivide access to the kubelet API, delegate authorization to the API server:
ensure the
authorization.k8s.io/v1beta1
--authorization-mode=Webhook
and the
flags
API on the configured API server to determine whether each request is authorized
The kubelet authorizes

 API requests using the same
request attributes
approach as the apiserver.
The verb is determined from the incoming request's HTTP verb:
HTTP verb
request verb
POST
create
GET, HEAD
get
PUT
update
PATCH
patch
DELETE
delete
The resource and subresource is determined from the incoming request's path:
Kubelet API
resource
subresource
/stats/*
nodes
stats
/metrics/*
nodes
metrics
/logs/*
nodes
log
/spec/*
nodes
spec
all others
nodes
proxy
The namespace and API group attributes are always an empty string, and
the resource name is always the name of the kubelet's
Node
When running in this mode, ensure the user identified by the
and
flags passed to the apiserver is authorized for the following attributes:
verb=*, resource=nodes, subresource=proxy
verb=*, resource=nodes, subresource=stats
verb=*, resource=nodes, subresource=log
verb=*, resource=nodes, subresource=spec
verb=*, resource=nodes, subresource=metrics
Feedback
Yes
No
.
or
.
Last modified October 24, 2022 at 1:47 PM PST:
Update content/en/docs/reference/access-authn-authz (f24c201017)
Overview
Home
Blog
Training
Partners
Community
CC BY 4.0

Common Parameters | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".
This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by
https://golang.org/pkg/unicode/#IsPrint
.
A selector to restrict the list of returned objects by their fields. Defaults to everything.
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the
ServerSideFieldValidation
feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the o

bject, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the
feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the
feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
A selector to restrict the list of returned objects by their labels. Defaults to everything.
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the
continue
field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.
The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
namespace
object name and auth scope, such as for teams and projects
pretty
If 'true', then the output is pretty printed.
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
resourceVersion sets a constraint on what resource versions a request may be served from. See
https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions
Defaults to unset
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See
Timeout for the l

ist/watch call. This limits the duration of the call, regardless of any activity or inactivity.
watch
Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Feedback
Yes
No
.
or
.
continue
dryRun
force
limit
namespace
pretty
watch
Home
Blog
Training
Partners
Community
CC BY 4.0

Reference | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Reference
1:
Glossary
2:
2.1:
2.2:
2.3:
2.4:
2.5:
2.6:
3:
3.1:
3.2:
3.3:
3.4:
3.5:
3.6:
3.7:
3.8:
3.9:
3.10:
3.11:
3.12:
3.13:
3.14:
3.15:
4:
4.1:
5:
5.1:
5.1.1:
Pod
5.1.2:
5.1.3:
5.1.4:
5.1.5:
5.1.6:
5.1.7:
5.1.8:
DaemonSet
5.1.9:
Job
5.1.10:
CronJob
5.1.11:
5.1.12:
5.1.13:
5.1.14:
5.1.15:
5.1.16:
5.1.17:
5.2:
5.2.1:
Service
5.2.2:
Endpoints
5.2.3:
5.2.4:
Ingress
5.2.5:
5.3:
5.3.1:
ConfigMap
5.3.2:
Secret
5.3.3:
Volume
5.3.4:
5.3.5:
5.3.6:
5.3.7:
5.3.8:
CSIDriver
5.3.9:
CSINode
5.3.10:
5.4:
5.4.1:
5.4.2:
5.4.3:
5.4.4:
5.5:
5.5.1:
5.5.2:
5.5.3:
5.5.4:
5.5.5:
5.5.6:
5.5.7:
5.5.8:
Role
5.5.9:
5.6:
5.6.1:
5.6.2:
5.6.3:
5.6.4:
5.7:
5.7.1:
5.7.2:
5.7.3:
5.7.4:
5.8:
5.8.1:
Node
5.8.2:
Namespace
5.8.3:
Event
5.8.4:
5.8.5:
Lease
5.8.6:
5.8.7:
5.8.8:
5.8.9:
Binding
5.8.10:
5.8.11:
5.9:
5.9.1:
5.9.2:
5.9.3:
ListMeta
5.9.4:
5.9.5:
5.9.6:
5.9.7:
5.9.8:
5.9.9:
Patch
5.9.10:
Quantity
5.9.11:
5.9.12:
Status
5.9.13:
5.10:
5.10.1:
5.11:
6:
6.1:
Kubernetes Component SLI Metrics
6.2:
6.3:
7:
7.1:
7.2:
7.3:
8:
8.1:
8.2:
8.3:
9:
9.1:
9.2:
9.3:
10:
10.1:
Kubeadm
10.1.1:

10.1.1.1:
10.1.1.2:
10.1.1.3:
10.1.1.4:
10.1.1.5:
10.1.1.6:
10.1.1.7:
10.1.1.8:
10.1.1.9:
10.1.1.10:
10.1.1.11:
10.1.1.12:
10.1.1.13:
10.1.1.14:
10.1.1.15:
10.1.1.16:
10.1.1.17:
10.1.1.18:
10.1.1.19:
10.1.1.20:
10.1.1.21:
10.1.1.22:
10.1.1.23:
10.1.1.24:
10.1.1.25:
10.1.1.26:
10.1.1.27:
10.1.1.28:
10.1.1.29:
10.1.1.30:
10.1.1.31:
10.1.1.32:
10.1.1.33:
10.1.1.34:
10.1.1.35:
10.1.1.36:
10.1.1.37:
10.1.1.38:
10.1.1.39:
10.1.1.40:
10.1.1.41:
10.1.1.42:
10.1.1.43:
10.1.1.44:
10.1.1.45:
10.1.1.46:
10.1.1.47:
10.1.1.48:
10.1.1.49:
10.1.1.50:
10.1.1.51:
10.1.1.52:
10.1.1.53:
10.1.1.54:
10.1.1.55:
10.1.1.56:
10.1.1.57:
10.1.1.58:
10.1.1.59:
10.1.1.60:
10.1.1.61:
10.1.1.62:
10.1.1.63:
10.1.1.64:
10.1.1.65:
10.1.1.66:
10.1.1.67:
10.1.1.68:
10.1.1.69:
10.1.1.70:
10.1.1.71:
10.1.1.72:
10.1.1.73:
10.1.1.74:
10.1.1.75:
10.1.1.76:
10.1.1.77:
10.1.1.78:
10.1.1.79:
10.1.1.80:
10.1.1.81:
10.1.1.82:
10.1.1.83:
10.1.1.84:
10.1.1.85:
10.1.1.86:
10.1.1.87:
10.1.1.88:
10.1.1.89:
10.1.1.90:
10.1.1.91:
10.1.1.92:
10.1.1.93:
10.1.1.94:
10.1.1.95:
10.1.1.96:
10.1.1.97:
10.1.1.98:
10.1.1.99:
10.1.1.100:
10.1.1.101:
10.1.1.102:
10.1.1.103:
10.1.1.104:
10.1.1.105:
10.1.1.106:
10.1.1.107:
10.1.1.108:
10.1.1.109:
10.1.2:
10.1.3:
10.1.4:
10.1.5:
10.1.6:
10.1.7:
10.1.8:
10.1.9:
10.1.10:
10.1.11:
10.1.12:
10.1.13:
10.1.14:
10.1.15:
10.1.16:
11:
11.1:
11.2:
11.3:
kubectl
11.4:
11.5:
11.6:
12:
12.1:
12.2:
12.3:
kubelet
12.4:
12.5:
12.6:
12.7:
13:
13.1:
13.2:
13.3:
13.4:
13.5:
13.6:
13.7:
13.8:
13.9:
13.10:
13.11:
13.12:
13.13:
13.14:
13.15:
13.16:
13.17:
13.18:
13.19:
13.20:
13.21:
13.22:
13.23:
14:
14.1:
14.2:
15:
15.1:
This section of the Kubernetes documentation contains references.
Glossary
- a comprehensive, standardized list of Kubernetes terminology
One-page API Reference for Kubernetes v1.26
Using The Kubernetes API
- overview of the API for Kubernetes.
API access control
- details on how Kubernetes controls API access
Officially supported client libraries
To call the Kubernetes API from a programming language, you can use
. Officially supported
client libraries:
Kubernetes Go client library
Kubernetes Python client library
Kubernetes Java client library
Kubernetes JavaScript client library
Kubernetes C# client library
Kubernetes Haskell client library
CLI
kubectl
- Main CLI tool for running commands and managing Kubernetes clusters.
JSONPath
- Syntax guide for using
JSONPath expressions
with kubectl.
kubeadm
- CLI tool to easily provision a secure Kubernetes cluster.
Components
kubel

et
- The
primary agent that runs on each node. The kubelet takes a set of PodSpecs
and ensures that the described containers are running and healthy.
-
REST API that validates and configures data for API objects such as pods,
services, replication controllers.
-
Daemon that embeds the core control loops shipped with Kubernetes.
- Can
do simple TCP/UDP stream forwarding or round-robin TCP/UDP forwarding across
a set of back-ends.
-
Scheduler that manages availability, performance, and capacity.
Scheduler Policies
Scheduler Profiles
List of
ports and protocols
that
should be open on control plane and worker nodes
Config APIs
This section hosts the documentation for "unpublished" APIs which are used to
configure kubernetes components or tools. Most of these APIs are not exposed
by the API server in a RESTful way though they are essential for a user or an
operator to use or manage a cluster.
kube-apiserver admission (v1)
kube-apiserver configuration (v1alpha1)
and
kube-apiserver configuration (v1)
kube-apiserver encryption (v1)
kube-apiserver event rate limit (v1alpha1)
kubelet configuration (v1alpha1)
and
kubelet configuration (v1beta1)
kubelet configuration (v1)
kubelet credential providers (v1alpha1)
,
kubelet credential providers (v1beta1)
and
kubelet credential providers (v1)
kube-scheduler configuration (v1beta2)
,
kube-scheduler configuration (v1beta3)
and
kube-scheduler configuration (v1)
kube-proxy configuration (v1alpha1)
API
Client authentication API (v1beta1)
and
Client authentication API (v1)
WebhookAdmission configuration (v1)
ImagePolicy API (v1alpha1)
Config API for kubeadm
v1beta2
v1beta3
Design Docs
An archive of the design docs for Kubernetes functionality. Good starting points are
Kubernetes Architecture
and
Kubernetes Design Overview
.
1 - Glossary
2 - API Overview
This section provides reference information for the Kubernetes API.
The REST API is the fundamental fabric of Kubernetes. All operations and
communications between components, and external user commands are REST API
calls that the API Server handles. Consequently, everything in the Kubernetes
platform is treated as an API object and has a corresponding entry in the
API
.
The
Kubernetes API reference
lists the API for Kubernetes version v1.26.
For general background information, read
.
describes how clients can authenticate to the Kubernetes API server, and how their
requests are authorized.
API versioning
The JSON and Protobuf serialization schemas follow the same guidelines for
schema changes. The following descriptions cover both formats.
The API versioning and software versioning are indirectly related.
The
API and release versioning proposal
describes the relationship between API versioning and software versioning.
Different API versions indicate different levels of stability and support. You
can find more information about the criteria for each level in the
API Changes documentation
.
Here's a summary of each level:
Alpha:
The version names contain
alpha
(for example,
v1alpha1
).
Built-in alpha API versions are disabled by default and must be explicitly enabled in the
configuration to be used.
The software may contain bugs. Enabling a feature may expose bugs.
Support for an alpha API may be dropped at any time without notice.
The API may change in incompatible ways in a later software release without notice.
The software is recommended for use only in short-lived testing clusters,
due to increased risk of bugs and lack of long-term support.
Beta:
beta
v2beta3
).
Built-in beta API versions are disabled by default and must be explicitly enabled in the
configuration to be used
(
except
for beta versions of APIs introduced prior to Kubernetes 1.22, which were enabled by default).
Built-in beta API versions have a maximum life

time of 9 months or 3 minor releases (whichever is longer) from introduction
to deprecation, and 9 months or 3 minor releases (whichever is longer) from deprecation to removal.
The software is well tested. Enabling a feature is considered safe.
The support for a feature will not be dropped, though the details may change.
The schema and/or semantics of objects may change in incompatible ways in
a subsequent beta or stable API version. When this happens, migration
instructions are provided. Adapting to a subsequent beta or stable API version
may require editing or re-creating API objects, and may not be straightforward.
The migration may require downtime for applications that rely on the feature.
The software is not recommended for production uses. Subsequent releases
may introduce incompatible changes. Use of beta API versions is
required to transition to subsequent beta or stable API versions
once the beta API version is deprecated and no longer served.
Note:
Please try beta features and provide feedback. After the features exit beta, it
may not be practical to make more changes.
Stable:
The version name is
vX
where
X
is an integer.
Stable API versions remain available for all future releases within a Kubernetes major version,
and there are no current plans for a major version revision of Kubernetes that removes stable APIs.
API groups
API groups
make it easier to extend the Kubernetes API.
The API group is specified in a REST path and in the
field of a
serialized object.
There are several API groups in Kubernetes:
The
core
(also called
legacy
) group is found at REST path
/api/v1
.
The core group is not specified as part of the
field, for
example,
.
The named groups are at REST path
/apis/$GROUP_NAME/$VERSION
and use
apiVersion: $GROUP_NAME/$VERSION
apiVersion: batch/v1
).
You can find the full list of supported API groups in
.
Enabling or disabling API groups
Certain resources and API groups are enabled by default. You can enable or
disable them by setting
on the API server. The
flag accepts comma separated
&lt;key&gt;[=&lt;value&gt;]
pairs
describing the runtime configuration of the API server. If the
=&lt;value&gt;
part is omitted, it is treated as if
=true
is specified. For example:
to disable
batch/v1
, set
--runtime-config=batch/v1=false
to enable
batch/v2alpha1
, set
--runtime-config=batch/v2alpha1
to enable a specific version of an API, such as
storage.k8s.io/v1beta1/csistoragecapacities
, set
--runtime-config=storage.k8s.io/v1beta1/csistoragecapacities
Note:
When you enable or disable groups or resources, you need to restart the API
server and controller manager to pick up the
changes.
Kubernetes stores its serialized state in terms of the API resources by writing them into
etcd
.
API conventions
Read the design documentation for
aggregator
2.1 - Kubernetes API Concepts
The Kubernetes API is a resource-based (RESTful) programmatic interface
provided via HTTP. It supports retrieving, creating, updating, and deleting
primary resources via the standard HTTP verbs (POST, PUT, PATCH, DELETE,
GET).
For some resources, the API includes additional subresources that allow
fine grained authorization (such as separate views for Pod details and
log retrievals), and can accept and serve those resources in different
representations for convenience or efficiency.
Kubernetes supports efficient change notifications on resources via
watches
.
Kubernetes also provides consistent list operations so that API clients can
effectively cache, track, and synchronize the state of resources.
You can view the
online,
or read on to learn about the API in general.
Kubernetes API terminology
Kubernetes generally leverages common RESTful terminology to describe the
API concepts:
A
is the name used in the URL (
pods
,
,
services
)
All resource typ

es have a concrete representation (their object schema) which is called a
kind
A list of instances of a resource is known as a
collection
A single instance of a resource type is called a
resource
, and also usually represents an
object
For some resource types, the API includes one or more
sub-resources
, which are represented as URI paths below the resource
Most Kubernetes API resource types are
objects
:
they represent a concrete instance of a concept on the cluster, like a
pod or namespace. A smaller number of API resource types are
virtual
in
that they often represent operations on objects, rather than objects, such
as a permission check
(use a POST with a JSON-encoded body of
to the
resource), or the
eviction
sub-resource of a Pod
(used to trigger
).
Object names
All objects you can create via the API have a unique object
name
to allow idempotent creation and
retrieval, except that virtual resource types may not have unique names if they are
not retrievable, or do not rely on idempotency.
Within a
namespace
, only one object
of a given kind can have a given name at a time. However, if you delete the object,
you can make a new object with the same name. Some objects are not namespaced (for
example: Nodes), and so their names must be unique across the whole cluster.
API verbs
Almost all object resource types support the standard HTTP verbs - GET, POST, PUT, PATCH,
and DELETE. Kubernetes also uses its own verbs, which are often written lowercase to distinguish
them from HTTP verbs.
Kubernetes uses the term
list
to describe returning a
of
resources to distinguish from retrieving a single resource which is usually called
a
get
. If you sent an HTTP GET request with the
?watch
query parameter,
Kubernetes calls this a
watch
and not a
get
(see
Efficient detection of changes
for more details).
For PUT requests, Kubernetes internally classifies these as either
create
or
update
based on the state of the existing object. An
update
is different from a
patch
; the
HTTP verb for a
patch
is PATCH.
Resource URIs
All resource types are either scoped by the cluster (
/apis/GROUP/VERSION/*
) or to a
namespace (
/apis/GROUP/VERSION/namespaces/NAMESPACE/*
). A namespace-scoped resource
type will be deleted when its namespace is deleted and access to that resource type
is controlled by authorization checks on the namespace scope.
Note: core resources use
/api
instead of
/apis
and omit the GROUP path segment.
Examples:
/api/v1/namespaces
/api/v1/pods
/api/v1/namespaces/my-namespace/pods
/apis/apps/v1/deployments
/apis/apps/v1/namespaces/my-namespace/deployments
/apis/apps/v1/namespaces/my-namespace/deployments/my-deployment
You can also access collections of resources (for example: listing all Nodes).
The following paths are used to retrieve collections and resources:
Cluster-scoped resources:
GET /apis/GROUP/VERSION/RESOURCETYPE
- return the collection of resources of the resource type
GET /apis/GROUP/VERSION/RESOURCETYPE/NAME
- return the resource with NAME under the resource type
Namespace-scoped resources:
- return the collection of all instances of the resource type across all namespaces
GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE
- return collection of all instances of the resource type in NAMESPACE
GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME
- return the instance of the resource type with NAME in NAMESPACE
Since a namespace is a cluster-scoped resource type, you can retrieve the list
(“collection”) of all namespaces with
GET /api/v1/namespaces
and details about
a particular namespace with
GET /api/v1/namespaces/NAME
.
Cluster-scoped subresource:
GET /apis/GROUP/VERSION/RESOURCETYPE/NAME/SUBRESOURCE
Namespace-scoped subresource:
GET /apis/GROUP/VERSION/namespaces/NAMESPACE/RESOURCETYPE/NAME/SUBRESOURCE
The verbs supported for each subresource will differ depending on the object -
see the
API refere

nce
for more information. It
is not possible to access sub-resources across multiple resources - generally a new
virtual resource type would be used if that becomes necessary.
The Kubernetes API allows clients to make an initial request for an object or a
collection, and then to track changes since that initial request: a
watch
. Clients
can send a
list
or a
get
and then make a follow-up
watch
request.
To make this change tracking possible, every Kubernetes object has a
field representing the version of that resource as stored in the underlying persistence
layer. When retrieving a collection of resources (either namespace or cluster scoped),
the response from the API server contains a
value. The client can
use that
to initiate a
watch
against the API server.
When you send a
watch
request, the API server responds with a stream of
changes. These changes itemize the outcome of operations (such as
create
,
delete
,
and
update
) that occurred after the
you specified as a parameter
to the
watch
request. The overall
watch
mechanism allows a client to fetch
the current state and then subscribe to subsequent changes, without missing any events.
If a client
watch
is disconnected then that client can start a new
watch
from
the last returned
; the client could also perform a fresh
get
/
list
request and begin again. See
Resource Version Semantics
for more detail.
List all of the pods in a given namespace.
GET /api/v1/namespaces/test/pods
---
200 OK
{
"kind": "PodList",
"metadata": {"resourceVersion":"10245"},
"items": [...]
}
Starting from resource version 10245, receive notifications of any API operations
(such as
create
,
delete
,
apply
or
update
) that affect Pods in the
test
namespace. Each change notification is a JSON document. The HTTP response body
(served as
) consists a series of JSON documents.
GET /api/v1/namespaces/test/pods?watch=1&amp;resourceVersion=10245
---
200 OK
Transfer-Encoding: chunked

{
  "type": "ADDED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
}
{
  "type": "MODIFIED",
  "object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "11020", ...}, ...}
}
...
A given Kubernetes server will only preserve a historical record of changes for a
limited time. Clusters using etcd 3 preserve changes in the last 5 minutes by default.
When the requested
watch
operations fail because the historical version of that
resource is not available, clients must handle the case by recognizing the status code
410 Gone
, clearing their local cache, performing a new
get
or
list
operation,
and starting the
watch
from the
that was returned.
For subscribing to collections, Kubernetes client libraries typically offer some form
of standard tool for this
list
-then-
watch
logic. (In the Go client library,
this is called a
Reflector
and is located in the
k8s.io/client-go/tools/cache
package.)
Watch bookmarks
To mitigate the impact of short history window, the Kubernetes API provides a watch
event named
BOOKMARK
. It is a special kind of event to mark that all changes up
to a given
the client is requesting have already been sent. The
document representing the
BOOKMARK
event is of the type requested by the request,
but only includes a
.metadata.resourceVersion
GET /api/v1/namespaces/test/pods?watch=1&amp;resourceVersion=10245&amp;allowWatchBookmarks=true
---
200 OK
{
"type": "ADDED",
"object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "10596", ...}, ...}
}
...
{
"type": "BOOKMARK",
"object": {"kind": "Pod", "apiVersion": "v1", "metadata": {"resourceVersion": "12746"} }
}
As a client, you can request
BOOKMARK
events by s

etting the
allowWatchBookmarks=true
query parameter to a
watch
request, but you shouldn't
assume bookmarks are returned at any specific interval, nor can clients assume that
the API server will send any
BOOKMARK
event even when requested.
Retrieving large results sets in chunks
Kubernetes v1.9 [beta]
On large clusters, retrieving the collection of some resource types may result in
very large responses that can impact the server and client. For instance, a cluster
may have tens of thousands of Pods, each of which is equivalent to roughly 2 KiB of
encoded JSON. Retrieving all pods across all namespaces may result in a very large
response (10-20MB) and consume a large amount of server resources.
Provided that you don't explicitly disable the
APIListChunking
, the
Kubernetes API server supports the ability to break a single large collection request
into many smaller chunks while preserving the consistency of the total request. Each
chunk can be returned sequentially which reduces both the total size of the request and
allows user-oriented clients to display results incrementally to improve responsiveness.
You can request that the API server handles a
list
by serving single collection
using pages (which Kubernetes calls
chunks
). To retrieve a single collection in
chunks, two query parameters
limit
and
continue
are supported on requests against
collections, and a response field
continue
is returned from all
list
operations
in the collection's
metadata
field. A client should specify the maximum results they
wish to receive in each chunk with
limit
and the server will return up to
limit
resources in the result and include a
continue
value if there are more resources
in the collection.
As an API client, you can then pass this
continue
value to the API server on the
next request, to instruct the server to return the next page (
chunk
) of results. By
continuing until the server returns an empty
continue
value, you can retrieve the
entire collection.
Like a
watch
operation, a
continue
token will expire after a short amount
of time (by default 5 minutes) and return a
410 Gone
if more results cannot be
returned. In this case, the client will need to start from the beginning or omit the
limit
parameter.
For example, if there are 1,253 pods on the cluster and you wants to receive chunks
of 500 pods at a time, request those chunks as follows:
List all of the pods on a cluster, retrieving up to 500 pods each time.
GET /api/v1/pods?limit=500
---
200 OK
{
"resourceVersion":"10245",
"continue": "ENCODED_CONTINUE_TOKEN",
"remainingItemCount": 753,
...
},
"items": [...] // returns pods 1-500
}
Continue the previous call, retrieving the next set of 500 pods.
GET /api/v1/pods?limit=500&amp;continue=ENCODED_CONTINUE_TOKEN
---
200 OK
{
"continue": "ENCODED_CONTINUE_TOKEN_2",
"remainingItemCount": 253,
...
},
"items": [...] // returns pods 501-1000
}
Continue the previous call, retrieving the last 253 pods.
GET /api/v1/pods?limit=500&amp;continue=ENCODED_CONTINUE_TOKEN_2
---
200 OK
{
"continue": "", // continue token is empty because we have reached the end of the list
...
},
"items": [...] // returns pods 1001-1253
}
Notice that the
of the collection remains constant across each request,
indicating the server is showing you a consistent snapshot of the pods. Pods that
are created, updated, or deleted after version
10245
would not be shown unless
you make a separate
list
request without the
continue
token. This allows you
to break large requests into smaller chunks and then perform a
watch
operation
on the full set without missing any updates.
remainingItemCount
is the number of subsequent items in the collec

tion that are not
included in this response. If the
list
request contained label or field
selectors
then the number of
remaining items is unknown and the API server does not include a
field in its response.
If the
list
is complete (either because it is not chunking, or because this is the
last chunk), then there are no more remaining items and the API server does not include a
field in its response. The intended use of the
is estimating the size of a collection.
Collections
In Kubernetes terminology, the response you get from a
list
is
a
. However, Kubernetes defines concrete kinds for
collections of different types of resource. Collections have a kind
named for the resource kind, with
List
appended.
When you query the API for a particular type, all items returned by that query are
of that type.
For example, when you
list
Services, the collection response
has
kind
set to
; each item in that collection represents a single Service. For example:
{
"kind":
"ServiceList"
,
"v1"
,
{
"2947301"
},
"items":
[
{
{
"name":
,
"default"
,
...
{
"name":
"kube-dns"
,
,
...
There are dozens of collection types (such as
PodList
,
,
and
NodeList
) defined in the Kubernetes API.
You can get more information about each collection type from the
Some tools, such as
kubectl
, represent the Kubernetes collection
mechanism slightly differently from the Kubernetes API itself.
Because the output of
kubectl
might include the response from
multiple
list
operations at the API level,
kubectl
represents
a list of items using
kind: List
kubectl get services -A -o yaml
:
v1
kind
:
List
metadata
:
:
""
selfLink
:
""
items
:
-
:
v1
kind
:
Service
metadata
:
:
"2021-06-03T14:54:12Z"
labels
:
component
:
apiserver
provider
:
name
:
namespace
:
default
...
-
:
v1
kind
:
Service
metadata
:
:
prometheus.io/port
:
"9153"
prometheus.io/scrape
:
"true"
:
"2021-06-03T14:54:14Z"
labels
:
k8s-app
:
kube-dns
:
"true"
kubernetes.io/name
:
CoreDNS
name
:
kube-dns
namespace
:
Note:
Keep in mind that the Kubernetes API does not have a
kind
named
List
.
kind: List
is a client-side, internal implementation detail for processing
collections that might be of different kinds of object. Avoid depending on
kind: List
in automation or other code.
Receiving resources as Tables
, the default output format is a simple tabular
representation of one or more instances of a particular resource type. In the past,
clients were required to reproduce the tabular and describe output implemented in
kubectl
to perform simple lists of objects.
A few limitations of that approach include non-trivial logic when dealing with
certain objects. Additionally, types provided by API aggregation or third party
resources are not known at compile time. This means that generic implementations
had to be in place for types unrecognized by a client.
In order to avoid potential limitations as described above, clients may request
the Table representation of objects, delegating specific details of printing to the
server. The Kubernetes API implements standard HTTP content type negotiation: passing
an
Accept
header containing a value of
application/json;as=Table;g=meta.k8s.io;v=v1
with a
GET
call will request that the server return objects in the Table content
type.
For example, list all of the pods on a cluster in the Table format.
GET /api/v1/pods
Accept: application/json;as=Table;g=meta.k8s.io;v=v1
---
200 OK
{
"kind": "Table",
"apiVersion": "meta.k8s.io/v1",
...
"columnDefinitions": [
...
]
}
For API resource types that do not ha

ve a custom Table definition known to the control
plane, the API server returns a default Table response that consists of the resource's
name
and
fields.
GET /apis/crd.example.com/v1alpha1/namespaces/default/resources
---
200 OK
...
{
...
{
"name": "Name",
"type": "string",
...
},
{
"name": "Created At",
"type": "date",
...
}
]
}
Not all API resource types support a Table response; for example, a
might not define field-to-table mappings, and an APIService that
extends the core Kubernetes API
might not serve Table responses at all. If you are implementing a client that
uses the Table information and must work against all resource types, including
extensions, you should make requests that specify multiple content types in the
Accept
header. For example:
Accept: application/json;as=Table;g=meta.k8s.io;v=v1, application/json
Alternate representations of resources
By default, Kubernetes returns objects serialized to JSON with content type
. This is the default serialization format for the API. However,
clients may request the more efficient
Protobuf representation
of these objects for better performance at scale.
The Kubernetes API implements standard HTTP content type negotiation: passing an
Accept
header with a
GET
call will request that the server tries to return
a response in your preferred media type, while sending an object in Protobuf to
the server for a
PUT
or
POST
call means that you must set the
Content-Type
header appropriately.
The server will return a response with a
header if the requested
format is supported, or the
406 Not acceptable
error if none of the media types you
requested are supported. All built-in resource types support the
media type.
See the Kubernetes
for a list of
supported content types for each API.
List all of the pods on a cluster in Protobuf format.
Accept: application/vnd.kubernetes.protobuf
---
200 OK
Content-Type: application/vnd.kubernetes.protobuf
... binary encoded PodList object
Create a pod by sending Protobuf encoded data to the server, but request a response
in JSON.
POST /api/v1/namespaces/test/pods
... binary encoded Pod object
---
200 OK
{
"kind": "Pod",
...
}
Not all API resource types support Protobuf; specifically, Protobuf isn't available for
resources that are defined as
or are served via the
.
As a client, if you might need to work with extension types you should specify multiple
content types in the request
Accept
header to support fallback to JSON.
Accept: application/vnd.kubernetes.protobuf, application/json
Kubernetes Protobuf encoding
Kubernetes uses an envelope wrapper to encode Protobuf responses. That wrapper starts
with a 4 byte magic number to help identify content in disk or in etcd as Protobuf
(as opposed to JSON), and then is followed by a Protobuf encoded wrapper message, which
describes the encoding and type of the underlying object and then contains the object.
The wrapper format is:
A four byte magic number prefix:
Bytes 0-3: "k8s\x00" [0x6b, 0x38, 0x73, 0x00]
An encoded Protobuf message with the following IDL:
message Unknown {
// typeMeta should have the string values for "kind" and "apiVersion" as set on the JSON object
optional TypeMeta typeMeta = 1;
// raw will hold the complete serialized object in protobuf. See the protobuf definitions in the client libraries for a given kind.
optional bytes raw = 2;
// contentEncoding is encoding used for the raw data. Unspecified means no encoding.
optional string contentEncoding = 3;
// contentType is the serialization method used to serialize 'raw'. Unspecified means application/vnd.kubernetes.protobuf an

d is usually
// omitted.
optional string contentType = 4;
}
message TypeMeta {
// apiVersion is the group/version for this type
optional string apiVersion = 1;
// kind is the name of the object schema. A protobuf definition should exist for this object.
optional string kind = 2;
}
Note:
Clients that receive a response in
application/vnd.kubernetes.protobuf
that does
not match the expected prefix should reject the response, as future versions may need
to alter the serialization format in an incompatible way and will do so by changing
the prefix.
Resource deletion
When you
delete
a resource this takes place in two phases.
finalization
removal
{
"kind":
"ConfigMap"
,
"v1"
,
{
"finalizers":
{
"url.io/neat-finalization"
,
"other-url.io/my-finalizer"
},
"deletionTimestamp":
nil,
}
}
When a client first sends a
delete
to request removal of a resource, the
is set to the current time.
Once the
is set, external controllers that act on finalizers
may start performing their cleanup work at any time, in any order.
Order is
not
enforced between finalizers because it would introduce significant
risk of stuck
.metadata.finalizers
.
The
field is shared: any actor with permission can reorder it.
If the finalizer list were processed in order, then this might lead to a situation
in which the component responsible for the first finalizer in the list is
waiting for some signal (field value, external system, or other) produced by a
component responsible for a finalizer later in the list, resulting in a deadlock.
Without enforced ordering, finalizers are free to order amongst themselves and are
not vulnerable to ordering changes in the list.
Once the last finalizer is removed, the resource is actually removed from etcd.
Single resource API
The Kubernetes API verbs
get
,
create
,
apply
,
update
,
patch
,
delete
and
proxy
support single resources only.
These verbs with single resource support have no support for submitting multiple
resources together in an ordered or unordered list or transaction.
When clients (including kubectl) act on a set of resources, the client makes a series
of single-resource API requests, then aggregates the responses if needed.
By contrast, the Kubernetes API verbs
list
and
watch
allow getting multiple
resources, and
allows deleting multiple resources.
Field validation
Kubernetes always validates the type of fields. For example, if a field in the
API is defined as a number, you cannot set the field to a text value. If a field
is defined as an array of strings, you can only provide an array. Some fields
allow you to omit them, other fields are required. Omitting a required field
from an API request is an error.
If you make a request with an extra field, one that the cluster's control plane
does not recognize, then the behavior of the API server is more complicated.
By default, the API server drops fields that it does not recognize
from an input that it receives (for example, the JSON body of a
PUT
request).
There are two situations where the API server drops fields that you supplied in
an HTTP request.
These situations are:
The field is unrecognized because it is not in the resource's OpenAPI schema. (One
exception to this is for
CRDs
that explicitly choose not to prune unknown
fields via
).
The field is duplicated in the object.
Setting the field validation level
Provided that the
is enabled (disabled
by default in 1.23 and 1.24, enabled by default starting in 1.25), you can take
advantage of server side field validation to catch these unrecognized fields.
When you use HTTP verbs that can submit data (
POST
,
PUT
, and
PATCH
), field
validation gives you the option to choose how you would like to be notified of
these fields that are being dropped by the API server. Possible levels of
validation 

are
Ignore
,
Warn
, and
Strict
.
Note:
If you submit a request that specifies an unrecognized field, and that is also invalid for
a different reason (for example, the request provides a string value where the API expects
an integer), then the API server responds with a 400 Bad Request error response.
You always receive an error response in this case, no matter what field validation level you requested.
Field validation is set by the
query parameter. The three
values that you can provide for this parameter are:
Ignore
The API server succeeds in handling the request as it would without the erroneous fields
being set, dropping all unknown and duplicate fields and giving no indication it
has done so.
Warn
(Default) The API server succeeds in handling the request, and reports a
warning to the client. The warning is sent using the
Warning:
response header,
adding one warning item for each unknown or duplicate field. For more
information about warnings and the Kubernetes API, see the blog article
Warning: Helpful Warnings Ahead
.
Strict
The API server rejects the request with a 400 Bad Request error when it
detects any unknown or duplicate fields. The response message from the API
server specifies all the unknown or duplicate fields that the API server has
detected.
Tools that submit requests to the server (such as
kubectl
), might set their own
defaults that are different from the
Warn
validation level that the API server uses
The
kubectl
tool uses the
flag to set the level of field validation.
Historically
was used to toggle client-side validation on or off as
a boolean flag. Since Kubernetes 1.25, kubectl uses
server-side field validation when sending requests to a server with this feature
enabled. Validation will fall back to client-side only when it cannot connect
to an API server with field validation enabled.
It accepts the values
ignore
,
warn
,
and
strict
while also accepting the values
true
(equivalent to
strict
) and
false
ignore
). The default validation setting for kubectl is
--validate=true
,
which means strict server-side field validation.
Dry-run
When you use HTTP verbs that can modify resources (
POST
,
PUT
,
PATCH
, and
DELETE
), you can submit your request in a
dry run
mode. Dry run mode helps to
evaluate a request through the typical request stages (admission chain, validation,
merge conflicts) up until persisting objects to storage. The response body for the
request is as close as possible to a non-dry-run response. Kubernetes guarantees that
dry-run requests will not be persisted in storage or have any other side effects.
Make a dry-run request
Dry-run is triggered by setting the
dryRun
query parameter. This parameter is a
string, working as an enum, and the only accepted values are:
[no value set]
Allow side effects. You request this with a query string such as
?dryRun
or
?dryRun&amp;pretty=true
. The response is the final object that would have been
persisted, or an error if the request could not be fulfilled.
All
Every stage runs as normal, except for the final storage stage where side effects
are prevented.
When you set
?dryRun=All
, any relevant
admission controllers
are run, validating admission controllers check the request post-mutation, merge is
performed on
PATCH
, fields are defaulted, and schema validation occurs. The changes
are not persisted to the underlying storage, but the final object which would have
been persisted is still returned to the user, along with the normal status code.
If the non-dry-run version of a request would trigger an admission controller that has
side effects, the request will be failed rather than risk an unwanted side effect. All
built in admission control plugins support dry-run. Additionally, admission webhooks can
declare in their
configuration object
that they do not have side effects, by setting their
sideEffects
field to
None
.
Note:
If a webhook actually does ha

ve side effects, then the
field should be
set to "NoneOnDryRun". That change is appropriate provided that the webhook is also
be modified to understand the
DryRun
field in AdmissionReview, and to prevent side
effects on any request marked as dry runs.
Here is an example dry-run request that uses
:
POST /api/v1/namespaces/test/pods?dryRun=All
The response would look the same as for non-dry-run request, but the values of some
generated fields may differ.
Generated values
Some values of an object are typically generated before the object is persisted. It
is important not to rely upon the values of these fields set by a dry-run request,
since these values will likely be different in dry-run mode from when the real
request is made. Some of these fields are:
name
: if
is set,
name
will have a unique random name
/
: records the time of creation/deletion
UID
:
uniquely identifies
the object and is randomly generated (non-deterministic)
: tracks the persisted version of the object
Any field set by a mutating admission controller
For the
Service
resource: Ports or IP addresses that the kube-apiserver assigns to Service objects
Dry-run authorization
Authorization for dry-run and non-dry-run requests is identical. Thus, to make
a dry-run request, you must be authorized to make the non-dry-run request.
For example, to run a dry-run
patch
for a Deployment, you must be authorized
to perform that
patch
. Here is an example of a rule for Kubernetes
RBAC
that allows patching
Deployments:
rules
:
-
apiGroups
:
[
"apps"
]
resources
:
[
"deployments"
]
verbs
:
[
"patch"
]
See
.
Server Side Apply
Kubernetes'
feature allows the control plane to track managed fields for newly created objects.
Server Side Apply provides a clear pattern for managing field conflicts,
offers server-side
Apply
and
Update
operations, and replaces the
client-side functionality of
.
The API verb for Server-Side Apply is
apply
.
See
Resource versions
Resource versions are strings that identify the server's internal version of an
object. Resource versions can be used by clients to determine when objects have
changed, or to express data consistency requirements when getting, listing and
watching resources. Resource versions must be treated as opaque by clients and passed
unmodified back to the server.
You must not assume resource versions are numeric or collatable. API clients may
only compare two resource versions for equality (this means that you must not compare
resource versions for greater-than or less-than relationships).
fields in metadata
Clients find resource versions in resources, including the resources from the response
stream for a
watch
, or when using
list
to enumerate resources.
v1.meta/ObjectMeta
- The
metadata.resourceVersion
of a resource instance identifies the resource version the instance was last modified at.
v1.meta/ListMeta
- The
of a resource collection (the response to a
list
) identifies the resource version at which the collection was constructed.
parameters in query strings
The
get
,
list
, and
watch
operations support the
From version v1.19, Kubernetes API servers also support the
parameter on
list
requests.
The API server interprets the
parameter differently depending
on the operation you request, and on the value of
. If you set
then this also affects the way matching happens.
Semantics for
get
and
list
For
get
and
list
, the semantics of
are:
get:
resourceVersion unset
resourceVersion="0"
resourceVersion="{value other than 0}"
Most Recent
Any
Not older than
list:
From version v1.19, Kubernetes API servers support the
para

meter
on
list
requests. If you set both
and
, the
parameter determines how the API server interprets
.
You should always set the
parameter when setting
on a
list
request. However, be prepared to handle the case
where the API server that responds is unaware of
and ignores it.
Unless you have strong consistency requirements, using
resourceVersionMatch=NotOlderThan
and
a known
is preferable since it can achieve better performance and scalability
of your cluster than leaving
and
unset, which requires
quorum read to be served.
Setting the
parameter without setting
is not valid.
This table explains the behavior of
list
requests with various combinations of
and
:
resourceVersionMatch and paging parameters for list
resourceVersionMatch param
paging params
resourceVersion not set
unset
limit unset
Any
unset
limit=&lt;n&gt;,
continue unset
Any
Exact
unset
limit=&lt;n&gt;, continue=&lt;token&gt;
Continue Token, Exact
Invalid, treated as Continue Token, Exact
Invalid, HTTP
400 Bad Request
resourceVersionMatch=Exact
Invalid
Invalid
Exact
Invalid
Invalid
Exact
Invalid
Any
Invalid
Any
Note:
If your cluster's API server does not honor the
parameter,
the behavior is the same as if you did not set it.
The meaning of the
get
and
list
semantics are:
Any
Return data at any resource version. The newest available resource version is preferred,
but strong consistency is not required; data at any resource version may be served. It is possible
for the request to return data at a much older resource version that the client has previously
observed, particularly in high availability configurations, due to partitions or stale
caches. Clients that cannot tolerate this should not use this semantic.
Most recent
Return data at the most recent resource version. The returned data must be
consistent (in detail: served from etcd via a quorum read).
Return data at least as new as the provided
. The newest
available data is preferred, but any data not older than the provided
may be
served. For
list
requests to servers that honor the
parameter, this
guarantees that the collection's
is not older than the requested
, but does not make any guarantee about the
of any
of the items in that collection.
Exact
Return data at the exact resource version provided. If the provided
is
unavailable, the server responds with HTTP 410 "Gone". For
list
parameter, this guarantees that the collection's
is the same as the
you requested in the query string. That guarantee does
not apply to the
of any items within that collection.
Return data at the resource version of the initial paginated
list
call. The returned
continue
tokens
are responsible for keeping track of the initially provided resource version for all paginated
list
calls after the initial paginated
list
.
Note:
When you
list
resources and receive a collection response, the response includes the
metadata
of the collection as
well as
for each item in that collection. For individual objects found within a collection response,
tracks when that object was last updated, and not how up-to-date
the object is when served.
W

hen using
and limit is set, clients must
handle HTTP 410 "Gone" responses. For example, the client might retry with a
newer
or fall back to
resourceVersion=""
.
When using
and
limit
is unset, clients must
verify that the collection's
matches
the requested
, and handle the case where it does not. For
example, the client might fall back to a request with
limit
set.
watch
For
watch
, the semantics of resource version are:
watch:
resourceVersion for watch
Get State and Start at Most Recent
Get State and Start at Any
Start at Exact
The meaning of those
watch
Caution:
Watches initialized this way may return arbitrarily stale
data. Please review this semantic before using it, and favor the other semantics
where possible.
Start a
watch
at any resource version; the most recent resource version
available is preferred, but not required. Any starting resource version is
allowed. It is possible for the
watch
to start at a much older resource
version that the client has previously observed, particularly in high availability
configurations, due to partitions or stale caches. Clients that cannot tolerate
this apparent rewinding should not start a
watch
with this semantic. To
establish initial state, the
watch
begins with synthetic "Added" events for
all resource instances that exist at the starting resource version. All following
watch events are for all changes that occurred after the resource version the
watch
started at.
Start a
watch
at the most recent resource version, which must be consistent
(in detail: served from etcd via a quorum read). To establish initial state,
the
watch
begins with synthetic "Added" events of all resources instances
that exist at the starting resource version. All following watch events are for
all changes that occurred after the resource version the
watch
Start a
watch
at an exact resource version. The watch events are for all changes
after the provided resource version. Unlike "Get State and Start at Most Recent"
and "Get State and Start at Any", the
watch
is not started with synthetic
"Added" events for the provided resource version. The client is assumed to already
have the initial state at the starting resource version since the client provided
the resource version.
"410 Gone" responses
Servers are not required to serve all older resource versions and may return a HTTP
410 (Gone)
status code if a client requests a
older than the
server has retained. Clients must be able to tolerate
410 (Gone)
responses. See
for details on
how to handle
410 (Gone)
responses when watching resources.
If you request a
outside the applicable limit then, depending
on whether a request is served from cache or not, the API server may reply with a
410 Gone
HTTP response.
Unavailable resource versions
Servers are not required to serve unrecognized resource versions. If you request
list
or
get
for a resource version that the API server does not recognize,
then the API server may either:
wait briefly for the resource version to become available, then timeout with a
504 (Gateway Timeout)
if the provided resource versions does not become available
in a reasonable amount of time;
respond with a
Retry-After
response header indicating how many seconds a client
should wait before retrying the request.
If you request a resource version that an API server does not recognize, the
kube-apiserver additionally identifies its error responses with a "Too large resource
version" message.
If you make a
watch
request for an unrecognized resource version, the API server
may wait indefinitely (until the request timeout) for the resource version to become
available.
2.2 - Server-Si

de Apply
Server-Side Apply helps users and controllers manage their resources through
declarative configurations. Clients can create and modify their
objects
declaratively by sending their fully specified intent.
A fully specified intent is a partial object that only includes the fields and
values for which the user has an opinion. That intent either creates a new
object or is
combined
, by the server, with the existing object.
The system supports multiple appliers collaborating on a single object.
Changes to an object's fields are tracked through a "
field management
"
mechanism. When a field's value changes, ownership moves from its current
manager to the manager making the change. When trying to apply an object,
fields that have a different value and are owned by another manager will
result in a
conflict
. This is done in order to signal that the
operation might undo another collaborator's changes. Conflicts can be forced,
in which case the value will be overridden, and the ownership will be
transferred.
If you remove a field from a configuration and apply the configuration,
Server-Side Apply checks if there are any other field managers that also
own the field. If the field is not owned by any other field managers, it
is either deleted from the live object or reset to its default value, if
it has one. The same rule applies to associative list or map items.
Server-Side Apply is meant both as a replacement for the original
and as a simpler mechanism for controllers to enact their changes.
If you have Server-Side Apply enabled, the control plane tracks managed fields
for all newly created objects.
Field Management
Compared to the
last-applied
annotation managed by
kubectl
, Server-Side
Apply uses a more declarative approach, which tracks a user's field management,
rather than a user's last applied state. This means that as a side effect of
using Server-Side Apply, information about which field manager manages each
field in an object also becomes available.
For a user to manage a field, in the Server-Side Apply sense, means that the
user relies on and expects the value of the field not to change. The user who
last made an assertion about the value of a field will be recorded as the
current field manager. This can be done either by changing the value with
POST
,
PUT
, or non-apply
PATCH
, or by including the field in a config sent
to the Server-Side Apply endpoint. When using Server-Side Apply, trying to
change a field which is managed by someone else will result in a rejected
request (if not forced, see
Conflicts
).
When two or more appliers set a field to the same value, they share ownership of
that field. Any subsequent attempt to change the value of the shared field, by any of
the appliers, results in a conflict. Shared field owners may give up ownership
of a field by removing it from their configuration.
Field management is stored in a
managedFields
field that is part of an object's
metadata
.
A simple example of an object created by Server-Side Apply could look like this:
:
v1
kind
:
ConfigMap
metadata
:
name
:
test-cm
namespace
:
default
labels
:
test-label
:
test
:
-
manager
:
kubectl
operation
:
Apply
:
v1
time
:
"2010-10-10T0:00:00Z"
fieldsType
:
FieldsV1
fieldsV1
:
f:metadata
:
f:labels
:
f:test-label
:
{}
f:data
:
f:key
:
{}
data
:
key
:
some value
The above object contains a single manager in
metadata.managedFields
. The
manager consists of basic information about the managing entity itself, like
operation type, API version, and the fields managed by it.
Note:
This field is managed by the API server and should not be changed by
the user.
Nevertheless it is possible to change
through an
Update
operation. Doing so is highly discouraged, but might be a reasonable
option to try if, for example, the
get into an inconsistent
state (which clearly should not happen).
The format of the

is described in the
API
.
Conflicts
A conflict is a special status error that occurs when an
Apply
operation tries
to change a field, which another user also claims to manage. This prevents an
applier from unintentionally overwriting the value set by another user. When
this occurs, the applier has 3 options to resolve the conflicts:
Overwrite value, become sole manager:
If overwriting the value was
intentional (or if the applier is an automated process like a controller) the
applier should set the
force
query parameter to true (in kubectl, it can be done by
using the
--force-conflicts
flag with the apply command) and make the request
again. This forces the operation to succeed, changes the value of the field,
and removes the field from all other managers' entries in managedFields.
Don't overwrite value, give up management claim:
If the applier doesn't
care about the value of the field anymore, they can remove it from their
config and make the request again. This leaves the value unchanged, and causes
the field to be removed from the applier's entry in managedFields.
Don't overwrite value, become shared manager:
If the applier still cares
about the value of the field, but doesn't want to overwrite it, they can
change the value of the field in their config to match the value of the object
on the server, and make the request again. This leaves the value unchanged,
and causes the field's management to be shared by the applier and all other
field managers that already claimed to manage it.
Managers
Managers identify distinct workflows that are modifying the object (especially
useful on conflicts!), and can be specified through the
query
parameter as part of a modifying request. It is required for the apply endpoint,
though kubectl will default it to
kubectl
. For other updates, its default is
computed from the user-agent.
Apply and Update
The two operation types considered by this feature are
Apply
(
PATCH
with
content type
application/apply-patch+yaml
) and
Update
(all other operations
which modify the object). Both operations update the
, but behave
a little differently.
Note:
Whether you are submitting JSON data or YAML data, use
as the
header value.
All JSON documents are valid YAML.
For instance, only the apply operation fails on conflicts while update does
not. Also, apply operations are required to identify themselves by providing a
query parameter, while the query parameter is optional for update
operations. Finally, when using the apply operation you cannot have
in the object that is being applied.
An example object with multiple managers could look like this:
:
v1
kind
:
ConfigMap
metadata
:
name
:
test-cm
namespace
:
default
labels
:
:
test
:
-
manager
:
kubectl
operation
:
Apply
:
v1
fields
:
:
f:labels
:
:
{}
-
manager
:
operation
:
Update
:
v1
time
:
'2019-03-30T16:00:00.000Z'
fields
:
f:data
:
f:key
:
{}
data
:
key
:
new value
In this example, a second operation was run as an
Update
by the manager called
. The update changed a value in the data field which
caused the field's management to change to the
.
If this update would have been an
Apply
operation, the operation
would have failed due to conflicting ownership.
Merge strategy
The merging strategy, implemented with Server-Side Apply, provides a generally
more stable object lifecycle. Server-Side Apply tries to merge fields based on
the actor who manages them instead of overruling based on values. This way
multiple actors can update the same object without causing unexpected interference.
When a user sends a "fully-specified intent" object to the Server-Side Apply
endpoint, the server merges it with the live object favoring the value in the
applied config if it is specified in both places. If the set of it

ems present in
the applied config is not a superset of the items applied by the same user last
time, each missing item not managed by any other appliers is removed. For
more information about how an object's schema is used to make decisions when
merging, see
sigs.k8s.io/structured-merge-diff
.
A number of markers were added in Kubernetes 1.16 and 1.17, to allow API
developers to describe the merge strategy supported by lists, maps, and
structs. These markers can be applied to objects of the respective type,
in Go files or in the OpenAPI schema definition of the
CRD
:
Golang marker
OpenAPI extension
Accepted values
Introduced in
//+listType
atomic
/
set
/
map
Applicable to lists.
set
applies to lists that include only scalar elements. These elements must be unique.
map
applies to lists of nested types only. The key values (see
listMapKey
) must be unique in the list.
atomic
can apply to any list. If configured as
atomic
, the entire list is replaced during merge. At any point in time, a single manager owns the list. If
set
or
map
, different managers can manage entries separately.
1.16
//+listMapKey
List of field names, e.g.
["port", "protocol"]
Only applicable when
+listType=map
. A list of field names whose values uniquely identify entries in the list. While there can be multiple keys,
is singular because keys need to be specified individually in the Go type. The key fields must be scalars.
1.16
//+mapType
atomic
/
granular
Applicable to maps.
atomic
means that the map can only be entirely replaced by a single manager.
granular
means that the map supports separate managers updating individual fields.
1.17
//+structType
atomic
/
granular
Applicable to structs; otherwise same usage and OpenAPI annotation as
.
1.17
If
listType
is missing, the API server interprets a
patchMergeStrategy=merge
marker as a
listType=map
and the
corresponding
patchMergeKey
marker as a
.
The
atomic
list type is recursive.
These markers are specified as comments and don't have to be repeated as
field tags.
Compatibility across topology changes
On rare occurrences, a CRD or built-in type author may want to change the
specific topology of a field in their resource without incrementing its
version. Changing the topology of types, by upgrading the cluster or
updating the CRD, has different consequences when updating existing
objects. There are two categories of changes: when a field goes from
map
/
set
/
granular
to
atomic
and the other way around.
When the
listType
,
mapType
, or
structType
changes from
map
/
set
/
granular
to
atomic
, the whole list, map, or struct of
existing objects will end-up being owned by actors who owned an element
of these types. This means that any further change to these objects
would cause a conflict.
When a list, map, or struct changes from
atomic
to
map
/
set
/
granular
, the API server won't be able to infer the new
ownership of these fields. Because of that, no conflict will be produced
when objects have these fields updated. For that reason, it is not
recommended to change a type from
atomic
to
map
/
set
/
granular
.
Take for example, the custom resource:
:
kind
:
Foo
metadata
:
name
:
foo-sample
:
-
manager
:
manager-one
operation
:
Apply
:
fields
:
f:spec
:
f:data
:
{}
spec
:
data
:
key1
:
val1
key2
:
val2
Before
spec.data
gets changed from
atomic
to
granular
,
owns the field
spec.data
, and all the fields within it
(
key1
and
key2
). When the CRD gets changed to make
spec.data
granular
,
continues to own the top-level field
spec.data
(meaning no other managers can delete the map called
data
without a conflict), but it no longer owns
key1
and
key2
, so another
manager can then modify or delete those fields without conflict.
By default, Server-Side Apply treats custom resources as

 unstructured data. All
keys are treated the same as struct fields, and all lists are considered atomic.
If the Custom Resource Definition defines a
schema
that contains annotations as defined in the previous "Merge Strategy"
section, these annotations will be used when merging objects of this
type.
Using Server-Side Apply in a controller
As a developer of a controller, you can use server-side apply as a way to
simplify the update logic of your controller. The main differences with a
read-modify-write and/or patch are the following:
the applied object must contain all the fields that the controller cares about.
there is no way to remove fields that haven't been applied by the controller
before (controller can still send a PATCH/UPDATE for these use-cases).
the object doesn't have to be read beforehand,
doesn't have
to be specified.
It is strongly recommended for controllers to always "force" conflicts, since they
might not be able to resolve or act on these conflicts.
Transferring Ownership
In addition to the concurrency controls provided by
conflict resolution
,
Server-Side Apply provides ways to perform coordinated
field ownership transfers from users to controllers.
This is best explained by example. Let's look at how to safely transfer
ownership of the
replicas
field from a user to a controller while enabling
automatic horizontal scaling for a Deployment, using the HorizontalPodAutoscaler
resource and its accompanying controller.
Say a user has defined deployment with
replicas
set to the desired value:
application/ssa/nginx-deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
nginx
spec
:
replicas
:
3
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
And the user has created the deployment using Server-Side Apply like so:
kubectl apply -f https://k8s.io/examples/application/ssa/nginx-deployment.yaml --server-side
Then later, HPA is enabled for the deployment, e.g.:
kubectl autoscale deployment nginx-deployment --cpu-percent
=
50
--min
=
1
--max
=
10
Now, the user would like to remove
replicas
from their configuration, so they
don't accidentally fight with the HPA controller. However, there is a race: it
might take some time before HPA feels the need to adjust
replicas
, and if
the user removes
replicas
before the HPA writes to the field and becomes
its owner, then apiserver will set
replicas
to 1, its default value. This
is not what the user wants to happen, even temporarily.
There are two solutions:
(basic) Leave
replicas
in the configuration; when HPA eventually writes to that
field, the system gives the user a conflict over it. At that point, it is safe
to remove from the configuration.
(more advanced) If, however, the user doesn't want to wait, for example
because they want to keep the cluster legible to coworkers, then they can take
the following steps to make it safe to remove
replicas
from their
configuration:
First, the user defines a new configuration containing only the
replicas
field:
application/ssa/nginx-deployment-replicas-only.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
3
The user applies that configuration using the field manager name
handover-to-hpa
:
kubectl apply -f https://k8s.io/examples/application/ssa/nginx-deployment-replicas-only.yaml
\
--server-side --field-manager
=
\
=
false
If the apply results in a conflict with the HPA controller, then do nothing. The
conflict indicates the controller has claimed the field earlier in the
process than it sometimes does.
At this point the user may remove the
replicas
field from their configuration.
application/ssa/nginx-deployment-no-replicas.yaml
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
nginx
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec

:
:
-
name
:
nginx
image
:
Note that whenever the HPA controller sets the
replicas
field to a new value,
the temporary field manager will no longer own any fields and will be
automatically deleted. No clean up is required.
Transferring Ownership Between Users
Users can transfer ownership of a field between each other by setting the field
to the same value in both of their applied configs, causing them to share
ownership of the field. Once the users share ownership of the field, one of them
can remove the field from their applied configuration to give up ownership and
complete the transfer to the other user.
Comparison with Client Side Apply
A consequence of the conflict detection and resolution implemented by Server-Side
Apply is that an applier always has up to date field values in their local
state. If they don't, they get a conflict the next time they apply. Any of the
three options to resolve conflicts results in the applied configuration being an
up to date subset of the object on the server's fields.
This is different from Client Side Apply, where outdated values which have been
overwritten by other users are left in an applier's local config. These values
only become accurate when the user updates that specific field, if ever, and an
applier has no way of knowing whether their next apply will overwrite other
users' changes.
Another difference is that an applier using Client Side Apply is unable to
change the API version they are using, but Server-Side Apply supports this use
case.
Upgrading from client-side apply to server-side apply
Client-side apply users who manage a resource with
can start
using server-side apply with the following flag.
kubectl apply --server-side
[
--dry-run
=
server
]
By default, field management of the object transfers from client-side apply to
kubectl server-side apply without encountering conflicts.
Caution:
Keep the
last-applied-configuration
annotation up to date.
The annotation infers client-side apply's managed fields.
Any fields not managed by client-side apply raise conflicts.
For example, if you used
to update the replicas field after
client-side apply, then this field is not owned by client-side apply and
creates conflicts on
.
This behavior applies to server-side apply with the
kubectl
field manager.
As an exception, you can opt-out of this behavior by specifying a different,
non-default field manager, as seen in the following example. The default field
manager for kubectl server-side apply is
kubectl
.
kubectl apply --server-side --field-manager
=
my-manager
[
--dry-run
=
server
]
Downgrading from server-side apply to client-side apply
If you manage a resource with
,
you can downgrade to client-side apply directly with
.
Downgrading works because kubectl server-side apply keeps the
annotation up-to-date if you use
.
kubectl
kubectl
.
=
[
--dry-run
=
server
]
API Endpoint
With the Server-Side Apply feature enabled, the
PATCH
endpoint accepts the
additional
content type. Users of Server-Side
Apply can send partially specified objects as YAML to this endpoint. When
applying a configuration, one should always include all the fields that they
have an opinion about.
RBAC and permissions
Since Server-Side Apply is a type of
PATCH
, a role will require the
PATCH
permission to edit resources, but will also need the
CREATE
verb permission in order to create resources with Server-Side Apply.
Clearing ManagedFields
It is possible to strip all managedFields from an object by overwriting them
usi

ng
MergePatch
,
StrategicMergePatch
,
JSONPatch
, or
Update
, so every
non-apply operation. This can be done by overwriting the managedFields field
with an empty entry. Two examples are:
PATCH /api/v1/namespaces/default/configmaps/example-cm
Content-Type: application/merge-patch+json
Data: {"metadata":{"managedFields": [{}]}}
Data: [{"op": "replace", "path": "/metadata/managedFields", "value": [{}]}]
This will overwrite the managedFields with a list containing a single empty
entry that then results in the managedFields being stripped entirely from the
object. Note that setting the managedFields to an empty list will not
reset the field. This is on purpose, so managedFields never get stripped by
clients not aware of the field.
In cases where the reset operation is combined with changes to other fields
than the managedFields, this will result in the managedFields being reset
first and the other changes being processed afterwards. As a result the
applier takes ownership of any fields updated in the same request.
2.3 - Client Libraries
,
.
Language
C
browse
dotnet
browse
Go
browse
Haskell
browse
Java
browse
browse
Perl
browse
Python
browse
Ruby
browse
Note:
Language
Clojure
DotNet
Elixir
Elixir
Go
Java
Lisp
Lisp
Node.js
Node.js
Node.js
Perl
PHP
PHP
PHP
github.com/travisghansen/kubernetes-client-php
PHP
Python
Python
Python
Python
Python
Ruby
github.com/abonas/kube

client
Ruby
Ruby
Rust
Rust
Scala
Scala
Scala
Swift
2.4 - Kubernetes Deprecation Policy
This document details the deprecation policy for various facets of the system.
Kubernetes is a large system with many components and many contributors. As
with any such software, the feature set naturally evolves over time, and
sometimes a feature may need to be removed. This could include an API, a flag,
or even an entire feature. To avoid breaking existing users, Kubernetes follows
a deprecation policy for aspects of the system that are slated to be removed.
Deprecating parts of the API
Since Kubernetes is an API-driven system, the API has evolved over time to
reflect the evolving understanding of the problem space. The Kubernetes API is
actually a set of APIs, called "API groups", and each API group is
independently versioned.
API versions
fall
into 3 main tracks, each of which has different policies for deprecation:
Example
Track
v1
GA (generally available, stable)
v1beta1
Beta (pre-release)
v1alpha1
Alpha (experimental)
A given release of Kubernetes can support any number of API groups and any
number of versions of each.
The following rules govern the deprecation of elements of the API. This
includes:
REST resources (aka API objects)
Fields of REST resources
Annotations on REST resources, including "beta" annotations but not
including "alpha" annotations.
Enumerated or constant values
Component config structures
These rules are enforced between official releases, not between
arbitrary commits to master or release branches.
Rule #1: API elements may only be removed by incrementing the version of the
API group.
Once an API element has been added to an API group at a particular version, it
can not be removed from that version or have its behavior significantly
changed, regardless of track.
Note:
For historical reasons, there are 2 "monolithic" API groups - "core" (no
group name) and "extensions". Resources will incrementally be moved from these
legacy API groups into more domain-specific API groups.
Rule #2: API objects must be able to round-trip between API versions in a given
release without information loss, with the exception of whole REST resources
that do not exist in some versions.
For example, an object can be written as v1 and then read back as v2 and
converted to v1, and the resulting v1 resource will be identical to the
original. The representation in v2 might be different from v1, but the system
knows how to convert between them in both directions. Additionally, any new
field added in v2 must be able to round-trip to v1 and back, which means v1
might have to add an equivalent field or represent it as an annotation.
Rule #3: An API version in a given track may not be deprecated in favor of a less stable API version.
GA API versions can replace beta and alpha API versions.
Beta API versions can replace earlier beta and alpha API versions, but
may not
replace GA API versions.
Alpha API versions can replace earlier alpha API versions, but
may not
replace GA or beta API versions.
Rule #4a: API lifetime is determined by the API stability level
GA API versions may be marked as deprecated, but must not be removed within a major version of Kubernetes
Beta API versions are deprecated no more than 9 months or 3 minor releases after introduction (whichever is longer),
and are no longer served 9 months or 3 minor releases after deprecation (whichever is longer)
Alpha API versions may be removed in any release without prior deprecation notice
This ensures beta API support covers the
maximum supported version skew of 2 releases
,
and that APIs don't stagnate on unstable beta versions, accumulating production usage that will be disrupted when support for the beta API ends.
Note:
There are no current plans for a maj

or version revision of Kubernetes that removes GA APIs.
Note:
Until
#52185
is
resolved, no API versions that have been persisted to storage may be removed.
Serving REST endpoints for those versions may be disabled (subject to the
deprecation timelines in this document), but the API server must remain capable
of decoding/converting previously persisted data from storage.
Rule #4b: The "preferred" API version and the "storage version" for a given
group may not advance until after a release has been made that supports both the
new version and the previous version
Users must be able to upgrade to a new release of Kubernetes and then roll back
to a previous release, without converting anything to the new API version or
suffering breakages (unless they explicitly used features only available in the
newer version). This is particularly evident in the stored representation of
objects.
All of this is best illustrated by examples. Imagine a Kubernetes release,
version X, which introduces a new API group. A new Kubernetes release is made
every approximately 4 months (3 per year). The following table describes which
API versions are supported in a series of subsequent releases.
Release
API Versions
Preferred/Storage Version
Notes
X
v1alpha1
v1alpha1
X+1
v1alpha2
v1alpha2
v1alpha1 is removed, "action required" relnote
X+2
v1beta1
v1beta1
v1alpha2 is removed, "action required" relnote
X+3
v1beta2, v1beta1 (deprecated)
v1beta1
v1beta1 is deprecated, "action required" relnote
X+4
v1beta2
X+5
v1, v1beta1 (deprecated), v1beta2 (deprecated)
v1beta2
v1beta2 is deprecated, "action required" relnote
X+6
v1, v1beta2 (deprecated)
v1
v1beta1 is removed, "action required" relnote
X+7
v1
X+8
v2alpha1, v1
v1
v1beta2 is removed, "action required" relnote
X+9
v2alpha2, v1
v1
v2alpha1 is removed, "action required" relnote
X+10
v2beta1, v1
v1
v2alpha2 is removed, "action required" relnote
X+11
v2beta2, v2beta1 (deprecated), v1
v1
v2beta1 is deprecated, "action required" relnote
X+12
v2, v2beta2 (deprecated), v2beta1 (deprecated), v1 (deprecated)
v1
v2beta2 is deprecated, "action required" relnote
v1 is deprecated in favor of v2, but will not be removed
X+13
v2, v2beta1 (deprecated), v2beta2 (deprecated), v1 (deprecated)
v2
X+14
v2, v2beta2 (deprecated), v1 (deprecated)
v2
v2beta1 is removed, "action required" relnote
X+15
v2, v1 (deprecated)
v2
v2beta2 is removed, "action required" relnote
Consider a hypothetical REST resource named Widget, which was present in API v1
in the above timeline, and which needs to be deprecated. We document and
announce
the
deprecation in sync with release X+1. The Widget resource still exists in API
version v1 (deprecated) but not in v2alpha1. The Widget resource continues to
exist and function in releases up to and including X+8. Only in release X+9,
when API v1 has aged out, does the Widget resource cease to exist, and the
behavior get removed.
Starting in Kubernetes v1.19, making an API request to a deprecated REST API endpoint:
Returns a
Warning
header (as defined in
RFC7234, Section 5.5
) in the API response.
Adds a
"k8s.io/deprecated":"true"
annotation to the
audit event
recorded for the request.
Sets an
apiserver_requested_deprecated_apis
gauge metric to
1
in the
process. The metric has labels for
group
,
version
,
resource
,
that can be joined
to the
apiserver_request_total
metric, and a
removed_release
label that indicates the
Kubernetes release in which the API will no longer be served. The following Prometheus query
returns information about requests made to deprecated APIs which will be removed in v1.22:
{
=
"
1.22
"}
*
on
(
group
,
version
,
resource
,
)
group_right
()
As with whole REST resources, an individual field which was present in API v1
must exist and function until API v1 is removed

. Unlike whole resources, the
v2 APIs may choose a different representation for the field, as long as it can
be round-tripped. For example a v1 field named "magnitude" which was
deprecated might be named "deprecatedMagnitude" in API v2. When v1 is
eventually removed, the deprecated field can be removed from v2.
As with whole REST resources and fields thereof, a constant value which was
supported in API v1 must exist and function until API v1 is removed.
Component configs are versioned and managed similar to REST resources.
Future work
Over time, Kubernetes will introduce more fine-grained API versions, at which
point these rules will be adjusted as needed.
Deprecating a flag or CLI
The Kubernetes system is comprised of several different programs cooperating.
Sometimes, a Kubernetes release might remove flags or CLI commands
(collectively "CLI elements") in these programs. The individual programs
naturally sort into two main groups - user-facing and admin-facing programs,
which vary slightly in their deprecation policies. Unless a flag is explicitly
prefixed or documented as "alpha" or "beta", it is considered GA.
CLI elements are effectively part of the API to the system, but since they are
not versioned in the same way as the REST API, the rules for deprecation are as
follows:
Rule #5a: CLI elements of user-facing components (e.g. kubectl) must function
after their announced deprecation for no less than:
GA: 12 months or 2 releases (whichever is longer)
Beta: 3 months or 1 release (whichever is longer)
Alpha: 0 releases
Rule #5b: CLI elements of admin-facing components (e.g. kubelet) must function
GA: 6 months or 1 release (whichever is longer)
Rule #6: Deprecated CLI elements must emit warnings (optionally disable)
when used.
Deprecating a feature or behavior
Occasionally a Kubernetes release needs to deprecate some feature or behavior
of the system that is not controlled by the API or CLI. In this case, the
rules for deprecation are as follows:
Rule #7: Deprecated behaviors must function for no less than 1 year after their
announced deprecation.
This does not imply that all changes to the system are governed by this policy.
This applies only to significant, user-visible behaviors which impact the
correctness of applications running on Kubernetes or that impact the
administration of Kubernetes clusters, and which are being removed entirely.
An exception to the above rule is
feature gates
. Feature gates are key=value
pairs that allow for users to enable/disable experimental features.
Feature gates are intended to cover the development life cycle of a feature - they
are not intended to be long-term APIs. As such, they are expected to be deprecated
and removed after a feature becomes GA or is dropped.
As a feature moves through the stages, the associated feature gate evolves.
The feature life cycle matched to its corresponding feature gate is:
Alpha: the feature gate is disabled by default and can be enabled by the user.
Beta: the feature gate is enabled by default and can be disabled by the user.
GA: the feature gate is deprecated (see
"Deprecation"
) and becomes
non-operational.
GA, deprecation window complete: the feature gate is removed and calls to it are
no longer accepted.
Deprecation
Features can be removed at any point in the life cycle prior to GA. When features are
removed prior to GA, their associated feature gates are also deprecated.
When an invocation tries to disable a non-operational feature gate, the call fails in order
to avoid unsupported scenarios that might otherwise run silently.
In some cases, removing pre-GA features requires considerable time. Feature gates can remain
operational until their associated feature is fully removed, at which point the feature gate
itself can be deprecated.
When removing a feature gate for a GA feat

ure also requires considerable time, calls to
feature gates may remain operational if the feature gate has no effect on the feature,
and if the feature gate causes no errors.
Features intended to be disabled by users should include a mechanism for disabling the
feature in the associated feature gate.
Versioning for feature gates is different from the previously discussed components,
therefore the rules for deprecation are as follows:
Rule #8: Feature gates must be deprecated when the corresponding feature they control
transitions a lifecycle stage as follows. Feature gates must function for no less than:
Beta feature to GA: 6 months or 2 releases (whichever is longer)
Beta feature to EOL: 3 months or 1 release (whichever is longer)
Alpha feature to EOL: 0 releases
Rule #9: Deprecated feature gates must respond with a warning when used. When a feature gate
is deprecated it must be documented in both in the release notes and the corresponding CLI help.
Both warnings and documentation must indicate whether a feature gate is non-operational.
Deprecating a metric
Each component of the Kubernetes control-plane exposes metrics (usually the
/metrics
endpoint), which are typically ingested by cluster administrators.
Not all metrics are the same: some metrics are commonly used as SLIs or used
to determine SLOs, these tend to have greater import. Other metrics are more
experimental in nature or are used primarily in the Kubernetes development
process.
Accordingly, metrics fall under two stability classes (
ALPHA
and
STABLE
);
this impacts removal of a metric during a Kubernetes release. These classes
are determined by the perceived importance of the metric. The rules for
deprecating and removing a metric are as follows:
Rule #9a: Metrics, for the corresponding stability class, must function for no less than:
STABLE: 4 releases or 12 months (whichever is longer)
ALPHA: 0 releases
Rule #9b: Metrics, after their
announced deprecation
, must function for no less than:
STABLE: 3 releases or 9 months (whichever is longer)
Deprecated metrics will have their description text prefixed with a deprecation notice
string '(Deprecated from x.y)' and a warning log will be emitted during metric
registration. Like their stable undeprecated counterparts, deprecated metrics will
be automatically registered to the metrics endpoint and therefore visible.
On a subsequent release (when the metric's
deprecatedVersion
is equal to
current_kubernetes_version - 3
)), a deprecated metric will become a
hidden
metric.
Unlike
their deprecated counterparts, hidden metrics will
no longer
be
automatically registered to the metrics endpoint (hence hidden). However, they
can be explicitly enabled through a command line flag on the binary
(
--show-hidden-metrics-for-version=
). This provides cluster admins an
escape hatch to properly migrate off of a deprecated metric, if they were not
able to react to the earlier deprecation warnings. Hidden metrics should be
deleted after one release.
Exceptions
No policy can cover every possible situation. This policy is a living
document, and will evolve over time. In practice, there will be situations
that do not fit neatly into this policy, or for which this policy becomes a
serious impediment. Such situations should be discussed with SIGs and project
leaders to find the best solutions for those specific cases, always bearing in
mind that Kubernetes is committed to being a stable system that, as much as
possible, never breaks users. Exceptions will always be announced in all
relevant release notes.
2.5 - Deprecated API Migration Guide
As the Kubernetes API evolves, APIs are periodically reorganized or upgraded.
When APIs evolve, the old API is deprecated and eventually removed.
This page contains information you need to know when migrating from
deprecated API versions to newer and more stable API versions.
Removed APIs by release
v1.29
The
v1.29
release will stop serving the following deprecated API versions:
Flow con

trol resources
The
API version of FlowSchema and PriorityLevelConfiguration will no longer be served in v1.29.
Migrate manifests and API clients to use the
flowcontrol.apiserver.k8s.io/v1beta3
API version, available since v1.26.
All existing persisted objects are accessible via the new API
Notable changes in
:
The PriorityLevelConfiguration
spec.limited.assuredConcurrencyShares
field is renamed to
spec.limited.nominalConcurrencyShares
v1.27
The
v1.27
The
storage.k8s.io/v1beta1
API version of CSIStorageCapacity will no longer be served in v1.27.
API version, available since v1.24.
No notable changes
v1.26
The
v1.26
release stopped serving the following deprecated API versions:
Flow control resources
The
flowcontrol.apiserver.k8s.io/v1beta1
API version of FlowSchema and PriorityLevelConfiguration is no longer served as of v1.26.
The
autoscaling/v2beta2
API version of HorizontalPodAutoscaler is no longer served as of v1.26.
API version, available since v1.23.
v1.25
The
v1.25
CronJob
The
batch/v1beta1
API version of CronJob is no longer served as of v1.25.
batch/v1
API version, available since v1.21.
The
discovery.k8s.io/v1beta1
API version of EndpointSlice is no longer served as of v1.25.
:
use per Endpoint
nodeName
field instead of deprecated
topology["kubernetes.io/hostname"]
field
zone
topology["topology.kubernetes.io/zone"]
field
topology
is replaced with the
deprecatedTopology
field which is not writable in v1
Event
The
events.k8s.io/v1beta1
API version of Event is no longer served as of v1.25.
API version, available since v1.19.
:
type
is limited to
Normal
and
Warning
is renamed to
regarding
action
,
reason
,
reportingController
, and
reportingInstance
are required when creating new
Events
use
eventTime
instead of the deprecated
firstTimestamp
field (which is renamed to
deprecatedFirstTimestamp
and not permitted in new
Events)
use
series.lastObservedTime
lastTimestamp
deprecatedLastTimestamp
Events)
use
series.count
count
deprecatedCount
Events)
use
source.component
deprecatedSource.component
Events)
use
source.host
deprecatedSource.host
Events)
The
autoscaling/v2beta1
API version of HorizontalPodAutoscaler is no longer served as of v1.25.
API version, available since v

1.23.
The
API version of PodDisruptionBudget is no longer served as of v1.25.
policy/v1
policy/v1
:
an empty
(
{}
) written to a
policy/v1
PodDisruptionBudget selects all pods in the namespace (in
an empty
selected no pods). An unset
selects no pods in either API version.
PodSecurityPolicy in the
API version is no longer served as of v1.25, and the PodSecurityPolicy admission controller will be removed.
Migrate to
or a
3rd party admission webhook
.
For a migration guide, see
.
For more information on the deprecation, see
PodSecurityPolicy Deprecation: Past, Present, and Future
.
RuntimeClass in the
node.k8s.io/v1beta1
API version is no longer served as of v1.25.
API version, available since v1.20.
v1.22
The
v1.22
Webhook resources
The
admissionregistration.k8s.io/v1beta1
API version of MutatingWebhookConfiguration and ValidatingWebhookConfiguration is no longer served as of v1.22.
API version, available since v1.16.
All existing persisted objects are accessible via the new APIs
Notable changes:
webhooks[*].failurePolicy
default changed from
Ignore
to
Fail
for v1
webhooks[*].matchPolicy
Exact
to
Equivalent
for v1
webhooks[*].timeoutSeconds
30s
to
10s
for v1
webhooks[*].sideEffects
default value is removed, and the field made required, and only
None
and
NoneOnDryRun
are permitted for v1
webhooks[*].admissionReviewVersions
default value is removed and the field made required for v1 (supported versions for AdmissionReview are
v1
and
v1beta1
)
webhooks[*].name
must be unique in the list for objects created via
The
API version of CustomResourceDefinition is no longer served as of v1.22.
spec.scope
is no longer defaulted to
and must be explicitly specified
spec.version
is removed in v1; use
instead
spec.validation
spec.versions[*].schema
instead
spec.subresources
spec.versions[*].subresources
instead
spec.additionalPrinterColumns
spec.versions[*].additionalPrinterColumns
instead
is moved to
spec.conversion.webhook.clientConfig
in v1
spec.conversion.conversionReviewVersions
is moved to
spec.conversion.webhook.conversionReviewVersions
in v1
is now required when creating v1 CustomResourceDefinition objects, and must be a
structural schema
spec.preserveUnknownFields: true
is disallowed when creating v1 CustomResourceDefinition objects; it must be specified within schema definitions as
x-kubernetes-preserve-unknown-fields: true
In
additionalPrinterColumns
items, the
JSONPath
field was renamed to
jsonPath
in v1 (fixes
#66531
)
The
apiregistration.k8s.io/v1beta1
API version of APIService is no longer served as of v1.22.
API version, available since v1.10.
The
authenti

cation.k8s.io/v1beta1
API version of TokenReview is no longer served as of v1.22.
API version, available since v1.6.
SubjectAccessReview resources
The
API version of LocalSubjectAccessReview, SelfSubjectAccessReview, SubjectAccessReview, and SelfSubjectRulesReview is no longer served as of v1.22.
spec.group
was renamed to
spec.groups
#32709
)
The
certificates.k8s.io/v1beta1
API version of CertificateSigningRequest is no longer served as of v1.22.
:
For API clients requesting certificates:
is now required (see
known Kubernetes signers
), and requests for
kubernetes.io/legacy-unknown
are not allowed to be created via the
API
spec.usages
is now required, may not contain duplicate values, and must only contain known usages
For API clients approving or signing certificates:
status.conditions
may not contain duplicate types
status.conditions[*].status
is now required
must be PEM-encoded, and contain only
CERTIFICATE
blocks
Lease
The
coordination.k8s.io/v1beta1
API version of Lease is no longer served as of v1.22.
API version, available since v1.14.
Ingress
The
extensions/v1beta1
and
networking.k8s.io/v1beta1
API versions of Ingress is no longer served as of v1.22.
spec.backend
spec.defaultBackend
The backend
Numeric backend
servicePort
fields are renamed to
service.port.number
String backend
service.port.name
pathType
is now required for each specified path. Options are
Prefix
,
Exact
, and
ImplementationSpecific
. To match the undefined
v1beta1
behavior, use
.
The
API version of IngressClass is no longer served as of v1.22.
RBAC resources
The
rbac.authorization.k8s.io/v1beta1
API version of ClusterRole, ClusterRoleBinding, Role, and RoleBinding is no longer served as of v1.22.
API version, available since v1.8.
The
scheduling.k8s.io/v1beta1
API version of PriorityClass is no longer served as of v1.22.
Storage resources
The
API version of CSIDriver, CSINode, StorageClass, and VolumeAttachment is no longer served as of v1.22.
API version
CSIDriver is available in
since v1.19.
CSINode is available in
since v1.17
StorageClass is available in
since v1.6
VolumeAttachment is available in
v1.13
v1.16
The
v1.16
release stopped serving the following deprecated API version

s:
The
API version of NetworkPolicy is no longer served as of v1.16.
DaemonSet
The
and
apps/v1beta2
API versions of DaemonSet are no longer served as of v1.16.
apps/v1
API version, available since v1.9.
spec.templateGeneration
is removed
is now required and immutable after creation; use the existing template labels as the selector for seamless upgrades
spec.updateStrategy.type
now defaults to
RollingUpdate
(the default in
was
OnDelete
)
The
,
apps/v1beta1
, and
API versions of Deployment are no longer served as of v1.16.
apps/v1
spec.rollbackTo
is removed
spec.progressDeadlineSeconds
600
seconds (the default in
was no deadline)
spec.revisionHistoryLimit
10
was
2
, the default in
was to retain all)
maxSurge
and
maxUnavailable
now default to
25%
was
1
)
The
and
API versions of StatefulSet are no longer served as of v1.16.
apps/v1
was
OnDelete
)
The
,
, and
API versions of ReplicaSet are no longer served as of v1.16.
apps/v1
The
API version of PodSecurityPolicy is no longer served as of v1.16.
Migrate manifests and API client to use the
API version of PodSecurityPolicy will be removed in v1.25.
What to do
Test with deprecated APIs disabled
You can test your clusters by starting an API server with specific API versions disabled
to simulate upcoming removals. Add the following flag to the API server startup arguments:
--runtime-config=&lt;group&gt;/&lt;version&gt;=false
--runtime-config=admissionregistration.k8s.io/v1beta1=false,apiextensions.k8s.io/v1beta1,...
Locate use of deprecated APIs
Use
client warnings, metrics, and audit information available in 1.19+
to locate use of deprecated APIs.
Migrate to non-deprecated APIs
Update custom integrations and controllers to call the non-deprecated APIs
Change YAML files to reference the non-deprecated APIs
command (
prior to v1.20)
to automatically convert an existing object:
kubectl-convert -f &lt;file&gt; --output-version &lt;group&gt;/&lt;version&gt;
.
For example, to convert an older Deployment to
apps/v1
, you can run:
kubectl-convert -f ./my-deployment.yaml --output-version apps/v1
Note that this may use non-ideal default values. To learn more about a specific
resource, check the Kubernetes
.
2.6 - Kubernetes API health endpoi

nts
API server
healthz
,
livez
and
readyz
The
healthz
livez
and
readyz
The
livez
flag
flag
with the
/readyz
endpoint.
healthz
/
livez
/
readyz
200
healthy
/
live
/
ready
verbose
=
[+]ping ok
[+]log ok
[+]etcd ok
curl -k
'https://localhost:6443/readyz?verbose&amp;exclude=etcd'
etcd
[+]ping ok
[+]log ok
where
livez
and
readyz
The
verbose
[+]
and
ok
.
These individual health checks should not be consumed by machines but can be helpful for a human operator to debug

 a system:
3 - API Access Control
For an introduction to how Kubernetes implements and controls API access,
read
.
Reference documentation:
Role Based Access Control
Attribute Based Access Control
Webhook Authorization
including
CSR approval
and
certificate signing
Service accounts
Developer guide
Kubelet Authentication &amp; Authorization
including kubelet
3.1 - Authenticating
This page provides an overview of authenticating.
Users in Kubernetes
All Kubernetes clusters have two categories of users: service accounts managed
by Kubernetes, and normal users.
It is assumed that a cluster-independent service manages normal users in the following ways:
an administrator distributing private keys
a user store like Keystone or Google Accounts
a file with a list of usernames and passwords
In this regard,
Kubernetes does not have objects which represent normal user
accounts.
Normal users cannot be added to a cluster through an API call.
Even though a normal user cannot be added via an API call, any user that
presents a valid certificate signed by the cluster's certificate authority
(CA) is considered authenticated. In this configuration, Kubernetes determines
the username from the common name field in the 'subject' of the cert (e.g.,
"/CN=bob"). From there, the role based access control (RBAC) sub-system would
determine whether the user is authorized to perform a specific operation on a
resource. For more details, refer to the normal users topic in
certificate request
for more details about this.
In contrast, service accounts are users managed by the Kubernetes API. They are
bound to specific namespaces, and created automatically by the API server or
manually through API calls. Service accounts are tied to a set of credentials
stored as
Secrets
, which are mounted into pods allowing in-cluster processes
to talk to the Kubernetes API.
API requests are tied to either a normal user or a service account, or are treated
as
anonymous requests
. This means every process inside or outside the cluster, from
a human user typing
kubectl
on a workstation, to
kubelets
on nodes, to members
of the control plane, must authenticate when making requests to the API server,
or be treated as an anonymous user.
Authentication strategies
Kubernetes uses client certificates, bearer tokens, or an authenticating proxy to
authenticate API requests through authentication plugins. As HTTP requests are
made to the API server, plugins attempt to associate the following attributes
with the request:
Username: a string which identifies the end user. Common values might be
kube-admin
or
jane@example.com
.
UID: a string which identifies the end user and attempts to be more consistent and unique than username.
Groups: a set of strings, each of which indicates the user's membership in a named logical collection of users. Common values might be
or
devops-team
.
Extra fields: a map of strings to list of strings which holds additional information authorizers may find useful.
All values are opaque to the authentication system and only hold significance
when interpreted by an
authorizer
.
You can enable multiple authentication methods at once. You should usually use at least two methods:
service account tokens for service accounts
at least one other method for user authentication.
When multiple authenticator modules are enabled, the first module
to successfully authenticate the request short-circuits evaluation.
The API server does not guarantee the order authenticators run in.
The
group is included in the list of groups for all authenticated users.
Integrations with other authentication protocols (LDAP, SAML, Kerberos, alternate x509 schemes, etc)
can be accomplished us

ing an
authenticating proxy
or the
authentication webhook
.
X509 Client Certs
Client certificate authentication is enabled by passing the
--client-ca-file=SOMEFILE
option to API server. The referenced file must contain one or more certificate authorities
to use to validate client certificates presented to the API server. If a client certificate
is presented and verified, the common name of the subject is used as the user name for the
request. As of Kubernetes 1.4, client certificates can also indicate a user's group memberships
using the certificate's organization fields. To include multiple group memberships for a user,
include multiple organization fields in the certificate.
For example, using the
openssl
command line tool to generate a certificate signing request:
openssl req -new -key jbeda.pem -out jbeda-csr.pem -subj
"/CN=jbeda/O=app1/O=app2"
This would create a CSR for the username "jbeda", belonging to two groups, "app1" and "app2".
See
Managing Certificates
for how to generate a client cert.
Static Token File
The API server reads bearer tokens from a file when given the
--token-auth-file=SOMEFILE
option on the command line. Currently, tokens last indefinitely, and the token list cannot be
changed without restarting the API server.
The token file is a csv file with a minimum of 3 columns: token, user name, user uid,
followed by optional group names.
Note:
If you have more than one group the column must be double quoted e.g.
token,user,uid,"group1,group2,group3"
Putting a Bearer Token in a Request
When using bearer token authentication from an http client, the API
server expects an
header with a value of
Bearer &lt;token&gt;
. The bearer token must be a character sequence that can be
put in an HTTP header value using no more than the encoding and
quoting facilities of HTTP. For example: if the bearer token is
31ada4fd-adec-460c-809a-9e56ceb75269
then it would appear in an HTTP
header as shown below.
Authorization: Bearer 31ada4fd-adec-460c-809a-9e56ceb75269
To allow for streamlined bootstrapping for new clusters, Kubernetes includes a
dynamically-managed Bearer token type called a
Bootstrap Token
. These tokens
are stored as Secrets in the
namespace, where they can be
dynamically managed and created. Controller Manager contains a TokenCleaner
controller that deletes bootstrap tokens as they expire.
The tokens are of the form
[a-z0-9]{6}.[a-z0-9]{16}
. The first component is a
Token ID and the second component is the Token Secret. You specify the token
in an HTTP header as follows:
Authorization: Bearer 781292.db7bc3a58fc5f07e
You must enable the Bootstrap Token Authenticator with the
flag on the API Server. You must enable
the TokenCleaner controller via the
flag on the Controller
Manager. This is done with something like
--controllers=*,tokencleaner
.
kubeadm
will do this for you if you are using it to bootstrap a cluster.
The authenticator authenticates as
system:bootstrap:&lt;Token ID&gt;
. It is
included in the
system:bootstrappers
group. The naming and groups are
intentionally limited to discourage users from using these tokens past
bootstrapping. The user names and group can be used (and are used by
kubeadm
)
to craft the appropriate authorization policies to support bootstrapping a
cluster.
Please see
for in depth
documentation on the Bootstrap Token authenticator and controllers along with
how to manage these tokens with
kubeadm
.
Service Account Tokens
A service account is an automatically enabled authenticator that uses signed
bearer tokens to verify requests. The plugin takes two optional flags:
File containing PEM-encoded x509 RSA or ECDSA
private or public keys, used to verify ServiceAccount tokens. The specified file
can contain multiple keys, and the flag can be specified multiple times with
different files. If unspecified, --tls-privat

e-key-file is used.
--service-account-lookup
If enabled, tokens which are deleted from the API will be revoked.
Service accounts are usually created automatically by the API server and
associated with pods running in the cluster through the
. Bearer tokens are
mounted into pods at well-known locations, and allow in-cluster processes to
talk to the API server. Accounts may be explicitly associated with pods using the
field of a
PodSpec
.
Note:
is usually omitted because this is done automatically.
:
apps/v1
# this apiVersion is relevant as of Kubernetes 1.9
kind
:
metadata
:
name
:
namespace
:
default
spec
:
replicas
:
3
template
:
metadata
:
# ...
spec
:
:
bob-the-bot
:
-
name
:
nginx
image
:
Service account bearer tokens are perfectly valid to use outside the cluster and
can be used to create identities for long standing jobs that wish to talk to the
Kubernetes API. To manually create a service account, use the
kubectl create serviceaccount (NAME)
command. This creates a service account in the current
kubectl create serviceaccount jenkins
serviceaccount/jenkins created
Create an associated token:
kubectl create token jenkins
eyJhbGciOiJSUzI1NiIsImtp...
The created token is a signed JSON Web Token (JWT).
The signed JWT can be used as a bearer token to authenticate as the given service
account. See
above
for how the token is included
in a request. Normally these tokens are mounted into pods for in-cluster access to
the API server, but can be used from outside the cluster as well.
Service accounts authenticate with the username
system:serviceaccount:(NAMESPACE):(SERVICEACCOUNT)
,
and are assigned to the groups
and
system:serviceaccounts:(NAMESPACE)
.
Warning:
Because service account tokens can also be stored in Secret API objects, any user with
write access to Secrets can request a token, and any user with read access to those
Secrets can authenticate as the service account. Be cautious when granting permissions
to service accounts and read or write capabilities for Secrets.
OpenID Connect Tokens
OpenID Connect
is a flavor of OAuth2 supported by
some OAuth2 providers, notably Azure Active Directory, Salesforce, and Google.
The protocol's main extension of OAuth2 is an additional field returned with
the access token called an
ID Token
.
This token is a JSON Web Token (JWT) with well known fields, such as a user's
email, signed by the server.
To identify the user, the authenticator uses the
id_token
(not the
access_token
)
from the OAuth2
token response
as a bearer token. See
above
for how the token
is included in a request.
sequenceDiagram
participant user as User
participant idp as Identity Provider
participant kube as Kubectl
participant api as API Server
user -&gt;&gt; idp: 1. Login to IdP
activate idp
idp --&gt;&gt; user: 2. Provide access_token,
id_token, and refresh_token
deactivate idp
activate user
user -&gt;&gt; kube: 3. Call Kubectl
with --token being the id_token
OR add tokens to .kube/config
deactivate user
activate kube
kube -&gt;&gt; api: 4. Authorization: Bearer...
deactivate kube
activate api
api -&gt;&gt; api: 5. Is JWT signature valid?
api -&gt;&gt; api: 6. Has the JWT expired? (iat+exp)
api -&gt;&gt; api: 7. User authorized?
api --&gt;&gt; kube: 8. Authorized: Perform
action and return result
deactivate api
kube --x user: 9. Return result
enabled
Login to your identity provider
Your identity provider will provide you with an
,
id_token
and a
refresh_token
When using
kubectl
, use your
id_token
with the
--token
flag or add it directly to your
kubectl
sends your
id_token
in a header called Authorization to the API server
The API server will make sure the JWT signature is valid by checking against the certificate named in the configura

tion
Check to make sure the
id_token
hasn't expired
Make sure the user is authorized
Once authorized the API server returns a response to
kubectl
kubectl
provides feedback to the user
Since all of the data needed to validate who you are is in the
id_token
, Kubernetes doesn't need to
"phone home" to the identity provider. In a model where every request is stateless this provides a very scalable solution for authentication. It does offer a few challenges:
Kubernetes has no "web interface" to trigger the authentication process. There is no browser or interface to collect credentials which is why you need to authenticate to your identity provider first.
The
id_token
can't be revoked, it's like a certificate so it should be short-lived (only a few minutes) so it can be very annoying to have to get a new token every few minutes.
To authenticate to the Kubernetes dashboard, you must use the
command or a reverse proxy that injects the
id_token
.
Configuring the API Server
To enable the plugin, configure the following flags on the API server:
Parameter
Example
Required
--oidc-issuer-url
URL of the provider which allows the API server to discover public signing keys. Only URLs which use the
https://
scheme are accepted. This is typically the provider's discovery URL without a path, for example "https://accounts.google.com" or "https://login.salesforce.com". This URL should point to the level below .well-known/openid-configuration
If the discovery URL is
https://accounts.google.com/.well-known/openid-configuration
, the value should be
https://accounts.google.com
Yes
--oidc-client-id
A client id that all tokens must be issued for.
Yes
--oidc-username-claim
JWT claim to use as the user name. By default
sub
, which is expected to be a unique identifier of the end user. Admins can choose other claims, such as
email
or
name
, depending on their provider. However, claims other than
email
will be prefixed with the issuer URL to prevent naming clashes with other plugins.
sub
No
--oidc-username-prefix
Prefix prepended to username claims to prevent clashes with existing names (such as
system:
users). For example, the value
oidc:
will create usernames like
oidc:jane.doe
. If this flag isn't provided and
is a value other than
email
the prefix defaults to
( Issuer URL )#
where
( Issuer URL )
. The value
-
can be used to disable all prefixing.
oidc:
No
--oidc-groups-claim
JWT claim to use as the user's group. If the claim is present it must be an array of strings.
groups
No
--oidc-groups-prefix
Prefix prepended to group claims to prevent clashes with existing names (such as
system:
groups). For example, the value
oidc:
will create group names like
oidc:engineering
and
oidc:infra
.
oidc:
No
--oidc-required-claim
claim=value
No
--oidc-ca-file
The path to the certificate for the CA that signed your identity provider's web certificate. Defaults to the host's root CAs.
/etc/kubernetes/ssl/kc-ca.pem
No
--oidc-signing-algs
The signing algorithms accepted. Default is "RS256".
RS512
No
Importantly, the API server is not an OAuth2 client, rather it can only be
configured to trust a single issuer. This allows the use of public providers,
such as Google, without trusting credentials issued to third parties. Admins who
wish to utilize multiple OAuth clients should explore providers which support the
azp
(authorized party) claim, a mechanism for allowing one client to issue
tokens on behalf of another.
Kubernetes does not provide an OpenID Connect Identity Provider.
You can use an existing public OpenID Connect Identity Provider (such as Google, or
others
).
Or, you can run your own Identity Provider, such as
dex
,
Keycloak
,
CloudFoundry
UAA
, or
Tremolo Security's
OpenUnison
.
Fo

r an identity provider to work with Kubernetes it must:
Support
OpenID connect discovery
; not all do.
Run in TLS with non-obsolete ciphers
Have a CA signed certificate (even if the CA is not a commercial CA or is self signed)
A note about requirement #3 above, requiring a CA signed certificate. If you deploy your own identity provider (as opposed to one of the cloud providers like Google or Microsoft) you MUST have your identity provider's web server certificate signed by a certificate with the
CA
flag set to
TRUE
, even if it is self signed. This is due to GoLang's TLS client implementation being very strict to the standards around certificate validation. If you don't have a CA handy, you can use
this script
from the Dex team to create a simple CA and a signed certificate and key pair.
Or you can use
this similar script
that generates SHA256 certs with a longer life and larger key size.
Setup instructions for specific systems:
UAA
Dex
Option 1 - OIDC Authenticator
The first option is to use the kubectl
oidc
authenticator, which sets the
id_token
as a bearer token for all requests and refreshes the token once it expires. After you've logged into your provider, use kubectl to add your
id_token
,
,
client_id
, and
client_secret
to configure the plugin.
Providers that don't return an
id_token
as part of their refresh token response aren't supported by this plugin and should use "Option 2" below.
kubectl config set-credentials USER_NAME
\
--auth-provider
=
oidc
\
--auth-provider-arg
=
idp-issuer-url
=(
issuer url
)
\
=
client-id
=(
your client id
)
\
=
client-secret
=(
your client secret
)
\
=
refresh-token
=(
your refresh token
)
\
=
idp-certificate-authority
=(
path to your ca certificate
)
\
=
id-token
=(
your id_token
)
As an example, running the below command after authenticating to your identity provider:
kubectl config set-credentials mmosley
\
=
oidc
\
=
=
https://oidcidp.tremolo.lan:8443/auth/idp/OidcIdP
\
=
client-id
=
\
=
=
1db158f6-177d-4d9c-8a8b-d36869918ec5
\
=
=
q1bKLFOyUiosTfawzA93TzZIDzH2TNa2SMm0zEiPKTUwME6BkEo6Sql5yUWVBSWpKUGphaWpxSVAfekBOZbBhaEW+VlFUeVRGcluyVF5JT4+haZmPsluFoFu5XkpXk5BXqHega4GAXlF+ma+vmYpFcHe5eZR+slBFpZKtQA
=
\
=
=
/root/ca.pem
\
=
id-token
=
eyJraWQiOiJDTj1vaWRjaWRwLnRyZW1vbG8ubGFuLCBPVT1EZW1vLCBPPVRybWVvbG8gU2VjdXJpdHksIEw9QXJsaW5ndG9uLCBTVD1WaXJnaW5pYSwgQz1VUy1DTj1rdWJlLWNhLTEyMDIxNDc5MjEwMzYwNzMyMTUyIiwiYWxnIjoiUlMyNTYifQ.eyJpc3MiOiJodHRwczovL29pZGNpZHAudHJlbW9sby5sYW46ODQ0My9hdXRoL2lkcC9PaWRjSWRQIiwiYXVkIjoia3ViZXJuZXRlcyIsImV4cCI6MTQ4MzU0OTUxMSwianRpIjoiMm96US15TXdFcHV4WDlHZUhQdy1hZyIsImlhdCI6MTQ4MzU0OTQ1MSwibmJmIjoxNDgzNTQ5MzMxLCJzdWIiOiI0YWViMzdiYS1iNjQ1LTQ4ZmQtYWIzMC0xYTAxZWU0MWUyMTgifQ.w6p4J_6qQ1HzTG9nrEOrubxIMb9K5hzcMPxc9IxPx2K4xO9l-oFiUw93daH3m5pluP6K7eOE6txBuRVfEcpJSwlelsOsW8gb8VJcnzMS9EnZpeA0tW_p-mnkFc3VcfyXuhe5R3G7aa5d8uHv70yJ9Y3-UhjiN9EhpMdfPAoEB9fYKKkJRzF7utTTIPGrSaSU6d2pcpfYKaxIwePzEkT4DfcQthoZdy9ucNvvLoi1DIC-UocFD8HLs8LYKEqSxQvOcvnThbObJ9af71EwmuE21fO5KzMW20KtAeget1gnldOosPtz1G5EwvaQ401-RPQzPGMVBld0_zMCAwZttJ4knw
Which would produce the below configuration:
users
:
-
name
:
mmosley
user
:
:
config
:
client-id
:
:
id-token
:
eyJraWQiOiJDTj1vaWRjaWRwLnRyZW1vbG8ubGFuLCBPVT1EZW1vLCBPPVRybWVvbG8gU2VjdXJpdHksIEw9QXJsaW5ndG9uLCBTVD1WaXJnaW5pYSwgQz1VUy1DTj1rdWJlLWNhLTEyMDIxNDc5MjEwMzYwNzMyMTUyIiwiYWxnIjoiUlMyNTYifQ.eyJpc3MiOiJodHRwczovL29pZGNpZHAudHJlbW9sby5sYW46ODQ0My9hdXRoL2lkcC9PaWRjSWRQIiwiYXVkIjoia3ViZXJuZXRlcyIsImV4cCI6MTQ4MzU0OTUxMSwianRpIjoiMm96US15TXdFcHV4WDlHZUhQdy1hZyIsImlhdCI6MTQ4MzU0OTQ1MSwibmJmIjoxNDgzNTQ5MzMxLCJzdWIiOiI0YWViMzdiYS1iNjQ1LTQ4ZmQtYWI

zMC0xYTAxZWU0MWUyMTgifQ.w6p4J_6qQ1HzTG9nrEOrubxIMb9K5hzcMPxc9IxPx2K4xO9l-oFiUw93daH3m5pluP6K7eOE6txBuRVfEcpJSwlelsOsW8gb8VJcnzMS9EnZpeA0tW_p-mnkFc3VcfyXuhe5R3G7aa5d8uHv70yJ9Y3-UhjiN9EhpMdfPAoEB9fYKKkJRzF7utTTIPGrSaSU6d2pcpfYKaxIwePzEkT4DfcQthoZdy9ucNvvLoi1DIC-UocFD8HLs8LYKEqSxQvOcvnThbObJ9af71EwmuE21fO5KzMW20KtAeget1gnldOosPtz1G5EwvaQ401-RPQzPGMVBld0_zMCAwZttJ4knw
:
:
:
q1bKLFOyUiosTfawzA93TzZIDzH2TNa2SMm0zEiPKTUwME6BkEo6Sql5yUWVBSWpKUGphaWpxSVAfekBOZbBhaEW+VlFUeVRGcluyVF5JT4+haZmPsluFoFu5XkpXk5BXq
name
:
oidc
Once your
id_token
expires,
kubectl
will attempt to refresh your
id_token
using your
and
storing the new values for the
and
id_token
in your
.
Option 2 - Use the
--token
Option
The
kubectl
command lets you pass in a token using the
--token
option. Copy and paste the
id_token
into this option:
kubectl --token
=
eyJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJodHRwczovL21sYi50cmVtb2xvLmxhbjo4MDQzL2F1dGgvaWRwL29pZGMiLCJhdWQiOiJrdWJlcm5ldGVzIiwiZXhwIjoxNDc0NTk2NjY5LCJqdGkiOiI2RDUzNXoxUEpFNjJOR3QxaWVyYm9RIiwiaWF0IjoxNDc0NTk2MzY5LCJuYmYiOjE0NzQ1OTYyNDksInN1YiI6Im13aW5kdSIsInVzZXJfcm9sZSI6WyJ1c2VycyIsIm5ldy1uYW1lc3BhY2Utdmlld2VyIl0sImVtYWlsIjoibXdpbmR1QG5vbW9yZWplZGkuY29tIn0.f2As579n9VNoaKzoF-dOQGmXkFKf1FMyNV0-va_B63jn-_n9LGSCca_6IVMP8pO-Zb4KvRqGyTP0r3HkHxYy5c81AnIh8ijarruczl-TK_yF5akjSTHFZD-0gRzlevBDiH8Q79NAr-ky0P4iIXS8lY9Vnjch5MF74Zx0c3alKJHJUnnpjIACByfF2SCaYzbWFMUNat-K1PaUk5-ujMBG7yYnr95xD-63n8CO8teGUAAEMx6zRjzfhnhbzX-ajwZLGwGUBT4WqjMs70-6a7_8gZmLZb2az1cZynkFRj2BaCkVT3A2RrjeEwZEtGXlMqKJ1_I2ulrOVsYx01_yD35-rw get nodes
Webhook Token Authentication
Webhook authentication is a hook for verifying bearer tokens.
--authentication-token-webhook-config-file
a configuration file describing how to access the remote webhook service.
--authentication-token-webhook-cache-ttl
how long to cache authentication decisions. Defaults to two minutes.
--authentication-token-webhook-version
determines whether to use
or
objects to send/receive information from the webhook. Defaults to
v1beta1
.
The configuration file uses the
file format. Within the file,
clusters
refers to the remote service and
users
refers to the API server webhook. An example would be:
# Kubernetes API version
:
v1
# kind of the API object
kind
:
Config
clusters
:
-
name
:
name-of-remote-authn-service
cluster
:
:
server
:
https://authn.example.com/authenticate
# URL of remote service to query. 'https' recommended for production.
users
:
-
name
:
user
:
:
# cert for the webhook plugin to use
:
# kubeconfig files require a context. Provide one for the API server.
:
webhook
contexts
:
-
context
:
cluster
:
user
:
name
:
webhook
When a client attempts to authenticate with the API server using a bearer token as discussed
above
,
the authentication webhook POSTs a JSON-serialized
object containing the token to the remote service.
Note that webhook API objects are subject to the same
versioning compatibility rules
as other Kubernetes API objects.
Implementers should check the
field of the request to ensure correct deserialization,
and
must
object of the same version as the request.
Note:
The Kubernetes API server defaults to sending
token reviews for backwards compatibility.
To opt into receiving
token reviews, the API server must be started wit

h
--authentication-token-webhook-version=v1
.
{
"authentication.k8s.io/v1"
,
"kind":
"TokenReview"
,
"spec":
{
# Opaque bearer token sent to the API server
"token":
"014fbff9a07c..."
,
# Optional list of the audience identifiers for the server the token was presented to.
# Audience-aware token authenticators (for example, OIDC token authenticators)
# should verify the token was intended for at least one of the audiences in this list,
# and return the intersection of this list and the valid audiences for the token in the response status.
# This ensures the token is valid to authenticate to the server it was presented to.
# If no audiences are provided, the token should be validated to authenticate to the Kubernetes API server.
"audiences":
[
"https://myserver.example.com"
,
"https://myserver.internal.example.com"
]
}
}
{
"authentication.k8s.io/v1beta1"
,
"kind":
,
"spec":
{
"token":
,
[
,
]
}
}
status
field of the request to indicate the success of the login.
The response body's
spec
field is ignored and may be omitted.
The remote service must return a response using the same
API version that it received.
A successful validation of the bearer token would return:
{
,
"kind":
,
"status":
{
"authenticated":
true
,
"user":
{
# Required
"username":
"janedoe@example.com"
,
# Optional
"uid":
"42"
,
# Optional group memberships
"groups":
[
"developers"
,
"qa"
],
# Optional additional information provided by the authenticator.
# This should not contain confidential data, as it can be recorded in logs
# or API objects, and is made available to admission webhooks.
"extra":
{
"extrafield1":
[
"extravalue1"
,
"extravalue2"
]
}
},
# Optional list audience-aware token authenticators can return,
# containing the audiences from the `spec.audiences` list for which the provided token was valid.
# If this is omitted, the token is considered to be valid to authenticate to the Kubernetes API server.
[
]
}
}
{
,
"kind":
,
"status":
{
true
,
"user":
{
# Required
,
# Optional
"uid":
"42"
,
"groups":
[
,
"qa"
],
"extra":
{
[
,
]
}
},
[
]
}
}
An unsuccessful request would return:
{
,
"kind":
,
"status":
{
false
,
# Optionally include details about why authentication failed.
# If no error is provided, the

 API will return a generic Unauthorized message.
# The error field is ignored when authenticated=true.
"error":
"Credentials are expired"
}
}
{
,
"kind":
,
"status":
{
false
,
# If no error is provided, the API will return a generic Unauthorized message.
"error":
}
}
Authenticating Proxy
The API server can be configured to identify users from request header values, such as
X-Remote-User
.
It is designed for use in combination with an authenticating proxy, which sets the request header value.
--requestheader-username-headers
Required, case-insensitive. Header names to check, in order, for the user identity. The first header containing a value is used as the username.
--requestheader-group-headers
1.6+. Optional, case-insensitive. "X-Remote-Group" is suggested. Header names to check, in order, for the user's groups. All values in all specified headers are used as group names.
--requestheader-extra-headers-prefix
1.6+. Optional, case-insensitive. "X-Remote-Extra-" is suggested. Header prefixes to look for to determine extra information about the user (typically used by the configured authorization plugin). Any headers beginning with any of the specified prefixes have the prefix removed. The remainder of the header name is lowercased and
percent-decoded
and becomes the extra key, and the header value is the extra value.
Note:
Prior to 1.11.3 (and 1.10.7, 1.9.11), the extra key could only contain characters which were
legal in HTTP header labels
.
For example, with this configuration:
this request:
GET
/
HTTP
/
1.1
:
fido
X-Remote-Group
:
dogs
:
dachshunds
X-Remote-Extra-Acme.com%2Fproject
:
some-project
X-Remote-Extra-Scopes
:
openid
:
profile
would result in this user info:
name
:
fido
groups
:
- dogs
- dachshunds
extra
:
acme.com/project
:
- some-project
scopes
:
- openid
- profile
In order to prevent header spoofing, the authenticating proxy is required to present a valid client
certificate to the API server for validation against the specified CA before the request headers are
checked. WARNING: do
not
reuse a CA that is used in a different context unless you understand
the risks and the mechanisms to protect the CA's usage.
Required. PEM-encoded certificate bundle. A valid client certificate must be presented and validated against the certificate authorities in the specified file before the request headers are checked for user names.
--requestheader-allowed-names
Optional. List of Common Name values (CNs). If set, a valid client certificate with a CN in the specified list must be presented before the request headers are checked for user names. If empty, any CN is allowed.
Anonymous requests
When enabled, requests that are not rejected by other configured authentication methods are
treated as anonymous requests, and given a username of
.
For example, on a server with token authentication configured, and anonymous access enabled,
a request providing an invalid bearer token would receive a
error.
A request providing no bearer token would be treated as an anonymous request.
In 1.5.1-1.5.x, anonymous access is disabled by default, and can be enabled by
passing the
--anonymous-auth=true
option to the API server.
In 1.6+, anonymous access is enabled by default if an authorization mode other than
is used, and can be disabled by passing the
Starting in 1.6, the ABAC and RBAC authorizers require explicit authorization of the
user or the
system:unauthen

ticated
group, so legacy policy rules
that grant access to the
*
user or
*
group do not include anonymous users.
User impersonation
A user can act as another user through impersonation headers. These let requests
manually override the user info a request authenticates as. For example, an admin
could use this feature to debug an authorization policy by temporarily
impersonating another user and seeing if a request was denied.
Impersonation requests first authenticate as the requesting user, then switch
to the impersonated user info.
A user makes an API call with their credentials
and
impersonation headers.
API server authenticates the user.
API server ensures the authenticated users have impersonation privileges.
Request user info is replaced with impersonation values.
Request is evaluated, authorization acts on impersonated user info.
The following HTTP headers can be used to performing an impersonation request:
Impersonate-User
: The username to act as.
Impersonate-Group
: A group name to act as. Can be provided multiple times to set multiple groups. Optional. Requires "Impersonate-User".
Impersonate-Extra-( extra name )
: A dynamic header used to associate extra fields with the user. Optional. Requires "Impersonate-User". In order to be preserved consistently,
( extra name )
must be lower-case, and any characters which aren't
MUST be utf8 and
percent-encoded
.
Impersonate-Uid
: A unique identifier that represents the user being impersonated. Optional. Requires "Impersonate-User". Kubernetes does not impose any format requirements on this string.
Note:
Prior to 1.11.3 (and 1.10.7, 1.9.11),
could only contain characters which were
.
Note:
is only available in versions 1.22.0 and higher.
An example of the impersonation headers used when impersonating a user with groups:
Impersonate-User: jane.doe@example.com
Impersonate-Group: developers
Impersonate-Group: admins
An example of the impersonation headers used when impersonating a user with a UID and
extra fields:
Impersonate-Extra-dn: cn=jane,ou=engineers,dc=example,dc=com
Impersonate-Extra-acme.com%2Fproject: some-project
Impersonate-Extra-scopes: view
Impersonate-Extra-scopes: development
Impersonate-Uid: 06f6ce97-e2c5-4ab8-7ba5-7654dd08d52b
When using
kubectl
set the
--as
flag to configure the
header, set the
--as-group
header.
kubectl drain mynode
Error from server (Forbidden): User "clark" cannot get nodes at the cluster scope. (get nodes mynode)
Set the
--as
and
flag:
kubectl drain mynode --as
=
superman --as-group
=
node/mynode cordoned
node/mynode drained
Note:
kubectl
cannot impersonate extra fields or UIDs.
To impersonate a user, group, user identifier (UID) or extra fields, the impersonating user must
have the ability to perform the "impersonate" verb on the kind of attribute
being impersonated ("user", "group", "uid", etc.). For clusters that enable the RBAC
authorization plugin, the following ClusterRole encompasses the rules needed to
set user and group impersonation headers:
:
kind
:
metadata
:
name
:
impersonator
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"users"
,
"groups"
,
"serviceaccounts"
]
verbs
:
[
"impersonate"
]
For impersonation, extra fields and impersonated UIDs are both under the "authentication.k8s.io"
apiGroup
.
Extra fields are evaluated as sub-resources of the resource "userextras". To
allow a user to use impersonation headers for the extra field "scopes" and
for UIDs, a user should be granted the following role:
:
kind
:
metadata
:
name
:
scopes-and-uid-impersonator
rules
:
# Can set "Impersonate-Extra-scopes" header and the "Impersonate-Uid" header.
-
apiGroups
:
[
"authentication.k8s.io"
]
resources
:
[
"userextras/scopes"
,
"uids"
]
ver

bs
:
[
]
The values of impersonation headers can also be restricted by limiting the set
of
a resource can take.
:
kind
:
metadata
:
name
:
limited-impersonator
rules
:
# Can impersonate the user "jane.doe@example.com"
-
apiGroups
:
[
""
]
resources
:
[
"users"
]
verbs
:
[
]
:
[
"jane.doe@example.com"
]
# Can impersonate the groups "developers" and "admins"
-
apiGroups
:
[
""
]
resources
:
[
"groups"
]
verbs
:
[
]
:
[
,
"admins"
]
# Can impersonate the extras field "scopes" with the values "view" and "development"
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
]
:
[
"view"
,
"development"
]
# Can impersonate the uid "06f6ce97-e2c5-4ab8-7ba5-7654dd08d52b"
-
apiGroups
:
[
]
resources
:
[
"uids"
]
verbs
:
[
]
:
[
"06f6ce97-e2c5-4ab8-7ba5-7654dd08d52b"
]
Note:
Impersonating a user or group allows you to perform any action as if you were that user or group;
for that reason, impersonation is not namespace scoped.
If you want to allow impersonation using Kubernetes RBAC,
this requires using a
and a
,
not a
Role
and
.
client-go credential plugins
k8s.io/client-go
and tools using it such as
kubectl
and
kubelet
are able to execute an
external command to receive user credentials.
This feature is intended for client side integrations with authentication protocols not natively
supported by
(LDAP, Kerberos, OAuth2, SAML, etc.). The plugin implements the
protocol specific logic, then returns opaque credentials to use. Almost all credential plugin
use cases require a server side component with support for the
webhook token authenticator
to interpret the credential format produced by the client plugin.
Example use case
In a hypothetical use case, an organization would run an external service that exchanges LDAP credentials
for user specific, signed tokens. The service would also be capable of responding to
webhook token
authenticator
requests to validate the tokens. Users would be required
to install a credential plugin on their workstation.
To authenticate against the API:
The user issues a
kubectl
command.
Credential plugin prompts the user for LDAP credentials, exchanges credentials with external service for a token.
Credential plugin returns token to client-go, which uses it as a bearer token against the API server.
API server uses the
to submit a
to the external service.
External service verifies the signature on the token and returns the user's username and groups.
Credential plugins are configured through
kubectl config files
as part of the user fields.
client.authentication.k8s.io/v1
client.authentication.k8s.io/v1beta1
:
v1
kind
:
Config
users
:
-
name
:
my-user
user
:
exec
:
# Command to execute. Required.
command
:
"example-client-go-exec-plugin"
# API version to use when decoding the ExecCredentials resource. Required.
#
# The API version returned by the plugin MUST match the version listed here.
#
# To integrate with tools that support multiple versions (such as client.authentication.k8s.io/v1beta1),
# set an environment variable, pass an argument to the tool that indicates which version the exec plugin expects,
# or read the version from the ExecCredential object in the KUBERNETES_EXEC_INFO environment variable.
:
"client.authentication.k8s.io/v1"
# Environment variables to set when executing the plugin. Optional.
env
:
-
name
:
"FOO"
value
:
"bar"
# Arguments to pass when executing the plugin. Optional.
args
:
-
"arg1"
-
"arg2"
# Text shown to the user when the executable doesn't seem to be present. Optional.
:
|
example-client-go-exec-plugin is required to authenticate
to the cu

rrent cluster.  It can be installed:
On macOS: brew install example-client-go-exec-plugin
On Ubuntu: apt-get install example-client-go-exec-plugin
On Fedora: dnf install example-client-go-exec-plugin
...
# Whether or not to provide cluster information, which could potentially contain
# very large CA data, to this exec plugin as a part of the KUBERNETES_EXEC_INFO
# environment variable.
:
true
# The contract between the exec plugin and the standard input I/O stream. If the
# contract cannot be satisfied, this plugin will not be run and an error will be
# returned. Valid values are "Never" (this exec plugin never uses standard input),
# "IfAvailable" (this exec plugin wants to use standard input if it is available),
# or "Always" (this exec plugin requires standard input to function). Required.
:
Never
clusters
:
-
name
:
my-cluster
cluster
:
server
:
"https://172.17.4.100:6443"
:
"/etc/kubernetes/ca.pem"
:
-
name
:
client.authentication.k8s.io/exec
# reserved extension name for per cluster exec config
extension
:
arbitrary
:
config
this
:
can be provided via the KUBERNETES_EXEC_INFO environment variable upon setting provideClusterInfo
you
:
[
"can"
,
"put"
,
"anything"
,
"here"
]
contexts
:
-
name
:
context
:
cluster
:
user
:
my-user
:
:
v1
kind
:
Config
users
:
-
name
:
my-user
user
:
exec
:
command
:
#
#
# To integrate with tools that support multiple versions (such as client.authentication.k8s.io/v1),
:
"client.authentication.k8s.io/v1beta1"
env
:
-
name
:
"FOO"
value
:
"bar"
args
:
-
"arg1"
-
"arg2"
:
|
to the current cluster.  It can be installed:
...
:
true
# or "Always" (this exec plugin requires standard input to function). Optional.
# Defaults to "IfAvailable".
:
Never
clusters
:
-
name
:
cluster
:
server
:
:
:
-
name
:
extension
:
arbitrary
:
config
this
:
you
:
[
"can"
,
"put"
,
,
"here"
]
contexts
:
-
name
:
context
:
cluster
:
user
:
my-user
:
Relative command paths are interpreted as relative to the directory of the config file. If
KUBECONFIG is set to
/home/jane/kubeconfig
and the exec command is
./bin/example-client-go-exec-plugin
,
the binary
/

home/jane/bin/example-client-go-exec-plugin
is executed.
-
name
:
my-user
user
:
exec
:
# Path relative to the directory of the kubeconfig
command
:
"./bin/example-client-go-exec-plugin"
:
:
Never
Input and output formats
The executed command prints an
ExecCredential
object to
stdout
.
authenticates against the Kubernetes API using the returned credentials in the
status
.
The executed command is passed an
object as input via the
KUBERNETES_EXEC_INFO
environment variable. This input contains helpful information like the expected API version
of the returned
object and whether or not the plugin can use
stdin
to interact
with the user.
When run from an interactive session (i.e., a terminal),
stdin
can be exposed directly
to the plugin. Plugins should use the
spec.interactive
field of the input
object from the
environment variable in order to
determine if
stdin
has been provided. A plugin's
stdin
requirements (i.e., whether
stdin
is optional, strictly required, or never used in order for the plugin
to run successfully) is declared via the
user.exec.interactiveMode
(see table
below for valid values). The
field is optional in
and required in
.
interactiveMode values
Value
Meaning
Never
This exec plugin never needs to use standard input, and therefore the exec plugin will be run regardless of whether standard input is available for user input.
IfAvailable
This exec plugin would like to use standard input if it is available, but can still operate if standard input is not available. Therefore, the exec plugin will be run regardless of whether stdin is available for user input. If standard input is available for user input, then it will be provided to this exec plugin.
Always
This exec plugin requires standard input in order to run, and therefore the exec plugin will only be run if standard input is available for user input. If standard input is not available for user input, then the exec plugin will not be run and an error will be returned by the exec plugin runner.
To use bearer token credentials, the plugin returns a token in the status of the
{
:
,
"kind"
:
"ExecCredential"
,
"status"
: {
"token"
:
"my-bearer-token"
}
}
{
:
,
"kind"
:
,
"status"
: {
"token"
:
}
}
Alternatively, a PEM-encoded client certificate and key can be returned to use TLS client auth.
If the plugin returns a different certificate and key on a subsequent call,
will close existing connections with the server to force a new TLS handshake.
If specified,
clientKeyData
and
clientCertificateData
must both must be present.
may contain additional intermediate certificates to send to the server.
{
:
,
"kind"
:
,
"status"
: {
"clientCertificateData"
:
"-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"
,
"clientKeyData"
:
"-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
}
}
{
:
,
"kind"
:
,
"status"
: {
:
,
:
}
}
Optionally, the response can include the expiry of the credential formatted as a
RFC 3339
timestamp.
Presence or absence of an expiry has the following impact:
If an expiry is included, the bearer token and TLS credentials are cached until

the expiry time is reached, or if the server responds with a 401 HTTP status code,
or when the process exits.
If an expiry is omitted, the bearer token and TLS credentials are cached until
the server responds with a 401 HTTP status code or until the process exits.
{
:
,
"kind"
:
,
"status"
: {
"token"
:
,
"expirationTimestamp"
:
"2018-03-05T17:30:20-08:00"
}
}
{
:
,
"kind"
:
,
"status"
: {
"token"
:
,
:
}
}
To enable the exec plugin to obtain cluster-specific information, set
on the
user.exec
.
The plugin will then be supplied this cluster-specific information in the
Information from this environment variable can be used to perform cluster-specific
credential acquisition logic.
The following
manifest describes a cluster information sample.
{
:
,
"kind"
:
,
"spec"
: {
"cluster"
: {
"server"
:
,
"certificate-authority-data"
:
"LS0t..."
,
"config"
: {
"arbitrary"
:
"config"
,
"this"
:
"can be provided via the KUBERNETES_EXEC_INFO environment variable upon setting provideClusterInfo"
,
"you"
: [
"can"
,
"put"
,
,
"here"
]
}
},
"interactive"
:
true
}
}
{
:
,
"kind"
:
,
"spec"
: {
"cluster"
: {
"server"
:
,
:
"LS0t..."
,
"config"
: {
:
"config"
,
"this"
:
,
"you"
: [
"can"
,
"put"
,
,
"here"
]
}
},
:
true
}
}
API access to authentication information for a client
If your cluster has the API enabled, you can use the
SelfSubjectReview
API to find out how your Kubernetes cluster maps your authentication
information to identify you as a client. This works whether you are authenticating as a user (typically representing
a real person) or as a ServiceAccount.
objects do not have any configurable fields. On receiving a request, the Kubernetes API server fills the status with the user attributes and returns it to the user.
Request example (the body would be a
):
POST /apis/authentication.k8s.io/v1alpha1/selfsubjectreviews
{
:
"authentication.k8s.io/v1alpha1"
,
"kind"
:
"SelfSubjectReview"
}
Response example:
{
:
,
"kind"
:
,
"status"
: {
"userInfo"
: {
"name"
:
"jane.doe"
,
"uid"
:
"b6c7cfd4-f166-11ec-8ea0-0242ac120002"
,
"groups"
: [
"viewers"
,
"editors"
,
],
"extra"
: {
"provider_id"
: [
"token.company.example"
]
}
}
}
}
For convenience, the
kubectl alpha auth whoami
command is present. Executing this command will produce the following output (yet different user attributes will be shown):
Simple output example
ATTRIBUTE         VALUE
Username          jane.doe
Groups            [system:authenticated]
Complex example including extra attributes
UID               b79dbf30-0c6a-11ed-861d-0242ac120002
Groups            [students teachers system:authenticated]
Extra: skills     [reading learning]
Extra: subjects   [math sports]
By providing the output flag, it is also possible to print the JSON or YAML representation of the result:
JSON
YAML
{
:
,
"kind"
:
,
"status"
: {
: {
:
,
"uid"
:
"b79dbf30-0c6a-11ed-861d-024

2ac120002"
,
"groups"
: [
"students"
,
"teachers"
,
],
"extra"
: {
"skills"
: [
"reading"
,
"learning"
],
"subjects"
: [
"math"
,
"sports"
]
}
}
}
}
:
authentication.k8s.io/v1alpha1
kind
:
status
:
userInfo
:
username
:
jane.doe
uid
:
b79dbf30-0c6a-11ed-861d-0242ac120002
groups
:
- students
- teachers
- system:authenticated
extra
:
skills
:
- reading
- learning
subjects
:
- math
- sports
This feature is extremely useful when a complicated authentication flow is used in a Kubernetes cluster,
for example, if you use
webhook token authentication
or
.
Note:
The Kubernetes API server fills the
userInfo
after all authentication mechanisms are applied,
including
impersonation
.
If you, or an authentication proxy, make a SelfSubjectReview using impersonation,
you see the user details and properties for the user that was impersonated.
By default, all authenticated users can create
objects when the
APISelfSubjectReview
feature is enabled. It is allowed by the
system:basic-user
cluster role.
Note:
You can only make
requests if:
the
is enabled for your cluster
the API server for your cluster has the
API group
enabled.
Read the
client authentication reference (v1beta1)
Read the
client authentication reference (v1)
3.2 - Authenticating with Bootstrap Tokens
Bootstrap tokens are a simple bearer token that is meant to be used when
creating new clusters or joining new nodes to an existing cluster. It was built
to support
kubeadm
, but can be used in other contexts
for users that wish to start clusters without
kubeadm
. It is also built to
work, via RBAC policy, with the
Kubelet TLS Bootstrapping
system.
Bootstrap Tokens Overview
Bootstrap Tokens are defined with a specific type
(
bootstrap.kubernetes.io/token
) of secrets that lives in the
namespace. These Secrets are then read by the Bootstrap Authenticator in the
API Server. Expired tokens are removed with the TokenCleaner controller in the
Controller Manager. The tokens are also used to create a signature for a
specific ConfigMap used in a "discovery" process through a BootstrapSigner
Token Format
Bootstrap Tokens take the form of
abcdef.0123456789abcdef
. More formally,
they must match the regular expression
.
The first part of the token is the "Token ID" and is considered public
information. It is used when referring to a token without leaking the secret
part used for authentication. The second part is the "Token Secret" and should
only be shared with trusted parties.
Enabling Bootstrap Token Authentication
The Bootstrap Token authenticator can be enabled using the following flag on the
API server:
When enabled, bootstrapping tokens can be used as bearer token credentials to
authenticate requests against the API server.
Authorization: Bearer 07401b.f395accd246ae52d
Tokens authenticate as the username
system:bootstrap:&lt;token id&gt;
and are members
of the group
. Additional groups may be specified in the
token's Secret.
Expired tokens can be deleted automatically by enabling the
tokencleaner
controller on the controller manager.
Bootstrap Token Secret Format
Each valid token is backed by a secret in the
namespace. You can
find the full design doc
here
.
Here is what the secret looks like.
:
v1
kind
:
Secret
metadata
:
# Name MUST be of form "bootstrap-token-&lt;token id&gt;"
name
:
bootstrap-token-07401b
namespace
:
# Type MUST be 'bootstrap.kubernetes.io/token'
type
:
stringData
:
# Human readable description. Optional.
:
"The default bootstrap token generated by 'kubeadm init'."
# Token ID and secret. Required.
token-id
:
07401b
token-secret
:
f395accd246ae52d

# Expiration. Optional.
expiration
:
2017-03-10T03:22:11Z
# Allowed usages.
usage-bootstrap-authentication
:
"true"
usage-bootstrap-signing
:
"true"
# Extra groups to authenticate the token as. Must start with "system:bootstrappers:"
auth-extra-groups
:
system:bootstrappers:worker,system:bootstrappers:ingress
The type of the secret must be
and the name must
be
bootstrap-token-&lt;token id&gt;
. It must also exist in the
The
usage-bootstrap-*
members indicate what this secret is intended to be used
for. A value must be set to
true
indicates that the token can be used to
authenticate to the API server as a bearer token.
indicates that the token may be used to sign the
ConfigMap as described below.
The
field controls the expiry of the token. Expired tokens are
rejected when used for authentication and ignored during ConfigMap signing.
The expiry value is encoded as an absolute UTC time using RFC3339. Enable the
controller to automatically delete expired tokens.
Token Management with kubeadm
kubeadm
tool to manage tokens on a running cluster. See the
kubeadm token docs
ConfigMap Signing
In addition to authentication, the tokens can be used to sign a ConfigMap. This
is used early in a cluster bootstrap process before the client trusts the API
server. The signed ConfigMap can be authenticated by the shared token.
Enable ConfigMap signing by enabling the
bootstrapsigner
controller on the
Controller Manager.
--controllers=*,bootstrapsigner
The ConfigMap that is signed is
in the
The typical flow is that a client reads this ConfigMap while unauthenticated and
ignoring TLS errors. It then validates the payload of the ConfigMap by looking
at a signature embedded in the ConfigMap.
The ConfigMap may look like this:
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
data
:
jws-kubeconfig-07401b
:
eyJhbGciOiJIUzI1NiIsImtpZCI6IjA3NDAxYiJ9..tYEfbo6zDNo40MQE07aZcQX2m3EB2rO3NuXtxVMYm9U
:
|
clusters:
- cluster:
certificate-authority-data: &lt;really long certificate data&gt;
server: https://10.138.0.2:6443
name: ""
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []
The
member of the ConfigMap is a config file with only the cluster
information filled out. The key thing being communicated here is the
. This may be expanded in the future.
The signature is a JWS signature using the "detached" mode. To validate the
signature, the user should encode the
payload according to JWS
rules (base64 encoded while discarding any trailing
=
). That encoded payload
is then used to form a whole JWS by inserting it between the 2 dots. You can
verify the JWS using the
HS256
scheme (HMAC-SHA256) with the full token (e.g.
07401b.f395accd246ae52d
) as the shared secret. Users
must
verify that HS256
is used.
Warning:
Any party with a bootstrapping token can create a valid signature for that
token. When using ConfigMap signing it's discouraged to share the same token with
many clients, since a compromised client can potentially man-in-the middle another
client relying on the signature to bootstrap TLS trust.
kubeadm implementation details
section for more information.
3.3 - Certificate Signing Requests
The Certificates API enables automation of
X.509
credential provisioning by providing
a programmatic interface for clients of the Kubernetes API to request and obtain
X.509
from a Certificate Authority (CA).
A CertificateSigningRequest (CSR) resource is used to request that a certificate be signed
by a denoted signer, after which the request may be approved or denied before
finally being signed.
Request signing process
The CertificateSigningRequest resource

 type allows a client to ask for an X.509 certificate
be issued, based on a signing request.
The CertificateSigningRequest object includes a PEM-encoded PKCS#10 signing request in
the
spec.request
field. The CertificateSigningRequest denotes the signer (the
recipient that the request is being made to) using the
field.
Note that
is a required key after API version
.
In Kubernetes v1.22 and later, clients may optionally set the
spec.expirationSeconds
field to request a particular lifetime for the issued certificate. The minimum valid
value for this field is
600
, i.e. ten minutes.
Once created, a CertificateSigningRequest must be approved before it can be signed.
Depending on the signer selected, a CertificateSigningRequest may be automatically approved
by a
.
Otherwise, a CertificateSigningRequest must be manually approved either via the REST API (or client-go)
or by running
. Likewise, a CertificateSigningRequest may also be denied,
which tells the configured signer that it must not sign the request.
For certificates that have been approved, the next step is signing. The relevant signing controller
first validates that the signing conditions are met and then creates a certificate.
The signing controller then updates the CertificateSigningRequest, storing the new certificate into
the
field of the existing CertificateSigningRequest object. The
field is either empty or contains a X.509 certificate, encoded in PEM format.
The CertificateSigningRequest
field is empty until the signer does this.
Once the
field has been populated, the request has been completed and clients can now
fetch the signed certificate PEM data from the CertificateSigningRequest resource.
The signers can instead deny certificate signing if the approval conditions are not met.
In order to reduce the number of old CertificateSigningRequest resources left in a cluster, a garbage collection
controller runs periodically. The garbage collection removes CertificateSigningRequests that have not changed
state for some duration:
Approved requests: automatically deleted after 1 hour
Denied requests: automatically deleted after 1 hour
Failed requests: automatically deleted after 1 hour
Pending requests: automatically deleted after 24 hours
All requests: automatically deleted after the issued certificate has expired
Signers
Custom signerNames can also be specified. All signers should provide information about how they work so that clients can predict what will happen to their CSRs.
This includes:
Trust distribution
: how trust (CA bundles) are distributed.
Permitted subjects
: any restrictions on and behavior when a disallowed subject is requested.
Permitted x509 extensions
: including IP subjectAltNames, DNS subjectAltNames, Email subjectAltNames, URI subjectAltNames etc, and behavior when a disallowed extension is requested.
Permitted key usages / extended key usages
: any restrictions on and behavior when usages different than the signer-determined usages are specified in the CSR.
Expiration/certificate lifetime
: whether it is fixed by the signer, configurable by the admin, determined by the CSR
field, etc
and the behavior when the signer-determined expiration is different from the CSR
field.
CA bit allowed/disallowed
: and behavior if a CSR contains a request a for a CA certificate when the signer does not permit it.
Commonly, the
field contains a single PEM-encoded X.509
certificate once the CSR is approved and the certificate is issued. Some
signers store multiple certificates into the
field. In
that case, the documentation for the signer should specify the meaning of
additional certificates; for example, this might be the certificate plus
intermediates to be presented during TLS handshakes.
The PKCS#10 signing reque

st format does not have a standard mechanism to specify a
certificate expiration or lifetime. The expiration or lifetime therefore has to be set
through the
field of the CSR object. The built-in signers
use the
ClusterSigningDuration
configuration option, which defaults to 1 year,
(the
--cluster-signing-duration
command-line flag of the kube-controller-manager)
as the default when no
is specified. When
is specified, the minimum of
and
is
used.
Note:
The
field was added in Kubernetes v1.22. Earlier versions of Kubernetes do not honor this field.
Kubernetes API servers prior to v1.22 will silently drop this field when the object is created.
Kubernetes signers
Kubernetes provides built-in signers that each have a well-known
signerName
:
: signs certificates that will be honored as client certificates by the API server.
Never auto-approved by
.
Trust distribution: signed certificates must be honored as client certificates by the API server. The CA bundle is not distributed by any other means.
Permitted subjects - no subject restrictions, but approvers and signers may choose not to approve or sign.
Certain subjects like cluster-admin level users or groups vary between distributions and installations,
but deserve additional scrutiny before approval and signing.
The
admission plugin is enabled by default to restrict
,
but it is often not the only cluster-admin subject in a cluster.
Permitted x509 extensions - honors subjectAltName and key usage extensions and discards other extensions.
Permitted key usages - must include
["client auth"]
. Must not include key usages beyond
["digital signature", "key encipherment", "client auth"]
.
Expiration/certificate lifetime - for the kube-controller-manager implementation of this signer, set to the minimum
of the
option or, if specified, the
field of the CSR object.
CA bit allowed/disallowed - not allowed.
kubernetes.io/kube-apiserver-client-kubelet
: signs client certificates that will be honored as client certificates by the
API server.
May be auto-approved by
.
Trust distribution: signed certificates must be honored as client certificates by the API server. The CA bundle
is not distributed by any other means.
Permitted subjects - organizations are exactly
["system:nodes"]
, common name starts with "
system:node:
".
Permitted x509 extensions - honors key usage extensions, forbids subjectAltName extensions and drops other extensions.
Permitted key usages - exactly
["key encipherment", "digital signature", "client auth"]
.
of the
: signs serving certificates that are honored as a valid kubelet serving certificate
by the API server, but has no other guarantees.
.
Trust distribution: signed certificates must be honored by the API server as valid to terminate connections to a kubelet.
The CA bundle is not distributed by any other means.
".
Permitted x509 extensions - honors key usage and DNSName/IPAddress subjectAltName extensions, forbids EmailAddress and
URI subjectAltName extensions, drops other extensions. At least one DNS or IP subjectAltName must be present.
["key encipherment", "digital signature", "server auth"]
.
Expiration/certificate lifetime - for the kube-controller-manager implementation of this signer, set t

o the minimum
of the
: has no guarantees for trust at all. Some third-party distributions of Kubernetes
may honor client certificates signed by it. The stable CertificateSigningRequest API (version
and later)
does not allow to set the
as
.
.
Trust distribution: None. There is no standard trust or distribution for this signer in a Kubernetes cluster.
Permitted subjects - any
Permitted key usages - any
of the
Note:
Failures for all of these are only reported in kube-controller-manager logs.
Note:
The
Distribution of trust happens out of band for these signers. Any trust outside of those described above are strictly
coincidental. For instance, some distributions may honor
as client certificates for the
kube-apiserver, but this is not a standard.
None of these usages are related to ServiceAccount token secrets
.data[ca.crt]
in any way. That CA bundle is only
guaranteed to verify a connection to the API server using the default service (
).
To allow creating a CertificateSigningRequest and retrieving any CertificateSigningRequest:
Verbs:
create
,
get
,
list
,
watch
, group:
, resource:
access/certificate-signing-request/clusterrole-create.yaml
:
kind
:
metadata
:
name
:
csr-creator
rules
:
-
apiGroups
:
- certificates.k8s.io
resources
:
- certificatesigningrequests
verbs
:
- create
- get
- list
- watch
To allow approving a CertificateSigningRequest:
Verbs:
get
,
list
,
watch
, group:
Verbs:
update
, group:
certificatesigningrequests/approval
Verbs:
approve
, group:
signers
, resourceName:
&lt;signerNameDomain&gt;/&lt;signerNamePath&gt;
or
&lt;signerNameDomain&gt;/*
access/certificate-signing-request/clusterrole-approve.yaml
:
kind
:
metadata
:
name
:
csr-approver
rules
:
-
apiGroups
:
resources
:
verbs
:
- get
- list
- watch
-
apiGroups
:
resources
:
- certificatesigningrequests/approval
verbs
:
- update
-
apiGroups
:
resources
:
- signers
:
- example.com/my-signer-name
# example.com/* can be used to authorize for all signers in the 'example.com' domain
verbs
:
- approve
To allow signing a CertificateSigningRequest:
Verbs:
get
,
list
,
watch
, group:
Verbs:
update
, group:
certificatesigningrequests/status
Verbs:
sign
, group:
signers
or
access/certificate-signing-request/clusterrole-sign.yaml
:
kind
:
metadata
:
name
:
csr-signer
rules
:
-
apiGroups
:
resources
:
verbs
:
- get
- list
- watch
-
apiGroups
:
- 

resources
:
- certificatesigningrequests/status
verbs
:
- update
-
apiGroups
:
resources
:
- signers
:
verbs
:
- sign
Normal user
A few steps are required in order to get a normal user to be able to
authenticate and invoke an API. First, this user must have a certificate issued
by the Kubernetes cluster, and then present that certificate to the Kubernetes API.
Create private key
The following scripts show how to generate PKI private key and CSR. It is
important to set CN and O attribute of the CSR. CN is the name of the user and
O is the group that this user will belong to. You can refer to
RBAC
for standard groups.
openssl genrsa -out myuser.key
2048
openssl req -new -key myuser.key -out myuser.csr
Create a CertificateSigningRequest and submit it to a Kubernetes Cluster via kubectl. Below is a script to generate the CertificateSigningRequest.
cat
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
name: myuser
spec:
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZqQ0NBVDRDQVFBd0VURVBNQTBHQTFVRUF3d0dZVzVuWld4aE1JSUJJakFOQmdrcWhraUc5dzBCQVFFRgpBQU9DQVE4QU1JSUJDZ0tDQVFFQTByczhJTHRHdTYxakx2dHhWTTJSVlRWMDNHWlJTWWw0dWluVWo4RElaWjBOCnR2MUZtRVFSd3VoaUZsOFEzcWl0Qm0wMUFSMkNJVXBGd2ZzSjZ4MXF3ckJzVkhZbGlBNVhwRVpZM3ExcGswSDQKM3Z3aGJlK1o2MVNrVHF5SVBYUUwrTWM5T1Nsbm0xb0R2N0NtSkZNMUlMRVI3QTVGZnZKOEdFRjJ6dHBoaUlFMwpub1dtdHNZb3JuT2wzc2lHQ2ZGZzR4Zmd4eW8ybmlneFNVekl1bXNnVm9PM2ttT0x1RVF6cXpkakJ3TFJXbWlECklmMXBMWnoyalVnald4UkhCM1gyWnVVV1d1T09PZnpXM01LaE8ybHEvZi9DdS8wYk83c0x0MCt3U2ZMSU91TFcKcW90blZtRmxMMytqTy82WDNDKzBERHk5aUtwbXJjVDBnWGZLemE1dHJRSURBUUFCb0FBd0RRWUpLb1pJaHZjTgpBUUVMQlFBRGdnRUJBR05WdmVIOGR4ZzNvK21VeVRkbmFjVmQ1N24zSkExdnZEU1JWREkyQTZ1eXN3ZFp1L1BVCkkwZXpZWFV0RVNnSk1IRmQycVVNMjNuNVJsSXJ3R0xuUXFISUh5VStWWHhsdnZsRnpNOVpEWllSTmU3QlJvYXgKQVlEdUI5STZXT3FYbkFvczFqRmxNUG5NbFpqdU5kSGxpT1BjTU1oNndLaTZzZFhpVStHYTJ2RUVLY01jSVUyRgpvU2djUWdMYTk0aEpacGk3ZnNMdm1OQUxoT045UHdNMGM1dVJVejV4T0dGMUtCbWRSeEgvbUNOS2JKYjFRQm1HCkkwYitEUEdaTktXTU0xMzhIQXdoV0tkNjVoVHdYOWl4V3ZHMkh4TG1WQzg0L1BHT0tWQW9FNkpsYWFHdTlQVmkKdjlOSjVaZlZrcXdCd0hKbzZXdk9xVlA3SVFjZmg3d0drWm89Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
signerName: kubernetes.io/kube-apiserver-client
expirationSeconds: 86400  # one day
usages:
- client auth
EOF
Some points to note:
usages
has to be '
client auth
'
could be made longer (i.e.
864000
for ten days) or shorter (i.e.
3600
for one hour)
request
is the base64 encoded value of the CSR file content.
You can get the content using this command:
cat myuser.csr | base64 | tr -d "\n"
Approve certificate signing request
Use kubectl to create a CSR and approve it.
Get the list of CSRs:
Approve the CSR:
kubectl certificate approve myuser
Get the certificate
Retrieve the certificate from the CSR:
kubectl get csr/myuser -o yaml
The certificate value is in Base64-encoded format under
.
Export the issued certificate from the CertificateSigningRequest.
kubectl get csr myuser -o
jsonpath
=
'{.status.certificate}'
| base64 -d &gt; myuser.crt
Create Role and RoleBinding
With the certificate created it is time to define the Role and RoleBinding for
this user to access Kubernetes cluster resources.
This is a sample command to create a Role for this new user:
kubectl create role developer --verb
=
create --verb
=
get --verb
=
list --verb
=
update --verb
=
delete --resource
=
pods
This is a sample command to create a RoleBinding for this new user:
kubectl create rolebinding developer-binding-myuser --role
=
developer --user
=
myuser
Add to kubeconfig
The last step is to add this user into the kubeconfig file.
First, you need to add new credentials:
kubectl config set-credentials myuser --client-key
=
myuser.key --client-certificate
=
myuser.crt --

embed-certs
=
true
Then, you need to add the context:
kubectl config set-context myuser --cluster
=
kubernetes --user
=
myuser
To test it, change the context to
myuser
:
kubectl config use-context myuser
Approval or rejection
Control plane automated approval
The kube-controller-manager ships with a built-in approver for certificates with
a signerName of
that delegates various
permissions on CSRs for node credentials to authorization.
The kube-controller-manager POSTs SubjectAccessReview resources to the API server
in order to check authorization for certificate approval.
Approval or rejection using
kubectl
A Kubernetes administrator (with appropriate permissions) can manually approve
(or deny) CertificateSigningRequests by using the
and
kubectl certificate deny
commands.
To approve a CSR with kubectl:
kubectl certificate approve &lt;certificate-signing-request-name&gt;
Likewise, to deny a CSR:
kubectl certificate deny &lt;certificate-signing-request-name&gt;
Approval or rejection using the Kubernetes API
Users of the REST API can approve CSRs by submitting an UPDATE request to the
approval
subresource of the CSR to be approved. For example, you could write an
operator
that watches for a particular
kind of CSR and then sends an UPDATE to approve them.
When you make an approval or rejection request, set either the
Approved
or
Denied
status condition based on the state you determine:
For
Approved
CSRs:
:
kind
:
...
status
:
:
-
:
"2020-02-08T11:37:35Z"
:
message
:
Approved by my custom approver controller
reason
:
ApprovedByMyPolicy
# You can set this to any string
type
:
Approved
For
Denied
CSRs:
:
kind
:
...
status
:
:
-
:
:
message
:
Denied by my custom approver controller
reason
:
DeniedByMyPolicy
type
:
Denied
It's usual to set
status.conditions.reason
to a machine-friendly reason
code using TitleCase; this is a convention but you can set it to anything
you like. If you want to add a note for human consumption, use the
status.conditions.message
field.
Signing
Control plane signer
The Kubernetes control plane implements each of the
,
as part of the kube-controller-manager.
Note:
Prior to Kubernetes v1.18, the kube-controller-manager would sign any CSRs that
were marked as approved.
Note:
The
API-based signers
Users of the REST API can sign CSRs by submitting an UPDATE request to the
status
subresource of the CSR to be signed.
As part of this request, the
field should be set to contain the
signed certificate. This field contains one or more PEM-encoded certificates.
All PEM blocks must have the "CERTIFICATE" label, contain no headers,
and the encoded data must be a BER-encoded ASN.1 Certificate structure
section 4 of RFC5280
.
Example certificate content:
-----BEGIN CERTIFICATE-----
MIIDgjCCAmqgAwIBAgIUC1N1EJ4Qnsd322BhDPRwmg3b/oAwDQYJKoZIhvcNAQEL
BQAwXDELMAkGA1UEBhMCeHgxCjAIBgNVBAgMAXgxCjAIBgNVBAcMAXgxCjAIBgNV
BAoMAXgxCjAIBgNVBAsMAXgxCzAJBgNVBAMMAmNhMRAwDgYJKoZIhvcNAQkBFgF4
MB4XDTIwMDcwNjIyMDcwMFoXDTI1MDcwNTIyMDcwMFowNzEVMBMGA1UEChMMc3lz
dGVtOm5vZGVzMR4wHAYDVQQDExVzeXN0ZW06bm9kZToxMjcuMC4wLjEwggEiMA0G
CSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDne5X2eQ1JcLZkKvhzCR4Hxl9+ZmU3
+e1zfOywLdoQxrPi+o4hVsUH3q0y52BMa7u1yehHDRSaq9u62cmi5ekgXhXHzGmm
kmW5n0itRECv3SFsSm2DSghRKf0mm6iTYHWDHzUXKdm9lPPWoSOxoR5oqOsm3JEh
Q7Et13wrvTJqBMJo1GTwQuF+HYOku0NF/DLqbZIcpI08yQKyrBgYz2uO51/oNp8a
sTCsV4OUfyHhx2BBLUo4g4SptHFySTBwl

pRWBnSjZPOhmN74JcpTLB4J5f4iEeA7
2QytZfADckG4wVkhH3C2EJUmRtFIBVirwDn39GXkSGlnvnMgF3uLZ6zNAgMBAAGj
YTBfMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggrBgEFBQcDAjAMBgNVHRMB
Af8EAjAAMB0GA1UdDgQWBBTREl2hW54lkQBDeVCcd2f2VSlB1DALBgNVHREEBDAC
ggAwDQYJKoZIhvcNAQELBQADggEBABpZjuIKTq8pCaX8dMEGPWtAykgLsTcD2jYr
L0/TCrqmuaaliUa42jQTt2OVsVP/L8ofFunj/KjpQU0bvKJPLMRKtmxbhXuQCQi1
qCRkp8o93mHvEz3mTUN+D1cfQ2fpsBENLnpS0F4G/JyY2Vrh19/X8+mImMEK5eOy
o0BMby7byUj98WmcUvNCiXbC6F45QTmkwEhMqWns0JZQY+/XeDhEcg+lJvz9Eyo2
aGgPsye1o3DpyXnyfJWAWMhOz7cikS5X2adesbgI86PhEHBXPIJ1v13ZdfCExmdd
M1fLPhLyR54fGaY+7/X8P9AZzPefAkwizeXwe9ii6/a08vWoiE4=
-----END CERTIFICATE-----
Non-PEM content may appear before or after the CERTIFICATE PEM blocks and is unvalidated,
to allow for explanatory text as described in
section 5.2 of RFC7468
.
When encoded in JSON or YAML, this field is base-64 encoded.
A CertificateSigningRequest containing the example certificate above would look like this:
:
kind
:
...
status
:
:
"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JS..."
Read
View the source code for the kube-controller-manager built in
signer
approver
For details of X.509 itself, refer to
RFC 5280
section 3.1
For information on the syntax of PKCS#10 certificate signing requests, refer to
RFC 2986
3.4 - Admission Controllers Reference
An
,
mutating
not
to read (
get
,
watch
or
list
) objects.
list
both.
takes a comma-delimited list of admission control plugins to invoke prior to modifying objects in the cluster.
For exampl

e, the following command line enables the
and the
=
Note:
=
Note:
The
and
the
API.
because
Always
required.
approve
See
for more
sign
See
for more
of
of
.
Ingress
objects that do not re

quest any specific
Ingress
Ingress
See the
Ingress
See
the taints
and
and
or
.
and
Service
field
. This
on
existing
Service
Service
objects.
of it.
:
:
kind
:
plugins
:
-
name
:
path
:
...
Server
Namespace
User
:
kind
:
limits
:
-
type
:
Namespace
qps
:
50
burs

t
:
100
cacheSize
:
2000
-
type
:
User
qps
:
10
burst
:
50
See the
:
:
allowTTL
:
50
denyTTL
:
50
:
500
:
true
:
:
kind
:
plugins
:
-
name
:
path
:
...
:
kind
:
plugins
:
-
name
:
:
:
:
allowTTL
:
50
denyTTL
:
50
:
500
:
true
cluster
user
field
clusters
:
-
name
:
cluster
:
:
server
:
users
:
-
name
:
user
:
:
:
.
Note:
group (
).
{
:
,
"kind"
:
,
"spec"
:{
:[
{
"image"
:
},
{
"image"
:
}
],
:{
:
},
:
}
}
status
field of the request and
re

spond to either allow or disallow access. The response body's
spec
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
true
}
}
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"reason"
:
}
}
API
.
in
.
object in a
Namespace
Pods in the
default
See the
and the
must
Namespace
itself.
This admission cont

roller enforces that a
Namespace
Namespace
default
,
,
.
A
Namespace
Node
and
Pod
.
Node
Pod
Node
The
Node
or
k8s.io/
Prevents
prefix.
Node
Allows
or
k8s.io
delete
update
owner
to
true
.
:
kind
:
metadata
:
name
:
:
:
resturl
:
restuser
:
""
:
""
:
""
:
true
.
may need to add the labels manually to prevent pods from moun

ting volumes from
the
.
:
:
:
:
:
:
kind
:
plugins
:
-
name
:
path
:
...
to assign
:
v1
kind
:
Namespace
metadata
:
:
:
name
:
If the
Namespace
,
Note:
.
See the
scheduler.alpha.kubernetes.io/t

olerationsWhitelist
:
v1
kind
:
Namespace
metadata
:
name
:
:
:
:
Priority
object in a
Namespace
. If you are
using
See the
and the
.spec
,
See also
task.
,
See
.
objects.
The
or
taints
Nodes as
NotReady
and
and
must
webhooks or other validating admission controllers will permit the request to finish.
If you disable 

the ValidatingAdmissionWebhook, you must also disable the
flag.
(shown
here
),
flag (
).
3.5 - Dynamic Admission Control
compiled-in admission plugins
,
admission plugins can be developed as extensions and run as webhooks configured at runtime.
This page describes how to build, configure, use, and monitor admission webhooks.
What are admission webhooks?
Admission webhooks are HTTP callbacks that receive admission requests and do
something with them. You can define two types of admission webhooks,
validating admission webhook
and
mutating admission webhook
.
Mutating admission webhooks are invoked first, and can modify objects sent to the API server to enforce custom defaults.
After all object modifications are complete, and after the incoming object is validated by the API server,
validating admission webhooks are invoked and can reject requests to enforce custom policies.
Note:
Admission webhooks that need to guarantee they see the final state of the object in order to enforce policy
should use a validating admission webhook, since objects can be modified after being seen by mutating webhooks.
Experimenting with admission webhooks
Admission webhooks are essentially part of the cluster control-plane. You should
write and deploy them with great caution. Please read the
user guides
for instructions if you intend to write/deploy production-grade admission webhooks.
In the following, we describe how to quickly experiment with admission webhooks.
Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook
admission controllers are enabled.
Here
is a recommended set of admission controllers to enable in general.
Ensure that the
API is enabled.
Write an admission webhook server
admission webhook server
AdmissionReview
request sent by the API servers, and sends back its decision
as an
object in the same version it received.
See the
webhook request
section for details on the data sent to webhooks.
See the
webhook response
section for the data expected from webhooks.
The example admission webhook server leaves the
field
empty
,
how to
.
Deploy the admission webhook service
The webhook server in the e2e test is deployed in the Kubernetes cluster, via
the
deployment API
.
The test also creates a
service
as the front-end of the webhook server. See
code
.
You may also deploy your webhooks outside of the cluster. You will need to update
your webhook configurations accordingly.
Configure admission webhooks on the fly
You can dynamically configure what resources are subject to what admission
webhooks via
or
.
The following is an example
, a mutating webhook configuration is similar.
See the
webhook configuration
section for details about each config field.
:
kind
:
metadata
:
name
:
"pod-policy.example.com"
webhooks
:
-
name
:
rules
:
-
apiGroups
:
[
""
]
apiVersions
:
[
"v1"
]
:
[
"CREATE"
]
resources
:
[
"pods"
]
scope
:
"Namespaced"
:
service
:
namespace
:
"example-namespace"
name
:
"e

xample-service"
caBundle
:
&lt;CA_BUNDLE&gt;
admissionReviewVersions
:
[
"v1"
]
:
None
:
5
Note:
You must replace the
in the above example by a valid CA bundle
which is a PEM-encoded CA bundle for validating the webhook's server certificate.
The
scope
field specifies if only cluster-scoped resources ("Cluster") or namespace-scoped
resources ("Namespaced") will match this rule. "∗" means that there are no scope restrictions.
Note:
When using
clientConfig.service
, the server cert must be valid for
&lt;svc_name&gt;.&lt;svc_namespace&gt;.svc
.
Note:
Default timeout for a webhook call is 10 seconds,
You can set the
timeout
and it is encouraged to use a short timeout for webhooks.
If the webhook call times out, the request is handled according to the webhook's
failure policy.
When an API server receives a request that matches one of the
rules
, the
API server sends an
admissionReview
request to webhook as specified in the
.
After you create the webhook configuration, the system will take a few seconds
to honor the new configuration.
Authenticate API servers
If your admission webhooks require authentication, you can configure the
API servers to use basic auth, bearer token, or a cert to authenticate itself to
the webhooks. There are three steps to complete the configuration.
When starting the API server, specify the location of the admission control
configuration file via the
flag.
In the admission control configuration file, specify where the
MutatingAdmissionWebhook controller and ValidatingAdmissionWebhook controller
should read the credentials. The credentials are stored in kubeConfig files
(yes, the same schema that's used by kubectl), so the field name is
. Here is an example admission control configuration file:
:
kind
:
plugins
:
-
name
:
:
:
kind
:
WebhookAdmissionConfiguration
:
"&lt;path-to-kubeconfig-file&gt;"
-
name
:
:
:
kind
:
:
# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1
:
kind
:
plugins
:
-
name
:
:
# Deprecated in v1.17 in favor of apiserver.config.k8s.io/v1, kind=WebhookAdmissionConfiguration
:
kind
:
:
-
name
:
:
:
kind
:
:
For more information about
, see the
AdmissionConfiguration (v1) reference
.
See the
In the kubeConfig file, provide the credentials:
:
v1
kind
:
Config
users
:
# name should be set to the DNS name of the service or the host (including port) of the URL the webhook is configured to speak to.
# If a non-443 port is used for services, it must be included in the name when configuring 1.16+ API servers.
#
# For a webhook configured to speak to a service on the default port (443), specify the DNS name of the service:
# - name: webhook1.ns1.svc
#   user: ...
#
# For a webhook configured to speak to a service on non-default port (e.g. 8443), specify the DNS name and port of the service in 1.16+:
# - name: webhook1.ns1.svc:8443
#   user: ...
# and optionally create a second stanza using only the DNS name of the service for compatibility with 1.15 API servers:
# - name: webhook1.

ns1.svc
#   user: ...
#
# For webhooks configured to speak to a URL, match the host (and port) specified in the webhook's URL. Examples:
# A webhook with `url: https://www.example.com`:
# - name: www.example.com
#   user: ...
#
# A webhook with `url: https://www.example.com:443`:
# - name: www.example.com:443
#   user: ...
#
# A webhook with `url: https://www.example.com:8443`:
# - name: www.example.com:8443
#   user: ...
#
-
name
:
'webhook1.ns1.svc'
user
:
:
"&lt;pem encoded certificate&gt;"
:
"&lt;pem encoded key&gt;"
# The `name` supports using * to wildcard-match prefixing segments.
-
name
:
'*.webhook-company.org'
user
:
password
:
"&lt;password&gt;"
username
:
"&lt;name&gt;"
# '*' is the default match.
-
name
:
'*'
user
:
token
:
"&lt;token&gt;"
Of course you need to set up the webhook server to handle these authentication requests.
Request
Webhooks are sent as POST requests, with
,
with an
admission.k8s.io
API group
with the
field in their configuration:
:
kind
:
webhooks
:
-
name
:
my-webhook.example.com
:
[
"v1"
,
"v1beta1"
]
is a required field when creating webhook configurations.
If none of the versions in the list are supported by the API server, the configuration will not be allowed to be created.
If an API server encounters a webhook configuration that was previously created and does not support any of the
versions the API server knows how to send, attempts to call to the webhook will fail and be subject to the
failure policy
.
object
for a request to update the
scale
subresource of an
apps/v1
:
:
admission.k8s.io/v1
kind
:
request
:
# Random uid uniquely identifying this admission call
uid
:
705ab4f5-6393-11e8-b7cc-42010a800002
# Fully-qualified group/version/kind of the incoming object
kind
:
group
:
autoscaling
version
:
v1
kind
:
Scale
# Fully-qualified group/version/kind of the resource being modified
resource
:
group
:
apps
version
:
v1
resource
:
# subresource, if the request is to a subresource
subResource
:
scale
# Fully-qualified group/version/kind of the incoming object in the original request to the API server.
# This only differs from `kind` if the webhook specified `matchPolicy: Equivalent` and the
# original request to the API server was converted to a version the webhook registered for.
requestKind
:
group
:
version
:
v1
kind
:
Scale
# Fully-qualified group/version/kind of the resource being modified in the original request to the API server.
# This only differs from `resource` if the webhook specified `matchPolicy: Equivalent` and the
requestResource
:
group
:
apps
version
:
v1
resource
:
# This only differs from `subResource` if the webhook specified `matchPolicy: Equivalent` and the
requestSubResource
:
scale
# Name of the resource being modified
name
:
my-deployment
# Namespace of the resource being modified, if the resource is namespaced (or is a Namespace object)
namespace
:
# operation can be CREATE, UPDATE, DELETE, or CONNECT
operation
:
UPDATE
userInfo
:
# Username of the authenticated user making the

 request to the API server
username
:
admin
# UID of the authenticated user making the request to the API server
uid
:
014fbff9a07c
# Group memberships of the authenticated user making the request to the API server
groups
:
- my-admin-group
# Arbitrary extra info associated with the user making the request to the API server.
# This is populated by the API server authentication layer and should be included
# if any SubjectAccessReview checks are performed by the webhook.
extra
:
some-key
:
- some-value1
- some-value2
# object is the new object being admitted.
# It is null for DELETE operations.
object
:
:
kind
:
Scale
# oldObject is the existing object.
# It is null for CREATE and CONNECT operations.
oldObject
:
:
kind
:
Scale
# options contains the options for the operation being admitted, like meta.k8s.io/v1 CreateOptions, UpdateOptions, or DeleteOptions.
# It is null for CONNECT operations.
options
:
:
meta.k8s.io/v1
kind
:
UpdateOptions
# dryRun indicates the API request is running in dry run mode and will not be persisted.
# Webhooks with side effects should avoid actuating those side effects when dryRun is true.
# See http://k8s.io/docs/reference/using-api/api-concepts/#make-a-dry-run-request for more details.
dryRun
:
False
Response
,
and a body containing an
with the
response
At a minimum, the
response
stanza must contain the following fields:
uid
allowed
, either set to
true
or
false
Example of a minimal response from a webhook to allow a request:
{
:
"admission.k8s.io/v1"
,
"kind"
:
"AdmissionReview"
,
"response"
: {
"uid"
:
,
"allowed"
:
true
}
}
Example of a minimal response from a webhook to forbid a request:
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
false
}
}
When rejecting a request, the webhook can customize the http code and message returned to the user
using the
status
field. The specified status object is returned to the user.
See the
API documentation
for details about the
status
type.
Example of a response to forbid a request, customizing the HTTP status code and message presented to the user:
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
false
,
"status"
: {
"code"
:
403
,
"message"
:
"You cannot do this because it is Tuesday and your name starts with A"
}
}
}
When allowing a request, a mutating admission webhook may optionally modify the incoming object as well.
This is done using the
patch
and
patchType
fields in the response.
The only currently supported
patchType
is
JSONPatch
.
See
JSON patch
For
patchType: JSONPatch
, the
patch
field contains a base64-encoded array of JSON patch operations.
As an example, a single patch operation that would set
would be
[{"op": "add", "path": "/spec/replicas", "value": 3}]
Base64-encoded, this would be
W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0=
So a webhook response to add that label would be:
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
,
"patchType"
:
"JSONPatch"
,
"patch"
:
"W3sib3AiOiAiYWRkIiwgInBhdGgiOiAiL3NwZWMvcmVwbGljYXMiLCAidmFsdWUiOiAzfV0="
}
}
Admission webhooks can optionally return warning messages that are returned to the requesting client
in HTTP
Warning
headers with a warning code of 299. Warnings can be sent with allowed or rejected admission responses.
If you're implementing a webhook that returns a warning:
Don't include a "Warning:" pre

fix in the message
Use warning messages to describe problems the client making the API request should correct or be aware of
Limit warnings to 120 characters if possible
Caution:
Individual warning messages over 256 characters may be truncated by the API server before being returned to clients.
If more than 4096 characters of warning messages are added (from all sources), additional warning messages are ignored.
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
,
"warnings"
: [
"duplicate envvar entries specified with name MY_ENV"
,
"memory request less than 4MB specified for container mycontainer, which will not start successfully"
]
}
}
Webhook configuration
To register admission webhooks, create
or
API objects.
The name of a
or a
object must be a valid
.
Each configuration can contain one or more webhooks.
If multiple webhooks are specified in a single configuration, each must be given a unique name.
This is required in order to make resulting audit logs and metrics easier to match up to active
configurations.
Each webhook defines the following things.
Matching requests: rules
Each webhook must specify a list of rules used to determine if a request to the API server should be sent to the webhook.
Each rule specifies one or more operations, apiGroups, apiVersions, and resources, and a resource scope:
lists one or more operations to match. Can be
"CREATE"
,
"UPDATE"
,
"DELETE"
,
"CONNECT"
,
or
"*"
to match all.
apiGroups
lists one or more API groups to match.
""
is the core API group.
"*"
matches all API groups.
lists one or more API versions to match.
"*"
matches all API versions.
resources
lists one or more resources to match.
"*"
matches all resources, but not subresources.
"*/*"
matches all resources and subresources.
"pods/*"
matches all subresources of pods.
"*/status"
matches all status subresources.
scope
specifies a scope to match. Valid values are
"Cluster"
,
, and
"*"
.
Subresources match the scope of their parent resource. Default is
"*"
.
"Cluster"
means that only cluster-scoped resources will match this rule (Namespace API objects are cluster-scoped).
means that only namespaced resources will match this rule.
"*"
means that there are no scope restrictions.
If an incoming request matches one of the specified
,
groups
,
versions
,
resources
, and
scope
for any of a webhook's
rules
, the request is sent to the webhook.
Here are other examples of rules that could be used to specify which resources should be intercepted.
Match
CREATE
or
UPDATE
requests to
apps/v1
and
and
:
:
kind
:
...
webhooks
:
-
name
:
rules
:
-
:
[
"CREATE"
,
"UPDATE"
]
apiGroups
:
[
"apps"
]
:
[
"v1"
,
"v1beta1"
]
resources
:
[
,
"replicasets"
]
scope
:
...
Match create requests for all resources (but not subresources) in all API groups and versions:
:
kind
:
webhooks
:
-
name
:
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
"*"
Match update requests for all
status
subresources in all API groups and versions:
:
kind
:
webhooks
:
-
name
:
rules
:
-
:
[
"UPDATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
]
scope
:
"*"
Matching requests: objectSelector
Webhooks may optionally limit which requests are intercepted based on the labels of the
objects they would be sent, by specifying an
objectSelect

or
. If specified, the objectSelector
is evaluated against both the object and oldObject that would be sent to the webhook,
and is considered to match if either object matches the selector.
A null object (
oldObject
in the case of create, or
newObject
in the case of delete),
or an object that cannot have labels (like a
DeploymentRollback
or a
PodProxyOptions
object)
is not considered to match.
Use the object selector only if the webhook is opt-in, because end users may skip
the admission webhook by setting the labels.
This example shows a mutating webhook that would match a
CREATE
of any resource with the label
foo: bar
:
:
kind
:
webhooks
:
-
name
:
objectSelector
:
:
foo
:
bar
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
"*"
See
labels concept
for more examples of label selectors.
Matching requests: namespaceSelector
Webhooks may optionally limit which requests for namespaced resources are intercepted,
based on the labels of the containing namespace, by specifying a
namespaceSelector
.
The
decides whether to run the webhook on a request for a namespaced resource
(or a Namespace object), based on whether the namespace's labels match the selector.
If the object itself is a namespace, the matching is performed on object.metadata.labels.
If the object is a cluster scoped resource other than a Namespace,
has no effect.
This example shows a mutating webhook that matches a
CREATE
of any namespaced resource inside a namespace
that does not have a "runlevel" label of "0" or "1":
:
kind
:
webhooks
:
-
name
:
:
:
-
key
:
runlevel
operator
:
NotIn
values
:
[
"0"
,
"1"
]
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
This example shows a validating webhook that matches a
CREATE
of any namespaced resource inside
a namespace that is associated with the "environment" of "prod" or "staging":
:
kind
:
webhooks
:
-
name
:
:
:
-
key
:
operator
:
In
values
:
[
"prod"
,
"staging"
]
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
See
Matching requests: matchPolicy
API servers can make objects available via multiple API groups or versions.
For example, if a webhook only specified a rule for some API groups/versions
(like
apiGroups:["apps"], apiVersions:["v1","v1beta1"]
),
and a request was made to modify the resource via another API group/version (like
),
the request would not be sent to the webhook.
The
matchPolicy
lets a webhook define how its
rules
are used to match incoming requests.
Allowed values are
Exact
or
.
Exact
means a request should be intercepted only if it exactly matches a specified rule.
means a request should be intercepted if modifies a resource listed in
rules
,
even via another API group or version.
In the example given above, the webhook that only registered for
apps/v1
could use
:
matchPolicy: Exact
would mean the
request would not be sent to the webhook
matchPolicy: Equivalent
means the
request would be sent to the webhook
(with the objects converted to a version the webhook had specified:
apps/v1
)
Specifying
is recommended, and ensures that webhooks continue to intercept the
resources they expect when upgrades enable new versions of the resource in the API server.
When a resource stops being served by the API server, it is no longer considered equivalent to
othe

r versions of that resource that are still served.
deployments were first deprecated and then removed (in Kubernetes v1.16).
Since that removal, a webhook with a
apiGroups:["extensions"], apiVersions:["v1beta1"], resources:["deployments"]
rule
does not intercept deployments created via
apps/v1
APIs. For that reason, webhooks should prefer registering
for stable versions of resources.
This example shows a validating webhook that intercepts modifications to deployments (no matter the API group or version),
and is always sent an
apps/v1
object:
:
kind
:
webhooks
:
-
name
:
:
rules
:
-
:
[
"CREATE"
,
"UPDATE"
,
"DELETE"
]
apiGroups
:
[
"apps"
]
:
[
"v1"
]
resources
:
[
]
scope
:
The
for an admission webhooks defaults to
.
Once the API server has determined a request should be sent to a webhook,
Webhooks can either be called via a URL or a service reference,
URL
url
(
).
The
host
service
The host might be resolved via external DNS in some API servers
(e.g.,
host
localhost
or
127.0.0.1
as a
host
is
which run an API server which might need to make calls to this
webhook. Such installations are likely to be non-portable or not readily
run in a new cluster.
Attempting to use a user or basic auth (for example
user:password@
) is not allowed.
Fragments (
#...
) and query parameters (
?...
) are also not allowed.
Here is an example of a mutating webhook configured to call a URL
:
kind
:
webhooks
:
-
name
:
:
url
:
Service reference
The
service
is a reference to the service for this webhook.
service
instead of
url
.
Here is an example of a mutating webhook configured to call a service on port "1234"
using a custom CA bundle:
:
kind
:
webhooks
:
-
name
:
:
caBundle
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
Note:
Side effects
Webhooks typically operate only on the content of the
sent to them.
Some webhooks, however, make out-of-band changes as part of processing admission requests.
Webhooks that make out-of-band changes ("side effects") must also have a reconciliation mechanism
(like a controller) that periodically determines the actual state of the world, and adjusts
the out-of-band dat

a modified by the admission webhook to reflect reality.
This is because a call to an admission webhook does not guarantee the admitted object will be persisted as is, or at all.
Later webhooks can modify the content of the object, a conflict could be encountered while writing to storage,
or the server could power off before persisting the object.
Additionally, webhooks with side effects must skip those side-effects when
dryRun: true
admission requests are handled.
A webhook must explicitly indicate that it will not have side-effects when run with
dryRun
,
or the dry-run request will not be sent to the webhook and the API request will fail instead.
Webhooks indicate whether they have side effects using the
field in the webhook configuration:
None
: calling the webhook will have no side effects.
: calling the webhook will possibly have side effects, but if a request with
is sent to the webhook, the webhook will suppress the side effects (the webhook
is
dryRun
-aware).
Here is an example of a validating webhook indicating it has no side effects on
requests:
:
kind
:
webhooks
:
-
name
:
:
Timeouts
Because webhooks add to API request latency, they should evaluate as quickly as possible.
allows configuring how long the API server should wait for a webhook to respond
before treating the call as a failure.
If the timeout expires before the webhook responds, the webhook call will be ignored or
the API call will be rejected based on the
.
The timeout value must be between 1 and 30 seconds.
Here is an example of a validating webhook with a custom timeout of 2 seconds:
:
kind
:
webhooks
:
-
name
:
:
2
The timeout for an admission webhook defaults to 10 seconds.
Reinvocation policy
A single ordering of mutating admissions plugins (including webhooks) does not work for all cases
(see
https://issue.k8s.io/64333
as an example). A mutating webhook can add a new sub-structure
to the object (like adding a
container
to a
pod
), and other mutating plugins which have already
run may have opinions on those new structures (like setting an
on all containers).
To allow mutating admission plugins to observe changes made by other plugins,
built-in mutating admission plugins are re-run if a mutating webhook modifies an object,
and mutating webhooks can specify a
reinvocationPolicy
to control whether they are reinvoked as well.
may be set to
Never
or
IfNeeded
. It defaults to
Never
.
Never
: the webhook must not be called more than once in a single admission evaluation.
IfNeeded
: the webhook may be called again as part of the admission evaluation if the object
being admitted is modified by other admission plugins after the initial webhook call.
The important elements to note are:
The number of additional invocations is not guaranteed to be exactly one.
If additional invocations result in further modifications to the object, webhooks are not
guaranteed to be invoked again.
Webhooks that use this option may be reordered to minimize the number of additional invocations.
To validate an object after all mutations are guaranteed complete, use a validating admission
webhook instead (recommended for webhooks with side-effects).
Here is an example of a mutating webhook opting into being re-invoked if later admission plugins
modify the object:
:
kind
:
webhooks
:
-
name
:
:
IfNeeded
Mutating webhooks must be
idempotent
, able to successfully process an object they have already admitted
and potentially modified. This is true for all mutating admission webhooks, since any change they can make
in an object could 

already exist in the user-provided object, but it is essential for webhooks that opt into reinvocation.
Failure policy
failurePolicy
defines how unrecognized errors and timeout errors from the admission webhook
are handled. Allowed values are
Ignore
or
Fail
.
Ignore
means that an error calling the webhook is ignored and the API request is allowed to continue.
Fail
means that an error calling the webhook causes the admission to fail and the API request to be rejected.
Here is a mutating webhook configured to reject an API request if errors are encountered calling the admission webhook:
:
kind
:
webhooks
:
-
name
:
:
Fail
for an admission webhooks is
Fail
.
Monitoring admission webhooks
The API server provides ways to monitor admission webhook behaviors. These
monitoring mechanisms help cluster admins to answer questions like:
Which mutating webhook mutated the object in a API request?
What change did the mutating webhook applied to the object?
Which webhooks are frequently rejecting API requests? What's the reason for a rejection?
Sometimes it's useful to know which mutating webhook mutated the object in a API request, and what change did the
webhook apply.
The Kubernetes API server performs
auditing
on each
mutating webhook invocation. Each invocation generates an auditing annotation
capturing if a request object is mutated by the invocation, and optionally generates an annotation
capturing the applied patch from the webhook admission response. The annotations are set in the
audit event for given request on given stage of its execution, which is then pre-processed
according to a certain policy and written to a backend.
The audit level of a event determines which annotations get recorded:
At
Metadata
audit level or higher, an annotation with key
mutation.webhook.admission.k8s.io/round_{round idx}_index_{order idx}
gets logged with JSON
payload indicating a webhook gets invoked for given request and whether it mutated the object or not.
For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is
ordered the third in the mutating webhook chain, and didn't mutated the request object during the
invocation.
# the audit event recorded
{
"kind":
"Event"
,
"audit.k8s.io/v1"
,
"annotations":
{
"mutation.webhook.admission.k8s.io/round_1_index_2": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook.example.com\",\"mutated\":
false
}"
# other annotations
...
}
# other fields
...
}
# the annotation value deserialized
{
"configuration":
"my-mutating-webhook-configuration.example.com"
,
"webhook":
"my-webhook.example.com"
,
"mutated":
false
}
The following annotation gets recorded for a webhook being invoked in the first round. The webhook
is ordered the first in the mutating webhook chain, and mutated the request object during the
{
"kind":
"Event"
,
,
{
"mutation.webhook.admission.k8s.io/round_0_index_0": "{\"configuration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"mutated\":
true
}"
...
}
...
}
{
,
"my-webhook-always-mutate.example.com"
,
true
}
At
Request
patch.webhook.admission.k8s.io/round_{round idx}_index_{order idx}
gets logged with JSON payload indicating
a webhook gets invoked for given request and what patch gets applied to the request object.
For example, the following annotation gets recorded for a webhook being reinvoked. The webhook is ordered the fourth in the
mutating webhook chain, and responded with a JSON patch 

which got applied to the request object.
{
"kind":
"Event"
,
,
{
"patch.webhook.admission.k8s.io/round_1_index_3":
"{\"configuration\":\"my-other-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"patch\":[{\"op\":\"add\",\"path\":\"/data/mutation-stage\",\"value\":\"yes\"}],\"patchType\":\"JSONPatch\"}"
...
}
...
}
{
"my-other-mutating-webhook-configuration.example.com"
,
,
"patchType":
,
"patch":
[
{
"op":
"add"
,
"path":
"/data/mutation-stage"
,
"value":
"yes"
}
]
}
Admission webhook metrics
The API server exposes Prometheus metrics from the
/metrics
endpoint, which can be used for monitoring and
diagnosing API server status. The following metrics record status related to admission webhooks.
API server admission webhook rejection count
Sometimes it's useful to know which admission webhooks are frequently rejecting API requests, and the
reason for a rejection.
The API server exposes a Prometheus counter metric recording admission webhook rejections. The
metrics are labelled to identify the causes of webhook rejection(s):
name
: the name of the webhook that rejected a request.
operation
: the operation type of the request, can be one of
CREATE
,
UPDATE
,
DELETE
and
CONNECT
.
type
: the admission webhook type, can be one of
admit
and
.
error_type
: identifies if an error occurred during the webhook invocation
that caused the rejection. Its value can be one of:
calling_webhook_error
: unrecognized errors or timeout errors from the admission webhook happened and the
webhook's
is set to
Fail
.
no_error
: no error occurred. The webhook rejected the request with
allowed: false
in the admission
response. The metrics label
rejection_code
records the
.status.code
set in the admission response.
apiserver_internal_error
: an API server internal error happened.
: the HTTP status code set in the admission response when a
webhook rejected a request.
Example of the rejection count metrics:
# HELP apiserver_admission_webhook_rejection_count [ALPHA] Admission webhook rejection count, identified by name and broken out for each admission type (validating or admit) and operation. Additional labels specify an error type (calling_webhook_error or apiserver_internal_error if an error occurred; no_error otherwise) and optionally a non-zero rejection code if the webhook rejects the request with an HTTP status code (honored by the apiserver when the code is greater or equal to 400). Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
# TYPE apiserver_admission_webhook_rejection_count counter
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="always-timeout-webhook.example.com",operation="CREATE",rejection_code="0",type="validating"} 1
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="invalid-admission-response-webhook.example.com",operation="CREATE",rejection_code="0",type="validating"} 1
apiserver_admission_webhook_rejection_count{error_type="no_error",name="deny-unwanted-configmap-data.example.com",operation="CREATE",rejection_code="400",type="validating"} 13
Best practices and warnings
Idempotence
An idempotent mutating admission webhook is able to successfully process an object it has already admitted
and potentially modified. The admission can be applied multiple times without changing the result beyond
the initial application.
Example of idempotent mutating admission webhooks:
For a
CREATE
pod request, set the field
.spec.securityContext.runAsNonRoot
of the
pod to true, to enforce security best practices.
For a
CREATE
pod request, if the field
.spec.containers[].resources.limits
of a container is not set, set default resource limits.
Fo

r a
CREATE
pod request, inject a sidecar container with name
foo-sidecar
if no container
with the name
already exists.
In the cases above, the webhook can be safely reinvoked, or admit an object that already has the fields set.
Example of non-idempotent mutating admission webhooks:
For a
CREATE
suffixed with the current timestamp (e.g.
foo-sidecar-19700101-000000
).
For a
CREATE
/
UPDATE
pod request, reject if the pod has label
"env"
set,
otherwise add an
"env": "prod"
label to the pod.
For a
CREATE
pod request, blindly append a sidecar container named
without looking to see if there is already a
container in the pod.
In the first case above, reinvoking the webhook can result in the same sidecar being injected multiple times to a pod, each time
with a different container name. Similarly the webhook can inject duplicated containers if the sidecar already exists in
a user-provided pod.
In the second case above, reinvoking the webhook will result in the webhook failing on its own output.
In the third case above, reinvoking the webhook will result in duplicated containers in the pod spec, which makes
the request invalid and rejected by the API server.
Intercepting all versions of an object
It is recommended that admission webhooks should always intercept all versions of an object by setting
.webhooks[].matchPolicy
to
. It is also recommended that admission webhooks should prefer registering for stable versions of resources.
Failure to intercept all versions of an object can result in admission policies not being enforced for requests in certain
versions. See
for examples.
Availability
It is recommended that admission webhooks should evaluate as quickly as possible (typically in
milliseconds), since they add to API request latency.
It is encouraged to use a small timeout for webhooks. See
Timeouts
It is recommended that admission webhooks should leverage some format of load-balancing, to
provide high availability and performance benefits. If a webhook is running within the cluster,
you can run multiple webhook backends behind a service to leverage the load-balancing that service
supports.
Guaranteeing the final state of the object is seen
For example, a mutating admission webhook is configured to inject a sidecar container with name
"foo-sidecar" on every
CREATE
pod request. If the sidecar
must
be present, a validating
admisson webhook should also be configured to intercept
CREATE
pod requests, and validate that a
container with name "foo-sidecar" with the expected configuration exists in the to-be-created
object.
Avoiding deadlocks in self-hosted webhooks
A webhook running inside the cluster might cause deadlocks for its own deployment if it is configured
to intercept resources required to start its own pods.
For example, a mutating admission webhook is configured to admit
CREATE
pod requests only if a certain label is set in the
pod (e.g.
). The webhook server runs in a deployment which doesn't set the
"env"
label.
When a node that runs the webhook server pods
becomes unhealthy, the webhook deployment will try to reschedule the pods to another node. However the requests will
get rejected by the existing webhook server since the
"env"
label is unset, and the migration cannot happen.
It is recommended to exclude the namespace where your webhook is running with a
.
It is recommended that admission webhooks should avoid side effects if possible, which means the webhooks operate only on the
content of the
sent to them, and do not make out-of-band changes. The
.webhooks[].sideEffects
field should
be set to

None
if a webhook doesn't have any side effect.
If side effects are required during the admission evaluation, they must be suppressed when processing an
object with
dryRun
set to
true
, and the
set to
. See
Avoiding operating on the kube-system namespace
The
namespace contains objects created by the Kubernetes system,
e.g. service accounts for the control plane components, pods like
kube-dns
.
Accidentally mutating or rejecting requests in the
namespace may
cause the control plane components to stop functioning or introduce unknown behavior.
If your admission webhooks don't intend to modify the behavior of the Kubernetes control
plane, exclude the
namespace from being intercepted using a
.
3.6 - Managing Service Accounts
A
.
minikube
examplens
.
)
adds a
to Pods,
...
-
name
:
projected
:
sources
:
-
:
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath

:
path
:
namespace
A
A
configMap
A
Note:
are
using the
API,
Note:
using
.
the
using the
using the
called an
.
default
.
The admission controller ensures that the ServiceAccount ref

erenced by the incoming Pod exists. If there
default
Pod's
false
:
volume
.
;
, then the
.
You use the
.
kubectl
, see
.
...
-
name
:
projected
:
:
420
sources
:
-
:
:
3607
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
A
A
configMap
A
Caution:
mechanism
expire, they represent a lower risk to information security.
:
v1
kind
:
Secret
type
:
metadata
:
name
:
:
:
To create a Secr

et based on this example, run:

Data
====
token:          ...
examplens
:
v1
kind
:
metadata
:
:
:
|
{
:
"v1"
,
"kind"
:
,
:{
:{},
"name"
:
,
:
}}
:
name
:
namespace
:
examplens
:
"777"
selfLink
:
uid
:
secrets
:
-
name
:
:
v1
kind
:
metadata
:
:
:
|
{
:
"v1"
,
"kind"
:
,
:{
:{},
"name"
:
,
:
}}
:
name
:
namespace
:
examplens
:
"1026"
selfLink
:
uid
:
secrets
:
-
name
:
Clean up
examplens
.
the
using the
using the
.
3.7 - Authorization Overview
Learn more about Kubernetes authorization, i

ncluding details about creating
policies using the supported authorization modules.
In Kubernetes, you must be authenticated (logged in) before your request can be
authorized (granted permission to access). For information about authentication,
see
.
Kubernetes expects attributes that are common to REST API requests. This means
that Kubernetes authorization works with existing organization-wide or
cloud-provider-wide access control systems which may handle other APIs besides
the Kubernetes API.
Determine Whether a Request is Allowed or Denied
Kubernetes authorizes API requests using the API server. It evaluates all of the
request attributes against all policies and allows or denies the request. All
parts of an API request must be allowed by some policy in order to proceed. This
means that permissions are denied by default.
(Although Kubernetes uses the API server, access controls and policies that
depend on specific fields of specific kinds of objects are handled by Admission
Controllers.)
When multiple authorization modules are configured, each is checked in sequence.
If any authorizer approves or denies a request, that decision is immediately
returned and no other authorizer is consulted. If all modules have no opinion on
the request, then the request is denied. A deny returns an HTTP status code 403.
Review Your Request Attributes
Kubernetes reviews only the following API request attributes:
user
- The
user
string provided during authentication.
group
- The list of group names to which the authenticated user belongs.
extra
- A map of arbitrary string keys to string values, provided by the authentication layer.
API
- Indicates whether the request is for an API resource.
Request path
- Path to miscellaneous non-resource endpoints like
/api
or
/healthz
.
API request verb
- API verbs like
get
,
list
,
create
,
update
,
patch
,
watch
,
delete
, and
are used for resource requests. To determine the request verb for a resource API endpoint, see
Determine the request verb
.
HTTP request verb
- Lowercased HTTP methods like
get
,
post
,
put
, and
delete
are used for non-resource requests.
Resource
- The ID or name of the resource that is being accessed (for resource requests only) -- For resource requests using
get
,
update
,
patch
, and
delete
verbs, you must provide the resource name.
Subresource
- The subresource that is being accessed (for resource requests only).
Namespace
- The namespace of the object that is being accessed (for namespaced resource requests only).
API group
- The
API Group
being accessed (for resource requests only). An empty string designates the
core
API group
.
Determine the Request Verb
Non-resource requests
Requests to endpoints other than
/api/v1/...
or
/apis/&lt;group&gt;/&lt;version&gt;/...
are considered "non-resource requests", and use the lower-cased HTTP method of the request as the verb.
For example, a
GET
request to endpoints like
/api
or
/healthz
would use
get
as the verb.
Resource requests
To determine the request verb for a resource API endpoint, review the HTTP verb
used and whether or not the request acts on an individual resource or a
collection of resources:
HTTP verb
POST
create
GET, HEAD
get (for individual resources), list (for collections, including full object content), watch (for watching an individual resource or collection of resources)
PUT
update
PATCH
patch
DELETE
delete (for individual resources), deletecollection (for collections)
Caution:
The
get
,
list
and
watch
verbs can all return the full details of a resource. In terms of the returned data they are equivalent. For example,
list
on
secrets
will still reveal the
data
attributes of any returned resources.
Kubernetes sometimes checks authorization for additional permissions using specialized verbs. For example:
RBAC
bind
and
escalate
verbs on
roles
and
resources in the
API group.
Authentication
impersonate
verb on
users

,
groups
, and
in the core API group, and the
userextras
in the
authentication.k8s.io
API group.
Authorization Modes
The Kubernetes API server may authorize a request using one of several authorization modes:
Node
- A special-purpose authorization mode that grants permissions to kubelets based on the pods they are scheduled to run. To learn more about using the Node authorization mode, see
.
ABAC
- Attribute-based access control (ABAC) defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together. The policies can use any type of attributes (user attributes, resource attributes, object, environment attributes, etc). To learn more about using the ABAC mode, see
ABAC Mode
.
RBAC
- Role-based access control (RBAC) is a method of regulating access to computer or network resources based on the roles of individual users within an enterprise. In this context, access is the ability of an individual user to perform a specific task, such as view, create, or modify a file. To learn more about using the RBAC mode, see
RBAC Mode
When specified RBAC (Role-Based Access Control) uses the
API group to drive authorization decisions, allowing admins to dynamically configure permission policies through the Kubernetes API.
To enable RBAC, start the apiserver with
--authorization-mode=RBAC
.
Webhook
- A WebHook is an HTTP callback: an HTTP POST that occurs when something happens; a simple event-notification via HTTP POST. A web application implementing WebHooks will POST a message to a URL when certain things happen. To learn more about using the Webhook mode, see
.
Checking API Access
kubectl
provides the
auth can-i
subcommand for quickly querying the API authorization layer.
The command uses the
API to determine if the current user can perform
a given action, and works regardless of the authorization mode used.
kubectl auth can-i create deployments --namespace dev
yes
kubectl auth can-i create deployments --namespace prod
no
Administrators can combine this with
user impersonation
to determine what action other users can perform.
kubectl auth can-i list secrets --namespace dev --as dave
no
Similarly, to check whether a ServiceAccount named
dev-sa
in Namespace
dev
can list Pods in the Namespace
target
:
kubectl auth can-i list pods
\
--namespace target
\
--as system:serviceaccount:dev:dev-sa
yes
is part of the
authorization.k8s.io
API group, which
exposes the API server authorization to external services. Other resources in
this group include:
- Access review for any user, not only the current one. Useful for delegating authorization decisions to the API server. For example, the kubelet and extension API servers use this to determine user access to their own APIs.
- Like
but restricted to a specific namespace.
- A review which returns the set of actions a user can perform within a namespace. Useful for users to quickly summarize their own access, or for UIs to hide/show actions.
These APIs can be queried by creating normal Kubernetes resources, where the response "status"
field of the returned object is the result of the query.
kubectl create -f - -o yaml
&lt;&lt; EOF
apiVersion: authorization.k8s.io/v1
kind: SelfSubjectAccessReview
spec:
resourceAttributes:
group: apps
resource: deployments
verb: create
namespace: dev
EOF
The generated
is:
:
kind
:
metadata
:
:
null
spec
:
resourceAttributes
:
group
:
apps
resource
:
namespace
:
dev
verb
:
create
status
:
allowed
:
true
denied
:
false
Using Flags for Your Authorization Module
You m

ust include a flag in your policy to indicate which authorization module
your policies include:
The following flags can be used:
--authorization-mode=ABAC
Attribute-Based Access Control (ABAC) mode allows you to configure policies using local files.
Role-based access control (RBAC) mode allows you to create and store policies using the Kubernetes API.
WebHook is an HTTP callback mode that allows you to manage authorization using a remote REST endpoint.
--authorization-mode=AlwaysDeny
This flag blocks all requests. Use this flag only for testing.
--authorization-mode=AlwaysAllow
This flag allows all requests. Use this flag only if you do not require authorization for your API requests.
You can choose more than one authorization module. Modules are checked in order
so an earlier module has higher priority to allow or deny a request.
Privilege escalation via workload creation or edits
Users who can create/edit pods in a namespace, either directly or through a
such as an operator, could escalate their privileges in that namespace.
Caution:
System administrators, use care when granting access to create or edit workloads.
Details of how these can be misused are documented in
escalation paths
Escalation paths
Mounting arbitrary secrets in that namespace
Can be used to access secrets meant for other workloads
Can be used to obtain a more privileged service account's service account token
Using arbitrary Service Accounts in that namespace
Can perform Kubernetes API actions as another workload (impersonation)
Can perform any privileged actions that Service Account has
Mounting configmaps meant for other workloads in that namespace
Can be used to obtain information meant for other workloads, such as DB host names.
Mounting volumes meant for other workloads in that namespace
Can be used to obtain information meant for other workloads, and change it.
Caution:
System administrators should be cautious when deploying CRDs that
change the above areas. These may open privilege escalations paths.
This should be considered when deciding on your RBAC controls.
To learn more about Authentication, see
in
.
To learn more about Admission Control, see
.
3.8 - Using RBAC Authorization
Role-based access control (RBAC) is a method of regulating access to computer or
network resources based on the roles of individual users within your organization.
RBAC authorization uses the
API group
to drive authorization
decisions, allowing you to dynamically configure policies through the Kubernetes API.
To enable RBAC, start the
API server
with the
--authorization-mode
flag set to a comma-separated list that includes
RBAC
;
for example:
kube-apiserver --authorization-mode
=
Example,RBAC --other-options --more-options
API objects
The RBAC API declares four kinds of Kubernetes object:
Role
,
,
and
. You can
describe objects
,
or amend them, using tools such as
kubectl
, just like any other Kubernetes object.
Caution:
These objects, by design, impose access restrictions. If you are making changes
to a cluster as you learn, see
privilege escalation prevention and bootstrapping
to understand how those restrictions can prevent you making some changes.
Role and ClusterRole
An RBAC
Role
or
contains rules that represent a set of permissions.
Permissions are purely additive (there are no "deny" rules).
A Role always sets permissions within a particular
namespace
;
when you create a Role, you have to specify the namespace it belongs in.
ClusterRole, by contrast, is a non-namespaced resource. The resources have different names (Role
and ClusterRole) because a Kubernetes object always has to be eith

er namespaced or not namespaced;
it can't be both.
ClusterRoles have several uses. You can use a ClusterRole to:
define permissions on namespaced resources and be granted access within individual namespace(s)
define permissions on namespaced resources and be granted access across all namespaces
define permissions on cluster-scoped resources
If you want to define a role within a namespace, use a Role; if you want to define
a role cluster-wide, use a ClusterRole.
Role example
Here's an example Role in the "default" namespace that can be used to grant read access to
pods
:
:
kind
:
Role
metadata
:
namespace
:
default
name
:
pod-reader
rules
:
-
apiGroups
:
[
""
]
# "" indicates the core API group
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"watch"
,
"list"
]
ClusterRole example
A ClusterRole can be used to grant the same permissions as a Role.
Because ClusterRoles are cluster-scoped, you can also use them to grant access to:
cluster-scoped resources (like
nodes
)
non-resource endpoints (like
/healthz
)
namespaced resources (like Pods), across all namespaces
For example: you can use a ClusterRole to allow a particular user to run
Here is an example of a ClusterRole that can be used to grant read access to
secrets
in any particular namespace,
or across all namespaces (depending on how it is
bound
):
:
kind
:
metadata
:
# "namespace" omitted since ClusterRoles are not namespaced
name
:
secret-reader
rules
:
-
apiGroups
:
[
""
]
#
# at the HTTP level, the name of the resource for accessing Secret
# objects is "secrets"
resources
:
[
"secrets"
]
verbs
:
[
"get"
,
"watch"
,
"list"
]
The name of a Role or a ClusterRole object must be a valid
path segment name
.
RoleBinding and ClusterRoleBinding
A role binding grants the permissions defined in a role to a user or set of users.
It holds a list of
subjects
(users, groups, or service accounts), and a reference to the
role being granted.
A RoleBinding grants permissions within a specific namespace whereas a ClusterRoleBinding
grants that access cluster-wide.
A RoleBinding may reference any Role in the same namespace. Alternatively, a RoleBinding
can reference a ClusterRole and bind that ClusterRole to the namespace of the RoleBinding.
If you want to bind a ClusterRole to all the namespaces in your cluster, you use a
ClusterRoleBinding.
The name of a RoleBinding or ClusterRoleBinding object must be a valid
.
RoleBinding examples
Here is an example of a RoleBinding that grants the "pod-reader" Role to the user "jane"
within the "default" namespace.
This allows "jane" to read pods in the "default" namespace.
:
# This role binding allows "jane" to read pods in the "default" namespace.
# You need to already have a Role named "pod-reader" in that namespace.
kind
:
metadata
:
name
:
read-pods
namespace
:
default
subjects
:
# You can specify more than one "subject"
-
kind
:
User
name
:
jane
# "name" is case sensitive
apiGroup
:
roleRef
:
# "roleRef" specifies the binding to a Role / ClusterRole
kind
:
Role
#this must be Role or ClusterRole
name
:
# this must match the name of the Role or ClusterRole you wish to bind to
apiGroup
:
A RoleBinding can also reference a ClusterRole to grant the permissions defined in that
ClusterRole to resources inside the RoleBinding's namespace. This kind of reference
lets you define a set of common roles across your cluster, then reuse them within
multiple namespaces.
For instance, even though the following RoleBinding refers to a ClusterRole,
"dave" (the subject, case sensitive) will only be able to read Secrets in the "development"
namespace, because the RoleBinding's namespace (in its metadata) is "development".
:
# This role binding allows "dave" to read secrets in the "de

velopment" namespace.
# You need to already have a ClusterRole named "secret-reader".
kind
:
metadata
:
name
:
read-secrets
#
# The namespace of the RoleBinding determines where the permissions are granted.
# This only grants permissions within the "development" namespace.
namespace
:
subjects
:
-
kind
:
User
name
:
dave
# Name is case sensitive
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
ClusterRoleBinding example
To grant permissions across a whole cluster, you can use a ClusterRoleBinding.
The following ClusterRoleBinding allows any user in the group "manager" to read
secrets in any namespace.
:
# This cluster role binding allows anyone in the "manager" group to read secrets in any namespace.
kind
:
metadata
:
name
:
read-secrets-global
subjects
:
-
kind
:
Group
name
:
manager
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
After you create a binding, you cannot change the Role or ClusterRole that it refers to.
If you try to change a binding's
roleRef
, you get a validation error. If you do want
to change the
roleRef
for a binding, you need to remove the binding object and create
a replacement.
There are two reasons for this restriction:
Making
roleRef
immutable allows granting someone
update
permission on an existing binding
object, so that they can manage the list of subjects, without being able to change
the role that is granted to those subjects.
A binding to a different role is a fundamentally different binding.
Requiring a binding to be deleted/recreated in order to change the
roleRef
ensures the full list of subjects in the binding is intended to be granted
the new role (as opposed to enabling or accidentally modifying only the roleRef
without verifying all of the existing subjects should be given the new role's
permissions).
The
kubectl auth reconcile
command-line utility creates or updates a manifest file containing RBAC objects,
and handles deleting and recreating binding objects if required to change the role they refer to.
See
command usage and examples
Referring to resources
In the Kubernetes API, most resources are represented and accessed using a string representation of
their object name, such as
pods
for a Pod. RBAC refers to resources using exactly the same
name that appears in the URL for the relevant API endpoint.
Some Kubernetes APIs involve a
, such as the logs for a Pod. A request for a Pod's logs looks like:
GET /api/v1/namespaces/{namespace}/pods/{name}/log
In this case,
pods
is the namespaced resource for Pod resources, and
log
is a
subresource of
pods
. To represent this in an RBAC role, use a slash (
/
) to
delimit the resource and subresource. To allow a subject to read
pods
and
also access the
log
subresource for each of those Pods, you write:
:
kind
:
Role
metadata
:
namespace
:
default
name
:
pod-and-pod-logs-reader
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"pods"
,
]
verbs
:
[
"get"
,
"list"
]
You can also refer to resources by name for certain requests through the
list.
When specified, requests can be restricted to individual instances of a resource.
Here is an example that restricts its subject to only
get
or
update
a
ConfigMap
named
my-configmap
:
:
kind
:
Role
metadata
:
namespace
:
default
name
:
configmap-updater
rules
:
-
apiGroups
:
[
""
]
#
# at the HTTP level, the name of the resource for accessing ConfigMap
# objects is "configmaps"
resources
:
[
]
:
[
"my-configmap"
]
verbs
:
[
"update"
,
"get"
]
Note:
You cannot restrict
create
or
requests by their resource name.
For
create
, this limitation is because the name of the new object may not be

 known at authorization time.
If you restrict
list
or
watch
by resourceName, clients must include a
field selector in their
list
or
watch
request that matches the specified resourceName in order to be authorized.
kubectl get configmaps --field-selector=metadata.name=my-configmap
Rather than referring to individual
resources
and
verbs
you can use the wildcard
*
symbol to refer to all such objects.
For
*
symbol as a suffix glob match and for
apiGroups
and
an empty set means that everything is allowed.
Here is an example that allows access to perform any current and future action on all current and future resources (note, this is similar to the built-in
role).
:
kind
:
Role
metadata
:
namespace
:
default
name
:
example.com-superuser
# DO NOT USE THIS ROLE, IT IS JUST AN EXAMPLE
rules
:
-
apiGroups
:
[
]
resources
:
[
"*"
]
verbs
:
[
"*"
]
Caution:
Using wildcards in resource and verb entries could result in overly permissive access being granted to sensitive resources.
For instance, if a new resource type is added, or a new subresource is added, or a new custom verb is checked, the wildcard entry automatically grants access, which may be undesirable.
The
principle of least privilege
should be employed, using specific resources and verbs to ensure only the permissions required for the workload to function correctly are applied.
Aggregated ClusterRoles
You can
aggregate
several ClusterRoles into one combined ClusterRole.
A controller, running as part of the cluster control plane, watches for ClusterRole
objects with an
set. The
defines a label
selector
that the controller
uses to match other ClusterRole objects that should be combined into the
rules
field of this one.
Caution:
The control plane overwrites any values that you manually specify in the
rules
field of an
aggregate ClusterRole. If you want to change or add rules, do so in the
objects
that are selected by the
.
Here is an example aggregated ClusterRole:
:
kind
:
metadata
:
name
:
monitoring
:
clusterRoleSelectors
:
-
:
rbac.example.com/aggregate-to-monitoring
:
"true"
rules
:
[]
# The control plane automatically fills in the rules
If you create a new ClusterRole that matches the label selector of an existing aggregated ClusterRole,
that change triggers adding the new rules into the aggregated ClusterRole.
Here is an example that adds rules to the "monitoring" ClusterRole, by creating another
ClusterRole labeled
rbac.example.com/aggregate-to-monitoring: true
.
:
kind
:
metadata
:
name
:
monitoring-endpoints
labels
:
:
"true"
# When you create the "monitoring-endpoints" ClusterRole,
# the rules below will be added to the "monitoring" ClusterRole.
rules
:
-
apiGroups
:
[
""
]
resources
:
[
,
"endpointslices"
,
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
The
default user-facing roles
use ClusterRole aggregation. This lets you,
as a cluster administrator, include rules for custom resources, such as those served by
or aggregated API servers, to extend the default roles.
For example: the following ClusterRoles let the "admin" and "edit" default roles manage the custom resource
named CronTab, whereas the "view" role can perform only read actions on CronTab resources.
You can assume that CronTab objects are named
"crontabs"
in URLs as seen by the API server.
:
kind
:
metadata
:
name
:
aggregate-cron-tabs-edit
labels
:
# Add these permissions to the "admin" and "edit" default roles.
rbac.authorization.k8s.io/aggregate-to-admin
:
"true"
rbac.authorization.k8s.io/aggregate-to-edit
:
"true"
rules
:
-
apiGroups
:
[
"stab

le.example.com"
]
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
---
kind
:
:
metadata
:
name
:
aggregate-cron-tabs-view
labels
:
# Add these permissions to the "view" default role.
rbac.authorization.k8s.io/aggregate-to-view
:
"true"
rules
:
-
apiGroups
:
[
"stable.example.com"
]
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
Role examples
The following examples are excerpts from Role or ClusterRole objects, showing only
the
rules
section.
Allow reading
"pods"
resources in the core
API Group
:
rules
:
-
apiGroups
:
[
""
]
#
# at the HTTP level, the name of the resource for accessing Pod
# objects is "pods"
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
Allow reading/writing Deployments (at the HTTP level: objects with
in the resource part of their URL) in the
"apps"
API groups:
rules
:
-
apiGroups
:
[
"apps"
]
#
# at the HTTP level, the name of the resource for accessing Deployment
# objects is "deployments"
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
Allow reading Pods in the core API group, as well as reading or writing Job
"batch"
API group:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
-
apiGroups
:
[
"batch"
]
#
# at the HTTP level, the name of the resource for accessing Job
# objects is "jobs"
resources
:
[
"jobs"
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
Allow reading a ConfigMap named "my-config" (must be bound with a
RoleBinding to limit to a single ConfigMap in a single namespace):
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
]
:
[
"my-config"
]
verbs
:
[
"get"
]
Allow reading the resource
"nodes"
in the core group (because a
Node is cluster-scoped, this must be in a ClusterRole bound with a
ClusterRoleBinding to be effective):
rules
:
-
apiGroups
:
[
""
]
#
# at the HTTP level, the name of the resource for accessing Node
# objects is "nodes"
resources
:
[
"nodes"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
Allow GET and POST requests to the non-resource endpoint
/healthz
and
all subpaths (must be in a ClusterRole bound with a ClusterRoleBinding
to be effective):
rules
:
-
:
[
,
"/healthz/*"
]
# '*' in a nonResourceURL is a suffix glob match
verbs
:
[
"get"
,
"post"
]
Referring to subjects
A RoleBinding or ClusterRoleBinding binds a role to subjects.
Subjects can be groups, users or
ServiceAccounts
.
Kubernetes represents usernames as strings.
These can be: plain names, such as "alice"; email-style names, like "bob@example.com";
or numeric user IDs represented as a string. It is up to you as a cluster administrator
to configure the
authentication modules
so that authentication produces usernames in the format you want.
Caution:
The prefix
system:
is reserved for Kubernetes system use, so you should ensure
that you don't have users or groups with names that start with
system:
by
accident.
Other than this special prefix, the RBAC authorization system does not require any format
for usernames.
In Kubernetes, Authenticator modules provide group information.
Groups, like users, are represented as strings, and that string has no format requirements,
other than that the prefix
system:
is reserved.
have names prefixed
with
system:serviceaccount:
, and belong to groups that have names prefixed with
system:serviceaccounts:
.
Note:
(singular) is the prefix for service account usernames.
(plural) is the prefix for service account groups.
The following examples are
excerpts that only
show the

subjects
section.
For a user named
alice@example.com
:
subjects
:
-
kind
:
User
name
:
"alice@example.com"
apiGroup
:
For a group named
frontend-admins
:
subjects
:
-
kind
:
Group
name
:
"frontend-admins"
apiGroup
:
For the default service account in the "kube-system" namespace:
subjects
:
-
kind
:
name
:
default
namespace
:
For all service accounts in the "qa" namespace:
subjects
:
-
kind
:
Group
name
:
system:serviceaccounts:qa
apiGroup
:
For all service accounts in any namespace:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
For all authenticated users:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
For all unauthenticated users:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
For all users:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
-
kind
:
Group
name
:
apiGroup
:
Default roles and role bindings
API servers create a set of default ClusterRole and ClusterRoleBinding objects.
Many of these are
system:
prefixed, which indicates that the resource is directly
managed by the cluster control plane.
All of the default ClusterRoles and ClusterRoleBindings are labeled with
kubernetes.io/bootstrapping=rbac-defaults
.
Caution:
Take care when modifying ClusterRoles and ClusterRoleBindings with names
that have a
system:
prefix.
Modifications to these resources can result in non-functional clusters.
Auto-reconciliation
At each start-up, the API server updates default cluster roles with any missing permissions,
and updates default cluster role bindings with any missing subjects.
This allows the cluster to repair accidental modifications, and helps to keep roles and role bindings
up-to-date as permissions and subjects change in new Kubernetes releases.
To opt out of this reconciliation, set the
rbac.authorization.kubernetes.io/autoupdate
annotation on a default cluster role or rolebinding to
false
.
Be aware that missing default permissions and subjects can result in non-functional clusters.
Auto-reconciliation is enabled by default if the RBAC authorizer is active.
API discovery roles
Default role bindings authorize unauthenticated and authenticated users to read API information that is deemed safe to be publicly accessible (including CustomResourceDefinitions). To disable anonymous unauthenticated access, add
to the API server configuration.
To view the configuration of these roles via
kubectl
run:
kubectl get clusterroles system:discovery -o yaml
Note:
If you edit that ClusterRole, your changes will be overwritten on API server restart
via
auto-reconciliation
. To avoid that overwriting,
either do not manually edit the role, or disable auto-reconciliation.
Kubernetes RBAC API discovery roles
Default ClusterRole
Default ClusterRoleBinding
group
Allows a user read-only access to basic information about themselves. Prior to v1.14, this role was also bound to
system:discovery
group
Allows read-only access to API discovery endpoints needed to discover and negotiate an API level. Prior to v1.14, this role was also bound to
system:public-info-viewer
and
groups
Allows read-only access to non-sensitive information about the cluster. Introduced in Kubernetes v1.14.
User-facing roles
Some of the default ClusterRoles are not
system:
prefixed. These are intended to be user-facing roles.
They include super-user roles (
), roles intended to be granted cluster-wide
using ClusterRoleBindings, and roles intended to be granted within particular
namespaces using RoleBindings (
admin
,
edit
,
view
).
User

-facing ClusterRoles use
ClusterRole aggregation
to allow admins to include
rules for custom resources on these ClusterRoles. To add rules to the
admin
,
edit
, or
view
roles, create
a ClusterRole with one or more of the following labels:
metadata
:
labels
:
:
"true"
:
"true"
:
"true"
group
Allows super-user access to perform any action on any resource.
When used in a
, it gives full control over every resource in the cluster and in all namespaces.
, it gives full control over every resource in the role binding's namespace, including the namespace itself.
admin
None
Allows admin access, intended to be granted within a namespace using a
.
If used in a
, allows read/write access to most resources in a namespace,
including the ability to create roles and role bindings within the namespace.
This role does not allow write access to resource quota or to the namespace itself.
This role also does not allow write access to EndpointSlices (or Endpoints) in clusters created
using Kubernetes v1.22+. More information is available in the
"Write Access for EndpointSlices and Endpoints" section
.
edit
None
Allows read/write access to most objects in a namespace.
This role does not allow viewing or modifying roles or role bindings.
However, this role allows accessing Secrets and running Pods as any ServiceAccount in
the namespace, so it can be used to gain the API access levels of any ServiceAccount in
the namespace. This role also does not allow write access to EndpointSlices (or Endpoints) in
clusters created using Kubernetes v1.22+. More information is available in the
.
view
None
Allows read-only access to see most objects in a namespace.
It does not allow viewing roles or role bindings.
This role does not allow viewing Secrets, since reading
the contents of Secrets enables access to ServiceAccount credentials
in the namespace, which would allow API access as any ServiceAccount
in the namespace (a form of privilege escalation).
Core component roles
user
Allows access to the resources required by the
scheduler
system:volume-scheduler
user
Allows access to the volume resources required by the kube-scheduler component.
user
controller manager
The permissions required by individual controllers are detailed in the
controller roles
.
None
Allows access to resources required by the kubelet,
including read access to all secrets, and write access to all pod status objects
.
You should use the
Node authorizer
and
NodeRestriction admission plugin
instead of the
role, and allow granting API access to kubelets based on the Pods scheduled to run on them.
The
role only exists for compatibility with Kubernetes clusters upgraded from versions prior to v1.8.
system:kube-proxy
user
Other component roles
system:auth-delegator
None
Allows delegated authentication and authorization checks.
This is commonly used by add-on API servers for unified authentication and authorization.
system:heapster
None
Role for the
Heapster
component (deprecated).
system:kube-aggregator
None
kube-aggregator
system:kube-dns
kube-dns
service account in the
namespace
kube-dns
system:kubelet-api-admin
None
Allows full access to the kubelet API.
system:nod

e-bootstrapper
None
Allows access to the resources required to perform
.
system:node-problem-detector
None
system:persistent-volume-provisioner
None
Allows access to the resources required by most
dynamic volume provisioners
.
system:monitoring
group
Allows read access to control-plane monitoring endpoints (i.e.
liveness and readiness endpoints (
/healthz
,
/livez
,
/readyz
), the individual health-check endpoints (
/healthz/*
,
/livez/*
,
/readyz/*
), and
/metrics
). Note that individual health check endpoints and the metric endpoint may expose sensitive information.
Roles for built-in controllers
runs
that are built in to the Kubernetes
control plane.
When invoked with
, kube-controller-manager starts each controller
using a separate service account.
Corresponding roles exist for each built-in controller, prefixed with
system:controller:
.
If the controller manager is not started with
, it runs all control loops
using its own credential, which must be granted all the relevant roles.
These roles include:
system:controller:attachdetach-controller
system:controller:certificate-controller
system:controller:clusterrole-aggregation-controller
system:controller:cronjob-controller
system:controller:daemon-set-controller
system:controller:deployment-controller
system:controller:disruption-controller
system:controller:endpoint-controller
system:controller:expand-controller
system:controller:generic-garbage-collector
system:controller:horizontal-pod-autoscaler
system:controller:job-controller
system:controller:namespace-controller
system:controller:node-controller
system:controller:persistent-volume-binder
system:controller:pod-garbage-collector
system:controller:pv-protection-controller
system:controller:pvc-protection-controller
system:controller:replicaset-controller
system:controller:replication-controller
system:controller:resourcequota-controller
system:controller:root-ca-cert-publisher
system:controller:route-controller
system:controller:service-account-controller
system:controller:service-controller
system:controller:statefulset-controller
system:controller:ttl-controller
Privilege escalation prevention and bootstrapping
The RBAC API prevents users from escalating privileges by editing roles or role bindings.
Because this is enforced at the API level, it applies even when the RBAC authorizer is not in use.
Restrictions on role creation or update
You can only create/update a role if at least one of the following things is true:
You already have all the permissions contained in the role, at the same scope as the object being modified
(cluster-wide for a ClusterRole, within the same namespace or cluster-wide for a Role).
You are granted explicit permission to perform the
escalate
verb on the
roles
or
resource in the
API group.
For example, if
user-1
does not have the ability to list Secrets cluster-wide, they cannot create a ClusterRole
containing that permission. To allow a user to create/update roles:
Grant them a role that allows them to create/update Role or ClusterRole objects, as desired.
Grant them permission to include specific permissions in the roles they create/update:
implicitly, by giving them those permissions (if they attempt to create or modify a Role or ClusterRole with permissions they themselves have not been granted, the API request will be forbidden)
or explicitly allow specifying any permission in a
Role
or
by giving them permission to perform the
escalate
verb on
roles
or
API group
Restrictions on role binding creation or update
You can only create/update a role binding if you already have all the permissions contained in the referenced role
(at the same scope as the role binding)
or
if y

ou have been authorized to perform the
bind
verb on the referenced role.
user-1
does not have the ability to list Secrets cluster-wide, they cannot create a ClusterRoleBinding
to a role that grants that permission. To allow a user to create/update role bindings:
Grant them a role that allows them to create/update RoleBinding or ClusterRoleBinding objects, as desired.
Grant them permissions needed to bind a particular role:
implicitly, by giving them the permissions contained in the role.
explicitly, by giving them permission to perform the
bind
verb on the particular Role (or ClusterRole).
For example, this ClusterRole and RoleBinding would allow
user-1
to grant other users the
admin
,
edit
, and
view
roles in the namespace
user-1-namespace
:
:
kind
:
metadata
:
name
:
role-grantor
rules
:
-
apiGroups
:
[
"rbac.authorization.k8s.io"
]
resources
:
[
"rolebindings"
]
verbs
:
[
"create"
]
-
apiGroups
:
[
]
resources
:
[
"clusterroles"
]
verbs
:
[
"bind"
]
# omit resourceNames to allow binding any ClusterRole
:
[
"admin"
,
"edit"
,
"view"
]
---
:
kind
:
metadata
:
name
:
role-grantor-binding
namespace
:
roleRef
:
apiGroup
:
kind
:
name
:
subjects
:
-
apiGroup
:
kind
:
User
name
:
user-1
When bootstrapping the first roles and role bindings, it is necessary for the initial user to grant permissions they do not yet have.
To bootstrap initial roles and role bindings:
Use a credential with the "system:masters" group, which is bound to the "cluster-admin" super-user role by the default bindings.
Command-line utilities
kubectl create role
Creates a Role object defining permissions within a single namespace. Examples:
Create a Role named "pod-reader" that allows users to perform
get
,
watch
and
list
on pods:
kubectl create role pod-reader --verb
=
get --verb
=
=
watch --resource
=
pods
Create a Role named "pod-reader" with resourceNames specified:
=
get --resource
=
pods --resource-name
=
readablepod --resource-name
=
anotherpod
Create a Role named "foo" with apiGroups specified:
kubectl create role foo --verb
=
get,list,watch --resource
=
replicasets.apps
Create a Role named "foo" with subresource permissions:
=
=
pods,pods/status
Create a Role named "my-component-lease-holder" with permissions to get/update a resource with a specific name:
kubectl create role my-component-lease-holder --verb
=
get,list,watch,update --resource
=
lease --resource-name
=
my-component
kubectl create clusterrole
Creates a ClusterRole. Examples:
Create a ClusterRole named "pod-reader" that allows user to perform
get
,
watch
and
list
on pods:
kubectl create clusterrole pod-reader --verb
=
=
pods
Create a ClusterRole named "pod-reader" with resourceNames specified:
=
=
=
=
Create a ClusterRole named "foo" with apiGroups specified:
kubectl create clusterrole foo --verb
=
=
Create a ClusterRole named "foo" with subresource permissions:
=
=
Create a ClusterRole named "foo" with nonResourceURL specified:
"foo"
--verb
=
get --non-resource-url
=
/logs/*
Create a ClusterRole named "monitoring" with an aggregationRule specified:
kubectl create clusterrole monitoring --aggregation-rule
=
"rbac.example.com/aggregate-to-monitoring=true"
kubectl create rolebinding
Grants a Role or ClusterRole within a specific namespace. Examples:
Within the namespace "acme", grant the permissions in the "admin" ClusterRole to a user named "b

ob":
kubectl create rolebinding bob-admin-binding --clusterrole
=
admin --user
=
bob --namespace
=
acme
Within the namespace "acme", grant the permissions in the "view" ClusterRole to the service account in the namespace "acme" named "myapp":
kubectl create rolebinding myapp-view-binding --clusterrole
=
view --serviceaccount
=
acme:myapp --namespace
=
acme
Within the namespace "acme", grant the permissions in the "view" ClusterRole to a service account in the namespace "myappnamespace" named "myapp":
kubectl create rolebinding myappnamespace-myapp-view-binding --clusterrole
=
=
myappnamespace:myapp --namespace
=
acme
kubectl create clusterrolebinding
Grants a ClusterRole across the entire cluster (all namespaces). Examples:
Across the entire cluster, grant the permissions in the "cluster-admin" ClusterRole to a user named "root":
kubectl create clusterrolebinding root-cluster-admin-binding --clusterrole
=
cluster-admin --user
=
root
Across the entire cluster, grant the permissions in the "system:node-proxier" ClusterRole to a user named "system:kube-proxy":
kubectl create clusterrolebinding kube-proxy-binding --clusterrole
=
system:node-proxier --user
=
Across the entire cluster, grant the permissions in the "view" ClusterRole to a service account named "myapp" in the namespace "acme":
kubectl create clusterrolebinding myapp-view-binding --clusterrole
=
=
acme:myapp
Creates or updates
API objects from a manifest file.
Missing objects are created, and the containing namespace is created for namespaced objects, if required.
Existing roles are updated to include the permissions in the input objects,
and remove extra permissions if
--remove-extra-permissions
Existing bindings are updated to include the subjects in the input objects,
and remove extra subjects if
--remove-extra-subjects
Examples:
Test applying a manifest file of RBAC objects, displaying changes that would be made:
kubectl auth reconcile -f my-rbac-rules.yaml --dry-run=client
Apply a manifest file of RBAC objects, preserving any extra permissions (in roles) and any extra subjects (in bindings):
kubectl auth reconcile -f my-rbac-rules.yaml
Apply a manifest file of RBAC objects, removing any extra permissions (in roles) and any extra subjects (in bindings):
kubectl auth reconcile -f my-rbac-rules.yaml --remove-extra-subjects --remove-extra-permissions
ServiceAccount permissions
Default RBAC policies grant scoped permissions to control-plane components, nodes,
and controllers, but grant
no permissions
to service accounts outside the
namespace
(beyond discovery permissions given to all authenticated users).
This allows you to grant particular roles to particular ServiceAccounts as needed.
Fine-grained role bindings provide greater security, but require more effort to administrate.
Broader grants can give unnecessary (and potentially escalating) API access to
ServiceAccounts, but are easier to administrate.
In order from most secure to least secure, the approaches are:
Grant a role to an application-specific service account (best practice)
This requires the application to specify a
in its pod spec,
and for the service account to be created (via the API, application manifest,
kubectl create serviceaccount
, etc.).
For example, grant read-only permission within "my-namespace" to the "my-sa" service account:
kubectl create rolebinding my-sa-view
\
--clusterrole
=
view
\
--serviceaccount
=
my-namespace:my-sa
\
=
Grant a role to the "default" service account in a namespace
If an application does not specify a
, it uses the "default" service account.
Note:
Permissions given to the "default" service account are available to any pod
in the namespace that does not specify a
.
For example, grant read-only permission within "my-namespace" to the "defa

ult" service account:
kubectl create rolebinding default-view
\
=
view
\
=
my-namespace:default
\
=
Many
add-ons
run as the
"default" service account in the
To allow those add-ons to run with super-user access, grant cluster-admin
permissions to the "default" service account in the
Caution:
Enabling this means the
namespace contains Secrets
that grant super-user access to your cluster's API.
kubectl create clusterrolebinding add-on-cluster-admin
\
=
\
=
kube-system:default
Grant a role to all service accounts in a namespace
If you want all applications in a namespace to have a role, no matter what service account they use,
you can grant a role to the service account group for that namespace.
For example, grant read-only permission within "my-namespace" to all service accounts in that namespace:
kubectl create rolebinding serviceaccounts-view
\
=
view
\
--group
=
system:serviceaccounts:my-namespace
\
=
Grant a limited role to all service accounts cluster-wide (discouraged)
If you don't want to manage permissions per-namespace, you can grant a cluster-wide role to all service accounts.
For example, grant read-only permission across all namespaces to all service accounts in the cluster:
kubectl create clusterrolebinding serviceaccounts-view
\
=
view
\
--group
=
Grant super-user access to all service accounts cluster-wide (strongly discouraged)
If you don't care about partitioning permissions at all, you can grant super-user access to all service accounts.
Warning:
This allows any application full access to your cluster, and also grants
any user with read access to Secrets (or the ability to create any pod)
full access to your cluster.
kubectl create clusterrolebinding serviceaccounts-cluster-admin
\
=
\
--group
=
Write access for EndpointSlices and Endpoints
Kubernetes clusters created before Kubernetes v1.22 include write access to
EndpointSlices (and Endpoints) in the aggregated "edit" and "admin" roles.
As a mitigation for
CVE-2021-25740
,
this access is not part of the aggregated roles in clusters that you create using
Kubernetes v1.22 or later.
Existing clusters that have been upgraded to Kubernetes v1.22 will not be
subject to this change. The
CVE
announcement
includes
guidance for restricting this access in existing clusters.
If you want new clusters to retain this level of access in the aggregated roles,
you can create the following ClusterRole:
access/endpoints-aggregated.yaml
:
kind
:
metadata
:
:
kubernetes.io/description
:
|-
Add endpoints write permissions to the edit and admin roles. This was
removed by default in 1.22 because of CVE-2021-25740. See
https://issue.k8s.io/103675. This can allow writers to direct LoadBalancer
or Ingress implementations to expose backend IPs that would not otherwise
be accessible, and can circumvent network policies or security controls
intended to prevent/isolate access to those backends.
EndpointSlices were never included in the edit or admin roles, so there
is nothing to restore for the EndpointSlice API.
labels
:
:
"true"
name
:
custom:aggregate-to-edit:endpoints
# you can change this if you wish
rules
:
-
apiGroups
:
[
""
]
resources
:
[
]
verbs
:
[
"create"
,
"delete"
,
"deletecollection"
,
"patch"
,
"update"
]
Upgrading from ABAC
Clusters that originally ran older Kubernetes versions often used
permissive ABAC policies, including granting full API access to all
service accounts.
namespace
(beyond discovery permissio

ns given to all authenticated users).
While far more secure, this can be disruptive to existing workloads expecting to automatically receive API permissions.
Here are two approaches for managing this transition:
Parallel authorizers
Run both the RBAC and ABAC authorizers, and specify a policy file that contains
the
legacy ABAC policy
:
--authorization-mode=...,RBAC,ABAC --authorization-policy-file=mypolicy.json
To explain that first command line option in detail: if earlier authorizers, such as Node,
deny a request, then the RBAC authorizer attempts to authorize the API request. If RBAC
also denies that API request, the ABAC authorizer is then run. This means that any request
allowed by
either
the RBAC or ABAC policies is allowed.
When the kube-apiserver is run with a log level of 5 or higher for the RBAC component
(
--vmodule=rbac*=5
or
--v=5
), you can see RBAC denials in the API server log
(prefixed with
RBAC
).
You can use that information to determine which roles need to be granted to which users, groups, or service accounts.
Once you have
granted roles to service accounts
and workloads
are running with no RBAC denial messages in the server logs, you can remove the ABAC authorizer.
Permissive RBAC permissions
You can replicate a permissive ABAC policy using RBAC role bindings.
Warning:
The following policy allows
ALL
service accounts to act as cluster administrators.
Any application running in a container receives service account credentials automatically,
and could perform any action against the API, including viewing secrets and modifying permissions.
This is not a recommended policy.
kubectl create clusterrolebinding permissive-binding
\
=
\
--user
=
admin
\
--user
=
kubelet
\
--group
=
After you have transitioned to use RBAC, you should adjust the access controls
for your cluster to ensure that these meet your information security needs.
3.9 - Using ABAC Authorization
Attribute-based access control (ABAC) defines an access control paradigm whereby access rights are granted to users through the use of policies which combine attributes together.
Policy File Format
To enable
ABAC
mode, specify
--authorization-policy-file=SOME_FILENAME
and
on startup.
The file format is
one JSON object per line
. There
should be no enclosing list or map, only one map per line.
Each line is a "policy object", where each such object is a map with the following
properties:
Versioning properties:
, type string; valid values are "abac.authorization.kubernetes.io/v1beta1". Allows versioning and conversion of the policy format.
kind
, type string: valid values are "Policy". Allows versioning and conversion of the policy format.
spec
property set to a map with the following properties:
Subject-matching properties:
user
, type string; the user-string from
--token-auth-file
. If you specify
user
, it must match the username of the authenticated user.
group
, type string; if you specify
group
, it must match one of the groups of the authenticated user.
matches all authenticated requests.
matches all unauthenticated requests.
Resource-matching properties:
apiGroup
, type string; an API group.
Ex:
apps
,
networking.k8s.io
Wildcard:
*
namespace
, type string; a namespace.
Ex:
Wildcard:
*
matches all resource requests.
resource
, type string; a resource type
Ex:
pods
,
Wildcard:
*
Non-resource-matching properties:
nonResourcePath
, type string; non-resource request paths.
Ex:
/version
or
/apis
Wildcard:
*
matches all non-resource requests.
/foo/*
matches all subpaths of
/foo/
.
readonly
, type boolean, when true, means that the Resource-matching policy only applies to get, list, and watch operations, Non-resource-matching policy only applies to get operation.
Note:
An unset property is the same as a property set to the zero value for its type
(e.

g. empty string, 0, false). However, unset should be preferred for
readability.
In the future, policies may be expressed in a JSON format, and managed via a
REST interface.
Authorization Algorithm
A request has attributes which correspond to the properties of a policy object.
When a request is received, the attributes are determined. Unknown attributes
are set to the zero value of its type (e.g. empty string, 0, false).
A property set to
"*"
will match any value of the corresponding attribute.
The tuple of attributes is checked for a match against every policy in the
policy file. If at least one line matches the request attributes, then the
request is authorized (but may fail later validation).
To permit any authenticated user to do something, write a policy with the
group property set to
.
To permit any unauthenticated user to do something, write a policy with the
"system:unauthenticated"
.
To permit a user to do anything, write a policy with the apiGroup, namespace,
resource, and nonResourcePath properties set to
"*"
.
Kubectl
Kubectl uses the
/api
and
/apis
endpoints of apiserver to discover
served resource types, and validates objects sent to the API by create/update
operations using schema information located at
.
When using ABAC authorization, those special resources have to be explicitly
exposed via the
property in a policy (see
examples
below):
/api
,
/api/*
,
/apis
, and
/apis/*
for API version negotiation.
/version
for retrieving the server version via
.
/swaggerapi/*
for create/update operations.
To inspect the HTTP calls involved in a specific kubectl operation you can turn
up the verbosity:
kubectl --v=8 version
Examples
Alice can do anything to all resources:
{
:
"abac.authorization.kubernetes.io/v1beta1"
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"alice"
,
:
"*"
,
"resource"
:
"*"
,
"apiGroup"
:
"*"
}}
The Kubelet can read any pods:
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"kubelet"
,
:
"*"
,
:
"pods"
,
"readonly"
:
true
}}
The Kubelet can read and write events:
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"kubelet"
,
:
"*"
,
:
"events"
}}
Bob can just read pods in namespace "projectCaribou":
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"bob"
,
:
"projectCaribou"
,
:
"pods"
,
:
true
}}
Anyone can make read-only requests to all non-resource paths:
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"group"
:
,
:
true
,
"nonResourcePath"
:
"*"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"group"
:
,
:
true
,
:
"*"
}}
Complete file example
A quick note on service accounts
Every service account has a corresponding ABAC username, and that service account's username is generated according to the naming convention:
system:serviceaccount:&lt;namespace&gt;:&lt;serviceaccountname&gt;
Creating a new namespace leads to the creation of a new service account in the following format:
system:serviceaccount:&lt;namespace&gt;:default
For example, if you wanted to grant the default service account (in the
namespace) full
privilege to the API using ABAC, you would add this line to your policy file:
{
:
,
"kind"
:
"Policy"
,
"spec"
:{
"user"
:
"system:serviceaccount:kube-system:default"
,
:
"*"
,
:
"*"
,
:
"*"
}}
The apiserver will need to be restarted to pick up the new policy lines.
3.10 - Using Node Authorization
Node authorization is a special-purpose authorization mode that specifically aut

horizes API requests made by kubelets.
Overview
services
endpoints
nodes
pods
events
.
.
must
hostname
hostname
and the
option.
.
.
group
Node
group,
but do not identify the particular node they are associated with,
format.
Node
The
RBAC
Node
Enable the
Node
) and the
Node
NODE DENY
Delete the
In 1.6, the
.
Node
and
R

BAC
3.11 - Mapping PodSecurityPolicies to Pod Security Standards
The tables below enumerate the configuration parameters on
objects, whether the field mutates
and/or validates pods, and how the configuration values map to the
.
For each applicable parameter, the allowed values for the
Baseline
and
profiles are listed.
Anything outside the allowed values for those profiles would fall under the
profile. "No opinion"
means all values are allowed under all Pod Security Standards.
For a step-by-step migration guide, see
.
PodSecurityPolicy Spec
The fields enumerated in this table are part of the
PodSecurityPolicySpec
, which is specified
under the
.spec
field path.
Mapping PodSecurityPolicySpec fields to Pod Security Standards
Type
Pod Security Standards Equivalent
Validating
Baseline &amp; Restricted
:
false
/ undefined / nil
defaultAddCapabilities
Mutating &amp; Validating
Requirements match
allowedCapabilities
below.
Baseline
: subset of
AUDIT_WRITE
CHOWN
DAC_OVERRIDE
FOWNER
FSETID
KILL
MKNOD
NET_BIND_SERVICE
SETFCAP
SETGID
SETPCAP
SETUID
SYS_CHROOT
: empty / undefined / nil OR a list containing
only
requiredDropCapabilities
Baseline
: no opinion
: must include
ALL
volumes
Baseline
: anything except
hostPath
*
: subset of
configMap
csi
emptyDir
ephemeral
projected
secret
:
false
hostPorts
: undefined / nil / empty
hostPID
:
false
hostIPC
:
false
seLinux
:
seLinux.rule
is
MustRunAs
, with the following
options
user
is unset (
""
/ undefined / nil)
role
is unset (
""
type
is unset or one of:
container_t, container_init_t, container_kvm_t
level
is anything
runAsUser
Baseline
: Anything
:
rule
is
MustRunAsNonRoot
Mutating (MustRunAs) &amp; Validating
No opinion
No opinion
fsGroup
No opinion
No opinion
defaultAllowPrivilegeEscalation
Mutating
No opinion (non-validating)
Only mutating if set to
false
Baseline
: No opinion
:
false
allowedHostPaths
No opinion (volumes takes precedence)
allowedFlexVolumes
allowedCSIDrivers
allowedUnsafeSysctls
forbiddenSysctls
No opinion
allowedProcMountTypes
(alpha feature)
:
["Default"]
OR undefined / nil / empty
runtimeClass
.defaultRuntimeClassName
Mutating
No opinion
.allowedRuntimeClassNames
No opinion
PodSecurityPolicy annotations
The
enumerated in this
table can be specified under
.metadata.annotations
on the PodSecurityPolicy object.
Mapping PodSecurityPolicy annotations to Pod Security Standards
PSP Annotation
Type
seccomp.security.alpha.kubernetes.io
/defaultProfileName
Mutating
No opinion
/allowedProfileName

s
Baseline
:
"runtime/default,"
(Trailing comma to allow unset)
:
"runtime/default"
(No trailing comma)
localhost/*
values are also permitted for both Baseline &amp; Restricted.
apparmor.security.beta.kubernetes.io
Mutating
No opinion
/allowedProfileNames
Baseline
:
:
3.12 - Webhook Mode
A WebHook is an HTTP callback: an HTTP POST that occurs when something happens; a simple event-notification via HTTP POST. A web application implementing WebHooks will POST a message to a URL when certain things happen.
When specified, mode
Webhook
causes Kubernetes to query an outside REST
service when determining user privileges.
Configuration File Format
Mode
Webhook
requires a file for HTTP configuration, specify by the
--authorization-webhook-config-file=SOME_FILENAME
flag.
file format. Within the file "users" refers to the API Server webhook and
"clusters" refers to the remote service.
A configuration example which uses HTTPS client auth:
:
v1
kind
:
Config
clusters
:
-
name
:
name-of-remote-authz-service
cluster
:
:
# URL of remote service to query. Must use 'https'. May not include parameters.
server
:
https://authz.example.com/authorize
# users refers to the API Server's webhook configuration.
users
:
-
name
:
user
:
:
:
# kubeconfig files require a context. Provide one for the API Server.
:
webhook
contexts
:
-
context
:
cluster
:
user
:
name
:
webhook
Request Payloads
When faced with an authorization decision, the API Server POSTs a JSON-
serialized
object describing the
action. This object contains fields describing the user attempting to make the
request, and either details about the resource being accessed or requests
attributes.
as other Kubernetes API objects. Implementers should be aware of looser
compatibility promises for beta objects and check the "apiVersion" field of the
request to ensure correct deserialization. Additionally, the API Server must
enable the
API extensions group (
--runtime-config=authorization.k8s.io/v1beta1=true
).
{
:
"authorization.k8s.io/v1beta1"
,
"kind"
:
"SubjectAccessReview"
,
"spec"
: {
"resourceAttributes"
: {
:
"kittensandponies"
,
"verb"
:
"get"
,
"group"
:
"unicorn.example.org"
,
:
"pods"
},
"user"
:
"jane"
,
"group"
: [
"group1"
,
"group2"
]
}
}
status
field of
the request and respond to either allow or disallow access. The response body's
spec
field is ignored and may be omitted. A permissive response would return:
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
true
}
}
For disallowing access there are two methods.
The first method is preferred in most cases, and indicates the authorization
webhook does not allow, or has "no opinion" about the request, but if other
authorizers are configured, they are given a chance to allow the request.
If there are no other authorizers, or none of them allow the request, the
request is forbidden. The webhook would return:
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"reason"
:
"user

 does not have read access to the namespace"
}
}
The second method denies immediately, short-circuiting evaluation by other
configured authorizers. This should only be used by webhooks that have
detailed knowledge of the full authorizer configuration of the cluster.
The webhook would return:
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"denied"
:
true
,
"reason"
:
"user does not have read access to the namespace"
}
}
Access to non-resource paths are sent as:
{
:
,
"kind"
:
,
"spec"
: {
"nonResourceAttributes"
: {
"path"
:
"/debug"
,
"verb"
:
"get"
},
"user"
:
"jane"
,
"group"
: [
"group1"
,
"group2"
]
}
}
Non-resource paths include:
/api
,
/apis
,
/metrics
,
/logs
,
/debug
,
/healthz
,
/livez
,
,
/readyz
, and
/version.
Clients require access to
/api
,
/api/*
,
/apis
,
/apis/*
,
and
/version
to discover what resources and versions are present on the server.
Access to other non-resource paths can be disallowed without restricting access
to the REST api.
For further documentation refer to the authorization.v1beta1 API objects and
webhook.go
.
3.13 - Kubelet authentication/authorization
Overview
.
flag
and
flags
see the
ensure the
and
flags
ensure the
and the
flags
The kubelet authorizes API requests using the same
HTTP verb
POST
create
GET, HEAD
get
PUT
update
PATCH
patch
DELETE
delete
The resource and subresource is determined from the inco

ming request's path:
resource
/stats/*
nodes
stats
nodes
metrics
/logs/*
nodes
log
/spec/*
nodes
spec
all others
nodes
proxy
Node
and
3.14 - TLS bootstrapping
In a Kubernetes cluster, the components on the worker nodes - kubelet and kube-proxy - need to communicate with Kubernetes control plane components, specifically kube-apiserver.
In order to ensure that communication is kept private, not interfered with, and ensure that each component of the cluster is talking to another trusted component, we strongly
recommend using client TLS certificates on nodes.
The normal process of bootstrapping these components, especially worker nodes that need certificates so they can communicate safely with kube-apiserver,
can be a challenging process as it is often outside of the scope of Kubernetes and requires significant additional work.
This in turn, can make it challenging to initialize or scale a cluster.
In order to simplify the process, beginning in version 1.4, Kubernetes introduced a certificate request and signing API. The proposal can be
found
here
.
This document describes the process of node initialization, how to set up TLS client certificate bootstrapping for
kubelets, and how it works.
Initialization Process
When a worker node starts up, the kubelet does the following:
Look for its
file
Retrieve the URL of the API server and credentials, normally a TLS key and signed certificate from the
file
Attempt to communicate with the API server using the credentials.
Assuming that the kube-apiserver successfully validates the kubelet's credentials, it will treat the kubelet as a valid node, and begin to assign pods to it.
Note that the above process depends upon:
Existence of a key and certificate on the local host in the
The certificate having been signed by a Certificate Authority (CA) trusted by the kube-apiserver
All of the following are responsibilities of whoever sets up and manages the cluster:
Creating the CA key and certificate
Distributing the CA certificate to the control plane nodes, where kube-apiserver is running
Creating a key and certificate for each kubelet; strongly recommended to have a unique one, with a unique CN, for each kubelet
Signing the kubelet certificate using the CA key
Distributing the kubelet key and signed certificate to the specific node on which the kubelet is running
The TLS Bootstrapping described in this document is intended to simplify, and partially or even completely automate, steps 3 onwards, as these are the most common when initializing or scaling
a cluster.
Bootstrap Initialization
In the bootstrap initialization process, the following occurs:
kubelet begins
kubelet sees that it does
not
have a
file
kubelet searches for and finds a
bootstrap-kubeconfig
file
kubelet reads its bootstrap file, retrieving the URL of the API server and a limited usage "token"
kubelet connects to the API server, authenticates using the token
kubelet now has limited credentials to create and retrieve a certificate signing request (CSR)
kubelet creates a CSR for itself with the signerName set to
CSR is approved in one of two ways:
If configured, kube-controller-manager automatically approves the CSR
If configured, an outside process, possibly a person, approves the CSR using the Kubernetes API or via
kubectl
Certificate is created for the kubelet
Certificate is issued to the 

kubelet
kubelet retrieves the certificate
kubelet creates a proper
with the key and signed certificate
kubelet begins normal operation
Optional: if configured, kubelet automatically requests renewal of the certificate when it is close to expiry
The renewed certificate is approved and issued, either automatically or manually, depending on configuration.
The rest of this document describes the necessary steps to configure TLS Bootstrapping, and its limitations.
To configure for TLS bootstrapping and optional automatic approval, you must configure options on the following components:
kubelet
in-cluster resources:
and potentially
In addition, you need your Kubernetes Certificate Authority (CA).
Certificate Authority
As without bootstrapping, you will need a Certificate Authority (CA) key and certificate. As without bootstrapping, these will be used
to sign the kubelet certificate. As before, it is your responsibility to distribute them to control plane nodes.
For the purposes of this document, we will assume these have been distributed to control plane nodes at
/var/lib/kubernetes/ca.pem
(certificate) and
/var/lib/kubernetes/ca-key.pem
(key).
We will refer to these as "Kubernetes CA certificate and key".
All Kubernetes components that use these certificates - kubelet, kube-apiserver, kube-controller-manager - assume the key and certificate to be PEM-encoded.
kube-apiserver configuration
The kube-apiserver has several requirements to enable TLS bootstrapping:
Recognizing CA that signs the client certificate
Authenticating the bootstrapping kubelet to the
group
Authorize the bootstrapping kubelet to create a certificate signing request (CSR)
Recognizing client certificates
This is normal for all client certificate authentication.
If not already set, add the
--client-ca-file=FILENAME
flag to the kube-apiserver command to enable
client certificate authentication, referencing a certificate authority bundle
containing the signing certificate, for example
--client-ca-file=/var/lib/kubernetes/ca.pem
.
Initial bootstrap authentication
In order for the bootstrapping kubelet to connect to kube-apiserver and request a certificate, it must first authenticate to the server.
You can use any
that can authenticate the kubelet.
While any authentication strategy can be used for the kubelet's initial
bootstrap credentials, the following two authenticators are recommended for ease
of provisioning.
Token authentication file
Using bootstrap tokens is a simpler and more easily managed method to authenticate kubelets, and does not require any additional flags when starting kube-apiserver.
Whichever method you choose, the requirement is that the kubelet be able to authenticate as a user with the rights to:
create and retrieve CSRs
be automatically approved to request node client certificates, if automatic approval is enabled.
A kubelet authenticating using bootstrap tokens is authenticated as a user in the group
, which is the standard method to use.
As this feature matures, you
should ensure tokens are bound to a Role Based Access Control (RBAC) policy
which limits requests (using the
bootstrap token
) strictly to client
requests related to certificate provisioning. With RBAC in place, scoping the
tokens to a group allows for great flexibility. For example, you could disable a
particular bootstrap group's access when you are done provisioning the nodes.
Bootstrap tokens
Bootstrap tokens are described in detail
here
. These are tokens that are stored as secrets in the Kubernetes cluster,
and then issued to the individual kubelet. You can use a single token for an entire cluster, or issue one per worker node.
The process is two-fold:
Create a Kubernetes secret with the token ID, secret and scope(s).
Issue the token to the kubelet
From the kubelet's perspective, one token is like another and h

as no special meaning.
From the kube-apiserver's perspective, however, the bootstrap token is special. Due to its
type
,
namespace
and
name
, kube-apiserver recognizes it as a special token,
and grants anyone authenticating with that token special bootstrap rights, notably treating them as a member of the
group. This fulfills a basic requirement
for TLS bootstrapping.
The details for creating the secret are available
here
.
If you want to use bootstrap tokens, you must enable it on kube-apiserver with the flag:
kube-apiserver has the ability to accept tokens as authentication.
These tokens are arbitrary but should represent at least 128 bits of entropy derived
from a secure random number generator (such as
/dev/urandom
on most modern Linux
systems). There are multiple ways you can generate a token. For example:
head -c
16
/dev/urandom | od -An -t x | tr -d
' '
This will generate tokens that look like
02b50b05283e98dd0fd71db496ef01e8
.
The token file should look like the following example, where the first three
values can be anything and the quoted group name should be as depicted:
02b50b05283e98dd0fd71db496ef01e8,kubelet-bootstrap,10001,"system:bootstrappers"
Add the
--token-auth-file=FILENAME
flag to the kube-apiserver command (in your
systemd unit file perhaps) to enable the token file. See docs
here
for
further details.
Authorize kubelet to create CSR
Now that the bootstrapping node is
authenticated
as part of the
group, it needs to be
authorized
to create a
certificate signing request (CSR) as well as retrieve it when done.
Fortunately, Kubernetes ships with a
with precisely these (and
only these) permissions,
.
To do this, you only need to create a
that binds the
group to the cluster role
.
# enable bootstrapping nodes to create CSR
:
kind
:
metadata
:
name
:
create-csrs-for-bootstrapping
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
kube-controller-manager configuration
While the apiserver receives the requests for certificates from the kubelet and authenticates those requests,
the controller-manager is responsible for issuing actual signed certificates.
The controller-manager performs this function via a certificate-issuing control loop.
This takes the form of a
cfssl
local signer using
assets on disk. Currently, all certificates issued have one year validity and a
default set of key usages.
In order for the controller-manager to sign certificates, it needs the following:
access to the "Kubernetes CA key and certificate" that you created and distributed
enabling CSR signing
Access to key and certificate
As described earlier, you need to create a Kubernetes CA key and certificate, and distribute it to the control plane nodes.
These will be used by the controller-manager to sign the kubelet certificates.
Since these signed certificates will, in turn, be used by the kubelet to authenticate as a regular kubelet to kube-apiserver, it is important that the CA
provided to the controller-manager at this stage also be trusted by kube-apiserver for authentication. This is provided to kube-apiserver
), as described in the kube-apiserver configuration section.
To provide the Kubernetes CA key and certificate to kube-controller-manager, use the following flags:
=
"/etc/path/to/kubernetes/ca/ca.crt"
=
"/etc/path/to/kubernetes/ca/ca.key"
=
"/var/lib/kubernetes/ca.pem"
=
"/var/lib/kubernetes/ca-key.pem"
The validity dur

ation of signed certificates can be configured with flag:
Approval
In order to approve CSRs, you need to tell the controller-manager that it is acceptable to approve them. This is done by granting
RBAC permissions to the correct group.
There are two distinct sets of permissions:
nodeclient
: If a node is creating a new certificate for a node, then it does not have a certificate yet. It is authenticating using one of the tokens listed above, and thus is part of the group
.
selfnodeclient
: If a node is renewing its certificate, then it already has a certificate (by definition), which it uses continuously to authenticate as part of the group
.
To enable the kubelet to request and receive a new certificate, create a
that binds the group in which the bootstrapping node is a member
to the
that grants it permission,
:
# Approve all CSRs for the group "system:bootstrappers"
:
kind
:
metadata
:
name
:
auto-approve-csrs-for-group
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
To enable the kubelet to renew its own client certificate, create a
that binds the group in which the fully functioning node is a member
to the
that
grants it permission,
:
# Approve renewal CSRs for the group "system:nodes"
:
kind
:
metadata
:
name
:
auto-approve-renewals-for-nodes
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
The
csrapproving
controller that ships as part of
and is enabled
by default. The controller uses the
API
to
determine if a given user is authorized to request a CSR, then approves based on
the authorization outcome. To prevent conflicts with other approvers, the
built-in approver doesn't explicitly deny CSRs. It only ignores unauthorized
requests. The controller also prunes expired certificates as part of garbage
Finally, with the control plane nodes properly set up and all of the necessary authentication and authorization in place, we can configure the kubelet.
The kubelet requires the following configuration to bootstrap:
A path to store the key and certificate it generates (optional, can use default)
A path to a
file that does not yet exist; it will place the bootstrapped config file here
A path to a bootstrap
file to provide the URL for the server and bootstrap credentials, e.g. a bootstrap token
Optional: instructions to rotate certificates
The bootstrap
should be in a path available to the kubelet, for example
/var/lib/kubelet/bootstrap-kubeconfig
.
Its format is identical to a normal
file. A sample file might look as follows:
:
v1
kind
:
Config
clusters
:
-
cluster
:
:
server
:
https://my.server.example.com:6443
name
:
bootstrap
contexts
:
-
context
:
cluster
:
bootstrap
user
:
kubelet-bootstrap
name
:
bootstrap
:
bootstrap
:
{}
users
:
-
name
:
user
:
token
:
: path to a CA file, used to validate the server certificate presented by kube-apiserver
server
: URL to kube-apiserver
token
: the token to use
The format of the token does not matter, as long as it matches what kube-

apiserver expects. In the above example, we used a bootstrap token.
As stated earlier,
any
valid authentication method can be used, not only tokens.
Because the bootstrap
is
a standard
, you can use
kubectl
to generate it. To create the above example file:
kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-cluster bootstrap --server='https://my.server.example.com:6443' --certificate-authority=/var/lib/kubernetes/ca.pem
kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-credentials kubelet-bootstrap --token=07401b.f395accd246ae52d
kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig set-context bootstrap --user=kubelet-bootstrap --cluster=bootstrap
kubectl config --kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig use-context bootstrap
To indicate to the kubelet to use the bootstrap
, use the following kubelet flag:
--bootstrap-kubeconfig="/var/lib/kubelet/bootstrap-kubeconfig" --kubeconfig="/var/lib/kubelet/kubeconfig"
When starting the kubelet, if the file specified via
does not
exist, the bootstrap kubeconfig specified via
--bootstrap-kubeconfig
is used
to request a client certificate from the API server. On approval of the
certificate request and receipt back by the kubelet, a kubeconfig file
referencing the generated key and obtained certificate is written to the path
specified by
. The certificate and key file will be placed in the
directory specified by
.
Client and Serving Certificates
All of the above relate to kubelet
client
certificates, specifically, the certificates a kubelet
uses to authenticate to kube-apiserver.
A kubelet also can use
serving
certificates. The kubelet itself exposes an https endpoint for certain features.
To secure these, the kubelet can do one of:
use provided key and certificate, via the
and
flags
create self-signed key and certificate, if a key and certificate are not provided
request serving certificates from the cluster server, via the CSR API
The client certificate provided by TLS bootstrapping is signed, by default, for
only, and thus cannot
be used as serving certificates, or
server auth
.
However, you
can
enable its server certificate, at least partially, via certificate rotation.
Kubernetes v1.8 and higher kubelet implements features for enabling
rotation of its client and/or serving certificates. Note, rotation of serving
certificate is a
beta
feature and requires the
RotateKubeletServerCertificate
feature flag on the kubelet (enabled by default).
You can configure the kubelet to rotate its client certificates by creating new CSRs
as its existing credentials expire. To enable this feature, use the
field of
or pass the following command line argument to the kubelet (deprecated):
--rotate-certificates
Enabling
causes the kubelet
both
to request a serving
certificate after bootstrapping its client credentials
and
to rotate that
certificate. To enable this behavior, use the field
of
the
--rotate-server-certificates
Note:
The CSR approving controllers implemented in core Kubernetes do not
approve node
serving
certificates for
security
reasons
. To use
operators need to run a custom approving
controller, or manually approve the serving certificate requests.
A deployment-specific approval process for kubelet serving certificates should typically only approve CSRs which:
are requested by nodes (ensure the
spec.username
field is of the form
and
contains
)
request usages for a serving certificate (ensure
contains
,
optionally contains
digital signature
and
key encipherment
, and contains no other usages)
on

ly have IP and DNS subjectAltNames that belong to the requesting node,
and have no URI and Email subjectAltNames (parse the x509 Certificate Signing Request
in
to verify
subjectAltNames
)
Other authenticating components
All of TLS bootstrapping described in this document relates to the kubelet. However,
other components may need to communicate directly with kube-apiserver. Notable is kube-proxy, which
is part of the Kubernetes node components and runs on every node, but may also include other components such as monitoring or networking.
Like the kubelet, these other components also require a method of authenticating to kube-apiserver.
You have several options for generating these credentials:
The old way: Create and distribute certificates the same way you did for kubelet before TLS bootstrapping
DaemonSet: Since the kubelet itself is loaded on each node, and is sufficient to start base services, you can run kube-proxy and other node-specific services not as a standalone process, but rather as a daemonset in the
namespace. Since it will be in-cluster, you can give it a proper service account with appropriate permissions to perform its activities. This may be the simplest way to configure such services.
kubectl approval
CSRs can be approved outside of the approval flows built into the controller
manager.
The signing controller does not immediately sign all certificate requests.
Instead, it waits until they have been flagged with an "Approved" status by an
appropriately-privileged user. This flow is intended to allow for automated
approval handled by an external approval controller or the approval controller
implemented in the core controller-manager. However cluster administrators can
also manually approve certificate requests using kubectl. An administrator can
list CSRs with
and describe one in detail with
kubectl describe csr &lt;name&gt;
. An administrator can approve or deny a CSR with
kubectl certificate approve &lt;name&gt;
and
kubectl certificate deny &lt;name&gt;
.
3.15 - Validating Admission Policy
This page provides an overview of Validating Admission Policy.
What is Validating Admission Policy?
Validating admission policies offer a declarative, in-process alternative to validating admission webhooks.
Validating admission policies use the Common Expression Language (CEL) to declare the validation
rules of a policy.
Validation admission policies are highly configurable, enabling policy authors to define policies
that can be parameterized and scoped to resources as needed by cluster administrators.
What Resources Make a Policy
A policy is generally made up of three resources:
The
describes the abstract logic of a policy
(think: "this policy makes sure a particular label is set to a particular value").
A
ValidatingAdmissionPolicyBinding
links the above resources together and provides scoping.
If you only want to require an
owner
label to be set for
Pods
, the binding is where you would
specify this restriction.
A parameter resource provides information to a ValidatingAdmissionPolicy to make it a concrete
statement (think "the
owner
label must be set to something that ends in
.company.com
").
A native type such as ConfigMap or a CRD defines the schema of a parameter resource.
objects specify what Kind they are expecting for their parameter resource.
At least a
and a corresponding
must be defined for a policy to have an effect.
If a
does not need to be configured via parameters, simply leave
spec.paramKind
in
unset.
Ensure the
Getting Started with Validating Admission Policy
Validating Admission Policy is part of the cluster control-plane. You sho

uld write and deploy them
with great caution. The following describes how to quickly experiment with Validating Admission Policy.
Creating a ValidatingAdmissionPolicy
The following is an example of a ValidatingAdmissionPolicy.
:
kind
:
metadata
:
name
:
"demo-policy.example.com"
spec
:
:
Fail
matchConstraints
:
:
-
apiGroups
:
[
"apps"
]
:
[
"v1"
]
:
[
"CREATE"
,
"UPDATE"
]
resources
:
[
]
validations
:
-
:
"object.spec.replicas &lt;= 5"
spec.validations
contains CEL expressions which use the
Common Expression Language (CEL)
to validate the request. If an expression evaluates to false, the validation check is enforced
according to the
spec.failurePolicy
field.
To configure a validating admission policy for use in a cluster, a binding is required.
The following is an example of a ValidatingAdmissionPolicyBinding.:
:
kind
:
metadata
:
name
:
"demo-binding-test.example.com"
spec
:
policyName
:
matchResources
:
:
:
:
test
When trying to create a deployment with replicas set not satisfying the validation expression, an
error will return containing message:
ValidatingAdmissionPolicy 'demo-policy.example.com' with binding 'demo-binding-test.example.com' denied request: failed expression: object.spec.replicas &lt;= 5
The above provides a simple example of using ValidatingAdmissionPolicy without a parameter configured.
Parameter resources
Parameter resources allow a policy configuration to be separate from its definition.
A policy can define paramKind, which outlines GVK of the parameter resource,
and then a policy binding ties a policy by name (via policyName) to a particular parameter resource via paramRef.
If parameter configuration is needed, the following is an example of a ValidatingAdmissionPolicy
with parameter configuration.
:
kind
:
metadata
:
name
:
"replicalimit-policy.example.com"
spec
:
:
Fail
paramKind
:
:
rules.example.com/v1
kind
:
ReplicaLimit
:
:
-
apiGroups
:
[
"apps"
]
:
[
"v1"
]
:
[
"CREATE"
,
"UPDATE"
]
resources
:
[
]
:
-
:
"object.spec.replicas &lt;= params.maxReplicas"
reason
:
Invalid
The
field of the ValidatingAdmissionPolicy specifies the kind of resources used
to parameterize this policy. For this example, it is configured by ReplicaLimit custom resources.
Note in this example how the CEL expression references the parameters via the CEL params variable,
e.g.
params.maxReplicas
.
spec.matchConstraints
specifies what resources this policy is
designed to validate. Note that the native types such like
ConfigMap
could also be used as
parameter reference.
The
fields contain CEL expressions. If an expression evaluates to false, the
validation check is enforced according to the
field.
The validating admission policy author is responsible for providing the ReplicaLimit parameter CRD.
To configure an validating admission policy for use in a cluster, a binding and parameter resource
are created. The following is an example of a ValidatingAdmissionPolicyBinding.
:
kind
:
metadata
:
name
:
"replicalimit-binding-test.example.com"
spec
:
:
paramRef
:
name
:
"replica-limit-test.example.com"
:
:
:
:
test
The parameter resource could be as following:
:
kind
:
metadata
:
name
:
:
3
This policy parameter resource limits depl

oyments to a max of 3 replicas in all namespaces in the
test environment. An admission policy may have multiple bindings. To bind all other environments
environment to have a maxReplicas limit of 100, create another ValidatingAdmissionPolicyBinding:
:
kind
:
metadata
:
name
:
"replicalimit-binding-nontest"
spec
:
:
paramRef
:
name
:
"replica-limit-clusterwide.example.com"
:
:
:
-
key
:
environment,
operator
:
NotIn,
values
:
[
"test"
]
And have a parameter resource like:
:
kind
:
metadata
:
name
:
:
100
Bindings can have overlapping match criteria. The policy is evaluated for each matching binding.
In the above example, the "nontest" policy binding could instead have been defined as a global policy:
:
kind
:
metadata
:
name
:
"replicalimit-binding-global"
spec
:
:
params
:
:
:
:
-
key
:
operator
:
Exists
The params object representing a parameter resource will not be set if a parameter resource has
not been bound, so for policies requiring a parameter resource, it can be useful to add a check to
ensure one has been bound.
For the use cases require parameter configuration, we recommend to add a param check in
spec.validations[0].expression
:
- expression: "params != null"
  message: "params missing but required to bind to this policy"
It can be convenient to be able to have optional parameters as part of a parameter resource, and
only validate them if present. CEL provides
has()
, which checks if the key passed to it exists.
CEL also implements Boolean short-circuiting. If the first half of a logical OR evaluates to true,
it won’t evaluate the other half (since the result of the entire OR will be true regardless).
Combining the two, we can provide a way to validate optional parameters:
!has(params.optionalNumber) || (params.optionalNumber &gt;= 5 &amp;&amp; params.optionalNumber &lt;= 10)
Here, we first check that the optional parameter is present with
!has(params.optionalNumber)
.
If
optionalNumber
hasn’t been defined, then the expression short-circuits since
will evaluate to true.
If
has been defined, then the latter half of the CEL expression will be
evaluated, and optionalNumber will be checked to ensure that it contains a value between 5 and
10 inclusive.
Authorization Check
We introduced the authorization check for parameter resources.
User is expected to have
read
access to the resources referenced by
paramKind
in
and
paramRef
in
.
Note that if a resource in
paramKind
fails resolving via the restmapper,
read
access to all
resources of groups is required.
Failure Policy
defines how mis-configurations and CEL expressions evaluating to error from the
admission policy are handled. Allowed values are
Ignore
or
Fail
.
Ignore
means that an error calling the ValidatingAdmissionPolicy is ignored and the API
request is allowed to continue.
Fail
means that an error calling the ValidatingAdmissionPolicy causes the admission to fail
and the API request to be rejected.
is defined inside
:
:
kind
:
spec
:
...
:
Ignore
# The default is "Fail"
:
-
:
"object.spec.xyz == params.x"
Validation Expression
spec.validations[i].expression
represents the expression which will be evaluated by CEL.
To learn more, see the
CEL language specification
CEL expressions have access to

 the contents of the Admission request/response, organized into CEL
variables as well as some other useful variables:
'object' - The object from the incoming request. The value is null for DELETE requests.
'oldObject' - The existing object. The value is null for CREATE requests.
'request' - Attributes of the
admission request
.
'params' - Parameter resource referred to by the policy binding being evaluated. The value is
null if
ParamKind
is unset.
The
,
kind
,
and
are always accessible from
the root of the object. No other metadata properties are accessible.
are accessible.
Accessible property names are escaped according to the following rules when accessed in the
expression:
escape sequence
property name equivalent
__underscores__
__
__dot__
.
__dash__
-
__slash__
/
__{keyword}__
CEL RESERVED keyword
Note:
A
CEL reserved
keyword only needs to be escaped if the token is an exact match
for the reserved keyword.
int
in the word “sprint” would not be escaped.
Examples on escaping:
property name
rule with escaped property name
namespace
self.__namespace__ &gt; 0
x-prop
self.x__dash__prop &gt; 0
redact__d
self.redact__underscores__d &gt; 0
string
self.startsWith('kube')
Equality on arrays with list type of 'set' or 'map' ignores element order, i.e. [1, 2] == [2, 1].
Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:
- 'set':
X + Y
X
Y
- 'map':
X + Y
X
Y
X
and
Y
Y
with
Validation expression examples
Expression
Purpose
object.minReplicas &lt;= object.replicas &amp;&amp; object.replicas &lt;= object.maxReplicas
Validate that the three fields defining replicas are ordered appropriately
'Available' in object.stateCounts
Validate that an entry with the 'Available' key exists in a map
(size(object.list1) == 0) != (size(object.list2) == 0)
Validate that one of two lists is non-empty, but not both
!('MY_KEY' in object.map1) || object['MY_KEY'].matches('^[a-zA-Z]*$')
Validate the value of a map for a specific key, if it is in the map
object.envars.filter(e, e.name == 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$')
Validate the 'value' field of a listMap entry where key field 'name' is 'MY_ENV'
has(object.expired) &amp;&amp; object.created + object.ttl &lt; object.expired
Validate that 'expired' date is after a 'create' date plus a 'ttl' duration
object.health.startsWith('ok')
Validate a 'health' string field has the prefix 'ok'
object.widgets.exists(w, w.key == 'x' &amp;&amp; w.foo &lt; 10)
Validate that the 'foo' property of a listMap item with a key 'x' is less than 10
type(object) == string ? object == '100%' : object == 1000
Validate an int-or-string field for both the int and string cases
object.metadata.name.startsWith(object.prefix)
Validate that an object's name has the prefix of another field value
object.set1.all(e, !(e in object.set2))
Validate that two listSets are disjoint
size(object.names) == size(object.details) &amp;&amp; object.names.all(n, n in object.details)
Validate the 'details' map is keyed by the items in the 'names' listSet
size(object.clusters.filter(c, c.name == object.primary)) == 1
Validate that the 'primary' property has one and only one occurrence in the 'clusters' listMap
Read
Supported evaluation on CEL
for more information about CEL rules.
spec.validation[i].reason
represents a machine-readable description of why this validation failed.
If this is the first validation in the list to fail, this reason, as well as the corresponding
HTTP response code, are used in the HTTP response

 to the client.
The currently supported reasons are:
Unauthorized
,
Forbidden
,
Invalid
,
RequestEntityTooLarge
.
If not set,
StatusReasonInvalid
is used in the response to the client.
4 - Well-Known Labels, Annotations and Taints
Kubernetes reserves all labels and annotations in the kubernetes.io and k8s.io namespaces.
This document serves both as a reference to the values and as a coordination point for assigning values.
Labels, annotations and taints used on API objects
Example:
app.kubernetes.io/component: "database"
Used on: All Objects (typically used on
workload resources
).
The component within the architecture.
One of the
recommended labels
.
app.kubernetes.io/created-by (deprecated)
Example:
app.kubernetes.io/created-by: "controller-manager"
).
The controller/user who created this resource.
Note:
Starting from v1.9, this label is deprecated.
Example:
app.kubernetes.io/instance: "mysql-abcxzy"
).
A unique name identifying the instance of an application. To assign a non-unique name, use
.
One of the
.
Example:
app.kubernetes.io/managed-by: "helm"
).
The tool being used to manage the operation of an application.
One of the
.
Example:
app.kubernetes.io/name: "mysql"
).
The name of the application.
One of the
.
Example:
app.kubernetes.io/part-of: "wordpress"
).
The name of a higher-level application this one is part of.
One of the
.
Example:
app.kubernetes.io/version: "5.7.21"
).
The current version of the application.
Common forms of values include:
semantic version
the Git
revision hash
for the source code.
One of the
.
cluster-autoscaler.kubernetes.io/safe-to-evict
Example:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
Used on: Pod
When this annotation is set to
"true"
, the cluster autoscaler is allowed to evict a Pod
even if other rules would normally prevent that.
The cluster autoscaler never evicts Pods that have this annotation explicitly set to
"false"
; you could set that on an important Pod that you want to keep running.
If this annotation is not set then the cluster autoscaler follows its Pod-level behavior.
Example:
kubernetes.io/arch: "amd64"
Used on: Node
The Kubelet populates this with
runtime.GOARCH
as defined by Go. This can be handy if you are mixing arm and x86 nodes.
Example:
kubernetes.io/os: "linux"
runtime.GOOS
as defined by Go. This can be handy if you are mixing operating systems in your cluster (for example: mixing Linux and Windows nodes).
Example:
kubernetes.io/metadata.name: "mynamespace"
Used on: Namespaces
The Kubernetes API server (part of the
)
sets this label on all namespaces. The label value is set
to the name of the namespace. You can't change this label's value.
This is useful if you want to target a specific namespace with a label
selector
.
kubernetes.io/limit-ranger
Example:
kubernetes.io/limit-ranger: "LimitRanger plugin set: cpu, memory request for container nginx; cpu, memory limit for container nginx"
Kubernetes by default doesn't provide any resource limit, that means unless you explicitly define limits,
your container can consume unlimited CPU and memory.
You can define a default request or default limit for pods. You do this by creating a LimitRange in the relevant namespace.
Pods deployed after you define a Lim

itRange will have these limits applied to them.
The annotation
records that resource defaults were specified for the Pod,
and they were applied successfully.
For more details, read about
LimitRanges
.
beta.kubernetes.io/arch (deprecated)
This label has been deprecated. Please use
instead.
beta.kubernetes.io/os (deprecated)
instead.
kube-aggregator.kubernetes.io/automanaged
Example:
kube-aggregator.kubernetes.io/automanaged: "onstart"
Used on: APIService
The
sets this label on any APIService object that the API server has created automatically. The label marks how the control plane should manage that APIService. You should not add, modify, or remove this label by yourself.
Note:
Automanaged APIService objects are deleted by kube-apiserver when it has no built-in or custom resource API corresponding to the API group/version of the APIService.
There are two possible values:
onstart
: The APIService should be reconciled when an API server starts up, but not otherwise.
true
: The API server should reconcile this APIService continuously.
Example:
kubernetes.io/hostname: "ip-172-20-114-199.ec2.internal"
The Kubelet populates this label with the hostname. Note that the hostname can be changed from the "actual" hostname by passing the
flag to the
kubelet
.
This label is also used as part of the topology hierarchy. See
kubernetes.io/change-cause
Example:
kubernetes.io/change-cause: "kubectl edit --record deployment foo"
Used on: All Objects
This annotation is a best guess at why something was changed.
It is populated when adding
--record
to a
kubectl
command that may change an object.
Example:
kubernetes.io/description: "Description of K8s object."
This annotation is used for describing specific behaviour of given object.
kubernetes.io/enforce-mountable-secrets
Example:
kubernetes.io/enforce-mountable-secrets: "true"
Used on: ServiceAccount
The value for this annotation must be
true
to take effect. This annotation indicates that pods running as this service account may only reference Secret API objects specified in the service account's
secrets
field.
node.kubernetes.io/exclude-from-external-load-balancer
Example:
Kubernetes automatically enables the
ServiceNodeExclusion
feature gate on the clusters it creates. With this feature gate enabled on a cluster,
you can add labels to particular worker nodes to exclude them from the list of backend servers.
The following command can be used to exclude a worker node from the list of backend servers in a backend set-
kubectl label nodes &lt;node-name&gt; node.kubernetes.io/exclude-from-external-load-balancers=true
controller.kubernetes.io/pod-deletion-cost
Example:
controller.kubernetes.io/pod-deletion-cost: "10"
This annotation is used to set
Pod Deletion Cost
which allows users to influence ReplicaSet downscaling order. The annotation parses into an
int32
type.
cluster-autoscaler.kubernetes.io/enable-ds-eviction
Example:
cluster-autoscaler.kubernetes.io/enable-ds-eviction: "true"
This annotation controls whether a DaemonSet pod should be evicted by a ClusterAutoscaler.
This annotation needs to be specified on DaemonSet pods in a DaemonSet manifest.
"true"
, the ClusterAutoscaler is allowed to evict a DaemonSet Pod,
even if other rules would normally prevent that. To disallow the ClusterAutoscaler from evicting DaemonSet pods,
you can set this annotation to
"false"
for important DaemonSet pods.
If this annotation is not set, then the Cluster Autoscaler follows its overall behaviour (i.e evict the DaemonSets based on its configuration).
Note:
This annotation only impacts DaemonSet pods.
kubernetes.io/in

gress-bandwidth
Note:
Ingress traffic shaping annotation is an experimental feature.
If you want to enable traffic shaping support, you must add the
bandwidth
plugin to your CNI configuration file (default
/etc/cni/net.d
) and
ensure that the binary is included in your CNI bin dir (default
/opt/cni/bin
).
Example:
kubernetes.io/ingress-bandwidth: 10M
You can apply quality-of-service traffic shaping to a pod and effectively limit its available bandwidth.
Ingress traffic (to the pod) is handled by shaping queued packets to effectively handle data.
To limit the bandwidth on a pod, write an object definition JSON file and specify the data traffic
speed using
kubernetes.io/ingress-bandwidth
annotation. The unit used for specifying ingress
rate is bits per second, as a
Quantity
.
10M
means 10 megabits per second.
kubernetes.io/egress-bandwidth
Note:
Egress traffic shaping annotation is an experimental feature.
bandwidth
) and
).
Example:
kubernetes.io/egress-bandwidth: 10M
Egress traffic (from the pod) is handled by policing, which simply drops packets in excess of the configured rate.
The limits you place on a pod do not affect the bandwidth of other pods.
annotation. The unit used for specifying egress
Quantity
.
10M
beta.kubernetes.io/instance-type (deprecated)
Note:
Starting in v1.17, this label is deprecated in favor of
.
Example:
node.kubernetes.io/instance-type: "m3.medium"
The Kubelet populates this with the instance type as defined by the
cloudprovider
.
This will be set only if you are using a
. This setting is handy
if you want to target certain workloads to certain instance types, but typically you want
to rely on the Kubernetes scheduler to perform resource-based scheduling. You should aim to schedule based on properties rather than on instance types (for example: require a GPU, instead of requiring a
g2.2xlarge
).
failure-domain.beta.kubernetes.io/region (deprecated)
See
.
Note:
.
failure-domain.beta.kubernetes.io/zone (deprecated)
See
.
Note:
.
statefulset.kubernetes.io/pod-name
Example:
statefulset.kubernetes.io/pod-name: "mystatefulset-7"
When a StatefulSet controller creates a Pod for the StatefulSet, the control plane
sets this label on that Pod. The value of the label is the name of the Pod being created.
See
Pod Name Label
in the
StatefulSet topic for more details.
Example:
scheduler.alpha.kubernetes.io/node-selector: "name-of-node-selector"
Used on: Namespace
The
uses this annotation key to assign node selectors to pods in namespaces.
Example:
topology.kubernetes.io/region: "us-east-1"
See
.
Example:
topology.kubernetes.io/zone: "us-east-1c"
Used on: Node, PersistentVolume
On Node: The
kubelet
or the external
populates this with the information as provided by the
. This will be set only if you are using a
. However, you should consider setting this on nodes if it makes sense in your topology.
On PersistentVolume: topology-aware volume provisioners will automatically set node affinity constraints on
.
A zone represents a logical failur

e domain. It is common for Kubernetes clusters to span multiple zones for increased availability. While the exact definition of a zone is left to infrastructure implementations, common properties of a zone include very low network latency within a zone, no-cost network traffic within a zone, and failure independence from other zones. For example, nodes within a zone might share a network switch, but nodes in different zones should not.
A region represents a larger domain, made up of one or more zones. It is uncommon for Kubernetes clusters to span multiple regions, While the exact definition of a zone or region is left to infrastructure implementations, common properties of a region include higher network latency between them than within them, non-zero cost for network traffic between them, and failure independence from other zones or regions. For example, nodes within a region might share power infrastructure (e.g. a UPS or generator), but nodes in different regions typically would not.
Kubernetes makes a few assumptions about the structure of zones and regions:
regions and zones are hierarchical: zones are strict subsets of regions and no zone can be in 2 regions
zone names are unique across regions; for example region "africa-east-1" might be comprised of zones "africa-east-1a" and "africa-east-1b"
It should be safe to assume that topology labels do not change. Even though labels are strictly mutable, consumers of them can assume that a given node is not going to be moved between zones without being destroyed and recreated.
Kubernetes can use this information in various ways. For example, the scheduler automatically tries to spread the Pods in a ReplicaSet across nodes in a single-zone cluster (to reduce the impact of node failures, see
). With multiple-zone clusters, this spreading behavior also applies to zones (to reduce the impact of zone failures). This is achieved via
SelectorSpreadPriority
.
is a best effort placement. If the zones in your cluster are heterogeneous (for example: different numbers of nodes, different types of nodes, or different pod resource requirements), this placement might prevent equal spreading of your Pods across zones. If desired, you can use homogenous zones (same number and types of nodes) to reduce the probability of unequal spreading.
The scheduler (through the
VolumeZonePredicate
predicate) also will ensure that Pods, that claim a given volume, are only placed into the same zone as that volume. Volumes cannot be attached across zones.
If
does not support automatic labeling of your PersistentVolumes, you should consider
adding the labels manually (or adding support for
). With
, the scheduler prevents Pods from mounting volumes in a different zone. If your infrastructure doesn't have this constraint, you don't need to add the zone labels to the volumes at all.
volume.beta.kubernetes.io/storage-provisioner (deprecated)
Example:
volume.beta.kubernetes.io/storage-provisioner: "k8s.io/minikube-hostpath"
Used on: PersistentVolumeClaim
This annotation has been deprecated.
volume.beta.kubernetes.io/mount-options (deprecated)
Example :
volume.beta.kubernetes.io/mount-options: "ro,soft"
Used on: PersistentVolume
A Kubernetes administrator can specify additional
mount options
for when a PersistentVolume is mounted on a node.
volume.kubernetes.io/storage-provisioner
This annotation will be added to dynamic provisioning required PVC.
If a node has set the annotation
on itself, then its storage attach and detach operations are being managed
by the
volume attach/detach
running within the
.
The value of the annotation isn't important; if this annotation exists on a node,

then storage attaches and detaches are controller managed.
node.kubernetes.io/windows-build
Example:
node.kubernetes.io/windows-build: "10.0.17763"
When the kubelet is running on Microsoft Windows, it automatically labels its node to record the version of Windows Server in use.
The label's value is in the format "MajorVersion.MinorVersion.BuildNumber".
service.kubernetes.io/headless
Example:
service.kubernetes.io/headless: ""
Used on: Service
The control plane adds this label to an Endpoints object when the owning Service is headless.
kubernetes.io/service-name
Example:
kubernetes.io/service-name: "my-website"
Used on: EndpointSlice
Kubernetes associates
with
Services
using this label.
This label records the
name
of the
Service that the EndpointSlice is backing. All EndpointSlices should have this label set to
the name of their associated Service.
Example:
kubernetes.io/service-account.name: "sa-name"
Used on: Secret
This annotation records the
name
of the
ServiceAccount that the token (stored in the Secret of type
) represents.
kubernetes.io/service-account.uid
Example:
kubernetes.io/service-account.uid: da68f9c6-9d26-11e7-b84e-002dc52800da
unique ID
of the
kubernetes.io/legacy-token-last-used
Example:
kubernetes.io/legacy-token-last-used: 2022-10-24
The control plane only adds this label for Secrets that have the type
.
The value of this label records the date (ISO 8601 format, UTC time zone) when the control plane last saw
a request where the client authenticated using the service account token.
If a legacy token was last used before the cluster gained the feature (added in Kubernetes v1.26), then
the label isn't set.
endpointslice.kubernetes.io/managed-by
Example:
endpointslice.kubernetes.io/managed-by: "controller"
Used on: EndpointSlices
The label is used to indicate the controller or entity that manages an EndpointSlice. This label aims to enable different EndpointSlice objects to be managed by different controllers or entities within the same cluster.
endpointslice.kubernetes.io/skip-mirror
Example:
endpointslice.kubernetes.io/skip-mirror: "true"
Used on: Endpoints
The label can be set to
"true"
on an Endpoints resource to indicate that the EndpointSliceMirroring controller should not mirror this resource with EndpointSlices.
service.kubernetes.io/service-proxy-name
Example:
service.kubernetes.io/service-proxy-name: "foo-bar"
The kube-proxy has this label for custom proxy, which delegates service control to custom proxy.
experimental.windows.kubernetes.io/isolation-type (deprecated)
Example:
experimental.windows.kubernetes.io/isolation-type: "hyperv"
The annotation is used to run Windows containers with Hyper-V isolation. To use Hyper-V isolation feature and create a Hyper-V isolated container, the kubelet should be started with feature gates HyperVContainer=true and the Pod should include the annotation
experimental.windows.kubernetes.io/isolation-type: hyperv
.
Note:
You can only set this annotation on Pods that have a single container.
Starting from v1.20, this annotation is deprecated. Experimental Hyper-V support was removed in 1.21.
ingressclass.kubernetes.io/is-default-class
Example:
ingressclass.kubernetes.io/is-default-class: "true"
Used on: IngressClass
When a single IngressClass resource has this annotation set to
"true"
, new Ingress resource without a class specified will be assigned this default class.
kubernetes.io/ingress.class (deprecated)
Note:
Starting in v1.18, this annotation is deprecated in favor of
spec.ingressClassName
.
Example:
storageclass.kubernetes.io/is-default-class: "true"
Used on: StorageClass
When a single Stora

geClass resource has this annotation set to
"true"
, new PersistentVolumeClaim
resource without a class specified will be assigned this default class.
alpha.kubernetes.io/provided-node-ip
Example:
alpha.kubernetes.io/provided-node-ip: "10.0.0.1"
The kubelet can set this annotation on a Node to denote its configured IPv4 address.
When kubelet is started with the
flag set to any value (includes both external and legacy in-tree cloud providers), it sets this annotation on the Node to denote an IP address set from the command line flag (
--node-ip
). This IP is verified with the cloud provider as valid by the cloud-controller-manager.
Example:
batch.kubernetes.io/job-completion-index: "3"
The Job controller in the kube-controller-manager sets this annotation for Pods
created with Indexed
completion mode
.
kubectl.kubernetes.io/default-container
Example:
kubectl.kubernetes.io/default-container: "front-end-app"
The value of the annotation is the container name that is default for this Pod. For example,
or
without
-c
or
flag will use this default container.
endpoints.kubernetes.io/over-capacity
Example:
endpoints.kubernetes.io/over-capacity:truncated
The
adds this annotation to an
Endpoints
object if the associated
Service
has more than 1000 backing endpoints. The annotation indicates that the Endpoints object is over capacity and the number of endpoints has been truncated to 1000.
If the number of backend endpoints falls below 1000, the control plane removes this annotation.
batch.kubernetes.io/job-tracking (deprecated)
Example:
batch.kubernetes.io/job-tracking: ""
Used on: Jobs
The presence of this annotation on a Job indicates that the control plane is
tracking the Job status using finalizers
.
The control plane uses this annotation to safely transition to tracking Jobs
using finalizers, while the feature is in development.
You should
not
manually add or remove this annotation.
Note:
Starting from Kubernetes 1.26, this annotation is deprecated.
Kubernetes 1.27 and newer will ignore this annotation and always track Jobs
using finalizers.
Example:
scheduler.alpha.kubernetes.io/defaultTolerations: '[{"operator": "Equal", "value": "value1", "effect": "NoSchedule", "key": "dedicated-node"}]'
This annotation requires the
admission controller to be enabled. This annotation key allows assigning tolerations to a namespace and any new pods created in this namespace would get these tolerations added.
scheduler.alpha.kubernetes.io/preferAvoidPods (deprecated)
Used on: Nodes
NodePreferAvoidPods scheduling plugin
to be enabled. The plugin is deprecated since Kubernetes 1.22.
Use
instead.
The taints listed below are always used on Nodes
Example:
node.kubernetes.io/not-ready: "NoExecute"
The node controller detects whether a node is ready by monitoring its health and adds or removes this taint accordingly.
Example:
node.kubernetes.io/unreachable: "NoExecute"
The node controller adds the taint to a node corresponding to the
NodeCondition
Ready
being
Unknown
.
node.kubernetes.io/unschedulable
Example:
node.kubernetes.io/unschedulable: "NoSchedule"
The taint will be added to a node when initializing the node to avoid race condition.
node.kubernetes.io/memory-pressure
Example:
node.kubernetes.io/memory-pressure: "NoSchedule"
The kubelet detects memory pressure based on
and
allocatableMemory.available
observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed.
node.kubernetes.io/disk-pressure
Example:
node.kubernetes.io/disk-pressure :"NoSchedule"
The 

kubelet detects disk pressure based on
imagefs.available
,
imagefs.inodesFree
,
nodefs.available
and
nodefs.inodesFree
(Linux only) observed on a Node. The observed values are then compared to the corresponding thresholds that can be set on the kubelet to determine if the Node condition and taint should be added/removed.
node.kubernetes.io/network-unavailable
Example:
node.kubernetes.io/network-unavailable: "NoSchedule"
This is initially set by the kubelet when the cloud provider used indicates a requirement for additional network configuration. Only when the route on the cloud is configured properly will the taint be removed by the cloud provider.
node.kubernetes.io/pid-pressure
Example:
node.kubernetes.io/pid-pressure: "NoSchedule"
The kubelet checks D-value of the size of
/proc/sys/kernel/pid_max
and the PIDs consumed by Kubernetes on a node to get the number of available PIDs that referred to as the
pid.available
metric. The metric is then compared to the corresponding threshold that can be set on the kubelet to determine if the node condition and taint should be added/removed.
node.kubernetes.io/out-of-service
Example:
node.kubernetes.io/out-of-service:NoExecute
A user can manually add the taint to a Node marking it out-of-service. If the
NodeOutOfServiceVolumeDetach
is enabled on
, and a Node is marked out-of-service with this taint, the pods on the node will be forcefully deleted if there are no matching tolerations on it and volume detach operations for the pods terminating on the node will happen immediately. This allows the Pods on the out-of-service node to recover quickly on a different node.
Caution:
Refer to
Non-graceful node shutdown
for further details about when and how to use this taint.
Example:
node.cloudprovider.kubernetes.io/uninitialized: "NoSchedule"
Sets this taint on a node to mark it as unusable, when kubelet is started with the "external" cloud provider, until a controller from the cloud-controller-manager initializes this node, and then removes the taint.
node.cloudprovider.kubernetes.io/shutdown
Example:
node.cloudprovider.kubernetes.io/shutdown: "NoSchedule"
If a Node is in a cloud provider specified shutdown state, the Node gets tainted accordingly with
and the taint effect of
.
Example:
pod-security.kubernetes.io/enforce: "baseline"
Value
must
be one of
,
baseline
, or
which correspond to
levels. Specifically,
the
enforce
label
prohibits
the creation of any Pod in the labeled Namespace which does not meet
the requirements outlined in the indicated level.
See
Enforcing Pod Security at the Namespace Level
Example:
pod-security.kubernetes.io/enforce-version: "1.26"
Value
must
be
latest
or a valid Kubernetes version in the format
v&lt;MAJOR&gt;.&lt;MINOR&gt;
.
This determines the version of the
policies to apply when validating a submitted Pod.
See
Example:
pod-security.kubernetes.io/audit: "baseline"
Value
must
be one of
,
baseline
, or
the
audit
label does not prevent the creation of a Pod in the labeled Namespace which does not meet
the requirements outlined in the indicated level, but adds an audit annotation to that Pod.
See
Example:
pod-security.kubernetes.io/audit-version: "1.26"
Value
must
be
latest
.

See
Example:
pod-security.kubernetes.io/warn: "baseline"
Value
must
be one of
,
baseline
, or
the
warn
label does not prevent the creation of a Pod in the labeled Namespace which does not meet the
requirements outlined in the indicated level, but returns a warning to the user after doing so.
Note that warnings are also displayed when creating or updating objects that contain Pod templates,
such as Deployments, Jobs, StatefulSets, etc.
See
Example:
pod-security.kubernetes.io/warn-version: "1.26"
Value
must
be
latest
.
policies to apply when validating a submitted Pod. Note that warnings are also displayed when creating
or updating objects that contain Pod templates, such as Deployments, Jobs, StatefulSets, etc.
See
kubernetes.io/psp (deprecated)
Example:
kubernetes.io/psp: restricted
This annotation was only relevant if you were using
.
Kubernetes v1.26 does not support the PodSecurityPolicy API.
When the PodSecurityPolicy admission controller admitted a Pod, the admission controller
modified the Pod to have this annotation.
The value of the annotation was the name of the PodSecurityPolicy that was used for validation.
seccomp.security.alpha.kubernetes.io/pod (deprecated)
This annotation has been deprecated since Kubernetes v1.19 and will become non-functional in a future release.
please use the corresponding pod or container
field in the Pod specification.
The
field within a Pod's
.spec
defines pod-level security attributes.
When you
specify the security context for a Pod
,
the settings you specify apply to all containers in that Pod.
container.seccomp.security.alpha.kubernetes.io/[NAME] (deprecated)
The tutorial
takes
you through the steps you follow to apply a seccomp profile to a Pod or to one of
its containers. That tutorial covers the supported mechanism for configuring seccomp in Kubernetes,
based on setting
within the Pod's
.spec
.
snapshot.storage.kubernetes.io/allowVolumeModeChange
Example:
snapshot.storage.kubernetes.io/allowVolumeModeChange: "true"
Used on: VolumeSnapshotContent
Value can either be
true
or
false
.
This determines whether a user can modify the mode of the source volume when a
is being
created from a VolumeSnapshot.
Refer to
Converting the volume mode of a Snapshot
and the
Kubernetes CSI Developer Documentation
scheduler.alpha.kubernetes.io/critical-pod (deprecated)
Example:
scheduler.alpha.kubernetes.io/critical-pod: ""
This annotation lets Kubernetes control plane know about a pod being a critical pod so that the descheduler will not remove this pod.
Note:
Starting in v1.16, this annotation was removed in favor of
Pod Priority
.
Annotations used for audit
authorization.k8s.io/decision
authorization.k8s.io/reason
insecure-sha1.invalid-cert.kubernetes.io/$hostname
missing-san.invalid-cert.kubernetes.io/$hostname
pod-security.kubernetes.io/audit-violations
pod-security.kubernetes.io/enforce-policy
pod-security.kubernetes.io/exempt
See more details on the
Au

dit Annotations
page.
kubeadm
Example:
kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/container.sock
Annotation that kubeadm uses to preserve the CRI socket information given to kubeadm at
init
/
join
time for later use.
kubeadm annotates the Node object with this information. The annotation remains "alpha", since ideally this should
be a field in KubeletConfiguration instead.
kubeadm.kubernetes.io/etcd.advertise-client-urls
Example:
kubeadm.kubernetes.io/etcd.advertise-client-urls: https://172.17.0.18:2379
Annotation that kubeadm places on locally managed etcd pods to keep track of a list of URLs where etcd clients
should connect to. This is used mainly for etcd cluster health check purposes.
Example:
kubeadm.kubernetes.io/kube-apiserver.advertise-address.endpoint: https://172.17.0.18:6443
Annotation that kubeadm places on locally managed kube-apiserver pods to keep track of the exposed advertise
address/port endpoint for that API server instance.
kubeadm.kubernetes.io/component-config.hash
Used on: ConfigMap
Example:
kubeadm.kubernetes.io/component-config.hash: 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
Annotation that kubeadm places on ConfigMaps that it manages for configuring components. It contains a hash (SHA-256)
used to determine if the user has applied settings different from the kubeadm defaults for a particular component.
Label that kubeadm applies on the control plane nodes that it manages.
Example:
Taint that kubeadm applies on control plane nodes to allow only critical workloads to schedule on them.
node-role.kubernetes.io/master (deprecated)
Example:
Taint that kubeadm previously applied on control plane nodes to allow only critical workloads to schedule on them.
Replaced by
; kubeadm
no longer sets or uses this deprecated taint.
4.1 - Audit Annotations
This page serves as a reference for the audit annotations of the kubernetes.io
namespace. These annotations apply to
Event
object from API group
.
Note:
The following annotations are not used within the Kubernetes API. When you
enable auditing
audit event data is written using
Event
from API group
.
The annotations apply to audit events. Audit events are different from objects in the
Event API
(API group
events.k8s.io
).
Example:
pod-security.kubernetes.io/exempt: namespace
Value
must
be one of
user
,
namespace
, or
Pod Security Exemption
dimensions. This annotation indicates on which dimension was based the exemption
from the PodSecurity enforcement.
Example:
pod-security.kubernetes.io/enforce-policy: restricted:latest
Value
must
be
privileged:&lt;version&gt;
,
baseline:&lt;version&gt;
,
restricted:&lt;version&gt;
Standard
levels accompanied by
a version which
must
be
latest
. This annotations informs about the enforcement level that
allowed or denied the pod during PodSecurity admission.
See
Example:
pod-security.kubernetes.io/audit-violations: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "example" must set securityContext.allowPrivilegeEscalation=false), ...
Value details an audit policy violation, it contains the
level
that was transgressed as well as the specific policies on the fields that were
violated from the PodSecurity enfo

rcement.
See
Example:
authorization.k8s.io/decision: "forbid"
This annotation indicates whether or not a request was authorized in Kubernetes audit logs.
See
Auditing
Example:
authorization.k8s.io/reason: "Human-readable reason for the decision"
This annotation gives reason for the
decision
in Kubernetes audit logs.
See
Auditing
Example:
missing-san.invalid-cert.kubernetes.io/example-svc.example-namespace.svc: "relies on a legacy Common Name field instead of the SAN extension for subject validation"
Used by Kubernetes version v1.24 and later
This annotation indicates a webhook or aggregated API server
is using an invalid certificate that is missing
.
Support for these certificates was disabled by default in Kubernetes 1.19,
and removed in Kubernetes 1.23.
Requests to endpoints using these certificates will fail.
Services using these certificates should replace them as soon as possible
to avoid disruption when running in Kubernetes 1.23+ environments.
There's more information about this in the Go documentation:
X.509 CommonName deprecation
.
Example:
insecure-sha1.invalid-cert.kubernetes.io/example-svc.example-namespace.svc: "uses an insecure SHA-1 signature"
is using an insecure certificate signed with a SHA-1 hash.
Support for these insecure certificates is disabled by default in Kubernetes 1.24,
and will be removed in a future release.
Services using these certificates should replace them as soon as possible,
to ensure connections are secured properly and to avoid disruption in future releases.
Rejecting SHA-1 certificates
.
5 - Kubernetes API
Kubernetes' API is the application that serves Kubernetes functionality through a RESTful interface and stores the state of the cluster.
Kubernetes resources and "records of intent" are all stored as API objects, and modified via RESTful calls to the API. The API allows configuration to be managed in a declarative way. Users can interact with the Kubernetes API directly, or via tools like
kubectl
. The core Kubernetes API is flexible and can also be extended to support custom resources.
5.1 - Workload Resources
5.1.1 - Pod
Pod is a collection of containers that can run on a host.
Pod
Pod is a collection of containers that can run on a host. This resource is created by clients and scheduled onto hosts.
: v1
kind
: Pod
metadata
(
)
spec
(
PodSpec
)
Specification of the desired behavior of the pod. More info:
status
(
PodStatus
)
Most recently observed status of the pod. This data may not be up to date. Populated by the system. Read-only. More info:
PodSpec
PodSpec is a description of a pod.
([]
Container
name
List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated.
([]
Container
)
name
List of initialization containers belonging to the pod. Init containers are executed in order prior to containers being started. If any init container fails, the pod is considered to have failed and is handled according to its restartPolicy. The name for an init container or normal container mus

t be unique among all containers. Init containers may not have Lifecycle actions, Readiness probes, Liveness probes, or Startup probes. The resourceRequirements of an init container are taken into account during scheduling by finding the highest request/limit for each resource type, and then using the max of of that value or the sum of the normal containers. Limits are applied to init containers in a similar fashion. Init containers cannot currently be added or removed. Cannot be updated. More info:
https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
ephemeralContainers
([]
EphemeralContainer
)
name
List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated actions such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod's ephemeralcontainers subresource.
([]
)
name
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. More info:
https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod
(boolean)
EnableServiceLinks indicates whether information about services should be injected into pod's environment variables, matching the syntax of Docker links. Optional: Defaults to true.
os
(PodOS)
Specifies the OS of the containers in the pod. Some pod and container fields are restricted if this is set.
If the OS field is set to linux, the following fields must be unset: -securityContext.windowsOptions
If the OS field is set to windows, following fields must be unset: - spec.hostPID - spec.hostIPC - spec.hostUsers - spec.securityContext.seLinuxOptions - spec.securityContext.seccompProfile - spec.securityContext.fsGroup - spec.securityContext.fsGroupChangePolicy - spec.securityContext.sysctls - spec.shareProcessNamespace - spec.securityContext.runAsUser - spec.securityContext.runAsGroup - spec.securityContext.supplementalGroups - spec.containers[
].securityContext.seLinuxOptions - spec.containers[
].securityContext.seccompProfile - spec.containers[
].securityContext.capabilities - spec.containers[
].securityContext.readOnlyRootFilesystem - spec.containers[
].securityContext.privileged - spec.containers[
].securityContext.allowPrivilegeEscalation - spec.containers[
].securityContext.procMount - spec.containers[
].securityContext.runAsUser - spec.containers[*].securityContext.runAsGroup
PodOS defines the OS parameters of a pod.
os.name
Name is the name of the operating system. The currently supported values are linux and windows. Additional value may be defined in future and can be one of:
https://github.com/opencontainers/runtime-spec/blob/master/config.md#platform-specific-configuration
Clients should expect to handle additional values and treat unrecognized values in this field as os: null
Volumes
volumes
([]
Volume
)
Patch strategies: retainKeys, merge on key
name
List of volumes that can be mounted by containers belonging to the pod. More info:
https://kubernetes.io/docs/concepts/storage/volumes
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info:
https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
nodeName
(string)
NodeName is a request to schedule this pod onto a specific node. If it is non-empty, the scheduler simply schedules this pod onto that node, assuming that it fits resource requirements.
affinity
(Affinity)
If specified, the pod's scheduling constraints
Affinity is a group of affinity schedulin

g rules.
affinity.nodeAffinity
(
NodeAffinity
)
Describes node affinity scheduling rules for the pod.
affinity.podAffinity
(
PodAffinity
)
Describes pod affinity scheduling rules (e.g. co-locate this pod in the same node, zone, etc. as some other pod(s)).
affinity.podAntiAffinity
(
PodAntiAffinity
)
Describes pod anti-affinity scheduling rules (e.g. avoid putting this pod in the same node, zone, etc. as some other pod(s)).
([]Toleration)
If specified, the pod's tolerations.
The pod this Toleration is attached to tolerates any taint that matches the triple &lt;key,value,effect&gt; using the matching operator
.
tolerations.key
(string)
Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys.
tolerations.operator
(string)
Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category.
tolerations.value
(string)
Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string.
tolerations.effect
(string)
Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
tolerations.tolerationSeconds
(int64)
TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
(string)
If specified, the pod will be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.
runtimeClassName
(string)
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the "legacy" RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info:
https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class
(string)
If specified, indicates the pod's priority. "system-node-critical" and "system-cluster-critical" are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default.
priority
(int32)
The priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority.
(string)
PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset.
topologySpreadConstraints
([]TopologySpreadConstraint)
topologyKey
topologyKey, whenUnsatisfiable
TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed.
TopologySpreadConstraint specifies how to spread matching pods among the given topology.
topologySpreadConstraints.maxSkew
MaxSkew describes the degree to which pods may be unevenly distributed. When
whenUnsatisfiable=DoNotSchedule
, it is the maximum permitted difference between the number of matchi

ng pods in the target topology and the global minimum. The global minimum is the minimum number of matching pods in an eligible domain or zero if the number of eligible domains is less than MinDomains. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 2/2/1: In this case, the global minimum is 1. | zone1 | zone2 | zone3 | | P P | P P | P | - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming pod can be scheduled onto any zone. When
whenUnsatisfiable=ScheduleAnyway
, it is used to give higher precedence to topologies that satisfy it. It's a required field. Default value is 1 and 0 is not allowed.
topologySpreadConstraints.topologyKey
TopologyKey is the key of node labels. Nodes that have a label with this key and identical values are considered to be in the same topology. We consider each &lt;key, value&gt; as a "bucket", and try to put balanced number of pods into each bucket. We define a domain as a particular instance of a topology. Also, we define an eligible domain as a domain whose nodes meet the requirements of nodeAffinityPolicy and nodeTaintsPolicy. e.g. If TopologyKey is "kubernetes.io/hostname", each Node is a domain of that topology. And, if TopologyKey is "topology.kubernetes.io/zone", each zone is a domain of that topology. It's a required field.
topologySpreadConstraints.whenUnsatisfiable
WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy the spread constraint.
DoNotSchedule (default) tells the scheduler not to schedule it.
ScheduleAnyway tells the scheduler to schedule the pod in any location,
but giving higher precedence to topologies that would help reduce the
skew.
A constraint is considered "Unsatisfiable" for an incoming pod if and only if every possible node assignment for that pod would violate "MaxSkew" on some topology. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 3/1/1: | zone1 | zone2 | zone3 | | P P P | P | P | If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler won't make it
more
imbalanced. It's a required field.
topologySpreadConstraints.labelSelector
(
)
LabelSelector is used to find matching pods. Pods that match this label selector are counted to determine the number of pods in their corresponding topology domain.
topologySpreadConstraints.matchLabelKeys
MatchLabelKeys is a set of pod label keys to select the pods over which spreading will be calculated. The keys are used to lookup values from the incoming pod labels, those key-value labels are ANDed with labelSelector to select the group of existing pods over which spreading will be calculated for the incoming pod. Keys that don't exist in the incoming pod labels will be ignored. A null or empty list means only match against labelSelector.
topologySpreadConstraints.minDomains
(int32)
MinDomains indicates a minimum number of eligible domains. When the number of eligible domains with matching topology keys is less than minDomains, Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. And when the number of eligible domains with matching topology keys equals or greater than minDomains, this value has no effect on scheduling. As a result, when the number of eligible domains is less than minDomains, scheduler won't schedule more than maxSkew Pods to those domains. If value is nil, the constraint behaves as if MinDomains is equal to 1. Valid values are integers greater than 0. When value is not nil, WhenUnsatisfiable must be DoNotSchedule

.
For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same labelSelector spread as 2/2/2: | zone1 | zone2 | zone3 | | P P | P P | P P | The number of domains is less than 5(MinDomains), so "global minimum" is treated as 0. In this situation, new pod with the same labelSelector cannot be scheduled, because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, it will violate MaxSkew.
This is a beta field and requires the MinDomainsInPodTopologySpread feature gate to be enabled (enabled by default).
topologySpreadConstraints.nodeAffinityPolicy
(string)
NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector when calculating pod topology spread skew. Options are: - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations. - Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations.
If this value is nil, the behavior is equivalent to the Honor policy. This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag.
topologySpreadConstraints.nodeTaintsPolicy
(string)
NodeTaintsPolicy indicates how we will treat node taints when calculating pod topology spread skew. Options are: - Honor: nodes without taints, along with tainted nodes for which the incoming pod has a toleration, are included. - Ignore: node taints are ignored. All nodes are included.
If this value is nil, the behavior is equivalent to the Ignore policy. This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag.
overhead
Quantity
)
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info:
https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md
Lifecycle
(string)
Restart policy for all containers within the pod. One of Always, OnFailure, Never. Default to Always. More info:
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy
(int64)
Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds.
activeDeadlineSeconds
(int64)
Optional duration in seconds the pod may be active on the node relative to StartTime before the system will actively try to mark it failed and kill associated containers. Value must be a positive integer.
readinessGates
([]PodReadinessGate)
If specified, all readiness gates will be evaluated for pod readiness. A pod is ready when all its containers are ready AND all conditions specified in the readiness gates have status equal to "True" More info:
https://git.k8s.io/enhancements/keps/sig-network/580-pod-readiness-gates
PodReadinessGate contains the reference to a pod condition
readinessGates.conditionType
ConditionType refers to a condition in the pod's condition list with matching type.
Hostname and Name resolution
hostname
(string)
Specifies the hostn

ame of the Pod If not specified, the pod's hostname will be set to a system-defined value.
setHostnameAsFQDN
(boolean)
If true the pod's hostname will be configured as the pod's FQDN, rather than the leaf name (the default). In Linux containers, this means setting the FQDN in the hostname field of the kernel (the nodename field of struct utsname). In Windows containers, this means setting the registry value of hostname for the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters to FQDN. If a pod does not have FQDN, this has no effect. Default to false.
subdomain
(string)
If specified, the fully qualified Pod hostname will be "&lt;hostname&gt;.&lt;subdomain&gt;.&lt;pod namespace&gt;.svc.&lt;cluster domain&gt;". If not specified, the pod will not have a domainname at all.
hostAliases
([]HostAlias)
ip
HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts file if specified. This is only valid for non-hostNetwork pods.
HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's hosts file.
hostAliases.hostnames
Hostnames for the above IP address.
hostAliases.ip
(string)
IP address of the host file entry.
dnsConfig
(PodDNSConfig)
Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy.
PodDNSConfig defines the DNS parameters of a pod in addition to those generated from DNSPolicy.
dnsConfig.nameservers
A list of DNS name server IP addresses. This will be appended to the base nameservers generated from DNSPolicy. Duplicated nameservers will be removed.
dnsConfig.options
([]PodDNSConfigOption)
A list of DNS resolver options. This will be merged with the base options generated from DNSPolicy. Duplicated entries will be removed. Resolution options given in Options will override those that appear in the base DNSPolicy.
PodDNSConfigOption defines DNS resolver options of a pod.
dnsConfig.options.name
(string)
Required.
dnsConfig.options.value
(string)
dnsConfig.searches
A list of DNS search domains for host-name lookup. This will be appended to the base search paths generated from DNSPolicy. Duplicated search paths will be removed.
dnsPolicy
(string)
Set DNS policy for the pod. Defaults to "ClusterFirst". Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to 'ClusterFirstWithHostNet'.
Hosts namespaces
(boolean)
Host networking requested for this pod. Use the host's network namespace. If this option is set, the ports that will be used must be specified. Default to false.
hostPID
(boolean)
Use the host's pid namespace. Optional: Default to false.
hostIPC
(boolean)
Use the host's ipc namespace. Optional: Default to false.
(boolean)
Share a single process namespace between all of the containers in a pod. When this is set containers will be able to view and signal processes from other containers in the same pod, and the first process in each container will not be assigned PID 1. HostPID and ShareProcessNamespace cannot both be set. Optional: Default to false.
Service account
(string)
ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
(boolean)
AutomountServiceAccountToken indicates whether a service account token should be automatically mounted.
Security context
(PodSecurityContext)
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field.
PodSecurityContext holds pod-level

 security attributes and common container settings. Some fields are also present in container.securityContext. Field values of container.securityContext take precedence over field values of PodSecurityContext.
securityContext.runAsUser
(int64)
The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
securityContext.runAsNonRoot
(boolean)
Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
securityContext.runAsGroup
(int64)
The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
securityContext.supplementalGroups
([]int64)
A list of groups applied to the first process run in each container, in addition to the container's primary GID, the fsGroup (if specified), and group memberships defined in the container image for the uid of the container process. If unspecified, no additional groups are added to any container. Note that group memberships defined in the container image for the uid of the container process are still effective, even if they are not included in this list. Note that this field cannot be set when spec.os.name is windows.
securityContext.fsGroup
(int64)
A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:
The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw----
If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows.
securityContext.fsGroupChangePolicy
(string)
fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows.
(SeccompProfile)
The seccomp options to use by the containers in this pod. Note that this field cannot be set when spec.os.name is windows.
SeccompProfile defines a pod/container's seccomp profile settings. Only one profile source may be set.
securityContext.seccompProfile.type
type indicates which kind of seccomp profile will be applied. Valid options are:
Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied.
securityContext.seccompProfile.localhostProfile
(string)
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must only be set if type is "Localhost".
securityContext.seLinuxOptions
(SELinuxOptions)
The SELinux context to be applied to all container

s. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
SELinuxOptions are the labels to be applied to the container
securityContext.seLinuxOptions.level
(string)
Level is SELinux level label that applies to the container.
securityContext.seLinuxOptions.role
(string)
Role is a SELinux role label that applies to the container.
securityContext.seLinuxOptions.type
(string)
Type is a SELinux type label that applies to the container.
securityContext.seLinuxOptions.user
(string)
User is a SELinux user label that applies to the container.
securityContext.sysctls
([]Sysctl)
Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch. Note that this field cannot be set when spec.os.name is windows.
Sysctl defines a kernel parameter to be set
securityContext.sysctls.name
Name of a property to set
securityContext.sysctls.value
Value of a property to set
securityContext.windowsOptions
(WindowsSecurityContextOptions)
The Windows specific settings applied to all containers. If unspecified, the options within a container's SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux.
WindowsSecurityContextOptions contain Windows-specific options and credentials.
securityContext.windowsOptions.gmsaCredentialSpec
(string)
GMSACredentialSpec is where the GMSA admission webhook (
https://github.com/kubernetes-sigs/windows-gmsa
) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field.
(string)
GMSACredentialSpecName is the name of the GMSA credential spec to use.
(boolean)
HostProcess determines if a container should be run as a 'Host Process' container. This field is alpha-level and will only be honored by components that enable the WindowsHostProcessContainers feature flag. Setting this field without the feature flag will result in errors when validating the Pod. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
(string)
The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
Alpha level
hostUsers
(boolean)
Use the host's user namespace. Optional: Default to true. If set to true or not present, the pod will be run in the host user namespace, useful for when the pod needs a feature only available to the host user namespace, such as loading a kernel module with CAP_SYS_MODULE. When set to false, a new userns is created for the pod. Setting false is useful for mitigating container breakout vulnerabilities even allowing users to run their containers as root without actually having root privileges on the host. This field is alpha-level and is only honored by servers that enable the UserNamespacesSupport feature.
resourceClaims
([]PodResourceClaim)
name
ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume t

hem by name.
This is an alpha field and requires enabling the DynamicResourceAllocation feature gate.
This field is immutable.
PodResourceClaim references exactly one ResourceClaim through a ClaimSource. It adds a name to it that uniquely identifies the ResourceClaim inside the Pod. Containers that need access to the ResourceClaim reference it with this name.
resourceClaims.name
Name uniquely identifies this resource claim inside the pod. This must be a DNS_LABEL.
resourceClaims.source
(ClaimSource)
Source describes where to find the ResourceClaim.
*ClaimSource describes a reference to a ResourceClaim.
Exactly one of these fields should be set. Consumers of this type must treat an empty object as if it has an unknown value.*
resourceClaims.source.resourceClaimName
(string)
ResourceClaimName is the name of a ResourceClaim object in the same namespace as this pod.
resourceClaims.source.resourceClaimTemplateName
(string)
ResourceClaimTemplateName is the name of a ResourceClaimTemplate object in the same namespace as this pod.
The template will be used to create a new ResourceClaim, which will be bound to this pod. When this pod is deleted, the ResourceClaim will also be deleted. The name of the ResourceClaim will be &lt;pod name&gt;-&lt;resource name&gt;, where &lt;resource name&gt; is the PodResourceClaim.Name. Pod validation will reject the pod if the concatenated name is not valid for a ResourceClaim (e.g. too long).
An existing ResourceClaim with that name that is not owned by the pod will not be used for the pod to avoid using an unrelated resource by mistake. Scheduling and pod startup are then blocked until the unrelated ResourceClaim is removed.
This field is immutable and no changes will be made to the corresponding ResourceClaim by the control plane after creating the ResourceClaim.
schedulingGates
([]PodSchedulingGate)
name
SchedulingGates is an opaque list of values that if specified will block scheduling the pod. More info:
https://git.k8s.io/enhancements/keps/sig-scheduling/3521-pod-scheduling-readiness
.
This is an alpha-level feature enabled by PodSchedulingReadiness feature gate.
PodSchedulingGate is associated to a Pod to guard its scheduling.
schedulingGates.name
Name of the scheduling gate. Each scheduling gate must have a unique name field.
(string)
DeprecatedServiceAccount is a depreciated alias for ServiceAccountName. Deprecated: Use serviceAccountName instead.
Container
A single application container that you want to run within a pod.
name
Name of the container specified as a DNS_LABEL. Each container in a pod must have a unique name (DNS_LABEL). Cannot be updated.
Image
image
(string)
Container image name. More info:
https://kubernetes.io/docs/concepts/containers/images
This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets.
(string)
Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info:
https://kubernetes.io/docs/concepts/containers/images#updating-images
Entrypoint
command
Entrypoint array. Not executed within a shell. The container image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info:
https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-con

tainer/#running-a-command-in-a-shell
args
Arguments to the entrypoint. The container image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info:
https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell
workingDir
(string)
Container's working directory. If not specified, the container runtime's default will be used, which might be configured in the container image. Cannot be updated.
Ports
ports
([]ContainerPort)
containerPort, protocol
List of ports to expose from the container. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Modifying this array with strategic merge patch may corrupt the data. For more information See
https://github.com/kubernetes/kubernetes/issues/108255
. Cannot be updated.
ContainerPort represents a network port in a single container.
ports.containerPort
Number of port to expose on the pod's IP address. This must be a valid port number, 0 &lt; x &lt; 65536.
ports.hostIP
(string)
What host IP to bind the external port to.
ports.hostPort
(int32)
Number of port to expose on the host. If specified, this must be a valid port number, 0 &lt; x &lt; 65536. If HostNetwork is specified, this must match ContainerPort. Most containers do not need this.
(string)
If specified, this must be an IANA_SVC_NAME and unique within the pod. Each named port in a pod must have a unique name. Name for the port that can be referred to by services.
(string)
Protocol for port. Must be UDP, TCP, or SCTP. Defaults to "TCP".
Environment variables
env
([]EnvVar)
name
List of environment variables to set in the container. Cannot be updated.
EnvVar represents an environment variable present in a Container.
env.name
Name of the environment variable. Must be a C_IDENTIFIER.
env.value
(string)
Variable references $(VAR_NAME) are expanded using the previously defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "".
env.valueFrom
(EnvVarSource)
Source for the environment variable's value. Cannot be used if value is not empty.
EnvVarSource represents a source for the value of an EnvVar.
env.valueFrom.configMapKeyRef
(ConfigMapKeySelector)
Selects a key of a ConfigMap.
Selects a key from a ConfigMap.
env.valueFrom.configMapKeyRef.key
The key to select.
env.valueFrom.configMapKeyRef.name
(string)
env.valueFrom.configMapKeyRef.optional
(boolean)
Specify whether the ConfigMap or its key must be defined
env.valueFrom.fieldRef
(
)
Selects a field of the pod: supports metadata.name, metadata.namespace,
metadata.labels['\&lt;KEY&gt;']
,
metadata.annotations['\&lt;KEY&gt;']
, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs.
env.valueFrom.resourceFieldRef
(
)

Selects a resource of the container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.
env.valueFrom.secretKeyRef
(SecretKeySelector)
Selects a key of a secret in the pod's namespace
SecretKeySelector selects a key of a Secret.
env.valueFrom.secretKeyRef.key
The key of the secret to select from. Must be a valid secret key.
env.valueFrom.secretKeyRef.name
(string)
env.valueFrom.secretKeyRef.optional
(boolean)
Specify whether the Secret or its key must be defined
envFrom
([]EnvFromSource)
List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be updated.
EnvFromSource represents the source of a set of ConfigMaps
envFrom.configMapRef
(ConfigMapEnvSource)
The ConfigMap to select from
*ConfigMapEnvSource selects a ConfigMap to populate the environment variables with.
The contents of the target ConfigMap's Data field will represent the key-value pairs as environment variables.*
envFrom.configMapRef.name
(string)
envFrom.configMapRef.optional
(boolean)
Specify whether the ConfigMap must be defined
envFrom.prefix
(string)
An optional identifier to prepend to each key in the ConfigMap. Must be a C_IDENTIFIER.
envFrom.secretRef
(SecretEnvSource)
The Secret to select from
*SecretEnvSource selects a Secret to populate the environment variables with.
The contents of the target Secret's Data field will represent the key-value pairs as environment variables.*
envFrom.secretRef.name
(string)
envFrom.secretRef.optional
(boolean)
Specify whether the Secret must be defined
Volumes
([]VolumeMount)
mountPath
Pod volumes to mount into the container's filesystem. Cannot be updated.
VolumeMount describes a mounting of a Volume within a container.
Path within the container at which the volume should be mounted. Must not contain ':'.
volumeMounts.name
This must match the Name of a Volume.
volumeMounts.mountPropagation
(string)
mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationNone is used. This field is beta in 1.10.
volumeMounts.readOnly
(boolean)
Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.
volumeMounts.subPath
(string)
Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).
volumeMounts.subPathExpr
(string)
Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPath are mutually exclusive.
volumeDevices
([]VolumeDevice)
devicePath
volumeDevices is the list of block devices to be used by the container.
volumeDevice describes a mapping of a raw block device within a container.
volumeDevices.devicePath
devicePath is the path inside of the container that the device will be mapped to.
volumeDevices.name
name must match the name of a persistentVolumeClaim in the pod
Resources
resources
(ResourceRequirements)
Comp

ute Resources required by this container. Cannot be updated. More info:
https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
ResourceRequirements describes the compute resource requirements.
resources.claims
([]ResourceClaim)
Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.
ResourceClaim references one entry in PodSpec.ResourceClaims.
resources.claims.name
Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container.
resources.limits
Quantity
)
Limits describes the maximum amount of compute resources allowed. More info:
resources.requests
Quantity
)
Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info:
Lifecycle
lifecycle
(Lifecycle)
Actions that the management system should take in response to container lifecycle events. Cannot be updated.
Lifecycle describes actions that the management system should take in response to container lifecycle events. For the PostStart and PreStop lifecycle handlers, management of the container blocks until the action is complete, unless the container process fails, in which case the handler is aborted.
lifecycle.postStart
(
LifecycleHandler
)
PostStart is called immediately after a container is created. If the handler fails, the container is terminated and restarted according to its restart policy. Other management of the container blocks until the hook completes. More info:
https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
lifecycle.preStop
(
)
PreStop is called immediately before a container is terminated due to an API request or management event such as liveness/startup probe failure, preemption, resource contention, etc. The handler is not called if the container crashes or exits. The Pod's termination grace period countdown begins before the PreStop hook is executed. Regardless of the outcome of the handler, the container will eventually terminate within the Pod's termination grace period (unless delayed by finalizers). Other management of the container blocks until the hook completes or until the termination grace period is reached. More info:
(string)
Optional: Path at which the file to which the container's termination message will be written is mounted into the container's filesystem. Message written is intended to be brief final status, such as an assertion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb. Defaults to /dev/termination-log. Cannot be updated.
(string)
Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is limited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.
(
Probe
)
Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated. More info:
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#contai

ner-probes
(
Probe
)
Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. More info:
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
(
Probe
)
StartupProbe indicates that the Pod has successfully initialized. If specified, no other probes are executed until this completes successfully. If this probe fails, the Pod will be restarted, just as if the livenessProbe failed. This can be used to provide different probe parameters at the beginning of a Pod's lifecycle, when it might take a long time to load data or warm a cache, than during steady-state operation. This cannot be updated. More info:
Security Context
(SecurityContext)
SecurityContext defines the security options the container should be run with. If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext. More info:
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
SecurityContext holds security configuration that will be applied to a container. Some fields are present in both SecurityContext and PodSecurityContext. When both are set, the values in SecurityContext take precedence.
(int64)
The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
(boolean)
Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
(int64)
The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
securityContext.readOnlyRootFilesystem
(boolean)
Whether this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows.
securityContext.procMount
(string)
procMount denotes the type of proc mount to use for the containers. The default is DefaultProcMount which uses the container runtime defaults for readonly paths and masked paths. This requires the ProcMountType feature flag to be enabled. Note that this field cannot be set when spec.os.name is windows.
securityContext.privileged
(boolean)
Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false. Note that this field cannot be set when spec.os.name is windows.
securityContext.allowPrivilegeEscalation
(boolean)
AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN Note that this field cannot be set when spec.os.name is windows.
securityContext.capabilities
(Capabilities)
The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime. Note that this field cannot be set when spec.os.name is windows.
Adds and removes POSIX capabilities from running containers.
securityCo

ntext.capabilities.add
Added capabilities
securityContext.capabilities.drop
Removed capabilities
The seccomp options to use by this container. If seccomp options are provided at both the pod &amp; container level, the container options override the pod options. Note that this field cannot be set when spec.os.name is windows.
(string)
The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
(string)
(string)
(string)
(string)
The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is linux.
(string)
(string)
(boolean)
(string)
Debugging
stdin
(boolean)
Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in E

OF. Default is false.
stdinOnce
(boolean)
Whether the container runtime should close the stdin channel after it has been opened by a single attach. When stdin is true the stdin stream will remain open across multiple attach sessions. If stdinOnce is set to true, stdin is opened on container start, is empty until the first client attaches to stdin, and then remains open and accepts data until the client disconnects, at which time stdin is closed and remains closed until the container is restarted. If this flag is false, a container processes that reads from stdin will never receive an EOF. Default is false
tty
(boolean)
Whether this container should allocate a TTY for itself, also requires 'stdin' to be true. Default is false.
An EphemeralContainer is a temporary container that you may add to an existing Pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a Pod is removed or restarted. The kubelet may evict a Pod if an ephemeral container causes the Pod to exceed its resource allocation.
To add an ephemeral container, use the ephemeralcontainers subresource of an existing Pod. Ephemeral containers may not be removed or restarted.
name
Name of the ephemeral container specified as a DNS_LABEL. This name must be unique among all containers, init containers and ephemeral containers.
targetContainerName
(string)
If set, the name of the container from PodSpec that this ephemeral container targets. The ephemeral container will be run in the namespaces (IPC, PID, etc) of this container. If not set then the ephemeral container uses the namespaces configured in the Pod spec.
The container runtime must implement support for this feature. If the runtime does not support namespace targeting then the result of setting this field is undefined.
Image
image
(string)
(string)
command
Entrypoint array. Not executed within a shell. The image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info:
args
Arguments to the entrypoint. The image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info:
(string)
env
name
EnvVar represents an environment var

iable present in a Container.
env.name
env.value
(string)
(string)
(boolean)
(
)
,
(
)
(string)
(boolean)
envFrom
(string)
(boolean)
(string)
(string)
(boolean)
Volumes
mountPath
Pod volumes to mount into the container's file

system. Subpath mounts are not allowed for ephemeral containers. Cannot be updated.
(string)
(boolean)
(string)
(string)
Lifecycle
(string)
(string)
Debugging
stdin
(boolean)
Whether this container should allocate a buffer for stdin in the container runtime. If this is not set, reads from stdin in the container will always result in EOF. Default is false.
stdinOnce
(boolean)
tty
(boolean)
Optional: SecurityContext defines the security options the ephemeral container should be run with. If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext.
(int64)
The UID to run the entrypoint of the container 

process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
(boolean)
(int64)
(boolean)
(string)
(boolean)
(boolean)
securityContext.capabilities.add
(string)
The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext

, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
(string)
(string)
(string)
(string)
(string)
(string)
(boolean)
(string)
Not allowed
ports
Ports are not allowed for ephemeral containers.
(string)
(int32)
(string)
(string)
resources
Resources are not allowed for ephemeral containers. Ephemeral containers use spare resources already allocated to the pod.
Name must m

atch the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container.
Quantity
)
Quantity
)
lifecycle
Lifecycle is not allowed for ephemeral containers.
(
)
(
)
(
Probe
)
Probes are not allowed for ephemeral containers.
(
Probe
)
(
Probe
)
LifecycleHandler defines a specific action that should be taken in a lifecycle hook. One and only one of the fields, except TCPSocket must be specified.
exec
(ExecAction)
Exec specifies the action to take.
ExecAction describes a "run in container" action.
exec.command
Command is the command line to execute inside the container, the working directory for the command is root ('/') in the container's filesystem. The command is simply exec'd, it is not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use a shell, you need to explicitly call out to that shell. Exit status of 0 is treated as live/healthy and non-zero is unhealthy.
httpGet
(HTTPGetAction)
HTTPGet specifies the http request to perform.
HTTPGetAction describes an action based on HTTP Get requests.
httpGet.port
(IntOrString), required
Name or number of the port to access on the container. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME.
httpGet.host
(string)
Host name to connect to, defaults to the pod IP. You probably want to set "Host" in httpHeaders instead.
httpGet.httpHeaders
([]HTTPHeader)
Custom headers to set in the request. HTTP allows repeated headers.
HTTPHeader describes a custom header to be used in HTTP probes
httpGet.httpHeaders.name
The header field name
httpGet.httpHeaders.value

The header field value
httpGet.path
(string)
Path to access on the HTTP server.
httpGet.scheme
(string)
Scheme to use for connecting to the host. Defaults to HTTP.
tcpSocket
(TCPSocketAction)
Deprecated. TCPSocket is NOT supported as a LifecycleHandler and kept for the backward compatibility. There are no validation of this field and lifecycle hooks will fail in runtime when tcp handler is specified.
TCPSocketAction describes an action based on opening a socket
tcpSocket.port
Number or name of the port to access on the container. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME.
tcpSocket.host
(string)
Optional: Host name to connect to, defaults to the pod IP.
Node affinity is a group of node affinity scheduling rules.
([]PreferredSchedulingTerm)
The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred.
An empty preferred scheduling term matches all objects with implicit weight 0 (i.e. it's a no-op). A null preferred scheduling term matches no objects (i.e. is also a no-op).
preferredDuringSchedulingIgnoredDuringExecution.preference
(NodeSelectorTerm), required
A node selector term, associated with the corresponding weight.
A null or empty node selector term matches no objects. The requirements of them are ANDed. The TopologySelectorTerm type implements a subset of the NodeSelectorTerm.
preferredDuringSchedulingIgnoredDuringExecution.preference.matchExpressions
([]
)
A list of node selector requirements by node's labels.
preferredDuringSchedulingIgnoredDuringExecution.preference.matchFields
([]
)
A list of node selector requirements by node's fields.
preferredDuringSchedulingIgnoredDuringExecution.weight
Weight associated with matching the corresponding nodeSelectorTerm, in the range 1-100.
(NodeSelector)
If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to an update), the system may or may not try to eventually evict the pod from its node.
A node selector represents the union of the results of one or more label queries over a set of nodes; that is, it represents the OR of the selectors represented by the node selector terms.
requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms
([]NodeSelectorTerm), required
Required. A list of node selector terms. The terms are ORed.
requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchExpressions
([]
)
requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms.matchFields
([]
)
Pod affinity is a group of inter pod affinity schedu

ling rules.
([]WeightedPodAffinityTerm)
The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred.
The weights of all of the matched WeightedPodAffinityTerm fields are added per-node to find the most preferred node(s)
preferredDuringSchedulingIgnoredDuringExecution.podAffinityTerm
(PodAffinityTerm), required
Required. A pod affinity term, associated with the corresponding weight.
Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key
matches that of any node on which a pod of the set of pods is running
preferredDuringSchedulingIgnoredDuringExecution.podAffinityTerm.topologyKey
This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.
preferredDuringSchedulingIgnoredDuringExecution.podAffinityTerm.labelSelector
(
)
A label query over a set of resources, in this case pods.
preferredDuringSchedulingIgnoredDuringExecution.podAffinityTerm.namespaceSelector
(
)
A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces.
preferredDuringSchedulingIgnoredDuringExecution.podAffinityTerm.namespaces
namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace".
weight associated with matching the corresponding podAffinityTerm, in the range 1-100.
([]PodAffinityTerm)
If the affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied.
requiredDuringSchedulingIgnoredDuringExecution.topologyKey
This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespaces, where co-located is defi

ned as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.
requiredDuringSchedulingIgnoredDuringExecution.labelSelector
(
)
requiredDuringSchedulingIgnoredDuringExecution.namespaceSelector
(
)
requiredDuringSchedulingIgnoredDuringExecution.namespaces
Pod anti affinity is a group of inter pod anti affinity scheduling rules.
The scheduler will prefer to schedule pods to nodes that satisfy the anti-affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling anti-affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most preferred.
(
)
(
)
weight associated with matching the corresponding podAffinityTerm, in the r

ange 1-100.
If the anti-affinity requirements specified by this field are not met at scheduling time, the pod will not be scheduled onto the node. If the anti-affinity requirements specified by this field cease to be met at some point during pod execution (e.g. due to a pod label update), the system may or may not try to eventually evict the pod from its node. When there are multiple elements, the lists of nodes corresponding to each podAffinityTerm are intersected, i.e. all terms must be satisfied.
(
)
(
)
Probe
Probe describes a health check to be performed against a container to determine whether it is alive or ready to receive traffic.
exec
httpGet
(string)
(string)
(string)
tcpSocket
(TCPSocketAct

ion)
TCPSocket specifies an action involving a TCP port.
(string)
(int32)
Number of seconds after the container has started before liveness probes are initiated. More info:
(int64)
Optional duration in seconds the pod needs to terminate gracefully upon probe failure. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this value overrides the value provided by the pod spec. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate. Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.
(int32)
How often (in seconds) to perform the probe. Default to 10 seconds. Minimum value is 1.
(int32)
Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1. More info:
(int32)
Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1.
(int32)
Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1.
grpc
(GRPCAction)
GRPC specifies an action involving a GRPC port. This is a beta field and requires enabling GRPCContainerProbe feature gate.
**
grpc.port
Port number of the gRPC service. Number must be in the range 1 to 65535.
grpc.service
(string)
Service is the name of the service to place in the gRPC HealthCheckRequest (see
https://github.com/grpc/grpc/blob/master/doc/health-checking.md)
.
If this is not specified, the default behavior is defined by gRPC.
PodStatus
PodStatus represents information about the status of a pod. Status may trail the actual state of a system, especially if the node that hosts the pod cannot contact the control plane.
nominatedNodeName
(string)
nominatedNodeName is set only when this pod preempts other pods on the node, but it cannot be scheduled right away as preemption victims receive their graceful termination periods. This field does not guarantee that the pod will be scheduled on this node. Scheduler may decide to place the pod elsewhere if other nodes become available sooner. Scheduler may also decide to give the resources on this node to a higher priority pod that is created after preemption. As a result, this field may be different than PodSpec.nodeName when the pod is scheduled.
hostIP
(string)
IP address of the host to which the pod is assigned. Empty if not yet scheduled.
startTime
(Time)
RFC 3339 date and time at which the object was acknowledged by the Kubelet. This is before the Kubelet pulled the container image(s) for the pod.
phase
(string)
The phase o

f a Pod is a simple, high-level summary of where the Pod is in its lifecycle. The conditions array, the reason and message fields, and the individual container status arrays contain more detail about the pod's status. There are five possible phase values:
Pending: The pod has been accepted by the Kubernetes system, but one or more of the container images has not been created. This includes time before being scheduled as well as time spent downloading images over the network, which could take a while. Running: The pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. Succeeded: All containers in the pod have terminated in success, and will not be restarted. Failed: All containers in the pod have terminated, and at least one container has terminated in failure. The container either exited with non-zero status or was terminated by the system. Unknown: For some reason the state of the pod could not be obtained, typically due to an error in communicating with the host of the pod.
More info:
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase
message
(string)
A human readable message indicating details about why the pod is in this condition.
reason
(string)
A brief CamelCase message indicating details about why the pod is in this state. e.g. 'Evicted'
podIP
(string)
IP address allocated to the pod. Routable at least within the cluster. Empty if not yet allocated.
podIPs
([]PodIP)
ip
podIPs holds the IP addresses allocated to the pod. If this field is specified, the 0th entry must match the podIP field. Pods may be allocated at most 1 value for each of IPv4 and IPv6. This list is empty if no IPs have been allocated yet.
*IP address information for entries in the (plural) PodIPs field. Each entry includes:
IP: An IP address allocated to the pod. Routable at least within the cluster.*
podIPs.ip
(string)
ip is an IP address (IPv4 or IPv6) assigned to the pod
([]PodCondition)
type
Current service state of pod. More info:
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-conditions
PodCondition contains details for the current condition of this pod.
Status is the status of the condition. Can be True, False, Unknown. More info:
Type is the type of the condition. More info:
conditions.lastProbeTime
(Time)
Last time we probed the condition.
(Time)
(string)
(string)
qosClass
(string)
The Quality of Service (QOS) classification assigned to the pod based on resource requirements See PodQOSClass type for available QOS classes More info:
https://git.k8s.io/community/contributors/design-proposals/node/resource-qos.md
initContainerStatuses
([]ContainerStatus)
The list has one entry per init container in the manifest. The most recent successful init container will have ready = true, the most recently started container will have startTime set. More info:
https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-and-container-status
ContainerStatus contains details for the current status of this c

ontainer.
initContainerStatuses.name
This must be a DNS_LABEL. Each container in a pod must have a unique name. Cannot be updated.
initContainerStatuses.image
The image the container is running. More info:
.
initContainerStatuses.imageID
ImageID of the container's image.
initContainerStatuses.containerID
(string)
Container's ID in the format '&lt;type&gt;://&lt;container_id&gt;'.
initContainerStatuses.state
(ContainerState)
Details about the container's current condition.
ContainerState holds a possible state of container. Only one of its members may be specified. If none of them is specified, the default one is ContainerStateWaiting.
initContainerStatuses.state.running
(ContainerStateRunning)
Details about a running container
ContainerStateRunning is a running state of a container.
initContainerStatuses.state.running.startedAt
(Time)
Time at which the container was last (re-)started
initContainerStatuses.state.terminated
(ContainerStateTerminated)
Details about a terminated container
ContainerStateTerminated is a terminated state of a container.
initContainerStatuses.state.terminated.containerID
(string)
Container's ID in the format '&lt;type&gt;://&lt;container_id&gt;'
initContainerStatuses.state.terminated.exitCode
Exit status from the last termination of the container
initContainerStatuses.state.terminated.startedAt
(Time)
Time at which previous execution of the container started
initContainerStatuses.state.terminated.finishedAt
(Time)
Time at which the container last terminated
initContainerStatuses.state.terminated.message
(string)
Message regarding the last termination of the container
initContainerStatuses.state.terminated.reason
(string)
(brief) reason from the last termination of the container
initContainerStatuses.state.terminated.signal
(int32)
Signal from the last termination of the container
initContainerStatuses.state.waiting
(ContainerStateWaiting)
Details about a waiting container
ContainerStateWaiting is a waiting state of a container.
initContainerStatuses.state.waiting.message
(string)
Message regarding why the container is not yet running.
initContainerStatuses.state.waiting.reason
(string)
(brief) reason the container is not yet running.
initContainerStatuses.lastState
Details about the container's last termination condition.
initContainerStatuses.lastState.running
initContainerStatuses.lastState.running.startedAt
(Time)
initContainerStatuses.lastState.terminated
initContainerStatuses.lastState.terminated.containerID
(string)
initContainerStatuses.lastState.terminated.exitCode
initContainerStatuses.lastSt

ate.terminated.startedAt
(Time)
initContainerStatuses.lastState.terminated.finishedAt
(Time)
initContainerStatuses.lastState.terminated.message
(string)
initContainerStatuses.lastState.terminated.reason
(string)
initContainerStatuses.lastState.terminated.signal
(int32)
initContainerStatuses.lastState.waiting
initContainerStatuses.lastState.waiting.message
(string)
initContainerStatuses.lastState.waiting.reason
(string)
initContainerStatuses.ready
Specifies whether the container has passed its readiness probe.
initContainerStatuses.restartCount
The number of times the container has been restarted.
initContainerStatuses.started
(boolean)
Specifies whether the container has passed its startup probe. Initialized as false, becomes true after startupProbe is considered successful. Resets to false when the container is restarted, or if kubelet loses state temporarily. Is always true when no startupProbe is defined.
The list has one entry per container in the manifest. More info:
ContainerStatus contains details for the current status of this container.
containerStatuses.name
containerStatuses.image
.
containerStatuses.imageID
containerStatuses.containerID
(string)
containerStatuses.state
containerStatuses.state.running
containerStatuses.state.running.startedAt
(Time)
containerStatuses.state.terminated
containerStatuses.state.terminated.containerID
(string)
containerStatuses.state.terminated.exitCode
containerStatuses.state.terminated.startedAt
(Time)
containerStatuses.state.terminated.finishedAt
(Time)
Time is a wrapper aroun

d time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.
containerStatuses.state.terminated.message
(string)
containerStatuses.state.terminated.reason
(string)
containerStatuses.state.terminated.signal
(int32)
containerStatuses.state.waiting
containerStatuses.state.waiting.message
(string)
containerStatuses.state.waiting.reason
(string)
containerStatuses.lastState
containerStatuses.lastState.running
containerStatuses.lastState.running.startedAt
(Time)
containerStatuses.lastState.terminated
containerStatuses.lastState.terminated.containerID
(string)
containerStatuses.lastState.terminated.exitCode
containerStatuses.lastState.terminated.startedAt
(Time)
containerStatuses.lastState.terminated.finishedAt
(Time)
containerStatuses.lastState.terminated.message
(string)
containerStatuses.lastState.terminated.reason
(string)
containerStatuses.lastState.terminated.signal
(int32)
containerStatuses.lastState.waiting
containerStatuses.lastState.waiting.message
(string)
containerStatuses.lastState.waiting.reason
(string)
containerStatuses.ready
containerStatuses.restartCount
containerStatuses.started
(boolean)
ephemeralContainerStatuses
Status for any ephemeral containers that have run in this pod.
ephemeralContainerStatuses.name
This must be a DNS_LABEL. Each container in a pod must have a unique n

ame. Cannot be updated.
ephemeralContainerStatuses.image
.
ephemeralContainerStatuses.imageID
ephemeralContainerStatuses.containerID
(string)
ephemeralContainerStatuses.state
ephemeralContainerStatuses.state.running
ephemeralContainerStatuses.state.running.startedAt
(Time)
ephemeralContainerStatuses.state.terminated
ephemeralContainerStatuses.state.terminated.containerID
(string)
ephemeralContainerStatuses.state.terminated.exitCode
ephemeralContainerStatuses.state.terminated.startedAt
(Time)
ephemeralContainerStatuses.state.terminated.finishedAt
(Time)
ephemeralContainerStatuses.state.terminated.message
(string)
ephemeralContainerStatuses.state.terminated.reason
(string)
ephemeralContainerStatuses.state.terminated.signal
(int32)
ephemeralContainerStatuses.state.waiting
ephemeralContainerStatuses.state.waiting.message
(string)
ephemeralContainerStatuses.state.waiting.reason
(string)
ephemeralContainerStatuses.lastState
ephemeralContainerStatuses.lastState.running
ephemeralContainerStatuses.lastState.running.startedAt
(Time)
ephemeralContainerStatuses.lastState.terminated
ephemeralContainerStatuses.lastState.terminated.containerID
(string)
ephemeralContainerStatuses.lastState.terminated.exitCode
ephemeralContainerStatuses.lastState.te

rminated.startedAt
(Time)
ephemeralContainerStatuses.lastState.terminated.finishedAt
(Time)
ephemeralContainerStatuses.lastState.terminated.message
(string)
ephemeralContainerStatuses.lastState.terminated.reason
(string)
ephemeralContainerStatuses.lastState.terminated.signal
(int32)
ephemeralContainerStatuses.lastState.waiting
ephemeralContainerStatuses.lastState.waiting.message
(string)
ephemeralContainerStatuses.lastState.waiting.reason
(string)
ephemeralContainerStatuses.ready
ephemeralContainerStatuses.restartCount
ephemeralContainerStatuses.started
(boolean)
PodList
PodList is a list of Pods.
items
([]
Pod
List of pods. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md
(string)
kind
(string)
metadata
(
ListMeta
)
get
read the specified Pod
GET /api/v1/namespaces/{namespace}/pods/{name}
name
(
in path
name of the Pod
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
read ephemeralcontainers of the specified Pod
GET /api/v1/namespaces/{namespace}/pods/{name}/ephemeralcontainers
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
read log of the specified Pod
name
(
in path
namespace
(
in path
namespace
container
(
in query
): string
The container for which to stream logs. Defaults to only container if there is one container in the pod.
follow
(
in query
): boolean
Follow the log stream of the pod. Defaults to false.
insecureSkipTLSVerifyBackend
(
in query
): boolean
insecureSkipTLSVerifyBackend indicates that the apiserver should not confirm the validity of the serving certificate of the backend it is conne

cting to. This will make the HTTPS connection between the apiserver and the backend insecure. This means the apiserver cannot verify the log data it is receiving came from the real kubelet. If the kubelet is configured to verify the apiserver's TLS credentials, it does not mean the connection to the real kubelet is vulnerable to a man in the middle attack (e.g. an attacker could not intercept the actual log data coming from the real kubelet).
limitBytes
(
in query
): integer
If set, the number of bytes to read from the server before terminating the log output. This may not display a complete final line of logging, and may return slightly more or slightly less than the specified limit.
pretty
(
in query
): string
pretty
previous
(
in query
): boolean
Return previous terminated container logs. Defaults to false.
sinceSeconds
(
in query
): integer
A relative time in seconds before the current time from which to show logs. If this value precedes the time a pod was started, only logs since the pod start will be returned. If this value is in the future, no logs will be returned. Only one of sinceSeconds or sinceTime may be specified.
tailLines
(
in query
): integer
If set, the number of lines from the end of the logs to show. If not specified, logs are shown from the creation of the container or sinceSeconds or sinceTime
timestamps
(
in query
): boolean
If true, add an RFC3339 or RFC3339Nano timestamp at the beginning of every line of log output. Defaults to false.
Response
200 (string): OK
get
read status of the specified Pod
GET /api/v1/namespaces/{namespace}/pods/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
list
list or watch objects of kind Pod
GET /api/v1/namespaces/{namespace}/pods
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
create
create a Pod
POST /api/v1/namespaces/{namespace}/pods
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
202 (
Pod
update
replace the specified Pod
PUT /api/v1/namespaces/{namespace}/pods/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
replace ephemeralcontainers of

 the specified Pod
PUT /api/v1/namespaces/{namespace}/pods/{name}/ephemeralcontainers
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
replace status of the specified Pod
PUT /api/v1/namespaces/{namespace}/pods/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
partially update the specified Pod
PATCH /api/v1/namespaces/{namespace}/pods/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
partially update ephemeralcontainers of the specified Pod
PATCH /api/v1/namespaces/{namespace}/pods/{name}/ephemeralcontainers
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
partially update status of the specified Pod
PATCH /api/v1/namespaces/{namespace}/pods/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
delete
delete a Pod
DELETE /api/v1/namespaces/{namespace}/pods/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Pod
): OK
202 (
Pod
delete collection of Pod
DELETE /api/v1/namespaces/{namespace}/pods
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.2 - PodTemplate
PodTemplate describes a template for creating copies of a predefined pod.
: v1
kind
: PodTempl

ate
metadata
(
)
template
(
)
Template defines the pods that will be created from this pod template.
PodTemplateSpec describes the data a pod should have when created from a template
metadata
(
)
spec
(
PodSpec
)
PodTemplateList
PodTemplateList is a list of PodTemplates.
: v1
kind
: PodTemplateList
metadata
(
ListMeta
)
items
([]
List of pod templates
get
read the specified PodTemplate
GET /api/v1/namespaces/{namespace}/podtemplates/{name}
name
(
in path
name of the PodTemplate
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind PodTemplate
GET /api/v1/namespaces/{namespace}/podtemplates
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/podtemplates
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a PodTemplate
POST /api/v1/namespaces/{namespace}/podtemplates
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified PodTemplate
PUT /api/v1/namespaces/{namespace}/podtemplates/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified PodTemplate
PATCH /api/v1/namespaces/{namespace}/podtemplates/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRu

n
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a PodTemplate
DELETE /api/v1/namespaces/{namespace}/podtemplates/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of PodTemplate
DELETE /api/v1/namespaces/{namespace}/podtemplates
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.3 - ReplicationController
ReplicationController represents the configuration of a replication controller.
: v1
kind
: ReplicationController
metadata
(
)
If the Labels of a ReplicationController are empty, they are defaulted to be the same as the Pod(s) that the replication controller manages. Standard object's metadata. More info:
spec
(
ReplicationControllerSpec
)
Spec defines the specification of the desired behavior of the replication controller. More info:
status
(
ReplicationControllerStatus
)
Status is the most recently observed status of the replication controller. This data may be out of date by some window of time. Populated by the system. Read-only. More info:
ReplicationControllerSpec is the specification of a replication controller.
selector
Selector is a label query over pods that should match the Replicas count. If Selector is empty, it is defaulted to the labels present on the Pod template. Label keys and values that must match in order to be controlled by this replication controller, if empty defaulted to labels on Pod template. More info:
template
(
)
Template is the object that describes the pod that will be created if insufficient replicas are detected. This takes precedence over a TemplateRef. More info:
replicas
(int32)
https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller
(int32)
Replica

tionControllerStatus
ReplicationControllerStatus represents the current status of a replication controller.
replicas
(int32)
The number of available replicas (ready for at least minReadySeconds) for this replication controller.
(int32)
The number of ready replicas for this replication controller.
(int32)
The number of pods that have labels matching the labels of the pod template of the replication controller.
([]ReplicationControllerCondition)
type
Represents the latest available observations of a replication controller's current state.
ReplicationControllerCondition describes the state of a replication controller at a certain point.
Type of replication controller condition.
(Time)
(string)
(string)
(int64)
ObservedGeneration reflects the generation of the most recently observed replication controller.
ReplicationControllerList
ReplicationControllerList is a collection of replication controllers.
: v1
kind
: ReplicationControllerList
metadata
(
ListMeta
)
items
([]
List of replication controllers. More info:
get
read the specified ReplicationController
GET /api/v1/namespaces/{namespace}/replicationcontrollers/{name}
name
(
in path
name of the ReplicationController
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified ReplicationController
GET /api/v1/namespaces/{namespace}/replicationcontrollers/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ReplicationController
GET /api/v1/namespaces/{namespace}/replicationcontrollers
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/replicationcontrollers
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query

): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ReplicationController
POST /api/v1/namespaces/{namespace}/replicationcontrollers
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ReplicationController
PUT /api/v1/namespaces/{namespace}/replicationcontrollers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified ReplicationController
PUT /api/v1/namespaces/{namespace}/replicationcontrollers/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ReplicationController
PATCH /api/v1/namespaces/{namespace}/replicationcontrollers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified ReplicationController
PATCH /api/v1/namespaces/{namespace}/replicationcontrollers/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ReplicationController
DELETE /api/v1/namespaces/{namespace}/replicationcontrollers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ReplicationController
DELETE /api/v1/namespaces/{namespace}/replicationcontrollers
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
graceP

eriodSeconds
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.4 - ReplicaSet
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
template
(
)
replicas
(int32)
(int32)
replicas
(int32)
The number of available replicas (ready for at least minReadySeconds) for this replica set.
(int32)
(int32)
type
(Time)
(string

)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
fiel

dValidation
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.5 - Deployment
Deployment enables declarative updates for Pods and ReplicaSets.
: apps/v1
kind
: Deployment
metadata
(
)
spec
(
DeploymentSpec
)
Specification of the desired behavior of the Deployment.
status
(
DeploymentStatus
)
Most recently observed status of the Deployment.
DeploymentSpec is the specification of the desired behavior of the Deployment.
selector
(
Label selector for pods. Existing ReplicaSets whose pods are selected by this will be the ones affected by this deployment. It must match the pod template's labels.
template
(
Template describes the pods that will be created.
replicas
(int32)
Number of desired pods. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1.
(int32)
strategy
(DeploymentStrategy)
Patch strategy: retainKeys
The deployment strategy to use to replace existing pods with new ones.
DeploymentStrategy describes how to replace

 existing pods with new ones.
strategy.type
(string)
Type of deployment. Can be "Recreate" or "RollingUpdate". Default is RollingUpdate.
strategy.rollingUpdate
(RollingUpdateDeployment)
Rolling update config params. Present only if DeploymentStrategyType = RollingUpdate.
Spec to control the desired behavior of rolling update.
strategy.rollingUpdate.maxSurge
The maximum number of pods that can be scheduled above the desired number of pods. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up. Defaults to 25%. Example: when this is set to 30%, the new ReplicaSet can be scaled up immediately when the rolling update starts, such that the total number of old and new pods do not exceed 130% of desired pods. Once old pods have been killed, new ReplicaSet can be scaled up further, ensuring that total number of pods running at any time during the update is at most 130% of desired pods.
strategy.rollingUpdate.maxUnavailable
The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. Defaults to 25%. Example: when this is set to 30%, the old ReplicaSet can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old ReplicaSet can be scaled down further, followed by scaling up the new ReplicaSet, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods.
(int32)
The number of old ReplicaSets to retain to allow rollback. This is a pointer to distinguish between explicit zero and not specified. Defaults to 10.
progressDeadlineSeconds
(int32)
The maximum time in seconds for a deployment to make progress before it is considered to be failed. The deployment controller will continue to process failed deployments and a condition with a ProgressDeadlineExceeded reason will be surfaced in the deployment status. Note that progress will not be estimated during the time a deployment is paused. Defaults to 600s.
paused
(boolean)
Indicates that the deployment is paused.
DeploymentStatus is the most recently observed status of the Deployment.
replicas
(int32)
Total number of non-terminated pods targeted by this deployment (their labels match the selector).
(int32)
Total number of available pods (ready for at least minReadySeconds) targeted by this deployment.
(int32)
readyReplicas is the number of pods targeted by this Deployment with a Ready Condition.
unavailableReplicas
(int32)
Total number of unavailable pods targeted by this deployment. This is the total number of pods that are still required for the deployment to have 100% available capacity. They may either be pods that are running but not yet available or pods that still have not been created.
(int32)
Total number of non-terminated pods targeted by this deployment that have the desired template spec.
(int32)
Count of hash collisions for the Deployment. The Deployment controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ReplicaSet.
([]DeploymentCondition)
type
Represents the latest

 available observations of a deployment's current state.
DeploymentCondition describes the state of a deployment at a certain point.
Type of deployment condition.
(Time)
conditions.lastUpdateTime
(Time)
The last time this condition was updated.
(string)
(string)
(int64)
The generation observed by the deployment controller.
DeploymentList
DeploymentList is a list of Deployments.
: apps/v1
kind
: DeploymentList
metadata
(
ListMeta
)
items
([]
Items is the list of Deployments.
get
read the specified Deployment
GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}
name
(
in path
name of the Deployment
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified Deployment
GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind Deployment
GET /apis/apps/v1/namespaces/{namespace}/deployments
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/apps/v1/deployments
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a Deployment
POST /apis/apps/v1/namespaces/{namespace}/deployments
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified Deployment
PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
fi

eldManager
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified Deployment
PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified Deployment
PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified Deployment
PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a Deployment
DELETE /apis/apps/v1/namespaces/{namespace}/deployments/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Deployment
DELETE /apis/apps/v1/namespaces/{namespace}/deployments
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.6 - StatefulSet
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
A StatefulSetSpe

c is the specification of a StatefulSet.
selector
(
template
(
replicas
(int32)
StatefulSetUpdateStrategy indicates the strategy that the StatefulSet controller will use to perform updates. It includes any additional parameters necessary to perform the update for the indicated strategy.
(string)
(int32)
(string)
Parallel
(int32)
revisionHistoryLimit is the maximum number of revisions

 that will be maintained in the StatefulSet's revision history. The revision history consists of all revisions not represented by a currently applied StatefulSetSpec version. The default value is 10.
([]
)
(int32)
(string)
WhenDeleted specifies what happens to PVCs created from StatefulSet VolumeClaimTemplates when the StatefulSet is deleted. The default policy of
Retain
Delete
(string)
Retain
Delete
ordinals
(int32)
replicas
(int32)
(int32)
(int32)
(int32)
(int32)
collisionCount is the count of hash collis

ions for the StatefulSet. The StatefulSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.
type
(Time)
(string)
(string)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
b

ody
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.7 - ControllerRevision
ControllerRevision implements an immutable snapshot of state data.
ControllerRevision implements an immutable snapshot of state data. Clients are res

ponsible for serializing and deserializing the objects that contain their internal state. Once a ControllerRevision has been successfully created, it can not be updated. The API Server will fail validation of all requests that attempt to mutate the Data field. ControllerRevisions may, however, be deleted. Note that, due to its use by both the DaemonSet and StatefulSet controllers for update and rollback, this object is beta. However, it may be subject to name and representation changes in future releases, and clients should not depend on its stability. It is primarily for internal use by controllers.
: apps/v1
kind
: ControllerRevision
metadata
(
)
revision
(int64), required
Revision indicates the revision of the state represented by Data.
data
(RawExtension)
Data is the serialized representation of the state.
*RawExtension is used to hold extensions in external versions.
To use this, make a field which has RawExtension as its type in your external, versioned struct, and Object in your internal struct. You also need to register your various plugin types.
// Internal package:
type MyAPIObject struct {
runtime.TypeMeta
json:",inline"
MyPlugin runtime.Object
json:"myPlugin"
}
type PluginA struct {
AOption string
json:"aOption"
}
// External package:
MyPlugin runtime.RawExtension
}
}
// On the wire, the JSON will look something like this:
{
"kind":"MyAPIObject",
"apiVersion":"v1",
"myPlugin": {
"kind":"PluginA",
"aOption":"foo",
},
}
So what happens? Decode first uses json or yaml to unmarshal the serialized data into your external MyAPIObject. That causes the raw JSON to be stored, but not unpacked. The next step is to copy (using pkg/conversion) into the internal struct. The runtime package's DefaultScheme has conversion functions installed which will unpack the JSON stored in RawExtension, turning it into the correct object type, and storing it in the Object. (TODO: In the case where the object is of an unknown type, a runtime.Unknown object will be created and stored.)*
ControllerRevisionList
ControllerRevisionList is a resource containing a list of ControllerRevision objects.
: apps/v1
kind
: ControllerRevisionList
metadata
(
ListMeta
)
More info:
items
([]
Items is the list of ControllerRevisions
get
read the specified ControllerRevision
GET /apis/apps/v1/namespaces/{namespace}/controllerrevisions/{name}
name
(
in path
name of the ControllerRevision
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ControllerRevision
GET /apis/apps/v1/namespaces/{namespace}/controllerrevisions
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/apps/v1/controllerrevisions
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
labelS

elector
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ControllerRevision
POST /apis/apps/v1/namespaces/{namespace}/controllerrevisions
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ControllerRevision
PUT /apis/apps/v1/namespaces/{namespace}/controllerrevisions/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ControllerRevision
PATCH /apis/apps/v1/namespaces/{namespace}/controllerrevisions/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ControllerRevision
DELETE /apis/apps/v1/namespaces/{namespace}/controllerrevisions/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ControllerRevision
DELETE /apis/apps/v1/namespaces/{namespace}/controllerrevisions
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.8 - DaemonSet
DaemonSet
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
The current status of this daemon set. This data may be out of date by so

me window of time. Populated by the system. Read-only. More info:
selector
(
template
(
(int32)
(string)
The maximum number of nodes with an existing available DaemonSet pod that can have an updated DaemonSet pod during during an update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up to a minimum of 1. Default value is 0. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their a new pod created before the old pod is marked as deleted. The update starts by launching new pods on 30% of nodes. Once an updated pod is available (Ready for at least minReadySeconds) the old DaemonSet pod on that node is marked deleted. If the old pod becomes unavailable for any reason (Ready transitions to false, is evicted, or is drained) an updated pod is immediatedly created on that node without considering surge limits. Allowing surge implies the possibility that the resources consumed by the daemonset on any given node can double if the readiness check fails, and so resource intensive daemonsets should take into account that they may cause evictions during disruption.
The maximum number of DaemonSet pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of total number of DaemonSet pods at the start of the update (ex: 10%). Absolute number is calculated from percentage by rounding up. This cannot be 0 if MaxSurge is 0 Default value is 1. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their pods stopped for an update at any given time. The update starts by stopping at most 30% of those DaemonSet pods and then brings up new DaemonSet pods in their place. Once the new pods are available, it then proceeds onto other DaemonSet pods, thus ensuring that at least 70% of original number of DaemonSet pods ar

e available at all times during the update.
(int32)
(int32)
(int32)
(int32)
(int32)
Count of hash collisions for the DaemonSet. The DaemonSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.
type
(Time)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
DaemonSet
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
list
GET /apis/apps/v1/namespaces/{nam

espace}/daemonsets
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
202 (
DaemonSet
update
name
(
in path
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
delete
name
(
in path
namespace
(
in path
): str

ing, required
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.9 - Job
Job represents the configuration of a single job.
import "k8s.io/api/batch/v1"
Job
: batch/v1
kind
: Job
metadata
(
)
spec
(
JobSpec
)
Specification of the desired behavior of a job. More info:
status
(
JobStatus
)
Current status of a job. More info:
JobSpec
JobSpec describes how the job execution will look like.
Replicas
template
(
Describes the pod that will be created when executing a job. More info:
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
(int32)
Specifies the maximum desired number of pods the job should run at any given time. The actual number of pods running in steady state will be less than this number when ((.spec.completions - .status.successful) &lt; .spec.parallelism), i.e. when the work left to do is less than max parallelism. More info:
Lifecycle
(int32)
Specifies the desired number of successfully finished pods the job should be run with. Setting to nil means that the success of any pod signals the success of all pods, and allows parallelism to have any positive value. Setting to 1 means that parallelism is limited to 1 and the success of that pod signals the success of the job. More info:
(string)
CompletionMode specifies how Pod completions are tracked. It can be
NonIndexed
(default) or
Indexed
.
means that the Job is considered complete when there have been .spec.completions successfully completed Pods. Each Pod completion is homologous to each other.
Indexed
means that the Pods of a Job get an associated completion index from 0 to (.spec.completions - 1), available in the annotation batch.kubernetes.io/job-completion-index. The Job is considered complete when there is one successfully completed Pod for each index. When value is
Indexed
, .spec.completions must be specified and
must be less than or equal to 10^5. In addition, The Pod name takes the form
$(job-name)-$(index)-$(random-string)
, the Pod hostname takes the form
$(job-name)-$(index)
.
More completion modes can be added in the future. If the Job controller observes a mode that it doesn't recognize, which is possible during upgrades due to version skew, the controller skips updates for the Job.
backoffLimit
(int32)
Specifies the number of retries before marking this job fa

iled. Defaults to 6
(int64)
Specifies the duration in seconds relative to the startTime that the job may be continuously active before the system tries to terminate it; value must be positive integer. If a Job is suspended (at creation or through an update), this timer will effectively be stopped and reset when the Job is resumed again.
ttlSecondsAfterFinished
(int32)
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won't be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes.
suspend
(boolean)
Suspend specifies whether the Job controller should create Pods or not. If a Job is created with suspend set to true, no Pods are created by the Job controller. If a Job is suspended after creation (i.e. the flag goes from false to true), the Job controller will delete all active Pods associated with this Job. Users must design their workload to gracefully handle this. Suspending a Job will reset the StartTime field of the Job, effectively resetting the ActiveDeadlineSeconds timer too. Defaults to false.
Selector
selector
(
)
A label query over pods that should match the pod count. Normally, the system sets this field for you. More info:
manualSelector
(boolean)
manualSelector controls generation of pod labels and pod selectors. Leave
unset unless you are certain what you are doing. When false or unset, the system pick labels unique to this job and appends those labels to the pod template. When true, the user is responsible for picking unique labels and specifying the selector. Failure to pick a unique label may cause this and other jobs to not function correctly. However, You may see
manualSelector=true
in jobs that were created with the old
API. More info:
https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#specifying-your-own-pod-selector
podFailurePolicy
(PodFailurePolicy)
Specifies the policy of handling failed pods. In particular, it allows to specify the set of actions and conditions which need to be satisfied to take the associated action. If empty, the default behaviour applies - the counter of failed pods, represented by the jobs's .status.failed field, is incremented and it is checked against the backoffLimit. This field cannot be used in combination with restartPolicy=OnFailure.
This field is alpha-level. To use this field, you must enable the
JobPodFailurePolicy
feature gate (disabled by default).
PodFailurePolicy describes how failed pods influence the backoffLimit.
podFailurePolicy.rules
([]PodFailurePolicyRule), required
A list of pod failure policy rules. The rules are evaluated in order. Once a rule matches a Pod failure, the remaining of the rules are ignored. When no rule matches the Pod failure, the default handling applies - the counter of pod failures is incremented and it is checked against the backoffLimit. At most 20 elements are allowed.
PodFailurePolicyRule describes how a pod failure is handled when the requirements are met. One of OnExitCodes and onPodConditions, but not both, can be used in each rule.
podFailurePolicy.rules.action
Specifies the action taken on a pod failure when the requirements are satisfied. Possible values are:
FailJob: indicates that the pod's job is marked as Failed and all
running pods are terminated.
Ignore: indicates that the counter towards the .backoffLimit is not
incremented and a replacement pod is created.
Count: indicates that the pod is handled in the defaul

t way - the
counter towards the .backoffLimit is incremented.
Additional values are considered to be added in the future. Clients should react to an unknown action by skipping the rule.
podFailurePolicy.rules.onPodConditions
([]PodFailurePolicyOnPodConditionsPattern), required
Represents the requirement on the pod conditions. The requirement is represented as a list of pod condition patterns. The requirement is satisfied if at least one pattern matches an actual pod condition. At most 20 elements are allowed.
PodFailurePolicyOnPodConditionsPattern describes a pattern for matching an actual pod condition type.
podFailurePolicy.rules.onPodConditions.status
Specifies the required Pod condition status. To match a pod condition it is required that the specified status equals the pod condition status. Defaults to True.
podFailurePolicy.rules.onPodConditions.type
Specifies the required Pod condition type. To match a pod condition it is required that specified type equals the pod condition type.
podFailurePolicy.rules.onExitCodes
(PodFailurePolicyOnExitCodesRequirement)
Represents the requirement on the container exit codes.
PodFailurePolicyOnExitCodesRequirement describes the requirement for handling a failed pod based on its container exit codes. In particular, it lookups the .state.terminated.exitCode for each app container and init container status, represented by the .status.containerStatuses and .status.initContainerStatuses fields in the Pod status, respectively. Containers completed with success (exit code 0) are excluded from the requirement check.
podFailurePolicy.rules.onExitCodes.operator
Represents the relationship between the container exit code(s) and the specified values. Containers completed with success (exit code 0) are excluded from the requirement check. Possible values are:
In: the requirement is satisfied if at least one container exit code
(might be multiple if there are multiple containers not restricted
by the 'containerName' field) is in the set of specified values.
NotIn: the requirement is satisfied if at least one container exit code
by the 'containerName' field) is not in the set of specified values.
Additional values are considered to be added in the future. Clients should react to an unknown operator by assuming the requirement is not satisfied.
podFailurePolicy.rules.onExitCodes.values
([]int32), required
Specifies the set of values. Each returned container exit code (might be multiple in case of multiple containers) is checked against this set of values with respect to the operator. The list of values must be ordered and must not contain duplicates. Value '0' cannot be used for the In operator. At least one element is required. At most 255 elements are allowed.
podFailurePolicy.rules.onExitCodes.containerName
(string)
Restricts the check for exit codes to the container with the specified name. When null, the rule applies to all containers. When specified, it should match one the container or initContainer names in the pod template.
JobStatus
JobStatus represents the current state of a Job.
startTime
(Time)
Represents time when the job controller started processing a job. When a Job is created in the suspended state, this field is not set until the first time it is resumed. This field is reset every time a Job is resumed from suspension. It is represented in RFC3339 form and is in UTC.
completionTime
(Time)
Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC. The completion time is only set when the job fi

nishes successfully.
active
(int32)
The number of pending and running pods.
failed
(int32)
The number of pods which reached phase Failed.
succeeded
(int32)
The number of pods which reached phase Succeeded.
completedIndexes
(string)
CompletedIndexes holds the completed indexes when .spec.completionMode = "Indexed" in a text format. The indexes are represented as decimal integers separated by commas. The numbers are listed in increasing order. Three or more consecutive numbers are compressed and represented by the first and last element of the series, separated by a hyphen. For example, if the completed indexes are 1, 3, 4, 5 and 7, they are represented as "1,3-5,7".
([]JobCondition)
type
The latest available observations of an object's current state. When a Job fails, one of the conditions will have type "Failed" and status true. When a Job is suspended, one of the conditions will have type "Suspended" and status true; when the Job is resumed, the status of this condition will become false. When a Job is completed, one of the conditions will have type "Complete" and status true. More info:
JobCondition describes current state of a job.
Type of job condition, Complete or Failed.
(Time)
Last time the condition was checked.
(Time)
Last time the condition transit from one status to another.
(string)
Human readable message indicating details about last transition.
(string)
(brief) reason for the condition's last transition.
uncountedTerminatedPods
(UncountedTerminatedPods)
UncountedTerminatedPods holds the UIDs of Pods that have terminated but the job controller hasn't yet accounted for in the status counters.
The job controller creates pods with a finalizer. When a pod terminates (succeeded or failed), the controller does three steps to account for it in the job status: (1) Add the pod UID to the arrays in this field. (2) Remove the pod finalizer. (3) Remove the pod UID from the arrays while increasing the corresponding
counter.
Old jobs might not be tracked using this field, in which case the field remains null.
UncountedTerminatedPods holds UIDs of Pods that have terminated but haven't been accounted in Job status counters.
uncountedTerminatedPods.failed
Failed holds UIDs of failed Pods.
uncountedTerminatedPods.succeeded
Succeeded holds UIDs of succeeded Pods.
Beta level
ready
(int32)
The number of pods which have a Ready condition.
This field is beta-level. The job controller populates the field when the feature gate JobReadyPods is enabled (enabled by default).
JobList
JobList is a collection of jobs.
: batch/v1
kind
: JobList
metadata
(
ListMeta
)
items
([]
Job
items is the list of Jobs.
get
read the specified Job
GET /apis/batch/v1/namespaces/{namespace}/jobs/{name}
name
(
in path
name of the Job
namespace
(
in path
): string, require

d
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
get
read status of the specified Job
GET /apis/batch/v1/namespaces/{namespace}/jobs/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
list
list or watch objects of kind Job
GET /apis/batch/v1/namespaces/{namespace}/jobs
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
list
GET /apis/batch/v1/jobs
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
create
create a Job
POST /apis/batch/v1/namespaces/{namespace}/jobs
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
202 (
Job
update
replace the specified Job
PUT /apis/batch/v1/namespaces/{namespace}/jobs/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
update
replace status of the specified Job
PUT /apis/batch/v1/namespaces/{namespace}/jobs/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch
partially update the specified Job
PATCH /apis/batch/v1/namespaces/{namespace}/jobs/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch
partially update status of the specified Job
PATCH /apis/batch/v1/namespaces/{namespace}/jobs/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(

in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
delete
delete a Job
DELETE /apis/batch/v1/namespaces/{namespace}/jobs/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Job
DELETE /apis/batch/v1/namespaces/{namespace}/jobs
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.10 - CronJob
CronJob represents the configuration of a single cron job.
CronJob
: batch/v1
kind
: CronJob
metadata
(
)
spec
(
CronJobSpec
)
Specification of the desired behavior of a cron job, including the schedule. More info:
status
(
CronJobStatus
)
Current status of a cron job. More info:
CronJobSpec describes how the job execution will look like and when it will actually run.
jobTemplate
(JobTemplateSpec), required
Specifies the job that will be created when executing a CronJob.
JobTemplateSpec describes the data a Job should have when created from a template
jobTemplate.metadata
(
)
Standard object's metadata of the jobs created from this template. More info:
jobTemplate.spec
(
JobSpec
)
Specification of the desired behavior of the job. More info:
schedule
The schedule in Cron format, see
https://en.wikipedia.org/wiki/Cron
.
timeZone
(string)
The time zone name for the given schedule, see
https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
. If not specified, this will default to the time zone of the kube-controller-manager process. The set of valid time zone names and the time zone offset is loaded from the system-wide time zone database by the API server during CronJob validation and the controller manager during execution. If no system-wide time zone database can be found a bundled version of the database is used instead. If the time zone name becomes invalid during the lifetime of a CronJob or due to a change in host configuration, the controller will stop creating new new Jobs and will create a system event with the reason UnknownTimeZone. More information can be found in
https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#time-zones
This is beta field and must be enabled via the
CronJobTimeZone
feature gate.
concurrencyPolicy
(string)
Specifies how to treat concurrent executions of a Job. Valid values are: - "Allow" (default): allows CronJobs to run concurrently; - "Forbid": forbids concurrent run

s, skipping next run if previous run hasn't finished yet; - "Replace": cancels currently running job and replaces it with a new one
startingDeadlineSeconds
(int64)
Optional deadline in seconds for starting the job if it misses scheduled time for any reason. Missed jobs executions will be counted as failed ones.
suspend
(boolean)
This flag tells the controller to suspend subsequent executions, it does not apply to already started executions. Defaults to false.
successfulJobsHistoryLimit
(int32)
The number of successful finished jobs to retain. Value must be non-negative integer. Defaults to 3.
failedJobsHistoryLimit
(int32)
The number of failed finished jobs to retain. Value must be non-negative integer. Defaults to 1.
CronJobStatus represents the current state of a cron job.
active
([]
)
A list of pointers to currently running jobs.
lastScheduleTime
(Time)
Information when was the last time the job was successfully scheduled.
lastSuccessfulTime
(Time)
Information when was the last time the job successfully completed.
CronJobList
CronJobList is a collection of cron jobs.
: batch/v1
kind
: CronJobList
metadata
(
ListMeta
)
items
([]
CronJob
items is the list of CronJobs.
get
read the specified CronJob
GET /apis/batch/v1/namespaces/{namespace}/cronjobs/{name}
name
(
in path
name of the CronJob
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
get
read status of the specified CronJob
GET /apis/batch/v1/namespaces/{namespace}/cronjobs/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
list
list or watch objects of kind CronJob
GET /apis/batch/v1/namespaces/{namespace}/cronjobs
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/batch/v1/cronjobs
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a CronJob
POST /apis/batch/v1/namespaces/{namespace}/cronjobs
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
prett

y
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
202 (
CronJob
update
replace the specified CronJob
PUT /apis/batch/v1/namespaces/{namespace}/cronjobs/{name}
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
update
replace status of the specified CronJob
PUT /apis/batch/v1/namespaces/{namespace}/cronjobs/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
partially update the specified CronJob
PATCH /apis/batch/v1/namespaces/{namespace}/cronjobs/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
partially update status of the specified CronJob
PATCH /apis/batch/v1/namespaces/{namespace}/cronjobs/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
delete
delete a CronJob
DELETE /apis/batch/v1/namespaces/{namespace}/cronjobs/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of CronJob
DELETE /apis/batch/v1/namespaces/{namespace}/cronjobs
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.11 - HorizontalPodAutoscaler
kind
metadata
(
)
spec
(
)
https://git.k8s.io/community/

contributors/devel/sig-architecture/api-conventions.md#spec-and-status
.
status
(
)
(string)
(int32)
(int32)
(int32)
current average CPU utilization over all pods, represented as a percentage of requested CPU, e.g. 70 means that an average pod is using now 70% of its requested CPU.
(Time)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
reso

urceVersion
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
DELE

TE /apis/autoscaling/v1/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.12 - HorizontalPodAutoscaler
HorizontalPodAutoscaler is the configuration for a horizontal pod autoscaler, which automatically manages the replica count of any resource implementing the scale subresource based on the metrics specified.
apiVersion: autoscaling/v2
import "k8s.io/api/autoscaling/v2"
: autoscaling/v2
kind
metadata
(
)
metadata is the standard object metadata. More info:
spec
(
)
spec is the specification for the behaviour of the autoscaler. More info:
.
status
(
)
status is the current information about the autoscaler.
HorizontalPodAutoscalerSpec describes the desired functionality of the HorizontalPodAutoscaler.
maxReplicas is the upper limit for the number of replicas to which the autoscaler can scale up. It cannot be less that minReplicas.
scaleTargetRef points to the target resource to scale, and is used to the pods for which metrics should be collected, as well as to actually change the replica count.
(string)
(int32)
behavior
(HorizontalPodAutoscalerBehavior)
behavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively). If not set, the default HPAScalingRules for scale up and scale down are used.
HorizontalPodAutoscalerBehavior configures the scaling behavior of

 the target in both Up and Down directions (scaleUp and scaleDown fields respectively).
behavior.scaleDown
(HPAScalingRules)
scaleDown is scaling policy for scaling Down. If not set, the default value is to allow to scale down to minReplicas pods, with a 300 second stabilization window (i.e., the highest recommendation for the last 300sec is used).
HPAScalingRules configures the scaling behavior for one direction. These Rules are applied after calculating DesiredReplicas from metrics for the HPA. They can limit the scaling velocity by specifying scaling policies. They can prevent flapping by specifying the stabilization window, so that the number of replicas is not set instantly, instead, the safest value from the stabilization window is chosen.
behavior.scaleDown.policies
([]HPAScalingPolicy)
policies is a list of potential scaling polices which can be used during scaling. At least one policy must be specified, otherwise the HPAScalingRules will be discarded as invalid
HPAScalingPolicy is a single policy which must hold true for a specified past interval.
behavior.scaleDown.policies.type
Type is used to specify the scaling policy.
behavior.scaleDown.policies.value
Value contains the amount of change which is permitted by the policy. It must be greater than zero
behavior.scaleDown.policies.periodSeconds
PeriodSeconds specifies the window of time for which the policy should hold true. PeriodSeconds must be greater than zero and less than or equal to 1800 (30 min).
behavior.scaleDown.selectPolicy
(string)
selectPolicy is used to specify which policy should be used. If not set, the default value Max is used.
behavior.scaleDown.stabilizationWindowSeconds
(int32)
StabilizationWindowSeconds is the number of seconds for which past recommendations should be considered while scaling up or scaling down. StabilizationWindowSeconds must be greater than or equal to zero and less than or equal to 3600 (one hour). If not set, use the default values: - For scale up: 0 (i.e. no stabilization is done). - For scale down: 300 (i.e. the stabilization window is 300 seconds long).
behavior.scaleUp
scaleUp is scaling policy for scaling Up. If not set, the default value is the higher of:
increase no more than 4 pods per 60 seconds
double the number of pods per 60 seconds
No stabilization is used.
behavior.scaleUp.policies
behavior.scaleUp.policies.type
behavior.scaleUp.policies.value
behavior.scaleUp.policies.periodSeconds
behavior.scaleUp.selectPolicy
(string)
behavior.scaleUp.stabilizationWindowSeconds
(int32)
StabilizationWindowSeconds is the number of seconds for which past recommendations should be considered while scaling up or scaling down. StabilizationWind

owSeconds must be greater than or equal to zero and less than or equal to 3600 (one hour). If not set, use the default values: - For scale up: 0 (i.e. no stabilization is done). - For scale down: 300 (i.e. the stabilization window is 300 seconds long).
metrics
([]MetricSpec)
metrics contains the specifications for which to use to calculate the desired replica count (the maximum replica count across all metrics will be used). The desired replica count is calculated multiplying the ratio between the target value and the current value by the current number of pods. Ergo, metrics used must decrease as the pod count is increased, and vice-versa. See the individual metric source types for more information about how each type of metric must respond. If not set, the default metric will be set to 80% average CPU utilization.
MetricSpec specifies how to scale based on a single metric (only
type
and one other matching field should be set at once).
metrics.type
type is the type of metric source. It should be one of "ContainerResource", "External", "Object", "Pods" or "Resource", each mapping to a matching field in the object. Note: "ContainerResource" type is available on when the feature-gate HPAContainerMetrics is enabled
metrics.containerResource
(ContainerResourceMetricSource)
containerResource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing a single container in each pod of the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source. This is an alpha feature and can be enabled by the HPAContainerMetrics feature flag.
ContainerResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). The values will be averaged together before being compared to the target. Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source. Only one "target" type should be set.
metrics.containerResource.container
container is the name of the container in the pods of the scaling target
metrics.containerResource.name
name is the name of the resource in question.
metrics.containerResource.target
(MetricTarget), required
target specifies the target value for the given metric
MetricTarget defines the target value, average value, or average utilization of a specific metric
metrics.containerResource.target.type
type represents whether the metric type is Utilization, Value, or AverageValue
metrics.containerResource.target.averageUtilization
(int32)
averageUtilization is the target value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods. Currently only valid for Resource metric source type
metrics.containerResource.target.averageValue
(
Quantity
)
averageValue is the target value of the average of the metric across all relevant pods (as a quantity)
metrics.containerResource.target.value
(
Quantity
)
value is the target value of the metric (as a quantity).
metrics.external
(ExternalMetricSource)
external refers to a global metric that is not associated with any Kubernetes object. It allows autoscaling based on information coming from components running outside of cluster (for example length of queue in cloud messaging service, or QPS from loadbalancer running outside of cluster).
ExternalMetricSource indicates how to scale on a metric not associated with any Kubernetes object (for example length of queue in cloud messaging service, or QPS from loadbalancer running outside of cluster).
metrics.external.metric
(MetricIdentifier), required
metric ident

ifies the target metric by name and selector
MetricIdentifier defines the name and optionally selector for a metric
metrics.external.metric.name
name is the name of the given metric
metrics.external.metric.selector
(
)
selector is the string-encoded form of a standard kubernetes label selector for the given metric When set, it is passed as an additional parameter to the metrics server for more specific metrics scoping. When unset, just the metricName will be used to gather metrics.
metrics.external.target
metrics.external.target.type
metrics.external.target.averageUtilization
(int32)
metrics.external.target.averageValue
(
Quantity
)
metrics.external.target.value
(
Quantity
)
metrics.object
(ObjectMetricSource)
object refers to a metric describing a single kubernetes object (for example, hits-per-second on an Ingress object).
ObjectMetricSource indicates how to scale on a metric describing a kubernetes object (for example, hits-per-second on an Ingress object).
metrics.object.describedObject
describedObject specifies the descriptions of a object,such as kind,name apiVersion
metrics.object.describedObject.kind
metrics.object.describedObject.name
metrics.object.describedObject.apiVersion
(string)
metrics.object.metric
metric identifies the target metric by name and selector
metrics.object.metric.name
metrics.object.metric.selector
(
)
metrics.object.target
metrics.object.target.type
metrics.object.target.averageUtilization
(int32)
metrics.object.target.averageValue
(
Quantity
)
metrics.object.target.value
(
Quantity
)
metrics.pods
(PodsMetricSource)
pods refers to a metric describing each pod in the current scale target (for example, transactions-processed-per-second). The values will be averaged together before being compared to the 

target value.
PodsMetricSource indicates how to scale on a metric describing each pod in the current scale target (for example, transactions-processed-per-second). The values will be averaged together before being compared to the target value.
metrics.pods.metric
metrics.pods.metric.name
metrics.pods.metric.selector
(
)
metrics.pods.target
metrics.pods.target.type
metrics.pods.target.averageUtilization
(int32)
metrics.pods.target.averageValue
(
Quantity
)
metrics.pods.target.value
(
Quantity
)
metrics.resource
(ResourceMetricSource)
resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
ResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). The values will be averaged together before being compared to the target. Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source. Only one "target" type should be set.
metrics.resource.name
metrics.resource.target
metrics.resource.target.type
metrics.resource.target.averageUtilization
(int32)
metrics.resource.target.averageValue
(
Quantity
)
metrics.resource.target.value
(
Quantity
)
HorizontalPodAutoscalerStatus describes the current status of a horizontal pod autoscaler.
desiredReplicas is the desired number of replicas of pods managed by this autoscaler, as last calculated by the autoscaler.
([]HorizontalPodAutoscalerCondition)
type
conditions is the set of conditions required for this autoscaler to scale its target, and indicates whether or not those conditions a

re met.
HorizontalPodAutoscalerCondition describes the state of a HorizontalPodAutoscaler at a certain point.
status is the status of the condition (True, False, Unknown)
type describes the current condition
(Time)
lastTransitionTime is the last time the condition transitioned from one status to another
(string)
message is a human-readable explanation containing details about the transition
(string)
reason is the reason for the condition's last transition.
currentMetrics
([]MetricStatus)
currentMetrics is the last read state of the metrics used by this autoscaler.
MetricStatus describes the last-read state of a single metric.
currentMetrics.type
type is the type of metric source. It will be one of "ContainerResource", "External", "Object", "Pods" or "Resource", each corresponds to a matching field in the object. Note: "ContainerResource" type is available on when the feature-gate HPAContainerMetrics is enabled
currentMetrics.containerResource
(ContainerResourceMetricStatus)
container resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing a single container in each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
ContainerResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing a single container in each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
currentMetrics.containerResource.container
Container is the name of the container in the pods of the scaling target
currentMetrics.containerResource.current
(MetricValueStatus), required
current contains the current value for the given metric
MetricValueStatus holds the current value for a metric
currentMetrics.containerResource.current.averageUtilization
(int32)
currentAverageUtilization is the current value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods.
currentMetrics.containerResource.current.averageValue
(
Quantity
)
averageValue is the current value of the average of the metric across all relevant pods (as a quantity)
currentMetrics.containerResource.current.value
(
Quantity
)
value is the current value of the metric (as a quantity).
currentMetrics.containerResource.name
Name is the name of the resource in question.
currentMetrics.external
(ExternalMetricStatus)
ExternalMetricStatus indicates the current value of a global metric not associated with any Kubernetes object.
currentMetrics.external.current
currentMetrics.external.current.averageUtilization
(int32)
currentMetrics.external.current.averageValue
(
Quantity
)
aver

ageValue is the current value of the average of the metric across all relevant pods (as a quantity)
currentMetrics.external.current.value
(
Quantity
)
currentMetrics.external.metric
currentMetrics.external.metric.name
currentMetrics.external.metric.selector
(
)
currentMetrics.object
(ObjectMetricStatus)
ObjectMetricStatus indicates the current value of a metric describing a kubernetes object (for example, hits-per-second on an Ingress object).
currentMetrics.object.current
currentMetrics.object.current.averageUtilization
(int32)
currentMetrics.object.current.averageValue
(
Quantity
)
currentMetrics.object.current.value
(
Quantity
)
currentMetrics.object.describedObject
DescribedObject specifies the descriptions of a object,such as kind,name apiVersion
currentMetrics.object.describedObject.kind
currentMetrics.object.describedObject.name
currentMetrics.object.describedObject.apiVersion
(string)
currentMetrics.object.metric
currentMetrics.object.metric.name
currentMetrics.object.metric.selector
(
)
currentMetrics.pods
(PodsMetricStatus)
pods refers to a metric describing each pod in the current scale target (for example, transactions-processed-per-second). The values will be averaged together before being compared to the target value.
PodsMetricStatus indicates the current value of a metric describing each pod in the current scale target (for example, transactions-processed-per-second).
currentMetrics.pods.current
currentMetrics.pods.current.averageUtilization
(int32)
currentMetrics.pods.current.averageValue
(
Quantity
)
averageValue is the current value of the 

average of the metric across all relevant pods (as a quantity)
currentMetrics.pods.current.value
(
Quantity
)
currentMetrics.pods.metric
currentMetrics.pods.metric.name
currentMetrics.pods.metric.selector
(
)
currentMetrics.resource
(ResourceMetricStatus)
ResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
currentMetrics.resource.current
currentMetrics.resource.current.averageUtilization
(int32)
currentMetrics.resource.current.averageValue
(
Quantity
)
currentMetrics.resource.current.value
(
Quantity
)
currentMetrics.resource.name
(int32)
currentReplicas is current number of replicas of pods managed by this autoscaler, as last seen by the autoscaler.
(Time)
lastScaleTime is the last time the HorizontalPodAutoscaler scaled the number of pods, used by the autoscaler to control how often the number of pods is changed.
(int64)
observedGeneration is the most recent generation observed by this autoscaler.
HorizontalPodAutoscalerList is a list of horizontal pod autoscaler objects.
kind
metadata
(
ListMeta
)
metadata is the standard list metadata.
items
([]
items is the list of horizontal pod autoscaler objects.
get
GET /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
GET /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of k

ind HorizontalPodAutoscaler
GET /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/autoscaling/v2/horizontalpodautoscalers
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
POST /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
PUT /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
PUT /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
name
(
in path
name of the HorizontalPodAut

oscaler
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
DELETE /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
DELETE /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.13 - PriorityClass
PriorityClass defines mapping from a priority class name to the priority integer value.
apiVersion: scheduling.k8s.io/v1
import "k8s.io/api/scheduling/v1"
PriorityClass defines mapping from a priority class name to the priority integer value. The value can be any valid integer.
: scheduling.k8s.io/v1
kind
: PriorityClass
metadata
(
)
value
The value of this priority class. This is the actual priority that pods receive when they have the name of this class in their pod spec.
(string)
description is an arbitrary string that usually provides guidelines on when this priority class should be used.
globalDefault
(boolean)
globalDefault specifies whether this PriorityClass should be considered as the default priority for pods that do not have any priority class. Only one PriorityClass can be marked as
. However, if more than one PriorityClasses exists with their
field set to true, the smallest value of such global default PriorityClasses will be used as the default priority.
(string)
PriorityClassList
PriorityClassList is a collection of priority classes.
kind
: PriorityClassList
metadata
(
ListMeta
)
items
([]
items is the list of PriorityClasses
get
read the specified PriorityClass
GET /apis/scheduling.k8s.io/v1/priorityclasses/{name}
name
(
in path
name of the PriorityClass
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind PriorityClass
GET /apis/scheduling.k8s.io/v1/priorityclasses
(
in query
): boolean
continue
(
in query
): string
continue

(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a PriorityClass
POST /apis/scheduling.k8s.io/v1/priorityclasses
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified PriorityClass
PUT /apis/scheduling.k8s.io/v1/priorityclasses/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified PriorityClass
PATCH /apis/scheduling.k8s.io/v1/priorityclasses/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a PriorityClass
DELETE /apis/scheduling.k8s.io/v1/priorityclasses/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of PriorityClass
DELETE /apis/scheduling.k8s.io/v1/priorityclasses
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.14 - PodScheduling v1alpha1
PodScheduling objects hold information that is needed to schedule a Pod with ResourceClaims that use "WaitForFirstConsumer" allocation mode.
apiVersion: resource.k8s.io/v1alpha1
import "k8s.io/api/resource/v1alpha1"
PodScheduling
This is an alpha type and requires enabling the DynamicResourceAllocation feature gate.
: resource.k8s.io/v1alpha1
kind
: PodScheduling
metadata
(
)
Standard object metadata
spec
(
PodSchedulingSpec
Spec describes where resources for the Pod are needed.
status
(
PodSchedulingStatus
)
Status describes where resources for the Pod can be allocated.
PodSchedulingSpec describes where resources for the Pod are needed.
potentialNodes
PotentialNodes lists nodes where the Pod might be able to run.
The 

size of this field is limited to 128. This is large enough for many clusters. Larger clusters may need more attempts to find a node that suits all pending resources. This may get increased in the future, but not reduced.
selectedNode
(string)
SelectedNode is the node for which allocation of ResourceClaims that are referenced by the Pod and that use "WaitForFirstConsumer" allocation is to be attempted.
PodSchedulingStatus describes where resources for the Pod can be allocated.
([]ResourceClaimSchedulingStatus)
ResourceClaims describes resource availability for each pod.spec.resourceClaim entry where the corresponding ResourceClaim uses "WaitForFirstConsumer" allocation mode.
ResourceClaimSchedulingStatus contains information about one particular ResourceClaim with "WaitForFirstConsumer" allocation mode.
(string)
Name matches the pod.spec.resourceClaims[*].Name field.
resourceClaims.unsuitableNodes
UnsuitableNodes lists nodes that the ResourceClaim cannot be allocated for.
The size of this field is limited to 128, the same as for PodSchedulingSpec.PotentialNodes. This may get increased in the future, but not reduced.
PodSchedulingList
PodSchedulingList is a collection of Pod scheduling objects.
kind
: PodSchedulingList
metadata
(
ListMeta
)
Standard list metadata
items
([]
Items is the list of PodScheduling objects.
get
read the specified PodScheduling
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings/{name}
name
(
in path
name of the PodScheduling
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified PodScheduling
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind PodScheduling
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/resource.k8s.io/v1alpha1/podschedulings
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a PodScheduling
POST /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pr

etty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified PodScheduling
PUT /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified PodScheduling
PUT /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified PodScheduling
PATCH /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified PodScheduling
PATCH /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a PodScheduling
DELETE /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of PodScheduling
DELETE /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/podschedulings
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.15 - ResourceClaim v1alpha1
ResourceClaim describes which resources are needed by a resource consumer.
ResourceClaim
ResourceClaim describes which resourc

es are needed by a resource consumer. Its status tracks whether the resource has been allocated and what the resulting attributes are.
kind
: ResourceClaim
metadata
(
)
spec
(
ResourceClaimSpec
Spec describes the desired attributes of a resource that then needs to be allocated. It can only be set once when creating the ResourceClaim.
status
(
ResourceClaimStatus
)
Status describes whether the resource is available and with which attributes.
ResourceClaimSpec defines how a resource is to be allocated.
resourceClassName
ResourceClassName references the driver and additional parameters via the name of a ResourceClass that was created as part of the driver deployment.
allocationMode
(string)
Allocation can start immediately or when a Pod wants to use the resource. "WaitForFirstConsumer" is the default.
parametersRef
(ResourceClaimParametersReference)
ParametersRef references a separate object with arbitrary parameters that will be used by the driver when allocating a resource for the claim.
The object must be in the same namespace as the ResourceClaim.
ResourceClaimParametersReference contains enough information to let you locate the parameters for a ResourceClaim. The object must be in the same namespace as the ResourceClaim.
parametersRef.kind
Kind is the type of resource being referenced. This is the same value as in the parameter object's metadata, for example "ConfigMap".
parametersRef.name
Name is the name of resource being referenced.
parametersRef.apiGroup
(string)
APIGroup is the group for the resource being referenced. It is empty for the core API. This matches the group in the APIVersion that is used when creating the resources.
ResourceClaimStatus tracks whether the resource has been allocated and what the resulting attributes are.
allocation
(AllocationResult)
Allocation is set by the resource driver once a resource has been allocated successfully. If this is not specified, the resource is not yet allocated.
AllocationResult contains attributed of an allocated resource.
allocation.availableOnNodes
This field will get set by the resource driver after it has allocated the resource driver to inform the scheduler where it can schedule Pods using the ResourceClaim.
Setting this field is optional. If null, the resource is available everywhere.
allocation.availableOnNodes.nodeSelectorTerms
allocation.availableOnNodes.nodeSelectorTerms.matchExpressions
([]
)
allocation.availableOnNodes.nodeSelectorTerms.matchFields
([]
)
allocation.resourceHandle
(string)
ResourceHandle contains arbitrary data returned by the driver after a successful allocation. This is opaque for Kubernetes. Driver documentation may explain to users how to interpret this data if needed.
The maximum size of this field is 16KiB. This may get increased in the future, but not reduced.
allocation.shareable
(boolean)
Shareable determines whether the resource supports more than one consumer at a time.
deallocationRequested
(boolean)
DeallocationRequested indicates that a ResourceClaim is to be deallocated.
The driver then must deallocate this claim and reset the field together with clearing t

he Allocation field.
While DeallocationRequested is set, no new consumers may be added to ReservedFor.
driverName
(string)
DriverName is a copy of the driver name from the ResourceClass at the time when allocation started.
reservedFor
([]ResourceClaimConsumerReference)
ReservedFor indicates which entities are currently allowed to use the claim. A Pod which references a ResourceClaim which is not reserved for that Pod will not be started.
There can be at most 32 such reservations. This may get increased in the future, but not reduced.
ResourceClaimConsumerReference contains enough information to let you locate the consumer of a ResourceClaim. The user must be a resource in the same namespace as the ResourceClaim.
reservedFor.name
reservedFor.resource
Resource is the type of resource being referenced, for example "pods".
reservedFor.uid
UID identifies exactly one incarnation of the resource.
reservedFor.apiGroup
(string)
ResourceClaimList
ResourceClaimList is a collection of claims.
kind
: ResourceClaimList
metadata
(
ListMeta
)
items
([]
Items is the list of resource claims.
get
read the specified ResourceClaim
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims/{name}
name
(
in path
name of the ResourceClaim
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified ResourceClaim
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ResourceClaim
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/resource.k8s.io/v1alpha1/resourceclaims
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ResourceClaim
POST /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (

): Created
202 (
update
replace the specified ResourceClaim
PUT /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified ResourceClaim
PUT /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ResourceClaim
PATCH /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified ResourceClaim
PATCH /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ResourceClaim
DELETE /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of ResourceClaim
DELETE /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaims
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.16 - ResourceClaimTemplate v1alpha1
ResourceClaimTemplate is used to produce ResourceClaim objects.
ResourceClaimTemplate
kin

d
: ResourceClaimTemplate
metadata
(
)
spec
(
ResourceClaimTemplateSpec
Describes the ResourceClaim that is to be generated.
This field is immutable. A ResourceClaim will get created by the control plane for a Pod when needed and then not get updated anymore.
ResourceClaimTemplateSpec contains the metadata and fields for a ResourceClaim.
spec
(
Spec for the ResourceClaim. The entire content is copied unchanged into the ResourceClaim that gets created from this template. The same fields as in a ResourceClaim are also valid here.
metadata
(
)
ObjectMeta may contain labels and annotations that will be copied into the PVC when creating it. No other fields are allowed and will be rejected during validation.
ResourceClaimTemplateList
ResourceClaimTemplateList is a collection of claim templates.
kind
: ResourceClaimTemplateList
metadata
(
ListMeta
)
items
([]
Items is the list of resource claim templates.
get
read the specified ResourceClaimTemplate
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaimtemplates/{name}
name
(
in path
name of the ResourceClaimTemplate
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ResourceClaimTemplate
GET /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaimtemplates
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/resource.k8s.io/v1alpha1/resourceclaimtemplates
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ResourceClaimTemplate
POST /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaimtemplates
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ResourceClaimTemplate
PUT /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaimtemplates/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
)

: OK
201 (
): Created
patch
partially update the specified ResourceClaimTemplate
PATCH /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaimtemplates/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ResourceClaimTemplate
DELETE /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaimtemplates/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of ResourceClaimTemplate
DELETE /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourceclaimtemplates
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.1.17 - ResourceClass v1alpha1
ResourceClass is used by administrators to influence how resources are allocated.
ResourceClass
kind
: ResourceClass
metadata
(
)
DriverName defines the name of the dynamic resource driver that is used for allocation of a ResourceClaim that uses this class.
Resource drivers have a unique name in forward domain order (acme.example.com).
(ResourceClassParametersReference)
ParametersRef references an arbitrary separate object that may hold parameters that will be used by the driver when allocating a resource that uses this class. A dynamic resource driver can distinguish between parameters stored here and and those stored in ResourceClaimSpec.
ResourceClassParametersReference contains enough information to let you locate the parameters for a ResourceClass.
Kind is the type of resource being referenced. This is the same value as in the parameter object's metadata.
(string)
parametersRef.namespace
(string)
Namespace that contains the referenced resource. Must be empty for cluster-scoped resources and non-empty for namespaced resources.
suitableNodes
Only nodes matching the selector will be considered by the scheduler when trying to find a Node that fits a Pod when that Pod uses a Resour

ceClaim that has not been allocated yet.
Setting this field is optional. If null, all nodes are candidates.
suitableNodes.nodeSelectorTerms
suitableNodes.nodeSelectorTerms.matchExpressions
([]
)
suitableNodes.nodeSelectorTerms.matchFields
([]
)
ResourceClassList
ResourceClassList is a collection of classes.
kind
: ResourceClassList
metadata
(
ListMeta
)
items
([]
Items is the list of resource classes.
get
read the specified ResourceClass
GET /apis/resource.k8s.io/v1alpha1/resourceclasses/{name}
name
(
in path
name of the ResourceClass
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ResourceClass
GET /apis/resource.k8s.io/v1alpha1/resourceclasses
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ResourceClass
POST /apis/resource.k8s.io/v1alpha1/resourceclasses
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ResourceClass
PUT /apis/resource.k8s.io/v1alpha1/resourceclasses/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ResourceClass
PATCH /apis/resource.k8s.io/v1alpha1/resourceclasses/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ResourceClass
DELETE /apis/resource.k8s.io/v1alpha1/resourceclasses/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of ResourceClass
DELETE /apis/resource.k8s.io/v1alph

a1/resourceclasses
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.2 - Service Resources
5.2.1 - Service
Service
: v1
kind
: Service
metadata
(
)
spec
(
)
status
(
)
selector
ports
port
(string)
(string)
(int32)
The port on each node on which this service is exposed when type is NodePort or LoadBalancer. Usually assigned 

by the system. If a value is specified, in-range, and not in use it will be used, otherwise the operation will fail. If not specified, a port will be allocated if this Service requires one. If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type from NodePort to ClusterIP). More info:
(string)
type
(string)
(string)
IPFamilyPolicy represents the dual-stack-ness requested or required by this Service. If there is no value provided, then this field will be set to SingleStack. Services can be "SingleStack" (a single IP family), "PreferDualStack" (two IP families on dual-stack configured clusters or a single IP family on single-stack clusters), or "RequireDualStack" (two IP families on dual-stack configured clusters, otherwise fail). The ipFamilies and clusterIPs fields depend on the value of this field. This field will be wiped when updating a service to type ExternalName.
clusterIP
(string)
clusterIP is the IP address of the service and is usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be blank) or the type field is being changed from ExternalName (in which

 case this field may optionally be specified, as describe above). Valid values are "None", empty string (""), or a valid IP address. Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. More info:
(string)
(string)
Only applies to Service Type: LoadBalancer. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature. Deprecated: This field was under-specified and its meaning varies across implementations, and it cannot support dual-stack. As of Kubernetes v1.24, users are encouraged to use implementation-specific annotations when available. This field may be removed in a future API version.
(string)
loadBalancerClass is the class of the load balancer implementation this Service belongs to. If specified, the value of this field must be a label-style identifier, with an optional prefix, e.g. "interna

l-vip" or "example.com/internal-vip". Unprefixed names are reserved for end-users. This field can only be set when the Service type is 'LoadBalancer'. If not set, the default load balancer implementation is used, today this is typically done through the cloud provider integration, but should apply for any default implementation. If set, it is assumed that a load balancer implementation is watching for Services with a matching class. Any default load balancer implementation (e.g. cloud providers) should ignore Services that set this field. This field can only be set when creating or updating a Service to type 'LoadBalancer'. Once set, it can not be changed. This field will be wiped when a service is updated to a non 'LoadBalancer' type.
(string)
type
(string)
(string)
(int32)
(boolean)
publishNotReadyAddresses indicates that any agent which deals with endpoints for this Service should disregard any indications of ready/not-ready. The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery. The Kubernetes controllers that generate Endpoints and EndpointSlice resources for Services interpret this to mean that all endpoints are considered "ready" even if the Pods themselves are not. Agents which consume only Kubernetes generated endpoints through the Endpoints or EndpointSlice resources can safely assume this behavior.
SessionAffinityConfig represents the configurati

ons of session affinity.
(int32)
(boolean)
type
(int64)
LoadBalancerIngress represents the status of a load-balancer ingress point: traffic intended for the service should be sent to an ingress point.
(string)
(string)
**
Protocol is the protocol of the service port of which status is recorded here Th

e supported values are: "TCP", "UDP", "SCTP"
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Service
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
202 (
Service
update
name
(
in path
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
4

patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Service
): OK
202 (
Service
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.2.2 - Endpoints
Endpoints
 Subsets: [
   {
   },
   {
   },
]
: v1
kind
metadata
(
)
subsets
{
}
I

P addresses which offer the related ports that are marked as ready. These endpoints should be considered safe for load balancers and clients to utilize.
(string)
(string)
(
)
IP addresses which offer the related ports but are not currently marked as ready because they have not yet finished starting, have recently failed a readiness check, or have recently failed a liveness check.
(string)
(string)
(
)
(string)
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Endpoints
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list

(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
202 (
Endpoints
update
name
(
in path
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.2.3 - EndpointSlice
kind
metadata
(
)
addressType specifies the type of address carried by this EndpointSlice. All addresses in this slice must be the same type. This field is immutable after creation. The following address types are currently supported: * IPv4: Repres

ents an IPv4 Address. * IPv6: Represents an IPv6 Address. * FQDN: Represents a Fully Qualified Domain Name.
endpoints
(boolean)
(boolean)
(boolean)
terminating indicates that this endpoint is terminating. A nil value indicates an unknown state. Consumers should interpret this unknown state to mean that the endpoint is not terminating.
(string)
(string)
(
)
(string)
ports
(int32

)
(string)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Crea

ted
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.2.4 - Ingress
Ingress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend.
apiVersion: networking.k8s.io/v1
import "k8s.io/api/networking/v1"
Ingress
Ingress is a collection of rules that allow inbound connections to reach the endpoints defined by a backend. An Ingress can be configured to give services externally-reachable urls, load balance traffic, terminate SSL, offer name based virtual hosting etc.
: networking.k8s.io/v1
kind
: Ingress
metadata
(
)
spec
(
IngressSpec
)
Spec is the desired state of the Ingress. More info:
status
(
IngressStatus
)
Status is the current state of the Ingress. More info:
IngressSpec describes the Ingress the user wishes to exist.
defaultBackend
(
IngressBackend
)
DefaultBackend is the backend that should handle requests that don't match any rule. If Rules are not specified, DefaultBackend must be specified. If DefaultBackend is not set, the handling of requests that do not match any of the rules will be up to the Ingress controller.
ingressClassName
(string)
IngressClassName is the name of an IngressClass cluster resource. Ingress controller implementations use this field to know whether they should be serving this Ingress resource, by a transitive connection (controller -&gt; IngressClass -&gt; Ingress resource). Although the
kubernetes.io/ingress.class
annotation (simple constant name) was never formally defined, it was widely supported by Ingress controllers to create a direct binding between Ingress controller and Ingress resources. Newly created Ingress resources should prefer using the field. However, even though the annotation is officially deprecated, for backwards compatibility reasons, ingress controlle

rs should still honor that annotation if present.
rules
([]IngressRule)
A list of host rules used to configure the Ingress. If unspecified, or no rule matches, all traffic is sent to the default backend.
IngressRule represents the rules mapping the paths under a specified host to the related backend services. Incoming requests are first evaluated for a host match, then routed to the backend associated with the matching IngressRuleValue.
rules.host
(string)
Host is the fully qualified domain name of a network host, as defined by RFC 3986. Note the following deviations from the "host" part of the URI as defined in RFC 3986: 1. IPs are not allowed. Currently an IngressRuleValue can only apply to
the IP in the Spec of the parent Ingress.
2. The
:
delimiter is not respected because ports are not allowed.
Currently the port of an Ingress is implicitly :80 for http and
:443 for https.
Both these may change in the future. Incoming requests are matched against the host before the IngressRuleValue. If the host is unspecified, the Ingress routes all traffic based on the specified IngressRuleValue.
Host can be "precise" which is a domain name without the terminating dot of a network host (e.g. "foo.bar.com") or "wildcard", which is a domain name prefixed with a single wildcard label (e.g. "
.foo.com"). The wildcard character '
' must appear by itself as the first DNS label and matches only a single label. You cannot have a wildcard label by itself (e.g. Host == "*"). Requests will be matched against the Host field in the following way: 1. If Host is precise, the request matches this rule if the http host header is equal to Host. 2. If Host is a wildcard, then the request matches this rule if the http host header is to equal to the suffix (removing the first label) of the wildcard rule.
rules.http
(HTTPIngressRuleValue)
HTTPIngressRuleValue is a list of http selectors pointing to backends. In the example: http://
/
?
-&gt; backend where where parts of the url correspond to RFC 3986, this resource will be used to match against everything after the last '/' and before the first '?' or '#'.
rules.http.paths
([]HTTPIngressPath), required
A collection of paths that map requests to backends.
HTTPIngressPath associates a path with a backend. Incoming urls matching the path are forwarded to the backend.
rules.http.paths.backend
(
Backend defines the referenced service endpoint to which the traffic will be forwarded to.
rules.http.paths.pathType
PathType determines the interpretation of the Path matching. PathType can be one of the following values: * Exact: Matches the URL path exactly. * Prefix: Matches based on a URL path prefix split by '/'. Matching is
done on a path element by element basis. A path element refers is the
list of labels in the path split by the '/' separator. A request is a
match for path p if every p is an element-wise prefix of p of the
request path. Note that if the last element of the path is a substring
of the last element in request path, it is not a match (e.g. /foo/bar
matches /foo/bar/baz, but does not match /foo/barbaz).
ImplementationSpecific: Interpretation of the Path matching is up to
the IngressClass. Implementations can treat this as a separate PathType
or treat it identically to Prefix or Exact path types.
Implementations are required to support all path types.
rules.http.paths.path
(string)
Path is matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/' and must be present when using PathType with value "Exact" or "Prefix".
tls
([]IngressTLS)
TLS configuration. Currently the Ingress only supports a single TLS port, 443. If multiple members of this list specify different hosts, they will be multiplexed on the same p

ort according to the hostname specified through the SNI TLS extension, if the ingress controller fulfilling the ingress supports SNI.
IngressTLS describes the transport layer security associated with an Ingress.
tls.hosts
Hosts are a list of hosts included in the TLS certificate. The values in this list must match the name/s used in the tlsSecret. Defaults to the wildcard host setting for the loadbalancer controller fulfilling this Ingress, if left unspecified.
tls.secretName
(string)
SecretName is the name of the secret used to terminate TLS traffic on port 443. Field is left optional to allow TLS routing based on SNI hostname alone. If the SNI host in a listener conflicts with the "Host" header field used by an IngressRule, the SNI host is used for termination and value of the Host header is used for routing.
IngressBackend describes all endpoints for a given service and port.
resource
(
)
Resource is an ObjectRef to another Kubernetes resource in the namespace of the Ingress object. If resource is specified, a service.Name and service.Port must not be specified. This is a mutually exclusive setting with "Service".
service
(IngressServiceBackend)
Service references a Service as a Backend. This is a mutually exclusive setting with "Resource".
IngressServiceBackend references a Kubernetes Service as a Backend.
Name is the referenced service. The service must exist in the same namespace as the Ingress object.
(ServiceBackendPort)
Port of the referenced service. A port name or port number is required for a IngressServiceBackend.
ServiceBackendPort is the service port being referenced.
(string)
Name is the name of the port on the Service. This is a mutually exclusive setting with "Number".
(int32)
Number is the numerical port number (e.g. 80) on the Service. This is a mutually exclusive setting with "Name".
IngressStatus describe the current state of the Ingress.
(IngressLoadBalancerStatus)
LoadBalancer contains the current status of the load-balancer.
IngressLoadBalancerStatus represents the status of a load-balancer.
([]IngressLoadBalancerIngress)
Ingress is a list containing ingress points for the load-balancer.
IngressLoadBalancerIngress represents the status of a load-balancer ingress point.
(string)
Hostname is set for load-balancer ingress points that are DNS based.
(string)
IP is set for load-balancer ingress points that are IP based.
([]IngressPortStatus)
Ports provides information about the ports exposed by this LoadBalancer.
IngressPortStatus represents the error condition of a service port
Port is the port number of the ingress port.
Protocol is the protocol of the ingress port. The supported values are: "TCP", "UDP", "SCTP"
(string)
IngressList
IngressList is a collection of Ingress.
items
([]
Ingress
Items is the list of Ingress.
(string)
kind
(string)
Kind is a string value representing the REST resource this object represent

s. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info:
metadata
(
ListMeta
)
get
read the specified Ingress
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses/{name}
name
(
in path
name of the Ingress
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
get
read status of the specified Ingress
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
list
list or watch objects of kind Ingress
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/networking.k8s.io/v1/ingresses
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create an Ingress
POST /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
202 (
Ingress
update
replace the specified Ingress
PUT /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
update
replace status of the specified Ingress
PUT /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
partially update the specified Ingress
PATCH /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses/{name}
name
(
in path
): 

string, required
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
partially update status of the specified Ingress
PATCH /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
delete
delete an Ingress
DELETE /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Ingress
DELETE /apis/networking.k8s.io/v1/namespaces/{namespace}/ingresses
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.2.5 - IngressClass
IngressClass represents the class of the Ingress, referenced by the Ingress Spec.
IngressClass represents the class of the Ingress, referenced by the Ingress Spec. The
annotation can be used to indicate that an IngressClass should be considered default. When a single IngressClass resource has this annotation set to true, new Ingress resources without a class specified will be assigned this default class.
kind
: IngressClass
metadata
(
)
spec
(
IngressClassSpec
)
Spec is the desired state of the IngressClass. More info:
IngressClassSpec provides information about the class of an Ingress.
(string)
Controller refers to the name of the controller that should handle this class. This allows for different "flavors" that are controlled by the same controller. For example, you may have different Parameters for the same implementing controller. This should be specified as a domain-prefixed path no more than 250 characters in length, e.g. "acme.io/ingress-controller". This field is immutable.
(IngressClassParametersReference)
Parameters is a link to a custom resource containing additional configuration for the controller. This is optional if the controller does not require extra parameters.
IngressClassParametersReference identifies an API object. This can be used to specify a cluster or nam

espace-scoped resource.
parameters.kind
Kind is the type of resource being referenced.
parameters.name
parameters.apiGroup
(string)
APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.
parameters.namespace
(string)
Namespace is the namespace of the resource being referenced. This field is required when scope is set to "Namespace" and must be unset when scope is set to "Cluster".
parameters.scope
(string)
Scope represents if this refers to a cluster or namespace scoped resource. This may be set to "Cluster" (default) or "Namespace".
IngressClassList
IngressClassList is a collection of IngressClasses.
kind
: IngressClassList
metadata
(
ListMeta
)
items
([]
Items is the list of IngressClasses.
get
read the specified IngressClass
GET /apis/networking.k8s.io/v1/ingressclasses/{name}
name
(
in path
name of the IngressClass
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind IngressClass
GET /apis/networking.k8s.io/v1/ingressclasses
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create an IngressClass
POST /apis/networking.k8s.io/v1/ingressclasses
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified IngressClass
PUT /apis/networking.k8s.io/v1/ingressclasses/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified IngressClass
PATCH /apis/networking.k8s.io/v1/ingressclasses/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete an IngressClass
DELETE /apis/networking.k8s.io/v1/ingressclasses/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of IngressClass
DELETE /apis/networking.k8s.io/v1/ingressclasses
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string

(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3 - Config and Storage Resources
5.3.1 - ConfigMap
ConfigMap
: v1
kind
metadata
(
)
data
immutable
(boolean)
: v1
kind
metadata
(
ListMeta
)
More info:
items
([]
ConfigMap
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
fieldValidat

ion
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
202 (
ConfigMap
update
name
(
in path
namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.2 - Secret
Secret holds secret data of a certain type.
Secret
Secret holds secret data of a certain type. The total bytes of the values in the Data field must be less than MaxSecretSize bytes.
: v1
kind
: Secret
metadata
(
)
data
Data contains the secret data. Each key must consist of alphanumeric characters, '-', '_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here. Described in
https://tools.ietf.org/html/rfc4648#section-4
immutable
(boolean)
Immutable, if set to true, ensures that data stored in the Secret cannot be updated (only object metadata can be modified). If not set to true, the field can be modified at any time. Defaulted to nil.
stringData allows specifying non-binary secret data in string form. It is provided as a write-only input field for convenience. All keys and values are merged into the data field on write, overwriting any existing values. The stringData field is never output when reading from the API.
type
(string)
Used to facilitate programmatic handling of secret data. More info:
https://kubernetes.io/docs/concepts/configuration/secret/#secret-types
SecretList
SecretList is a list of Secret.
: v1
kind
: SecretList
metadata
(
ListMeta
)

items
([]
Secret
Items is a list of secret objects. More info:
https://kubernetes.io/docs/concepts/configuration/secret
get
read the specified Secret
GET /api/v1/namespaces/{namespace}/secrets/{name}
name
(
in path
name of the Secret
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
list
list or watch objects of kind Secret
GET /api/v1/namespaces/{namespace}/secrets
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/secrets
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a Secret
POST /api/v1/namespaces/{namespace}/secrets
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
202 (
Secret
update
replace the specified Secret
PUT /api/v1/namespaces/{namespace}/secrets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
patch
partially update the specified Secret
PATCH /api/v1/namespaces/{namespace}/secrets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
delete
delete a Secret
DELETE /api/v1/namespaces/{namespace}/secrets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Secret
DELETE /api/v1/namespaces/{namespace}/secrets
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in

 query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.3 - Volume
Volume
name
(boolean)
configMap
(string)
(boolean)
(int32)
([]
KeyToPath
)
secret
(string)
secretNam

e is the name of the secret in the pod's namespace to use. More info:
(boolean)
(int32)
([]
KeyToPath
)
(int32)
([]
)
projected
(int32)
The contents of the target ConfigMap's Data field will be presented in a projected volume as files using the keys in the Data field as the file names, unless the items element is populated with specific mappings of keys to paths. Note that this is identical to a configmap volume source without the default mode.*
(string)
(boolean)
([]
KeyToPath
)
items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. I

f a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
([]
)
(string)
(boolean)
([]
KeyToPath
)
(string)
(int64)
emptyDir
emptyDir represents a temporary directory that shares a pod's lifetime. More info:
(string)
(
Quantity
)
sizeLimit is the total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means tha

t the limit is undefined. More info:
hostPath
(string)
(string)
(int32)
(boolean)
azureDisk
(string)
(string)
fsType is Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified.
(string)
(boolean)
azureFile
shar

eName is the azure share Name
(boolean)
cephfs
(string)
(boolean)
(string)
(
)
(string)
cinder
(string)
(boolean)
(
)
csi
(string)
fsType to mount. Ex. "ext4", "xfs", "ntfs". If not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply.
(
)
(boolean)
ephemeral
ephemeral represents a volume that is 

handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed.
where
not
(
(
)
fc
fc.fsType
(string)
fc.lun
(int32)
(boolean)
fc.wwids
driver is t

he name of the driver to use for this volume.
(string)
(boolean)
(
)
flocker
(string)
(string)
(string)
(int32)
(boolean)
iscsi
iscsi.iqn
iscsi.lun
(boolean)
(boolean

)
(string)
(string)
(string)
(boolean)
(
)
nfs
nfs.path
(boolean)
(string)
(string)
(boolean)
quobyte
(string)
(boolean)
(string)
tenant owning the given Quobyte volume in the Backend Used with 

dynamically provisioned Quobyte volumes, value is set by the plugin
(string)
rbd
rbd.image
(string)
(string)
rbd.pool
(string)
(boolean)
(
)
rbd.user
(string)
scaleIO
(
(string)
(string)
(boolean)
(boolean)
(string)
storageMode indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned.
(string)
(string)
storageos
(string)
(boolean)
(
)
secretRef specifies the secret to use for obtaining the Storag

eOS API credentials. If not specified, default values will be attempted.
(string)
(string)
(string)
(string)
(string)
gitRepo
(string)
(string)
path
fieldRef
(
)
mode
(int32)
Optional: mode bits used to set permissions on this file, must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
(
)
KeyToPath
key
path
mode
(int32)
mode is Optional: mode bits used to set permissions on this file. Mu

st be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
5.3.4 - PersistentVolumeClaim
PersistentVolumeClaim is a user's request for and claim to a persistent volume.
PersistentVolumeClaim is a user's request for and claim to a persistent volume
: v1
kind
: PersistentVolumeClaim
metadata
(
)
spec
(
)
spec defines the desired characteristics of a volume requested by a pod author. More info:
status
(
PersistentVolumeClaimStatus
)
status represents the current information/status of a persistent volume claim. Read-only. More info:
PersistentVolumeClaimSpec describes the common attributes of storage devices and allows a Source for provider-specific attributes
accessModes contains the desired access modes the volume should have. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
selector
(
)
selector is a label query over volumes to consider for binding.
resources
resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure feature is enabled users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources
Quantity
)
Quantity
)
volumeName
(string)
volumeName is the binding reference to the PersistentVolume backing this claim.
(string)
storageClassName is the name of the StorageClass required by the claim. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1
volumeMode
(string)
volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.
Beta level
dataSource
(
)
dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume

 based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.
dataSourceRef
(TypedObjectReference)
dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef: * While dataSource only allows two specific types of objects, dataSourceRef
allows any non-core object, as well as PersistentVolumeClaim objects.
While dataSource ignores disallowed values (dropping them), dataSourceRef
preserves all values, and generates an error if a disallowed value is
specified.
While dataSource only allows local objects, dataSourceRef allows objects
in any namespaces.
(Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled.
**
dataSourceRef.kind
Kind is the type of resource being referenced
dataSourceRef.name
Name is the name of resource being referenced
dataSourceRef.apiGroup
(string)
dataSourceRef.namespace
(string)
Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled.
PersistentVolumeClaimStatus is the current status of a persistent volume claim.
accessModes contains the actual access modes the volume backing the PVC has. More info:
allocatedResources
Quantity
)
allocatedResources is the storage resource within AllocatedResources tracks the capacity allocated to a PVC. It may be larger than the actual capacity when a volume expansion operation is requested. For storage quota, the larger value from allocatedResources and PVC.spec.resources is used. If allocatedResources is not set, PVC.spec.resources alone is used for quota calculation. If a volume expansion capacity request is lowered, allocatedResources is only lowered if there are no expansion operations in progress and if the actual volume capacity is equal or lower than the requested capacity. This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.
capacity
Quantity
)
capacity represents the actual resources of the underlying volume.
([]PersistentVolumeClaimCondition)
type
conditions is the current Condition of persistent volume claim. If underlying persisten

t volume is being resized then the Condition will be set to 'ResizeStarted'.
PersistentVolumeClaimCondition contails details about state of pvc
(Time)
lastProbeTime is the time we probed the condition.
(Time)
lastTransitionTime is the time the condition transitioned from one status to another.
(string)
message is the human-readable message indicating details about last transition.
(string)
reason is a unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
phase
(string)
phase represents the current phase of PersistentVolumeClaim.
resizeStatus
(string)
resizeStatus stores status of resize operation. ResizeStatus is not set by default but when expansion is complete resizeStatus is set to empty string by resize controller or kubelet. This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.
PersistentVolumeClaimList
PersistentVolumeClaimList is a list of PersistentVolumeClaim items.
: v1
kind
: PersistentVolumeClaimList
metadata
(
ListMeta
)
items
([]
items is a list of persistent volume claims. More info:
get
read the specified PersistentVolumeClaim
GET /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
name
(
in path
name of the PersistentVolumeClaim
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified PersistentVolumeClaim
GET /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind PersistentVolumeClaim
GET /api/v1/namespaces/{namespace}/persistentvolumeclaims
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/persistentvolumeclaims
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
wat

ch
Response
200 (
): OK
create
create a PersistentVolumeClaim
POST /api/v1/namespaces/{namespace}/persistentvolumeclaims
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified PersistentVolumeClaim
PUT /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified PersistentVolumeClaim
PUT /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified PersistentVolumeClaim
PATCH /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified PersistentVolumeClaim
PATCH /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a PersistentVolumeClaim
DELETE /api/v1/namespaces/{namespace}/persistentvolumeclaims/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of PersistentVolumeClaim
DELETE /api/v1/namespaces/{namespace}/persistentvolumeclaims
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
prett

y
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.5 - PersistentVolume
PersistentVolume (PV) is a storage resource provisioned by an administrator.
PersistentVolume (PV) is a storage resource provisioned by an administrator. It is analogous to a node. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes
: v1
kind
: PersistentVolume
metadata
(
)
spec
(
)
spec defines a specification of a persistent volume owned by the cluster. Provisioned by an administrator. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
status
(
PersistentVolumeStatus
)
status represents the current information/status for the persistent volume. Populated by the system. Read-only. More info:
PersistentVolumeSpec is the specification of a persistent volume.
accessModes contains all ways the volume can be mounted. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes
capacity
Quantity
)
capacity is the description of the persistent volume's resources and capacity. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity
claimRef
(
)
claimRef is part of a bi-directional binding between PersistentVolume and PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName is the authoritative bind between PV and PVC. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding
mountOptions
mountOptions is the list of mount options, e.g. ["ro", "soft"]. Not validated - mount will simply fail if one is invalid. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options
(VolumeNodeAffinity)
nodeAffinity defines constraints that limit what nodes this volume can be accessed from. This field influences the scheduling of pods that use this volume.
VolumeNodeAffinity defines constraints that limit what nodes this volume can be accessed from.
nodeAffinity.required
required specifies hard node constraints that must be met.
nodeAffinity.required.nodeSelectorTerms
nodeAffinity.required.nodeSelectorTerms.matchExpressions
([]
)
nodeAffinity.required.nodeSelectorTerms.matchFields
([]
)
persistentVolumeReclaimPolicy
(string)
persistentVolumeReclaimPolicy defines what happens to a persistent volume when released from its claim. Valid options are Retain (default for manually created PersistentVolumes), Delete (default for dynamically provisioned PersistentVolumes), and Recycle (deprecated). Recycle must be supported by the volume plugin underlying this PersistentVolume. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#reclaiming
(string)
storageClassName is the name of StorageClass to which thi

s persistent volume belongs. Empty value means that this volume does not belong to any StorageClass.
(string)
volumeMode defines if a volume is intended to be used with a formatted filesystem or to remain in raw block state. Value of Filesystem is implied when not included in spec.
Local
hostPath
hostPath represents a directory on the host. Provisioned by a developer or tester. This is useful for single-node development and testing only! On-host storage is not supported in any way and WILL NOT WORK in a multi-node cluster. More info:
(string)
local
(LocalVolumeSource)
local represents directly-attached storage with node affinity
Local represents directly-attached storage with node affinity (Beta feature)
local.path
path of the full path to the volume on the node. It can be either a directory or block device (disk, partition, ...).
local.fsType
(string)
fsType is the filesystem type to mount. It applies only when the Path is a block device. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default value is to auto-select a filesystem if unspecified.
(string)
(int32)
(boolean)
azureDisk
(string)
(string)
fsType is Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext

4" if unspecified.
(string)
(boolean)
azureFile
(AzureFilePersistentVolumeSource)
shareName is the azure Share Name
(boolean)
azureFile.secretNamespace
(string)
secretNamespace is the namespace of the secret that contains Azure Storage Account Name and Key default is the same as the Pod
cephfs
(CephFSPersistentVolumeSource)
(string)
(boolean)
(string)
(SecretReference)
SecretReference represents a Secret Reference. It has enough information to retrieve secret in any namespace
cephfs.secretRef.name
(string)
name is unique within a namespace to reference a secret resource.
cephfs.secretRef.namespace
(string)
namespace defines the space within which the secret name must be unique.
(string)
user is Optional: User is the rados user name, default is admin More info:
cinder
(CinderPersistentVolumeSource)
(string)
fsType Filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info:
(boolean)
secretRef is Optional: points to a secret object containing parameters used to connect to OpenStack.
SecretReference represents a Secret Reference. It has enough information to retrieve secret in

 any namespace
cinder.secretRef.name
(string)
cinder.secretRef.namespace
(string)
csi
(CSIPersistentVolumeSource)
csi represents storage that is handled by an external CSI driver (Beta feature).
Represents storage that is managed by an external CSI volume driver (Beta feature)
driver is the name of the driver to use for this volume. Required.
csi.volumeHandle
volumeHandle is the unique volume name returned by the CSI volume plugin’s CreateVolume to refer to the volume on all subsequent calls. Required.
csi.controllerExpandSecretRef
controllerExpandSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerExpandVolume call. This is an beta field and requires enabling ExpandCSIVolumes feature gate. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
csi.controllerExpandSecretRef.name
(string)
csi.controllerExpandSecretRef.namespace
(string)
csi.controllerPublishSecretRef
controllerPublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerPublishVolume and ControllerUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
csi.controllerPublishSecretRef.name
(string)
csi.controllerPublishSecretRef.namespace
(string)
(string)
fsType to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs".
csi.nodeExpandSecretRef
nodeExpandSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeExpandVolume call. This is an alpha field and requires enabling CSINodeExpandSecret feature gate. This field is optional, may be omitted if no secret is required. If the secret object contains more than one secret, all secrets are passed.
csi.nodeExpandSecretRef.name
(string)
csi.nodeExpandSecretRef.namespace
(string)
nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
csi.nodePublishSecretRef.name
(string)
csi.nodePublishSecretRef.namespace
(string)
csi.nodeStageSecretRef
nodeStageSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeStageVolume and NodeStageVolu

me and NodeUnstageVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
csi.nodeStageSecretRef.name
(string)
csi.nodeStageSecretRef.namespace
(string)
(boolean)
readOnly value to pass to ControllerPublishVolumeRequest. Defaults to false (read/write).
volumeAttributes of the volume to publish.
fc
fc.fsType
(string)
fc.lun
(int32)
(boolean)
fc.wwids
(FlexPersistentVolumeSource)
FlexPersistentVolumeSource represents a generic persistent volume resource that is provisioned/attached using an exec based plugin.
(string)
fsType is the Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default filesystem depends on FlexVolume script.
(boolean)
secretRef is Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts.
flexVolume.secretRef.name
(string)
flexVolume.secretRef.namespace
(string)
flocker
flocker represents a Flocker volume attached to a kubelet's host machine and exposed to the pod for its usage. This depends on the Flocker control service being running
(string)
(string)
gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin. More i

nfo:
(string)
(int32)
(boolean)
iscsi
(ISCSIPersistentVolumeSource)
iscsi represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin.
ISCSIPersistentVolumeSource represents an ISCSI disk. ISCSI volumes can only be mounted as read/write once. ISCSI volumes support ownership management and SELinux relabeling.
iscsi.iqn
iqn is Target iSCSI Qualified Name.
iscsi.lun
lun is iSCSI Target Lun number.
(boolean)
(boolean)
(string)
(string)
(string)
portals is the iSCSI Target Portal List. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).
(boolean)
iscsi.secretRef.name
(string)
iscsi.secretRef.namespace
(string)
nfs
nfs represents an NFS mount on the host. Provisioned by an admin. More info:
Represents an NFS mount that

 lasts the lifetime of a pod. NFS volumes do not support ownership management or SELinux relabeling.
nfs.path
(boolean)
(string)
(string)
(boolean)
quobyte
(string)
(boolean)
(string)
(string)
rbd
(RBDPersistentVolumeSource)
rbd.image
(string)
(string)
rbd.pool
(string)
https://examples.k8s.io/volumes/rbd/README.md#how-to-use

-it
(boolean)
rbd.secretRef.name
(string)
rbd.secretRef.namespace
(string)
rbd.user
(string)
scaleIO
(ScaleIOPersistentVolumeSource)
ScaleIOPersistentVolumeSource represents a persistent ScaleIO volume
(SecretReference), required
scaleIO.secretRef.name
(string)
scaleIO.secretRef.namespace
(string)
(string)
fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Default is "xfs"
(string)
(boolean)
(boolean)
sslEnabled is the flag to enable/disable SSL communication with Gateway, default false
(string)
(string)
(string)
storageos
(StorageOSPersistentVolumeSource)
storageOS represents a StorageOS volume that is attached to the kubelet's host machine and mounted into the pod More info:
https://examples.k8s.io/volumes/storageos/README.md
(string)
(boolean)
(
)
(string)
(string)
volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within Stor

ageOS. Namespaces that do not pre-exist within StorageOS will be created.
(string)
(string)
(string)
PersistentVolumeStatus is the current status of a persistent volume.
message
(string)
message is a human-readable message indicating details about why the volume is in this state.
phase
(string)
phase indicates if a volume is available, bound to a claim, or released by a claim. More info:
https://kubernetes.io/docs/concepts/storage/persistent-volumes#phase
reason
(string)
reason is a brief CamelCase string that describes any failure and is meant for machine parsing and tidy display in the CLI.
PersistentVolumeList
PersistentVolumeList is a list of PersistentVolume items.
: v1
kind
: PersistentVolumeList
metadata
(
ListMeta
)
items
([]
items is a list of persistent volumes. More info:
get
read the specified PersistentVolume
GET /api/v1/persistentvolumes/{name}
name
(
in path
name of the PersistentVolume
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified PersistentVolume
GET /api/v1/persistentvolumes/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind PersistentVolume
GET /api/v1/persistentvolumes
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a PersistentVolume
POST /api/v1/persistentvolumes
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified PersistentVolume
PUT /api/v1/persistentvolumes/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified PersistentVolume
PUT /api/v1/persistentvolumes/{name}/status
name
(
in path
nam

e of the PersistentVolume
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified PersistentVolume
PATCH /api/v1/persistentvolumes/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified PersistentVolume
PATCH /api/v1/persistentvolumes/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a PersistentVolume
DELETE /api/v1/persistentvolumes/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of PersistentVolume
DELETE /api/v1/persistentvolumes
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.6 - StorageClass
StorageClass describes the parameters for a class of storage for which PersistentVolumes can be dynamically provisioned.
StorageClasses are non-namespaced; the name of the storage class according to etcd is in ObjectMeta.Name.
kind
: StorageClass
metadata
(
)
Provisioner indicates the type of the provisioner.
(boolean)
AllowVolumeExpansion shows whether the storage class allow volume expand
allowedTopologies
([]TopologySelectorTerm)
Restrict the node topologies where volumes can be dynamically provisioned. Each volume plugin defines its own supported topology specifications. An empty TopologySelectorTerm list means there is no topology restriction. This field is only honored by servers that enable the VolumeScheduling feature.
A topology selector term represents the result of label queries. A null or empty topology selector term matches no objects. The requirements of them are ANDed. It provides a subset of functionality as NodeSelectorTerm. This is an alpha feature and may change in the future.
allowe

dTopologies.matchLabelExpressions
([]TopologySelectorLabelRequirement)
A list of topology selector requirements by labels.
A topology selector requirement is a selector that matches given label. This is an alpha feature and may change in the future.
allowedTopologies.matchLabelExpressions.key
The label key that the selector applies to.
allowedTopologies.matchLabelExpressions.values
An array of string values. One value must match the label to be selected. Each entry in Values is ORed.
Dynamically provisioned PersistentVolumes of this storage class are created with these mountOptions, e.g. ["ro", "soft"]. Not validated - mount of the PVs will simply fail if one is invalid.
Parameters holds the parameters for the provisioner that should create volumes of this storage class.
reclaimPolicy
(string)
Dynamically provisioned PersistentVolumes of this storage class are created with this reclaimPolicy. Defaults to Delete.
volumeBindingMode
(string)
VolumeBindingMode indicates how PersistentVolumeClaims should be provisioned and bound. When unset, VolumeBindingImmediate is used. This field is only honored by servers that enable the VolumeScheduling feature.
StorageClassList
StorageClassList is a collection of storage classes.
kind
: StorageClassList
metadata
(
ListMeta
)
items
([]
Items is the list of StorageClasses
get
read the specified StorageClass
GET /apis/storage.k8s.io/v1/storageclasses/{name}
name
(
in path
name of the StorageClass
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind StorageClass
GET /apis/storage.k8s.io/v1/storageclasses
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a StorageClass
POST /apis/storage.k8s.io/v1/storageclasses
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified StorageClass
PUT /apis/storage.k8s.io/v1/storageclasses/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified StorageClass
PATCH /apis/storage.k8s.io/v1/storageclasses/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a StorageClass
DELETE /apis/storage.k8s.io/v1/storageclasses/{name}
name
(
in path

body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of StorageClass
DELETE /apis/storage.k8s.io/v1/storageclasses
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.7 - VolumeAttachment
VolumeAttachment captures the intent to attach or detach the specified volume to/from the specified node.
VolumeAttachment objects are non-namespaced.
kind
: VolumeAttachment
metadata
(
)
spec
(
VolumeAttachmentSpec
Specification of the desired attach/detach volume behavior. Populated by the Kubernetes system.
status
(
VolumeAttachmentStatus
)
Status of the VolumeAttachment request. Populated by the entity completing the attach or detach operation, i.e. the external-attacher.
VolumeAttachmentSpec is the specification of a VolumeAttachment request.
attacher
Attacher indicates the name of the volume driver that MUST handle this request. This is the name returned by GetPluginName().
nodeName
The node that the volume should be attached to.
source
(VolumeAttachmentSource), required
Source represents the volume that should be attached.
VolumeAttachmentSource represents a volume that should be attached. Right now only PersistenVolumes can be attached via external attacher, in future we may allow also inline volumes in pods. Exactly one member can be set.
source.inlineVolumeSpec
(
)
inlineVolumeSpec contains all the information necessary to attach a persistent volume defined by a pod's inline VolumeSource. This field is populated only for the CSIMigration feature. It contains translated fields from a pod's inline VolumeSource to a PersistentVolumeSpec. This field is beta-level and is only honored by servers that enabled the CSIMigration feature.
source.persistentVolumeName
(string)
Name of the persistent volume to attach.
VolumeAttachmentStatus is the status of a VolumeAttachment request.
attached
Indicates the volume is successfully attached. This field must only be set by the entity completing the attach operation, i.e. the external-attacher.
attachError
(VolumeError)
The last error encountered during attach operation, if any. This field must only be set by the entity completing the attach operation, i.e. the external-attacher.
VolumeError captures an error encountered during a volume operation.
attachError.message
(string)
String detailing the error encountered during Attach or Detach operation. This string may be logged, so it should not contain sensitive information.
attachError.time
(Time)
Time the error was encountered.
att

achmentMetadata
Upon successful attach, this field is populated with any information returned by the attach operation that must be passed into subsequent WaitForAttach or Mount calls. This field must only be set by the entity completing the attach operation, i.e. the external-attacher.
detachError
The last error encountered during detach operation, if any. This field must only be set by the entity completing the detach operation, i.e. the external-attacher.
detachError.message
(string)
detachError.time
(Time)
VolumeAttachmentList
VolumeAttachmentList is a collection of VolumeAttachment objects.
kind
: VolumeAttachmentList
metadata
(
ListMeta
)
items
([]
Items is the list of VolumeAttachments
get
read the specified VolumeAttachment
GET /apis/storage.k8s.io/v1/volumeattachments/{name}
name
(
in path
name of the VolumeAttachment
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified VolumeAttachment
GET /apis/storage.k8s.io/v1/volumeattachments/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind VolumeAttachment
GET /apis/storage.k8s.io/v1/volumeattachments
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a VolumeAttachment
POST /apis/storage.k8s.io/v1/volumeattachments
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified VolumeAttachment
PUT /apis/storage.k8s.io/v1/volumeattachments/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified VolumeAttachment
PUT /apis/storage.k8s.io/v1/volumeattachments/{name}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially up

date the specified VolumeAttachment
PATCH /apis/storage.k8s.io/v1/volumeattachments/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified VolumeAttachment
PATCH /apis/storage.k8s.io/v1/volumeattachments/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a VolumeAttachment
DELETE /apis/storage.k8s.io/v1/volumeattachments/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of VolumeAttachment
DELETE /apis/storage.k8s.io/v1/volumeattachments
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.8 - CSIDriver
CSIDriver captures information about a Container Storage Interface (CSI) volume driver deployed on the cluster.
CSIDriver
CSIDriver captures information about a Container Storage Interface (CSI) volume driver deployed on the cluster. Kubernetes attach detach controller uses this object to determine whether attach is required. Kubelet uses this object to determine whether pod information needs to be passed on mount. CSIDriver objects are non-namespaced.
kind
: CSIDriver
metadata
(
)
Standard object metadata. metadata.Name indicates the name of the CSI driver that this object refers to; it MUST be the same name returned by the CSI GetPluginName() call for that driver. The driver name must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), dots (.), and alphanumerics between. More info:
spec
(
CSIDriverSpec
Specification of the CSI Driver.
CSIDriverSpec is the specification of a CSIDriver.
attachRequired
(boolean)
attachRequired indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume() method), and that the Kubernetes attach detach controller should call the attach volume interface which checks the volumeattachment status and waits until the volume is attached before proceeding to mounting. The CSI external-attacher coordinates with CSI volume driver and updates the volumeattachment status when the attach operation is complete. If the CSIDriverRegistry feature gate is enabled and the value is sp

ecified to false, the attach operation will be skipped. Otherwise the attach operation will be called.
fsGroupPolicy
(string)
Defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupPolicy values for additional details.
Defaults to ReadWriteOnceWithFSType, which will examine each volume to determine if Kubernetes should modify ownership and permissions of the volume. With the default policy the defined fsGroup will only be applied if a fstype is defined and the volume's access mode contains ReadWriteOnce.
podInfoOnMount
(boolean)
If set to true, podInfoOnMount indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during mount operations. If set to false, pod information will not be passed on mount. Default is false. The CSI driver specifies podInfoOnMount as part of driver deployment. If true, Kubelet will pass pod information as VolumeContext in the CSI NodePublishVolume() calls. The CSI driver is responsible for parsing and validating the information passed in as VolumeContext. The following VolumeConext will be passed if podInfoOnMount is set to true. This list might grow, but the prefix will be used. "csi.storage.k8s.io/pod.name": pod.Name "csi.storage.k8s.io/pod.namespace": pod.Namespace "csi.storage.k8s.io/pod.uid": string(pod.UID) "csi.storage.k8s.io/ephemeral": "true" if the volume is an ephemeral inline volume
defined by a CSIVolumeSource, otherwise "false"
"csi.storage.k8s.io/ephemeral" is a new feature in Kubernetes 1.16. It is only required for drivers which support both the "Persistent" and "Ephemeral" VolumeLifecycleMode. Other drivers can leave pod info disabled and/or ignore this field. As Kubernetes 1.15 doesn't support this field, drivers can only support one mode when deployed on such a cluster and the deployment determines which mode that is, for example via a command line parameter of the driver.
requiresRepublish
(boolean)
RequiresRepublish indicates the CSI driver wants
NodePublishVolume
being periodically called to reflect any possible change in the mounted volume. This field defaults to false.
Note: After a successful initial NodePublishVolume call, subsequent calls to NodePublishVolume should only update the contents of the volume. New mount points will not be seen by a running container.
seLinuxMount
(boolean)
SELinuxMount specifies if the CSI driver supports "-o context" mount option.
When "true", the CSI driver must ensure that all volumes provided by this CSI driver can be mounted separately with different
-o context
options. This is typical for storage backends that provide volumes as filesystems on block devices or as independent shared volumes. Kubernetes will call NodeStage / NodePublish with "-o context=xyz" mount option when mounting a ReadWriteOncePod volume used in Pod that has explicitly set SELinux context. In the future, it may be expanded to other volume AccessModes. In any case, Kubernetes will ensure that the volume is mounted only with a single SELinux context.
When "false", Kubernetes won't pass any special SELinux mount options to the driver. This is typical for volumes that represent subdirectories of a bigger shared filesystem.
Default is "false".
storageCapacity
(boolean)
If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.
The check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.
Alternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.
This field was immutab

le in Kubernetes &lt;= 1.22 and now is mutable.
tokenRequests
([]TokenRequest)
TokenRequests indicates the CSI driver needs pods' service account tokens it is mounting volume for to do necessary authentication. Kubelet will pass the tokens in VolumeContext in the CSI NodePublishVolume calls. The CSI driver should parse and validate the following VolumeContext: "csi.storage.k8s.io/serviceAccount.tokens": {
"&lt;audience&gt;": {
"token": &lt;token&gt;,
"expirationTimestamp": &lt;expiration timestamp in RFC3339&gt;,
},
...
}
Note: Audience in each TokenRequest should be different and at most one token is empty string. To receive a new token after expiry, RequiresRepublish can be used to trigger NodePublishVolume periodically.
TokenRequest contains parameters of a service account token.
tokenRequests.audience
Audience is the intended audience of the token in "TokenRequestSpec". It will default to the audiences of kube apiserver.
tokenRequests.expirationSeconds
(int64)
ExpirationSeconds is the duration of validity of the token in "TokenRequestSpec". It has the same default value of "ExpirationSeconds" in "TokenRequestSpec".
volumeLifecycleModes
volumeLifecycleModes defines what kind of volumes this CSI volume driver supports. The default if the list is empty is "Persistent", which is the usage defined by the CSI specification and implemented in Kubernetes via the usual PV/PVC mechanism. The other mode is "Ephemeral". In this mode, volumes are defined inline inside the pod spec with CSIVolumeSource and their lifecycle is tied to the lifecycle of that pod. A driver has to be aware of this because it is only going to get a NodePublishVolume call for such a volume. For more information about implementing this mode, see
https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html
A driver can support one or more of these modes and more modes may be added in the future. This field is beta.
CSIDriverList
CSIDriverList is a collection of CSIDriver objects.
kind
: CSIDriverList
metadata
(
ListMeta
)
items
([]
CSIDriver
items is the list of CSIDriver
get
read the specified CSIDriver
GET /apis/storage.k8s.io/v1/csidrivers/{name}
name
(
in path
name of the CSIDriver
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
list
list or watch objects of kind CSIDriver
GET /apis/storage.k8s.io/v1/csidrivers
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a CSIDriver
POST /apis/storage.k8s.io/v1/csidrivers
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
202 (
CSIDriver
update
replace the specified CSIDriver
PUT /apis/storage.k8s.io/v1/csidrivers/{name}
name
(
in path
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pre

tty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
patch
partially update the specified CSIDriver
PATCH /apis/storage.k8s.io/v1/csidrivers/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
delete
delete a CSIDriver
DELETE /apis/storage.k8s.io/v1/csidrivers/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
CSIDriver
): OK
202 (
CSIDriver
delete collection of CSIDriver
DELETE /apis/storage.k8s.io/v1/csidrivers
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.9 - CSINode
CSINode
kind
: CSINode
metadata
(
)
spec
(
drivers
name
VolumeNodeResour

ces is a set of resource limits for scheduling of volumes.
(int32)
Maximum number of unique volumes managed by the CSI driver that can be used on a node. A volume that is both attached and mounted on a node is considered to be used once, not twice. The same rule applies for a unique volume that is shared among multiple pods on the same node. If this field is not specified, then the supported number of volumes on this node is unbounded.
kind
metadata
(
ListMeta
)
items
([]
CSINode
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
202 (
CSINode
update
name
(
in path
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (

CSINode
): OK
202 (
CSINode
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.3.10 - CSIStorageCapacity
kind
metadata
(
)
More info:
capacity
(
Quantity
)
The semantic is currently (CSI spec 1.2) defined as: The available capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable.
(
Quantity
)
(
)
No

deTopology defines which nodes have access to the storage for which capacity was reported. If not set, the storage is not accessible from any node in the cluster. If empty, the storage is accessible from all nodes. This field is immutable.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean

force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.4 - Authentication Resources
5.4.1 - ServiceAccount
ServiceAccount binds together: * a name, understood by users, and perhaps by peripheral systems, for an identity * a principal that can be authenticated and authorized * a set of secrets.
ServiceAccount binds together: * a name, understood by users, and perhaps by peripheral systems, for an identity * a principal that can be authenticated and authorized * a set of secrets
: v1
kind
: ServiceAccount
metadata
(
)
(boolean)
AutomountServiceAccountToken indicates whether pods running as this service account should have an API token automatically mounted. Can be overridden at the pod level.
([]
)
ImagePullSecrets is a list of references to secrets in the same namespace to use for pulling any images in pods that reference this ServiceAccount. ImagePullSecrets are distinct from Secrets because Secrets can be mounted in the pod, but ImagePullSecrets are only accessed by the kubelet. More info:
https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
secrets
([]
)
name
Secrets is a list of the secrets in the same namespace that pods running using this ServiceAccount are allowed to use. Pods are only limited to this list if this service account has a "kubernetes.io/enforce-mountable-secrets" annotation set to "true". This field should not be used to find auto-generated service account token secrets for use outside of pods. Instead, tokens can be requested directly using the TokenRequest API, or service account token secrets can be manually created. More info:
ServiceAccountList
ServiceAccountList is a list of ServiceAccount objects
: v1
kind
: ServiceAccountList
metadata
(
ListMeta
)
items
([]
List of ServiceAccounts. More info:
get
read the specified ServiceAccount
GET /api/v1/namespaces/{namespace}/serviceaccounts/{name}
name
(
in path
): string, req

uired
name of the ServiceAccount
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ServiceAccount
GET /api/v1/namespaces/{namespace}/serviceaccounts
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/serviceaccounts
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ServiceAccount
POST /api/v1/namespaces/{namespace}/serviceaccounts
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ServiceAccount
PUT /api/v1/namespaces/{namespace}/serviceaccounts/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ServiceAccount
PATCH /api/v1/namespaces/{namespace}/serviceaccounts/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ServiceAccount
DELETE /api/v1/namespaces/{namespace}/serviceaccounts/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of ServiceAccount
DELETE /api/v1/namespaces/{namespace}/serviceaccounts
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in 

query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.4.2 - TokenRequest
TokenRequest requests a token for a given service account.
apiVersion: authentication.k8s.io/v1
import "k8s.io/api/authentication/v1"
: authentication.k8s.io/v1
kind
: TokenRequest
metadata
(
)
spec
(
TokenRequestSpec
Spec holds information about the request being evaluated
status
(
TokenRequestStatus
)
Status is filled in by the server and indicates whether the token can be authenticated.
TokenRequestSpec contains client provided parameters of a token request.
audiences
Audiences are the intendend audiences of the token. A recipient of a token must identify themself with an identifier in the list of audiences of the token, and otherwise should reject the token. A token issued for multiple audiences may be used to authenticate against any of the audiences listed but implies a high degree of trust between the target audiences.
boundObjectRef
(BoundObjectReference)
BoundObjectRef is a reference to an object that the token will be bound to. The token will only be valid for as long as the bound object exists. NOTE: The API server's TokenReview endpoint will validate the BoundObjectRef, but other audiences may not. Keep ExpirationSeconds small if you want prompt revocation.
BoundObjectReference is a reference to an object that a token is bound to.
boundObjectRef.apiVersion
(string)
API version of the referent.
boundObjectRef.kind
(string)
Kind of the referent. Valid kinds are 'Pod' and 'Secret'.
boundObjectRef.name
(string)
Name of the referent.
boundObjectRef.uid
(string)
UID of the referent.
(int64)
ExpirationSeconds is the requested duration of validity of the request. The token issuer may return a token with a different validity duration so a client needs to check the 'expiration' field in a response.
TokenRequestStatus is the result of a token request.
expirationTimestamp
ExpirationTimestamp is the time of expiration of the returned token.
token
Token is the opaque bearer token.
create
create token of a ServiceAccount
POST /api/v1/namespaces/{namespace}/serviceaccounts/{name}/token
name
(
in path
name of the TokenRequest
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.4.3 - TokenReview
TokenReview attempts to authenticate a token to a known user.
TokenReview attempts to authenticate a token to a known user. Note: TokenReview requests may be cached by the webhook token authenticator plugin in the kube-apiserver.
kind
: TokenReview
metadata
(
)
spec
(
TokenReviewSpec
Spec holds information about

 the request being evaluated
status
(
TokenReviewStatus
)
Status is filled in by the server and indicates whether the request can be authenticated.
TokenReviewSpec is a description of the token authentication request.
audiences
Audiences is a list of the identifiers that the resource server presented with the token identifies as. Audience-aware token authenticators will verify that the token was intended for at least one of the audiences in this list. If no audiences are provided, the audience will default to the audience of the Kubernetes apiserver.
token
(string)
TokenReviewStatus is the result of the token authentication request.
audiences
Audiences are audience identifiers chosen by the authenticator that are compatible with both the TokenReview and token. An identifier is any identifier in the intersection of the TokenReviewSpec audiences and the token's audiences. A client of the TokenReview API that sets the spec.audiences field should validate that a compatible audience identifier is returned in the status.audiences field to ensure that the TokenReview server is audience aware. If a TokenReview returns an empty status.audience field where status.authenticated is "true", the token is valid against the audience of the Kubernetes API server.
(boolean)
Authenticated indicates that the token was associated with a known user.
error
(string)
Error indicates that the token couldn't be checked
user
(UserInfo)
User is the UserInfo associated with the provided token.
UserInfo holds the information about the user needed to implement the user.Info interface.
user.extra
(map[string][]string)
Any additional information provided by the authenticator.
user.groups
The names of groups this user is a part of.
user.uid
(string)
A unique value that identifies this user across time. If this user is deleted and another user by the same name is added, they will have different UIDs.
user.username
(string)
The name that uniquely identifies this user among all active users.
create
create a TokenReview
POST /apis/authentication.k8s.io/v1/tokenreviews
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.4.4 - CertificateSigningRequest
CertificateSigningRequest objects provide a mechanism to obtain x509 certificates by submitting a certificate signing request, and having it asynchronously approved and issued.
import "k8s.io/api/certificates/v1"
Kubelets use this API to obtain:
client certificates to authenticate to kube-apiserver (with the "kubernetes.io/kube-apiserver-client-kubelet" signerName).
serving certificates for TLS endpoints kube-apiserver can connect to securely (with the "kubernetes.io/kubelet-serving" signerName).
This API can be used to request client certificates to authenticate to kube-apiserver (with the "kubernetes.io/kube-apiserver-client" signerName), or to obtain certificates from custom non-Kubernetes signers.
: certificates.k8s.io/v1
kind
: CertificateSigningRequest
metadata
(
)
spec
(
CertificateSigningRequestSpec
spec contains the certificate request, and is immutable after creation. Only the request, signerName, expirationSeconds, and usages fields can be set on creation. Other fields are derived by Kubernetes and cannot be modified by users.
status
(
CertificateSigningRequestStatus
)
status contains information about whether the request is approved or de

nied, and the certificate issued by the signer, or the failure condition indicating signer failure.
CertificateSigningRequestSpec contains the certificate request.
request
([]byte), required
request contains an x509 certificate signing request encoded in a "CERTIFICATE REQUEST" PEM block. When serialized as JSON or YAML, the data is additionally base64-encoded.
signerName indicates the requested signer, and is a qualified name.
List/watch requests for CertificateSigningRequests can filter on this field using a "spec.signerName=NAME" fieldSelector.
Well-known Kubernetes signers are:
"kubernetes.io/kube-apiserver-client": issues client certificates that can be used to authenticate to kube-apiserver.
Requests for this signer are never auto-approved by kube-controller-manager, can be issued by the "csrsigning" controller in kube-controller-manager.
"kubernetes.io/kube-apiserver-client-kubelet": issues client certificates that kubelets use to authenticate to kube-apiserver.
Requests for this signer can be auto-approved by the "csrapproving" controller in kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager.
"kubernetes.io/kubelet-serving" issues serving certificates that kubelets use to serve TLS endpoints, which kube-apiserver can connect to securely.
Requests for this signer are never auto-approved by kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager.
More details are available at
https://k8s.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers
Custom signerNames can also be specified. The signer defines:
Trust distribution: how trust (CA bundles) are distributed.
Permitted subjects: and behavior when a disallowed subject is requested.
Required, permitted, or forbidden x509 extensions in the request (including whether subjectAltNames are allowed, which types, restrictions on allowed values) and behavior when a disallowed extension is requested.
Required, permitted, or forbidden key usages / extended key usages.
Expiration/certificate lifetime: whether it is fixed by the signer, configurable by the admin.
Whether or not requests for CA certificates are allowed.
(int32)
expirationSeconds is the requested duration of validity of the issued certificate. The certificate signer may issue a certificate with a different validity duration so a client must check the delta between the notBefore and and notAfter fields in the issued certificate to determine the actual duration.
The v1.22+ in-tree implementations of the well-known Kubernetes signers will honor this field as long as the requested duration is not greater than the maximum duration they will honor per the --cluster-signing-duration CLI flag to the Kubernetes controller manager.
Certificate signers may not honor this field for various reasons:
Old signer that is unaware of the field (such as the in-tree
implementations prior to v1.22)
Signer whose configured maximum is shorter than the requested duration
Signer whose configured minimum is longer than the requested duration
The minimum valid value for expirationSeconds is 600, i.e. 10 minutes.
extra
extra contains extra attributes of the user that created the CertificateSigningRequest. Populated by the API server on creation and immutable.
groups
groups contains group membership of the user that created the CertificateSigningRequest. Populated by the API server on creation and immutable.
uid
(string)
uid contains the uid of the user that created the CertificateSigningRequest. Populated by the API server on creation and immutable.
usages
usages specifies a set of key usages requested in the issued certificate.
Requests for TLS client certificates typically request: 

"digital signature", "key encipherment", "client auth".
Requests for TLS serving certificates typically request: "key encipherment", "digital signature", "server auth".
Valid values are:
"signing", "digital signature", "content commitment",
"key encipherment", "key agreement", "data encipherment",
"cert sign", "crl sign", "encipher only", "decipher only", "any",
"server auth", "client auth",
"code signing", "email protection", "s/mime",
"ipsec end system", "ipsec tunnel", "ipsec user",
"timestamping", "ocsp signing", "microsoft sgc", "netscape sgc"
username
(string)
username contains the name of the user that created the CertificateSigningRequest. Populated by the API server on creation and immutable.
CertificateSigningRequestStatus contains conditions used to indicate approved/denied/failed status of the request, and the issued certificate.
([]byte)
certificate is populated with an issued certificate by the signer after an Approved condition is present. This field is set via the /status subresource. Once populated, this field is immutable.
If the certificate signing request is denied, a condition of type "Denied" is added and this field remains empty. If the signer cannot issue the certificate, a condition of type "Failed" is added and this field remains empty.
Validation requirements:
certificate must contain one or more PEM blocks.
All PEM blocks must have the "CERTIFICATE" label, contain no headers, and the encoded data
must be a BER-encoded ASN.1 Certificate structure as described in section 4 of RFC5280.
Non-PEM content may appear before or after the "CERTIFICATE" PEM blocks and is unvalidated,
to allow for explanatory text as described in section 5.2 of RFC7468.
If more than one PEM block is present, and the definition of the requested spec.signerName does not indicate otherwise, the first block is the issued certificate, and subsequent blocks should be treated as intermediate certificates and presented in TLS handshakes.
The certificate is encoded in PEM format.
When serialized as JSON or YAML, the data is additionally base64-encoded, so it consists of:
base64(
...
)
([]CertificateSigningRequestCondition)
conditions applied to the request. Known conditions are "Approved", "Denied", and "Failed".
CertificateSigningRequestCondition describes a condition of a CertificateSigningRequest object
status of the condition, one of True, False, Unknown. Approved, Denied, and Failed conditions may not be "False" or "Unknown".
type of the condition. Known conditions are "Approved", "Denied", and "Failed".
An "Approved" condition is added via the /approval subresource, indicating the request was approved and should be issued by the signer.
A "Denied" condition is added via the /approval subresource, indicating the request was denied and should not be issued by the signer.
A "Failed" condition is added via the /status subresource, indicating the signer failed to issue the certificate.
Approved and Denied conditions are mutually exclusive. Approved, Denied, and Failed conditions cannot be removed once added.
Only one condition of a given type is allowed.
(Time)
lastTransitionTime is the time the condition last transitioned from one status to another. If unset, when a new condition type is added or an existing condition's status is changed, the server defaults this to the current time.
(Time)
lastUpdateTime is the time of the last update to this condition
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON

. Wrappers are provided for many of the factory methods that the time package offers.
(string)
message contains a human readable message with details about the request state
(string)
reason indicates a brief reason for the request state
CertificateSigningRequestList
CertificateSigningRequestList is a collection of CertificateSigningRequest objects
kind
: CertificateSigningRequestList
metadata
(
ListMeta
)
items
([]
items is a collection of CertificateSigningRequest objects
get
read the specified CertificateSigningRequest
GET /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}
name
(
in path
name of the CertificateSigningRequest
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read approval of the specified CertificateSigningRequest
GET /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}/approval
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified CertificateSigningRequest
GET /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind CertificateSigningRequest
GET /apis/certificates.k8s.io/v1/certificatesigningrequests
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a CertificateSigningRequest
POST /apis/certificates.k8s.io/v1/certificatesigningrequests
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified CertificateSigningRequest
PUT /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace approval of the specified CertificateSigningRequest
PUT /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}/approval
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified CertificateSigningRequest
PUT /apis/certificates.k8s.io/v1/certificatesigningrequests/{name

}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified CertificateSigningRequest
PATCH /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update approval of the specified CertificateSigningRequest
PATCH /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}/approval
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified CertificateSigningRequest
PATCH /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a CertificateSigningRequest
DELETE /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of CertificateSigningRequest
DELETE /apis/certificates.k8s.io/v1/certificatesigningrequests
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.5 - Authorization Resources
5.5.1 - LocalSubjectAccessReview
import "k8s.io/api/authorization/v1"
LocalSubjectAccessReview checks whether or not a user or group can perform an action in a given namespace. Having a namespace scoped resource makes it much easier to grant namespace scoped policy that includes permissions checking.
: authorization.k8s.io/v1
kind
: LocalSubjectAccessReview
metadata
(
)
Standard list metadata. Mor

e info:
spec
(
SubjectAccessReviewSpec
Spec holds information about the request being evaluated. spec.namespace must be equal to the namespace you made the request against. If empty, it is defaulted.
status
(
SubjectAccessReviewStatus
)
Status is filled in by the server and indicates whether the request is allowed or not
create
create a LocalSubjectAccessReview
POST /apis/authorization.k8s.io/v1/namespaces/{namespace}/localsubjectaccessreviews
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.5.2 - SelfSubjectAccessReview
SelfSubjectAccessReview checks whether or the current user can perform an action. Not filling in a spec.namespace means "in all namespaces". Self is a special case, because users should always be able to check whether they can perform an action
kind
: SelfSubjectAccessReview
metadata
(
)
spec
(
SelfSubjectAccessReviewSpec
Spec holds information about the request being evaluated. user and groups must be empty
status
(
)
SelfSubjectAccessReviewSpec is a description of the access request. Exactly one of ResourceAuthorizationAttributes and NonResourceAuthorizationAttributes must be set
nonResourceAttributes
(NonResourceAttributes)
NonResourceAttributes describes information for a non-resource access request
NonResourceAttributes includes the authorization attributes available for non-resource requests to the Authorizer interface
nonResourceAttributes.path
(string)
Path is the URL path of the request
nonResourceAttributes.verb
(string)
Verb is the standard HTTP verb
(ResourceAttributes)
ResourceAuthorizationAttributes describes information for a resource access request
ResourceAttributes includes the authorization attributes available for resource requests to the Authorizer interface
resourceAttributes.group
(string)
Group is the API Group of the Resource. "*" means all.
resourceAttributes.name
(string)
Name is the name of the resource being requested for a "get" or deleted for a "delete". "" (empty) means all.
resourceAttributes.namespace
(string)
Namespace is the namespace of the action being requested. Currently, there is no distinction between no namespace and all namespaces "" (empty) is defaulted for LocalSubjectAccessReviews "" (empty) is empty for cluster-scoped resources "" (empty) means "all" for namespace scoped resources from a SubjectAccessReview or SelfSubjectAccessReview
resourceAttributes.resource
(string)
Resource is one of the existing resource types. "*" means all.
resourceAttributes.subresource
(string)
Subresource is one of the existing resource types. "" means none.
resourceAttributes.verb
(string)
Verb is a kubernetes resource API verb, like: get, list, watch, create, update, delete, proxy. "*" means all.
resourceAttributes.version
(string)
Version is the API Version of the Resource. "*" means all.
create
create a SelfSubjectAccessReview
POST /apis/authorization.k8s.io/v1/selfsubjectaccessreviews
body
:
, required
dryRun
(

in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.5.3 - SelfSubjectRulesReview
SelfSubjectRulesReview enumerates the set of actions the current user can perform within a namespace. The returned list of actions may be incomplete depending on the server's authorization mode, and any errors experienced during the evaluation. SelfSubjectRulesReview should be used by UIs to show/hide actions, or to quickly let an end user reason about their permissions. It should NOT Be used by external systems to drive authorization decisions as this raises confused deputy, cache lifetime/revocation, and correctness concerns. SubjectAccessReview, and LocalAccessReview are the correct way to defer authorization decisions to the API server.
kind
: SelfSubjectRulesReview
metadata
(
)
spec
(
SelfSubjectRulesReviewSpec
Spec holds information about the request being evaluated.
status
(SubjectRulesReviewStatus)
Status is filled in by the server and indicates the set of actions a user can perform.
SubjectRulesReviewStatus contains the result of a rules check. This check can be incomplete depending on the set of authorizers the server is configured with and any errors experienced during evaluation. Because authorization rules are additive, if a rule appears in a list it's safe to assume the subject has that permission, even if that list is incomplete.
status.incomplete
Incomplete is true when the rules returned by this call are incomplete. This is most commonly encountered when an authorizer, such as an external authorizer, doesn't support rules evaluation.
status.nonResourceRules
([]NonResourceRule), required
NonResourceRules is the list of actions the subject is allowed to perform on non-resources. The list ordering isn't significant, may contain duplicates, and possibly be incomplete.
NonResourceRule holds information that describes a rule for the non-resource
status.nonResourceRules.verbs
Verb is a list of kubernetes non-resource API verbs, like: get, post, put, delete, patch, head, options. "*" means all.
status.nonResourceRules.nonResourceURLs
NonResourceURLs is a set of partial urls that a user should have access to.
s are allowed, but only as the full, final step in the path. "
" means all.
status.resourceRules
([]ResourceRule), required
ResourceRules is the list of actions the subject is allowed to perform on resources. The list ordering isn't significant, may contain duplicates, and possibly be incomplete.
ResourceRule is the list of actions the subject is allowed to perform on resources. The list ordering isn't significant, may contain duplicates, and possibly be incomplete.
status.resourceRules.verbs
Verb is a list of kubernetes resource API verbs, like: get, list, watch, create, update, delete, proxy. "*" means all.
status.resourceRules.apiGroups
APIGroups is the name of the APIGroup that contains the resources. If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed. "*" means all.
status.resourceRules.resourceNames
ResourceNames is an optional white list of names that the rule applies to. An empty set means that everything is allowed. "*" means all.
status.resourceRules.resources
Resources is a list of resources this rule applies 

to. "
" means all in the specified apiGroups.
"
/foo" represents the subresource 'foo' for all resources in the specified apiGroups.
status.evaluationError
(string)
EvaluationError can appear in combination with Rules. It indicates an error occurred during rule evaluation, such as an authorizer that doesn't support rule evaluation, and that ResourceRules and/or NonResourceRules may be incomplete.
SelfSubjectRulesReviewSpec defines the specification for SelfSubjectRulesReview.
namespace
(string)
Namespace to evaluate rules for. Required.
create
create a SelfSubjectRulesReview
POST /apis/authorization.k8s.io/v1/selfsubjectrulesreviews
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.5.4 - SubjectAccessReview
kind
: SubjectAccessReview
metadata
(
)
spec
(
status
(
)
SubjectAccessReviewSpec is a description of the access request. Exactly one of ResourceAuthorizationAttributes and NonResourceAuthorizationAttributes must be set
extra
Extra corresponds to the user.Info.GetExtra() method from the authenticator. Since that is input to the authorizer it needs a reflection here.
groups
Groups is the groups you're testing for.
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
uid
(string)
UID information about the requesting user.
user
(string)
User is the user you're testing for. If you speci

fy "User" but not "Groups", then is it interpreted as "What if User were not a member of any groups
allowed
Allowed is required. True if the action would be allowed, false otherwise.
denied
(boolean)
Denied is optional. True if the action would be denied, otherwise false. If both allowed is false and denied is false, then the authorizer has no opinion on whether to authorize the action. Denied may not be true if Allowed is true.
evaluationError
(string)
EvaluationError is an indication that some error occurred during the authorization check. It is entirely possible to get an error and be able to continue determine authorization status in spite of it. For instance, RBAC can be missing a role, but enough roles are still present and bound to reason about the request.
reason
(string)
Reason is optional. It indicates why a request was allowed or denied.
create
create a SubjectAccessReview
POST /apis/authorization.k8s.io/v1/subjectaccessreviews
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.5.5 - SelfSubjectReview v1alpha1
apiVersion: authentication.k8s.io/v1alpha1
import "k8s.io/api/authentication/v1alpha1"
SelfSubjectReview contains the user information that the kube-apiserver has about the user making this request. When using impersonation, users will receive the user info of the user being impersonated.
: authentication.k8s.io/v1alpha1
kind
: SelfSubjectReview
metadata
(
)
status
(
SelfSubjectReviewStatus
)
Status is filled in by the server with the user attributes.
SelfSubjectReviewStatus is filled by the kube-apiserver and sent back to a user.
userInfo
User attributes of the user making this request.
userInfo.extra
userInfo.groups
userInfo.uid
(string)
userInfo.username
(string)
create
create a SelfSubjectReview
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.5.6 - ClusterRole
kind
metadata
(
)
AggregationRule is an optional field that describes how to build the Rules for this ClusterRole. If AggregationRule is set, then the Rul

es are controller managed and direct changes to Rules will be stomped by the controller.
([]
)
rules
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
): string, requir

ed
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.5.7 - ClusterRoleBinding
ClusterRoleBinding references a ClusterRole, but not contain it. It can reference a ClusterRole in the global namespace, and adds who information via Subject.
kind
: ClusterRoleBinding
metadata
(
)
roleRef
(RoleRef), required
RoleRef can only reference a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error.
RoleRef contains information that points to the role being used
roleRef.apiGroup
APIGroup is the group for the resource being referenced
roleRef.kind
roleRef.name
subjects
([]Subject)
Subjects holds references to the objects the role applies to.
Subject contains a reference to the object or user identities a role binding applies to. This can either hold a direct API object reference, or a value for non-objects such as user and group names.
subjects.kind
Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not recognized the kind value, the Authorizer should report an error.
subjects.name
Name of the object being referenced.
subjects.apiGroup
(string)
APIGroup holds the API group of the referenced subject. Defaults to "" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io" for User and Group subjects.
subjects.namespace
(string)
Namespace of the referenced object. If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
ClusterRoleBindingList
ClusterRoleBindingList is a collection of ClusterRoleBindings
kind
: ClusterRoleBindingList
metadata
(
ListMeta
)
items
([]
Items is a list of ClusterRoleBindings
get
read the specified ClusterRoleBinding
GET /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
name
(
in path
name of the ClusterRoleBinding
pretty
(
in query
): string
pretty
Response
200 (
): OK
401: U

nauthorized
list
list or watch objects of kind ClusterRoleBinding
GET /apis/rbac.authorization.k8s.io/v1/clusterrolebindings
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ClusterRoleBinding
POST /apis/rbac.authorization.k8s.io/v1/clusterrolebindings
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ClusterRoleBinding
PUT /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ClusterRoleBinding
PATCH /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ClusterRoleBinding
DELETE /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ClusterRoleBinding
DELETE /apis/rbac.authorization.k8s.io/v1/clusterrolebindings
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.5.8 - Role
Role
kind
: Role
metadata
(
)
rules
Rules holds all the PolicyRules for this Role
PolicyRule holds information that describes a policy rule, but does not contain information about who the rule applies to or which namespace t

he rule applies to.
RoleList
RoleList is a collection of Roles
kind
: RoleList
metadata
(
ListMeta
)
items
([]
Role
Items is a list of Roles
get
read the specified Role
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
name
(
in path
name of the Role
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
list
list or watch objects of kind Role
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
list
GET /apis/rbac.authorization.k8s.io/v1/roles
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
create
create a Role
POST /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
202 (
Role
update
replace the specified Role
PUT /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
patch
partially update the specified Role
PATCH

 /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
delete
delete a Role
DELETE /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Role
DELETE /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.5.9 - RoleBinding
RoleBinding references a role, but does not contain it. It can reference a Role in the same namespace or a ClusterRole in the global namespace. It adds who information via Subjects and namespace information by which namespace it exists in. RoleBindings in a given namespace only have effect in that namespace.
kind
: RoleBinding
metadata
(
)
roleRef
RoleRef can reference a Role in the current namespace or a ClusterRole in the global namespace. If the RoleRef cannot be resolved, the Authorizer must return an error.
subjects
(string)
(string)
RoleBindingList
RoleBindingList is a collection of RoleBindings
kind
: RoleBindingList
metadata
(
ListMeta
)
Standard object's me

tadata.
items
([]
Items is a list of RoleBindings
get
read the specified RoleBinding
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}
name
(
in path
name of the RoleBinding
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind RoleBinding
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/rbac.authorization.k8s.io/v1/rolebindings
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a RoleBinding
POST /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified RoleBinding
PUT /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified RoleBinding
PATCH /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a RoleBinding
DELETE /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of RoleBinding
DELETE /apis/rbac.author

ization.k8s.io/v1/namespaces/{namespace}/rolebindings
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.6 - Policy Resources
5.6.1 - LimitRange
LimitRange sets resource usage limits for each kind of resource in a Namespace.
: v1
kind
: LimitRange
metadata
(
)
spec
(
LimitRangeSpec
)
Spec defines the limits enforced. More info:
LimitRangeSpec defines a min/max usage limit for resources that match on kind.
limits
([]LimitRangeItem), required
Limits is the list of LimitRangeItem objects that are enforced.
LimitRangeItem defines a min/max usage limit for any resource that matches on kind.
limits.type
Type of resource that this limit applies to.
limits.default
Quantity
)
Default resource requirement limit value by resource name if resource limit is omitted.
limits.defaultRequest
Quantity
)
DefaultRequest is the default resource requirement request value by resource name if resource request is omitted.
limits.max
Quantity
)
Max usage constraints on this kind by resource name.
limits.maxLimitRequestRatio
Quantity
)
MaxLimitRequestRatio if specified, the named resource must have a request and limit that are both non-zero where limit divided by request is less than or equal to the enumerated value; this represents the max burst for the named resource.
limits.min
Quantity
)
Min usage constraints on this kind by resource name.
LimitRangeList
LimitRangeList is a list of LimitRange items.
: v1
kind
: LimitRangeList
metadata
(
ListMeta
)
items
([]
Items is a list of LimitRange objects. More info:
get
read the specified LimitRange
GET /api/v1/namespaces/{namespace}/limitranges/{name}
name
(
in path
name of the LimitRange
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind LimitRange
GET /api/v1/namespaces/{namespace}/limitranges
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/limitranges
(
in query
): boolean
allowWatchBo

okmarks
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a LimitRange
POST /api/v1/namespaces/{namespace}/limitranges
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified LimitRange
PUT /api/v1/namespaces/{namespace}/limitranges/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified LimitRange
PATCH /api/v1/namespaces/{namespace}/limitranges/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a LimitRange
DELETE /api/v1/namespaces/{namespace}/limitranges/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of LimitRange
DELETE /api/v1/namespaces/{namespace}/limitranges
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.6.2 - ResourceQuota
ResourceQuota sets aggregate quota restrictions enforced per namespace.
ResourceQuota sets aggregate quota restrictions enforced per namespace
: v1
kind
: ResourceQuota
metadata
(
)
spec
(
ResourceQuotaSpec
)
Spec defines the desired quota.
status
(
ResourceQuotaStatus
)
Status defines the actual enforced quota and its current usage.

ResourceQuotaSpec defines the desired hard limits to enforce for Quota.
hard
Quantity
)
hard is the set of desired hard limits for each named resource. More info:
https://kubernetes.io/docs/concepts/policy/resource-quotas/
scopeSelector
(ScopeSelector)
scopeSelector is also a collection of filters like scopes that must match each object tracked by a quota but expressed using ScopeSelectorOperator in combination with possible values. For a resource to match, both scopes AND scopeSelector (if specified in spec), must be matched.
A scope selector represents the AND of the selectors represented by the scoped-resource selector requirements.
scopeSelector.matchExpressions
([]ScopedResourceSelectorRequirement)
A list of scope selector requirements by scope of the resources.
A scoped-resource selector requirement is a selector that contains values, a scope name, and an operator that relates the scope name and values.
scopeSelector.matchExpressions.operator
Represents a scope's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist.
scopeSelector.matchExpressions.scopeName
The name of the scope that the selector applies to.
scopeSelector.matchExpressions.values
An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
scopes
A collection of filters that must match each object tracked by a quota. If not specified, the quota matches all objects.
ResourceQuotaStatus defines the enforced hard limits and observed use.
hard
Quantity
)
Hard is the set of enforced hard limits for each named resource. More info:
used
Quantity
)
Used is the current observed total usage of the resource in the namespace.
ResourceQuotaList
ResourceQuotaList is a list of ResourceQuota items.
: v1
kind
: ResourceQuotaList
metadata
(
ListMeta
)
items
([]
Items is a list of ResourceQuota objects. More info:
get
read the specified ResourceQuota
GET /api/v1/namespaces/{namespace}/resourcequotas/{name}
name
(
in path
name of the ResourceQuota
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified ResourceQuota
GET /api/v1/namespaces/{namespace}/resourcequotas/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ResourceQuota
GET /api/v1/namespaces/{namespace}/resourcequotas
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /api/v1/resourcequotas
(
in query
): boolean
continue
(
in query
): string
continue
(
in 

query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ResourceQuota
POST /api/v1/namespaces/{namespace}/resourcequotas
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ResourceQuota
PUT /api/v1/namespaces/{namespace}/resourcequotas/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified ResourceQuota
PUT /api/v1/namespaces/{namespace}/resourcequotas/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ResourceQuota
PATCH /api/v1/namespaces/{namespace}/resourcequotas/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified ResourceQuota
PATCH /api/v1/namespaces/{namespace}/resourcequotas/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ResourceQuota
DELETE /api/v1/namespaces/{namespace}/resourcequotas/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
delete collection of ResourceQuota
DELETE /api/v1/namespaces/{namespace}/resourcequotas
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): s

tring
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.6.3 - NetworkPolicy
NetworkPolicy describes what network traffic is allowed for a set of Pods.
NetworkPolicy describes what network traffic is allowed for a set of Pods
kind
: NetworkPolicy
metadata
(
)
spec
(
NetworkPolicySpec
)
Specification of the desired behavior for this NetworkPolicy.
status
(
NetworkPolicyStatus
)
Status is the current state of the NetworkPolicy. More info:
NetworkPolicySpec provides the specification of a NetworkPolicy
(
Selects the pods to which this NetworkPolicy object applies. The array of ingress rules is applied to any pods selected by this field. Multiple network policies can select the same set of pods. In this case, the ingress rules for each are combined additively. This field is NOT optional and follows standard label selector semantics. An empty podSelector matches all pods in this namespace.
policyTypes
List of rule types that the NetworkPolicy relates to. Valid options are ["Ingress"], ["Egress"], or ["Ingress", "Egress"]. If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you want to write an egress-only policy, you must explicitly specify policyTypes [ "Egress" ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include "Egress" (since such a policy would not include an Egress section and would otherwise default to just [ "Ingress" ]). This field is beta-level in 1.8
ingress
([]NetworkPolicyIngressRule)
List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic source is the pod's local node, OR if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are isolated by default)
NetworkPolicyIngressRule describes a particular set of traffic that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and from.
ingress.from
([]NetworkPolicyPeer)
List of sources which should be able to access the pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all sources (traffic not restricted by source). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the from list.
NetworkPolicyPeer describes a peer to allow traffic to/from. Only certain combinations of fields are allowed
ingress.from.ipBlock
(IPBlock)
IPBlock defines policy on a particular IPBlock. If this field is set then neither of the other fields can be.
IPBlock describes a particular CIDR (Ex. "192.168.1.0/24","2001:db8::/64") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs 

that should not be included within this rule.
ingress.from.ipBlock.cidr
CIDR is a string representing the IP Block Valid examples are "192.168.1.0/24" or "2001:db8::/64"
ingress.from.ipBlock.except
Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.0/24" or "2001:db8::/64" Except values will be rejected if they are outside the CIDR range
ingress.from.namespaceSelector
(
)
Selects Namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.
If PodSelector is also set, then the NetworkPolicyPeer as a whole selects the Pods matching PodSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects all Pods in the Namespaces selected by NamespaceSelector.
ingress.from.podSelector
(
)
This is a label selector which selects Pods. This field follows standard label selector semantics; if present but empty, it selects all pods.
If NamespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the Pods matching PodSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the Pods matching PodSelector in the policy's own Namespace.
ingress.ports
([]NetworkPolicyPort)
List of ports which should be made accessible on the pods selected for this rule. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list.
NetworkPolicyPort describes a port to allow traffic on
ingress.ports.port
The port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the specified protocol AND port will be matched.
ingress.ports.endPort
(int32)
If set, indicates that the range of ports from port to endPort, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port.
ingress.ports.protocol
(string)
The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults to TCP.
egress
([]NetworkPolicyEgressRule)
List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic matches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8
NetworkPolicyEgressRule describes a particular set of traffic that is allowed out of pods matched by a NetworkPolicySpec's podSelector. The traffic must match both ports and to. This type is beta-level in 1.8
egress.to
List of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows traffic only if the traffic matches at least one item in the to list.
egress.to.ipBlock
(IP

Block)
IPBlock describes a particular CIDR (Ex. "192.168.1.0/24","2001:db8::/64") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should not be included within this rule.
egress.to.ipBlock.cidr
egress.to.ipBlock.except
egress.to.namespaceSelector
(
)
egress.to.podSelector
(
)
egress.ports
List of destination ports for outgoing traffic. Each item in this list is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list.
egress.ports.port
egress.ports.endPort
(int32)
egress.ports.protocol
(string)
NetworkPolicyStatus describe the current state of the NetworkPolicy.
type
Conditions holds an array of metav1.Condition that describe the state of the NetworkPolicy. Current service state
reason contai

ns a programmatic identifier indicating the reason for the condition's last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
(int64)
NetworkPolicyList
NetworkPolicyList is a list of NetworkPolicy objects.
kind
: NetworkPolicyList
metadata
(
ListMeta
)
items
([]
get
read the specified NetworkPolicy
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
name
(
in path
name of the NetworkPolicy
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified NetworkPolicy
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind NetworkPolicy
GET /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/networking.k8s.io/v1/networkpolicies
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a NetworkPolicy
POST /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified NetworkPolicy
PUT /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Netwo

rkPolicy
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified NetworkPolicy
PUT /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified NetworkPolicy
PATCH /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified NetworkPolicy
PATCH /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a NetworkPolicy
DELETE /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of NetworkPolicy
DELETE /apis/networking.k8s.io/v1/namespaces/{namespace}/networkpolicies
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.6.4 - PodDisruptionBudget
PodDisruptionBudget is an object to define the max disruption that can be caused to a collection of pods.
apiVersion: policy/v1
import "k8s.io/api/policy/v1"
PodDisruptionBudget is an object to define the max disruption that can be caused to a collection of pods
: policy/v1
kind
: PodDisruptionBudget
metadata
(
)
spec
(
PodDisruptionBudgetSpec
)
Specification of the desired behavior of the PodDisruptionBudget.
status
(
PodDisruptionBudgetStatus
)
Most recently observed statu

s of the PodDisruptionBudget.
PodDisruptionBudgetSpec is a description of a PodDisruptionBudget.
An eviction is allowed if at most "maxUnavailable" pods selected by "selector" are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. This is a mutually exclusive setting with "minAvailable".
minAvailable
An eviction is allowed if at least "minAvailable" pods selected by "selector" will still be available after the eviction, i.e. even in the absence of the evicted pod. So for example you can prevent all voluntary evictions by specifying "100%".
selector
(
)
Label query over pods whose evictions are managed by the disruption budget. A null selector will match no pods, while an empty ({}) selector will select all pods within the namespace.
unhealthyPodEvictionPolicy
(string)
UnhealthyPodEvictionPolicy defines the criteria for when unhealthy pods should be considered for eviction. Current implementation considers healthy pods, as pods that have status.conditions item with type="Ready",status="True".
Valid policies are IfHealthyBudget and AlwaysAllow. If no policy is specified, the default behavior will be used, which corresponds to the IfHealthyBudget policy.
IfHealthyBudget policy means that running pods (status.phase="Running"), but not yet healthy can be evicted only if the guarded application is not disrupted (status.currentHealthy is at least equal to status.desiredHealthy). Healthy pods will be subject to the PDB for eviction.
AlwaysAllow policy means that all running pods (status.phase="Running"), but not yet healthy are considered disrupted and can be evicted regardless of whether the criteria in a PDB is met. This means perspective running pods of a disrupted application might not get a chance to become healthy. Healthy pods will be subject to the PDB for eviction.
Additional policies may be added in the future. Clients making eviction decisions should disallow eviction of unhealthy pods if they encounter an unrecognized policy in this field.
This field is alpha-level. The eviction API uses this field when the feature gate PDBUnhealthyPodEvictionPolicy is enabled (disabled by default).
PodDisruptionBudgetStatus represents information about the status of a PodDisruptionBudget. Status may trail the actual state of a system.
currentHealthy
current number of healthy pods
desiredHealthy
minimum desired number of healthy pods
disruptionsAllowed
Number of pod disruptions that are currently allowed.
expectedPods
total number of pods counted by this disruption budget
type
Conditions contain conditions for PDB. The disruption controller sets the DisruptionAllowed condition. The following are known values for the reason field (additional reasons could be added in the future): - SyncFailed: The controller encountered an error and wasn't able to compute
the number of allowed disruptions. Therefore no disruptions are
allowed and the status of the condition will be False.
InsufficientPods: The number of pods are either at or below the number
required by the PodDisruptionBudget. No disruptions are
SufficientPods: There are more pods than re

quired by the PodDisruptionBudget.
The condition will be True, and the number of allowed
disruptions are provided by the disruptionsAllowed property.
(int64)
disruptedPods
(map[string]Time)
DisruptedPods contains information about pods whose eviction was processed by the API server eviction subresource handler but has not yet been observed by the PodDisruptionBudget controller. A pod will be in this map from the time when the API server processed the eviction request to the time when the pod is seen by PDB controller as having been marked for deletion (or after a timeout). The key in the map is the name of the pod and the value is the time when the API server processed the eviction request. If the deletion didn't occur and a pod is still there it will be removed from the list automatically by PodDisruptionBudget controller after some time. If everything goes smooth this map should be empty for the most of the time. Large number of entries in the map may indicate problems with pod deletions.
(int64)
Most recent generation observed when updating this PDB status. DisruptionsAllowed and other status information is valid only if observedGeneration equals to PDB's object generation.
PodDisruptionBudgetList
PodDisruptionBudgetList is a collection of PodDisruptionBudgets.
kind
: PodDisruptionBudgetList
metadata
(
ListMeta
)
items
([]
Items is a list of PodDisruptionBudgets
get
read the specified PodDisruptionBudget
GET /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}
name
(
in path
name of the PodDisruptionBudget
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified PodDisruptionBudget
GET /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
PodDisruptionBud

get
): OK
list
list or watch objects of kind PodDisruptionBudget
GET /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
GET /apis/policy/v1/poddisruptionbudgets
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a PodDisruptionBudget
POST /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified PodDisruptionBudget
PUT /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified PodDisruptionBudget
PUT /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified PodDisruptionBudget
PATCH /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified PodDisruptionBudget
PATCH /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): s

tring
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a PodDisruptionBudget
DELETE /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of PodDisruptionBudget
DELETE /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.7 - Extend Resources
5.7.1 - CustomResourceDefinition
kind
metadata
(
)
spec
(
status
(
)
group
).
names
kind
.
(string)
kind
List".
shortNames are short names for the resource, exposed in API discovery documents, and used b

y clients to support invocations like
(string)
kind
.
scope
Cluster
and
.
versions
if
served
is true.
(string)
(string)
(int32)
(boolean)
(string)
(CustomResourceValidati

on)
CustomResourceValidation is a list of validation methods for CustomResources.
(
)
/scale
.spec
/scale
.status
/scale
(string)
.status
or
.spec
/scale
/status
status
/status
status
.status
None
Webhook
strategy
is set to
Webhook
.
WebhookConversion describes how to call a

 conversion webhook
versions the Webhook expects. The API server will use the first version in the list which it supports. If none of the versions specified in this list are supported by API server, conversion will fail for the custom resource. If a persisted Webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail.
Webhook
.
([]byte)
service
.
(string)
(int32)
port
(string)
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
(boolean)
to true in
. See
.
$ref
(string)
$schema
(string)
allOf
([]
)
anyOf
([]
JSONSchemaPro

ps
)
default
(JSON)
default is a default value for undefined object fields. Defaulting is a beta feature under the CustomResourceDefaulting feature gate. Defaulting requires spec.preserveUnknownFields to be false.
)
(string)
enum
([]JSON)
example
(JSON)
(boolean)
(boolean)
(string)
(string)
format
(string)
id
(string)
items
maxItems
(int64)
maxLength
(int64)
(int64)
maximum
(double)
minItems
(int64)
minLength
(int64)
(int64)
minimum
(double)
(double)
not
(
)
nullable
(boolean)
oneOf
([]
)
pattern
(string)
)
)
required
title
(string)
type
(string)
(boolean)
(boolean)
x-kubernetes-embedded-

resource defines that the value is an embedded Kubernetes runtime.Object, with TypeMeta and ObjectMeta. The type must be object. It is allowed to further restrict the embedded object. kind, apiVersion and metadata are validated automatically. x-kubernetes-preserve-unknown-fields is allowed to be true, but does not have to be if the object is fully specified (up to kind, apiVersion, metadata).
(boolean)
anyOf:
allOf:
anyOf:
map
(string)
atomic
set
:
atomic
or an
atomic
.
map
:
(string)
granular
:
atomic
(boolean)
rule
self
and f

ield presence can be checked via
. Null valued fields are treated as absent fields in CEL expressions. If the Rule is scoped to an object with additionalProperties (i.e. a map) the value of the map are accessible via
self[i]
self
The
,
kind
,
and
dot
dash
slash
Examples:
namespace
&gt; 0"}
'set':
X + Y
X
Y
'map':
X + Y
X
Y
X
and
Y
Y
with
(string)
kind
(s

tring), required
.
(string)
kind
List".
(string)
kind
.
(Time)
(string)
(string)
items
([]
(string)
kind
(string)
metadata
(
ListMeta
)
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
name of the CustomResourceDefi

nition
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
deletecoll

ection
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.7.2 - MutatingWebhookConfiguration
MutatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and may change the object.
apiVersion: admissionregistration.k8s.io/v1
import "k8s.io/api/admissionregistration/v1"
: admissionregistration.k8s.io/v1
kind
: MutatingWebhookConfiguration
metadata
(
)
Standard object metadata; More info:
.
webhooks
([]MutatingWebhook)
name
Webhooks is a list of webhooks and the affected resources and operations.
MutatingWebhook describes an admission webhook and the resources and operations it applies to.
webhooks.admissionReviewVersions
AdmissionReviewVersions is an ordered list of preferred
versions the Webhook expects. API server will try to use first version in the list which it supports. If none of the versions specified in this list supported by API server, validation will fail for this object. If a persisted webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail and be subject to the failure policy.
webhooks.clientConfig
(WebhookClientConfig), required
ClientConfig defines how to communicate with the hook. Required
WebhookClientConfig contains the information to make a TLS connection with the webhook
webhooks.clientConfig.caBundle
([]byte)
caBundle
is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used.
webhooks.clientConfig.service
service
is a reference to the service for this webhook. Either
service
or
url
service
.
webhooks.clientConfig.service.name
name
is the name of the service. Required
webhooks.clientConfig.service.namespace
namespace
is the namespace of the service. Required
webhooks.clientConfig.service.path
(string)
path
is an optional URL path which will be sent in any request to this service.
webhooks.clientConfig.service.port
(int32)
port
webhooks.clientConfig.url
(string)
url
gives the location of the webhook, in standard URL form (
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make 

calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster.
webhooks.name
The name of the admission webhook. Name should be fully qualified, e.g., imagepolicy.kubernetes.io, where "imagepolicy" is the name of the webhook, and kubernetes.io is the name of the organization. Required.
webhooks.sideEffects
SideEffects states whether this webhook has side effects. Acceptable values are: None, NoneOnDryRun (webhooks created via v1beta1 may also specify Some or Unknown). Webhooks with side effects MUST implement a reconciliation system, since a request may be rejected by a future step in the admission chain and the side effects therefore need to be undone. Requests with the dryRun attribute will be auto-rejected if they match a webhook with sideEffects == Unknown or Some.
webhooks.failurePolicy
(string)
FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Fail.
webhooks.matchPolicy
(string)
matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent".
Exact: match a request only if it exactly matches a specified rule. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, but "rules" only included
apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]
, a request to apps/v1beta1 or extensions/v1beta1 would not be sent to the webhook.
Equivalent: match a request if modifies a resource listed in rules, even via another API group or version. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, and "rules" only included
, a request to apps/v1beta1 or extensions/v1beta1 would be converted to apps/v1 and sent to the webhook.
Defaults to "Equivalent"
webhooks.namespaceSelector
(
)
NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.
For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1"; you will set the selector as follows: "namespaceSelector": {
"matchExpressions": [
{
"key": "runlevel",
"operator": "NotIn",
"values": [
"0",
"1"
]
}
]
}
If instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {
{
"key": "environment",
"operator": "In",
"prod",
"staging"
]
}
]
}
See
https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
Default to the empty LabelSelector, which matches everything.
webhooks.objectSelector
(
)
ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in

, because end users may skip the admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
webhooks.reinvocationPolicy
(string)
reinvocationPolicy indicates whether this webhook should be called multiple times as part of a single admission evaluation. Allowed values are "Never" and "IfNeeded".
Never: the webhook will not be called more than once in a single admission evaluation.
IfNeeded: the webhook will be called at least one additional time as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call. Webhooks that specify this option
must
be idempotent, able to process objects they previously admitted. Note: * the number of additional invocations is not guaranteed to be exactly one. * if additional invocations result in further modifications to the object, webhooks are not guaranteed to be invoked again. * webhooks that use this option may be reordered to minimize the number of additional invocations. * to validate an object after all mutations are guaranteed complete, use a validating admission webhook instead.
Defaults to "Never".
webhooks.rules
([]RuleWithOperations)
Rules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches
any
Rule. However, in order to prevent ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks from putting the cluster in a state which cannot be recovered from without completely disabling the plugin, ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks are never called on admission requests for ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects.
RuleWithOperations is a tuple of Operations and Resources. It is recommended to make sure that all the tuple expansions are valid.
webhooks.rules.apiGroups
APIGroups is the API groups the resources belong to. '
' is all groups. If '
' is present, the length of the slice must be one. Required.
webhooks.rules.apiVersions
APIVersions is the API versions the resources belong to. '
' is all versions. If '
webhooks.rules.operations
Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If '*' is present, the length of the slice must be one. Required.
webhooks.rules.resources
Resources is a list of resources this rule applies to.
For example: 'pods' means pods. 'pods/log' means the log subresource of pods. '
' means all resources, but not subresources. 'pods/
' means all subresources of pods. '
/scale' means all scale subresources. '
/*' means all resources and their subresources.
If wildcard is present, the validation rule will ensure resources do not overlap with each other.
Depending on the enclosing object, subresources might not be allowed. Required.
webhooks.rules.scope
(string)
scope specifies the scope of this rule. Valid values are "Cluster", "Namespaced", and "
" "Cluster" means that only cluster-scoped resources will match this rule. Namespace API objects are cluster-scoped. "Namespaced" means that only namespaced resources will match this rule. "
" means that there are no scope restrictions. Subresources match the scope of their parent resource. Default is "*".
webhooks.timeoutSeconds
(int32)
TimeoutSeconds specifies the timeout for this webhook. After the timeout passes, the webhook call will be ignored or the API call will fail based on the failure policy. The timeout value must be between 1 and 30 seconds. Default to 10 seconds.
MutatingWebhookConfigurationList
MutatingWebhookConfigurationList is a list of MutatingWebhookConfiguration.
apiVers

ion
kind
: MutatingWebhookConfigurationList
metadata
(
ListMeta
)
items
([]
List of MutatingWebhookConfiguration.
get
read the specified MutatingWebhookConfiguration
GET /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}
name
(
in path
name of the MutatingWebhookConfiguration
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind MutatingWebhookConfiguration
GET /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a MutatingWebhookConfiguration
POST /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified MutatingWebhookConfiguration
PUT /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified MutatingWebhookConfiguration
PATCH /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a MutatingWebhookConfiguration
DELETE /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of MutatingWebhookConfiguration
DELETE /apis/admissionregistration.k8s.io/v1/mutatingwebhookconfigurations
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
lim

it
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.7.3 - ValidatingWebhookConfiguration
ValidatingWebhookConfiguration describes the configuration of and admission webhook that accept or reject and object without changing it.
kind
: ValidatingWebhookConfiguration
metadata
(
)
.
webhooks
([]ValidatingWebhook)
name
ValidatingWebhook describes an admission webhook and the resources and operations it applies to.
([]byte)
caBundle
service
service
or
url
service
.
name
namespace
(string)
path
(int32)
port
(string)
url
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
Attempting to use a user or basic auth e.g. "user

:password@" is not allowed. Fragments ("#...") and query parameters ("?...") are not allowed, either.
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
https://kubernetes.io/docs/concepts/overview/working-with-objects/labels
(
)
ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
any
Rule. However, in order to prevent ValidatingAdmissionWebhooks and MutatingAdmission

Webhooks from putting the cluster in a state which cannot be recovered from without completely disabling the plugin, ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks are never called on admission requests for ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects.
(string)
(int32)
ValidatingWebhookConfigurationList
ValidatingWebhookConfigurationList is a list of ValidatingWebhookConfiguration.
kind
: ValidatingWebhookConfigurationList
metadata
(
ListMeta
)
items
([]
List of ValidatingWebhookConfiguration.
get
read the specified ValidatingWebhookConfiguration
GET /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/{name}
name
(
in path
name of the ValidatingWebhookConfiguration
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ValidatingWebhookConfiguration
GET /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ValidatingWebhookConfigurati

on
POST /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ValidatingWebhookConfiguration
PUT /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ValidatingWebhookConfiguration
PATCH /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ValidatingWebhookConfiguration
DELETE /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ValidatingWebhookConfiguration
DELETE /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.7.4 - ValidatingAdmissionPolicy v1alpha1
ValidatingAdmissionPolicy describes the definition of an admission validation policy that accepts or rejects an object without changing it.
apiVersion: admissionregistration.k8s.io/v1alpha1
import "k8s.io/api/admissionregistration/v1alpha1"
: admissionregistration.k8s.io/v1alpha1
kind
: ValidatingAdmissionPolicy
metadata
(
)
.
spec
(ValidatingAdmissionPolicySpec)
Specification of the desired behavior of the ValidatingAdmissionPolicy.
ValidatingAdmissionPolicySpec is the specification of the desired behavior of the AdmissionPolicy.
([]Validation), required
Validations contain CEL expressions which is used to apply the validation. A m

inimum of one validation is required for a policy definition. Required.
Validation specifies the CEL expression which is used to apply the validation.
spec.validations.expression
Expression represents the expression which will be evaluated by CEL. ref:
CEL expressions have access to the contents of the Admission request/response, organized into CEL variables as well as some other useful variables:
'object' - The object from the incoming request. The value is null for DELETE requests. 'oldObject' - The existing object. The value is null for CREATE requests. 'request' - Attributes of the admission request(
ref
). 'params' - Parameter resource referred to by the policy binding being evaluated. Only populated if the policy has a ParamKind.
The
,
kind
,
and
are always accessible from the root of the object. No other metadata properties are accessible.
dot
dash
slash
Examples:
Expression accessing a property named "namespace": {"Expression": "object.
namespace
&gt; 0"}
Expression accessing a property named "x-prop": {"Expression": "object.x__dash__prop &gt; 0"}
Expression accessing a property named "redact__d": {"Expression": "object.redact__underscores__d &gt; 0"}
Equality on arrays with list type of 'set' or 'map' ignores element order, i.e. [1, 2] == [2, 1]. Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:
'set':
X + Y
X
Y
'map':
X + Y
X
Y
X
and
Y
Y
with
Required.
spec.validations.message
(string)
Message represents the message displayed when validation fails. The message is required if the Expression contains line breaks. The message must not contain line breaks. If unset, the message is "failed rule: {Rule}". e.g. "must be a URL with the host matching spec.host" If the Expression contains line breaks. Message is required. The message must not contain line breaks. If unset, the message is "failed Expression: {Expression}".
spec.validations.reason
(string)
Reason represents a machine-readable description of why this validation failed. If this is the first validation in the list to fail, this reason, as well as the corresponding HTTP response code, are used in the HTTP response to the client. The currently supported reasons are: "Unauthorized", "Forbidden", "Invalid", "RequestEntityTooLarge". If not set, StatusReasonInvalid is used in the response to the client.
(string)
FailurePolicy defines how to handle failures for the admission policy. Failures can occur from invalid or mis-configured policy definitions or bindings. A policy is invalid if spec.paramKind refers to a non-existent Kind. A binding is invalid if spec.paramRef.name refers to a non-existent resource. Allowed values are Ignore or Fail. Defaults to Fail.
(MatchResources)
MatchConstraints specifies what resources this policy is designed to validate. The AdmissionPolicy cares about a request if it matches
all
Constraints. However, in order to prevent clusters from being put into an unstable

 state that cannot be recovered from via the API ValidatingAdmissionPolicy cannot match ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding. Required.
MatchResources decides whether to run the admission control policy on an object based on whether it meets the match criteria. The exclude rules take precedence over include rules (if a resource matches both, it is excluded)
spec.matchConstraints.excludeResourceRules
([]NamedRuleWithOperations)
ExcludeResourceRules describes what operations on what resources/subresources the ValidatingAdmissionPolicy should not care about. The exclude rules take precedence over include rules (if a resource matches both, it is excluded)
NamedRuleWithOperations is a tuple of Operations and Resources with ResourceNames.
spec.matchConstraints.excludeResourceRules.apiGroups
spec.matchConstraints.excludeResourceRules.apiVersions
spec.matchConstraints.excludeResourceRules.operations
spec.matchConstraints.excludeResourceRules.resourceNames
spec.matchConstraints.excludeResourceRules.resources
spec.matchConstraints.excludeResourceRules.scope
(string)
spec.matchConstraints.matchPolicy
(string)
matchPolicy defines how the "MatchResources" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent".
, a request to apps/v1beta1 or extensions/v1beta1 would not be sent to the ValidatingAdmissionPolicy.
, a request to apps/v1beta1 or extensions/v1beta1 would be converted to apps/v1 and sent to the ValidatingAdmissionPolicy.
spec.matchConstraints.namespaceSelector
(
)
NamespaceSelector decides whether to run

 the admission control policy on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the policy.
{
"0",
"1"
]
}
]
}
If instead you want to only run the policy on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {
{
"prod",
"staging"
]
}
]
}
See
spec.matchConstraints.objectSelector
(
)
ObjectSelector decides whether to run the validation based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the cel validation, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
spec.matchConstraints.resourceRules
ResourceRules describes what operations on what resources/subresources the ValidatingAdmissionPolicy matches. The policy cares about an operation if it matches
any
Rule.
spec.matchConstraints.resourceRules.apiGroups
spec.matchConstraints.resourceRules.apiVersions
spec.matchConstraints.resourceRules.operations
spec.matchConstraints.resourceRules.resourceNames
spec.matchConstraints.resourceRules.resources
spec.matchConstraints.resourceRules.scope
(string)
" "Cluster" means that only cluster-scoped resources will match this rule. Namespace API objects are cluster-scoped. "Namespaced" 

means that only namespaced resources will match this rule. "
(ParamKind)
ParamKind specifies the kind of resources used to parameterize this policy. If absent, there are no parameters for this policy and the param CEL variable will not be provided to validation expressions. If ParamKind refers to a non-existent kind, this policy definition is mis-configured and the FailurePolicy is applied. If paramKind is specified but paramRef is unset in ValidatingAdmissionPolicyBinding, the params variable will be null.
ParamKind is a tuple of Group Kind and Version.
spec.paramKind.apiVersion
(string)
APIVersion is the API group version the resources belong to. In format of "group/version". Required.
spec.paramKind.kind
(string)
Kind is the API kind the resources belong to. Required.
ValidatingAdmissionPolicyList
ValidatingAdmissionPolicyList is a list of ValidatingAdmissionPolicy.
(string)
items
([]
)
List of ValidatingAdmissionPolicy.
kind
(string)
metadata
(
ListMeta
)
ValidatingAdmissionPolicyBinding binds the ValidatingAdmissionPolicy with paramerized resources. ValidatingAdmissionPolicyBinding and parameter CRDs together define how cluster administrators configure policies for clusters.
(string)
kind
(string)
metadata
(
)
.
spec
(ValidatingAdmissionPolicyBindingSpec)
Specification of the desired behavior of the ValidatingAdmissionPolicyBinding.
ValidatingAdmissionPolicyBindingSpec is the specification of the ValidatingAdmissionPolicyBinding.
spec.matchResources
MatchResources declares what resources match this binding and will be validated by it. Note that this is intersected with the policy's matchConstraints, so only requests that are matched by the policy can be selected by this. If this is unset, all resources matched by the policy are validated by this binding When resourceRules is unset, it does not constrain resource matching. If a resource is matched by the other fields of this object, it will be validated. Note that this is differs from ValidatingAdmissionPolicy matchConstraints, where resourceRules are required.
spec.matchResources.excludeResourceRules
([]NamedRuleWithOper

ations)
spec.matchResources.excludeResourceRules.apiGroups
spec.matchResources.excludeResourceRules.apiVersions
spec.matchResources.excludeResourceRules.operations
spec.matchResources.excludeResourceRules.resourceNames
spec.matchResources.excludeResourceRules.resources
spec.matchResources.excludeResourceRules.scope
(string)
spec.matchResources.matchPolicy
(string)
spec.matchResources.namespaceSelector
(
)
NamespaceSelector decides whether to run the admission control policy on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the policy.
"matchExpress

ions": [
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
spec.matchResources.objectSelector
(
)
spec.matchResources.resourceRules
any
Rule.
spec.matchResources.resourceRules.apiGroups
spec.matchResources.resourceRules.apiVersions
spec.matchResources.resourceRules.operations
spec.matchResources.resourceRules.resourceNames
spec.matchResources.resourceRules.resources
spec.matchResources.resourceRules.scope
(string)
spec.paramRef
(ParamRef)
ParamRef specifies the parameter resource used to configure the admission control policy. It should point to a resource of the type specified in ParamKind of the bound ValidatingAdmissionPolicy. If the policy specifies a ParamKind and the resource referred to by ParamRef does n

ot exist, this binding is considered mis-configured and the FailurePolicy of the ValidatingAdmissionPolicy applied.
ParamRef references a parameter resource
spec.paramRef.name
(string)
Name of the resource being referenced.
spec.paramRef.namespace
(string)
Namespace of the referenced resource. Should be empty for the cluster-scoped resources
spec.policyName
(string)
PolicyName references a ValidatingAdmissionPolicy name which the ValidatingAdmissionPolicyBinding binds to. If the referenced resource does not exist, this binding is considered invalid and will be ignored Required.
get
read the specified ValidatingAdmissionPolicy
GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
name
(
in path
name of the ValidatingAdmissionPolicy
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ValidatingAdmissionPolicy
GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ValidatingAdmissionPolicy
POST /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ValidatingAdmissionPolicy
PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ValidatingAdmissionPolicy
PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ValidatingAdmissionPolicy
DELETE /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ValidatingAdmissionPolicy
DELETE /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicies
body
:
continue
(
in query
): string
contin

ue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8 - Cluster Resources
5.8.1 - Node
Node
Node is a worker node in Kubernetes. Each node will have a unique identifier in the cache (i.e. in etcd).
: v1
kind
: Node
metadata
(
)
spec
(
NodeSpec
)
Spec defines the behavior of a node.
status
(
NodeStatus
)
Most recently observed status of the node. Populated by the system. Read-only. More info:
NodeSpec
NodeSpec describes the attributes that a node is created with.
configSource
(NodeConfigSource)
Deprecated: Previously used to specify the source of the node's configuration for the DynamicKubeletConfig feature. This feature is removed.
NodeConfigSource specifies a source of node configuration. Exactly one subfield (excluding metadata) must be non-nil. This API is deprecated since 1.22
configSource.configMap
(ConfigMapNodeConfigSource)
ConfigMap is a reference to a Node's ConfigMap
ConfigMapNodeConfigSource contains the information to reference a ConfigMap as a config source for the Node. This API is deprecated since 1.22:
https://git.k8s.io/enhancements/keps/sig-node/281-dynamic-kubelet-configuration
configSource.configMap.kubeletConfigKey
KubeletConfigKey declares which key of the referenced ConfigMap corresponds to the KubeletConfiguration structure This field is required in all cases.
configSource.configMap.name
Name is the metadata.name of the referenced ConfigMap. This field is required in all cases.
configSource.configMap.namespace
Namespace is the metadata.namespace of the referenced ConfigMap. This field is required in all cases.
configSource.configMap.resourceVersion
(string)
ResourceVersion is the metadata.ResourceVersion of the referenced ConfigMap. This field is forbidden in Node.Spec, and required in Node.Status.
configSource.configMap.uid
(string)
UID is the metadata.UID of the referenced ConfigMap. This field is forbidden in Node.Spec, and required in Node.Status.
externalID
(string)
Deprecated. Not all kubelets will set this field. Remove field after 1.13. see:
https://issues.k8s.io/61966
podCIDR
(string)
PodCIDR represents the pod IP range assigned to the node.
podCIDRs
podCIDRs represents the IP ranges assigned to the node for usage by Pods on that node. If this field is specified, the 0th entry must match the podCIDR field. It may contain at most 1 value for each of IPv4 and IPv6.
providerID
(string)
ID of the node assigned by the cloud provider in the format: &lt;ProviderName&gt;://&lt;ProviderSpecificNodeID&gt;
taints
([]Taint)
If specified, the node's taints.
The node this Taint is attached to has the "effect" on any pod that does not tolerate the Taint.
taints.effect
Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute.
taints.key
Required. The taint key to be applied to a node.
taints.timeAdded
(Time)
TimeAdded represents the time at which the taint was added. It is only written for NoExecute taints.
Time is a wrapper ar

ound time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.
taints.value
(string)
The taint value corresponding to the taint key.
unschedulable
(boolean)
Unschedulable controls node schedulability of new pods. By default, node is schedulable. More info:
https://kubernetes.io/docs/concepts/nodes/node/#manual-node-administration
NodeStatus is information about the current status of a node.
addresses
([]NodeAddress)
type
List of addresses reachable to the node. Queried from cloud provider, if available. More info:
https://kubernetes.io/docs/concepts/nodes/node/#addresses
Note: This field is declared as mergeable, but the merge key is not sufficiently unique, which can cause data corruption when it is merged. Callers should instead use a full-replacement patch. See
https://pr.k8s.io/79391
for an example.
NodeAddress contains information for the node's address.
addresses.address
The node address.
addresses.type
Node address type, one of Hostname, ExternalIP or InternalIP.
Quantity
)
Allocatable represents the resources of a node that are available for scheduling. Defaults to Capacity.
capacity
Quantity
)
Capacity represents the total resources of a node. More info:
([]NodeCondition)
type
Conditions is an array of current observed node conditions. More info:
https://kubernetes.io/docs/concepts/nodes/node/#condition
NodeCondition contains condition information for a node.
Type of node condition.
conditions.lastHeartbeatTime
(Time)
Last time we got an update on a given condition.
(Time)
(string)
(string)
config
(NodeConfigStatus)
Status of the config assigned to the node via the dynamic Kubelet config feature.
NodeConfigStatus describes the status of the config assigned by Node.Spec.ConfigSource.
config.active
Active reports the checkpointed config the node is actively using. Active will represent either the current version of the Assigned config, or the current LastKnownGood config, depending on whether attempting to use the Assigned config results in an error.
config.active.configMap
config.active.configMap.kubeletConfigKey
config.active.configMap.name
config.active.configMap.namespace
Namespace is the metadata.namespace of the referenced ConfigMap. This field is req

uired in all cases.
config.active.configMap.resourceVersion
(string)
config.active.configMap.uid
(string)
config.assigned
Assigned reports the checkpointed config the node will try to use. When Node.Spec.ConfigSource is updated, the node checkpoints the associated config payload to local disk, along with a record indicating intended config. The node refers to this record to choose its config checkpoint, and reports this record in Assigned. Assigned only updates in the status after the record has been checkpointed to disk. When the Kubelet is restarted, it tries to make the Assigned config the Active config by loading and validating the checkpointed payload identified by Assigned.
config.assigned.configMap
config.assigned.configMap.kubeletConfigKey
config.assigned.configMap.name
config.assigned.configMap.namespace
config.assigned.configMap.resourceVersion
(string)
config.assigned.configMap.uid
(string)
config.error
(string)
Error describes any problems reconciling the Spec.ConfigSource to the Active config. Errors may occur, for example, attempting to checkpoint Spec.ConfigSource to the local Assigned record, attempting to checkpoint the payload associated with Spec.ConfigSource, attempting to load or validate the Assigned config, etc. Errors may occur at different points while syncing config. Earlier errors (e.g. download or checkpointing errors) will not result in a rollback to LastKnownGood, and may resolve across Kubelet retries. Later errors (e.g. loading or validating a checkpointed config) will result in a rollback to LastKnownGood. In the latter case, it is usually possible to resolve the error by fixing the config assigned in Spec.ConfigSource. You can find additional information for debugging by searching the error message in the Kubelet log. Error is a human-readable description of the error state; machines can check whether or not Error is empty, but should not rely on the stability of the Error text across Kubelet versions.
config.lastKnownGood
LastKnownGood reports the checkpointed config the node will fall back to when it encounters an error attempting to use the Assigned config. The Assigned config becomes the LastKnownGood config when the node determines that the Assigned config is stable and correct. This is currently implemented as a 10-minute soak period starting when the local record of Assigned config is updated. If the Assigned config is Active at the end of this period, it becomes the LastKnownGood. Note that if Spec.ConfigSource is reset to nil (use local defaults), the LastKnownGood is also immediately reset to

 nil, because the local default config is always assumed good. You should not make assumptions about the node's method of determining config stability and correctness, as this may change or become configurable in the future.
config.lastKnownGood.configMap
config.lastKnownGood.configMap.kubeletConfigKey
config.lastKnownGood.configMap.name
config.lastKnownGood.configMap.namespace
config.lastKnownGood.configMap.resourceVersion
(string)
config.lastKnownGood.configMap.uid
(string)
(NodeDaemonEndpoints)
Endpoints of daemons running on the Node.
NodeDaemonEndpoints lists ports opened by daemons running on the Node.
daemonEndpoints.kubeletEndpoint
(DaemonEndpoint)
Endpoint on which Kubelet is listening.
DaemonEndpoint contains information about a single Daemon endpoint.
daemonEndpoints.kubeletEndpoint.Port
Port number of the given endpoint.
images
([]ContainerImage)
List of container images on this node
Describe a container image
images.names
Names by which this image is known. e.g. ["kubernetes.example/hyperkube:v1.0.7", "cloud-vendor.registry.example/cloud-vendor/hyperkube:v1.0.7"]
images.sizeBytes
(int64)
The size of the image in bytes.
nodeInfo
(NodeSystemInfo)
Set of ids/uuids to uniquely identify the node. More info:
https://kubernetes.io/docs/concepts/nodes/node/#info
NodeSystemInfo is a set of ids/uuids to uniquely identify the node.
nodeInfo.architecture
The Architecture reported by the node
nodeInfo.bootID
Boot ID reported by the node.
nodeInfo.containerRuntimeVersion
ContainerRuntime Version reported by the node through runtime remote API (e.g. containerd://1.4.2).
nodeInfo.kernelVersion
Kernel Version reported by the node from 'uname -r' (e.g. 3.16.0-0.bpo.4-amd64).
nodeInfo.kubeProxyVersion
KubeProxy Version reported by the node.
nodeInfo.kubeletVersion
Kubelet Version reported by the node.
nodeInfo.machineID
MachineID reported by the node. For unique machine identification in the cluster this field is preferred. Learn more from man(5) machine-id:
http://man7.org/linux/man-pages/man5/machine-id.5.html
nodeInfo.operatingSystem
The Operating System reported by the node
nodeInfo.osImage
OS Image reported by the node from /etc/os-release (e.g. Debian GNU/Linux 7 (wheezy)).
nodeInfo.systemUUID
SystemUUID reported by the node. For unique machine identification MachineID is preferred. This field is specific to Red Hat hosts
https://access.redhat.com/documentation/en-us/red_hat_subscription_management/1/html/rhsm/uuid
phase
(string)
NodePhase is the recently observed lifecycle phase of the node. More info:
https://kubernetes.io/docs/concepts/nodes/node/#phase
The field is never populated, and now is depr

ecated.
volumesAttached
([]AttachedVolume)
List of volumes that are attached to the node.
AttachedVolume describes a volume attached to a node
volumesAttached.devicePath
DevicePath represents the device path where the volume should be available
volumesAttached.name
Name of the attached volume
volumesInUse
List of attachable volumes in use (mounted) by the node.
NodeList
NodeList is the whole list of all Nodes which have been registered with master.
: v1
kind
: NodeList
metadata
(
ListMeta
)
items
([]
Node
List of nodes
get
read the specified Node
GET /api/v1/nodes/{name}
name
(
in path
name of the Node
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
get
read status of the specified Node
GET /api/v1/nodes/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
list
list or watch objects of kind Node
GET /api/v1/nodes
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
NodeList
): OK
create
create a Node
POST /api/v1/nodes
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
202 (
Node
update
replace the specified Node
PUT /api/v1/nodes/{name}
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
update
replace status of the specified Node
PUT /api/v1/nodes/{name}/status
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
partially update the specified Node
PATCH /api/v1/nodes/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
partially update status of the specified Node
PATCH /api/v1/nodes/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
delete
delete a Node
DELETE /api/v1/nodes/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
g

racePeriodSeconds
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Node
DELETE /api/v1/nodes
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8.2 - Namespace
Namespace
Namespace provides a scope for Names. Use of multiple namespaces is optional.
: v1
kind
: Namespace
metadata
(
)
spec
(
NamespaceSpec
)
Spec defines the behavior of the Namespace. More info:
status
(
NamespaceStatus
)
Status describes the current status of a Namespace. More info:
NamespaceSpec describes the attributes on a Namespace.
Finalizers is an opaque list of values that must be empty to permanently remove object from storage. More info:
https://kubernetes.io/docs/tasks/administer-cluster/namespaces/
NamespaceStatus is information about the current status of a Namespace.
([]NamespaceCondition)
type
Represents the latest available observations of a namespace's current state.
NamespaceCondition contains details about state of namespace.
Type of namespace controller condition.
(Time)
(string)
(string)
phase
(string)
Phase is the current lifecycle phase of the namespace. More info:
NamespaceList
NamespaceList is a list of Namespaces.
: v1
kind
: NamespaceList
metadata
(
ListMeta
)
items
([]
Namespace
Items is the list of Namespace objects in the list. More info:
https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
get
read the specified Namespace
GET /api/v1/namespaces/{name}
name
(
in path
name of the Namespace
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
get
read status of the specified Namespace
GET /api/v1/namespaces/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
list
list or watch objects of kind Namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): intege

r
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a Namespace
POST /api/v1/namespaces
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
202 (
Namespace
update
replace the specified Namespace
PUT /api/v1/namespaces/{name}
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
replace finalize of the specified Namespace
PUT /api/v1/namespaces/{name}/finalize
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
replace status of the specified Namespace
PUT /api/v1/namespaces/{name}/status
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
partially update the specified Namespace
PATCH /api/v1/namespaces/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
partially update status of the specified Namespace
PATCH /api/v1/namespaces/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
delete
delete a Namespace
DELETE /api/v1/namespaces/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
5.8.3 - Event
apiVersion: events.k8s.io/v1
import "k8s.io/api/events/v1"
Event
Event is a report of an event somewhere in the cluster. It generally denotes some state change in the system. Events have a limited retention time and triggers and messages may evolve with time. Event consumers should not rely on the timing of an event with a given Reason reflecting a consistent underlying trigger, or the continued existence of events with that Reason. Events should be treated as informative, best-effort, supplemental data.
: events.k8s.io/v1
kind
: Event
metadata
(
)
Stand

ard object's metadata. More info:
eventTime
(MicroTime), required
eventTime is the time when this Event was first observed. It is required.
action
(string)
action is what action was taken/failed regarding to the regarding object. It is machine-readable. This field cannot be empty for new Events and it can have at most 128 characters.
(int32)
deprecatedCount is the deprecated field assuring backward compatibility with core.v1 Event type.
(Time)
deprecatedFirstTimestamp is the deprecated field assuring backward compatibility with core.v1 Event type.
(Time)
deprecatedLastTimestamp is the deprecated field assuring backward compatibility with core.v1 Event type.
deprecatedSource
(EventSource)
deprecatedSource is the deprecated field assuring backward compatibility with core.v1 Event type.
EventSource contains information for an event.
(string)
Component from which the event is generated.
(string)
Node name on which the event is generated.
note
(string)
note is a human-readable description of the status of this operation. Maximal length of the note is 1kB, but libraries should be prepared to handle values up to 64kB.
reason
(string)
reason is why the action was taken. It is human-readable. This field cannot be empty for new Events and it can have at most 128 characters.
regarding
(
)
regarding contains the object this Event is about. In most cases it's an Object reporting controller implements, e.g. ReplicaSetController implements ReplicaSets and this event is emitted because it acts on some changes in a ReplicaSet object.
related
(
)
related is the optional secondary object for more complex actions. E.g. when regarding object triggers a creation or deletion of related object.
(string)
reportingController is the name of the controller that emitted this Event, e.g.
kubernetes.io/kubelet
. This field cannot be empty for new Events.
(string)
reportingInstance is the ID of the controller instance, e.g.
kubelet-xyzf
. This field cannot be empty for new Events and it can have at most 128 characters.
series
(EventSeries)
series is data about the Event series this event represents or nil if it's a singleton Event.
EventSeries contain information on series of events, i.e. thing that was/is happening continuously for some time. How often to update the EventSeries is up to the event reporters. The default event reporter in "k8s.io/client-go/tools/events/event_broadcaster.go" shows how this struct is updated on heartbeats and can guide customized reporter implementations.
count is the number of occurrences in this series up to the last heartbeat time.
lastObservedTime is the time when last Event from the series was seen before last heartbeat.
type
(string)
type is the type of this event (Normal, Warning), new types could be added in the future. It is machine-readable. This field cannot be empty for new Events.
EventList
EventList is a list of Event objects.
kind
: EventList
metadata
(
ListMeta
)
items
([]
Event
items is a list of schema objects.
get
read the sp

ecified Event
GET /apis/events.k8s.io/v1/namespaces/{namespace}/events/{name}
name
(
in path
name of the Event
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
list
list or watch objects of kind Event
GET /apis/events.k8s.io/v1/namespaces/{namespace}/events
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
list
GET /apis/events.k8s.io/v1/events
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
create
create an Event
POST /apis/events.k8s.io/v1/namespaces/{namespace}/events
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
202 (
Event
update
replace the specified Event
PUT /apis/events.k8s.io/v1/namespaces/{namespace}/events/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
patch
partially update the specified Event
PATCH /apis/events.k8s.io/v1/namespaces/{namespace}/events/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
delete
delete an Event
DELETE /apis/events.k8s.io/v1/namespaces/{namespace}/events/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Event
DELETE /apis/events.k8s.io/v1/namespaces/{namespace}/events
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
)

: integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8.4 - APIService
kind
metadata
(
)
spec
(
)
status
(
)
caBundle
([]byte)
CABundle is a PEM encoded CA bundle which will be used to validate an API server's serving certificate. If unspecified, system trust roots on the apiserver are used.
group
(string)
(boolean)
service
(string)
(string)
(int32)
port
vers

ion
(string)
type
(Time)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/apiregistration.k8s.io/v1/apise

rvices/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8.5 - Lease
Lease
kind
: Lease
metadata
(
)
More info:
spec
(
LeaseSpec
)
LeaseSpec
(string)
(int32)
(int32)
renewTime
LeaseList
kind
metadata
(
ListMeta
)
items
([]
Lease
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
list
GET /apis/coordination

.k8s.io/v1/namespaces/{namespace}/leases
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
create
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
202 (
Lease
update
name
(
in path
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK

5.8.6 - RuntimeClass
apiVersion: node.k8s.io/v1
import "k8s.io/api/node/v1"
RuntimeClass defines a class of container runtime supported in the cluster. The RuntimeClass is used to determine which container runtime is used to run all containers in a pod. RuntimeClasses are manually defined by a user or cluster provisioner, and referenced in the PodSpec. The Kubelet is responsible for resolving the RuntimeClassName reference before running the pod. For more details, see
https://kubernetes.io/docs/concepts/containers/runtime-class/
: node.k8s.io/v1
kind
: RuntimeClass
metadata
(
)
More info:
handler
Handler specifies the underlying runtime and configuration that the CRI implementation will use to handle pods of this class. The possible values are specific to the node &amp; CRI configuration. It is assumed that all handlers are available on every node, and handlers of the same name are equivalent on every node. For example, a handler called "runc" might specify that the runc OCI runtime (using native Linux containers) will be used to run the containers in a pod. The Handler must be lowercase, conform to the DNS Label (RFC 1123) requirements, and is immutable.
overhead
(Overhead)
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see
https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/
Overhead structure represents the resource overhead associated with running a pod.
overhead.podFixed
Quantity
)
PodFixed represents the fixed resource overhead associated with running a pod.
scheduling
(Scheduling)
Scheduling holds the scheduling constraints to ensure that pods running with this RuntimeClass are scheduled to nodes that support it. If scheduling is nil, this RuntimeClass is assumed to be supported by all nodes.
Scheduling specifies the scheduling constraints for nodes supporting a RuntimeClass.
scheduling.nodeSelector
nodeSelector lists labels that must be present on nodes that support this RuntimeClass. Pods using this RuntimeClass can only be scheduled to a node matched by this selector. The RuntimeClass nodeSelector is merged with a pod's existing nodeSelector. Any conflicts will cause the pod to be rejected in admission.
scheduling.tolerations
tolerations are appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass.
.
scheduling.tolerations.key
(string)
scheduling.tolerations.operator
(string)
scheduling.tolerations.value
(string)
scheduling.tolerations.effect
(string)
scheduling.tolerations.tolerationSeconds
(int64)
TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means 

tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
RuntimeClassList
RuntimeClassList is a list of RuntimeClass objects.
kind
: RuntimeClassList
metadata
(
ListMeta
)
items
([]
get
read the specified RuntimeClass
GET /apis/node.k8s.io/v1/runtimeclasses/{name}
name
(
in path
name of the RuntimeClass
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind RuntimeClass
GET /apis/node.k8s.io/v1/runtimeclasses
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a RuntimeClass
POST /apis/node.k8s.io/v1/runtimeclasses
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified RuntimeClass
PUT /apis/node.k8s.io/v1/runtimeclasses/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified RuntimeClass
PATCH /apis/node.k8s.io/v1/runtimeclasses/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a RuntimeClass
DELETE /apis/node.k8s.io/v1/runtimeclasses/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of RuntimeClass
DELETE /apis/node.k8s.io/v1/runtimeclasses
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8.7 - FlowSchema v1beta3
apiVersion: flowcontrol.apiserver.k8s.io/v1b

eta3
import "k8s.io/api/flowcontrol/v1beta3"
FlowSchema defines the schema of a group of flows. Note that a flow is made up of a set of inbound API requests with similar attributes and is identified by a pair of strings: the name of the FlowSchema and a "flow distinguisher".
: flowcontrol.apiserver.k8s.io/v1beta3
kind
: FlowSchema
metadata
(
)
metadata
is the standard object's metadata. More info:
spec
(
FlowSchemaSpec
)
spec
is the specification of the desired behavior of a FlowSchema. More info:
status
(
FlowSchemaStatus
)
status
is the current status of a FlowSchema. More info:
FlowSchemaSpec describes how the FlowSchema's specification looks like.
(PriorityLevelConfigurationReference), required
should reference a PriorityLevelConfiguration in the cluster. If the reference cannot be resolved, the FlowSchema will be ignored and marked as invalid in its status. Required.
PriorityLevelConfigurationReference contains information that points to the "request-priority" being used.
priorityLevelConfiguration.name
name
is the name of the priority level configuration being referenced Required.
(FlowDistinguisherMethod)
defines how to compute the flow distinguisher for requests that match this schema.
nil
specifies that the distinguisher is disabled and thus will always be the empty string.
FlowDistinguisherMethod specifies the method of a flow distinguisher.
type
is the type of flow distinguisher method The supported types are "ByUser" and "ByNamespace". Required.
(int32)
is used to choose among the FlowSchemas that match a given request. The chosen FlowSchema is among those with the numerically lowest (which we take to be logically highest) MatchingPrecedence. Each MatchingPrecedence value must be ranged in [1,10000]. Note that if the precedence is not specified, it will be set to 1000 as default.
rules
([]PolicyRulesWithSubjects)
rules
describes which requests will match this flow schema. This FlowSchema matches a request if and only if at least one member of rules matches the request. if it is an empty slice, there will be no requests matching the FlowSchema.
PolicyRulesWithSubjects prescribes a test that applies to a request to an apiserver. The test considers the subject making the request, the verb being requested, and the resource to be acted upon. This PolicyRulesWithSubjects matches a request if and only if both (a) at least one member of subjects matches the request and (b) at least one member of resourceRules or nonResourceRules matches the request.
rules.subjects
([]Subject), required
subjects is the list of normal user, serviceaccount, or group that this rule cares about. There must be at least one member in this slice. A slice that includes both the system:authenticated and system:unauthenticated user groups matches every request. Required.
Subject matches the originator of a request, as identified by the request authentication system. There are three ways of matching an originator; by user, group, or service account.
rules.subjects.kind
kind
indicates which one of the other fields is non-empty. Required
rules.subjects.group
(GroupSubject)
group
matches based on user group name.
GroupSubject holds detailed information for group-kind subject.
rules.subjects.group.name
name is the user group that matches, or "*" to match all user groups. See
https://github.com/kubernetes/apiserver

/blob/master/pkg/authentication/user/user.go
for some well-known group names. Required.
rules.subjects.serviceAccount
(ServiceAccountSubject)
matches ServiceAccounts.
ServiceAccountSubject holds detailed information for service-account-kind subject.
rules.subjects.serviceAccount.name
name
is the name of matching ServiceAccount objects, or "*" to match regardless of name. Required.
rules.subjects.serviceAccount.namespace
namespace
is the namespace of matching ServiceAccount objects. Required.
rules.subjects.user
(UserSubject)
user
matches based on username.
UserSubject holds detailed information for user-kind subject.
rules.subjects.user.name
name
is the username that matches, or "*" to match all usernames. Required.
rules.nonResourceRules
([]NonResourcePolicyRule)
is a list of NonResourcePolicyRules that identify matching requests according to their verb and the target non-resource URL.
NonResourcePolicyRule is a predicate that matches non-resource requests according to their verb and the target non-resource URL. A NonResourcePolicyRule matches a request if and only if both (a) at least one member of verbs matches the request and (b) at least one member of nonResourceURLs matches the request.
rules.nonResourceRules.nonResourceURLs
is a set of url prefixes that a user should have access to and may not be empty. For example:
"/healthz" is legal
"/hea*" is illegal
"/hea" is legal but matches nothing
"/hea/*" also matches nothing
"/healthz/
" matches all per-component health checks.
"
" matches all non-resource urls. if it is present, it must be the only entry. Required.
rules.nonResourceRules.verbs
verbs
is a list of matching verbs and may not be empty. "*" matches all verbs. If it is present, it must be the only entry. Required.
rules.resourceRules
([]ResourcePolicyRule)
is a slice of ResourcePolicyRules that identify matching requests according to their verb and the target resource. At least one of
and
has to be non-empty.
ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one member of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) either (d1) the request does not specify a namespace (i.e.,
Namespace==""
) and clusterScope is true or (d2) the request specifies a namespace and least one member of namespaces matches the request's namespace.
rules.resourceRules.apiGroups
apiGroups
is a list of matching API groups and may not be empty. "*" matches all API groups and, if present, must be the only entry. Required.
rules.resourceRules.resources
resources
is a list of matching resources (i.e., lowercase and plural) with, if desired, subresource. For example, [ "services", "nodes/status" ]. This list may not be empty. "*" matches all resources and, if present, must be the only entry. Required.
rules.resourceRules.verbs
verbs
is a list of matching verbs and may not be empty. "*" matches all verbs and, if present, must be the only entry. Required.
rules.resourceRules.clusterScope
(boolean)
clusterScope
indicates whether to match requests that do not specify a namespace (which happens either because the resource is not namespaced or the request targets all namespaces). If this field is omitted or false then the
field mus

t contain a non-empty list.
rules.resourceRules.namespaces
is a list of target namespaces that restricts matches. A request that specifies a target namespace matches only if either (a) this list contains that target namespace or (b) this list contains "
". Note that "
" matches any specified namespace but does not match a request that
does not specify
a namespace (see the
field for that). This list may be empty, but only if
is true.
FlowSchemaStatus represents the current state of a FlowSchema.
([]FlowSchemaCondition)
type
is a list of the current states of FlowSchema.
FlowSchemaCondition describes conditions for a FlowSchema.
(Time)
is the last time the condition transitioned from one status to another.
(string)
message
is a human-readable message indicating details about last transition.
(string)
reason
is a unique, one-word, CamelCase reason for the condition's last transition.
(string)
status
is the status of the condition. Can be True, False, Unknown. Required.
(string)
type
is the type of the condition. Required.
FlowSchemaList
FlowSchemaList is a list of FlowSchema objects.
kind
: FlowSchemaList
metadata
(
ListMeta
)
metadata
is the standard list metadata. More info:
items
([]
items
is a list of FlowSchemas.
get
read the specified FlowSchema
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas/{name}
name
(
in path
name of the FlowSchema
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified FlowSchema
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind FlowSchema
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a FlowSchema
POST /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified FlowSchema
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
401: Un

update
replace status of the specified FlowSchema
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas/{name}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified FlowSchema
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified FlowSchema
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a FlowSchema
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of FlowSchema
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta3/flowschemas
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8.8 - PriorityLevelConfiguration v1beta3
apiVersion: flowcontrol.apiserver.k8s.io/v1beta3
kind
: PriorityLevelConfiguration
metadata
(
)
metadata
spec
(
PriorityLevelConfigurationSpec
)
spec
is the specification of the desired behavior of a "request-priority". More info:
status
(
PriorityLevelConfigurationStatus
)
status
is the current status of a "request-priority". More info:
PriorityLevelConfigurationSpec specifies the configuration of a priority level.
type
type
indicates whether this priority level is subject to limitation on request execution. A value of
"Exempt"
means that requests of this priority level are n

ot subject to a limit (and thus are never queued) and do not detract from the capacity made available to other priority levels. A value of
"Limited"
means that (a) requests of this priority level
are
subject to limits and (b) some of the server's limited capacity is made available exclusively to this priority level. Required.
limited
(LimitedPriorityLevelConfiguration)
limited
specifies how requests are handled for a Limited priority level. This field must be non-empty if and only if
type
is
"Limited"
.
*LimitedPriorityLevelConfiguration specifies how to handle requests that are subject to limits. It addresses two issues:
How are requests for this priority level limited?
What should be done with requests that exceed the limit?*
limited.borrowingLimitPercent
(int32)
borrowingLimitPercent
, if present, configures a limit on how many seats this priority level can borrow from other priority levels. The limit is known as this level's BorrowingConcurrencyLimit (BorrowingCL) and is a limit on the total number of seats that this level may borrow at any one time. This field holds the ratio of that limit to the level's nominal concurrency limit. When this field is non-nil, it must hold a non-negative integer and the limit is calculated as follows.
BorrowingCL(i) = round( NominalCL(i) * borrowingLimitPercent(i)/100.0 )
The value of this field can be more than 100, implying that this priority level can borrow a number of seats that is greater than its own nominal concurrency limit (NominalCL). When this field is left
nil
, the limit is effectively infinite.
limited.lendablePercent
(int32)
lendablePercent
prescribes the fraction of the level's NominalCL that can be borrowed by other priority levels. The value of this field must be between 0 and 100, inclusive, and it defaults to 0. The number of seats that other levels can borrow from this level, known as this level's LendableConcurrencyLimit (LendableCL), is defined as follows.
LendableCL(i) = round( NominalCL(i) * lendablePercent(i)/100.0 )
limited.limitResponse
(LimitResponse)
limitResponse
indicates what to do with requests that can not be executed right now
LimitResponse defines how to handle requests that can not be executed right now.
limited.limitResponse.type
type
is "Queue" or "Reject". "Queue" means that requests that can not be executed upon arrival are held in a queue until they can be executed or a queuing limit is reached. "Reject" means that requests that can not be executed upon arrival are rejected. Required.
limited.limitResponse.queuing
(QueuingConfiguration)
queuing
holds the configuration parameters for queuing. This field may be non-empty only if
type
is
"Queue"
.
QueuingConfiguration holds the configuration parameters for queuing
limited.limitResponse.queuing.handSize
(int32)
handSize
is a small positive number that configures the shuffle sharding of requests into queues. When enqueuing a request at this priority level the request's flow identifier (a string pair) is hashed and the hash value is used to shuffle the list of queues and deal a hand of the size specified here. The request is put into one of the shortest queues in that hand.
handSize
must be no larger than
queues
, and should be significantly smaller (so that a few heavy flows do not saturate most of the queues). See the user-facing documentation for more extensive guidance on setting this field. This field has a default value of 8.
limited.limitResponse.queuing.queueLengthLimit
(int32)
is the maximum number of requests allowed to be waiting in a given queue of this priority level at a time; excess requests are rejected. This value must be positive. If not specified, it will be defaulted to 50.
limited.limitResponse.queuing.queues
(int32)
queues
is the number of queues for this priority level. The queues exist independently at each apiserver. The value must be positive. Setting it to 1 effectively precludes shufflesharding and thus makes the distinguisher method of ass

ociated flow schemas irrelevant. This field has a default value of 64.
limited.nominalConcurrencyShares
(int32)
nominalConcurrencyShares
(NCS) contributes to the computation of the NominalConcurrencyLimit (NominalCL) of this level. This is the number of execution seats available at this priority level. This is used both for requests dispatched from this priority level as well as requests dispatched from other priority levels borrowing seats from this level. The server's concurrency limit (ServerCL) is divided among the Limited priority levels in proportion to their NCS values:
NominalCL(i) = ceil( ServerCL * NCS(i) / sum_ncs ) sum_ncs = sum[limited priority level k] NCS(k)
Bigger numbers mean a larger nominal concurrency limit, at the expense of every other Limited priority level. This field has a default value of 30.
PriorityLevelConfigurationStatus represents the current state of a "request-priority".
([]PriorityLevelConfigurationCondition)
type
is the current state of "request-priority".
PriorityLevelConfigurationCondition defines the condition of priority level.
(Time)
(string)
message
(string)
reason
(string)
status
(string)
type
PriorityLevelConfigurationList
PriorityLevelConfigurationList is a list of PriorityLevelConfiguration objects.
kind
: PriorityLevelConfigurationList
metadata
(
ListMeta
)
metadata
items
([]
items
is a list of request-priorities.
get
read the specified PriorityLevelConfiguration
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations/{name}
name
(
in path
name of the PriorityLevelConfiguration
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the specified PriorityLevelConfiguration
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind PriorityLevelConfiguration
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a PriorityLevelConfiguration
POST /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations
body
:
, required
dryRun
(
in query
): string
dryRun

(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified PriorityLevelConfiguration
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
replace status of the specified PriorityLevelConfiguration
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations/{name}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified PriorityLevelConfiguration
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update status of the specified PriorityLevelConfiguration
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations/{name}/status
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a PriorityLevelConfiguration
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of PriorityLevelConfiguration
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8.9 - Binding
import "k8s.

io/api/core/v1"
Binding
Binding ties one object to another; for example, a pod is bound to a node by a scheduler. Deprecated in 1.7, please use the bindings subresource of pods instead.
: v1
kind
: Binding
metadata
(
)
target
(
The target object that you want to bind to the standard object.
create
create a Binding
POST /api/v1/namespaces/{namespace}/bindings
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
create
create binding of a Pod
POST /api/v1/namespaces/{namespace}/pods/{name}/binding
name
(
in path
name of the Binding
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
5.8.10 - ComponentStatus
ComponentStatus (and ComponentStatusList) holds the cluster validation info. Deprecated: This API is deprecated in v1.19+
: v1
kind
: ComponentStatus
metadata
(
)
([]ComponentCondition)
type
List of component conditions observed
Information about the condition of a component.
Status of the condition for a component. Valid values for "Healthy": "True", "False", or "Unknown".
Type of condition for a component. Valid value: "Healthy"
conditions.error
(string)
Condition error code for a component. For example, a health check error code.
(string)
Message about the condition for a component. For example, information about a health check.
ComponentStatusList
Status of all the conditions for the component as a list of ComponentStatus objects. Deprecated: This API is deprecated in v1.19+
: v1
kind
: ComponentStatusList
metadata
(
ListMeta
)
items
([]
List of ComponentStatus objects.
get
read the specified ComponentStatus
GET /api/v1/componentstatuses/{name}
name
(
in path
name of the ComponentStatus
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list objects of kind ComponentStatus
GET /api/v1/componentstatuses
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
5.8.11 - ClusterCIDR v1alpha1
ClusterCIDR represents a single configuration for per-Node Pod CIDR allocations when the MultiCIDRRangeAllocator is enabled (see the config for ku

be-controller-manager).
apiVersion: networking.k8s.io/v1alpha1
import "k8s.io/api/networking/v1alpha1"
ClusterCIDR
ClusterCIDR represents a single configuration for per-Node Pod CIDR allocations when the MultiCIDRRangeAllocator is enabled (see the config for kube-controller-manager). A cluster may have any number of ClusterCIDR resources, all of which will be considered when allocating a CIDR for a Node. A ClusterCIDR is eligible to be used for a given Node when the node selector matches the node in question and has free CIDRs to allocate. In case of multiple matching ClusterCIDR resources, the allocator will attempt to break ties using internal heuristics, but any ClusterCIDR whose node selector matches the Node may be used.
: networking.k8s.io/v1alpha1
kind
: ClusterCIDR
metadata
(
)
spec
(
ClusterCIDRSpec
)
Spec is the desired state of the ClusterCIDR. More info:
ClusterCIDRSpec defines the desired state of ClusterCIDR.
perNodeHostBits
PerNodeHostBits defines the number of host bits to be configured per node. A subnet mask determines how much of the address is used for network bits and host bits. For example an IPv4 address of 192.168.0.0/24, splits the address into 24 bits for the network portion and 8 bits for the host portion. To allocate 256 IPs, set this field to 8 (a /24 mask for IPv4 or a /120 for IPv6). Minimum value is 4 (16 IPs). This field is immutable.
ipv4
(string)
IPv4 defines an IPv4 IP block in CIDR notation(e.g. "10.0.0.0/8"). At least one of IPv4 and IPv6 must be specified. This field is immutable.
ipv6
(string)
IPv6 defines an IPv6 IP block in CIDR notation(e.g. "2001:db8::/64"). At least one of IPv4 and IPv6 must be specified. This field is immutable.
NodeSelector defines which nodes the config is applicable to. An empty or nil NodeSelector selects all nodes. This field is immutable.
nodeSelector.nodeSelectorTerms
nodeSelector.nodeSelectorTerms.matchExpressions
([]
)
nodeSelector.nodeSelectorTerms.matchFields
([]
)
ClusterCIDRList
ClusterCIDRList contains a list of ClusterCIDR.
kind
: ClusterCIDRList
metadata
(
ListMeta
)
items
([]
Items is the list of ClusterCIDRs.
get
read the specified ClusterCIDR
GET /apis/networking.k8s.io/v1alpha1/clustercidrs/{name}
name
(
in path
name of the ClusterCIDR
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
list or watch objects of kind ClusterCIDR
GET /apis/networking.k8s.io/v1alpha1/clustercidrs
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
r

esourceVersionMatch
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ClusterCIDR
POST /apis/networking.k8s.io/v1alpha1/clustercidrs
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
replace the specified ClusterCIDR
PUT /apis/networking.k8s.io/v1alpha1/clustercidrs/{name}
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified ClusterCIDR
PATCH /apis/networking.k8s.io/v1alpha1/clustercidrs/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a ClusterCIDR
DELETE /apis/networking.k8s.io/v1alpha1/clustercidrs/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of ClusterCIDR
DELETE /apis/networking.k8s.io/v1alpha1/clustercidrs
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.9 - Common Definitions
5.9.1 - DeleteOptions
DeleteOptions may be provided when deleting an API object.
(string)
dryRun
(int64)
kind
(string)
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be 

updated. In CamelCase. More info:
orphanDependents
(boolean)
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
preconditions
(Preconditions)
Must be fulfilled before a deletion is carried out. If not possible, a 409 Conflict status will be returned.
Preconditions must be fulfilled before an operation (update, delete, etc.) is carried out.
preconditions.resourceVersion
(string)
Specifies the target ResourceVersion
preconditions.uid
(string)
Specifies the target UID.
(string)
5.9.2 - LabelSelector
key
5.9.3 - ListMeta
ListMeta describes metadata that synthetic resources must have, including lists and various status objects.
ListMeta describes metadata that synthetic resources must have, including lists and various status objects. A resource may have only one of {ObjectMeta, ListMeta}.
continue
(string)
continue may be set if the user set a limit on the number of items returned, and indicates that the server has more data available. The value is opaque and may be used to issue another request to the endpoint that served this list to retrieve the next set of available objects. Continuing a consistent list may not be possible if the server configuration has changed or more than a few minutes have passed. The resourceVersion field returned when using this continue value will be identical to the value in the first response, unless you have received this token from an error message.
(int64)
remainingItemCount is the number of subsequent items in the list which are not included in this list response. If the list request contained label or field selectors, then the number of remaining items is unknown and the field will be left unset and omitted during serialization. If the list is comple

te (either because it is not chunking or because this is the last chunk), then there are no more remaining items and this field will be left unset and omitted during serialization. Servers older than v1.15 do not set this field. The intended use of the remainingItemCount is
estimating
the size of a collection. Clients should not rely on the remainingItemCount to be set or to be exact.
(string)
String that identifies the server's internal version of this object that can be used by clients to determine when objects have changed. Value must be treated as opaque by clients and passed unmodified back to the server. Populated by the system. Read-only. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
selfLink
(string)
Deprecated: selfLink is a legacy read-only field that is no longer populated by the system.
5.9.4 - LocalObjectReference
LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
name
(string)
5.9.5 - NodeSelectorRequirement
A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
key
operator
Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt.
values
An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch.
5.9.6 - ObjectFieldSelector
fieldPath
(string)
5.9.7 - ObjectMeta
ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create.
name
(string)
Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info:
(string)
GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.
If this field is specified and the generated name exists, the server will return a 409.
Applied only if Name is not specified. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency
namespace
(string)
Namespace defines the space within which each name must be unique. A

n empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.
Must be a DNS_LABEL. Cannot be updated. More info:
http://kubernetes.io/docs/user-guide/namespaces
labels
Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info:
http://kubernetes.io/docs/user-guide/labels
Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info:
http://kubernetes.io/docs/user-guide/annotations
System
Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list.
([]ManagedFieldsEntry)
ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object.
ManagedFieldsEntry is a workflow-id, a FieldSet and the group version of the resource that the fieldset applies to.
managedFields.apiVersion
(string)
APIVersion defines the version of this resource that this field set applies to. The format is "group/version" just like the top-level APIVersion field. It is necessary to track the version of a field set because it cannot be automatically converted.
managedFields.fieldsType
(string)
FieldsType is the discriminator for the different fields format and version. There is currently only one possible value: "FieldsV1"
managedFields.fieldsV1
(FieldsV1)
FieldsV1 holds the first JSON version format as described in the "FieldsV1" type.
*FieldsV1 stores a set of fields in a data structure like a Trie, in JSON format.
Each key is either a '.' representing the field itself, and will always map to an empty set, or a string representing a sub-field or item. The string will follow one of these four formats: 'f:
', where
is the name of a field in a struct, or key in a map 'v:
', where
is the exact json formatted value of a list item 'i:
', where
is position of a item in a list 'k:
', where
is a map of a list item's key fields to their unique values If a key maps to an empty Fields value, the field that key represents is part of the set.
The exact format is defined in sigs.k8s.io/structured-merge-diff*
managedFields.manager
(string)
Manager is an identifier of the workflow managing these fields.
managedFields.operation
(string)
Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for this field are 'Apply' and 'Update'.
managedFields.subresource
(string)
Subresource is the name of the subresource used to update that object, or empty

 string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource.
managedFields.time
(Time)
Time is the timestamp of when the ManagedFields entry was added. The timestamp will also be updated if a field is added, the manager changes any of the owned fields value or removes a field. The timestamp does not update when a field is removed from the entry because another manager took it over.
([]OwnerReference)
uid
List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will point to this controller, with the controller field set to true. There cannot be more than one managing controller.
OwnerReference contains enough information to let you identify an owning object. An owning object must be in the same namespace as the dependent, or be cluster-scoped, so there is no namespace field.
ownerReferences.apiVersion
ownerReferences.kind
Kind of the referent. More info:
ownerReferences.name
ownerReferences.uid
UID of the referent. More info:
http://kubernetes.io/docs/user-guide/identifiers#uids
(boolean)
If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See
https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion
for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned.
ownerReferences.controller
(boolean)
If true, this reference points to the managing controller.
Read-only
(Time)
CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.
Populated by the system. Read-only. Null for lists. More info:
deletionGracePeriodSeconds
(int64)
Number of seconds allowed for this object to gracefully terminate before it will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only.
(Time)
DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the fut

ure, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.
Populated by the system when a graceful deletion is requested. Read-only. More info:
generation
(int64)
A sequence number representing a specific generation of the desired state. Populated by the system. Read-only.
(string)
An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.
Populated by the system. Read-only. Value must be treated as opaque by clients and . More info:
selfLink
(string)
uid
(string)
UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.
Populated by the system. Read-only. More info:
5.9.8 - ObjectReference
ObjectReference contains enough information to let you inspect or modify the referred object.
(string)
fieldPath
(string)
If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
kind
(string)
name
(string)
namespace
(string)
Namespace of the referent. More info:
(string)
Specific resourceVersion to which this reference is made, if any. More info:
uid
(string)
https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
5.9.9 - Patch
Patch is provided to give a concrete name and type to the Kubernetes PATCH request body.
Patch is provided to give a concrete name and t

ype to the Kubernetes PATCH request body.
5.9.10 - Quantity
Quantity is a fixed-point representation of a number.
import "k8s.io/apimachinery/pkg/api/resource"
Quantity is a fixed-point representation of a number. It provides convenient marshaling/unmarshaling in JSON and YAML, in addition to String() and AsInt64() accessors.
The serialization format is:
(Note that \&lt;suffix&gt; may be empty, from the "" case in \&lt;decimalSI&gt;.)

\&lt;digit&gt;           ::= 0 | 1 | ... | 9 \&lt;digits&gt;          ::= \&lt;digit&gt; | \&lt;digit&gt;\&lt;digits&gt; \&lt;number&gt;          ::= \&lt;digits&gt; | \&lt;digits&gt;.\&lt;digits&gt; | \&lt;digits&gt;. | .\&lt;digits&gt; \&lt;sign&gt;            ::= "+" | "-" \&lt;signedNumber&gt;    ::= \&lt;number&gt; | \&lt;sign&gt;\&lt;number&gt; \&lt;suffix&gt;          ::= \&lt;binarySI&gt; | \&lt;decimalExponent&gt; | \&lt;decimalSI&gt; \&lt;binarySI&gt;        ::= Ki | Mi | Gi | Ti | Pi | Ei

	(International System of units; See: http://physics.nist.gov/cuu/Units/binary.html)

\&lt;decimalSI&gt;       ::= m | "" | k | M | G | T | P | E

	(Note that 1024 = 1Ki but 1000 = 1k; I didn't choose the capitalization.)

\&lt;decimalExponent&gt; ::= "e" \&lt;signedNumber&gt; | "E" \&lt;signedNumber&gt; ```

No matter which of the three exponent forms is used, no quantity may represent a number greater than 2^63-1 in magnitude, nor may it have more than 3 decimal places. Numbers larger or more precise will be capped or rounded up. (E.g.: 0.1m will rounded up to 1m.) This may be extended in the future if we require larger or smaller quantities.

When a Quantity is parsed from a string, it will remember the type of suffix it had, and will use the same type again when it is serialized.

Before serializing, Quantity will be put in "canonical form". This means that Exponent/suffix will be adjusted up or down (with a corresponding increase or decrease in Mantissa) such that:

- No precision is lost - No fractional digits will be emitted - The exponent (or suffix) is as large as possible.

The sign will be omitted unless the number is negative.

Examples:

- 1.5 will be serialized as "1500m" - 1.5Gi will be serialized as "1536Mi"

Note that the quantity will NEVER be internally represented by a floating point number. That is the whole point of this exercise.

Non-canonical values will still parse as long as they are well formed, but will be re-emitted in their canonical form. (So always use canonical form, or don't diff.)

This format is intended to make it difficult to use these numbers without writing some sort of special handling code in the hopes that that will cause implementors to also use a fixed point implementation.

&lt;hr&gt;
5.9.11 - ResourceFieldSelector
ResourceFieldSelector represents container resources (cpu, memory) and their output format.
ResourceFieldSelector represents container resources (cpu, memory) and their output format
resource
Required: resource to select
containerName
(string)
Container name: required for volumes, optional for env vars
divisor
(
Quantity
)
Specifies the output format of the exposed resources, defaults to "1"
5.9.12 - Status
Status is a return value for calls that don't return other objects.
(string)
code
(int32)
Suggested HTTP return code for this status, 0 if not set.
details
(StatusDetails)
Extended data associated with the reason. Each reason may define its own extended details. This field is optional and the data returned is not guaranteed to conform to any schema except that defined by the reason type

.
StatusDetails is a set of additional properties that MAY be set by the server to provide additional information about a response. The Reason field of a Status object defines what attributes will be set. Clients must ignore fields that do not match the defined type of each attribute, and should assume that any attribute may be empty, invalid, or under defined.
details.causes
([]StatusCause)
The Causes array includes more details associated with the StatusReason failure. Not all StatusReasons may provide detailed causes.
StatusCause provides more information about an api.Status failure, including cases when multiple errors are encountered.
details.causes.field
(string)
The field of the resource that has caused this error, as named by its JSON serialization. May include dot and postfix notation for nested attributes. Arrays are zero-indexed. Fields may appear more than once in an array of causes due to fields having multiple errors. Optional.
Examples:
"name" - the field "name" on the current resource
"items[0].name" - the field "name" on the first array entry in "items"
details.causes.message
(string)
A human-readable description of the cause of the error. This field may be presented as-is to a reader.
details.causes.reason
(string)
A machine-readable description of the cause of the error. If this value is empty there is no information available.
details.group
(string)
The group attribute of the resource associated with the status StatusReason.
details.kind
(string)
The kind attribute of the resource associated with the status StatusReason. On some operations may differ from the requested resource Kind. More info:
details.name
(string)
The name attribute of the resource associated with the status StatusReason (when there is a single name which can be described).
details.retryAfterSeconds
(int32)
If specified, the time in seconds before the operation should be retried. Some errors may indicate the client must take an alternate action - for those errors this field may indicate how long to wait before taking the alternate action.
details.uid
(string)
UID of the resource. (when there is a single resource which can be described). More info:
kind
(string)
message
(string)
A human-readable description of the status of this operation.
metadata
(
ListMeta
)
reason
(string)
A machine-readable description of why this operation is in the "Failure" status. If this value is empty there is no information available. A Reason clarifies an HTTP status code but does not override it.
status
(string)
Status of the operation. One of: "Success" or "Failure". More info:
5.9.13 - TypedLocalObjectReference
TypedLocalObjectReference contains enough information to let you locate the typed referenced object inside the same namespace.
kind
name
apiGroup
(string)
5.10 - Other Resources
5.10.1 - ValidatingAdmissionPolicyBindingList

 v1alpha1
5.11 - Common Parameters
continue
dryRun
.
feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the
force
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if 

not specified. zero means delete immediately.
limit
continue
namespace
pretty
Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.
watch
6 - Instrumentation
6.1 - Kubernetes Component SLI Metrics
As an alpha feature, Kubernetes lets you configure Service Level Indicator (SLI) metrics
for each Kubernetes component binary. This metric endpoint is exposed on the serving
HTTPS port of each component, at the path
/metrics/slis
. You must enable the
ComponentSLIs
for every component from which you want to scrape SLI metrics.
SLI Metrics
With SLI metrics enabled, each Kubernetes component exposes two metrics,
labeled per healthcheck:
a gauge (which represents the current state of the healthcheck)
a counter (which records the cumulative counts observed for each healthcheck state)
You can use the metric information to calculate per-component availability statistics.
For example, the API server checks the health of etcd. You can work out and report how
available or unavailable etcd has been - as reported by its client, the API server.
The prometheus gauge data looks like this:
# HELP kubernetes_healthcheck [ALPHA] This metric records 

the result of a single healthcheck.
# TYPE kubernetes_healthcheck gauge
kubernetes_healthcheck{name="autoregister-completion",type="healthz"} 1
kubernetes_healthcheck{name="autoregister-completion",type="readyz"} 1
kubernetes_healthcheck{name="etcd",type="healthz"} 1
kubernetes_healthcheck{name="etcd",type="readyz"} 1
kubernetes_healthcheck{name="etcd-readiness",type="readyz"} 1
kubernetes_healthcheck{name="informer-sync",type="readyz"} 1
kubernetes_healthcheck{name="log",type="healthz"} 1
kubernetes_healthcheck{name="log",type="readyz"} 1
kubernetes_healthcheck{name="ping",type="healthz"} 1
kubernetes_healthcheck{name="ping",type="readyz"} 1
While the counter data looks like this:
# HELP kubernetes_healthchecks_total [ALPHA] This metric records the results of all healthcheck.
# TYPE kubernetes_healthchecks_total counter
kubernetes_healthchecks_total{name="autoregister-completion",status="error",type="readyz"} 1
kubernetes_healthchecks_total{name="autoregister-completion",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="autoregister-completion",status="success",type="readyz"} 14
kubernetes_healthchecks_total{name="etcd",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="etcd",status="success",type="readyz"} 15
kubernetes_healthchecks_total{name="etcd-readiness",status="success",type="readyz"} 15
kubernetes_healthchecks_total{name="informer-sync",status="error",type="readyz"} 1
kubernetes_healthchecks_total{name="informer-sync",status="success",type="readyz"} 14
kubernetes_healthchecks_total{name="log",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="log",status="success",type="readyz"} 15
kubernetes_healthchecks_total{name="ping",status="success",type="healthz"} 15
kubernetes_healthchecks_total{name="ping",status="success",type="readyz"} 15
Using this data
The component SLIs metrics endpoint is intended to be scraped at a high frequency. Scraping
at a high frequency means that you end up with greater granularity of the gauge's signal, which
can be then used to calculate SLOs. The
endpoint provides the raw data necessary
to calculate an availability SLO for the respective Kubernetes component.
6.2 - Node metrics data
Mechanisms for accessing metrics at node, volume, pod and container level, as seen by the kubelet.
The
kubelet
gathers metric statistics at the node, volume, pod and container level,
and emits this information in the
.
You can send a proxied request to the stats summary API via the
Kubernetes API server.
Here is an example of a Summary API request for a node named
minikube
:
"/api/v1/nodes/minikube/proxy/stats/summary"
curl
:
# You need to run "kubectl proxy" first
# Change 8080 to the port that "kubectl proxy" assigns
curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary
Note:
Beginning with
0.6.x,
queries the
kubelet endpoint, and not
.
Summary metrics API source
By default, Kubernetes fetches node summary metrics data using an embedded
cAdvisor
that runs within the kubelet.
Summary API data via CRI
Kubernetes v1.23 [alpha]
If you enable the
PodAndContainerStatsFromCRI
in your
cluster, and you use a container runtime that supports statistics access via
(CRI), then
the kubelet fetches Pod- and container-level metric data using CRI, and not via cAdvisor.
The task pages for
discuss
how to use a metrics pipeline that rely on these data.
6.3 - Kubernetes Metrics Reference
Details of the metric data that Kubernetes components export.
Metrics (auto-generated 2022 Nov 01)
This page details the metrics that different Kubernetes components export. You can query the metrics endpoint for these
components using an HTTP scrape, and fetch the current metrics data in Prometheus format.
List of Stable Kubernetes Metrics
N

ame
Stability Level
Type
Help
Labels
Const Labels
apiserver_admission_controller_admission_duration_seconds
STABLE
Histogram
Admission controller latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).
name
operation
rejected
type
None
apiserver_admission_step_admission_duration_seconds
STABLE
Histogram
Admission sub-step latency histogram in seconds, broken out for each operation and API resource and step type (validate or admit).
operation
rejected
type
None
apiserver_admission_webhook_admission_duration_seconds
STABLE
Histogram
Admission webhook latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).
name
operation
rejected
type
None
STABLE
Gauge
Maximal number of currently used inflight request limit of this apiserver per request kind in last second.
None
apiserver_longrunning_requests
STABLE
Gauge
Gauge of all active long-running apiserver requests broken out by verb, group, version, resource, scope and component. Not all requests are tracked this way.
component
group
resource
scope
verb
version
None
apiserver_request_duration_seconds
STABLE
Histogram
Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.
component
dry_run
group
resource
scope
verb
version
None
STABLE
Counter
Counter of apiserver requests broken out for each verb, dry run value, group, version, resource, scope, component, and HTTP response code.
code
component
dry_run
group
resource
scope
verb
version
None
STABLE
Gauge
Gauge of deprecated APIs that have been requested, broken out by API group, version, resource, subresource, and removed_release.
group
resource
version
None
apiserver_response_sizes
STABLE
Histogram
Response size distribution in bytes for each group, version, verb, resource, subresource, scope and component.
component
group
resource
scope
verb
version
None
apiserver_storage_objects
STABLE
Gauge
Number of stored objects at the time of last check split by kind.
resource
None
node_collector_evictions_total
STABLE
Counter
Number of Node evictions that happened since current instance of NodeController started.
zone
None
scheduler_framework_extension_point_duration_seconds
STABLE
Histogram
Latency for running all plugins of a specific extension point.
extension_point
profile
status
None
scheduler_pending_pods
STABLE
Gauge
Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulablePods.
queue
None
scheduler_pod_scheduling_attempts
STABLE
Histogram
Number of attempts to successfully schedule a pod.
None
None
scheduler_pod_scheduling_duration_seconds
STABLE
Histogram
E2e latency for a pod being scheduled which may include multiple scheduling attempts.
attempts
None
scheduler_preemption_attempts_total
STABLE
Counter
Total preemption attempts in the cluster till now
None
None
scheduler_preemption_victims
STABLE
Histogram
Number of selected preemption victims
None
None
scheduler_queue_incoming_pods_total
STABLE
Counter
Number of pods added to scheduling queues by event and queue type.
event
queue
None
scheduler_schedule_attempts_total
STABLE
Counter
Number of attempts to schedule pods, by the result. 'unschedulable' means a pod could not be scheduled, while 'error' means an internal scheduler problem.
profile
result
None
scheduler_scheduling_attempt_duration_seconds
STABLE
Histogram
Scheduling attempt latency in seconds (scheduling algorithm + binding)
profile
result
None
List of Alpha Kubernetes Metrics
Name
Type
Help
Labels
aggregator_openapi_v2_regeneration_count
ALPHA
Counter
Counter of OpenAPI v2 

spec regeneration count broken down by causing APIService name and reason.
apiservice
reason
None
aggregator_openapi_v2_regeneration_duration
ALPHA
Gauge
Gauge of OpenAPI v2 spec regeneration duration in seconds.
reason
None
aggregator_unavailable_apiservice
ALPHA
Custom
Gauge of APIServices which are marked as unavailable broken down by APIService name.
name
None
aggregator_unavailable_apiservice_total
ALPHA
Counter
Counter of APIServices which are marked as unavailable broken down by APIService name and reason.
name
reason
None
apiextensions_openapi_v2_regeneration_count
ALPHA
Counter
Counter of OpenAPI v2 spec regeneration count broken down by causing CRD name and reason.
crd
reason
None
apiextensions_openapi_v3_regeneration_count
ALPHA
Counter
Counter of OpenAPI v3 spec regeneration count broken down by group, version, causing CRD and reason.
crd
group
reason
version
None
apiserver_admission_step_admission_duration_seconds_summary
ALPHA
Summary
Admission sub-step latency summary in seconds, broken out for each operation and API resource and step type (validate or admit).
operation
rejected
type
None
apiserver_admission_webhook_fail_open_count
ALPHA
Counter
Admission webhook fail open count, identified by name and broken out for each admission type (validating or mutating).
name
type
None
apiserver_admission_webhook_rejection_count
ALPHA
Counter
Admission webhook rejection count, identified by name and broken out for each admission type (validating or admit) and operation. Additional labels specify an error type (calling_webhook_error or apiserver_internal_error if an error occurred; no_error otherwise) and optionally a non-zero rejection code if the webhook rejects the request with an HTTP status code (honored by the apiserver when the code is greater or equal to 400). Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
name
operation
type
None
apiserver_admission_webhook_request_total
ALPHA
Counter
Admission webhook request total, identified by name and broken out for each admission type (validating or mutating) and operation. Additional labels specify whether the request was rejected or not and an HTTP status code. Codes greater than 600 are truncated to 600, to keep the metrics cardinality bounded.
code
name
operation
rejected
type
None
ALPHA
Counter
Counter of audit events that failed to be audited properly. Plugin identifies the plugin affected by the error.
plugin
None
ALPHA
Counter
Counter of audit events generated and sent to the audit backend.
None
None
apiserver_audit_level_total
ALPHA
Counter
Counter of policy levels for audit events (1 per request).
level
None
apiserver_audit_requests_rejected_total
ALPHA
Counter
Counter of apiserver requests rejected due to an error in audit logging backend.
None
None
apiserver_cache_list_fetched_objects_total
ALPHA
Counter
Number of objects read from watch cache in the course of serving a LIST request
index
resource_prefix
None
apiserver_cache_list_returned_objects_total
ALPHA
Counter
Number of objects returned for a LIST request from watch cache
None
apiserver_cache_list_total
ALPHA
Counter
Number of LIST requests served from watch cache
index
None
apiserver_cel_compilation_duration_seconds
ALPHA
Histogram
None
None
apiserver_cel_evaluation_duration_seconds
ALPHA
Histogram
None
None
apiserver_certificates_registry_csr_honored_duration_total
ALPHA
Counter
Total number of issued CSRs with a requested duration that was honored, sliced by signer (only kubernetes.io signer names are specifically identified)
None
apiserver_certificates_registry_csr_requested_duration_total
ALPHA
Counter
Total number of issued CSRs with a requested duration, sliced by signer (only kubernetes.io signer names are specifically identified)
None
apiserver_client_certificate_expiration_seconds
ALPHA
Histogram
Distribution of the remainin

g lifetime on the certificate used to authenticate a request.
None
None
apiserver_crd_webhook_conversion_duration_seconds
ALPHA
Histogram
CRD webhook conversion duration in seconds
crd_name
from_version
succeeded
to_version
None
ALPHA
Gauge
Maximal number of queued requests in this apiserver per request kind in last second.
None
apiserver_delegated_authn_request_duration_seconds
ALPHA
Histogram
Request latency in seconds. Broken down by status code.
code
None
apiserver_delegated_authn_request_total
ALPHA
Counter
Number of HTTP requests partitioned by status code.
code
None
apiserver_delegated_authz_request_duration_seconds
ALPHA
Histogram
code
None
apiserver_delegated_authz_request_total
ALPHA
Counter
code
None
apiserver_egress_dialer_dial_duration_seconds
ALPHA
Histogram
Dial latency histogram in seconds, labeled by the protocol (http-connect or grpc), transport (tcp or uds)
protocol
transport
None
apiserver_egress_dialer_dial_failure_count
ALPHA
Counter
Dial failure count, labeled by the protocol (http-connect or grpc), transport (tcp or uds), and stage (connect or proxy). The stage indicates at which stage the dial failed
protocol
stage
transport
None
apiserver_envelope_encryption_dek_cache_fill_percent
ALPHA
Gauge
Percent of the cache slots currently occupied by cached DEKs.
None
None
apiserver_envelope_encryption_dek_cache_inter_arrival_time_seconds
ALPHA
Histogram
Time (in seconds) of inter arrival of transformation requests.
transformation_type
None
ALPHA
Gauge
Number of requests in initial (for a WATCH) or any (for a non-WATCH) execution stage in the API Priority and Fairness subsystem
None
ALPHA
Gauge
Number of requests currently pending in queues of the API Priority and Fairness subsystem
None
apiserver_flowcontrol_current_r
ALPHA
Gauge
R(time of last change)
None
apiserver_flowcontrol_dispatch_r
ALPHA
Gauge
R(time of last dispatch)
None
ALPHA
Counter
Number of requests executed by API Priority and Fairness subsystem
None
apiserver_flowcontrol_epoch_advance_total
ALPHA
Counter
Number of times the queueset's progress meter jumped backward
success
None
apiserver_flowcontrol_latest_s
ALPHA
Gauge
S(most recently dispatched request)
None
apiserver_flowcontrol_next_discounted_s_bounds
ALPHA
Gauge
min and max, over queues, of S(oldest waiting request in queue) - estimated work in progress
bound
None
apiserver_flowcontrol_next_s_bounds
ALPHA
Gauge
min and max, over queues, of S(oldest waiting request in queue)
bound
None
ALPHA
TimingRatioHistogram
Observations, at the end of every nanosecond, of number of requests (as a fraction of the relevant limit) waiting or in any stage of execution (but only initial stage for WATCHes)
phase
None
ALPHA
Observations, at the end of every nanosecond, of utilization of seats for any stage of execution (but only initial stage for WATCHes)
map[phase:executing]
ALPHA
Observations, at the end of every nanosecond, of the number of requests (as a fraction of the relevant limit) waiting or in regular stage of execution
phase
None
ALPHA
Counter
Number of requests rejected by API Priority and Fairness subsystem
reason
None
ALPHA
Gauge
Concurrency (number of seats) occupied 

by the currently executing (initial stage for a WATCH, any stage otherwise) requests in the API Priority and Fairness subsystem
None
ALPHA
Gauge
Shared concurrency limit in the API Priority and Fairness subsystem
None
ALPHA
Counter
Number of times a dispatch attempt resulted in a non accommodation due to lack of available seats
None
ALPHA
Histogram
Duration of initial stage (for a WATCH) or any (for a non-WATCH) stage of request execution in the API Priority and Fairness subsystem
type
None
ALPHA
Histogram
Length of queue in the API Priority and Fairness subsystem, as seen by each request after it is enqueued
None
ALPHA
Histogram
Length of time a request spent waiting in its queue
execute
None
ALPHA
Histogram
count of watchers for mutating requests in API Priority and Fairness
None
ALPHA
Histogram
Number of estimated seats (maximum of initial and final seats) associated with requests in API Priority and Fairness
None
apiserver_init_events_total
ALPHA
Counter
Counter of init events processed in watch cache broken by resource type.
resource
None
apiserver_kube_aggregator_x509_insecure_sha1_total
ALPHA
Counter
Counts the number of requests to servers with insecure SHA1 signatures in their serving certificate OR the number of connection failures due to the insecure SHA1 signatures (either/or, based on the runtime environment)
None
None
apiserver_kube_aggregator_x509_missing_san_total
ALPHA
Counter
Counts the number of requests to servers missing SAN extension in their serving certificate OR the number of connection failures due to the lack of x509 certificate SAN extension missing (either/or, based on the runtime environment)
None
None
apiserver_request_aborts_total
ALPHA
Counter
Number of requests which apiserver aborted possibly due to a timeout, for each group, version, verb, resource, subresource and scope
group
resource
scope
verb
version
None
apiserver_request_body_sizes
ALPHA
Histogram
Apiserver request body sizes broken out by size.
resource
verb
None
apiserver_request_filter_duration_seconds
ALPHA
Histogram
Request filter latency distribution in seconds, for each filter type
filter
None
apiserver_request_post_timeout_total
ALPHA
Counter
Tracks the activity of the request handlers after the associated requests have been timed out by the apiserver
source
status
None
apiserver_request_slo_duration_seconds
ALPHA
Histogram
Response latency distribution (not counting webhook duration) in seconds for each verb, group, version, resource, subresource, scope and component.
component
group
resource
scope
verb
version
None
apiserver_request_terminations_total
ALPHA
Counter
Number of requests which apiserver terminated in self-defense.
code
component
group
resource
scope
verb
version
None
apiserver_request_timestamp_comparison_time
ALPHA
Histogram
Time taken for comparison of old vs new objects in UPDATE or PATCH requests
code_path
None
apiserver_selfrequest_total
ALPHA
Counter
Counter of apiserver self-requests broken out for each verb, API resource and subresource.
resource
verb
None
apiserver_storage_data_key_generation_duration_seconds
ALPHA
Histogram
Latencies in seconds of data encryption key(DEK) generation operations.
None
None
apiserver_storage_data_key_generation_failures_total
ALPHA
Counter
Total number of failed data encryption key(DEK) generation operations.
None
None
apiserver_storage_db_total_size_in_bytes
ALPHA
Gauge
Total siz

e of the storage database file physically allocated in bytes.
endpoint
None
apiserver_storage_envelope_transformation_cache_misses_total
ALPHA
Counter
Total number of cache misses while accessing key decryption key(KEK).
None
None
apiserver_storage_list_evaluated_objects_total
ALPHA
Counter
Number of objects tested in the course of serving a LIST request from storage
resource
None
apiserver_storage_list_fetched_objects_total
ALPHA
Counter
Number of objects read from storage in the course of serving a LIST request
resource
None
apiserver_storage_list_returned_objects_total
ALPHA
Counter
Number of objects returned for a LIST request from storage
resource
None
apiserver_storage_list_total
ALPHA
Counter
Number of LIST requests served from storage
resource
None
apiserver_storage_transformation_duration_seconds
ALPHA
Histogram
Latencies in seconds of value transformation operations.
None
apiserver_storage_transformation_operations_total
ALPHA
Counter
Total number of transformations.
status
transformer_prefix
None
apiserver_terminated_watchers_total
ALPHA
Counter
Counter of watchers closed due to unresponsiveness broken by resource type.
resource
None
apiserver_tls_handshake_errors_total
ALPHA
Counter
Number of requests dropped with 'TLS handshake error from' error
None
None
apiserver_validating_admission_policy_check_duration_seconds
ALPHA
Histogram
Validation admission latency for individual validation expressions in seconds, labeled by policy and param resource, further including binding, state and enforcement action taken.
enforcement_action
params
policy
policy_binding
state
validation_expression
None
apiserver_validating_admission_policy_check_total
ALPHA
Counter
Validation admission policy check total, labeled by policy and param resource, and further identified by binding, validation expression, enforcement action taken, and state.
params
policy
state
None
apiserver_validating_admission_policy_definition_total
ALPHA
Counter
Validation admission policy count total, labeled by state and enforcement action.
state
None
apiserver_watch_cache_events_dispatched_total
ALPHA
Counter
Counter of events dispatched in watch cache broken by resource type.
resource
None
apiserver_watch_cache_initializations_total
ALPHA
Counter
Counter of watch cache initializations broken by resource type.
resource
None
apiserver_watch_events_sizes
ALPHA
Histogram
Watch event size distribution in bytes
group
kind
version
None
apiserver_watch_events_total
ALPHA
Counter
Number of events sent in watch clients
group
kind
version
None
apiserver_webhooks_x509_insecure_sha1_total
ALPHA
Counter
None
None
apiserver_webhooks_x509_missing_san_total
ALPHA
Counter
None
None
attachdetach_controller_forced_detaches
ALPHA
Counter
Number of times the A/D Controller performed a forced detach
None
None
attachdetach_controller_total_volumes
ALPHA
Custom
Number of volumes in A/D Controller
plugin_name
state
None
authenticated_user_requests
ALPHA
Counter
Counter of authenticated requests broken out by username.
username
None
authentication_attempts
ALPHA
Counter
Counter of authenticated attempts.
result
None
authentication_duration_seconds
ALPHA
Histogram
Authentication duration in seconds broken out by result.
result
None
authentication_token_cache_active_fetch_count
ALPHA
Gauge
status
None
authentication_token_cache_fetch_total
ALPHA
Counter
status
None
authentication_token_cache_request_duration_seconds
ALPHA
Histogram

status
None
authentication_token_cache_request_total
ALPHA
Counter
status
None
cloudprovider_aws_api_request_duration_seconds
ALPHA
Histogram
Latency of AWS API calls
request
None
cloudprovider_aws_api_request_errors
ALPHA
Counter
AWS API errors
request
None
cloudprovider_aws_api_throttled_requests_total
ALPHA
Counter
AWS API throttled requests
operation_name
None
cloudprovider_azure_api_request_duration_seconds
ALPHA
Histogram
Latency of an Azure API call
request
resource_group
source
subscription_id
None
cloudprovider_azure_api_request_errors
ALPHA
Counter
Number of errors for an Azure API call
request
source
None
cloudprovider_azure_api_request_ratelimited_count
ALPHA
Counter
Number of rate limited Azure API calls
request
source
None
cloudprovider_azure_api_request_throttled_count
ALPHA
Counter
Number of throttled Azure API calls
request
source
None
cloudprovider_azure_op_duration_seconds
ALPHA
Histogram
Latency of an Azure service operation
request
source
None
cloudprovider_azure_op_failure_count
ALPHA
Counter
Number of failed Azure service operations
request
source
None
cloudprovider_gce_api_request_duration_seconds
ALPHA
Histogram
Latency of a GCE API call
region
request
version
zone
None
cloudprovider_gce_api_request_errors
ALPHA
Counter
Number of errors for an API call
region
request
version
zone
None
cloudprovider_vsphere_api_request_duration_seconds
ALPHA
Histogram
Latency of vsphere api call
request
None
cloudprovider_vsphere_api_request_errors
ALPHA
Counter
vsphere Api errors
request
None
cloudprovider_vsphere_operation_duration_seconds
ALPHA
Histogram
Latency of vsphere operation call
operation
None
cloudprovider_vsphere_operation_errors
ALPHA
Counter
vsphere operation errors
operation
None
cloudprovider_vsphere_vcenter_versions
ALPHA
Custom
Versions for connected vSphere vCenters
hostname
version
build
None
container_cpu_usage_seconds_total
ALPHA
Custom
Cumulative cpu time consumed by the container in core-seconds
container
pod
namespace
None
container_memory_working_set_bytes
ALPHA
Custom
Current working set of the container in bytes
container
pod
namespace
None
container_start_time_seconds
ALPHA
Custom
Start time of the container since unix epoch in seconds
container
pod
namespace
None
cronjob_controller_cronjob_job_creation_skew_duration_seconds
ALPHA
Histogram
Time between when a cronjob is scheduled to be run, and when the corresponding job is created
None
None
csi_operations_seconds
ALPHA
Histogram
Container Storage Interface operation duration with gRPC error code status total
driver_name
grpc_status_code
method_name
migrated
None
endpoint_slice_controller_changes
ALPHA
Counter
Number of EndpointSlice changes
operation
None
endpoint_slice_controller_desired_endpoint_slices
ALPHA
Gauge
Number of EndpointSlices that would exist with perfect endpoint allocation
None
None
endpoint_slice_controller_endpoints_added_per_sync
ALPHA
Histogram
Number of endpoints added on each Service sync
None
None
endpoint_slice_controller_endpoints_desired
ALPHA
Gauge
Number of endpoints desired
None
None
endpoint_slice_controller_endpoints_removed_per_sync
ALPHA
Histogram
Number of endpoints removed on each Service sync
None
None
endpoint_slice_controller_endpointslices_changed_per_sync
ALPHA
Histogram
Number of EndpointSlices changed on each Service sync
topology
None
endpoint_slice_controller_num_endpoint_slices
ALPHA
Gauge
Number of EndpointSlices
None
None
endpoint_slice_controller_syncs
ALPHA
Counter
Number of EndpointSlice syncs
result
None
endpoint_slice_mirroring_controller_addresses_skipped_per_sync
ALPHA
Histogram
Number of addresses skipped on each Endpoints sync due to being invalid or exceeding MaxEndpointsPerSubset
None
None
endpoint_slice_mirroring_controller_changes
ALPHA
Counter
operation
None
endpoint_slice_mirroring_controller

_desired_endpoint_slices
ALPHA
Gauge
None
None
endpoint_slice_mirroring_controller_endpoints_added_per_sync
ALPHA
Histogram
Number of endpoints added on each Endpoints sync
None
None
endpoint_slice_mirroring_controller_endpoints_desired
ALPHA
Gauge
None
None
endpoint_slice_mirroring_controller_endpoints_removed_per_sync
ALPHA
Histogram
Number of endpoints removed on each Endpoints sync
None
None
endpoint_slice_mirroring_controller_endpoints_sync_duration
ALPHA
Histogram
Duration of syncEndpoints() in seconds
None
None
endpoint_slice_mirroring_controller_endpoints_updated_per_sync
ALPHA
Histogram
Number of endpoints updated on each Endpoints sync
None
None
endpoint_slice_mirroring_controller_num_endpoint_slices
ALPHA
Gauge
None
None
ephemeral_volume_controller_create_failures_total
ALPHA
Counter
Number of PersistenVolumeClaims creation requests
None
None
ephemeral_volume_controller_create_total
ALPHA
Counter
None
None
etcd_bookmark_counts
ALPHA
Gauge
Number of etcd bookmarks (progress notify events) split by kind.
resource
None
etcd_lease_object_counts
ALPHA
Histogram
Number of objects attached to a single etcd lease.
None
None
etcd_request_duration_seconds
ALPHA
Histogram
Etcd request latency in seconds for each operation and object type.
operation
type
None
etcd_version_info
ALPHA
Gauge
Etcd server's binary version
binary_version
None
field_validation_request_duration_seconds
ALPHA
Histogram
Response latency distribution in seconds for each field validation value and whether field validation is enabled or not
enabled
field_validation
None
garbagecollector_controller_resources_sync_error_total
ALPHA
Counter
Number of garbage collector resources sync errors
None
None
get_token_count
ALPHA
Counter
Counter of total Token() requests to the alternate token source
None
None
get_token_fail_count
ALPHA
Counter
Counter of failed Token() requests to the alternate token source
None
None
job_controller_job_finished_total
ALPHA
Counter
The number of finished job
completion_mode
reason
result
None
job_controller_job_pods_finished_total
ALPHA
Counter
The number of finished Pods that are fully tracked
result
None
job_controller_job_sync_duration_seconds
ALPHA
Histogram
The time it took to sync a job
action
result
None
job_controller_job_sync_total
ALPHA
Counter
The number of job syncs
action
result
None
job_controller_pod_failures_handled_by_failure_policy_total
ALPHA
Counter
`The number of failed Pods handled by failure policy with, respect to the failure policy action applied based on the matched, rule. Possible values of the action label correspond to the, possible values for the failure policy rule action, which are:, "FailJob", "Ignore" and "Count".`
action
None
job_controller_terminated_pods_tracking_finalizer_total
ALPHA
Counter
`The number of terminated pods (phase=Failed|Succeeded), that have the finalizer batch.kubernetes.io/job-tracking, The event label can be "add" or "delete".`
event
None
kube_apiserver_clusterip_allocator_allocated_ips
ALPHA
Gauge
Gauge measuring the number of allocated IPs for Services
cidr
None
kube_apiserver_clusterip_allocator_allocation_errors_total
ALPHA
Counter
Number of errors trying to allocate Cluster IPs
cidr
scope
None
kube_apiserver_clusterip_allocator_allocation_total
ALPHA
Counter
Number of Cluster IPs allocations
cidr
scope
None
kube_apiserver_clusterip_allocator_available_ips
ALPHA
Gauge
Gauge measuring the number of available IPs for Services
cidr
None
kube_apiserver_pod_logs_pods_logs_backend_tls_failure_total
ALPHA
Counter
Total number of requests for pods/logs that failed due to kubelet server TLS verification
None
None
kube_apiserver_pod_logs_pods_logs_insecure_backend_total
ALPHA
Counter
Total number of requests for pods/logs sliced by usage type: enforce_tls, skip_tl

s_allowed, skip_tls_denied
usage
None
ALPHA
Custom
Resources limit for workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.
namespace
pod
node
scheduler
priority
resource
unit
None
ALPHA
Custom
Resources requested by workloads on the cluster, broken down by pod. This shows the resource usage the scheduler and kubelet expect per pod for resources along with the unit for the resource if any.
namespace
pod
node
scheduler
priority
resource
unit
None
kubelet_certificate_manager_client_expiration_renew_errors
ALPHA
Counter
Counter of certificate renewal errors.
None
None
kubelet_certificate_manager_client_ttl_seconds
ALPHA
Gauge
Gauge of the TTL (time-to-live) of the Kubelet's client certificate. The value is in seconds until certificate expiry (negative if already expired). If client certificate is invalid or unused, the value will be +INF.
None
None
kubelet_certificate_manager_server_rotation_seconds
ALPHA
Histogram
Histogram of the number of seconds the previous certificate lived before being rotated.
None
None
kubelet_certificate_manager_server_ttl_seconds
ALPHA
Gauge
Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate. The value is in seconds until certificate expiry (negative if already expired). If serving certificate is invalid or unused, the value will be +INF.
None
None
kubelet_cgroup_manager_duration_seconds
ALPHA
Histogram
Duration in seconds for cgroup manager operations. Broken down by method.
operation_type
None
kubelet_container_log_filesystem_used_bytes
ALPHA
Custom
Bytes used by the container's logs on the filesystem.
uid
namespace
pod
container
None
kubelet_containers_per_pod_count
ALPHA
Histogram
The number of containers per pod.
None
None
kubelet_cpu_manager_pinning_errors_total
ALPHA
Counter
The number of cpu core allocations which required pinning failed.
None
None
kubelet_cpu_manager_pinning_requests_total
ALPHA
Counter
The number of cpu core allocations which required pinning.
None
None
kubelet_device_plugin_alloc_duration_seconds
ALPHA
Histogram
Duration in seconds to serve a device plugin Allocation request. Broken down by resource name.
resource_name
None
kubelet_device_plugin_registration_total
ALPHA
Counter
Cumulative number of device plugin registrations. Broken down by resource name.
None
kubelet_eviction_stats_age_seconds
ALPHA
Histogram
Time between when stats are collected, and when pod is evicted based on those stats by eviction signal
eviction_signal
None
kubelet_evictions
ALPHA
Counter
Cumulative number of pod evictions by eviction signal
None
kubelet_graceful_shutdown_end_time_seconds
ALPHA
Gauge
Last graceful shutdown start time since unix epoch in seconds
None
None
kubelet_graceful_shutdown_start_time_seconds
ALPHA
Gauge
None
None
kubelet_http_inflight_requests
ALPHA
Gauge
Number of the inflight http requests
long_running
method
path
server_type
None
kubelet_http_requests_duration_seconds
ALPHA
Histogram
Duration in seconds to serve http requests
method
path
None
kubelet_http_requests_total
ALPHA
Counter
Number of the http requests received since the server started
method
path
None
kubelet_kubelet_credential_provider_plugin_duration
ALPHA
Histogram
Duration of execution in seconds for credential provider plugin
None
kubelet_kubelet_credential_provider_plugin_errors
ALPHA
Counter
Number of errors from credential provider plugin
None
kubelet_lifecycle_handler_http_fallbacks_total
ALPHA
Counter
The number of times lifecycle handlers successfully fell back to http from https.
None
None
kubelet_managed_ephemeral_containers
ALPHA
Gauge
Current number of ephemeral containers in pods managed by this kubelet. Ephemeral containers will be ignor

ed if disabled by the EphemeralContainers feature gate, and this number will be 0.
None
None
kubelet_node_name
ALPHA
Gauge
The node's name. The count is always 1.
node
None
kubelet_pleg_discard_events
ALPHA
Counter
The number of discard events in PLEG.
None
None
kubelet_pleg_last_seen_seconds
ALPHA
Gauge
Timestamp in seconds when PLEG was last seen active.
None
None
kubelet_pleg_relist_duration_seconds
ALPHA
Histogram
Duration in seconds for relisting pods in PLEG.
None
None
kubelet_pleg_relist_interval_seconds
ALPHA
Histogram
Interval in seconds between relisting in PLEG.
None
None
kubelet_pod_resources_endpoint_errors_get_allocatable
ALPHA
Counter
Number of requests to the PodResource GetAllocatableResources endpoint which returned error. Broken down by server api version.
server_api_version
None
kubelet_pod_resources_endpoint_errors_list
ALPHA
Counter
Number of requests to the PodResource List endpoint which returned error. Broken down by server api version.
None
kubelet_pod_resources_endpoint_requests_get_allocatable
ALPHA
Counter
Number of requests to the PodResource GetAllocatableResources endpoint. Broken down by server api version.
None
kubelet_pod_resources_endpoint_requests_list
ALPHA
Counter
Number of requests to the PodResource List endpoint. Broken down by server api version.
None
kubelet_pod_resources_endpoint_requests_total
ALPHA
Counter
Cumulative number of requests to the PodResource endpoint. Broken down by server api version.
None
kubelet_pod_start_duration_seconds
ALPHA
Histogram
Duration in seconds from kubelet seeing a pod for the first time to the pod starting to run
None
None
kubelet_pod_status_sync_duration_seconds
ALPHA
Histogram
Duration in seconds to sync a pod status update. Measures time from detection of a change to pod status until the API is successfully updated for that pod, even if multiple intevening changes to pod status occur.
None
None
kubelet_pod_worker_duration_seconds
ALPHA
Histogram
Duration in seconds to sync a single pod. Broken down by operation type: create, update, or sync
None
kubelet_pod_worker_start_duration_seconds
ALPHA
Histogram
Duration in seconds from kubelet seeing a pod to starting a worker.
None
None
kubelet_preemptions
ALPHA
Counter
Cumulative number of pod preemptions by preemption resource
preemption_signal
None
kubelet_run_podsandbox_duration_seconds
ALPHA
Histogram
Duration in seconds of the run_podsandbox operations. Broken down by RuntimeClass.Handler.
runtime_handler
None
kubelet_run_podsandbox_errors_total
ALPHA
Counter
Cumulative number of the run_podsandbox operation errors by RuntimeClass.Handler.
None
kubelet_running_containers
ALPHA
Gauge
Number of containers currently running
container_state
None
kubelet_running_pods
ALPHA
Gauge
Number of pods that have a running pod sandbox
None
None
kubelet_runtime_operations_duration_seconds
ALPHA
Histogram
Duration in seconds of runtime operations. Broken down by operation type.
None
kubelet_runtime_operations_errors_total
ALPHA
Counter
Cumulative number of runtime operation errors by operation type.
None
kubelet_runtime_operations_total
ALPHA
Counter
Cumulative number of runtime operations by operation type.
None
kubelet_server_expiration_renew_errors
ALPHA
Counter
None
None
kubelet_started_containers_errors_total
ALPHA
Counter
Cumulative number of errors when starting containers
code
container_type
None
kubelet_started_containers_total
ALPHA
Counter
Cumulative number of containers started
None
kubelet_started_host_process_containers_errors_total
ALPHA
Counter
Cumulative number of errors when starting hostprocess containers. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.
code
None
kubelet_started_host_process_containers_total
ALPHA

Counter
Cumulative number of hostprocess containers started. This metric will only be collected on Windows and requires WindowsHostProcessContainers feature gate to be enabled.
None
kubelet_started_pods_errors_total
ALPHA
Counter
Cumulative number of errors when starting pods
None
None
kubelet_started_pods_total
ALPHA
Counter
Cumulative number of pods started
None
None
kubelet_volume_metric_collection_duration_seconds
ALPHA
Histogram
Duration in seconds to calculate volume stats
metric_source
None
kubelet_volume_stats_available_bytes
ALPHA
Custom
Number of available bytes in the volume
namespace
persistentvolumeclaim
None
kubelet_volume_stats_capacity_bytes
ALPHA
Custom
Capacity in bytes of the volume
namespace
None
kubelet_volume_stats_health_status_abnormal
ALPHA
Custom
Abnormal volume health status. The count is either 1 or 0. 1 indicates the volume is unhealthy, 0 indicates volume is healthy
namespace
None
kubelet_volume_stats_inodes
ALPHA
Custom
Maximum number of inodes in the volume
namespace
None
kubelet_volume_stats_inodes_free
ALPHA
Custom
Number of free inodes in the volume
namespace
None
kubelet_volume_stats_inodes_used
ALPHA
Custom
Number of used inodes in the volume
namespace
None
kubelet_volume_stats_used_bytes
ALPHA
Custom
Number of used bytes in the volume
namespace
None
kubeproxy_network_programming_duration_seconds
ALPHA
Histogram
In Cluster Network Programming Latency in seconds
None
None
kubeproxy_sync_proxy_rules_duration_seconds
ALPHA
Histogram
SyncProxyRules latency in seconds
None
None
kubeproxy_sync_proxy_rules_endpoint_changes_pending
ALPHA
Gauge
Pending proxy rules Endpoint changes
None
None
kubeproxy_sync_proxy_rules_endpoint_changes_total
ALPHA
Counter
Cumulative proxy rules Endpoint changes
None
None
kubeproxy_sync_proxy_rules_iptables_restore_failures_total
ALPHA
Counter
Cumulative proxy iptables restore failures
None
None
kubeproxy_sync_proxy_rules_iptables_total
ALPHA
Gauge
Number of proxy iptables rules programmed
table
None
kubeproxy_sync_proxy_rules_last_queued_timestamp_seconds
ALPHA
Gauge
The last time a sync of proxy rules was queued
None
None
kubeproxy_sync_proxy_rules_last_timestamp_seconds
ALPHA
Gauge
The last time proxy rules were successfully synced
None
None
kubeproxy_sync_proxy_rules_no_local_endpoints_total
ALPHA
Gauge
Number of services with a Local traffic policy and no endpoints
traffic_policy
None
kubeproxy_sync_proxy_rules_service_changes_pending
ALPHA
Gauge
Pending proxy rules Service changes
None
None
kubeproxy_sync_proxy_rules_service_changes_total
ALPHA
Counter
Cumulative proxy rules Service changes
None
None
kubernetes_build_info
ALPHA
Gauge
A metric with a constant '1' value labeled by major, minor, git version, git commit, git tree state, build date, Go version, and compiler from which Kubernetes was built, and platform on which it is running.
build_date
compiler
git_commit
git_tree_state
git_version
go_version
major
minor
platform
None
kubernetes_feature_enabled
ALPHA
Gauge
This metric records the data about the stage and enablement of a k8s feature.
name
stage
None
kubernetes_healthcheck
ALPHA
Gauge
This metric records the result of a single healthcheck.
name
type
None
kubernetes_healthchecks_total
ALPHA
Counter
This metric records the results of all healthcheck.
name
status
type
None
leader_election_master_status
ALPHA
Gauge
Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
name
None
node_authorizer_graph_actions_duration_seconds
ALPHA
Histogram
Histogram of duration of graph actions in node authorizer.
operation
None
node_collector_evictions_number
ALPHA
Counter
Number of Node evictions that happened since current instance of NodeController started, This metric is replaced by node_c

ollector_evictions_total.
zone
None
node_collector_unhealthy_nodes_in_zone
ALPHA
Gauge
Gauge measuring number of not Ready Nodes per zones.
zone
None
node_collector_zone_health
ALPHA
Gauge
Gauge measuring percentage of healthy nodes per zone.
zone
None
node_collector_zone_size
ALPHA
Gauge
Gauge measuring number of registered Nodes per zones.
zone
None
node_cpu_usage_seconds_total
ALPHA
Custom
Cumulative cpu time consumed by the node in core-seconds
None
None
node_ipam_controller_cidrset_allocation_tries_per_request
ALPHA
Histogram
None
node_ipam_controller_cidrset_cidrs_allocations_total
ALPHA
Counter
Counter measuring total number of CIDR allocations.
None
node_ipam_controller_cidrset_cidrs_releases_total
ALPHA
Counter
Counter measuring total number of CIDR releases.
None
node_ipam_controller_cidrset_usage_cidrs
ALPHA
Gauge
Gauge measuring percentage of allocated CIDRs.
None
node_ipam_controller_multicidrset_allocation_tries_per_request
ALPHA
Histogram
Histogram measuring CIDR allocation tries per request.
None
node_ipam_controller_multicidrset_cidrs_allocations_total
ALPHA
Counter
None
node_ipam_controller_multicidrset_cidrs_releases_total
ALPHA
Counter
None
node_ipam_controller_multicidrset_usage_cidrs
ALPHA
Gauge
None
node_memory_working_set_bytes
ALPHA
Custom
Current working set of the node in bytes
None
None
number_of_l4_ilbs
ALPHA
Gauge
Number of L4 ILBs
feature
None
plugin_manager_total_plugins
ALPHA
Custom
Number of plugins in Plugin Manager
socket_path
state
None
pod_cpu_usage_seconds_total
ALPHA
Custom
Cumulative cpu time consumed by the pod in core-seconds
pod
namespace
None
pod_memory_working_set_bytes
ALPHA
Custom
Current working set of the pod in bytes
pod
namespace
None
pod_security_errors_total
ALPHA
Counter
Number of errors preventing normal evaluation. Non-fatal errors may result in the latest restricted profile being used for evaluation.
fatal
request_operation
resource
None
pod_security_evaluations_total
ALPHA
Counter
Number of policy evaluations that occurred, not counting ignored or exempt requests.
decision
mode
policy_level
policy_version
resource
None
pod_security_exemptions_total
ALPHA
Counter
Number of exempt requests, not counting ignored or out of scope requests.
resource
None
prober_probe_duration_seconds
ALPHA
Histogram
Duration in seconds for a probe response.
container
namespace
pod
probe_type
None
prober_probe_total
ALPHA
Counter
Cumulative number of a liveness, readiness or startup probe for a container by result.
container
namespace
pod
pod_uid
result
None
pv_collector_bound_pv_count
ALPHA
Custom
Gauge measuring number of persistent volume currently bound
storage_class
None
pv_collector_bound_pvc_count
ALPHA
Custom
Gauge measuring number of persistent volume claim currently bound
namespace
None
pv_collector_total_pv_count
ALPHA
Custom
Gauge measuring total number of persistent volumes
volume_mode
None
pv_collector_unbound_pv_count
ALPHA
Custom
Gauge measuring number of persistent volume currently unbound
None
pv_collector_unbound_pvc_count
ALPHA
Custom
Gauge measuring number of persistent volume claim currently unbound
namespace
None
replicaset_controller_sorting_deletion_age_ratio
ALPHA
Histogram
The ratio of chosen deleted pod's ages to the current youngest pod's age (at the time). Should be &lt;2.The intent of this metric is to measure the rough efficacy of the LogarithmicScaleDown feature gate's effect onthe sorting (and deletion) of pods when a replicaset scales down. This only considers Ready pods when calculating and reporting.
None
None
rest_client_exec_plugin_call_total
ALPHA
Counter
Numbe

r of calls to an exec plugin, partitioned by the type of event encountered (no_error, plugin_execution_error, plugin_not_found_error, client_internal_error) and an optional exit code. The exit code will be set to 0 if and only if the plugin call was successful.
call_status
code
None
rest_client_exec_plugin_certificate_rotation_age
ALPHA
Histogram
Histogram of the number of seconds the last auth exec plugin client certificate lived before being rotated. If auth exec plugin client certificates are unused, histogram will contain no data.
None
None
rest_client_exec_plugin_ttl_seconds
ALPHA
Gauge
Gauge of the shortest TTL (time-to-live) of the client certificate(s) managed by the auth exec plugin. The value is in seconds until certificate expiry (negative if already expired). If auth exec plugins are unused or manage no TLS certificates, the value will be +INF.
None
None
rest_client_rate_limiter_duration_seconds
ALPHA
Histogram
Client side rate limiter latency in seconds. Broken down by verb, and host.
host
verb
None
rest_client_request_duration_seconds
ALPHA
Histogram
Request latency in seconds. Broken down by verb, and host.
host
verb
None
rest_client_request_size_bytes
ALPHA
Histogram
Request size in bytes. Broken down by verb and host.
host
verb
None
rest_client_requests_total
ALPHA
Counter
Number of HTTP requests, partitioned by status code, method, and host.
code
host
method
None
rest_client_response_size_bytes
ALPHA
Histogram
Response size in bytes. Broken down by verb and host.
host
verb
None
retroactive_storageclass_errors_total
ALPHA
Counter
Total number of failed retroactive StorageClass assignments to persistent volume claim
None
None
retroactive_storageclass_total
ALPHA
Counter
Total number of retroactive StorageClass assignments to persistent volume claim
None
None
root_ca_cert_publisher_sync_duration_seconds
ALPHA
Histogram
Number of namespace syncs happened in root ca cert publisher.
code
None
root_ca_cert_publisher_sync_total
ALPHA
Counter
code
None
running_managed_controllers
ALPHA
Gauge
Indicates where instances of a controller are currently running
manager
name
None
scheduler_e2e_scheduling_duration_seconds
ALPHA
Histogram
E2e scheduling latency in seconds (scheduling algorithm + binding). This metric is replaced by scheduling_attempt_duration_seconds.
profile
result
None
scheduler_goroutines
ALPHA
Gauge
Number of running goroutines split by the work they do such as binding.
operation
None
scheduler_permit_wait_duration_seconds
ALPHA
Histogram
Duration of waiting on permit.
result
None
scheduler_plugin_execution_duration_seconds
ALPHA
Histogram
Duration for running a plugin at a specific extension point.
plugin
status
None
scheduler_scheduler_cache_size
ALPHA
Gauge
Number of nodes, pods, and assumed (bound) pods in the scheduler cache.
type
None
scheduler_scheduler_goroutines
ALPHA
Gauge
Number of running goroutines split by the work they do such as binding. This metric is replaced by the \"goroutines\" metric.
work
None
scheduler_scheduling_algorithm_duration_seconds
ALPHA
Histogram
Scheduling algorithm latency in seconds
None
None
scheduler_unschedulable_pods
ALPHA
Gauge
The number of unschedulable pods broken down by plugin name. A pod will increment the gauge for all plugins that caused it to not schedule and so this metric have meaning only when broken down by plugin.
plugin
profile
None
scheduler_volume_binder_cache_requests_total
ALPHA
Counter
Total number for request volume binding cache
operation
None
scheduler_volume_scheduling_stage_error_total
ALPHA
Counter
Volume scheduling stage error count
operation
None
scrape_error
ALPHA
Custom
1 if there was an error while getting container metrics, 0 otherwise
None
None
service_controller_nodesync_latency_seconds
ALPHA
Histogram
A metric measuring the latency for nodesync which updates loadbalancer hosts on cluster node updates.
None
None
service_controller_update_loadbalancer_host_l

atency_seconds
ALPHA
Histogram
A metric measuring the latency for updating each load balancer hosts.
None
None
serviceaccount_legacy_tokens_total
ALPHA
Counter
Cumulative legacy service account tokens used
None
None
serviceaccount_stale_tokens_total
ALPHA
Counter
Cumulative stale projected service account tokens used
None
None
serviceaccount_valid_tokens_total
ALPHA
Counter
Cumulative valid projected service account tokens used
None
None
storage_count_attachable_volumes_in_use
ALPHA
Custom
Measure number of volumes in use
node
volume_plugin
None
storage_operation_duration_seconds
ALPHA
Histogram
Storage operation duration
migrated
status
None
ttl_after_finished_controller_job_deletion_duration_seconds
ALPHA
Histogram
The time it took to delete the job since it became eligible for deletion
None
None
volume_manager_selinux_container_errors_total
ALPHA
Gauge
Number of errors when kubelet cannot compute SELinux context for a container. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of containers.
None
None
volume_manager_selinux_container_warnings_total
ALPHA
Gauge
Number of errors when kubelet cannot compute SELinux context for a container that are ignored. They will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
None
None
volume_manager_selinux_pod_context_mismatch_errors_total
ALPHA
Gauge
Number of errors when a Pod defines different SELinux contexts for its containers that use the same volume. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of Pods.
None
None
volume_manager_selinux_pod_context_mismatch_warnings_total
ALPHA
Gauge
Number of errors when a Pod defines different SELinux contexts for its containers that use the same volume. They are not errors yet, but they will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
None
None
volume_manager_selinux_volume_context_mismatch_errors_total
ALPHA
Gauge
Number of errors when a Pod uses a volume that is already mounted with a different SELinux context than the Pod needs. Kubelet can't start such a Pod then and it will retry, therefore value of this metric may not represent the actual nr. of Pods.
None
None
volume_manager_selinux_volume_context_mismatch_warnings_total
ALPHA
Gauge
Number of errors when a Pod uses a volume that is already mounted with a different SELinux context than the Pod needs. They are not errors yet, but they will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
None
None
volume_manager_selinux_volumes_admitted_total
ALPHA
Gauge
Number of volumes whose SELinux context was fine and will be mounted with mount -o context option.
None
None
volume_manager_total_volumes
ALPHA
Custom
Number of volumes in Volume Manager
state
None
volume_operation_total_errors
ALPHA
Counter
Total volume operation errors
None
volume_operation_total_seconds
ALPHA
Histogram
Storage operation end to end duration in seconds
None
watch_cache_capacity
ALPHA
Gauge
Total capacity of watch cache broken by resource type.
resource
None
watch_cache_capacity_decrease_total
ALPHA
Counter
Total number of watch cache capacity decrease events broken by resource type.
resource
None
watch_cache_capacity_increase_total
ALPHA
Counter
Total number of watch cache capacity increase events broken by resource type.
resource
None
workqueue_adds_total
ALPHA
Counter
Total number of adds handled by workqueue
name
None
workqueue_depth
ALPHA
Gauge
Current depth of workqueue
name
None
workqueue_longest_running_processor_seconds
ALPHA
Gauge
How many seconds has the longest running processor for workqueue been running.
name
None
workqueue_queue_duration_seconds
ALPHA
Histogram
How long in seconds an item stays in workqueue before b

eing requested.
name
None
workqueue_retries_total
ALPHA
Counter
Total number of retries handled by workqueue
name
None
workqueue_unfinished_work_seconds
ALPHA
Gauge
How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
name
None
workqueue_work_duration_seconds
ALPHA
Histogram
How long in seconds processing an item from workqueue takes.
name
None
7 - Kubernetes Issues and Security
7.1 - Kubernetes Issue Tracker
.
.
Official
7.2 - Kubernetes Security and Disclosure Information
This page describes Kubernetes security and disclosure information.
Security Announcements
Join the
kubernetes-security-announce
group for emails about security and major API announcements.
Report a Vulnerability
We're extremely grateful for security researchers and users that report vulnerabilities to the Kubernetes Open Source Community. All reports are thoroughly investigated by a set of community volunteers.
To make a report, submit your vulnerability to the
Kubernetes bug bounty program
. This allows triage and handling of the vulnerability with standardized response times.
You can also email the private
security@kubernetes.io
list with the security details and the details expected for
all Kubernetes bug reports
.
You may encrypt your email to this list using the GPG keys of the
Security Response Committee members
. Encryption using GPG is NOT required to make a disclosure.
When Should I Report a Vulnerability?
You think you discovered a potential security vulnerability in Kubernetes
You are unsure how a vulnerability affects Kubernetes
You think you discovered a vulnerability in another project that Kubernetes depends on
For projects with their own vulnerability reporting and disclosure process, please report it directly there
When Should I NOT Report a Vulnerability?
You need help tuning Kubernetes components for security
You need help applying security related updates
Your issue is not security related
Security Vulnerability Response
Each report is acknowledged and analyzed by Security Response Committee members within 3 working days. This will set off the
Security Release Process
.
Any vulnerability information shared with Security Response Committee stays within Kubernetes project and will not be disseminated to other projects unless it is necessary to get the issue fixed.
As the security issue moves from triage, to identified fix, to release planning we will keep the reporter updated.
Public Disclosure Timing
A public disclosure date is negotiated by the Kubernetes Security Response Committee and the bug submitter. We prefer to fully disclose the bug as soon as possible once a user mitigation is available. It is reasonable to delay disclosure when the bug or the fix is not yet fully understood, the solution is not well-tested, or for vendor coordination. The timeframe for disclosure is from immediate (especially if it's already publicly known) to a few weeks. For a vulnerability with a straightforward mitigation, we expect report date to disclosure date to be on the order of 7 days. The Kubernetes Security Response Committee holds the final say when setting a disclosure date.
7.3 - Official CVE Feed
This is a community maintained list of official CVEs announced by
the Kubernetes Security Response Committee. See
The Kubernetes project publishes a programmatically accessible
JSON Feed
of

published security issues. You can access it by executing the following command:
curl -Lv https://k8s.io/docs/reference/issues-security/official-cve-feed/index.json
Official Kubernetes CVE List
CVE ID
Issue Summary
CVE GitHub Issue URL
CVE-2022-3294
Node address isn't always verified when proxying
#113757
CVE-2022-3162
Unauthorized read of Custom Resources
#113756
CVE-2022-3172
Aggregated API server can cause clients to be redirected (SSRF)
#112513
CVE-2021-25749
`runAsNonRoot` logic bypass for Windows containers
#112192
CVE-2021-25741
Symlink Exchange Can Allow Host Filesystem Access
#104980
CVE-2021-25737
Holes in EndpointSlice Validation Enable Host Network Hijack
#102106
CVE-2021-3121
Processes may panic upon receipt of malicious protobuf messages
#101435
CVE-2021-25735
Validating Admission Webhook does not observe some previous fields
#100096
CVE-2020-8554
Man in the middle using LoadBalancer or ExternalIPs
#97076
CVE-2020-8566
Ceph RBD adminSecrets exposed in logs when loglevel &gt;= 4
#95624
CVE-2020-8565
Incomplete fix for CVE-2019-11250 allows for token leak in logs when logLevel &gt;= 9
#95623
CVE-2020-8564
Docker config secrets leaked when file is malformed and log level &gt;= 4
#95622
CVE-2020-8563
Secret leaks in kube-controller-manager when using vSphere provider
#95621
CVE-2020-8557
Node disk DOS by writing to container /etc/hosts
#93032
CVE-2020-8559
Privilege escalation from compromised node to cluster
#92914
CVE-2020-8558
Node setting allows for neighboring hosts to bypass localhost boundary
#92315
CVE-2020-8555
Half-Blind SSRF in kube-controller-manager
#91542
CVE-2020-10749
IPv4 only clusters susceptible to MitM attacks via IPv6 rogue router advertisements
#91507
CVE-2019-11254
kube-apiserver Denial of Service vulnerability from malicious YAML payloads
#89535
CVE-2020-8552
apiserver DoS (oom)
#89378
CVE-2020-8551
Kubelet DoS via API
#89377
CVE-2019-11251
kubectl cp symlink vulnerability
#87773
CVE-2018-1002102
Unvalidated redirect
#85867
CVE-2019-11255
CSI volume snapshot, cloning and resizing features can result in unauthorized volume data access or mutation
#85233
CVE-2019-11253
Kubernetes API Server JSON/YAML parsing vulnerable to resource exhaustion attack
#83253
CVE-2019-11250
Bearer tokens are revealed in logs
#81114
CVE-2019-11248
/debug/pprof exposed on kubelet's healthz port
#81023
CVE-2019-11249
Incomplete fixes for CVE-2019-1002101 and CVE-2019-11246, kubectl cp potential directory traversal
#80984
CVE-2019-11247
API server allows access to custom resources via wrong scope
#80983
CVE-2019-11245
container uid changes to root after first restart or if image is already pulled to the node
#78308
CVE-2019-11243
rest.AnonymousClientConfig() does not remove the serviceaccount credentials from config created by rest.InClusterConfig()
#76797
CVE-2019-11244
`kubectl:-http-cache=&lt;world-accessible dir&gt;` creates world-writeable cached schema files
#76676
CVE-2019-1002100
json-patch requests can exhaust apiserver resources
#74534
CVE-2018-1002105
proxy request handling in kube-apiserver can leave vulnerable TCP connections
#71411
CVE-2018-1002101
smb mount security issue
#65750
CVE-2018-1002100
Kubectl copy doesn't check for paths outside of it's destination directory.
#61297
CVE-2017-1002102
atomic writer volume handling allows arbitrary file deletion in host filesystem
#60814
CVE-2017-1002101
subpath volume mount handling allows arbitrary file access in host filesystem
#60813
CVE-2017-1002100
Azure PV should be Private scope not Container scope
#47611
CVE-2017-1000056
PodSecurityPolicy admission plugin authorizes incorrectly
#43459
This feed is auto-refreshing with a noticeable but small lag (minutes to hours)
from the time a CVE is announced to the time it is accessible in this feed.
The source of truth of this feed is a set of GitHub Issues, filtered by a controlled and
restricted label
official-cve-feed
. The raw data is stored in a Google Cloud
Bucket which is writable only by a small number of tr

usted members of the
Community.
8 - Node Reference Information
a list of
.
8.1 - Kubelet Checkpoint API
Checkpointing a container is the functionality to create a stateful copy of a
running container. Once you have a stateful copy of a container, you could
move it to a different computer for debugging or similar purposes.
If you move the checkpointed container data to a computer that's able to restore
it, that restored container continues to run at exactly the same
point it was checkpointed. You can also inspect the saved data, provided that you
have suitable tools for doing so.
Creating a checkpoint of a container might have security implications. Typically
a checkpoint contains all memory pages of all processes in the checkpointed
container. This means that everything that used to be in memory is now available
on the local disk. This includes all private data and possibly keys used for
encryption. The underlying CRI implementations (the container runtime on that node)
should create the checkpoint archive to be only accessible by the
root
user. It
is still important to remember if the checkpoint archive is transferred to another
system all memory pages will be readable by the owner of the checkpoint archive.
post
checkpoint the specified container
Tell the kubelet to checkpoint a specific container from the specified Pod.
for more information about how access to the kubelet checkpoint interface is
controlled.
The kubelet will request a checkpoint from the underlying
CRI
implementation. In the checkpoint
request the kubelet will specify the name of the checkpoint archive as
checkpoint-&lt;podFullName&gt;-&lt;containerName&gt;-&lt;timestamp&gt;.tar
and also request to
store the checkpoint archive in the
checkpoints
directory below its root
directory (as defined by
). This defaults to
/var/lib/kubelet/checkpoints
.
The checkpoint archive is in
tar
format, and could be listed using an implementation of
tar
. The contents of the
archive depend on the underlying CRI implementation (the container runtime on that node).
POST /checkpoint/{namespace}/{pod}/{container}
namespace
(
in path
Namespace
pod
(
in path
Pod
container
(
in path
Container
timeout
(
in query
): integer
Timeout in seconds to wait until the checkpoint creation is finished.
If zero or no timeout is specfied the default
CRI
timeout value will be used. Checkpoint
creation time depends directly on the used memory of the container.
The more memory a container uses the more time is required to create
the corresponding checkpoint.
Response
200: OK
404: Not Found (if the
ContainerCheckpoint
feature gate is disabled)
404: Not Found (if the specified
namespace
,
pod
or
container
cannot be found)
500: Internal Server Error (if the CRI implementation encounter an error during checkpointing (see error message for further details))
500: Internal Server Error (if the CRI implementation does not implement the checkpoint CRI API (see error message for further details))
8.2 - Kubelet Device Manager API Versions
,
v1alpha1
v1beta1
-
✓
-
✓
-
✓
-
✓
-
✓
-
✓
Key:
✓
+
Kuber

netes cluster, either because the device plugin API has added additional new API
-
8.3 - Articles on dockershim Removal and on Using CRI-compatible Runtimes
This is a list of articles and other pages that are either
about the Kubernetes' deprecation and removal of
,
or about using CRI-compatible container runtimes,
in connection with that removal.
Kubernetes project
Kubernetes blog:
(originally published 2020/12/02)
Updated: Dockershim Removal FAQ
(updated published 2022/02/17)
Kubernetes is Moving on From Dockershim: Commitments and Next Steps
(published 2022/01/07)
Dockershim removal is coming. Are you ready?
(published 2021/11/12)
Kubernetes documentation:
Kubernetes enhancement proposal:
KEP-2221: Removing dockershim from kubelet
Kubernetes enhancement proposal issue:
Removing dockershim from kubelet
(
k/enhancements#2221
)
You can provide feedback via the GitHub issue
Dockershim removal feedback &amp; issues
. (
k/kubernetes/#106917
)
External sources
Amazon Web Services EKS documentation:
Amazon EKS is ending support for Dockershim
CNCF conference video:
Lessons Learned Migrating Kubernetes from Docker to containerd Runtime
(Ana Caylin, at KubeCon Europe 2019)
Docker.com blog:
What developers need to know about Docker, Docker Engine, and Kubernetes v1.20
(published 2020/12/04)
"
Google Open Source
" channel on YouTube:
Learn Kubernetes with Google - Migrating from Dockershim to Containerd
Microsoft Apps on Azure blog:
Dockershim deprecation and AKS
(published 2022/01/21)
Mirantis blog:
The Future of Dockershim is cri-dockerd
(published 2021/04/21)
Mirantis:
Mirantis/cri-dockerd
Git repository (on GitHub)
Tripwire:
How Dockershim’s Forthcoming Deprecation Affects Your Kubernetes
(published 2021/07/01)
9 - Networking Reference
9.1 - Protocols for Services
If you configure a
Service
,
you can select from any network protocol that Kubernetes supports.
Kubernetes supports the following protocols with Services:
SCTP
TCP
(the default)
UDP
When you define a Service, you can also specify the
application protocol
that it uses.
This document details some special cases, all of them typically using TCP
as a transport protocol:
HTTP
and
HTTPS
PROXY protocol
TLS
termination at the load balancer
Supported protocols
There are 3 valid values for the
protocol
of a port for a Service:
SCTP
When using a network plugin that supports SCTP traffic, you can use SCTP for
most Services. For
Services, SCTP support depends on the cloud
provider offering this facility. (Most do not).
SCTP is not supported on nodes that run Windows.
Support for multihomed SCTP associations
The support of multihomed SCTP associations requires that the CNI plugin can support the assignment of multiple interfaces and IP addresses to a Pod.
NAT for multihomed SCTP associations requires special logic in the corresponding kernel modules.
TCP
You can use TCP for any kind of Service, and it's the default network protocol.
UDP
You can use UDP for most Services. For
Services,
UDP support depends on the cloud provider offering this facility.
Special cases
HTTP
If your cloud provider supports it, you can use a Service in LoadBalancer mode to
configure a load balancer outside of your Kubernetes cluste

r, in a special mode
where your cloud provider's load balancer implements HTTP / HTTPS reverse proxying,
with traffic forwarded to the backend endpoints for that Service.
Typically, you set the protocol for the Service to
TCP
and add an
(usually specific to your cloud provider) that configures the load balancer
to handle traffic at the HTTP level.
This configuration might also include serving HTTPS (HTTP over TLS) and
reverse-proxying plain HTTP to your workload.
Note:
You can also use an
Ingress
to expose
HTTP/HTTPS Services.
You might additionally want to specify that the
of the connection is
http
or
https
. Use
http
if the session from the
load balancer to your workload is HTTP without TLS, and use
https
if the
session from the load balancer to your workload uses TLS encryption.
If your cloud provider supports it, you can use a Service set to
to configure a load balancer outside of Kubernetes itself, that will forward connections
wrapped with the
.
The load balancer then sends an initial series of octets describing the
incoming connection, similar to this example (PROXY protocol v1):
PROXY TCP4 192.0.2.202 10.0.42.7 12345 7\r\n
The data after the proxy protocol preamble are the original
data from the client. When either side closes the connection,
the load balancer also triggers a connection close and sends
any remaining data where feasible.
Typically, you define a Service with the protocol to
TCP
.
You also set an annotation, specific to your
cloud provider, that configures the load balancer to wrap each incoming connection in the PROXY protocol.
TLS
as
a way to set up external reverse proxying, where the connection from client to load
balancer is TLS encrypted and the load balancer is the TLS server peer.
The connection from the load balancer to your workload can also be TLS,
or might be plain text. The exact options available to you depend on your
cloud provider or custom Service implementation.
Typically, you set the protocol to
TCP
and set an annotation
to act as a TLS server. You would configure the TLS identity (as server,
and possibly also as a client that connects to your workload) using
mechanisms that are specific to your cloud provider.
9.2 - Ports and Protocols
When running Kubernetes in an environment with strict network boundaries, such
as on-premises datacenter with physical network firewalls or Virtual
Networks in Public Cloud, it is useful to be aware of the ports and protocols
used by Kubernetes components.
Control plane
Protocol
Direction
Port Range
Purpose
Used By
TCP
Inbound
6443
Kubernetes API server
All
TCP
Inbound
2379-2380
etcd server client API
kube-apiserver, etcd
TCP
Inbound
10250
Self, Control plane
TCP
Inbound
10259
Self
TCP
Inbound
10257
Self
Although etcd ports are included in control plane section, you can also host your own
etcd cluster externally or on custom ports.
Worker node(s)
Protocol
Direction
Port Range
Purpose
Used By
TCP
Inbound
10250
TCP
Inbound
30000-32767
NodePort Services†
All
† Default port range for
NodePort Services
.
All default port numbers can be overridden. When custom ports are used those
ports need to be open instead of defaults mentioned here.
One common example is API server port that is sometimes switched
to 443. Alternatively, the default port is kept as is and API server is put
behind a load balancer that listens on 443 and routes the requests to API server
on the default port.
9.3 - Virtual IPs and Service Proxies
Every
node
in a Kubernetes
cluster runs a
(unless you have deployed your own alternative component in place of
).
The
component is responsible for implementing a
virtual IP
mechanism for
Services
of
t

ype
other than
ExternalName
.
A question that pops up every now and then is why Kubernetes relies on
proxying to forward inbound traffic to backends. What about other
approaches? For example, would it be possible to configure DNS records that
have multiple A values (or AAAA for IPv6), and rely on round-robin name
resolution?
There are a few reasons for using proxying for Services:
There is a long history of DNS implementations not respecting record TTLs,
and caching the results of name lookups after they should have expired.
Some apps do DNS lookups only once and cache the results indefinitely.
Even if apps and libraries did proper re-resolution, the low or zero TTLs
on the DNS records could impose a high load on DNS that then becomes
difficult to manage.
Later in this page you can read about how various kube-proxy implementations work.
Overall, you should note that, when running
, kernel level rules may be modified
(for example, iptables rules might get created), which won't get cleaned up, in some
cases until you reboot. Thus, running kube-proxy is something that should only be done
by an administrator which understands the consequences of having a low level, privileged
network proxying service on a computer. Although the
executable supports a
cleanup
function, this function is not an official feature and thus is only available
to use as-is.
Some of the details in this reference refer to an example: the back end Pods for a stateless
image-processing workload, running with three replicas. Those replicas are
fungible—frontends do not care which backend they use. While the actual Pods that
compose the backend set may change, the frontend clients should not need to be aware of that,
nor should they need to keep track of the set of backends themselves.
Proxy modes
Note that the kube-proxy starts up in different modes, which are determined by its configuration.
The kube-proxy's configuration is done via a ConfigMap, and the ConfigMap for
kube-proxy effectively deprecates the behavior for almost all of the flags for
the kube-proxy.
The ConfigMap for the kube-proxy does not support live reloading of configuration.
The ConfigMap parameters for the kube-proxy cannot all be validated and verified on startup.
For example, if your operating system doesn't allow you to run iptables commands,
the standard kernel kube-proxy implementation will not work.
iptables
proxy mode
In this mode, kube-proxy watches the Kubernetes control plane for the addition and
removal of Service and EndpointSlice objects. For each Service, it installs
iptables rules, which capture traffic to the Service's
clusterIP
and
port
,
and redirect that traffic to one of the Service's
backend sets. For each endpoint, it installs iptables rules which
select a backend Pod.
By default, kube-proxy in iptables mode chooses a backend at random.
Using iptables to handle traffic has a lower system overhead, because traffic
is handled by Linux netfilter without the need to switch between userspace and the
kernel space. This approach is also likely to be more reliable.
If kube-proxy is running in iptables mode and the first Pod that's selected
does not respond, the connection fails. This is different from the old
userspace
mode: in that scenario, kube-proxy would detect that the connection to the first
Pod had failed and would automatically retry with a different backend Pod.
You can use Pod
readiness probes
to verify that backend Pods are working OK, so that kube-proxy in iptables mode
only sees backends that test out as healthy. Doing this means you avoid
having traffic sent via kube-proxy to a Pod that's known to have failed.
Services overview diagram for iptables proxy
Example
As an example, consider the image processing application described
earlier
in the page.
When the backend Service is created, the Kubernetes control plane assigns a virtual
IP address, for example 10.0.0.1. For this example, assume that the
Service port is 1234.
All of the kube-proxy i

nstances in the cluster observe the creation of the new
Service.
When kube-proxy on a node sees a new Service, it installs a series of iptables rules
which redirect from the virtual IP address to more iptables rules, defined per Service.
The per-Service rules link to further rules for each backend endpoint, and the per-
endpoint rules redirect traffic (using destination NAT) to the backends.
When a client connects to the Service's virtual IP address the iptables rule kicks in.
A backend is chosen (either based on session affinity or randomly) and packets are
redirected to the backend without rewriting the client IP address.
This same basic flow executes when traffic comes in through a node-port or
through a load-balancer, though in those cases the client IP address does get altered.
Optimizing iptables mode performance
In large clusters (with tens of thousands of Pods and Services), the
iptables mode of kube-proxy may take a long time to update the rules
in the kernel when Services (or their EndpointSlices) change. You can adjust the syncing
behavior of kube-proxy via options in the
iptables
section
of the
(which you specify via
kube-proxy --config &lt;path&gt;
):
...
iptables
:
:
1s
:
30s
...
The
parameter sets the minimum duration between
attempts to resynchronize iptables rules with the kernel. If it is
0s
, then kube-proxy will always immediately synchronize the rules
every time any Service or Endpoint changes. This works fine in very
small clusters, but it results in a lot of redundant work when lots of
things change in a small time period. For example, if you have a
Service backed by a Deployment with 100 pods, and you delete the
Deployment, then with
minSyncPeriod: 0s
, kube-proxy would end up
removing the Service's Endpoints from the iptables rules one by one,
for a total of 100 updates. With a larger
, multiple
Pod deletion events would get aggregated together, so kube-proxy might
instead end up making, say, 5 updates, each removing 20 endpoints,
which will be much more efficient in terms of CPU, and result in the
full set of changes being synchronized faster.
The larger the value of
, the more work that can be
aggregated, but the downside is that each individual change may end up
waiting up to the full
before being processed, meaning
that the iptables rules spend more time being out-of-sync with the
current apiserver state.
The default value of
1s
is a good compromise for small and medium
clusters. In large clusters, it may be necessary to set it to a larger
value. (Especially, if kube-proxy's
sync_proxy_rules_duration_seconds
metric indicates an average
time much larger than 1 second, then bumping up
may
make updates more efficient.)
The
parameter controls a handful of synchronization
operations that are not directly related to changes in individual
Services and Endpoints. In particular, it controls how quickly
kube-proxy notices if an external component has interfered with
kube-proxy's iptables rules. In large clusters, kube-proxy also only
performs certain cleanup operations once every
to avoid
unnecessary work.
For the most part, increasing
is not expected to have much
impact on performance, but in the past, it was sometimes useful to set
it to a very large value (eg,
1h
). This is no longer recommended,
and is likely to hurt functionality more than it improves performance.
Experimental performance improvements
In Kubernetes 1.26, some new performance improvements were made to the
iptables proxy mode, but they are not enabled by default (and should
probably not be enabled in production clusters yet). To try them out,
enable the
MinimizeIPTablesRestore
feature
gate
for
kube-proxy with
--feature-gates=MinimizeIPTablesRestore=true,…
.
If you enable that feature gate and you were previously overrid

ing
, you should try removing that override and letting
kube-proxy use the default value (
1s
) or at least a smaller value
than you were using before.
If you notice kube-proxy's
sync_proxy_rules_iptables_restore_failures_total
or
sync_proxy_rules_iptables_partial_restore_failures_total
metrics
increasing after enabling this feature, that likely indicates you are
encountering bugs in the feature and you should file a bug report.
IPVS proxy mode
In
ipvs
mode, kube-proxy watches Kubernetes Services and EndpointSlices,
calls
netlink
interface to create IPVS rules accordingly and synchronizes
IPVS rules with Kubernetes Services and EndpointSlices periodically.
This control loop ensures that IPVS status matches the desired
state.
When accessing a Service, IPVS directs traffic to one of the backend Pods.
The IPVS proxy mode is based on netfilter hook function that is similar to
iptables mode, but uses a hash table as the underlying data structure and works
in the kernel space.
That means kube-proxy in IPVS mode redirects traffic with lower latency than
kube-proxy in iptables mode, with much better performance when synchronizing
proxy rules. Compared to the other proxy modes, IPVS mode also supports a
higher throughput of network traffic.
IPVS provides more options for balancing traffic to backend Pods;
these are:
rr
: round-robin
lc
: least connection (smallest number of open connections)
dh
: destination hashing
sh
: source hashing
sed
: shortest expected delay
nq
: never queue
Note:
To run kube-proxy in IPVS mode, you must make IPVS available on
the node before starting kube-proxy.
When kube-proxy starts in IPVS proxy mode, it verifies whether IPVS
kernel modules are available. If the IPVS kernel modules are not detected, then kube-proxy
falls back to running in iptables proxy mode.
Services overview diagram for IPVS proxy
Session affinity
In these proxy models, the traffic bound for the Service's IP:Port is
proxied to an appropriate backend without the clients knowing anything
about Kubernetes or Services or Pods.
If you want to make sure that connections from a particular client
are passed to the same Pod each time, you can select the session affinity based
on the client's IP addresses by setting
.spec.sessionAffinity
to
ClientIP
for a Service (the default is
None
).
Session stickiness timeout
You can also set the maximum session sticky time by setting
.spec.sessionAffinityConfig.clientIP.timeoutSeconds
appropriately for a Service.
(the default value is 10800, which works out to be 3 hours).
Note:
On Windows, setting the maximum session sticky time for Services is not supported.
IP address assignment to Services
Unlike Pod IP addresses, which actually route to a fixed destination,
Service IPs are not actually answered by a single host. Instead, kube-proxy
uses packet processing logic (such as Linux iptables) to define
virtual
IP
addresses which are transparently redirected as needed.
When clients connect to the VIP, their traffic is automatically transported to an
appropriate endpoint. The environment variables and DNS for Services are actually
populated in terms of the Service's virtual IP address (and port).
Avoiding collisions
One of the primary philosophies of Kubernetes is that you should not be
exposed to situations that could cause your actions to fail through no fault
of your own. For the design of the Service resource, this means not making
you choose your own port number if that choice might collide with
someone else's choice. That is an isolation failure.
In order to allow you to choose a port number for your Services, we must
ensure that no two Services can collide. Kubernetes does that by allocating each
Service its own IP address from within the
CIDR range that is configured for the API server.
To ensure each Service receives a unique IP, an internal allocator atomically
updates a global allocation map in
etcd
prior to creating each Service. The map object must exist in the re

gistry for
Services to get IP address assignments, otherwise creations will
fail with a message indicating an IP address could not be allocated.
In the control plane, a background controller is responsible for creating that
map (needed to support migrating from older versions of Kubernetes that used
in-memory locking). Kubernetes also uses controllers to check for invalid
assignments (e.g. due to administrator intervention) and for cleaning up allocated
IP addresses that are no longer used by any Services.
IP address ranges for Service virtual IP addresses
Kubernetes divides the
ClusterIP
range into two bands, based on
the size of the configured
by using the following formula
min(max(16, cidrSize / 16), 256)
. That formula paraphrases as
never less than 16 or
more than 256, with a graduated step function between them
.
Kubernetes prefers to allocate dynamic IP addresses to Services by choosing from the upper band,
which means that if you want to assign a specific IP address to a
type: ClusterIP
Service, you should manually assign an IP address from the
lower
band. That approach
reduces the risk of a conflict over allocation.
ServiceIPStaticSubrange
then Kubernetes
uses a single shared pool for both manually and dynamically assigned IP addresses,
that are used for
Services.
Traffic policies
.spec.internalTrafficPolicy
and
.spec.externalTrafficPolicy
fields
to control how Kubernetes routes traffic to healthy (“ready”) backends.
Internal traffic policy
field to control how traffic from
internal sources is routed. Valid values are
Cluster
and
Local
. Set the field to
Cluster
to route internal traffic to all ready endpoints and
Local
to only route
to ready node-local endpoints. If the traffic policy is
Local
and there are no
node-local endpoints, traffic is dropped by kube-proxy.
External traffic policy
external sources is routed. Valid values are
Cluster
and
Local
. Set the field
to
Cluster
to route external traffic to all ready endpoints and
Local
to only
route to ready node-local endpoints. If the traffic policy is
Local
and there are
are no node-local endpoints, the kube-proxy does not forward any traffic for the
relevant Service.
Traffic to terminating endpoints
If the
ProxyTerminatingEndpoints
is enabled in kube-proxy and the traffic policy is
Local
, that node's
kube-proxy uses a more complicated algorithm to select endpoints for a Service.
With the feature enabled, kube-proxy checks if the node
has local endpoints and whether or not all the local endpoints are marked as terminating.
If there are local endpoints and
all
of them are terminating, then kube-proxy
will forward traffic to those terminating endpoints. Otherwise, kube-proxy will always
prefer forwarding traffic to endpoints that are not terminating.
This forwarding behavior for terminating endpoints exist to allow
NodePort
and
Services to gracefully drain connections when using
externalTrafficPolicy: Local
.
As a deployment goes through a rolling update, nodes backing a load balancer may transition from
N to 0 replicas of that deployment. In some cases, external load balancers can send traffic to
a node with 0 replicas in between health check probes. Routing traffic to terminating endpoints
ensures that Node's that are scaling down Pods can gracefully receive and drain traffic to
those terminating Pods. By the time the Pod completes termination, the external load balancer
should have seen the node's health check failing and fully removed the node from the backend pool.
To learn more about Services,
read
.
You can also:
Read about
Services
as a concept
Read about
Ingres

ses
Read the
for the Service API
10 - Setup tools
10.1 - Kubeadm
and
.
or
10.1.1 - Kubeadm Generated
10.1.1.1 -
Synopsis
│ KUBEADM                                                  │
│                                                          │

│ worker# kubeadm join &amp;lt;arguments-returned-from-init&amp;gt;      │

Options
-h, --help
10.1.1.2 -
Synopsis
Options
-h, --help
10.1.1.3 -
Synopsis
Options
-h, --help
10.1.1.4 -
Synopsis

Options
-h, --help
10.1.1.5 -
Synopsis
Examples
Options
-h, --help
10.1.1.6 -
Synopsis
Options
-h, --help
10.1.1.7 -
Synopsis
Options
-h, --help
10.1.1.8 -
Synopsis
Options
--cert-dir string     Default: "/etc/kuber

netes/pki"
-h, --help
10.1.1.9 -
Synopsis
Options
-h, --help
10.1.1.10 -
Synopsis
Options
-h, --help
10.1.1.11 -
Synopsis
Options
Path 

to a kubeadm configuration file.
-h, --help
10.1.1.12 -
Synopsis
Options
-h, --help
10.1.1.13 -
Synopsis
Options
-h, --help
10.1.1.14 -
Synopsis
Options
The path where to save the certi

ficates
-h, --help
10.1.1.15 -
Synopsis
Options
-h, --help
10.1.1.16 -
Synopsis
Options
-h, --help
10.1.1.17 -
Synopsis
Options
-h, --help
help for sche

duler.conf
10.1.1.18 -
Synopsis
Examples

Options
-h, --help
10.1.1.19 -
Synopsis
Options
-h, --help
10.1.1.20 -
Synopsis
Options
-h, --help
10.1.1.21 -
Synopsis
Options
If true, ignore any errors in templates when a field or map key is missing in the template. Only app

lies to golang and jsonpath output formats.
-h, --help
10.1.1.22 -
Synopsis
Options
-h, --help
10.1.1.23 -
Synopsis
Options
-h, --help
10.1.1.24 -
Synopsis
This co

mmand prints configurations for subcommands provided.
Options
-h, --help
10.1.1.25 -
Synopsis
Options
-h, --help
10.1.1.26 -
Synopsis
Options
-h, --help
10.1.1.27 -
Synopsis
  /sa                          Generate a private key for 

signing service account tokens along with its public key
Options
--dry-run
-h, --help
Path to a directory that contains files na

med "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
10.1.1.28 -
Synopsis
Options
-h, --help
10.1.1.29 -
Synopsis
Options
-h, --help
10.1.1.30 -
Synopsis
Options
--dry-run
-h, --help
[E

XPERIMENTAL] The path to the 'real' host root filesystem.
10.1.1.31 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.32 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.33 -
Synopsis
Examples
Options
--dry-run
-h, --help
Skip pri

nting of the default bootstrap token generated by 'kubeadm init'.
10.1.1.34 -
Synopsis
Options
-h, --help
10.1.1.35 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.36 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.37 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.38 -
Synopsis
Options
--api

server-advertise-address string
--dry-run
-h, --help
10.1.1.39 -
Synopsis
Options
--dry-run
-h, --help
help for ca
10.1.1.40 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.41 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.42 -
Synopsis
Generate the certificate for etcd nodes to com

municate with each other, and save them into etcd/peer.crt and etcd/peer.key files.
Options
--dry-run
-h, --help
10.1.1.43 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.44 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.45 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.46 -
Synopsis
Options
--cert-dir string     Default: "/

etc/kubernetes/pki"
-h, --help
help for sa
10.1.1.47 -
Synopsis
Options
-h, --help
10.1.1.48 -
Synopsis
Examples
  
Options
--dry-run
-h, --help
10.1.1.49 -
Synopsis
Options
--apiserver-bind-port int32     Def

ault: 6443
--dry-run
-h, --help
10.1.1.50 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.51 -
Synopsis
Options
--dry-run
-h,

 --help
10.1.1.52 -
Synopsis
Options
-h, --help
10.1.1.53 -
Synopsis
Examples
  
Options
--dry-run
-h, --help
10.1.1.54 -
Synopsis
Options
-h, --help
10.1.1.55 -
Synopsis
Options
--cert-dir string 

    Default: "/etc/kubernetes/pki"
--dry-run
-h, --help
10.1.1.56 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.57 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.58 -
only
Synopsis
only
Options
--dry-run
-h, --help
The path where to save th

e kubeconfig file.
10.1.1.59 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.60 -
Synopsis
Examples
Options
-h, --help
10.1.1.61 -
Synopsis
Examples
Options
--dry-run
-h, --help
10.1.1.62 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.63 -
Synopsis
Examples
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension".

 For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
10.1.1.64 -
Synopsis
Examples
  
Options
--dry-run
-h, --help
10.1.1.65 -
Synopsis
Examples
Options
--dry-run
-h, --help
10.1.1.66 -
Synopsis
Options
-h, --help
10.1.1.67 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.68 -
Synopsis
Options
-h, --help
10.1.1.69 -
Synopsis
Options
--dry-run
-h, --help
help for 

all
10.1.1.70 -
Synopsis
Examples
Options
--dry-run
-h, --help
10.1.1.71 -
Synopsis
Examples
Options
--dry-run
-h, --help
10.1.1.72 -
Synopsis
If you cannot know the CA public key hash ahead of time, 

you can pass
Options
--dry-run
-h, --help
--skip-phase

s strings
10.1.1.73 -
Synopsis
Options
-h, --help
10.1.1.74 -
Synopsis
Examples
Options
-h, --help
10.1.1.75 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.76 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.77 -
Synopsis
Options
Create a new contro

l plane instance on this node
--dry-run
-h, --help
10.1.1.78 -
Synopsis
Options
-h, --help
10.1.1.79 -
Synopsis
Examples
Options
-h, --help
10.1.1.80 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.81 -
Generate the certificates for the ne

w control plane components
Synopsis
Options
--dry-run
-h, --help
10.1.1.82 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.83 -
Synopsis
Options
Fo

r token-based discovery, allow joining without --discovery-token-ca-cert-hash pinning.
--dry-run
-h, --help
10.1.1.84 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.85 -
Synopsis
Options
--dry-run
-h, --help
Use this token for both discovery-token and

 tls-bootstrap-token when those values are not provided.
10.1.1.86 -
Synopsis
Examples
Options
--dry-run
-h, --help
10.1.1.87 -
Synopsis
Options
-h, --help
10.1.1.88 -
Synopsis
Examples
Options
-h, --help
10.1.1.89 -
Synopsis
c

leanup-node        Run cleanup node.
Options
--dry-run
-h, --help
10.1.1.90 -
Synopsis
Options
-h, --help
10.1.1.91 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.92 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.93 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.94 -
Synopsis
(the token serves as a way for the API Serve

r to trust the client), for example for doing the TLS Bootstrap.
Options
--dry-run
-h, --help
10.1.1.95 -
Synopsis
Options
-h, --help
--dry-run
10.1.1.96 -
Synopsis
Options
-h, --help
--dry-run
10.1.1.97 -
Synopsis
yourself as long as it is in t

he format "[a-z0-9]{6}.[a-z0-9]{16}". This
Options
-h, --help
--dry-run
10.1.1.98 -
Synopsis
Options
-h, --help
--dry-run
10.1.1.99 -
Synopsis
Options
-h, --help
10.1.1.100 -
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by ku

bectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
-y, --yes
10.1.1.101 -
Synopsis
Options
-h, --help
10.1.1.102 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.103 -
Synopsis
Options
-h, --help
10.1.1.104 -
Synopsis
kubeadm upgrade node phase control-plane [flag

s]
Options
--dry-run
-h, --help
10.1.1.105 -
Synopsis
Options
--dry-run
-h, --help
10.1.1.106 -
Synopsis
Options
-h, --help
10.1.1.107 -
Synopsis
Options
RootlessControlPlane=true|false (ALPHA - de

fault=false)
-h, --help
10.1.1.108 -
Synopsis
Options
-h, --help
10.1.1.109 -
10.1.2 - kubeadm init
Synopsis
upload-config                Upload the kubeadm and kubelet configuration 

to a ConfigMap
Options
--dry-run
-h, --help
The token to use for establishing bidirectional trust between nodes and control-plane nodes. The format is [a-z0-9]{6}.[a-z0-

9]{16} - e.g. abcdef.0123456789abcdef
.
(
.
run there.
token via
--token
docs.
and
See
Warning:
command.
. The list
--help
--help
=
local
--config
=
=
=
configfi

le.yaml
.
Caution:
--config
contain a
---\n
Mixing
--config
command.
the
command.
.
flag for
a
using
--config
.
Passing
.
Feature
Default
Alpha
Beta
GA
false
1.19
-
-
false
1.22
-
-
true
1.22
1.23
1.25
Note:
true
for
,
,
and
etcd
ConfigMap
true
.
false
or
(during
,
,
IPVS
Passing custom flags to 

control plane components
--config
and
fields.
. If the
ci/latest
)
is used.
.
To provide
.
and
imageTag
to
using
.
,
where
, and/or
imageTag
to
.
topic.
to
and
to
.
=
Note:
with
--config
.
and
For detailed information on certificate management with kubeadm see
.
The
kubeadm
kubelet
by
systemd
.
.
see the
.
Setting the node 

name
kubeadm
flag.
.
in the
,
.
--token
and
.
at
.
phases
or
10.1.3 - kubeadm join
Synopsis
verification. This weakens the kubeadm security model since other node

s
Options
--dry-run
-h, --help
Specify the token used to temporarily authenticate with the Kubernetes Control Plane w

hile joining the node.
process.
.
--help
command,
flag.
=
=
field in
.
RFC7469
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2&gt;/dev/null | openssl dgst -sha256 -hex | sed
's/^.* //'
Example
commands:
join
if the
.
control-plane node even if other worker nodes or the n

etwork are compromised.
command.
Example
command:
file.
Example
commands:
Using
, you can see that the original CSR is in the Pending state.
kube

ctl get csr
ConfigMap
flag of the
Fetch the
jsonpath
=
:
v1
kind
:
Config
clusters
:
-
cluster
:
:
server
:
name
:
""
contexts
:
[]
:
""
:
{}
users
:
[]
Use the
.
but before
.
Caution:
--config
contain a
--config
command.
the
command.
.
.
or
.
10.1.4 - kubeadm upgrade
.
You can use
and
will also
.
Note:
and
--config
Synopsis
Check which versions are available to upgrade to

 and validate whether your current cluster is upgradeable. To skip the internet check, pass in the optional [version] parameter
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
-y, --yes
Perform the upgrade and do not prompt for confirmation (non-in

teractive mode).
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
10.1.5 - kubeadm config
During
in the
,
and
.
You can use
uses for
and
.
Note:
init
and
join
Using kubeadm init with a

 configuration file
or
.
.
You can use
and
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
--new-config if spe

cified.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
10.1.6 - kubeadm reset
or
.
Synopsis

Options
--dry-run
-h, --help
the
or
reset
the
and
""
--prefix
See the
10.1.7 - kubeadm token
.
Synopsis
Options
-h, --help
Instead of printing only the token, print the full 'kubeadm join' flag neede

d to join the cluster using the token.
--dry-run
Synopsis
Options
-h, --help
--dry-run
Synopsis
Options
-h, --help
--dry-run
Synopsis
Options
-h, --help
--dry-run
10.1.8 - kubeadm version
Synopsis
Options
-h, --help
[EXPERIMENTAL] The 

path to the 'real' host root filesystem.
10.1.9 - kubeadm alpha
Caution:
.
or
10.1.10 - kubeadm certs
.
overview
Synopsis
Options
-h, --help
all
.
renew
all
apiserver
etcd-peer
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewals run unconditionally, rega

rdless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewals run unconditionally, regardless of certificate expiration date

; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resupply them.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will be based on the existing file/certificates, there is no need to resu

pply them.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
to
and
Synopsis
This command will print out a secure randomly-generated cert

ificate key that can be used with
Options
-h, --help
.
Synopsis
Options
-h, --help
Synopsis
Examples
Options
-h, --help
or
10.1.11 - kubeadm init phase
,
preflight
Synopsis
Examples
Options
--dry-run
-h, --help
--ignore-preflight-errors stri

ngs
Synopsis
Examples
Options
--dry-run
-h, --help
certs
all
ca
apiserver
etcd-ca
etcd-peer
sa
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
If both files already exist, kubeadm skips the generation step and existing files will be 

used.
Options
--dry-run
-h, --help
help for ca
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Generate the cer

tificate for the front proxy client, and save them into front-proxy-client.crt and front-proxy-client.key files.
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--cert-dir string     Default: "/etc/kubernetes/pki

"
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
-h, --help
help for sa
all
all
admin
kubelet
scheduler
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
The IP address the API Server will advertise it's listening on. If not s

et the default network interface will be used.
--dry-run
-h, --help
only
Synopsis
only
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
Specify a stable IP address or DNS 

name for the control plane.
--dry-run
-h, --help
all
apiserver
scheduler
Synopsis
Options
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
[EXPERIMENTAL] The path t

o the 'real' host root filesystem.
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Generates the kube-schedul

er static Pod manifest
Synopsis
Options
--dry-run
-h, --help
etcd
local
Synopsis
Options
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
.
all
kubeadm
kubelet
Synopsis
Options
-h, --help
Options i

nherited from parent commands
Synopsis
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Examples
# Applies control-plane label and t

aint to the current node, functionally equivalent to what executed by kubeadm init.
  
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
all
phases.
Synopsis
Examples
Options
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
all
addon
all
coredns
Synopsis
This command is not meant to be run on its own. See list of ava

ilable subcommands.
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
-

-dry-run
-h, --help
v1beta3
or
10.1.12 - kubeadm join phase
,
phase
Synopsis
Options
-h, --help
preflight
Synopsis
Examples
Options
--dry-run
-h, --help
--tok

en string
all
certs
Synopsis
Examples
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
For token-bas

ed discovery, validate that the root CA public key matches this hash (format: "&lt;type&gt;:&lt;value&gt;").
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
If the node should host a new control plane 

instance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
all
etcd
Synopsis
Examples
Options
-h, --help
--root

fs string
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help

or
10.1.13 - kubeadm kubeconfig
see
.
overview
Synopsis
Options
-h, --help
user
Synopsis
Examples
Options
-h, --help
10.1.14 - kubeadm reset phase
,
phase
Synopsis
Options
-h, --help
preflight
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Ru

n cleanup node.
Synopsis
Options
--dry-run
-h, --help
or
10.1.15 - kubeadm upgrade phase
phases.
apply
phase
preflight
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
--kubeconfig string     Defa

ult: "/etc/kubernetes/admin.conf"
or
10.1.16 - Implementation details
and
how
and
Secure
:
It should
not
etcd.yaml
(
ca.crt
,
ca.key
,
,
sa.pub
,
sa.key
,
,
for 

the front proxy client
The
.
The
option.
flag) is
[Error] if
[Error] if
[Error] if
conntrack
,
ip
,
iptables
,
mount
,
nsenter
ebtables
,
ethtool
,
socat
,
tc
,
touch
,
crictl
command
ca.crt
file and
ca.key
private key file
ca.crt
10.96.0.1
)
if
,
,
The
A c

lient certificate for the API server to connect to the kubelets securely, generated using
ca.crt
.
sa.key
sa.pub
saved into
using the
flag.
See also
ca.crt
ca.key
csrsigner
;
--dry-run
command
Be in the
module
have the CN
root
by default
.
CN.
ca.crt
,
--dry-run
Kubeconfig files generation can be invoked individually w

ith the
command
namespace
and
labels
The
address
127.0.0.1
See
--dry-run
the
command
API server
The
and
The
(
,
,
);
with the
See
to
true
to
to:
and
using
gce
or
aws
to enforce default storage class on
objects
to
this makes
to
ca.crt
to
to
to
to
to
sa

.pub
to
to
to
(
)
the users:
and
with the
and
to
true
to
ca.crt
to
ca.crt
""
to
ca.key
""
to
sa.key
Scheduler
listen on
and use
make a
hostPath
dataDir
--dry-run
command.
ok
.
ok
under
determine the actual/current cluster state and make new decisions based on that data.
.
kubeadm init p

hase mark-control-plane
command.
and
The
.
,
--token
under
group
flag)
.
.
The role
.
role
.
Additionally it creates a Role and a RoleBinding granting access to the ConfigMap for
).
ConfigMap
is not
.
proxy

ca.crt
and
token
The
DNS
kube-dns
the
--config
here
.
The
coredns
kube-dns
kube-dns
.
, also
see
.
kubeadm
with the
option.
If
users for
)
flag;
This weakens the kubeadm security model since others can potentially impersonate the Kubernetes Master.
If
server
and
Config

Map, and if available, uses it.
ca.crt
file and
)
during the
process
process
11 - Command line tool (kubectl)
,
kubectl
.
kubectl
config
in the
flag.
kubectl
kubectl
;
.
docker
kubectl
Syntax
kubectl
kubectl
[
command
]
[
TYPE
]
[
NAME
]
[
flags
]
where
command
,
TYPE
,
NAME
, and
flags
are:
command
create
,
get
,
describe
,
delete
.
TYPE
NAME
.
.
Example:
.
Example:
Example:
flags
-s
or
--server
flags to specify the address and port of the Kubernetes API server.
Caution:
By default
kubectl
and
/var/run/secrets/kubernetes.io/service

account/token
If the
seattle
,
seattle
If:
, and
the
the
=
kubectl
Operation
Syntax
alpha
annotate
apply
attach
auth
autoscale
config
convert
kubectl convert -f FILENAME [options]
cordon
cp
create
delete
kubectl delete (-f FILENAME | TYPE [NAME | /NAME | -l label | --all]) [flags

]
describe
diff
drain
edit
events
exec
explain
expose
get
kustomize
label
logs
options
patch
plugin
proxy
replace
rollout
run
scale
set
taint
top
uncordon
kubectl uncordon NODE [options]
version
wait
kubectl
The following table includes a list of all the supported resource ty

pes and their abbreviated aliases.
NAME
KIND
bindings
v1
true
Binding
cs
v1
false
cm
v1
true
ConfigMap
endpoints
ep
v1
true
Endpoints
events
ev
v1
true
Event
limits
v1
true
ns
v1
false
Namespace
nodes
no
v1
false
Node
pvc
v1
true
pv
v1
false
pods
po
v1
true
Pod
v1
true
rc
v1
true
quota
v1
true
secrets
v1
true
Secret
sa
v1
true
services
svc
v1
true
Service
false
false
crd,crds
false
false
apps/v1
true
ds
apps/v1
true
DaemonSet
deploy
apps/v1
true
rs
apps/v1
true
sts
apps/v1
true
false
true
false
false
false
hpa
true
cronjobs
cj
batch/v1
true
CronJob
jobs
batch/v1
true
Job
csr
false
leases
true
Lease
true
events
ev
true
Event
false
false
false
ingresses
ing
true
Ingress
netpol
true
false
pdb
policy/v1
true
psp
false
false
false
true
roles
true
Role
pc
false
false
CSIDriver
csinodes
false
CSINode
true
sc
false
false
kubectl
kubectl
commands is the human readable plain-text format. To output details to your terminal window in a specific format, you can add either the
-o
or
--output
kubectl
command.
Syntax
kubectl
[
command
]
[
TYPE
]
[
NAME
]
kubectl
.
file.
-o json
Output a 

JSON formatted API object.
jsonpath
jsonpath
file.
-o name
-o wide
-o yaml
Example
kubectl
option.
or
.
Examples
Inline:
=
=
where the
NAME          RSRC
NAME           RSRC
kubectl
flag to the
command.
Examples
=
false
NAME       AGE
--sort-by
kubectl
--sort-by
jsonpath
Syntax
kubectl
[
command
]
[
TYPE
]
[
NAME
]
--sort-by
=
Example
=
kubectl
# List the replication controller with the specified name in plain-text output format. Tip: You can shorten and replace the 'replicationcontroller' resource type with the alias 'rc'.
=
=
server01
# Display the 

details of the node with name &lt;node-name&gt;.
Note:
The
-o
or
--output
-w
or
--watch
=
kubectl
exec
kubectl
exec
kubectl
exec
kubectl
plugins:
#!/bin/sh
echo
# You can "uninstall" a plugin, by removing it from the folder in your
kubectl
, use
the

sudo chmod -x /usr/local/bin/ku

bectl-foo

have
=
$(
)
Read the
kubectl
the
reference
kubectl
Read about
in kubectl
.
11.1 - kubectl Cheat Sheet
kubectl
BASH
source
&lt;
(
)
echo
kubectl
alias
k
=
kubectl
complete
ZSH
source
&lt;
(
)
echo
A note on
Appending
:
kubectl -A
kubectl
=
jsonpath
=
jsonpath
=
jsonpath
=
=
=
=
=
ggckad-s2
# set a context utilizing a specific username and namespace.
=
cluster-admin --names

pace
=
foo
\
unset
users.foo
alias
kx
=
alias
kn
=
apply
.
.yaml
,
.yml
, and
.json
=
nginx
=
echo
=
=
"*/1 * * * *"
--
echo
cat
kind: Pod
metadata:
spec:
args:
- sleep
---
kind: Pod
metadata:
spec:
args:
- sleep
- "1000"
EOF
cat
metadata:
data:
EOF
=
=
=
=
app
=
\
jsonpath
=
\
-o
jsonpath
=
=
=
=
=
Running
jsonpath
=
# "jq" command useful for transformations that are too complex for jsonpath

, it can be found at https://stedolan.github.io/jq/
sel
=
${
$(
=
json | jq -j
)
%?
}
echo
$(
=
$sel
--output
=
jsonpath
={
}
)
JSONPATH
=
\
jsonpath
=
"
$JSONPATH
"
| grep
=
jsonpath
=
=
=
Warning
for
pod in
$(
=
jsonpath
={
}
)
;
do
echo
$pod
kubectl
exec
-it
$pod
-- env;
done
=
status
kubectl
set
www
=
image:v2
history
=
2
=
80
=
8000
=
awesome
=
=
2
--max
=
10
# Update a container's image; spec.containers[*].name is required because it's a merge key
'{"spec":{"containers"

:[{"name":"kubernetes-serve-hostname","image":"new image"}]}}'
=
'json'
-p
=
=
=
'json'
-p
=
=
'scale'
--type
=
'merge'
-p
=
"nano"
=
3
rs/foo
=
3
=
2
=
3
=
5
name
=
myLabel
=
true
| awk
name
=
myLabel
name
=
name
=
=
=
=
=
kubectl
exec
kubectl
exec
kubectl
exec
=
cpu
# Show metrics for a given pod and sort it by 'cpu' or 'memory'
Copying files and directories to

 and from containers
Note:
kubectl cp
kubectl cp
will fail.
.
exec
kubectl
exec
5000
kubectl
exec
=
=
dedicated
=
API group
, and
Kind
:
=
true
=
false
=
list,get
=
-o
(or
--output
kubectl
command.
file
-o=json
jsonpath
P

rint the fields defined by the
jsonpath
file
-o=name
-o=wide
-o=yaml
:
=
=
=
=
=
=
=
=
.
-v
or
--v
here
.
Verbosity
--v=0
always
--v=1
--v=2
--v=3
--v=4
--v=5
--v=6
--v=7
--v=8
--v=9
Read the
JsonPath
.
See
kubectl
options.
Also read
.
11.2 - kubectl Commands
kubectl Command Reference
11.3 - kubectl
Synopsis
kubectl controls the Kubernetes cluster manager.
Find more information at:
https://kubernetes.io/docs/reference/kubectl/overview/
kubectl [flags]
Options
If true, adds the file directory to the header of the log messages
log to standard error as well as files
--as string
Username to impersonate for the operation
--as-group stringArray
Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
--cache-dir string     Default: "$HOME/.kube/cache"
Default cache directory
--certificate-authority string
Path to a cert file for the certificate authority
--client-certificate string
Path to a client certificate file for TLS
--client-key string
Path to a client key file for TLS
--cloud-provider-gce-lb-src-cidrs cidrs     Default: 130.211.0.0/22,209.85.152.0/22,209.85.204.0/22,35.191.0.0/16
CIDRs opened in GCE firewall for L4 LB traffic proxy &amp; health checks
--cluster string
The name of the kubeconfig cluster to use
--context string
The name of the kubeconfig context to use
-h, --help
help for kubectl
--insecure-skip-tls-verify
If true, the server's certificate will not be checked for validi

ty. This will make your HTTPS connections insecure
--kubeconfig string
Path to the kubeconfig file to use for CLI requests.
--log-backtrace-at traceLocation     Default: :0
when logging hits line file:N, emit a stack trace
--log-dir string
If non-empty, write log files in this directory
--log-file string
If non-empty, use this log file
--log-file-max-size uint     Default: 1800
Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited.
--logtostderr     Default: true
log to standard error instead of files
--match-server-version
Require server version to match client version
-n, --namespace string
If present, the namespace scope for this CLI request
If true, only write logs to their native severity level (vs also writing to each lower severity level
--password string
Password for basic authentication to the API server
--profile string     Default: "none"
Name of profile to capture. One of (none|cpu|heap|goroutine|threadcreate|block|mutex)
--profile-output string     Default: "profile.pprof"
Name of the file to write the profile to
--request-timeout string     Default: "0"
The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests.
-s, --server string
The address and port of the Kubernetes API server
If true, avoid header prefixes in the log messages
If true, avoid headers when opening log files
--stderrthreshold severity     Default: 2
logs at or above this threshold go to stderr
--tls-server-name string
Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
Bearer token for authentication to the API server
--user string
The name of the kubeconfig user to use
--username string
Username for basic authentication to the API server
-v, --v Level
number for the log level verbosity
--vmodule moduleSpec
comma-separated list of pattern=N settings for file-filtered logging
--warnings-as-errors
Treat warnings received from the server as errors and exit with a non-zero exit code
Path to the kubectl configuration ("kubeconfig") file. Default: "$HOME/.kube/config"
KUBECTL_COMMAND_HEADERS
When set to false, turns off extra HTTP headers detailing invoked kubectl command (Kubernetes version v1.22 or later)
KUBECTL_EXPLAIN_OPENAPIV3
Toggles whether calls to `kubectl explain` use the new OpenAPIv3 data source available. OpenAPIV3 is enabled by default since Kubernetes 1.24.
See Also
- Update the annotations on a resource
- Print the supported API resources on the server
kubectl api-versions
- Print the supported API versions on the server, in the form of "group/version"
- Apply a configuration to a resource by filename or stdin
- Attach to a running container
kubectl auth
- Inspect authorization
- Auto-scale a Deployment, ReplicaSet, or ReplicationController
kubectl certificate
- Modify certificate resources.
- Display cluster info
kubectl completion
- Output shell completion code for the specified shell (bash or zsh)
- Modify kubeconfig files
kubectl cordon
- Mark node as unschedulable
kubectl cp
- Copy files and directories to and from containers.
kubectl create
- Create a resource from a file or from stdin.
- Create debugging sessions for troubleshooting workloads and nodes
- Delete resources by filenames, stdin, resources and names, or by resources and label selector
- Show details of a specific resource or group of resources
- Diff live version against woul

d-be applied version
- Drain node in preparation for maintenance
- Edit a resource on the server
- List events
- Execute a command in a container
kubectl explain
- Documentation of resources
- Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service
- Display one or many resources
kubectl kustomize
- Build a kustomization target from a directory or a remote url.
- Update the labels on a resource
- Print the logs for a container in a pod
- Print the list of flags inherited by all commands
- Update field(s) of a resource
kubectl plugin
- Provides utilities for interacting with plugins.
- Forward one or more local ports to a pod
- Run a proxy to the Kubernetes API server
- Replace a resource by filename or stdin
- Manage the rollout of a resource
- Run a particular image on the cluster
- Set a new size for a Deployment, ReplicaSet or Replication Controller
kubectl set
- Set specific features on objects
kubectl taint
- Update the taints on one or more nodes
- Display Resource (CPU/Memory/Storage) usage.
kubectl uncordon
- Mark node as schedulable
- Print the client and server version information
kubectl wait
- Experimental: Wait for a specific condition on one or many resources.
11.4 - JSONPath Support
Kubectl supports JSONPath template.
JSONPath template is composed of JSONPath expressions enclosed by curly braces {}.
Kubectl uses JSONPath expressions to filter on specific fields in the JSON object and format the output.
In addition to the original JSONPath template syntax, the following functions and syntax are valid:
Use double quotes to quote text inside JSONPath expressions.
Use the
range
,
end
operators to iterate lists.
Use negative slice indices to step backwards through a list. Negative indices do not "wrap around" a list and are valid as long as
-index + listLength &gt;= 0
.
Note:
The
$
operator is optional since the expression always starts from the root object by default.
The result object is printed as its String() function.
Given the JSON input:
{
"kind"
:
"List"
,
"items"
:[
{
"kind"
:
"None"
,
:{
"name"
:
},
"status"
:{
"capacity"
:{
"cpu"
:
"4"
},
"addresses"
:[{
"type"
:
"LegacyHostIP"
,
"address"
:
}]
}
},
{
"kind"
:
"None"
,
:{
"name"
:
"127.0.0.2"
},
"status"
:{
:{
"cpu"
:
"8"
},
:[
{
"type"
:
,
"address"
:
},
{
"type"
:
"another"
,
"address"
:
"127.0.0.3"
}
]
}
}
],
"users"
:[
{
"name"
:
"myself"
,
"user"
: {}
},
{
"name"
:
"e2e"
,
"user"
: {
:
"admin"
,
:
"secret"
}
}
]
}
Function
Example
Result
text
the plain text
kind is {.kind}
kind is List
@
the current object
{@}
the same as input
.
or
[]
child operator
{.kind}
,
{['kind']}
or
{['name\.type']}
List
..
recursive descent
{..name}
127.0.0.1 127.0.0.2 myself e2e
*
wildcard. Get all objects
{.items[*].metadata.name}
[127.0.0.1 127.0.0.2]
[start:end:step]
subscript operator
{.users[0].name}
myself
[,]
union operator
{.items[*]['metadata.name', 'status.capacity']}
127.0.0.1 127.0.0.2 map[cpu:4] map[cpu:8]
?()
filter
{.users[?(@.name=="e2e")].user.password}
secret
range
,
end
iterate list
{range .items[*]}[{.metadata.name}, {.status.capacity}] {end}
[127.0.0.1, map[cpu:4]] [127.0.0.2, map[cpu:8]]
''
quote interpreted string
{range .items[*]}{.metadata.name}{'\t'}{end}
127.0.0.1 127.0.0.2
kubectl
and JSONPath expressions:
kubectl get pods -o json
kubectl get pods -o
=
jsonpath
=
'{@}'
=
jsonpath
=
'{.items[0]}'
=
jsonpath
=
'{.items[0].metadata.name}'
=
jsonpath
=
"{.items[*]['metadata.name', 'status.capacity']}"
=
jsonpath
=
'{range .items[*]}

{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'
Note:
double
quote any JSONPath template that contains spaces (not single quote as shown above for bash). This in turn means that you must use a single quote or escaped double quote around any literals in the template. For example:
kubectl get pods -o=jsonpath=
"{range .items[*]}{.metadata.name}{'\t'}{.status.startTime}{'\n'}{end}"
"{range .items[*]}{.metadata.name}{\"
\t\
"}{.status.startTime}{\"
\n\
"}{end}"
Note:
JSONPath regular expressions are not supported. If you want to match using regular expressions, you can use a tool such as
jq
.
# kubectl does not support regular expressions for JSONpath output
# The following command does not work
jsonpath
=
'{.items[?(@.metadata.name=~/^test$/)].metadata.name}'
# The following command achieves the desired result
kubectl get pods -o json | jq -r
'.items[] | select(.metadata.name | test("test-")).spec.containers[].image'
11.5 - kubectl for Docker Users
You can use the Kubernetes command line tool
kubectl
to interact with the API Server. Using kubectl is straightforward if you are familiar with the Docker command line tool. However, there are a few differences between the Docker commands and the kubectl commands. The following sections show a Docker sub-command and describe the equivalent
kubectl
command.
docker run
To run an nginx Deployment and expose the Deployment, see
.
docker:
docker run -d --restart
=
always -e
DOMAIN
=
cluster --name nginx-app -p 80:80 nginx
55c103fa129692154a7652490236fee9be47d70a8dd562281ae7d2f9a339a6db
docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
55c103fa1296        nginx               "nginx -g 'daemon of…"   9 seconds ago       Up 9 seconds        0.0.0.0:80-&gt;80/tcp   nginx-app
kubectl:
# start the pod running nginx
kubectl create deployment --image
=
nginx nginx-app
deployment.apps/nginx-app created
# add env to nginx-app
kubectl
set
env deployment/nginx-app
DOMAIN
=
cluster
deployment.apps/nginx-app env updated
Note:
kubectl
commands print the type and name of the resource created or mutated, which can then be used in subsequent commands. You can expose a new Service after a Deployment is created.
# expose a port through with a service
kubectl expose deployment nginx-app --port
=
80
--name
=
nginx-http
service "nginx-http" exposed
By using kubectl, you can create a
to ensure that N pods are running nginx, where N is the number of replicas stated in the spec and defaults to 1. You can also create a
service
with a selector that matches the pod labels. For more information, see
.
By default images run in the background, similar to
docker run -d ...
. To run things in the foreground, use
to create pod:
[
-i
]
[
--tty
]
--attach &lt;name&gt; --image
=
&lt;image&gt;
Unlike
docker run ...
, if you specify
--attach
, then you attach
stdin
,
stdout
and
stderr
. You cannot control which streams are attached (
docker -a ...
).
To detach from the container, you can type the escape sequence Ctrl+P followed by Ctrl+Q.
docker ps
To list what is currently running, see
.
docker:
docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                     PORTS                NAMES
14636241935f        ubuntu:16.04        "echo test"              5 seconds ago        Exited (0) 5 seconds ago                        cocky_fermi
55c103fa1296        nginx               "nginx -g 'daemon of…"   About a minute ago   Up About a minute          0.0.0.0:80-&gt;80/tcp   nginx-app
kubectl:
kubectl get po
NAME                        READY     STATUS      RESTARTS   AGE
nginx-app-8df569cb7-4gd89   1/1       Running     0          3m
ubuntu                      0/1       Completed   0       

   20s
docker attach
To attach a process that is already running in a container, see
.
docker:
docker ps
55c103fa1296        nginx               "nginx -g 'daemon of…"   5 minutes ago       Up 5 minutes        0.0.0.0:80-&gt;80/tcp   nginx-app
docker attach 55c103fa1296
...
kubectl:
nginx-app-5jyvm   1/1       Running   0          10m
kubectl attach -it nginx-app-5jyvm
...
To execute a command in a container, see
.
docker:
docker ps
55c103fa1296        nginx               "nginx -g 'daemon of…"   6 minutes ago       Up 6 minutes        0.0.0.0:80-&gt;80/tcp   nginx-app
docker
exec
55c103fa1296 cat /etc/hostname
55c103fa1296
kubectl:
kubectl
exec
nginx-app-5jyvm -- cat /etc/hostname
nginx-app-5jyvm
To use interactive commands.
docker:
docker
exec
-ti 55c103fa1296 /bin/sh
# exit
kubectl:
kubectl
exec
-ti nginx-app-5jyvm -- /bin/sh
# exit
.
To follow stdout/stderr of a process that is running, see
.
docker:
docker logs -f a9e
192.168.9.1 - - [14/Jul/2015:01:04:02 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.35.0" "-"
192.168.9.1 - - [14/Jul/2015:01:04:03 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.35.0" "-"
kubectl:
kubectl logs -f nginx-app-zibvs
10.240.63.110 - - [14/Jul/2015:01:09:01 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.26.0" "-"
10.240.63.110 - - [14/Jul/2015:01:09:02 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.26.0" "-"
There is a slight difference between pods and containers; by default pods do not terminate if their processes exit. Instead the pods restart the process. This is similar to the docker run option
--restart=always
with one major difference. In docker, the output for each invocation of the process is concatenated, but for Kubernetes, each invocation is separate. To see the output from a previous run in Kubernetes, do this:
kubectl logs --previous nginx-app-zibvs
.
docker stop and docker rm
To stop and delete a running process, see
.
docker:
docker ps
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                         NAMES
a9ec34d98787        nginx               "nginx -g 'daemon of"  22 hours ago        Up 22 hours         0.0.0.0:80-&gt;80/tcp, 443/tcp   nginx-app
docker stop a9ec34d98787
a9ec34d98787
docker rm a9ec34d98787
kubectl:
kubectl get deployment nginx-app
nginx-app    1/1     1            1           2m
kubectl get po -l
app
=
nginx-app
nginx-app-2883164633-aklf7   1/1       Running   0          2m
kubectl delete deployment nginx-app
deployment "nginx-app" deleted
app
=
nginx-app
# Return nothing
Note:
When you use kubectl, you don't delete the pod directly. You have to first delete the Deployment that owns the pod. If you delete the pod directly, the Deployment recreates the pod.
There is no direct analog of
in kubectl. If you are interested in using Kubernetes with a private registry, see
Using a Private Registry
.
docker version
To get the version of client

 and server, see
.
docker:
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): linux/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64
kubectl:
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.9+a3d1dfa6f4335", GitCommit:"9b77fed11a9843ce3780f70dd251e92901c43072", GitTreeState:"dirty", BuildDate:"2017-08-29T20:32:58Z", OpenPaasKubernetesVersion:"v1.03.02", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.9+a3d1dfa6f4335", GitCommit:"9b77fed11a9843ce3780f70dd251e92901c43072", GitTreeState:"dirty", BuildDate:"2017-08-29T20:32:58Z", OpenPaasKubernetesVersion:"v1.03.02", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
docker info
To get miscellaneous information about the environment and configuration, see
.
docker:
Containers: 40
Images: 168
Storage Driver: aufs
 Root Dir: /usr/local/google/docker/aufs
 Backing Filesystem: extfs
 Dirs: 248
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-53-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 12
Total Memory: 31.32 GiB
Name: k8s-is-fun.mtv.corp.google.com
ID: ADUV:GCYR:B3VJ:HMPO:LNPQ:KD5S:YKFQ:76VN:IANZ:7TFV:ZBF4:BYJO
WARNING: No swap limit support
kubectl:
Kubernetes master is running at https://203.0.113.141
KubeDNS is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/kube-dns/proxy
kubernetes-dashboard is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy
Grafana is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy
Heapster is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/monitoring-heapster/proxy
InfluxDB is running at https://203.0.113.141/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
11.6 - kubectl Usage Conventions
Recommended usage conventions for
kubectl
.
Using
kubectl
in Reusable Scripts
For a stable output in a script:
Request one of the machine-oriented output forms, such as
-o name
,
-o json
,
-o yaml
,
, or
-o jsonpath
.
Fully-qualify the version. For example,
jobs.v1.batch/myjob
. This will ensure that kubectl does not use its default version that can change over time.
Don't rely on context, preferences, or other implicit states.
Subresources
alpha flag for kubectl commands like
get
,
patch
,
edit
and
replace
to fetch and update subresources for all resources that
support them. Currently, only the
status
and
scale
subresources are supported.
The API contract against a subresource is identical to a full resource. While updating the
status
subresource to a new value, keep in mind that the subresource could be potentially
reconciled by a controller to a different value.
Best Practices
For
to satisfy infrastructure as code:
Tag the image with a version-specific tag and don't move that tag to a new version. For example, use
:v1234
,
v1.2.3
,
r03062016-1-4
, rather than
:latest
(For more information, see
Best Practices for Configuration
).
Check in the script for an image that is heavily parameterized.
Switch to configuration files checked into source control for features that are needed, but not expressible via
flags.
--dry-run=client
flag to preview the object that would be sent to your cluster, without really submitting it.
You can use
to create or update resources. For more information about using kubectl apply to update resources, see
.
12 - Component tools
12.1 - Feature Gates
This page contains an overview of the various feature gates an administrato

r
can specify on different Kubernetes components.
See
feature stages
for an explanation of the stages for a feature.
Overview
Feature gates are a set of key=value pairs that describe Kubernetes features.
You can turn these features on or off using the
on each Kubernetes component.
Each Kubernetes component lets you enable or disable a set of feature gates that
are relevant to that component.
Use
-h
flag to see a full set of feature gates for all components.
To set feature gates for a component, such as kubelet, use the
flag assigned to a list of feature pairs:
=
...,GracefulNodeShutdown
=
true
The following tables are a summary of the feature gates that you can set on
different Kubernetes components.
The "Since" column contains the Kubernetes release when a feature is introduced
or its release stage is changed.
The "Until" column, if not empty, contains the last Kubernetes release in which
you can still use a feature gate.
If a feature is in the Alpha or Beta state, you can find the feature listed
in the
Alpha/Beta feature gate table
.
If a feature is stable you can find all stages for that feature listed in the
Graduated/Deprecated feature gate table
.
The
also lists deprecated and withdrawn features.
Note:
For a reference to old feature gates that are removed, please refer to
feature gates removed
.
Feature gates for Alpha or Beta features
Feature gates for features in Alpha or Beta states
Feature
Default
Stage
Since
Until
false
Alpha
1.8
1.8
true
Beta
1.9
false
Alpha
1.18
1.19
true
Beta
1.20
APIResponseCompression
false
Alpha
1.7
1.15
true
Beta
1.16
false
Alpha
1.26
APIServerIdentity
false
Alpha
1.20
1.25
true
Beta
1.26
false
Alpha
1.22
AggregatedDiscoveryEndpoint
false
Alpha
1.26
AnyVolumeDataSource
false
Alpha
1.18
1.23
true
Beta
1.24
AppArmor
true
Beta
1.4
false
Alpha
1.23
true
Beta
1.23
false
Alpha
1.22
1.22
true
Beta
1.23
CSIMigrationPortworx
false
Alpha
1.23
1.24
false
Beta
1.25
CSIMigrationRBD
false
Alpha
1.23
CSINodeExpandSecret
false
Alpha
1.25
CSIVolumeHealth
false
Alpha
1.21
false
Alpha
1.26
false
Alpha
1.25
false
Alpha
1.24
false
Alpha
1.24
1.24
true
Beta
1.25
CrossNamespaceVolumeDataSource
false
Alpha
1.26
CustomCPUCFSQuotaPeriod
false
Alpha
1.12
false
Alpha
1.23
1.24
true
Beta
1.25
DisableCloudProviders
false
Alpha
1.22
DisableKubeletCloudCredentialProviders
false
Alpha
1.23
DownwardAPIHugePages
false
Alpha
1.20
1.20
false
Beta
1.21
1.21
true
Beta
1.22
DynamicResourceAllocation
false
Alpha
1.26
EventedPLEG
false
Alpha
1.26
-
ExpandedDNSConfig
false
Alpha
1.22
1.25
true
Beta
1.26
ExperimentalHostUserNamespaceDefaulting
false
Beta
1.5
false
Alpha
1.23
1.23
true
Beta
1.24
GracefulNodeShutdown
false
Alpha
1.20
1.20
true
Beta
1.21
GracefulNodeShutdownBasedOnPodPriority
false
Alpha
1.23
1.23
true
Beta
1.24
HPAContainerMetrics
false
Alpha
1.20
HPAScaleToZero
false
Alpha
1.16
HonorPVReclaimPolicy
false
Alpha
1.23
IPTablesOwnershipCleanup
false
Alpha
1.25
InTreePluginAWSUnregister
false
Alpha
1.21
InTreePluginAzureDiskUnregister
false
Alpha
1.21
InTreePluginAzureFileUnregister
false
Alpha
1.21
InTreePluginGCEUnregister
false
Alpha
1.21
InTreePluginOpenStackUnregister
false
Alpha
1.21
InTreePluginPortworxUnregister
false
Alpha
1.23
InTreePluginRBDUnregister
false
Alpha
1.23
InTreePluginvSphereUnregister
false
A

lpha
1.21
JobMutableNodeSchedulingDirectives
true
Beta
1.23
false
Alpha
1.25
1.25
true
Beta
1.26
JobReadyPods
false
Alpha
1.23
1.23
true
Beta
1.24
KMSv2
false
Alpha
1.25
false
Alpha
1.22
KubeletPodResources
false
Alpha
1.13
1.14
true
Beta
1.15
KubeletPodResourcesGetAllocatable
false
Alpha
1.21
1.22
true
Beta
1.23
false
Alpha
1.25
LegacyServiceAccountTokenTracking
false
Alpha
1.25
LocalStorageCapacityIsolationFSQuotaMonitoring
false
Alpha
1.15
-
LogarithmicScaleDown
false
Alpha
1.21
1.21
true
Beta
1.22
LoggingAlphaOptions
false
Alpha
1.24
-
LoggingBetaOptions
true
Beta
1.24
-
MatchLabelKeysInPodTopologySpread
false
Alpha
1.25
MaxUnavailableStatefulSet
false
Alpha
1.24
false
Alpha
1.21
1.21
true
Beta
1.22
MemoryQoS
false
Alpha
1.22
MinDomainsInPodTopologySpread
false
Alpha
1.24
1.24
false
Beta
1.25
false
Alpha
1.26
-
MultiCIDRRangeAllocator
false
Alpha
1.25
false
Alpha
1.24
NodeInclusionPolicyInPodTopologySpread
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.24
1.25
true
Beta
1.26
NodeSwap
false
Alpha
1.22
OpenAPIEnums
false
Alpha
1.23
1.23
true
Beta
1.24
OpenAPIV3
false
Alpha
1.23
1.23
OpenAPIV3
true
Beta
1.24
PDBUnhealthyPodEvictionPolicy
false
Alpha
1.26
false
Alpha
1.23
PodDeletionCost
false
Alpha
1.21
1.21
true
Beta
1.22
PodDisruptionConditions
false
Alpha
1.25
1.25
true
Beta
1.26
PodHasNetworkCondition
false
Alpha
1.25
PodSchedulingReadiness
false
Alpha
1.26
false
Alpha
1.21
1.21
false
Beta
1.22
1.24
true
Beta
1.25
ProcMountType
false
Alpha
1.12
false
Alpha
1.22
1.25
true
Beta
1.26
QOSReserved
false
Alpha
1.11
false
Alpha
1.22
RecoverVolumeExpansionFailure
false
Alpha
1.23
RemainingItemCount
false
Alpha
1.15
1.15
true
Beta
1.16
RetroactiveDefaultStorageClass
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.7
1.11
true
Beta
1.12
false
Alpha
1.25
false
Alpha
1.22
1.24
true
Beta
1.25
false
Alpha
1.23
1.24
true
Beta
1.25
SizeMemoryBackedVolumes
false
Alpha
1.20
1.21
true
Beta
1.22
StatefulSetAutoDeletePVC
false
Alpha
1.22
StatefulSetStartOrdinal
false
Alpha
1.26
StorageVersionAPI
false
Alpha
1.20
StorageVersionHash
false
Alpha
1.14
1.14
true
Beta
1.15
TopologyAwareHints
false
Alpha
1.21
1.22
false
Beta
1.23
1.23
true
Beta
1.24
false
Alpha
1.16
1.17
true
Beta
1.18
false
Alpha
1.26
false
Beta
1.26
false
Alpha
1.26
false
Alpha
1.25
false
Alpha
1.26
VolumeCapacityPriority
false
Alpha
1.21
-
WinDSR
false
Alpha
1.14
WinOverlay
false
Alpha
1.14
1.19
true
Beta
1.20
WindowsHostNetwork
false
Alpha
1.26
Feature gates for graduated or deprecated features
Feature Gates for Graduated or Deprecated Features
Feature
Default
Stage
Since
Until
AdvancedAuditing
false
Alpha
1.7
1.7
true
Beta
1.8
1.11
true
GA
1.12
-
false
Alpha
1.8
1.9
true
Beta
1.10
1.25
true
GA
1.26
-
CSIInlineVolume
false
Alpha
1.15
1.15
true
Beta
1.16
1.24
true
GA
1.25
-
CSIMigration
false
Al

pha
1.14
1.16
true
Beta
1.17
1.24
true
GA
1.25
-
CSIMigrationAWS
false
Alpha
1.14
1.16
false
Beta
1.17
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
CSIMigrationAzureDisk
false
Alpha
1.15
1.18
false
Beta
1.19
1.22
true
Beta
1.23
1.23
true
GA
1.24
CSIMigrationAzureFile
false
Alpha
1.15
1.20
false
Beta
1.21
1.23
true
Beta
1.24
1.25
true
GA
1.26
CSIMigrationGCE
false
Alpha
1.14
1.16
false
Beta
1.17
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
CSIMigrationvSphere
false
Alpha
1.18
1.18
false
Beta
1.19
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
false
Alpha
1.19
1.20
true
Beta
1.21
1.23
true
GA
1.24
-
ConsistentHTTPGetHandlers
true
GA
1.25
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
-
DaemonSetUpdateSurge
false
Alpha
1.21
1.21
true
Beta
1.22
1.24
true
GA
1.25
-
DelegateFSGroupToCSIDriver
false
Alpha
1.22
1.22
true
Beta
1.23
1.25
true
GA
1.26
-
DevicePlugins
false
Alpha
1.8
1.9
true
Beta
1.10
1.25
true
GA
1.26
-
false
Alpha
1.19
1.19
true
Beta
1.20
1.24
true
GA
1.25
-
DryRun
false
Alpha
1.12
1.12
DryRun
true
Beta
1.13
1.18
DryRun
true
GA
1.19
-
EfficientWatchResumption
false
Alpha
1.20
1.20
true
Beta
1.21
1.23
true
GA
1.24
-
EndpointSliceTerminatingCondition
false
Alpha
1.20
1.21
true
Beta
1.22
1.25
true
GA
1.26
EphemeralContainers
false
Alpha
1.16
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
true
GA
1.20
-
ExpandCSIVolumes
false
Alpha
1.14
1.15
true
Beta
1.16
1.23
true
GA
1.24
-
ExpandInUsePersistentVolumes
false
Alpha
1.11
1.14
true
Beta
1.15
1.23
true
GA
1.24
-
ExpandPersistentVolumes
false
Alpha
1.8
1.10
true
Beta
1.11
1.23
true
GA
1.24
-
IdentifyPodOS
false
Alpha
1.23
1.23
true
Beta
1.24
1.24
true
GA
1.25
-
JobTrackingWithFinalizers
false
Alpha
1.22
1.22
false
Beta
1.23
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
KubeletCredentialProviders
false
Alpha
1.20
1.23
true
Beta
1.24
1.25
true
GA
1.26
-
LegacyServiceAccountTokenNoAutoGeneration
true
Beta
1.24
1.25
true
GA
1.26
-
LocalStorageCapacityIsolation
false
Alpha
1.7
1.9
true
Beta
1.10
1.24
true
GA
1.25
-
MixedProtocolLBService
false
Alpha
1.20
1.23
true
Beta
1.24
1.25
true
GA
1.26
-
NetworkPolicyEndPort
false
Alpha
1.21
1.21
true
Beta
1.22
1.24
true
GA
1.25
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.24
true
GA
1.25
RemoveSelfLink
false
Alpha
1.16
1.19
true
Beta
1.20
1.23
true
GA
1.24
-
ServerSideApply
false
Alpha
1.14
1.15
true
Beta
1.16
1.21
true
GA
1.22
-
false
Alpha
1.24
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
ServiceInternalTrafficPolicy
false
Alpha
1.21
1.21
true
Be

ta
1.22
1.25
true
GA
1.26
-
StatefulSetMinReadySeconds
false
Alpha
1.22
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
WatchBookmark
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
-
WindowsHostProcessContainers
false
Alpha
1.22
1.22
true
Beta
1.23
1.25
true
GA
1.26
-
Using a feature
Feature stages
A feature can be in
Alpha
,
Beta
or
GA
stage.
An
Alpha
feature means:
Disabled by default.
Might be buggy. Enabling the feature may expose bugs.
Support for feature may be dropped at any time without notice.
Recommended for use only in short-lived testing clusters, due to increased
risk of bugs and lack of long-term support.
A
Beta
Enabled by default.
The feature is well tested. Enabling the feature is considered safe.
Support for the overall feature will not be dropped, though details may change.
The schema and/or semantics of objects may change in incompatible ways in a
subsequent beta or stable release. When this happens, we will provide instructions
for migrating to the next version. This may require deleting, editing, and
re-creating API objects. The editing process may require some thought.
This may require downtime for applications that rely on the feature.
Recommended for only non-business-critical uses because of potential for
incompatible changes in subsequent releases. If you have multiple clusters
that can be upgraded independently, you may be able to relax this restriction.
Note:
Please do try
Beta
features and give feedback on them!
After they exit beta, it may not be practical for us to make more changes.
A
General Availability
(GA) feature is also referred to as a
stable
feature. It means:
The feature is always enabled; you cannot disable it.
The corresponding feature gate is no longer needed.
Stable versions of features will appear in released software for many subsequent versions.
List of feature gates
Each feature gate is designed for enabling/disabling a specific feature:
: Enable the API clients to retrieve (
LIST
or
GET
)
resources from API server in chunks.
: Enable managing request concurrency with
prioritization and fairness at each server. (Renamed from
RequestManagement
)
: Compress the API responses for
LIST
or
GET
requests.
: Activate the
API which allows users
to see the requesting subject's authentication information.
See
: Assign each API server an ID in a cluster.
: Add support for distributed tracing in the API server.
See
Traces for Kubernetes System Components
: Enable
advanced auditing
: Enable a single HTTP endpoint
/discovery/&lt;version&gt;
which
supports native HTTP caching with ETags containing all APIResources known to the API server.
: Enable use of any custom resource as the
DataSource
of a
PVC
.
AppArmor
: Enable use of AppArmor mandatory access control for Pods running on Linux nodes.
See
AppArmor Tutorial
: Enables the kubelet
checkpoint
API.
See
: Enables Leader Migration for
and
which allows a cluster operator to live migrate
controllers from the kube-controller-manager into an external controller-manager
(e.g. the cloud-controller-manager) in an HA cluster without downtime.
: Enable container level CPU affinity support, see
.
: This allows fine-tuning of CPUManager policies,
experimental, Alpha-quality options
Thi

s feature gate guards
a group
of CPUManager options whose quality level is alpha.
This feature gate will never graduate to beta or stable.
experimental, Beta-quality options
This feature gate guards
a group
of CPUManager options whose quality level is beta.
This feature gate will never graduate to stable.
: Allow fine-tuning of CPUManager policies.
: Enable CSI Inline volumes support for pods.
: Enables shims and translation logic to route volume
operations from in-tree plugins to corresponding pre-installed CSI plugins
operations from the AWS-EBS in-tree plugin to EBS CSI plugin. Supports
falling back to in-tree EBS plugin for mount operations to nodes that have
the feature disabled or that do not have EBS CSI plugin installed and
configured. Does not support falling back for provision operations, for those
the CSI plugin must be installed and configured.
operations from the Azure-Disk in-tree plugin to AzureDisk CSI plugin.
Supports falling back to in-tree AzureDisk plugin for mount operations to
nodes that have the feature disabled or that do not have AzureDisk CSI plugin
installed and configured. Does not support falling back for provision
operations, for those the CSI plugin must be installed and configured.
Requires CSIMigration feature flag enabled.
operations from the Azure-File in-tree plugin to AzureFile CSI plugin.
Supports falling back to in-tree AzureFile plugin for mount operations to
nodes that have the feature disabled or that do not have AzureFile CSI plugin
operations from the GCE-PD in-tree plugin to PD CSI plugin. Supports falling
back to in-tree GCE plugin for mount operations to nodes that have the
feature disabled or that do not have PD CSI plugin installed and configured.
Does not support falling back for provision operations, for those the CSI
plugin must be installed and configured. Requires CSIMigration feature flag
enabled.
operations from the RBD in-tree plugin to Ceph RBD CSI plugin. Requires
CSIMigration and csiMigrationRBD feature flags enabled and Ceph CSI plugin
installed and configured in the cluster. This flag has been deprecated in
favor of the
feature flag which prevents the registration of
in-tree RBD plugin.
: Enables shims and translation logic to route volume operations
from the vSphere in-tree plugin to vSphere CSI plugin. Supports falling back
to in-tree vSphere plugin for mount operations to nodes that have the feature
disabled or that do not have vSphere CSI plugin installed and configured.
enabled.
from the Portworx in-tree plugin to Portworx CSI plugin.
Requires Portworx CSI driver to be installed and configured in the cluster.
: Enable passing secret authentication data to a CSI driver for use
during a
NodeExpandVolume
CSI operation.
: Enables CSI drivers to publish storage capacity information
and the Kubernetes scheduler to use that information when scheduling pods. See
.
Check the
csi
volume type
: Enable support for CSI volume heal

th monitoring on node.
: Enable the
endpoint on Kubernetes components like
kubelet, kube-scheduler, kube-proxy, kube-controller-manager, cloud-controller-manager
allowing you to scrape health check metrics.
: Normalize HTTP get URL and Header passing for lifecycle
handlers with probers.
: When you enable this feature gate, Kubernetes components that support
contextual logging add extra detail to log output.
: Enables leader migration for
and
.
: Allow the use of the
timeZone
optional field in
CronJobs
: Enable the usage of cross namespace volume data source
to allow you to specify a source namespace in the
field of a
PersistentVolumeClaim.
: Enable nodes to change
cpuCFSQuotaPeriod
in
kubelet config
.
: Enable expression language validation in CRD
which will validate customer resource based on validation rules written in
the
extension.
: Enables the DaemonSet workloads to maintain
availability during update per node.
See
.
: If supported by the CSI driver, delegates the
role of applying
fsGroup
from a Pod's
to the driver by
passing
fsGroup
through the NodeStageVolume and NodePublishVolume CSI calls.
device-plugins
based resource provisioning on nodes.
:
Disable accelerator metrics collected by the kubelet
.
: Disables any functionality in
,
and
kubelet
related to the
component flag.
: Disable the in-tree functionality in kubelet
to authenticate to a cloud provider container registry for image pull credentials.
: Enables usage of hugepages in
.
DryRun
: Enable server-side
dry run
requests
so that validation, merging, and mutation can be tested without committing.
`DynamicResourceAllocation": Enables support for resources with custom parameters and a lifecycle
that is independent of a Pod.
: Enables EndpointSlice
terminating
and
serving
condition fields.
: Allows for storage-originated bookmark (progress
notify) events to be delivered to the users. This is only applied to watch operations.
: Enable the ability to add
to running pods.
: Enable support for the kubelet to receive container life cycle events from the
via
an extension to
CRI
.
(PLEG is an abbreviation for “Pod lifecycle event generator”).
For this feature to be useful, you also need to enable support for container lifecycle events
in each container runtime running in your cluster. If the container runtime does not announce
support for container lifecycle events then the kubelet automatically switches to the legacy
generic PLEG mechanism, even if you have this feature gate enabled.
: Ensure kubelet respects exec probe timeouts.
This feature gate exists in case any of your existing workloads depend on a
now-corrected fault where Kubernetes ignored exec probe timeouts. See
.
: Enable the expanding of CSI volumes.
: Enable kubelet and kube-apiserver to allow more DNS
search paths and longer list of DNS search paths. This feature requires container
runtime support(Containerd: v1.5.6 or higher, CRI-O: v1.22 or higher). See
Expanded DNS Configuration
.
: Enable expanding in-use PVCs. See
Resizing an in-use PersistentVolumeClaim
.
: Enable the expanding of persistent volumes. See
Expanding Persistent Volumes Claims
.
Experimen

talHostUserNamespaceDefaulting
: Enabling the defaulting user
namespace to host. This is for containers that are using other host namespaces,
host mounts, or containers that are privileged or using specific non-namespaced
capabilities (e.g.
MKNODE
,
SYS_MODULE
etc.). This should only be enabled
if user namespace remapping is enabled in the Docker daemon.
: Enables support for graceful shutdown in kubelet.
During a system shutdown, kubelet will attempt to detect the shutdown event
and gracefully terminate pods running on the node. See
Graceful Node Shutdown
: Enables the kubelet to check Pod priorities
when shutting down a node gracefully.
: Enables the gRPC probe method for {Liveness,Readiness,Startup}Probe.
See
.
: Honor persistent volume reclaim policy when it is
Delete
irrespective of PV-PVC deletion ordering.
For more details, check the
PersistentVolume deletion protection finalizer
to scale based on
metrics from individual containers in target pods.
: Enables setting
to 0 for
resources when using custom or external metrics.
: This causes kubelet to no longer create legacy iptables rules.
: Allows the Pod OS field to be specified. This helps in identifying
the OS of the pod authoritatively during the API server admission time.
In Kubernetes 1.26, the allowed values for the
pod.spec.os.name
are
windows
and
linux
.
: Stops registering the aws-ebs in-tree plugin in kubelet
and volume controllers.
: Stops registering the azuredisk in-tree plugin in kubelet
: Stops registering the azurefile in-tree plugin in kubelet
: Stops registering the gce-pd in-tree plugin in kubelet
: Stops registering the OpenStack cinder in-tree plugin in kubelet
: Stops registering the Portworx in-tree plugin in kubelet
: Stops registering the RBD in-tree plugin in kubelet
: Stops registering the vSphere in-tree plugin in kubelet
: Allows updating node scheduling directives in
the pod template of
Job
.
: Allow users to specify handling of pod failures based on container
exit codes and pod conditions.
: Enables tracking the number of Pods that have a
Ready
condition
.
The count of
Ready
pods is recorded in the
status
of a
Job
status.
: Enables tracking
Job
completions without relying on Pods remaining in the cluster indefinitely.
The Job controller uses Pod finalizers and a field in the Job status to keep
track of the finished Pods to count towards completion.
KMSv2
: Enables KMS v2 API for encryption at rest. See
Using a KMS Provider for data encryption
: Enable kubelet exec credential providers for
image pull credentials.
: Enables support for running kubelet in a
.
See
.
: Enable the kubelet's pod resources gRPC endpoint. See
Support Device Monitoring
: Enable the kubelet's pod resources
GetAllocatableResources
functionality. This API augments the
resource allocation reporting
with informations about the allocatable resources, enabling clients to properly
track the free compute resources on a node.
: Add support for dist

ributed tracing in the kubelet.
When enabled, kubelet CRI interface and authenticated http servers are instrumented to generate
OpenTelemetry trace spans.
See
: Stop auto-generation of Secret-based
service account tokens
.
: Track usage of Secret-based
.
: Enable the consumption of
local ephemeral storage
and also the
sizeLimit
property of an
.
: When
is enabled for
and the backing filesystem for
emptyDir volumes
supports project quotas and they are enabled, use project quotas to monitor
storage consumption rather than
filesystem walk for better performance and accuracy.
: Enable semi-random selection of pods to evict on controller scaledown
based on logarithmic bucketing of pod timestamps.
: Allow fine-tuing of experimental, alpha-quality logging options.
: Allow fine-tuing of experimental, beta-quality logging options.
matchLabelKeys
field for
Pod topology spread constraints
.
: Enables setting the
field for the
rolling update strategy
of a StatefulSet. The field specifies the maximum number of Pods
that can be unavailable during the update.
: Allows setting memory affinity for a container based on
NUMA topology.
MemoryQoS
: Enable memory protection and usage throttle on pod / container using
cgroup v2 memory controller.
: Enable
minDomains
in
.
: Enables new performance improvement logics
in the kube-proxy iptables mode.
: Enable using different protocols in the same
type
Service instance.
: Enables the MultiCIDR range allocator.
: Enable use of the field
endPort
in NetworkPolicy objects,
allowing the selection of a port range instead of a single port.
status
subresource for NetworkPolicy objects.
: Enable using
nodeAffinityPolicy
and
nodeTaintsPolicy
in
when calculating pod topology spread skew.
: When a Node is marked out-of-service using the
taint, Pods on the node will be forcefully deleted
if they can not tolerate this taint, and the volume detach operations for Pods terminating
on the node will happen immediately. The deleted Pods can recover quickly on different nodes.
NodeSwap
: Enable the kubelet to allocate swap memory for Kubernetes workloads on a node.
Must be used with
KubeletConfiguration.failSwapOn
set to false.
For more details, please see
swap memory
: Enables populating "enum" fields of OpenAPI schemas in the
spec returned from the API server.
OpenAPIV3
: Enables the API server to publish OpenAPI v3.
: Enables the
field of a
. This specifies
when unhealthy pods should be considered for eviction. Please see
Unhealthy Pod Eviction Policy
feature which allows users to influence ReplicaSet downscaling order.
: Configure the kubelet to gather container and pod stats from the CRI container runtime rather than gathering them from cAdvisor.
As of 1.26, this also includes gathering metrics from CRI and emitting them over
(rather than having cAdvisor emit them directly).
: Enables support for appending a dedicated pod condition indicating that the pod is being deleted due to a disruption.
PodHas

NetworkCondition
: Enable the kubelet to mark the
PodHasNetwork
condition on pods.
: Enable setting
field to control a Pod's
scheduling readiness
.
: Enable
setting probe-level
on pods. See the
: Enables control over the type proc mounts for containers
by setting the
procMount
field of a SecurityContext.
: Enable the kube-proxy to handle terminating
endpoints when
ExternalTrafficPolicy=Local
.
: Allows resource reservations at the QoS level preventing pods
at lower QoS levels from bursting into resources requested at higher QoS levels
(memory only for now).
: Enables the usage of
access mode.
: Enables users to edit their PVCs to smaller
sizes so as they can recover from previously issued volume expansion failures.
See
Recovering from Failure when Expanding Volumes
: Allow the API servers to show a count of remaining
items in the response to a
chunking list request
.
: Sets the
.metadata.selfLink
field to blank (empty string) for all
objects and collections. This field has been deprecated since the Kubernetes v1.16
release. When this feature is enabled, the
field remains part of
the Kubernetes API, but is always unset.
: Allow assigning StorageClass to unbound PVCs retroactively.
: Enable the rotation of the server TLS certificate on the kubelet.
See
: Speed up container startup by mounting volumes with the correct
SELinux label instead of changing each file on the volumes recursively. The initial implementation
focused on ReadWriteOncePod volumes.
: Enables the use of
as the default seccomp profile
for all workloads.
The seccomp profile is specified in the
of a Pod and/or a Container.
: Allows kubelet to mount volumes for a Pod directly with the
right SELinux label instead of applying the SELinux label recursively on every file on the
volume.
Sever Side Apply (SSA)
feature on the API Server.
: Enables server-side field validation. This means the validation
of resource schema is performed at the API server side rather than the client side
(for example, the
or
command line).
field on Services
: Enables a strategy for Services ClusterIP allocations, whereby the
ClusterIP range is subdivided. Dynamic allocated ClusterIP addresses will be allocated preferently
from the upper range allowing users to assign static ClusterIPs from the lower range with a low
risk of collision. See
: Enable kubelets to determine the size limit for
memory-backed volumes (mainly
emptyDir
volumes).
: Allows
to be respected by
the StatefulSet controller.
: Allow configuration of the start ordinal in a
StatefulSet. See
Start ordinal
storage version API
.
: Allow API servers to expose the storage version hash in the
discovery.
: Enables topology aware routing based on topology hints
in EndpointSlices. See
Topology Aware
Hints
for more
details.
: Enable a mechanism to coordinate fine-grained hardware resource
assignments for different components in Kubernetes. See
.
: Allow fine-tuning of to

pology manager policies,
experimental, Alpha-quality options.
a group
of topology manager options whose quality level is alpha.
: Allow fine-tuning of topology manager policies,
experimental, Beta-quality options.
a group
of topology manager options whose quality level is beta.
: Enable user namespace support for stateless Pods.
: Enable
support for CEL validations be used in Admission Control.
: Enable support for prioritizing nodes in different
topologies based on available PV capacity.
: Enable support for watch bookmark events.
WinDSR
: Allows kube-proxy to create DSR loadbalancers for Windows.
: Allows kube-proxy to run in overlay mode for Windows.
: Enables support for joining Windows containers to a hosts' network namespace.
: Enables support for Windows HostProcess containers.
The
deprecation policy
for Kubernetes explains
the project's approach to removing features and components.
Since Kubernetes 1.24, new beta APIs are not enabled by default. When enabling a beta
feature, you will also need to enable any associated API resources.
For example, to enable a particular resource like
, set
.
See
API Versioning
for more details on the command line flags.
12.2 - Feature Gates (removed)
This page contains list of feature gates that have been removed. The information on this page is for reference.
A removed feature gate is different from a GA'ed or deprecated one in that a removed one is
no longer recognized as a valid feature gate.
However, a GA'ed or a deprecated feature gate is still recognized by the corresponding Kubernetes
components although they are unable to cause any behavior differences in a cluster.
For feature gates that are still recognized by the Kubernetes components, please refer to
the
or the
Feature gates that are removed
In the following table:
The "From" column contains the Kubernetes release when a feature is introduced
The "To" column, if not empty, contains the last Kubernetes release in which
you can still use a feature gate. If the feature stage is either "Deprecated"
or "GA", the "To" column is the Kubernetes release when the feature is removed.
Feature Gates Removed
Feature
Default
Stage
From
To
Accelerators
false
Alpha
1.6
1.10
-
1.11
1.11
AffinityInAnnotations
false
Alpha
1.6
1.7
-
1.8
1.8
AllowExtTrafficLocalEndpoints
false
Beta
1.4
1.6
true
GA
1.7
1.9
AllowInsecureBackendProxy
true
Beta
1.17
1.20
true
GA
1.21
1.25
AttachVolumeLimit
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.21
BalanceAttachedNodeVolumes
false
Alpha
1.11
1.21
false
1.22
1.22
BlockVolume
false
Alpha
1.9
1.12
true
Beta
1.13
1.17
true
GA
1.18
1.21
BoundServiceAccountTokenVolume
false
Alpha
1.13
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.23
CRIContainerLogRotation
false
Alpha
1.10
1.10
true
Beta
1.11
1.20
true
GA
1.21
1.22
CSIBlockVolume
false
Alpha
1.11
1.13
true
Beta
1.14
1.17
true
GA
1.18
1.21
CSIDriverRegistry
false
Alpha
1.12
1.13
true
Beta
1.14
1.17
true
GA
1.18
1.21
CSIMigrati

onAWSComplete
false
Alpha
1.17
1.20
CSIMigrationAWSComplete
-
1.21
1.21
CSIMigrationAzureDiskComplete
false
Alpha
1.17
1.20
-
1.21
1.21
CSIMigrationAzureFileComplete
false
Alpha
1.17
1.20
-
1.21
1.21
CSIMigrationGCEComplete
false
Alpha
1.17
1.20
-
1.21
1.21
CSIMigrationOpenStack
false
Alpha
1.14
1.17
true
Beta
1.18
1.23
true
GA
1.24
1.25
CSIMigrationOpenStackComplete
false
Alpha
1.17
1.20
-
1.21
1.21
CSIMigrationvSphereComplete
false
Beta
1.19
1.21
-
1.22
1.22
CSINodeInfo
false
Alpha
1.12
1.13
true
Beta
1.14
1.16
true
GA
1.17
1.22
CSIPersistentVolume
false
Alpha
1.9
1.9
true
Beta
1.10
1.12
true
GA
1.13
1.16
CSIServiceAccountToken
false
Alpha
1.20
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.24
CSIVolumeFSGroupPolicy
false
Alpha
1.19
1.19
true
Beta
1.20
1.22
true
GA
1.23
1.25
ConfigurableFSGroupPolicy
false
Alpha
1.18
1.19
true
Beta
1.20
1.22
true
GA
1.23
1.25
CronJobControllerV2
false
Alpha
1.20
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.23
CSRDuration
true
Beta
1.22
1.23
true
GA
1.24
1.25
CustomPodDNS
false
Alpha
1.9
1.9
true
Beta
1.10
1.13
true
GA
1.14
1.16
CustomResourceDefaulting
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
1.18
CustomResourcePublishOpenAPI
false
Alpha
1.14
1.14
true
Beta
1.15
1.15
true
GA
1.16
1.18
CustomResourceSubresources
false
Alpha
1.10
1.10
true
Beta
1.11
1.15
true
GA
1.16
1.18
CustomResourceValidation
false
Alpha
1.8
1.8
true
Beta
1.9
1.15
true
GA
1.16
1.18
false
Alpha
1.13
1.14
true
Beta
1.15
1.15
true
GA
1.16
1.18
DefaultPodTopologySpread
false
Alpha
1.19
1.19
true
Beta
1.20
1.23
true
GA
1.24
1.25
DynamicAuditing
false
Alpha
1.13
1.18
-
1.19
1.19
false
Alpha
1.4
1.10
true
Beta
1.11
1.21
false
1.22
1.25
DynamicProvisioningScheduling
false
Alpha
1.11
1.11
-
1.12
-
DynamicVolumeProvisioning
true
Alpha
1.3
1.7
true
GA
1.8
1.12
EnableAggregatedDiscoveryTimeout
true
1.16
1.17
EnableEquivalenceClassCache
false
Alpha
1.8
1.12
-
1.13
1.23
false
Alpha
1.16
1.16
false
Beta
1.17
1.17
true
Beta
1.18
1.20
true
GA
1.21
1.24
EndpointSliceNodeName
false
Alpha
1.20
1.20
true
GA
1.21
1.24
EndpointSliceProxying
false
Alpha
1.18
1.18
true
Beta
1.19
1.21
true
GA
1.22
1.24
EvenPodsSpread
false
Alpha
1.16
1.17
true
Beta
1.18
1.18
true
GA
1.19
1.21
ExperimentalCriticalPodAnnotation
false
Alpha
1.5
1.12
false
1.13
1.16
ExternalPolicyForExternalIP
true
GA
1.18
1.22
GCERegionalPersistentDisk
true
Beta
1.10
1.12
true
GA
1.13
1.16
GenericEphemeralVolume
false
Alpha
1.19
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
HugePageStorageMediumSize
false
Alpha
1.18
1.18
true
Beta
1.19
1.21
true
GA
1.22
1.24
HugePages
false
Alph

a
1.8
1.9
HugePages
true
Beta
1.10
1.13
HugePages
true
GA
1.14
1.16
HyperVContainer
false
Alpha
1.10
1.19
false
1.20
1.20
IPv6DualStack
false
Alpha
1.15
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
ImmutableEphemeralVolumes
false
Alpha
1.18
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.24
IndexedJob
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
IngressClassNamespacedParams
false
Alpha
1.21
1.21
true
Beta
1.22
1.22
true
GA
1.23
1.24
Initializers
false
Alpha
1.7
1.13
-
1.14
1.14
KubeletConfigFile
false
Alpha
1.8
1.9
-
1.10
1.10
KubeletPluginsWatcher
false
Alpha
1.11
1.11
true
Beta
1.12
1.12
true
GA
1.13
1.16
LegacyNodeRoleBehavior
false
Alpha
1.16
1.18
true
Beta
1.19
1.20
false
GA
1.21
1.22
MountContainers
false
Alpha
1.9
1.16
false
1.17
1.17
MountPropagation
false
Alpha
1.8
1.9
true
Beta
1.10
1.11
true
GA
1.12
1.14
true
Beta
1.21
1.21
true
GA
1.22
1.23
NodeDisruptionExclusion
false
Alpha
1.16
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.22
NodeLease
false
Alpha
1.12
1.13
NodeLease
true
Beta
1.14
1.16
NodeLease
true
GA
1.17
1.23
NonPreemptingPriority
false
Alpha
1.15
1.18
true
Beta
1.19
1.23
true
GA
1.24
1.25
PVCProtection
false
Alpha
1.9
1.9
-
1.10
1.10
PersistentLocalVolumes
false
Alpha
1.7
1.9
true
Beta
1.10
1.13
true
GA
1.14
1.16
PodAffinityNamespaceSelector
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.3
1.4
true
Beta
1.5
1.20
true
GA
1.21
1.25
PodOverhead
false
Alpha
1.16
1.17
true
Beta
1.18
1.23
true
GA
1.24
1.25
PodPriority
false
Alpha
1.8
1.10
true
Beta
1.11
1.13
true
GA
1.14
1.18
PodReadinessGates
false
Alpha
1.11
1.11
true
Beta
1.12
1.13
true
GA
1.14
1.16
PodShareProcessNamespace
false
Alpha
1.10
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.19
PreferNominatedNode
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.15
1.16
-
1.17
1.17
ResourceLimitsPriorityFunction
false
Alpha
1.9
1.18
-
1.19
1.19
ResourceQuotaScopeSelectors
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
RootCAConfigMap
false
Alpha
1.13
1.19
true
Beta
1.20
1.20
true
GA
1.21
1.22
RotateKubeletClientCertificate
true
Beta
1.8
1.18
true
GA
1.19
1.21
RunAsGroup
true
Beta
1.14
1.20
true
GA
1.21
1.22
false
Alpha
1.12
1.13
true
Beta
1.14
1.19
true
GA
1.20
1.24
SCTPSupport
false
Alpha
1.12
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.22
ScheduleDaemonSetPods
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
SelectorIndex
false
Alpha
1.18
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.25
ServiceAccountIssuerDiscovery
false
Alpha
1.18
1.19
true
Beta
1.20
1.20
true
GA
1.21
1.23
ServiceAppProtocol
false
Alpha
1.18
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.22
ServiceLBNodePortControl
fals

e
Alpha
1.20
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
ServiceLoadBalancerClass
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
ServiceLoadBalancerFinalizer
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
1.20
false
Alpha
1.8
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.22
ServiceTopology
false
Alpha
1.17
1.19
false
1.20
1.22
SetHostnameAsFQDN
false
Alpha
1.19
1.19
true
Beta
1.20
1.21
true
GA
1.22
1,24
StartupProbe
false
Alpha
1.16
1.17
true
Beta
1.18
1.19
true
GA
1.20
1.23
true
Beta
1.10
1.10
true
GA
1.11
1.24
StreamingProxyRedirects
false
Beta
1.5
1.5
true
Beta
1.6
1.17
true
1.18
1.21
false
1.22
1.24
SupportIPVSProxyMode
false
Alpha
1.8
1.8
false
Beta
1.9
1.9
true
Beta
1.10
1.10
true
GA
1.11
1.20
SupportNodePidsLimit
false
Alpha
1.14
1.14
true
Beta
1.15
1.19
true
GA
1.20
1.23
SupportPodPidsLimit
false
Alpha
1.10
1.13
true
Beta
1.14
1.19
true
GA
1.20
1.23
SuspendJob
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
Sysctls
true
Beta
1.11
1.20
Sysctls
true
GA
1.21
1.22
TTLAfterFinished
false
Alpha
1.12
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
TaintBasedEvictions
false
Alpha
1.6
1.12
true
Beta
1.13
1.17
true
GA
1.18
1.20
false
Alpha
1.8
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
false
Alpha
1.10
1.11
true
Beta
1.12
1.19
true
GA
1.20
1.21
TokenRequestProjection
false
Alpha
1.11
1.11
true
Beta
1.12
1.19
true
GA
1.20
1.21
ValidateProxyRedirects
false
Alpha
1.12
1.13
true
Beta
1.14
1.21
true
1.22
1.24
VolumePVCDataSource
false
Alpha
1.15
1.15
true
Beta
1.16
1.17
true
GA
1.18
1.21
VolumeScheduling
false
Alpha
1.9
1.9
true
Beta
1.10
1.12
true
GA
1.13
1.16
VolumeSnapshotDataSource
false
Alpha
1.12
1.16
true
Beta
1.17
1.19
true
GA
1.20
1.22
VolumeSubpath
true
GA
1.10
1.24
VolumeSubpathEnvExpansion
false
Alpha
1.14
1.14
true
Beta
1.15
1.16
true
GA
1.17
1.24
WarningHeaders
true
Beta
1.19
1.21
true
GA
1.22
1.24
WindowsEndpointSliceProxying
false
Alpha
1.19
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.24
WindowsGMSA
false
Alpha
1.14
1.15
true
Beta
1.16
1.17
true
GA
1.18
1.20
WindowsRunAsUserName
false
Alpha
1.16
1.16
true
Beta
1.17
1.17
true
GA
1.18
1.20
Descriptions for removed feature gates
: Provided an early form of plugin to enable Nvidia GPU support when using
Docker Engine; no longer available. See
for
an alternative.
Pod affinity or anti-affinity
.
: Enable a service to route external requests to node local endpoints.
: Enable the users to skip TLS verification of
kubelets on Pod log requests.
: Enable volume plugins to report limits on number of volumes
that can be attached to a node.
See
dynamic volume limits
: Include volu

me count on node to be considered for
balanced resource allocation while scheduling. A node which has closer CPU,
memory utilization, and volume count is favored by the scheduler while making decisions.
: Enable the definition and consumption of raw block devices in Pods.
See
Raw Block Volume Support
: Migrate ServiceAccount volumes to use a projected volume
consisting of a ServiceAccountTokenVolumeProjection. Cluster admins can use metric
to monitor workloads that are depending on the extended
tokens. If there are no such workloads, turn off extended tokens by starting
with
flag
--service-account-extend-token-expiration=false
.
Check
Bound Service Account Tokens
: Enable container log rotation for CRI container runtime.
The default max size of a log file is 10MB and the default max number of
log files allowed for a container is 5.
These values can be configured in the kubelet config.
See
logging at node level
: Enable external CSI volume drivers to support block storage.
See
csi
raw block volume support
: Enable all logic related to the CSIDriver API object in
csi.storage.k8s.io
.
: Stops registering the EBS in-tree plugin in
kubelet and volume controllers and enables shims and translation logic to
route volume operations from the AWS-EBS in-tree plugin to EBS CSI plugin.
Requires CSIMigration and CSIMigrationAWS feature flags enabled and EBS CSI
plugin installed and configured on all nodes in the cluster. This flag has
been deprecated in favor of the
feature flag
which prevents the registration of in-tree EBS plugin.
: Stops registering the Azure-Disk in-tree
plugin in kubelet and volume controllers and enables shims and translation
logic to route volume operations from the Azure-Disk in-tree plugin to
AzureDisk CSI plugin. Requires CSIMigration and CSIMigrationAzureDisk feature
flags enabled and AzureDisk CSI plugin installed and configured on all nodes
in the cluster. This flag has been deprecated in favor of the
feature flag which prevents the registration
of in-tree AzureDisk plugin.
: Stops registering the Azure-File in-tree
logic to route volume operations from the Azure-File in-tree plugin to
AzureFile CSI plugin. Requires CSIMigration and CSIMigrationAzureFile feature
flags enabled and AzureFile CSI plugin installed and configured on all nodes
of in-tree AzureFile plugin.
: Stops registering the GCE-PD in-tree plugin in
route volume operations from the GCE-PD in-tree plugin to PD CSI plugin.
Requires CSIMigration and CSIMigrationGCE feature flags enabled and PD CSI
feature flag which
prevents the registration of in-tree GCE PD plugin.
operations from the Cinder in-tree plugin to Cinder CSI plugin. Supports
falling back to in-tree Cinder plugin for mount operations to nodes that have
the feature disabled or that do not have Cinder CSI plugin installed and
the CSI plugin must be installed and configured. Requires CSIMigration
feature flag enabled.
: Stops registering the Cinder in-tree plugin in
kubelet and volume controllers and enables shims an

d translation logic to route
volume operations from the Cinder in-tree plugin to Cinder CSI plugin.
Requires CSIMigration and CSIMigrationOpenStack feature flags enabled and Cinder
CSI plugin installed and configured on all nodes in the cluster. This flag has
which prevents the registration of in-tree openstack cinder plugin.
and volume controllers and enables shims and translation logic to route volume operations
from the vSphere in-tree plugin to vSphere CSI plugin. Requires CSIMigration and
CSIMigrationvSphere feature flags enabled and vSphere CSI plugin installed and
configured on all nodes in the cluster. This flag has been deprecated in favor
of the
feature flag which prevents the
registration of in-tree vsphere plugin.
: Enable all logic related to the CSINodeInfo API object in
.
: Enable discovering and mounting volumes provisioned through a
CSI (Container Storage Interface)
compatible volume plugin.
: Enable CSI drivers to receive the pods' service account token
that they mount volumes for. See
Token Requests
.
: Allows CSIDrivers to use the
field.
This field controls whether volumes created by a CSIDriver support volume ownership
and permission modifications when these volumes are mounted.
: Allows clients to request a duration for certificates issued
via the Kubernetes CSR API.
: Allows user to configure volume permission change policy
for fsGroups when mounting a volume in a Pod. See
: Use an alternative implementation of the
CronJob
controller. Otherwise,
version 1 of the same controller is selected.
: Enable customizing the DNS settings for a Pod using its
dnsConfig
property.
Check
Pod's DNS Config
: Enable CRD support for default values in OpenAPI v3 validation schemas.
: Enables publishing of CRD OpenAPI specs.
: Enable
/status
and
/scale
subresources
on resources created from
.
: Enable schema based validation on resources created from
.
: Enable webhook-based conversion
.
PodTopologySpread
scheduling plugin to do
default spreading
.
: Used to enable dynamic auditing before v1.19.
: Enable the dynamic configuration of kubelet. The
feature is no longer supported outside of supported skew policy. The feature
gate was removed from kubelet in 1.24. See
Reconfigure kubelet
.
: Extend the default scheduler to be aware of
volume topology and handle PV provisioning.
This feature was superseded by the
feature in v1.12.
of persistent volumes to Pods.
: Enable the five second
timeout on aggregated discovery calls.
: Enable the scheduler to cache equivalence of
nodes when scheduling Pods.
: Enables EndpointSlices for more scalable and extensible
network endpoints. See
Enabling EndpointSlices
.
nodeName
field.
: When enabled, kube-proxy running
on Linux will use EndpointSlices as the primary data source instead of
Endpoints, enabling scalability and performance improvements. See
Enabling Endpoint Slices
.
: Enable pods to be scheduled evenly across topology domains. See
Pod Topolo

gy Spread Constraints
.
: Enable annotating specific pods as
critical
so that their
scheduling is guaranteed
.
This feature is deprecated by Pod Priority and Preemption as of v1.13.
: Fix a bug where ExternalTrafficPolicy is not
applied to Service ExternalIPs.
: Enable the regional PD feature on GCE.
: Enables ephemeral, inline volumes that support all features
of normal volumes (can be provided by third-party storage vendors, storage capacity tracking,
restore from snapshot, etc.).
See
.
: Enable support for multiple sizes pre-allocated
huge pages
.
HugePages
: Enable the allocation and consumption of pre-allocated
huge pages
.
: Enable
Hyper-V isolation
for Windows containers.
: Enable
dual stack
support for IPv6.
: Allows for marking individual Secrets and ConfigMaps as
immutable for better safety and performance.
: Allows the
Job
controller to manage Pod completions per completion index.
: Allow namespace-scoped parameters reference in
resource. This feature adds two fields -
Scope
and
Namespace
to
IngressClass.spec.parameters
.
: Allow asynchronous coordination of object creation using the
Initializers admission plugin.
: Enable loading kubelet configuration from
a file specified using a config file.
See
setting kubelet parameters via a config file
: Enable probe-based plugin watcher utility to enable kubelet
to discover plugins such as
CSI volume drivers
.
: When disabled, legacy behavior in service load balancers and
node disruption will ignore the
label in favor of the
feature-specific labels provided by
and
.
: Enable using utility containers on host as the volume mounter.
: Enable sharing volume mounted by one container to other containers or pods.
mount propagation
.
: Configure the API Server to set an immutable
label
on all namespaces, containing the namespace name.
: Enable use of the Node label
node.kubernetes.io/exclude-disruption
which prevents nodes from being evacuated during zone failures.
NodeLease
: Enable the new Lease API to report node heartbeats, which could be used as a node health signal.
: Enable
field for PriorityClass and Pod.
: Enable the prevention of a PersistentVolumeClaim (PVC) from
being deleted when it is still used by any Pod.
: Enable the usage of
local
volume type in Pods.
Pod affinity has to be specified if requesting a
local
volume.
Pod Affinity Namespace Selector
and
CrossNamespacePodAffinity
quota scope features.
feature.
feature to account for pod overheads.
: Enable the descheduling and preemption of Pods based on their
priorities
.
: Enable the setting of
PodReadinessGate
field for extending
Pod readiness evaluation. See
Pod readiness gate
in a Pod for sharing
a single process namespace between containers running in a pod. More details can be found in
.
: This flag tells the scheduler whether the nominated
nodes will be checked first before looping through all the other nodes in
the cluster.
: Enables managing request concurrency with prioritization and fairness
at each API server. Deprecated by
APIPriorityA

ndFairness
since 1.17.
: Enable a scheduler priority function that
assigns a lowest possible score of 1 to a node that satisfies at least one of
the input Pod's cpu and memory limits. The intent is to break ties between
nodes with same scores.
: Enable resource quota scope selectors.
: Configure the
to publish a
ConfigMap
named
to every namespace. This ConfigMap contains a CA bundle used for verifying connections
to the kube-apiserver. See
: Enable the rotation of the client TLS certificate on the kubelet.
See
: Enable control over the primary group ID set on the init processes of containers.
feature for
selecting container runtime configurations.
SCTP
protocol
value in Pod, Service, Endpoints, EndpointSlice,
and NetworkPolicy definitions.
: Enable DaemonSet Pods to be scheduled by the default scheduler instead
of the DaemonSet controller.
: Allows label and field based indexes in API server watch cache to accelerate
list operations.
: Enable OIDC discovery endpoints (issuer and JWKS URLs) for the
service account issuer in the API server. See
appProtocol
field on Services and Endpoints.
field on Services. See
Specifying class of load balancer implementation
: Enable finalizer protection for Service load balancers.
field on Services.
: Enable the exclusion of nodes from load balancers created by a cloud provider.
A node is eligible for exclusion if labelled with "
node.kubernetes.io/exclude-from-external-load-balancers
".
: Enable service to route traffic based upon the Node topology of the cluster.
See
: Enable the ability of setting Fully Qualified Domain Name(FQDN) as the
hostname of a pod. See
Pod's
field
.
startup
probe in the kubelet.
: Postpone the deletion of PersistentVolume or
PersistentVolumeClaim objects if they are still being used.
: Instructs the API server to intercept (and follow) redirects from the
backend (kubelet) for streaming requests. Examples of streaming requests include the
exec
,
attach
and
requests.
: Enable providing in-cluster service load balancing using IPVS.
See
service proxies
: Enable the support to limiting PIDs on the Node. The parameter
pid=&lt;number&gt;
in the
and
options can be specified to
ensure that the specified number of process IDs will be reserved for the system as a whole and for
Kubernetes system daemons respectively.
: Enable the support to limiting PIDs in Pods.
: Enable support to suspend and resume Jobs. For more details, see
the Jobs docs
.
Sysctls
: Enable support for namespaced kernel parameters (sysctls) that can be set for each
pod. See
sysctls
: Allow a
TTL controller
to clean up resources after they finish execution.
: Enable evicting pods from nodes based on taints on Nodes and tolerations
on Pods. See
taints and tolerations
: Enable automatic tainting nodes based on
node conditions
.
endpoint on service account resources.
: Enable the injection of service account tokens 

into a Pod through a
projected
volume
.
: This flag controls whether the API server should validate that redirects
are only followed to the same host. Only used if the
flag is enabled.
: Enable support for specifying an existing PVC as a DataSource.
: Enable volume topology aware scheduling and make the PersistentVolumeClaim
(PVC) binding aware of scheduling decisions. It also enables the usage of
local
volume type when used together with the
: Enable volume snapshot data source support.
: Allow mounting a subpath of a volume in a container.
: Enable
subPathExpr
field for expanding environment
variables into a
subPath
.
: Allow sending warning headers in API responses.
: When enabled, kube-proxy running on Windows will use
EndpointSlices as the primary data source instead of Endpoints, enabling scalability and
performance improvements. See
.
: Enables passing of GMSA credential specs from pods to container runtimes.
: Enable support for running applications in Windows containers with as a
non-default user. See
Configuring RunAsUserName
12.3 - kubelet
Synopsis
The kubelet is the primary "node agent" that runs on each node. It can
register the node with the apiserver using one of: the hostname; a flag to
override the hostname; or specific logic for a cloud provider.
The kubelet works in terms of a PodSpec. A PodSpec is a YAML or JSON object
that describes a pod. The kubelet takes a set of PodSpecs that are provided
through various mechanisms (primarily through the apiserver) and ensures that
the containers described in those PodSpecs are running and healthy. The
kubelet doesn't manage containers which were not created by Kubernetes.
Other than from a PodSpec from the apiserver, there are two ways that a
container manifest can be provided to the Kubelet.
File: Path passed as a flag on the command line. Files under this path will be
monitored periodically for updates. The monitoring period is 20s by default
and is configurable via a flag.
HTTP endpoint: HTTP endpoint passed as a parameter on the command line. This
endpoint is checked every 20 seconds (also configurable with a flag).
kubelet [flags]
Options
If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see
here
.)
--address string     Default: 0.0.0.0
The IP address for the Kubelet to serve on (set to
0.0.0.0
or
::
for listening in all interfaces and IP families) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
kubelet-config-file
for more information.)
--allowed-unsafe-sysctls strings
Comma-separated whitelist of unsafe sysctls or unsafe sysctl patterns (ending in
*
). Use these at your own risk. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
Log to standard error as well as files (DEPRECATED: will be removed in a future release, see
here
.)
Enables anonymous requests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of
, and a group name of
. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
Use the
API to determine authentication for bearer tokens. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See

)
--authentication-token-webhook-cache-ttl duration     Default:
2m0s
The duration to cache responses from the webhook token authenticator. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--authorization-mode string     Default:
Authorization mode for Kubelet server. Valid options are AlwaysAllow or Webhook. Webhook mode uses the SubjectAccessReview API to determine authorization. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See
--authorization-webhook-cache-authorized-ttl duration     Default:
5m0s
The duration to cache 'authorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--authorization-webhook-cache-unauthorized-ttl duration     Default:
30s
The duration to cache 'unauthorized' responses from the webhook authorizer. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--bootstrap-kubeconfig string
Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by
does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by
. The client certificate and key file will be stored in the directory pointed by
.
--cert-dir string     Default:
The directory where the TLS certs are located. If
and
are provided, this flag will be ignored.
--cgroup-driver string     Default:
cgroupfs
Driver that the kubelet uses to manipulate cgroups on the host. Possible values:
cgroupfs
,
systemd
--config
flag. See
--cgroup-root string     Default:
''
Optional root cgroup to use for pods. This is handled by the container runtime on a best effort basis. Default: '', which means use the container runtime default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See
--cgroups-per-qos     Default:
true
Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the
of the client certificate. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See
The path to the cloud provider configuration file. Empty string for no configuration file. (DEPRECATED: will be removed in 1.24 or later, in favor of removing cloud providers code from kubelet.)
The provider for cloud services. Set to empty string for running with no cloud provider. If set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used). (DEPRECATED: will be removed in 1.24 or later, in favor of removing cloud provider code from Kubelet.)
--cluster-dns strings
Comma-separated list of DNS server IP address. This value is used

 for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst".
Note:
all DNS servers appearing in the list MUST serve the same set of records otherwise name resolution within the cluster may not work correctly. There is no guarantee as to which DNS server may be contacted for name resolution. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--cluster-domain string
Domain for this cluster. If set, kubelet will configure all containers to search this domain in addition to the host's search domains (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
The Kubelet will load its initial configuration from this file. The path may be absolute or relative; relative paths start at the Kubelet's current working directory. Omit this flag to use the built-in default configuration values. Command-line flags override configuration from this file.
--container-log-max-files int32     Default: 5
&lt;Warning: Beta feature&gt; Set the maximum number of container log files that can be present for a container. The number must be &gt;= 2. This flag can only be used with
. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See
--container-log-max-size string     Default:
10Mi
&lt;Warning: Beta feature&gt; Set the maximum size (e.g.
10Mi
) of container log file before it is rotated. This flag can only be used with
--config
flag. See
--container-runtime string     Default:
remote
The container runtime to use. Possible values:
docker
,
remote
. (DEPRECATED: will be removed in 1.27 as the only valid value is 'remote')
--container-runtime-endpoint string
The endpoint of remote runtime service. Unix Domain Sockets are supported on Linux, while npipe and tcp endpoints are supported on windows. Examples:
unix:///path/to/runtime.sock
,
npipe:////./pipe/runtime
.
Enable lock contention profiling, if profiling is enabled (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--cpu-cfs-quota     Default:
true
Enable CPU CFS quota enforcement for containers that specify CPU limits (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--cpu-cfs-quota-period duration     Default:
100ms
Sets CPU CFS quota period value,
cpu.cfs_period_us
, defaults to Linux Kernel default. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--cpu-manager-policy string     Default:
none
CPU Manager policy to use. Possible values:
none
,
static
--config
flag. See
--cpu-manager-policy-options mapStringString
A set of key=value CPU Manager policy options to use, to fine tune their behaviour. If not supplied, keep the default behaviour. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--cpu-manager-reconcile-period duration     Default:
10s
&lt;Warning: Alpha feature&gt; CPU Manager reconciliation period. Examples:
10s
, or
1m
. If not supplied, defaults to node status update frequency. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--enable-con

troller-attach-detach     Default:
true
Enables the Attach/Detach controller to manage attachment/detachment of volumes scheduled to this node, and disables kubelet from executing any attach/detach operations. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--enable-debugging-handlers     Default:
true
Enables server endpoints for log collection and local running of containers and commands. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--enable-server     Default:
true
Enable the Kubelet's server. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--enforce-node-allocatable strings     Default:
pods
A comma separated list of levels of node allocatable enforcement to be enforced by kubelet. Acceptable options are
none
,
pods
,
, and
. If the latter two options are specified,
and
must also be set, respectively. If
none
is specified, no additional options should be set. See
here
for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See
--event-burst int32     Default: 10
Maximum size of a bursty event records, temporarily allows event records to burst to this number, while still not exceeding
--event-qps
. The number must be &gt;= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--event-qps int32     Default: 5
QPS to limit event creations. The number must be &gt;= 0. If 0 will use default QPS (5). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--eviction-hard mapStringString     Default:
imagefs.available&lt;15%,memory.available&lt;100Mi,nodefs.available&lt;10%
A set of eviction thresholds (e.g.
memory.available&lt;1Gi
) that if met would trigger a pod eviction. On a Linux node, the default value also includes
nodefs.inodesFree&lt;5%
--config
flag. See
--eviction-max-pod-grace-period int32
Maximum allowed grace period (in seconds) to use when terminating pods in response to a soft eviction threshold being met. If negative, defer to pod specified value. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--eviction-minimum-reclaim mapStringString
A set of minimum reclaims (e.g.
imagefs.available=2Gi
) that describes the minimum amount of resource the kubelet will reclaim when performing a pod eviction if that resource is under pressure. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--eviction-pressure-transition-period duration     Default:
5m0s
Duration for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--eviction-soft mapStringString
memory.available&lt;1.5Gi
) that if met over a corresponding grace period would trigger a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--eviction-soft-grace-period mapStringString
A set of eviction 

grace periods (e.g.
memory.available=1m30s
) that correspond to how long a soft eviction threshold must hold before triggering a pod eviction. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--exit-on-lock-contention
Whether kubelet should exit upon lock-file contention.
--experimental-allocatable-ignore-eviction     Default:
false
When set to
true
, hard eviction thresholds will be ignored while calculating node allocatable. See
here
for more details. (DEPRECATED: will be removed in 1.24 or later)
--experimental-kernel-memcg-notification
Use kernelMemcgNotification configuration, this flag will be removed in 1.24 or later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--experimental-mounter-path string     Default:
mount
[Experimental] Path of mounter binary. Leave empty to use the default
mount
. (DEPRECATED: will be removed in 1.24 or later, in favor of using CSI.)
--fail-swap-on     Default:
true
Makes the Kubelet fail to start if swap is enabled on the node. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--feature-gates &lt;A list of 'key=true/false' pairs&gt;
A set of
key=value
pairs that describe feature gates for alpha/experimental features. Options are:
APIServerIdentity=true|false (ALPHA - default=false)
CPUManager=true|false (BETA - default=true)
CSIInlineVolume=true|false (BETA - default=true)
CSIMigration=true|false (BETA - default=true)
CSIMigrationAWS=true|false (BETA - default=true)
CSIMigrationAzureFile=true|false (BETA - default=true)
CSIMigrationGCE=true|false (BETA - default=true)
CSIMigrationPortworx=true|false (ALPHA - default=false)
CSIMigrationvSphere=true|false (BETA - default=false)
CronJobTimeZone=true|false (ALPHA - default=false)
CustomResourceValidationExpressions=true|false (ALPHA - default=false)
DaemonSetUpdateSurge=true|false (BETA - default=true)
DelegateFSGroupToCSIDriver=true|false (BETA - default=true)
DevicePlugins=true|false (BETA - default=true)
DisableAcceleratorUsageMetrics=true|false (BETA - default=true)
EndpointSliceTerminatingCondition=true|false (BETA - default=true)
EphemeralContainers=true|false (BETA - default=true)
ExpandedDNSConfig=true|false (ALPHA - default=false)
IdentifyPodOS=true|false (BETA - default=true)
InTreePluginAzureDiskUnregister=true

|false (ALPHA - default=false)
JobTrackingWithFinalizers=true|false (BETA - default=false)
KubeletCredentialProviders=true|false (BETA - default=true)
LegacyServiceAccountTokenNoAutoGeneration=true|false (BETA - default=true)
LocalStorageCapacityIsolation=true|false (BETA - default=true)
MinDomainsInPodTopologySpread=true|false (ALPHA - default=false)
MixedProtocolLBService=true|false (BETA - default=true)
NetworkPolicyEndPort=true|false (BETA - default=true)
NodeOutOfServiceVolumeDetach=true|false (ALPHA - default=false)
PodSecurity=true|false (BETA - default=true)
ProbeTerminationGracePeriod=true|false (BETA - default=false)
ProxyTerminatingEndpoints=true|false (ALPHA - default=false)
SeccompDefault=true|false (ALPHA - default=false)
ServerSideFieldValidation=true|false (ALPHA - default=false)
ServiceIPStaticSubrange=true|false (ALPHA - default=false)
ServiceInternalTrafficPolicy=true|false (BETA - default=true)
StatefulSetMinReadySeconds=true|false (BETA - default=true)
WindowsHostProcessContainers=true|false (BETA - default=true)
(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--file-check-frequency duration     Default:
20s
Duration between checking config files for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--hairpin-mode string     Default:
How should the kubelet setup hairpin NAT. This allows endpoints of a Service to load balance back to themselves if they should try to access their own Service. Valid values are
,
and
none
--config
flag. See
--healthz-bind-address string     Default:
127.0.0.1
Th

e IP address for the healthz server to serve on (set to
0.0.0.0
or
::
for listening in all interfaces and IP families). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--healthz-port int32     Default: 10248
The port of the localhost healthz endpoint (set to
0
to disable). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
-h, --help
--hostname-override string
If non-empty, will use this string as identification instead of the actual hostname. If
is set, the cloud provider determines the name of the node (consult cloud provider documentation to determine if and how the hostname is used).
--http-check-frequency duration     Default:
20s
Duration between checking HTTP for new data. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--image-credential-provider-bin-dir string
The path to the directory where credential provider plugin binaries are located.
--image-credential-provider-config string
The path to the credential provider plugin config file.
--image-gc-high-threshold int32     Default: 85
The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--image-gc-low-threshold int32     Default: 80
The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of
--image-gc-high-threshold
--config
flag. See
--image-service-endpoint string
[Experimental] The endpoint of remote image service. If not specified, it will be the same with
by default. Unix Domain Socket are supported on Linux, while npipe and TCP endpoints are supported on Windows. Examples:
unix:///var/run/dockershim.sock
,
npipe:////./pipe/dockershim
--iptables-drop-bit int32     Default: 15
The bit of the
fwmark
space to mark packets for dropping. Must be within the range [0, 31]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--iptables-masquerade-bit int32     Default: 14
fwmark
space to mark packets for SNAT. Must be within the range [0, 31]. Please match this parameter with corresponding parameter in
--config
flag. See
--keep-terminated-pod-volumes
Keep terminated pod volumes mounted to the node after the pod terminates. Can be useful for debugging volume related issues. (DEPRECATED: will be removed in a future version)
--kernel-memcg-notification
If enabled, the kubelet will integrate with the kernel memcg notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--kube-api-burst int32     Default: 10
Burst to use while talking with kubernetes API server. The number must be &gt;= 0. If 0 will use default burst (10). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--kube-api-content-type string     Default:
Con

tent type of requests sent to apiserver. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--kube-api-qps int32     Default: 5
QPS to use while talking with kubernetes API server. The number must be &gt;= 0. If 0 will use default QPS (5). Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--kube-reserved mapStringString     Default: &lt;None&gt;
A set of
&lt;resource name&gt;=&lt;resource quantity&gt;
(e.g.
cpu=200m,memory=500Mi,ephemeral-storage=1Gi,pid='100'
) pairs that describe resources reserved for kubernetes system components. Currently
cpu
,
memory
and local
for root file system are supported. See
here
for more detail. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--kube-reserved-cgroup string     Default:
''
Absolute name of the top level cgroup that is used to manage kubernetes components for which compute resources were reserved via
flag. Ex.
/kube-reserved
--config
flag. See
Path to a kubeconfig file, specifying how to connect to the API server. Providing
enables API server mode, omitting
enables standalone mode.
--kubelet-cgroups string
Optional absolute name of cgroups to create and run the Kubelet in. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--lock-file string
&lt;Warning: Alpha feature&gt; The path to file for kubelet to use as a lock file.
--log-backtrace-at &lt;A string of format 'file:line'&gt;     Default:
":0"
When logging hits line
:
, emit a stack trace. (DEPRECATED: will be removed in a future release, see
here
.)
If non-empty, write log files in this directory. (DEPRECATED: will be removed in a future release, see
here
.)
If non-empty, use this log file. (DEPRECATED: will be removed in a future release, see
here
.)
Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see
here
.)
--log-flush-frequency duration     Default:
5s
Maximum number of seconds between log flushes.
--log-json-info-buffer-size string     Default:
'0'
[Experimental] In JSON format with split output streams, the info messages can be buffered for a while to increase performance. The default value of zero bytes disables buffering. The size can be specified as number of bytes (512), multiples of 1000 (1K), multiples of 1024 (2Ki), or powers of those (3M, 4G, 5Mi, 6Gi). (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--log-json-split-stream
[Experimental] In JSON format, write error messages to stderr and info messages to stdout. The default is to write a single stream to stdout. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--logging-format string     Default:
text
Sets the log format. Permitted formats:
text
,
json
.
Non-default formats don't honor these flags:
,
,
,
--log-dir
,
,
,
,
--skip_headers
,
--skip_log_headers
,
,
--log-flush-frequency
.
Non-default choices are currently alpha and subject

 to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--logtostderr     Default:
true
log to standard error instead of files. (DEPRECATED: will be removed in a future release, see
here
.)
--make-iptables-util-chains     Default:
true
If true, kubelet will ensure
iptables
utility rules are present on host. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--manifest-url string
URL for accessing additional Pod specifications to run (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--manifest-url-header string
Comma-separated list of HTTP headers to use when accessing the URL provided to
--manifest-url
. Multiple headers with the same name will be added in the same order provided. This flag can be repeatedly invoked. For example:
--manifest-url-header 'a:hello,b:again,c:world' --manifest-url-header 'b:beautiful'
--config
flag. See
--master-service-namespace string     Default:
default
The namespace from which the kubernetes master services should be injected into pods. (DEPRECATED: This flag will be removed in a future version.)
--max-open-files int     Default: 1000000
Number of files that can be opened by Kubelet process. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--max-pods int32     Default: 110
Number of Pods that can run on this Kubelet. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--maximum-dead-containers int32     Default: -1
Maximum number of old instances of containers to retain globally. Each container takes up some disk space. To disable, set to a negative number. (DEPRECATED: Use
or
--eviction-soft
instead. Will be removed in a future version.)
--maximum-dead-containers-per-container int32     Default: 1
Maximum number of old instances to retain per container. Each container takes up some disk space. (DEPRECATED: Use
or
--memory-manager-policy string     Default:
None
Memory Manager policy to use. Possible values:
'None'
,
'Static'
--config
flag. See
--minimum-container-ttl-duration duration
Minimum age for a finished container before it is garbage collected. Examples:
'300ms'
,
'10s'
or
'2h45m'
(DEPRECATED: Use
or
--minimum-image-ttl-duration duration     Default:
2m0s
Minimum age for an unused image before it is garbage collected. Examples:
'300ms'
,
'10s'
or
'2h45m'
--config
flag. See
--node-ip string
IP address (or comma-separated dual-stack IP addresses) of the node. If unset, kubelet will use the node's default IPv4 address, if any, or its default IPv6 address if it has no IPv4 addresses. You can pass
'::'
to make it prefer the default IPv6 address rather than the default IPv4 address.
--node-labels mapStringString
&lt;Warning: Alpha feature&gt;Labels to add when registering the node in the cluster. Labels must be
key=value pairs
separated by
','
. Labels in the
'kubernetes.io'
namespace must begin with an allowed prefix (
'kubelet.kubernetes.io'
,
'node.kubernetes.io'
) or be in the specifically allowed set (
'beta.

kubernetes.io/arch'
,
'beta.kubernetes.io/instance-type'
,
'beta.kubernetes.io/os'
,
'failure-domain.beta.kubernetes.io/region'
,
'failure-domain.beta.kubernetes.io/zone'
,
'kubernetes.io/arch'
,
'kubernetes.io/hostname'
,
'kubernetes.io/os'
,
'node.kubernetes.io/instance-type'
,
'topology.kubernetes.io/region'
,
'topology.kubernetes.io/zone'
)
--node-status-max-images int32     Default: 50
The maximum number of images to report in
node.status.images
. If
-1
is specified, no cap will be applied. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--node-status-update-frequency duration     Default:
10s
Specifies how often kubelet posts node status to master. Note: be cautious when changing the constant, it must work with
nodeMonitorGracePeriod
in Node controller. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
If true, only write logs to their native severity level (vs also writing to each lower severity level). (DEPRECATED: will be removed in a future release, see
here
.)
--oom-score-adj int32     Default: -999
The
oom-score-adj
value for kubelet process. Values must be within the range [-1000, 1000]. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--pod-cidr string
The CIDR to use for pod IP addresses, only used in standalone mode. In cluster mode, this is obtained from the master. For IPv6, the maximum number of IP's allocated is 65536 (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--pod-infra-container-image string     Default:
registry.k8s.io/pause:3.6
Specified image will not be pruned by the image garbage collector. When container-runtime is set to
docker
, all containers in each pod will use the network/IPC namespaces from this image. Other CRI implementations have their own configuration to set this image.
--pod-manifest-path string
Path to the directory containing static pod files to run, or the path to a single static pod file. Files starting with dots will be ignored. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--pod-max-pids int     Default: -1
Set the maximum number of processes per pod. If
-1
, the kubelet defaults to the node allocatable PID capacity. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--pods-per-core int32
Number of Pods per core that can run on this kubelet. The total number of pods on this kubelet cannot exceed
--max-pods
, so
will be used if this calculation results in a larger number of pods allowed on the kubelet. A value of
0
disables this limit. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--port int32     Default: 10250
The port for the kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the kubelet's
--config
flag. See
--protect-kernel-defaults
Default kubelet behaviour for kernel tuning. If set, kubelet errors if any of kernel tunables is different than kubelet defaults. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--provider-id string
Unique identifier for identifying the node in a machine database, i.e cloud provider. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
for more informati

on.)
--qos-reserved mapStringString
&lt;Warning: Alpha feature&gt; A set of
&lt;resource name&gt;=&lt;percentage&gt;
(e.g.
memory=50%
) pairs that describe how pod resource requests are reserved at the QoS level. Currently only
memory
is supported. Requires the
feature gate to be enabled. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--read-only-port int32     Default: 10255
The read-only port for the kubelet to serve on with no authentication/authorization (set to
0
--config
flag. See
--register-node     Default:
true
Register the node with the API server. If
is not provided, this flag is irrelevant, as the Kubelet won't have an API server to register with. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See
--register-schedulable     Default:
true
Register the node as schedulable. Won't have any effect if
--register-node
is
false
. (DEPRECATED: will be removed in a future version)
--register-with-taints mapStringString
Register the node with the given list of taints (comma separated
&lt;key&gt;=&lt;value&gt;:&lt;effect&gt;
). No-op if
is
false
--registry-burst int32     Default: 10
Maximum size of a bursty pulls, temporarily allows pulls to burst to this number, while still not exceeding
--registry-qps
. Only used if
is greater than 0. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--registry-qps int32     Default: 5
If &gt; 0, limit registry pull QPS to this value. If
0
, unlimited. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--reserved-cpus string
A comma-separated list of CPUs or CPU ranges that are reserved for system and kubernetes usage. This specific list will supersede cpu counts in
and
--config
flag. See
--reserved-memory string
A comma-separated list of memory reservations for NUMA nodes. (e.g.
--reserved-memory 0:memory=1Gi,hugepages-1M=2Gi --reserved-memory 1:memory=2Gi
). The total sum for each memory type should be equal to the sum of
,
and
. See
here
for more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--resolv-conf string     Default:
Resolver configuration file used as the basis for the container DNS resolution configuration. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--root-dir string     Default:
Directory path for managing kubelet files (volume mounts, etc).
&lt;Warning: Beta feature&gt; Auto rotate the kubelet client certificates by requesting new certificates from the
when the certificate expiration approaches. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
Auto-request and rotate the kubelet serving certificates by requesting new certificates from the
when the certificate expiration approaches. 

Requires the
feature gate to be enabled, and approval of the submitted
objects. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--runonce
If
true
, exit after spawning pods from local manifests or remote urls. Exclusive with
--enable-server
(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See
--runtime-cgroups string
Optional absolute name of cgroups to create and run the runtime in.
--runtime-request-timeout duration     Default:
2m0s
Timeout of all runtime requests except long running request -
pull
,
logs
,
exec
and
attach
. When timeout exceeded, kubelet will cancel the request, throw out an error and retry later. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--seccomp-default string
&lt;Warning: Alpha feature&gt; Enable the use of
as the default seccomp profile for all workloads. The
feature gate must be enabled to allow this flag, which is disabled by default.
--serialize-image-pulls     Default:
true
Pull images one at a time. We recommend *not* changing the default value on nodes that run docker daemon with version &lt; 1.9 or an
aufs
storage backend. Issue #10959 has more details. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
If
true
, avoid header prefixes in the log messages. (DEPRECATED: will be removed in a future release, see
here
.)
If
true
, avoid headers when opening log files. (DEPRECATED: will be removed in a future release, see
here
.)
--stderrthreshold int     Default: 2
logs at or above this threshold go to stderr. (DEPRECATED: will be removed in a future release, see
here
.)
--streaming-connection-idle-timeout duration     Default:
4h0m0s
Maximum time a streaming connection can be idle before the connection is automatically closed.
0
indicates no timeout. Example:
5m
. Note: All connections to the kubelet server have a maximum duration of 4 hours. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--sync-frequency duration     Default:
1m0s
Max period between synchronizing running containers and config. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--system-cgroups string
Optional absolute name of cgroups in which to place all non-kernel processes that are not already inside a cgroup under
'/'
. Empty for no container. Rolling back the flag requires a reboot. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--system-reserved mapStringString     Default: &lt;none&gt;
A set of
(e.g.
) pairs that describe resources reserved for non-kubernetes components. Currently only
cpu
and
memory
are supported. See
here
--config
flag. See
--system-reserved-cgroup string     Default:
''
Absolute name of the top level cgroup that is used to manage non-kubernetes components for which compute resources were reserved via
flag. Ex.
/system-reserved
--config
flag. See
File containing x5

09 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If
and
are not provided, a self-signed certificate and key are generated for the public address and saved to the directory passed to
--config
flag. See
Preferred values:
`TLS_AES_128_GCM_SHA256`, `TLS_AES_256_GCM_SHA384`, `TLS_CHACHA20_POLY1305_SHA256`, `TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA`, `TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA`, `TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305`, `TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256`, `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256`, `TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384`, `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305`, `TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256`, `TLS_RSA_WITH_AES_128_CBC_SHA`, `TLS_RSA_WITH_AES_128_GCM_SHA256`, `TLS_RSA_WITH_AES_256_CBC_SHA`, `TLS_RSA_WITH_AES_256_GCM_SHA384`
Insecure values:
`TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDHE_ECDSA_WITH_RC4_128_SHA`, `TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256`, `TLS_ECDHE_RSA_WITH_RC4_128_SHA`, `TLS_RSA_WITH_3DES_EDE_CBC_SHA`, `TLS_RSA_WITH_AES_128_CBC_SHA256`, `TLS_RSA_WITH_RC4_128_SHA`.
(DEPRECATED: This parameter should be set via the config file specified by the Kubelet's `--config` flag. See
Minimum TLS version supported. Possible values:
VersionTLS10
,
VersionTLS11
,
VersionTLS12
,
VersionTLS13
--config
flag. See
File containing x509 private key matching
--config
flag. See
--topology-manager-policy string     Default:
'none'
Topology Manager policy to use. Possible values:
'none'
,
'best-effort'
,
'restricted'
,
'single-numa-node'
--config
flag. See
--topology-manager-scope string     Default:
container
Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values:
'container'
,
'pod'
Number for the log level verbosity
--vmodule &lt;A list of 'pattern=N' strings&gt;
Comma-separated list of
pattern=N
settings for file-filtered logging
--volume-plugin-dir string     Default:
/usr/libexec/kubernetes/kubelet-plugins/volume/exec/
The full path of the directory in which to search for additional third party volume plugins. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--volume-stats-agg-period duration     Default:
1m0s
Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes. To disable volume calculations, set to
0
--config
flag. See
12.4 - kube-apiserver
Synopsis
The Kubernetes API server validates and confi

gures data
Options
Maximum s

ize of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded.
API group and version used for serializing audit events written to webhook.
--authorization-webhook-version string 

    Default: "v1beta1"
admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
admission plugins that should be enabled in addition to default enabled ones 

(NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
--etcd-cafile string
The hostname to use when generating externa

lized URLs for this master (e.g. Swagger API Docs or OpenID Discovery).
InTreePluginAzureDiskUnregister=true|false (ALPHA - default=false)
MinimizeIPTablesRestore=true|false (ALPHA 

- default=false)
To prevent HTTP/2 clients from getting stuck on a single apiserver, randomly close a connection (GOAWAY). The client's other in-flight requests won't be affected, and the client will reconnect, likely landing on a different apiserver after going through the load balancer again. This argument sets the fraction of requests that will be sent a GOAWAY. Clusters with single apiservers, or which don't use a load balancer, should NOT enable this. Min is 0 (off), Max is .02 (1/50 requests); .001 (1/1000) is a recommended starting point.
-h, --help
The time in seconds that each lease is reused. A lower value could avoid large number of objects reusing the sam

e lease. Notice that a too small value may cause performance problems at storage layer.
The URL of the OpenID issuer, only HTTPS scheme will be accepted. If set, it will be used to verify the OIDC JSON Web Token (JWT).
If true, SO_REUSEPORT will be used whe

n binding the port, which allows more than one instance to bind on the same address and port. [default=false]
&lt;group&gt;/&lt;version&gt;=true|false for a specific API group and version (e.g. apps/v1=true)
Identifier of the service account token issuer. The issuer will assert this identifier in "iss" claim of issued tokens. This value is a string or URI. If this option is not a valid URI per the OpenID Discovery 1.0 spec, the ServiceAccountIssuerDiscovery feature will remain disabled, even if the feature gate is set to true. It is highly recommended that this value comply with the OpenID spec: https://openid.net/specs/openid-connect-discovery-1_0.html. In practice, this means that service-account-issuer must be an https URL. It is also h

ighly recommended that this URL be capable of serving OpenID discovery documents at {service-account-issuer}/.well-known/openid-configuration. When this flag is specified multiple times, the first is used to generate tokens and all are used to determine which issuers are accepted.
The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is &lt;major&gt;.&lt;minor&gt;, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.
File containing the default x509 Certificate for HTTPS. (CA cert, if any, concaten

ated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
-v, --v int
12.5 - kube-controller-manager
Synopsis
The Kubernetes controller manager is a daemon that embeds
the core control loops shipped with Kubernetes. In applications of robotics and
automation, a control loop is a non-terminating loop that regulates the state of
the system. In Kubernetes, a controller is a control loop that watches the shared
state of the cluster through the apiserver and makes changes attempting to move the
current state towards the desired state. Ex

amples of controllers that ship with
Kubernetes today are the replication controller, endpoints controller, namespace
controller, and serviceaccounts controller.
kube-controller-manager [flags]
Options
--allocate-node-cidrs
Should CIDRs for Pods be allocated and set on the cloud provider.
--attach-detach-reconcile-sync-period duration     Default: 1m0s
The reconciler sync wait time between volume attach detach. This duration must be larger than one second, and increasing this value from the default may allow for volumes to be mismatched with pods.
--authentication-kubeconfig string
kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenreviews.authentication.k8s.io. This is optional. If empty, all token requests are considered to be anonymous and no client CA is looked up in the cluster.
--authentication-skip-lookup
If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
--authentication-token-webhook-cache-ttl duration     Default: 10s
--authentication-tolerate-lookup-failure
If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
--authorization-always-allow-paths strings     Default: "/healthz,/readyz,/livez"
A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.
--authorization-kubeconfig string
kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped by authorization are forbidden.
--authorization-webhook-cache-authorized-ttl duration     Default: 10s
--authorization-webhook-cache-unauthorized-ttl duration     Default: 10s
--cidr-allocator-type string     Default: "RangeAllocator"
Type of CIDR allocator to use
--cluster-cidr string
CIDR Range for Pods in cluster. Requires --allocate-node-cidrs to be true
--cluster-name string     Default: "kubernetes"
The instance prefix for the cluster.
--cluster-signing-cert-file string
Filename containing a PEM-encoded X509 CA certificate used to issue cluster-scoped certificates. If specified, no more specific --cluster-signing-* flag may be specified.
--cluster-signing-duration duration     Default: 8760h0m0s
The max length of duration signed certificates will be gi

ven. Individual CSRs may request shorter certs by setting spec.expirationSeconds.
--cluster-signing-key-file string
Filename containing a PEM-encoded RSA or ECDSA private key used to sign cluster-scoped certificates. If specified, no more specific --cluster-signing-* flag may be specified.
--cluster-signing-kube-apiserver-client-cert-file string
Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/kube-apiserver-client signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--cluster-signing-kube-apiserver-client-key-file string
Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/kube-apiserver-client signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--cluster-signing-kubelet-client-cert-file string
Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/kube-apiserver-client-kubelet signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--cluster-signing-kubelet-client-key-file string
Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/kube-apiserver-client-kubelet signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--cluster-signing-kubelet-serving-cert-file string
Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/kubelet-serving signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--cluster-signing-kubelet-serving-key-file string
Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/kubelet-serving signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--cluster-signing-legacy-unknown-cert-file string
Filename containing a PEM-encoded X509 CA certificate used to issue certificates for the kubernetes.io/legacy-unknown signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--cluster-signing-legacy-unknown-key-file string
Filename containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/legacy-unknown signer. If specified, --cluster-signing-{cert,key}-file must not be set.
--concurrent-deployment-syncs int32     Default: 5
The number of deployment objects that are allowed to sync concurrently. Larger number = more responsive deployments, but more CPU (and network) load
--concurrent-endpoint-syncs int32     Default: 5
The number of endpoint syncing operations that will be done concurrently. Larger number = faster endpoint updating, but more CPU (and network) load
--concurrent-ephemeralvolume-syncs int32     Default: 5
The number of ephemeral volume syncing operations that will be done concurrently. Larger number = faster ephemeral volume updating, but more CPU (and network) load
--concurrent-gc-syncs int32     Default: 20
The number of garbage collector workers that are allowed to sync concurrently.
--concurrent-horizontal-pod-autoscaler-syncs int32     Default: 5
The number of horizontal pod autoscaler objects that are allowed to sync concurrently. Larger number = more responsive horizontal pod autoscaler objects processing, but more CPU (and network) load.
--concurrent-namespace-syncs int32     Default: 10
The number of namespace objects that are allowed to sync concurrently. Larger number = more responsive namespace termination, but more CPU (and network) load
--concurrent-rc-syncs int32     Default: 5
The number of replication controllers that are allowed to sync concurrently. Larger number = more responsive replica management, but more CPU (and network) load
--concurrent-replicaset-syncs int32     Default: 5
The number of replica sets that are allowed to sync concurrently. Larger number = more responsive replica management, but more CPU (and network) load
--concurrent-resource-quota-syncs int32     Default: 5
The number of resource quotas that

 are allowed to sync concurrently. Larger number = more responsive quota management, but more CPU (and network) load
--concurrent-service-endpoint-syncs int32     Default: 5
The number of service endpoint syncing operations that will be done concurrently. Larger number = faster endpoint slice updating, but more CPU (and network) load. Defaults to 5.
--concurrent-service-syncs int32     Default: 1
The number of services that are allowed to sync concurrently. Larger number = more responsive service management, but more CPU (and network) load
--concurrent-serviceaccount-token-syncs int32     Default: 5
The number of service account token objects that are allowed to sync concurrently. Larger number = more responsive token generation, but more CPU (and network) load
--concurrent-statefulset-syncs int32     Default: 5
The number of statefulset objects that are allowed to sync concurrently. Larger number = more responsive statefulsets, but more CPU (and network) load
--concurrent-ttl-after-finished-syncs int32     Default: 5
The number of TTL-after-finished controller workers that are allowed to sync concurrently.
--configure-cloud-routes     Default: true
Should CIDRs allocated by allocate-node-cidrs be configured on the cloud provider.
--controller-start-interval duration
Interval between starting controller managers.
--controllers strings     Default: "*"
A list of controllers to enable. '*' enables all on-by-default controllers, 'foo' enables the controller named 'foo', '-foo' disables the controller named 'foo'.
All controllers: attachdetach, bootstrapsigner, cloud-node-lifecycle, clusterrole-aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, endpointslice, endpointslicemirroring, ephemeral-volume, garbagecollector, horizontalpodautoscaling, job, namespace, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv-protection, pvc-protection, replicaset, replicationcontroller, resourcequota, root-ca-cert-publisher, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after-finished
Disabled-by-default controllers: bootstrapsigner, tokencleaner
--disable-attach-detach-reconcile-sync
Disable volume attach detach reconciler sync. Disabling this may cause volumes to be mismatched with pods. Use wisely.
--enable-dynamic-provisioning     Default: true
Enable dynamic provisioning for environments that support it.
Enables the generic garbage collector. MUST be synced with the corresponding flag of the kube-apiserver.
--enable-hostpath-provisioner
Enable HostPath PV provisioning when running without a cloud provider. This allows testing and development of provisioning features. HostPath provisioning is not supported in any way, won't work in a multi-node cluster, and should not be used for anything other than testing or development.
Whether to enable controller leader migration.
--endpoint-updates-batch-period duration
The length of endpoint updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated
--endpointslice-updates-batch-period duration
The length of endpoint slice updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower n

umber of endpoints revision generated
--external-cloud-volume-plugin string
The plugin to use when cloud provider is set to external. Can be empty, should only be set when cloud-provider is external. Currently used to allow node and volume controllers to work for in tree cloud providers.
MaxUnavailableStatefulSet=true|false (ALPH

A - default=false)
--flex-volume-plugin-dir string     Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
Full path of the directory in which the flex volume plugin should search for additional third party volume plugins.
-h, --help
help for kube-controller-manager
--horizontal-pod-autoscaler-cpu-initialization-period duration     Default: 5m0s
The period after pod start when CPU samples might be skipped.
--horizontal-pod-autoscaler-downscale-stabilization duration     Default: 5m0s
The period for which autoscaler will look backwards and not scale down below any recommendation it made during that period.
--horizontal-pod-autoscaler-initial-readiness-delay duration     Default: 30s
The period after pod start during which readiness changes will be treated as initial readiness.
--horizontal-pod-autoscaler-sync-period duration     Default: 15s
The period for syncing the number of pods in horizontal pod autoscaler.
--horizontal-pod-autoscaler-tolerance float     Default: 0.1
The minimum change (from 1.0) in the desired-to-actual metrics ratio for the horizontal pod autoscaler to consider scaling.
--kube-api-burst int32     Default: 30
Burst to use while talking with kubernetes apiserver.
--kube-api-content-type string     Default: "application/vnd.kuberne

tes.protobuf"
Content type of requests sent to apiserver.
--kube-api-qps float     Default: 20
QPS to use while talking with kubernetes apiserver.
Path to kubeconfig file with authorization and master location information.
--large-cluster-size-threshold int32     Default: 50
Number of nodes from which NodeController treats the cluster as large for the eviction logic purposes. --secondary-node-eviction-rate is implicitly overridden to 0 for clusters this size or smaller.
--leader-elect     Default: true
Start a leader election client and gain leadership before executing the main loop. Enable this when running replicated components for high availability.
--leader-elect-lease-duration duration     Default: 15s
The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is replaced by another candidate. This is only applicable if leader election is enabled.
--leader-elect-renew-deadline duration     Default: 10s
The interval between attempts by the acting master to renew a leadership slot before it stops leading. This must be less than the lease duration. This is only applicable if leader election is enabled.
--leader-elect-resource-lock string     Default: "leases"
The type of resource object that is used for locking during leader election. Supported options are 'leases', 'endpointsleases' and 'configmapsleases'.
--leader-elect-resource-name string     Default: "kube-controller-manager"
The name of resource object that is used for locking during leader election.
--leader-elect-resource-namespace string     Default: "kube-system"
The namespace of resource object that is used for locking during leader election.
--leader-elect-retry-period duration     Default: 2s
The duration the clients should wait between attempting acquisition and renewal of a leadership. This is only applicable if leader election is enabled.
--leader-migration-config string
Path to the config file for controller leader migration, or empty to use the value that reflects default configuration of the controller manager. The config file should be of type LeaderMigrationConfiguration, group controllermanager.config.k8s.io, version v1alpha1.
--master string
The address of the Kubernetes API server (overrides any value in kubeconfig).
--max-endpoints-per-slice int32     Default: 100
The maximum number of endpoints that will be added to an EndpointSlice. More endpoints per slice will result in less endpoint slices, but larger resources. Defaults to 100.
--min-resync-period duration     Default: 12h0m0s
The resync period in reflectors will be random between MinResyncPeriod and 2*MinResyncPeriod.
--mirroring-concurrent-service-endpoint-syncs int32     Default: 5
The number of service endpoint syncing operations that will be done concurrently by the EndpointSliceMirroring controller. Larger number = faster endpoint slice updating, but more CPU (and network) load. Defaults to 5.
--mirroring-endpointslice-updates-batch-period duration
The length of EndpointSlice updates batching period for EndpointSliceMirroring controller. Processing of EndpointSlice changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of EndpointSlice updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated
--mirroring-max-endpoints-per-subset int32     Default: 1000
The maximum number of endpoints that will be added to an EndpointSlice by the EndpointSliceMirroring controller. More endpoints per slice will result in less endpoint slices, but larger resources. Defaults to 100.
--namespace-sync-period duration  

   Default: 5m0s
The period for syncing namespace life-cycle updates
--node-cidr-mask-size int32
Mask size for node cidr in cluster. Default is 24 for IPv4 and 64 for IPv6.
--node-cidr-mask-size-ipv4 int32
Mask size for IPv4 node cidr in dual-stack cluster. Default is 24.
--node-cidr-mask-size-ipv6 int32
Mask size for IPv6 node cidr in dual-stack cluster. Default is 64.
--node-eviction-rate float     Default: 0.1
Number of nodes per second on which pods are deleted in case of node failure when a zone is healthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters.
--node-monitor-grace-period duration     Default: 40s
Amount of time which we allow running Node to be unresponsive before marking it unhealthy. Must be N times more than kubelet's nodeStatusUpdateFrequency, where N means number of retries allowed for kubelet to post node status.
--node-monitor-period duration     Default: 5s
The period for syncing NodeStatus in NodeController.
--node-startup-grace-period duration     Default: 1m0s
Amount of time which we allow starting Node to be unresponsive before marking it unhealthy.
--pv-recycler-increment-timeout-nfs int32     Default: 30
the increment of time added per Gi to ActiveDeadlineSeconds for an NFS scrubber pod
--pv-recycler-minimum-timeout-hostpath int32     Default: 60
The minimum ActiveDeadlineSeconds to use for a HostPath Recycler pod. This is for development and testing only and will not work in a multi-node cluster.
--pv-recycler-minimum-timeout-nfs int32     Default: 300
The minimum ActiveDeadlineSeconds to use for an NFS Recycler pod
--pv-recycler-pod-template-filepath-hostpath string
The file path to a pod definition used as a template for HostPath persistent volume recycling. This is for development and testing only and will not work in a multi-node cluster.
--pv-recycler-pod-template-filepath-nfs string
The file path to a pod definition used as a template for NFS persistent volume recycling
--pv-recycler-timeout-increment-hostpath int32     Default: 30
the increment of time added per Gi to ActiveDeadlineSeconds for a HostPath scrubber pod. This is for development and testing only and will not work in a multi-node cluster.
--pvclaimbinder-sync-period duration     Default: 15s
The period for syncing persistent volumes and persistent volume claims
--requestheader-extra-headers-prefix strings     Default: "x-remote-extra-"
--requestheader-group-headers strings     Default: "x-remote-group"
--requestheader-username-headers strings     Default: "x-remote-user"
--resource-quota-sync-period duration     Default: 5m0s
The period for syncing quota usage status in

 the system
--root-ca-file string
If set, this root certificate authority will be included in service account's token secret. This must be a valid PEM-encoded CA bundle.
--route-reconciliation-period duration     Default: 10s
The period for reconciling routes created for Nodes by cloud provider.
--secondary-node-eviction-rate float     Default: 0.01
Number of nodes per second on which pods are deleted in case of node failure when a zone is unhealthy (see --unhealthy-zone-threshold for definition of healthy/unhealthy). Zone refers to entire cluster in non-multizone clusters. This value is implicitly overridden to 0 if the cluster size is smaller than --large-cluster-size-threshold.
--secure-port int     Default: 10257
The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all.
--service-account-private-key-file string
Filename containing a PEM-encoded private RSA or ECDSA key used to sign service account tokens.
CIDR Range for Services in cluster. Requires --allocate-node-cidrs to be true
--terminated-pod-gc-threshold int32     Default: 12500
Number of terminated pods that can exist before the terminated pod garbage collector starts deleting terminated pods. If &lt;= 0, the terminated pod garbage collector is disabled.
A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over ex

tracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com".
--unhealthy-zone-threshold float     Default: 0.55
Fraction of Nodes in a zone which needs to be not Ready (minimum 3) for zone to be treated as unhealthy.
If true, use individual service account credentials for each controller.
-v, --v int
--volume-host-allow-local-loopback     Default: true
If false, deny local loopback IPs in addition to any CIDR ranges in --volume-host-cidr-denylist
--volume-host-cidr-denylist strings
A comma-separated list of CIDR ranges to avoid from volume plugins.
12.6 - kube-proxy
Synopsis
The Kubernetes network proxy runs on each node. This
reflects services as defined in the Kubernetes API on each node and can do simple
TCP, UDP, and SCTP stream forwarding or round robin TCP, UDP, and SCTP forwarding across a set of backends.
Service cluster IPs and ports are currently found through Docker-links-compatible
environment variables specifying ports opened by the service proxy. There is an optional
addon that provides cluster DNS for these cluster IPs. The user must create a service
with the apiserver API to configure the proxy.
kube-proxy [flags]
Options
--add_dir_header
log to standard error as well as files (no effect when -logtostderr=true)
The IP address for the proxy server to serve on (set to '0.0.0.0' for all IPv4 interfaces and '::' for all IPv6 interfaces). This parameter is ignored if a config file is specified by --config.
--bind-address-hard-fail
If true kube-proxy will treat failure to bind to a port as fatal and exit
--boot_id_file string     Default: "/proc/sys/kernel/random/boot_id"
Comma-separated list of files to check for boot-id. Use the first one that exists.
--cleanup
If true cleanup iptables and ipvs rules and exit.
The CIDR range of pods in the cluster. When configured, traffic sent to a Service cluster IP from outside this range will be masqueraded and traffic sent from pods to an external LoadBalancer IP will be directed to the respective cluster IP instead. For dual-stack clusters, a comma-separated list is accepted with at least one CIDR per IP family (IPv4 and IPv6). This parameter is ignored if a config file is specified by --config.
The path to the configuration file.
--config-sync-period duration     Default: 15m0s
How often configuration from the apiserver is refreshed. Must be greater than 0.
--conntrack-max-per-core int32     Default: 32768
Maximum number of NAT connections to track per CPU core (0 to leave the limit as-is and ignore conntrack-min).
--conntrack-min int32     Default: 131072
Minimum number of conntrack entries to allocate, regardless of conntrack-max-per-core (set conntrack-max-per-core=0 to leave the limit as-is).
--conntrack-tcp-timeout-close-wait duration     Default: 1h0m0s
NAT timeout for TCP connections in the CLOSE_WAIT state
--conntrack-tcp-timeout-established duration     Default: 24h0m0s
Idle timeout for established TCP connections (0 to leave as-is)
--detect-local-mode LocalMode
Mode to use to detect local traffic. This parameter is ignored if a config file is specified by --config.
APIServe

rIdentity=true|false (BETA - default=true)
PodDisruptionConditions=true|false (BETA - d

efault=true)
This parameter is ignored if a config file is specified by --config.
--healthz-bind-address ipport     Default: 0.0.0.0:10256
The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '[::]:10256' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.
-h, --help
If non-empty, will use this string as identification instead of the actual hostname.
--iptables-localhost-nodeports     Default: true
If false Kube-proxy will disable the legacy behavior of allowing NodePort services to be accessed via localhost, This only applies to iptables mode and ipv4.
If using the pure iptables proxy, the bit of the fwmark space to mark packets requiring SNAT with. Must be within the range [0, 31].
--iptables-min-sync-period duration     Default: 1s
The minimum interval of how often the iptables rules can be refreshed as endpoints and services change (e.g. '5s', '1m', '2h22m').
--iptables-sync-period duration     Default: 30s
The maximum interval of how often iptables rules are refreshed (e.g. '5s', '1m', '2h22m'). Must be greater than 0.
--ipvs-exclude-cidrs strings
A comma-separated list of CIDR's which the ipvs proxier should not touch when cleaning up IPVS rules.
--ipvs-min-sync-period duration
The minimum interval of how often the ipvs rules can be refreshed as endpoints and services change (e.g. '5s', '1m', '2h22m').
--ipvs-scheduler string
The ipvs scheduler type when proxy mode is ipvs
--ipvs-strict-arp
Enable strict ARP by setting arp_ignore to 1 and arp_announce to 2
--ipvs-sync-period duration     Default: 30s
The maximum interval of how often ipvs rules are refreshed (e.g. '5s', '1m', '2h22m'). Must be greater than 0.
--ipvs-tcp-timeout duration
The timeout for idle IPVS TCP connections, 0 to leave as-is. (e.g. '5s', '1m', '2h22m').
--ipvs-tcpfin-timeout duration
The timeout for IPVS TCP connections after receiving a FIN packet, 0 to leave as-is. (e.g. '5s', '1m', '2h22m').
--ipvs-udp-timeout duration
The timeout for IPVS UDP packets, 0 to leave as-is. (e.g. '5s', '1m', '2h22m').
Burst to use wh

ile talking with kubernetes apiserver
--kube-api-content-type string     Default: "application/vnd.kubernetes.protobuf"
--kube-api-qps float     Default: 5
QPS to use while talking with kubernetes apiserver
Path to kubeconfig file with authorization information (the master location can be overridden by the master flag).
--log_backtrace_at &lt;a string in the form 'file:N'&gt;     Default: :0
--log_dir string
If non-empty, write log files in this directory (no effect when -logtostderr=true)
--log_file string
If non-empty, use this log file (no effect when -logtostderr=true)
--log_file_max_size uint     Default: 1800
Defines the maximum size a log file can grow to (no effect when -logtostderr=true). Unit is megabytes. If the value is 0, the maximum file size is unlimited.
--machine_id_file string     Default: "/etc/machine-id,/var/lib/dbus/machine-id"
Comma-separated list of files to check for machine-id. Use the first one that exists.
--masquerade-all
If using the pure iptables proxy, SNAT all traffic sent via Service cluster IPs (this not commonly needed)
The address of the Kubernetes API server (overrides any value in kubeconfig)
--metrics-bind-address ipport     Default: 127.0.0.1:10249
The IP address with port for the metrics server to serve on (set to '0.0.0.0:10249' for all IPv4 interfaces and '[::]:10249' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.
--nodeport-addresses strings
A string slice of values which specify the addresses to use for NodePorts. Values may be valid IP blocks (e.g. 1.2.3.0/24, 1.2.3.4/32). The default empty string slice ([]) means to use all local addresses. This parameter is ignored if a config file is specified by --config.
--one_output
If true, only write logs to their native severity level (vs also writing to each lower severity level; no effect when -logtostderr=true)
The oom-score-adj value for kube-proxy process. Values must be within the range [-1000, 1000]. This parameter is ignored if a config file is specified by --config.
--pod-bridge-interface string
A bridge interface name in the cluster. Kube-proxy considers traffic as local if originating from an interface which matches the value. This argument should be set if DetectLocalMode is set to BridgeInterface.
--pod-interface-name-prefix string
An interface prefix in the cluster. Kube-proxy considers traffic as local if originating from interfaces that match the given prefix. This argument should be set if DetectLocalMode is set to InterfaceNamePrefix.
--profiling
If true enables profiling via web interface on /debug/pprof handler. This parameter is ignored if a config file is specified by --config.
--proxy-mode ProxyMode
Which proxy mode to use: on Linux this can be 'iptables' (default) or 'ipvs'. On Windows the only supported value is 'kernelspace'.This parameter is ignored if a config file is specified by --config.
--proxy-port-range port-range
Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports will be randomly chosen.
The previous version for which you want to show hidden metrics. Only the previous minor version is meaningful, other values will not be allowed. The format is &lt;major&gt;.&lt;minor&gt;, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that. This parameter is ignored if a config file is specified by --config.
--skip_lo

g_headers
If true, avoid headers when opening log files (no effect when -logtostderr=true)
logs at or above this threshold go to stderr when writing to files and stderr (no effect when -logtostderr=true or -alsologtostderr=false)
-v, --v int
--vmodule &lt;comma-separated 'pattern=N' settings&gt;
--write-config-to string
If set, write the default configuration values to this file and exit.
12.7 - kube-scheduler
Synopsis
The Kubernetes scheduler is a control plane process which assigns
Pods to Nodes. The scheduler determines which Nodes are valid placements for
each Pod in the scheduling queue according to constraints and available
resources. The scheduler then ranks each valid Node and binds the Pod to a
suitable Node. Multiple different schedulers may be used within a cluster;
kube-scheduler is the reference implementation.
See
for more information about scheduling and the kube-scheduler component.
kube-scheduler [flags]
Options
--authentication-tolerate-lookup-failure     Default: true
--contention-profiling     Default: true
DEPRECATED: enable lock contention profiling, if profiling is enabled. This para

meter is ignored if a config file is specified in --config.
MaxUnavailableStatefulSet=tr

ue|false (ALPHA - default=false)
-h, --help
help for kube-scheduler
--kube-api-burst int32     Default: 100
DEPRECATED: burst to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.
DEPRECATED: content type of requests sent to apiserver. This parameter is ignored if a config file is specified in --config.
--kube-api-qps float     Default: 50
DEPRECATED: QPS to use while talking with kubernetes apiserver. This parameter is ignored if a config file is specified in --config.
DEPRECATED: path to kubeconfig file with authorization and master location information. This parameter is ignored if a config file is specified in --config.
The duration that non-leader candidates will wait after observing a leadership renewal until attempting to acquire leadership of a led but unrenewed leader slot. This is effectively the maximum duration that a leader can be stopped before it is

 replaced by another candidate. This is only applicable if leader election is enabled.
--leader-elect-resource-name string     Default: "kube-scheduler"
--lock-object-name string     Default: "kube-scheduler"
DEPRECATED: define the name of the lock object. Will be removed in favor of leader-elect-resource-name. This parameter is ignored if a config file is specified in --config.
--lock-object-namespace string     Default: "kube-system"
DEPRECATED: define the namespace of the lock object. Will be removed in favor of leader-elect-resource-namespace. This parameter is ignored if a config file is specified in --config.
--pod-max-in-unschedulable-pods-duration duration     Default: 5m0s
DEPRECATED: the maximum time a pod can stay in unschedulablePods. If a pod stays in unschedulablePods for longer than this value, the pod will be moved from unschedulablePods to backoffQ or activeQ. This flag is deprecated and will be removed in 1.26
DEPRECATED: enable profiling via web interface host:port/debug/pprof/. This parameter is ignored if a config file is specified in --config.
--secure-port int     Default: 10259
The previous version for which you want to show hidden metrics. Only the previous minor version i

s meaningful, other values will not be allowed. The format is &lt;major&gt;.&lt;minor&gt;, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that.
-v, --v int
If set, write the configuration values to this file and exit.
13 - Configuration APIs
13.1 - Client Authentication (v1)
ExecCredential is used by exec-based plugins to communicate credentials to
HTTP transports.
Field
string
kind
string
spec
ExecCredentialSpec
Spec holds information passed to the plugin by the transport.
status
ExecCredentialStatus
Status is filled in by the plugin and holds the credentials that the transport
should use to contact the API.
Cluster
Cluster contains information to allow an exec plugin to communicate
with the kubernetes cluster being authenticated to.
To ensure that this struct contains everything someone would need to communicate
with a kubernetes cluster (just like they would via a kubeconfig), the fields
should shadow "k8s.io/client-go/tools/clientcmd/api/v1".Cluster, with the exception
of CertificateAuthority, since CA dat

a will always be passed to the plugin as bytes.
Field
server
string
string
TLSServerName is passed to the server for SNI and is used in the client to
check server certificates against. If ServerName is empty, the hostname
used to contact the server is used.
bool
InsecureSkipTLSVerify skips the validity check for the server's certificate.
This will make your HTTPS connections insecure.
[]byte
CAData contains PEM-encoded certificate authority certificates.
If empty, system roots should be used.
proxy-url
string
ProxyURL is the URL to the proxy to be used for all requests to this
cluster.
bool
config
Config holds additional config data that is specific to the exec
plugin with regards to the cluster being authenticated to.
This data is sourced from the clientcmd Cluster object's
extensions[client.authentication.k8s.io/exec] field:
clusters:
name: my-cluster
cluster:
...
extensions:
name: client.authentication.k8s.io/exec # reserved extension name for per cluster exec config
extension:
audience: 06e3fbd18de8 # arbitrary config
In some environments, the user config may be exactly the same across many clusters
(i.e. call this exec plugin) minus some details that are specific to each cluster
such as the audience. This field allows the per cluster config to be directly
specified with the cluster info. Using this field to store secret data is not
recommended as one of the prime benefits of exec plugins is that no secrets need
to be stored directly in the kubeconfig.
ExecCredentialSpec holds request and runtime specific information provided by
the transport.
Field
cluster
Cluster
Cluster contains information to allow an exec plugin to communicate with the
kubernetes cluster being authenticated to. Note that Cluster is non-nil only
when provideClusterInfo is set to true in the exec provider config (i.e.,
ExecConfig.ProvideClusterInfo).
bool
Interactive declares whether stdin has been passed to this exec plugin.
ExecCredentialStatus holds credentials for the transport to use.
Token and ClientKeyData are sensitive fields. This data should only be
transmitted in-memory between client and exec plugin process. Exec plugin
itself should at least be protected via file permissions.
Field
ExpirationTimestamp indicates a time when the provided credentials expire.
token
string
Token is a bearer token used by the client for request authentication.
string
PEM-encoded client TLS certificates (including intermediates, if any).
string
PEM-encoded private key for the above certificate.
13.2 - Client Authentication (v1beta1)
Field
string
kind
string
spec
status
Cluster
To ensure that this

 struct contains everything someone would need to communicate
of CertificateAuthority, since CA data will always be passed to the plugin as bytes.
Field
server
string
string
bool
[]byte
proxy-url
string
cluster.
bool
config
clusters:
cluster:
...
Field
cluster
Cluster
bool
Field
token
string
string
string
13.3 - Event Rate Limit Configuration (v1alpha1)
Configuration provides configuration for the EventRateLimit admission
Field
string
kind
string
limits
[]Limit
limits are the limits to place on event queries received.
Limits can be placed on events received server-wide, per namespace,

per user, and per source+object.
At least one limit is required.
Limit
Limit is the configuration for a particular limit type
Field
type
LimitType
type is the type of limit to which this configuration applies
qps
int32
qps is the number of event queries per second that are allowed for this
type of limit. The qps and burst fields are used together to determine if
a particular event query is accepted. The qps determines how many queries
are accepted once the burst amount of queries has been exhausted.
burst
int32
burst is the burst number of event queries that are allowed for this type
of limit. The qps and burst fields are used together to determine if a
particular event query is accepted. The burst determines the maximum size
of the allowance granted for a particular bucket. For example, if the burst
is 10 and the qps is 3, then the admission control will accept 10 queries
before blocking any queries. Every second, 3 more queries will be allowed.
If some of that allowance is not used, then it will roll over to the next
second, until the maximum allowance of 10 is reached.
cacheSize
int32
cacheSize is the size of the LRU cache for this type of limit. If a bucket
is evicted from the cache, then the allowance for that bucket is reset. If
more queries are later received for an evicted bucket, then that bucket
will re-enter the cache with a clean slate, giving that bucket a full
allowance of burst queries.
The default cache size is 4096.
If limitType is 'server', then cacheSize is ignored.
LimitType
(Alias of
string
)
Limit
LimitType is the type of the limit (e.g., per-namespace)
13.4 - Image Policy API (v1alpha1)
ImageReview checks if the set of images in a pod are allowed.
Field
string
kind
string
metadata
meta/v1.ObjectMeta
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
Refer to the Kubernetes API documentation for the fields of the
metadata
field.
spec
ImageReviewSpec
Spec holds information about the pod being evaluated
status
ImageReviewStatus
Status is filled in by the backend and indicates whether the pod should be allowed.
ImageReviewContainerSpec
ImageReviewContainerSpec is a description of a container within the pod creation request.
Field
image
string
This can be in the form image:tag or image@SHA:012345679abcdef.
ImageReviewSpec is a description of the pod creation request.
Field
[]ImageReviewContainerSpec
Containers is a list of a subset of the information in each container of the Pod being created.
Annotations is a list of key-value pairs extracted from the Pod's annotations.
It only includes keys which match the pattern
.
It is up to each webhook backend to determine how to interpret these annotations, if at all.
namespace
string
Namespace is the namespace the pod is being created in.
ImageReviewStatus is the result of the review for the pod creation request.
Field
allowed
bool
Allowed indicates that all images were allowed to be run.
reason
string
Reason should be empty unless Allowed is false in which case it
may contain a short description of what is wrong. Kubernetes
may truncate excessively long errors when displaying to the user.
auditAnnotations
AuditAnnotations will be added to the attributes object of the
admission controller request using 'AddAnnotation'. The keys should
be prefix-less (i.e., the admission controller will add an
appropriate prefix).
13.5 - kube-apiserver Admission (v1)
AdmissionReview describes an admission review request/response.

Field
string
kind
string
request
AdmissionRequest
Request describes the attributes for the admission request.
response
AdmissionResponse
Response describes the attributes for the admission response.
AdmissionRequest describes the admission.Attributes for the admission request.
Field
uid
k8s.io/apimachinery/pkg/types.UID
UID is an identifier for the individual request/response. It allows us to distinguish instances of requests which are
otherwise identical (parallel requests, requests when earlier requests did not modify etc)
The UID is meant to track the round trip (request/response) between the KAS and the WebHook, not the user request.
It is suitable for correlating log entries between the webhook and apiserver, for either auditing or debugging.
kind
meta/v1.GroupVersionKind
Kind is the fully-qualified type of object being submitted (for example, v1.Pod or autoscaling.v1.Scale)
resource
meta/v1.GroupVersionResource
Resource is the fully-qualified resource being requested (for example, v1.pods)
string
SubResource is the subresource being requested, if any (for example, "status" or "scale")
RequestKind is the fully-qualified type of the original API request (for example, v1.Pod or autoscaling.v1.Scale).
If this is specified and differs from the value in "kind", an equivalent match and conversion was performed.
For example, if deployments can be modified via apps/v1 and apps/v1beta1, and a webhook registered a rule of
and
,
an API request to apps/v1beta1 deployments would be converted and sent to the webhook
with
kind: {group:"apps", version:"v1", kind:"Deployment"}
(matching the rule the webhook registered for),
and
requestKind: {group:"apps", version:"v1beta1", kind:"Deployment"}
(indicating the kind of the original API request).
See documentation for the "matchPolicy" field in the webhook configuration type for more details.
RequestResource is the fully-qualified resource of the original API request (for example, v1.pods).
If this is specified and differs from the value in "resource", an equivalent match and conversion was performed.
and
,
with
resource: {group:"apps", version:"v1", resource:"deployments"}
(matching the resource the webhook registered for),
and
requestResource: {group:"apps", version:"v1beta1", resource:"deployments"}
(indicating the resource of the original API request).
See documentation for the "matchPolicy" field in the webhook configuration type.
string
RequestSubResource is the name of the subresource of the original API request, if any (for example, "status" or "scale")
If this is specified and differs from the value in "subResource", an equivalent match and conversion was performed.
name
string
Name is the name of the object as presented in the request. On a CREATE operation, the client may omit name and
rely on the server to generate the name. If that is the case, this field will contain an empty string.
namespace
string
Namespace is the namespace associated with the request (if any).
operation
Operation
Operation is the operation being performed. This may be different than the operation
requested. e.g. a patch can result in either a CREATE or UPDATE Operation.
userInfo
authentication/v1.UserInfo
UserInfo is information about the requesting user
object
k8s.io/apimachinery

/pkg/runtime.RawExtension
Object is the object from the incoming request.
oldObject
OldObject is the existing object. Only populated for DELETE and UPDATE requests.
dryRun
bool
DryRun indicates that modifications will definitely not be persisted for this request.
Defaults to false.
options
Options is the operation option structure of the operation being performed.
e.g.
meta.k8s.io/v1.DeleteOptions
or
meta.k8s.io/v1.CreateOptions
different than the options the caller provided. e.g. for a patch request the performed
Operation might be a CREATE, in which case the Options will a
even though the caller provided
meta.k8s.io/v1.PatchOptions
.
AdmissionResponse describes an admission response.
Field
uid
UID is an identifier for the individual request/response.
This must be copied over from the corresponding AdmissionRequest.
allowed
bool
Allowed indicates whether or not the admission request was permitted.
status
meta/v1.Status
Result contains extra details into why an admission request was denied.
This field IS NOT consulted in any way if "Allowed" is "true".
patch
[]byte
The patch body. Currently we only support "JSONPatch" which implements RFC 6902.
patchType
PatchType
The type of Patch. Currently we only allow "JSONPatch".
AuditAnnotations is an unstructured key value map set by remote admission controller (e.g. error=image-blacklisted).
MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controller will prefix the keys with
admission webhook name (e.g. imagepolicy.example.com/error=image-blacklisted). AuditAnnotations will be provided by
the admission webhook to add additional context to the audit log for this request.
warnings
[]string
warnings is a list of warning messages to return to the requesting API client.
Warning messages describe a problem the client making the API request should correct or be aware of.
Limit warnings to 120 characters if possible.
Warnings over 256 characters and large numbers of warnings may be truncated.
Operation
(Alias of
string
)
Operation is the type of resource operation being checked for admission control
PatchType
(Alias of
string
)
PatchType is the type of patch being used to represent the mutated object
13.6 - kube-apiserver Audit Configuration (v1)
Event
EventList
Policy
PolicyList
Event
EventList
Event captures all the information that can be included in an API audit log.
Field
string
kind
string
Event
level
Level
AuditLevel at which event was generated
auditID
Unique audit ID, generated for each request.
stage
Stage
Stage of the request handling when this event instance was generated.
requestURI
string
RequestURI is the request URI as sent by the client to a server.
verb
string
Verb is the kubernetes verb associated with the request.
For non-resource requests, this is the lower-cased HTTP method.
user
Authenticated user information.
impersonatedUser
Impersonated user information.
sourceIPs
[]string
Source IPs, from where the request originated and intermediate proxies.
The source IPs are listed from (in order):
X-Forwarded-For request header IPs
X-Real-Ip header, if not present in the X-Forwarded-For list
The remote address for the connection, if it doesn't match the last
IP in the list up to here (X-Forwarded-For or X-Real-Ip).
Note: All but the last IP can be arbitrarily set by the client.
userAgent
string
UserAgent records the user agent string reported by the client.
Note that the UserAgent is provided by the client, and

 must not be trusted.
objectRef
Object reference this request is targeted at.
Does not apply for List-type requests, or non-resource requests.
responseStatus
The response status, populated even when the ResponseObject is not a Status type.
For successful responses, this will only include the Code and StatusSuccess.
For non-status type error responses, this will be auto-populated with the error Message.
requestObject
API object from the request, in JSON format. The RequestObject is recorded as-is in the request
(possibly re-encoded as JSON), prior to version conversion, defaulting, admission or
merging. It is an external versioned object type, and may not be a valid object on its own.
Omitted for non-resource requests. Only logged at Request Level and higher.
responseObject
API object returned in the response, in JSON. The ResponseObject is recorded after conversion
to the external type, and serialized as JSON. Omitted for non-resource requests. Only logged
at Response Level.
requestReceivedTimestamp
meta/v1.MicroTime
Time the request reached the apiserver.
stageTimestamp
Time the request reached current audit stage.
Annotations is an unstructured key value map stored with an audit event that may be set by
plugins invoked in the request serving chain, including authentication, authorization and
admission plugins. Note that these annotations are for the audit event, and do not correspond
to the metadata.annotations of the submitted object. Keys should uniquely identify the informing
component to avoid name collisions (e.g. podsecuritypolicy.admission.k8s.io/policy). Values
should be short. Annotations are included in the Metadata level.
EventList
EventList is a list of audit Events.
Field
string
kind
string
EventList
metadata
meta/v1.ListMeta
items
[]Event
Policy
Policy defines the configuration of audit logging, and the rules for how different request
categories are logged.
Field
string
kind
string
Policy
metadata
ObjectMeta is included for interoperability with API infrastructure.
metadata
field.
rules
[]PolicyRule
Rules specify the audit Level a request should be recorded at.
A request may match multiple rules, in which case the FIRST matching rule is used.
The default audit level is None, but can be overridden by a catch-all rule at the end of the list.
PolicyRules are strictly ordered.
[]Stage
OmitStages is a list of stages for which no events are created. Note that this can also
be specified per rule in which case the union of both are omitted.
omitManagedFields
bool
OmitManagedFields indicates whether to omit the managed fields of the request
and response bodies from being written to the API audit log.
This is used as a global default - a value of 'true' will omit the managed fileds,
otherwise the managed fields will be included in the API audit log.
Note that this can also be specified per rule in which case the value specified
in a rule will override the global default.
PolicyList is a list of audit Policies.
Field
string
kind
string
metadata
items
[]Policy
GroupResources
PolicyRule
GroupResources represents resource kinds in an API group.
Field
group
string
Group is the name of the API group that contains the resources.
The empty string represents the core API group.
resources
[]string
'pods' matches pods.
'pods/log' matches the log subresource of pods.
'
' matches all resou

rces and their subresources.
'pods/
' matches all subresources of pods.
'*/scale' matches all scale subresources.
If wildcard is present, the validation rule will ensure resources do not
overlap with each other.
An empty list implies all resources and subresources in this API groups apply.
[]string
ResourceNames is a list of resource instance names that the policy matches.
Using this field requires Resources to be specified.
An empty list implies that every instance of the resource is matched.
Level
(Alias of
string
)
Event
Level defines the amount of information logged during auditing
Event
Field
resource
string
namespace
string
name
string
uid
apiGroup
string
APIGroup is the name of the API group that contains the referred object.
string
APIVersion is the version of the API group that contains the referred object.
string
string
Policy
PolicyRule maps requests based off metadata to an audit Level.
Requests must match the rules of every field (an intersection of rules).
Field
level
Level
The Level that requests matching this rule are recorded at.
users
[]string
The users (by authenticated user name) this rule applies to.
An empty list implies every user.
[]string
The user groups this rule applies to. A user is considered matching
if it is a member of any of the UserGroups.
An empty list implies every user group.
verbs
[]string
The verbs that match this rule.
An empty list implies every verb.
resources
[]GroupResources
Resources that this rule matches. An empty list implies all kinds in all API groups.
[]string
Namespaces that this rule matches.
The empty string "" matches non-namespaced resources.
An empty list implies every namespace.
[]string
NonResourceURLs is a set of URL paths that should be audited.
s are allowed, but only as the full, final step in the path.
Examples:
"/metrics" - Log requests for apiserver metrics
"/healthz
" - Log all health checks
[]Stage
be specified policy wide in which case the union of both are omitted.
An empty list means no restrictions will apply.
bool
a value of 'true' will drop the managed fields from the API audit log
a value of 'false' indicates that the managed fileds should be included
in the API audit log
Note that the value, if specified, in this rule will override the global default
If a value is not specified then the global default specified in
Policy.OmitManagedFields will stand.
Stage
(Alias of
string
)
Event
Policy
Stage defines the stages in request handling that audit events may be generated.
13.7 - kube-apiserver Configuration (v1)
Field
string
kind
string
plugins
Field
name
string
It must match the registered admission plugin

 name.
path
string
13.8 - kube-apiserver Configuration (v1alpha1)
Field
string
kind
string
plugins
Field
string
kind
string
Field
string
kind
string
endpoint
string
int32
Field
name
string
path
string
Field
transport
Transport
This is required if ProxyProtocol is HTTPConnect or GRPC.
Field
name
string
(Alias of
string
)
Transport
Field
url
string
tlsConfig
TLSConfig
TLSConfig
Field
caBundle
string
caBundl

e is the file location of the CA to be used to determine trust with the konnectivity server.
clientKey
string
string
Transport
Field
tcp
uds
Transport
Field
udsName
string
13.9 - kube-apiserver Encryption Configuration (v1)
Field
string
kind
string
resources
Field
keys
[]Key
Field
string
name
string
cachesize
int32
endpoint
string
timeout
Key
Field
name
string
secret
string
Field
aesgcm
aescbc
secretbox
secretbox is the configuration for the Secretbox based tran

sformer.
identity
kms
Field
resources
[]string
providers
Field
keys
[]Key
13.10 - kube-proxy Configuration (v1alpha1)
Field
string
kind
string
string
string
string
bool
bool
string
string
iptables
ipvs
int32
mode
ProxyMode
portRange
string
conntrack
configSyncPeriod is how often configuration from the apiserver is refreshed. Must be greater
than 0.
[]string
If set it to "127.0.0.0/8", kube-

proxy will only select the loopback interface for NodePort.
winkernel
string
LocalMode
Field
string
string
Field
int32
min
int32
Field
int32
bool
bool
'2h22m').
Field
'2h22m').
scheduler
string
[]string
strictARP
bool
strict ARP c

onfigure arp_ignore and arp_announce to avoid answering ARP queries
Field
string
sourceVip
string
enableDSR
bool
with DSR
string
bool
Windows
LocalMode
(Alias of
string
)
ProxyMode
(Alias of
string
)
is
iptables
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
enableContentionProfiling enables lock contention profiling, if
Field
bool
before executing the main loop. Enab

le this when running replicated
enabled.
string
string
string
13.11 - kube-scheduler Configuration (v1)
DefaultPreemptionArgs
InterPodAffinityArgs
NodeAffinityArgs
NodeResourcesBalancedAllocationArgs
NodeResourcesFitArgs
PodTopologySpreadArgs
VolumeBindingArgs
DefaultPreemptionArgs holds arguments used to configure the
DefaultPreemption plugin.
Field
string
kubescheduler.config.k8s.io/v1
kind
string
minCandidateNodesPercentage
int32
MinCandidateNodesPercentage is the minimum number of candidates to
shortlist when dry running preemption as a percentage of number of nodes.
Must be in the range [0, 100]. Defaults to 10% of the cluster size if
unspecified.
minCandidateNodesAbsolute
int32
MinCandidateNodesAbsolute is the absolute minimum number of candidates to
shortlist. The likely number of candidates enumerated for dry running
preemption is given by the formula:
numCandidates = max(numNodes * minCandidateNodesPercentage, minCandidateNodesAbsolute)
We say "likely" because there are other factors such as PDB violations
that play a role in the number of candidates shortlisted. Must be at least
0 nodes. Defaults to 100 nodes if unspecified.
InterPodAffinityArgs holds arguments used to configure the InterPodAffinity plugin.
Field
string
kind
string
hardPodAffinityWeight
int32
HardPodAffinityWeight is the scoring weight for existing pods with a
matching hard affinity to the incoming pod.
KubeSchedulerConfiguration configures a scheduler
Field
string
kind
string
int32
Parallelism defines the amount of parallelism in algorithms for scheduling a Pods. Must be greater than 0. Defaults to 16
leaderElection
LeaderElection defines the configuration of leader election client.
ClientConnection specifies the kubeconfig file and client connection
settings for the proxy server to use when communicating with the apiserver.
DebuggingConfiguration holds configuration for Debugging related features
TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfiguration
percentageOfNodesToScore
int32
PercentageOfNodesToScore is the percentage of all nodes 

that once found feasible
for running a pod, the scheduler stops its search for more feasible nodes in
the cluster. This helps improve scheduler's performance. Scheduler always tries to find
at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is.
Example: if the cluster size is 500 nodes and the value of this flag is 30,
then scheduler stops finding further feasible nodes once it finds 150 feasible ones.
When the value is 0, default percentage (5%--50% based on the size of the cluster) of the
nodes will be scored. It is overridden by profile level PercentageofNodesToScore.
podInitialBackoffSeconds
int64
PodInitialBackoffSeconds is the initial backoff for unschedulable pods.
If specified, it must be greater than 0. If this value is null, the default value (1s)
will be used.
podMaxBackoffSeconds
int64
PodMaxBackoffSeconds is the max backoff for unschedulable pods.
If specified, it must be greater than podInitialBackoffSeconds. If this value is null,
the default value (10s) will be used.
profiles
[]KubeSchedulerProfile
Profiles are scheduling profiles that kube-scheduler supports. Pods can
choose to be scheduled under a particular profile by setting its associated
scheduler name. Pods that don't specify any scheduler name are scheduled
with the "default-scheduler" profile, if present here.
extenders
[]Extender
Extenders are the list of scheduler extenders, each holding the values of how to communicate
with the extender. These extenders are shared by all scheduler profiles.
NodeAffinityArgs holds arguments to configure the NodeAffinity plugin.
Field
string
kind
string
addedAffinity
core/v1.NodeAffinity
AddedAffinity is applied to all Pods additionally to the NodeAffinity
specified in the PodSpec. That is, Nodes need to satisfy AddedAffinity
AND .spec.NodeAffinity. AddedAffinity is empty by default (all Nodes
match).
When AddedAffinity is used, some Pods with affinity requirements that match
a specific Node (such as Daemonset Pods) might remain unschedulable.
NodeResourcesBalancedAllocationArgs holds arguments used to configure NodeResourcesBalancedAllocation plugin.
Field
string
kind
string
resources
[]ResourceSpec
Resources to be managed, the default is "cpu" and "memory" if not specified.
NodeResourcesFitArgs holds arguments used to configure the NodeResourcesFit plugin.
Field
string
kind
string
ignoredResources
[]string
IgnoredResources is the list of resources that NodeResources fit filter
should ignore. This doesn't apply to scoring.
ignoredResourceGroups
[]string
IgnoredResourceGroups defines the list of resource groups that NodeResources fit filter should ignore.
e.g. if group is ["example.com"], it will ignore all resource names that begin
with "example.com", such as "example.com/aaa" and "example.com/bbb".
A resource group name can't contain '/'. This doesn't apply to scoring.
scoringStrategy
ScoringStrategy
ScoringStrategy selects the node resource scoring strategy.
The default strategy is LeastAllocated with an equal "cpu" and "memory" weight.
PodTopologySpreadArgs holds arguments used to configure the PodTopologySpread plugin.
Field
string
kind
string
defaultConstraints
[]core/v1.TopologySpreadConstraint
DefaultConstraints defines topology spread constraints to be applied to
Pods that don't define any in
pod.spec.topologySpreadConstraints
.
.defaultConstraints[*].labelSelectors
must be empty, as they are
deduced from the Pod's membership to Services, ReplicationControllers,
ReplicaSets or StatefulSe

ts.
When not empty, .defaultingType must be "List".
defaultingType
PodTopologySpreadConstraintsDefaulting
DefaultingType determines how .defaultConstraints are deduced. Can be one
of "System" or "List".
"System": Use kubernetes defined constraints that spread Pods among
Nodes and Zones.
"List": Use constraints defined in .defaultConstraints.
Defaults to "System".
VolumeBindingArgs holds arguments used to configure the VolumeBinding plugin.
Field
string
kind
string
bindTimeoutSeconds
int64
BindTimeoutSeconds is the timeout in seconds in volume binding operation.
Value must be non-negative integer. The value zero indicates no waiting.
If this value is nil, the default value (600) will be used.
shape
[]UtilizationShapePoint
Shape specifies the points defining the score function shape, which is
used to score nodes based on the utilization of statically provisioned
PVs. The utilization is calculated by dividing the total requested
storage of the pod by the total capacity of feasible PVs on each node.
Each point contains utilization (ranges from 0 to 100) and its
associated score (ranges from 0 to 10). You can turn the priority by
specifying different scores for different utilization numbers.
The default shape points are:
0 for 0 utilization
10 for 100 utilization
All points must be sorted in increasing order by utilization.
Extender
Extender holds the parameters used to communicate with the extender. If a verb is unspecified/empty,
it is assumed that the extender chose not to provide that extension.
Field
urlPrefix
string
URLPrefix at which the extender is available
filterVerb
string
Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter call to extender.
preemptVerb
string
Verb for the preempt call, empty if not supported. This verb is appended to the URLPrefix when issuing the preempt call to extender.
prioritizeVerb
string
Verb for the prioritize call, empty if not supported. This verb is appended to the URLPrefix when issuing the prioritize call to extender.
weight
int64
The numeric multiplier for the node scores that the prioritize call generates.
The weight should be a positive integer
bindVerb
string
Verb for the bind call, empty if not supported. This verb is appended to the URLPrefix when issuing the bind call to extender.
If this method is implemented by the extender, it is the extender's responsibility to bind the pod to apiserver. Only one extender
can implement this function.
enableHTTPS
bool
EnableHTTPS specifies whether https should be used to communicate with the extender
tlsConfig
ExtenderTLSConfig
TLSConfig specifies the transport layer security config
httpTimeout
HTTPTimeout specifies the timeout duration for a call to the extender. Filter timeout fails the scheduling of the pod. Prioritize
timeout is ignored, k8s/other extenders priorities are used to select the node.
nodeCacheCapable
bool
NodeCacheCapable specifies that the extender is capable of caching node information,
so the scheduler should only send minimal information about the eligible nodes
assuming that the extender already cached full details of all nodes in the cluster
managedResources
[]ExtenderManagedResource
ManagedResources is a list of extended resources that are managed by
this extender.
A pod will be sent to the extender on the Filter, Prioritize and Bind
(if the extender is the binder) phases iff the pod requests at least
one of the extended resources in this list. If empty or unspecified,
all pods will be sent to this extender.
If IgnoredByScheduler is set to true for a resource, kube-scheduler
will skip checking the resource in predicates.
ignorable
bool
Ignorable specifies if the extender is ignor

able, i.e. scheduling should not
fail when the extender returns an error or is not reachable.
ExtenderManagedResource
Extender
ExtenderManagedResource describes the arguments of extended resources
managed by an extender.
Field
name
string
Name is the extended resource name.
ignoredByScheduler
bool
IgnoredByScheduler indicates whether kube-scheduler should ignore this
resource when applying predicates.
Extender
ExtenderTLSConfig contains settings to enable TLS with extender
Field
insecure
bool
Server should be accessed without verifying the TLS certificate. For testing only.
serverName
string
ServerName is passed to the server for SNI and is used in the client to check server
certificates against. If ServerName is empty, the hostname used to contact the
server is used.
certFile
string
Server requires TLS client certificate authentication
keyFile
string
caFile
string
Trusted root certificates for server
certData
[]byte
CertData holds PEM-encoded bytes (typically read from a client certificate file).
CertData takes precedence over CertFile
keyData
[]byte
KeyData holds PEM-encoded bytes (typically read from a client certificate key file).
KeyData takes precedence over KeyFile
caData
[]byte
CAData holds PEM-encoded bytes (typically read from a root certificates bundle).
CAData takes precedence over CAFile
KubeSchedulerProfile
KubeSchedulerProfile is a scheduling profile.
Field
string
SchedulerName is the name of the scheduler associated to this profile.
If SchedulerName matches with the pod's "spec.schedulerName", then the pod
is scheduled with this profile.
int32
PercentageOfNodesToScore is the percentage of all nodes that once found feasible
nodes will be scored. It will override global PercentageOfNodesToScore. If it is empty,
global PercentageOfNodesToScore will be used.
plugins
Plugins
Plugins specify the set of plugins that should be enabled or disabled.
Enabled plugins are the ones that should be enabled in addition to the
default plugins. Disabled plugins are any of the default plugins that
should be disabled.
When no enabled or disabled plugin is specified for an extension point,
default plugins for that extension point will be used if there is any.
If a QueueSort plugin is specified, the same QueueSort Plugin and
PluginConfig must be specified for all profiles.
pluginConfig
[]PluginConfig
PluginConfig is an optional set of custom plugin arguments for each plugin.
Omitting config args for a plugin is equivalent to using the default config
for that plugin.
Plugin
PluginSet
Plugin specifies a plugin name and its weight when applicable. Weight is used only for Score plugins.
Field
name
string
Name defines the name of plugin
weight
int32
Weight defines the weight of plugin, only used for Score plugins.
PluginConfig
PluginConfig specifies arguments that should be passed to a plugin at the time of initialization.
A plugin that is invoked at multiple extension points is initialized once. Args can have arbitrary structure.
It is up to the plugin to process these Args.
Field
name
[Require

d]
string
Name defines the name of plugin being configured
args
Args defines the arguments passed to the plugins at the time of initialization. Args can have arbitrary structure.
PluginSet
Plugins
PluginSet specifies enabled and disabled plugins for an extension point.
If an array is empty, missing, or nil, default plugins at that extension point will be used.
Field
enabled
[]Plugin
Enabled specifies plugins that should be enabled in addition to default plugins.
If the default plugin is also configured in the scheduler config file, the weight of plugin will
be overridden accordingly.
These are called after default plugins and in the same order specified here.
disabled
[]Plugin
Disabled specifies default plugins that should be disabled.
When all default plugins need to be disabled, an array containing only one "*" should be provided.
Plugins
Plugins include multiple extension points. When specified, the list of plugins for
a particular extension point are the only ones enabled. If an extension point is
omitted from the config, then the default set of plugins is used for that extension point.
Enabled plugins are called in the order specified here, after default plugins. If they need to
be invoked before default plugins, default plugins must be disabled and re-enabled here in desired order.
Field
preEnqueue
PluginSet
PreEnqueue is a list of plugins that should be invoked before adding pods to the scheduling queue.
queueSort
PluginSet
QueueSort is a list of plugins that should be invoked when sorting pods in the scheduling queue.
preFilter
PluginSet
PreFilter is a list of plugins that should be invoked at "PreFilter" extension point of the scheduling framework.
filter
PluginSet
Filter is a list of plugins that should be invoked when filtering out nodes that cannot run the Pod.
postFilter
PluginSet
PostFilter is a list of plugins that are invoked after filtering phase, but only when no feasible nodes were found for the pod.
preScore
PluginSet
PreScore is a list of plugins that are invoked before scoring.
score
PluginSet
Score is a list of plugins that should be invoked when ranking nodes that have passed the filtering phase.
reserve
PluginSet
Reserve is a list of plugins invoked when reserving/unreserving resources
after a node is assigned to run the pod.
permit
PluginSet
Permit is a list of plugins that control binding of a Pod. These plugins can prevent or delay binding of a Pod.
preBind
PluginSet
PreBind is a list of plugins that should be invoked before a pod is bound.
bind
PluginSet
Bind is a list of plugins that should be invoked at "Bind" extension point of the scheduling framework.
The scheduler call these plugins in order. Scheduler skips the rest of these plugins as soon as one returns success.
postBind
PluginSet
PostBind is a list of plugins that should be invoked after a pod is successfully bound.
multiPoint
PluginSet
MultiPoint is a simplified config section to enable plugins for all valid extension points.
Plugins enabled through MultiPoint will automatically register for every individual extension
point the plugin has implemented. Disabling a plugin through MultiPoint disables that behavior.
The same is true for disabling "*" through MultiPoint (no default plugins will be automatically registered).
Plugins can still be disabled through their individual extension points.
In terms of precedence, plugin config follows this basic hierarchy
Specific extension points
Explicitly configured MultiPoint plugins
The set of default plugins, as MultiPoint plugins
This implies that a higher precedence plugin will run first and overwrite any settings within MultiPoint.
Explicitly user-configured plugins also take a higher precedence over default p

lugins.
Within this hierarchy, an Enabled setting takes precedence over Disabled. For example, if a plugin is
set in both
multiPoint.Enabled
and
multiPoint.Disabled
, the plugin will be enabled. Similarly,
including
multiPoint.Disabled = '*'
and
multiPoint.Enabled = pluginA
will still register that specific
plugin through MultiPoint. This follows the same behavior as all other extension point configurations.
(Alias of
string
)
PodTopologySpreadConstraintsDefaulting defines how to set default constraints
for the PodTopologySpread plugin.
RequestedToCapacityRatioParam
RequestedToCapacityRatioParam define RequestedToCapacityRatio parameters
Field
shape
Shape is a list of points defining the scoring function shape.
ResourceSpec
ResourceSpec represents a single resource.
Field
name
string
Name of the resource.
weight
int64
Weight of the resource.
ScoringStrategy define ScoringStrategyType for node resource plugin
Field
type
ScoringStrategyType
Type selects which strategy to run.
resources
Resources to consider when scoring.
The default resource set includes "cpu" and "memory" with an equal weight.
Allowed weights go from 1 to 100.
Weight defaults to 1 if not specified or explicitly set to 0.
requestedToCapacityRatio
Arguments specific to RequestedToCapacityRatio strategy.
(Alias of
string
)
ScoringStrategyType the type of scoring strategy used in NodeResourcesFit plugin.
UtilizationShapePoint
UtilizationShapePoint represents single point of priority function shape.
Field
utilization
int32
Utilization (x axis). Valid values are 0 to 100. Fully utilized node maps to 100.
score
int32
Score assigned to given utilization (y axis). Valid values are 0 to 10.
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
leaseDuratio

n
enabled.
string
string
string
13.12 - kube-scheduler Configuration (v1beta2)
Field
string
kubescheduler.config.k8s.io/v1beta2
kind
string
int32
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
string
Note: Both HealthzBindAddress and MetricsBindAddress fields are deprecated.
Only empty address or port 0 is allowed. Anything else will fail validation.
HealthzBindAddress is the IP address and port for the health check server to serve on.
string
MetricsBindAddress is the IP address and port for the metrics server to serve on.
(Members

 of
int32
nodes will be scored.
int64
int64
profiles
extenders
Field
string
kind
string
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
Field
string
kind
string
De

faultConstraints defines topology spread constraints to be applied to
.
ReplicaSets or StatefulSets.
Field
string
kind
string
int64
shape
Extender
Field
urlPrefix
string
string
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool
(if the extender is the binder) phases iff the pod requests 

at least
ignorable
bool
Ignorable specifies if the extender is ignorable, i.e. scheduling should not
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
Field
string
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
Field
name
string
args
PluginSet
Plugins
Field
enabled
[Req

uired]
[]Plugin
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSet
PluginSet
preScore
PluginSet
score
PluginSet
reserve
PluginSet
permit
PluginSet
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
(Alias of
string
)
Field
shape
Field
name
string
weight
int64
Field
type
resources
RequestedToCapacityRatioPara

m
(Alias of
string
)
Field
int32
score
int32
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
enabled.
string
string
string
13.13 - kube-scheduler Configuration (v1beta3)
Field
string
kubescheduler.config.k8s.io/v1beta3
kind
string
[Requir

ed]
int32
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
int32
int64
int64
profiles
extenders
Field
string
kind
string
specified in the PodSpec. T

hat is, Nodes need to satisfy AddedAffinity
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
Field
string
kind
string
.
Field
string
kind
string
int64
shape
Extender
Field
urlPrefix
string
string
Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix 

when issuing the filter call to extender.
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool
ignorable
bool
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
Field
string
is scheduled wi

th this profile.
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
Field
name
string
args
PluginSet
Plugins
Field
enabled
[]Plugin
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSet
PluginSet
preScore
PluginSet
score
PluginSet
reserve
PluginSet
permit
PluginSet
Permit is a list of plugins that control binding of a Pod. These plugins can pr

event or delay binding of a Pod.
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
Explicitly user-configured plugins also take a higher precedence over default plugins.
set in both
and
including
and
(Alias of
string
)
Field
shape
Field
name
string
weight
int64
Field
type
resources
(Alias of
string
)
Field
int32
score
int32
Field
string
string
default value of 'applicati

on/json'. This field will control all connections to the server used by a particular
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
enabled.
string
string
string
13.14 - kubeadm Configuration (v1beta2)
Overview
Basics
--config
configuration options defined in the kubeadm config file ar

e also available as command line flags, but only
---
).
:
kind
:
:
kind
:
:
kind
:
:
kind
:
:
kind
:
init
or
join
provided by kubeadm includes also enforcing consistency of values across components when required (e.g.
--config
:
kind
:
:
...
:
...
:
kind
:
:
...
etcd
:
...
apiServer
:
extraArgs
:
...
:
...
...
:
kind
:
...
:
kind
:
...
The KubeletConfiguration type should be used to change the configu

rations that will be passed to all kubelet instances
run.
:
kind
:
:
-
token
:
:
ttl
:
"24h"
-
token
:
:
usages
:
- signing
groups
:
:
name
:
criSocket
:
taints
:
-
key
:
value
:
effect
:
:
v
:
4
:
:
:
bindPort
:
6443
:
---
:
kind
:
etcd
:
local
:
:
imageTag
:
"3.2.24"
dataDir
:
extraArgs
:
:
:
-
:
-
:
:
podSubnet
:
dnsDomain
:
:
"v1.12.0"
:
apiServer
:
extraArgs
:
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
certSANs
:
-
-
:
4m0s
:
extraArgs
:
"20"
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
scheduler
:
extraArgs
:
address
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
:
:
:
false
:
---
:
kind
:
---
:
kind
:
--config
:
kind
:
...
Field
string
kind

string
etcd
Etcd
etcd
string
string
In case the
+
bindPort
the
bindPort
is used.
apiServer
APIServer
apiServer
scheduler
scheduler
dns
DNS
dns
string
string
If empty,
ci/
)
bool
hyperkube
string
Field
string
kind
string
Field
string
kind
string
is the global endpoint for the cluster, which then load-balances the requests to each individual API server.

 This
string
phase.
Field
string
kind
string
string
discovery
Discovery
discovery
Field
string
bindPort
int32
bindPort
APIServer
Field
certSANs
[]string
certSANs
to appear.
Field
token
token
string
ttl
ttl
expires
and
ttl
expires
expires
ttl
.
expires
and
ttl
usages
[]string
usages
groups
[]string
groups
Discovery
Field
token
string
token
string
[]string

bool
Field
-
string
-
string
APIServer
Field
extraArgs
extraArgs
DNS
Field
type
type
ImageMeta
ImageMeta
ImageMeta
(Alias of
string
)
DNS
Discovery
Field
and
file
file
file
and
file
string
If
,
If
file
must be set
timeout
timeout
Etcd
Field
local
LocalEtcd
local
local
and
external
external
external
local
and
external
Etcd
Field
endpoints
[]string
endpoints
caFile
string
caFile
certFile
string
certFile
keyFile
string
keyFile
Discovery
Field
Descripti

on
string
Field
name
string
name
hostPath
string
hostPath
mountPath
string
mountPath
readOnly
bool
readOnly
pathType
pathType
ImageMeta
DNS
LocalEtcd
Field
string
imageTag
string
imageTag
Field
string
LocalEtcd
Etcd
Field
ImageMeta
ImageMeta
ImageMeta
dataDir
string
dataDir
extraArgs
extraArgs
[]string
[]string
Field
string
podSubnet
string
podSubnet
dnsDomain
string
dnsDomain
Field
name
string
name
is the
in this
or
criSocket
string
criSocket
taints
taints
If this field is unset, i.e. nil, 

in the
taints: []
[]string
13.15 - kubeadm Configuration (v1beta3)
Overview
Package v1beta3 defines the v1beta3 version of the kubeadm configuration file format.
This version improves on the v1beta2 format by fixing some minor issues and adding a few new fields.
A list of changes since v1beta2:
The deprecated "ClusterConfiguration.useHyperKubeImage" field has been removed.
Kubeadm no longer supports the hyperkube image.
The "ClusterConfiguration.DNS.Type" field has been removed since CoreDNS is the only supported
DNS server type by kubeadm.
Include "datapolicy" tags on the fields that hold secrets.
This would result in the field values to be omitted when API structures are printed with klog.
Add "InitConfiguration.SkipPhases", "JoinConfiguration.SkipPhases" to allow skipping
a list of phases during kubeadm init/join command execution.
Add "InitConfiguration.NodeRegistration.ImagePullPolicy" and "JoinConfiguration.NodeRegistration.ImagePullPolicy"
to allow specifying the images pull policy during kubeadm "init" and "join".
The value must be one of "Always", "Never" or "IfNotPresent".
"IfNotPresent" is the default, which has been the existing behavior prior to this addition.
Add "InitConfiguration.Patches.Directory", "JoinConfiguration.Patches.Directory" to allow
the user to configure a directory from which to take patches for components deployed by kubeadm.
Move the BootstrapToken* API and related utilities out of the "kubeadm" API group to a new group
"bootstraptoken". The kubeadm API version v1beta3 no longer contains the BootstrapToken* structures.
kubeadm v1.15.x and newer can be used to migrate from v1beta1 to v1beta2.
kubeadm v1.22.x and newer no longer support v1beta1 and older APIs, but can be used to migrate v1beta2 to v1beta3.
Basics
--config
---
).
:
kind
:
:
kind
:
:
kind
:
:
kind
:
:
kind
:
init
or
join
) and by the configuration options you are going to use (defaults or advanced
customization).
If some configuration types are not provided, or provided only partially, kubead

m will use default values; defaults
relevance for security (e.g. enforce authorization-mode Node and RBAC on api server)
--config
:
kind
:
:
...
:
...
The InitConfiguration type should be used to configure runtime settings, that in case of kubeadm init
are the configuration of the bootstrap token and all the setting which are specific to the node where
kubeadm is executed, including:
NodeRegistration, that holds fields that relate to registering the new node to the cluster;
LocalAPIEndpoint, that represents the endpoint of the instance of the API server to be deployed on this node;
:
kind
:
:
...
etcd
:
...
apiServer
:
extraArgs
:
...
:
...
...
that holds configuration for the networking topology of the cluster; use it e.g. to customize
Pod subnet or services subnet.
etcd
: use it e.g. to customize the local etcd or to configure the API server
:
kind
:
...
The KubeProxyConfiguration type should be used to change the configuration passed to kube-proxy instances
for kube-proxy official documentation.
:
kind
:
...
run.
:
kind
:
:
-
token
:
:
ttl
:
"24h"
-
token
:
:
usages
:
- signing
groups
:
:
name
:
criSocket
:
taints
:
-
key
:
value
:
effect
:
:
v
:
4
:
:
"IfNotPresent"

:
:
bindPort
:
6443
:
:
- addon/kube-proxy
---
:
kind
:
etcd
:
local
:
:
imageTag
:
"3.2.24"
dataDir
:
extraArgs
:
:
:
-
:
-
# endpoints:
# - "10.100.0.1:2379"
# - "10.100.0.2:2379"
# caFile: "/etcd/kubernetes/pki/etcd/etcd-ca.crt"
# certFile: "/etcd/kubernetes/pki/etcd/etcd.crt"
# keyFile: "/etcd/kubernetes/pki/etcd/etcd.key"
:
:
podSubnet
:
dnsDomain
:
:
"v1.21.0"
:
apiServer
:
extraArgs
:
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
certSANs
:
-
-
:
4m0s
:
extraArgs
:
"20"
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
scheduler
:
extraArgs
:
address
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
:
:
:
---
:
kind
:
---
:
kind
:
When executing
with the
--config
:
kind
:
...
Field
token
token
string
ttl
ttl
defines the time to live for this token. Defaults to
24h
.
expires
and
ttl
expires
expires
ttl
.
expires
and
ttl
usages
[]string
usages
groups
[]string
groups
used for authentication
BootstrapTokenString is a token of the format
abcdef.abcdef0123456789
that is used
for both validation of the practically of the API se

rver from a joining node's point
"kubeadm join". This token is and should be short-lived.
Field
-
string
-
string
Field
string
kind
string
etcd
Etcd
etcd
holds the configuration for etcd.
string
string
sets a stable IP address or DNS name for the control plane.
It can be a valid IP address or a RFC-1123 DNS subdomain, both with optional TCP port.
In case the
+
bindPort
the
bindPort
is used.
could
be used for assigning a stable DNS to the control plane.
apiServer
APIServer
apiServer
scheduler
scheduler
dns
DNS
dns
string
string
If empty,
will be used by default.
In case of kubernetes version is a CI build (kubernetes version starts with
ci/
)
will be used as a default for control plane components
and for kube-proxy, while
string
-only information. These fields are solely used the first time
runs.
After that, the information in the fields IS NOT uploaded to the
ConfigMap
that is used by
for instance. These fields must be omitempty.
Field
string
kind
string
time and describes a set of Bootstrap Tokens to create.
This information IS NOT uploaded to the kubeadm cluster configmap, partly because of its sensitive nature
holds fields that relate to registering the new control-plane node
to the cluster.
represents the endpoint of the API server instance that's deployed on this
control plane node. In HA setups, this differs from
in the sense that
is the global endpoint for the cluster, which then
load-balances the requests to each individual API server.
This configuration object lets you customize what IP/DNS name and port the local API server
advertises it's accessible on. By default, kubeadm tries to auto-detect the IP of the default
interface and use that, but in case that process fails you may set the desired value here.
string
sets the key with which certificates and keys are encrypted prior to being
uploaded in a Secret in

 the cluster during the
phase.
[]string
is a list of phases to skip during command execution.
The list of phases can be obtained with the
command.
The flag "--skip-phases" takes precedence over this field.
patches
Patches
patches
contains options related to applying patches to components deployed by kubeadm during
.
Field
string
kind
string
control-plane node to the cluster.
string
is the path to the SSL certificate authority used to secure
comunications between a node and the control-plane.
discovery
Discovery
discovery
[]string
command.
The flag
takes precedence over this field.
patches
Patches
patches
contains options related to applying patches to components deployed
by kubeadm during
.
Field
string
bindPort
int32
bindPort
APIServer
APIServer holds settings necessary for API server deployments in the cluster
Field
certSANs
[]string
certSANs
sets extra Subject Alternative Names (SANs) for the API Server signing
controls the timeout that we wait for API server to appear.
Discovery
Field
token
string
token
is a token used to validate cluster information fetched from the
control-plane.
string
is an IP or domain name to the API server from which
information will be fetched.
[]string
Each hash is specified as
&lt;type&gt;:&lt;value&gt;
, where the only currently supported type is
"sha256". This is a hex-encoded SHA-256 hash of the Subject Public Key Info (SPKI)
object in DER-encoded ASN.1. These hashes can be calculated using, for example, OpenSSL.
bool
allows token-based discovery without CA verification
via
APIServer
Field
extraArgs
extraArgs
is an extra set of flags to pass to the control plane component.
is an extra set of host volumes, mounted to the control pl

ane component.
DNS
Field
ImageMeta
ImageMeta
ImageMeta
imageMeta
allows to customize the image used for the DNS component.
Discovery
Discovery specifies the options for the kubelet to use during the TLS Bootstrap process.
Field
and
file
file
file
and
file
string
If
, but
can be overridden. If
file
must be set
in case the KubeConfigFile
does not contain any other authentication information
timeout
timeout
Etcd
Field
local
LocalEtcd
local
local
and
external
external
external
local
and
external
Etcd
Field
endpoints
[]string
endpoints
contains the list of etcd members.
caFile
string
caFile
certFile
string
certFile
keyFile
string
keyFile
Discovery
FileDiscovery is used to specify a file or URL to a kubeconfig file from which to load
Field
string
is used to specify the actual file path or URL to the kubeconfig
file from which to load cluster information.
Field
name
string
name
is the name of the volume inside the Pod template.
hostPath
string
hostPath
mountPath
string
mountPath
is the path inside the Pod where
hostPath
will be mounted.
readOnly
bool
readOnly
pathType
pathType
is the type of the
hostPath
.
ImageMeta
DNS
LocalEtcd
Field
string
defined in ClusterConfiguration will be used instead.
imageTag
string
imageTag
allows to specify a tag for the image.
In case this value is set, kubeadm does not change automatically the version of
the above components during upgrades.
Field
represents the endpoint of the API server instance to be
deployed on this node.
string
they are downloade

d from the secret upon joining a new control plane node.
LocalEtcd
Etcd
LocalEtcd describes that kubeadm should run an etcd cluster locally
Field
ImageMeta
ImageMeta
ImageMeta
dataDir
string
dataDir
extraArgs
extraArgs
inside a static Pod. A key in this map is the flag name as it appears on the
[]string
sets extra Subject Alternative Names (SANs) for the etcd
server signing certificate.
[]string
sets extra Subject Alternative Names (SANs) for the etcd peer
Field
string
is the subnet used by Kubernetes Services. Defaults to "10.96.0.0/12".
podSubnet
string
podSubnet
dnsDomain
string
dnsDomain
is the DNS domain used by Kubernetes Services. Defaults to "cluster.local".
NodeRegistrationOptions holds fields that relate to registering a new control-plane or
node to the cluster, either via "kubeadm init" or "kubeadm join"
Field
name
string
name
is the
field of the Node API object that will be created in this
or
field of the kubelet's client certificate to
the API server.
criSocket
string
criSocket
is used to retrieve container runtime info.
This information will be annotated to the Node API object, for later re-use
taints
taints
If this field is unset, i.e. nil, it will be defaulted with a control-plane taint for control-plane nodes.
If you don't want to taint your control-plane node, set this field to an empty list,
i.e.
taints: []
in the YAML file. This field is solely used for Node registration.
passes through extra arguments to the kubelet.
The arguments here are passed to the kubelet command line via the environment file
kubeadm writes at runtime for the kubelet to source.
This overrides the generic base-level configuration in the
kubeadm is executing on. A key in this map is the flag name as it appears on the
[]string
provides a list of pre-flight errors to be ignored when
the current node is registered.
core/v1.PullPolicy
specifies the policy for image pulling during kubeadm "init" and
"join" operations.
The value of this field must be one of "Always", "IfNotPresent" or "Never".
If this field is not set, kubeadm will default it to "IfNotPresent", or pull the required
images if not present on the host.
Patches
Patches contains options related to applying patches to components deployed by kubeadm.
Field
directory
string
directory
is a path to a directory that contains files named
"target[suffix][+patchtype].extension".
For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of
"kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd". "patchtype" can
be one of "strategic" "merge" or "json" and they match the patch formats supported by
kubectl.
The default "patchtype" is "str

ategic". "extension" must be either "json" or "yaml".
"suffix" is an optional string that can be used to determine which patches are applied
first alpha-numerically.
13.16 - kubeconfig (v1)
Config
AuthInfo
Field
string
[]byte
string
[]byte
token
string
tokenFile
string
as
string
as-uid
string
as-groups
[]string
username
string
password
string
exec
AuthInfo
Field
name
string
config
Cluster
Field
server
string
string
TLSServerName is used to check server certificate. If TLSServerName is empty, the hostname used to contact the server is used.
bool
string
[]byte
proxy-url
string
bool
Context
Appears i

n:
Field
cluster
string
user
string
namespace
string
AuthInfo
Field
command
string
args
[]string
env
string
string
bool
details.
to "IfAvailable" when unset. Otherwise, this field is required.
Field
name
string
value
string
(Alias of
string
)
Config
Field
name
string
user
AuthInfo
Config
Field
name
string
cluster
Cluster
Config
Field
name
string
context
Context
Config
AuthInfo
Cluster
Context
Field
name
string
extension
Config
Field
co

lors
bool
13.17 - Kubelet Configuration (v1)
Field
string
kind
string
providers
Field
name
string
[]string
'.
gcr.io
.
string
args
[]string
env
Field
name
string
value
string
Field
json
Only ava

ilable when the LoggingAlphaOptions feature gate is enabled.
Field
bool
Field
format
string
text
verbosity
vmodule
options
Field
endpoint
string
int32
(Alias of
)
(Alias of
uint32
)
13.18 - Kubelet Configuration (v1alpha1)
Field
string
kubelet.config.k8s.io/v1alpha1
kind
string
providers
Field
name

string
[]string
as subdomains like
*.k8s.io
or
k8s.*.io
, and top-level-domains such as
k8s.*
.
Matching partial subdomains like
*.io
does not match
*.k8s.io
.
gcr.io
string
args
[]string
env
Field
name
string
value
string
13.19 - Kubelet Configuration (v1beta1)
SerializedNodeConfigSource
Field
string
kind
string
providers
KubeletConfiguration contains the configuration for the Kubelet
Field
string
kind
string
enableServer
bool
enableServer enables Kubelet's secured server.
Note: Kubelet's insecure port is controlled by the readOnlyPort option.
Default: true
staticPodPath
string
staticPodPath is the path to the directory containing local (static) pods to
run, or the path to a single static pod file.
Default: ""
syncFrequency
syncFrequency is the max period between synchronizing running
containers and config.
D

efault: "1m"
fileCheckFrequency is the duration between checking config files for
new data.
Default: "20s"
httpCheckFrequency
httpCheckFrequency is the duration between checking http for new data.
staticPodURL
string
staticPodURL is the URL for accessing static pods to run.
staticPodURLHeader
staticPodURLHeader is a map of slices with HTTP headers to use when accessing the podURL.
Default: nil
address
string
address is the IP address for the Kubelet to serve on (set to 0.0.0.0
for all interfaces).
Default: "0.0.0.0"
port
int32
port is the port for the Kubelet to serve on.
The port number must be between 1 and 65535, inclusive.
Default: 10250
readOnlyPort
int32
readOnlyPort is the read-only port for the Kubelet to serve on with
no authentication/authorization.
Setting this field to 0 disables the read-only service.
Default: 0 (disabled)
tlsCertFile
string
tlsCertFile is the file containing x509 Certificate for HTTPS. (CA cert,
if any, concatenated after server cert). If tlsCertFile and
tlsPrivateKeyFile are not provided, a self-signed certificate
and key are generated for the public address and saved to the directory
passed to the Kubelet's --cert-dir flag.
tlsPrivateKeyFile
string
tlsPrivateKeyFile is the file containing x509 private key matching tlsCertFile.
tlsCipherSuites
[]string
tlsCipherSuites is the list of allowed cipher suites for the server.
Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants).
tlsMinVersion
string
tlsMinVersion is the minimum TLS version supported.
bool
rotateCertificates enables client certificate rotation. The Kubelet will request a
new certificate from the certificates.k8s.io API. This requires an approver to approve the
certificate signing requests.
Default: false
bool
serverTLSBootstrap enables server certificate bootstrap. Instead of self
signing a serving certificate, the Kubelet will request a certificate from
the 'certificates.k8s.io' API. This requires an approver to approve the
certificate signing requests (CSR). The RotateKubeletServerCertificate feature
must be enabled when setting this field.
authentication
KubeletAuthentication
authentication specifies how requests to the Kubelet's server are authenticated.
Defaults:
anonymous:
enabled: false
webhook:
enabled: true
cacheTTL: "2m"
authorization
KubeletAuthorization
authorization specifies how requests to the Kubelet's server are authorized.
Defaults:
mode: Webhook
webhook:
cacheAuthorizedTTL: "5m"
cacheUnauthorizedTTL: "30s"
registryPullQPS
int32
registryPullQPS is the limit of registry pulls per second.
The value must not be a negative number.
Setting it to 0 means no limit.
Default: 5
registryBurst
int32
registryBurst is the maximum size of bursty pulls, temporarily allows
pulls to burst to this number, while still not exceeding registryPullQPS.
Only used if registryPullQPS is greater than 0.
Default: 10
eventRecordQPS
int32
eventRecordQPS is the maximum event creations per second. If 0, there
is no limit enforced. The value cannot be a negative number.
Default: 5
eventBurst
int32
eventBurst is the maximum size of a burst of event creations, temporarily
allows event creations to burst to this number, while still not exceeding
eventRecordQPS. This field canot be a negative number and it is only used
when eventRecordQPS &gt; 0.
enableDebuggingHandlers
bool
enableDebuggingHandlers enables server endpoints for log access
and local running of containers and commands, including the exec,
attach, logs, and portforward features.
bool
enableContentionProfiling enables lock c

ontention profiling, if enableDebuggingHandlers is true.
healthzPort
int32
healthzPort is the port of the localhost healthz endpoint (set to 0 to disable).
A valid number is between 1 and 65535.
Default: 10248
string
healthzBindAddress is the IP address for the healthz server to serve on.
Default: "127.0.0.1"
int32
oomScoreAdj is The oom-score-adj value for kubelet process. Values
must be within the range [-1000, 1000].
Default: -999
clusterDomain
string
clusterDomain is the DNS domain for this cluster. If set, kubelet will
configure all containers to search this domain in addition to the
host's search domains.
clusterDNS
[]string
clusterDNS is a list of IP addresses for the cluster DNS server. If set,
kubelet will configure all containers to use this for DNS resolution
instead of the host's DNS servers.
streamingConnectionIdleTimeout
streamingConnectionIdleTimeout is the maximum time a streaming connection
can be idle before the connection is automatically closed.
Default: "4h"
nodeStatusUpdateFrequency
nodeStatusUpdateFrequency is the frequency that kubelet computes node
status. If node lease feature is not enabled, it is also the frequency that
kubelet posts node status to master.
Note: When node lease feature is not enabled, be cautious when changing the
constant, it must work with nodeMonitorGracePeriod in nodecontroller.
Default: "10s"
nodeStatusReportFrequency
nodeStatusReportFrequency is the frequency that kubelet posts node
status to master if node status does not change. Kubelet will ignore this
frequency and post node status immediately if any change is detected. It is
only used when node lease feature is enabled. nodeStatusReportFrequency's
default value is 5m. But if nodeStatusUpdateFrequency is set explicitly,
nodeStatusReportFrequency's default value will be set to
nodeStatusUpdateFrequency for backward compatibility.
Default: "5m"
nodeLeaseDurationSeconds
int32
nodeLeaseDurationSeconds is the duration the Kubelet will set on its corresponding Lease.
NodeLease provides an indicator of node health by having the Kubelet create and
periodically renew a lease, named after the node, in the kube-node-lease namespace.
If the lease expires, the node can be considered unhealthy.
The lease is currently renewed every 10s, per KEP-0009. In the future, the lease renewal
interval may be set based on the lease duration.
The field value must be greater than 0.
Default: 40
imageMinimumGCAge
imageMinimumGCAge is the minimum age for an unused image before it is
garbage collected.
Default: "2m"
imageGCHighThresholdPercent
int32
imageGCHighThresholdPercent is the percent of disk usage after which
image garbage collection is always run. The percent is calculated by
dividing this field value by 100, so this field must be between 0 and
100, inclusive. When specified, the value must be greater than
imageGCLowThresholdPercent.
Default: 85
imageGCLowThresholdPercent
int32
imageGCLowThresholdPercent is the percent of disk usage before which
image garbage collection is never run. Lowest disk usage to garbage
collect to. The percent is calculated by dividing this field value by 100,
so the field value must be between 0 and 100, inclusive. When specified, the
value must be less than imageGCHighThresholdPercent.
Default: 80
volumeStatsAggPeriod
volumeStatsAggPeriod is the frequency for calculating and caching volume
disk usage for all pods.
Default: "1m"
kubeletCgroups
string
kubeletCgroups is the absolute name of cgroups to isolate the kubelet in
systemCgroups
string
systemCgroups is absolute name of cgroups in which to place
all non-kernel processes that are not already in a container. Empty
for no container. Rolling back the flag requires a reboot.
The cgroupRoot must be specified if this field is not empty.
cgroupRoot
string
cgroupRoot is the root cgroup to use f

or pods. This is handled by the
container runtime on a best effort basis.
cgroupsPerQOS
bool
cgroupsPerQOS enable QoS based CGroup hierarchy: top level CGroups for QoS classes
and all Burstable and BestEffort Pods are brought up under their specific top level
QoS CGroup.
string
cgroupDriver is the driver kubelet uses to manipulate CGroups on the host (cgroupfs
or systemd).
Default: "cgroupfs"
string
cpuManagerPolicy is the name of the policy to use.
Requires the CPUManager feature gate to be enabled.
Default: "None"
cpuManagerPolicyOptions
cpuManagerPolicyOptions is a set of key=value which allows to set extra options
to fine tune the behaviour of the cpu manager policies.
Requires both the "CPUManager" and "CPUManagerPolicyOptions" feature gates to be enabled.
cpuManagerReconcilePeriod
cpuManagerReconcilePeriod is the reconciliation period for the CPU Manager.
memoryManagerPolicy
string
memoryManagerPolicy is the name of the policy to use by memory manager.
Requires the MemoryManager feature gate to be enabled.
Default: "none"
topologyManagerPolicy
string
topologyManagerPolicy is the name of the topology manager policy to use.
Valid values include:
: kubelet only allows pods with optimal NUMA node alignment for
requested resources;
: kubelet will favor pods with NUMA alignment of CPU and device
resources;
none
: kubelet has no knowledge of NUMA alignment of a pod's CPU and device resources.
: kubelet only allows pods with a single NUMA alignment
of CPU and device resources.
Policies other than "none" require the TopologyManager feature gate to be enabled.
topologyManagerScope
string
topologyManagerScope represents the scope of topology hint generation
that topology manager requests and hint providers generate. Valid values include:
container
: topology policy is applied on a per-container basis.
pod
: topology policy is applied on a per-pod basis.
"pod" scope requires the TopologyManager feature gate to be enabled.
Default: "container"
topologyManagerPolicyOptions
TopologyManagerPolicyOptions is a set of key=value which allows to set extra options
to fine tune the behaviour of the topology manager policies.
Requires both the "TopologyManager" and "TopologyManagerPolicyOptions" feature gates to be enabled.
qosReserved
qosReserved is a set of resource name to percentage pairs that specify
the minimum percentage of a resource reserved for exclusive use by the
guaranteed QoS tier.
Currently supported resources: "memory"
Requires the QOSReserved feature gate to be enabled.
runtimeRequestTimeout
runtimeRequestTimeout is the timeout for all runtime requests except long running
requests - pull, logs, exec and attach.
hairpinMode
string
hairpinMode specifies how the Kubelet should configure the container
bridge for hairpin packets.
Setting this flag allows endpoints in a Service to loadbalance back to
themselves if they should try to access their own Service. Values:
"promiscuous-bridge": make the container bridge promiscuous.
"hairpin-veth": set the hairpin flag on container veth interfaces.
"none": do nothing.
Generally, one must set
--hairpin-mode=hairpin-veth to
achieve hairpin NAT,
because promiscuous-bridge assumes the existence of a container bridge named cbr0.
Default: "promiscuous-bridge"
maxPods
int32
maxPods is the maximum number of Pods that can run on this Kubelet.
The value must be a non-negative integer.
Default: 110
podCIDR
string
podCIDR is the CIDR to use for pod IP addresses, only used in standalone mode.
In cluster mode, this is obtained from the control plane.
podPidsLimit
int64
podPidsLimit is the maximum number of PIDs in any pod.
Default: -1
resolvConf
string
resolvConf is the resolver configurati

on file used as the basis
for the container DNS resolution configuration.
If set to the empty string, will override the default and effectively disable DNS lookups.
Default: "/etc/resolv.conf"
runOnce
bool
runOnce causes the Kubelet to check the API server once for pods,
run those in addition to the pods specified by static pod files, and exit.
cpuCFSQuota
bool
cpuCFSQuota enables CPU CFS quota enforcement for containers that
specify CPU limits.
cpuCFSQuotaPeriod is the CPU CFS quota period value,
.
The value must be between 1 ms and 1 second, inclusive.
Requires the CustomCPUCFSQuotaPeriod feature gate to be enabled.
Default: "100ms"
nodeStatusMaxImages
int32
nodeStatusMaxImages caps the number of images reported in Node.status.images.
The value must be greater than -2.
Note: If -1 is specified, no cap will be applied. If 0 is specified, no image is returned.
Default: 50
maxOpenFiles
int64
maxOpenFiles is Number of files that can be opened by Kubelet process.
The value must be a non-negative number.
Default: 1000000
string
contentType is contentType of requests sent to apiserver.
Default: "application/vnd.kubernetes.protobuf"
kubeAPIQPS
int32
kubeAPIQPS is the QPS to use while talking with kubernetes apiserver.
Default: 5
kubeAPIBurst
int32
kubeAPIBurst is the burst to allow while talking with kubernetes API server.
This field cannot be a negative number.
bool
serializeImagePulls when enabled, tells the Kubelet to pull images one
at a time. We recommend
not
changing the default value on nodes that
run docker daemon with version &lt; 1.9 or an Aufs storage backend.
Issue #10959 has more details.
evictionHard is a map of signal names to quantities that defines hard eviction
thresholds. For example:
{"memory.available": "300Mi"}
.
To explicitly disable, pass a 0% or 100% threshold on an arbitrary resource.
Default:
memory.available: "100Mi"
nodefs.available: "10%"
nodefs.inodesFree: "5%"
imagefs.available: "15%"
evictionSoft
evictionSoft is a map of signal names to quantities that defines soft eviction thresholds.
.
evictionSoftGracePeriod
evictionSoftGracePeriod is a map of signal names to quantities that defines grace
periods for each soft eviction signal. For example:
{"memory.available": "30s"}
.
evictionPressureTransitionPeriod
evictionPressureTransitionPeriod is the duration for which the kubelet has to wait
before transitioning out of an eviction pressure condition.
evictionMaxPodGracePeriod
int32
evictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use
when terminating pods in response to a soft eviction threshold being met. This value
effectively caps the Pod's terminationGracePeriodSeconds value during soft evictions.
Note: Due to issue #64530, the behavior has a bug where this value currently just
overrides the grace period during soft eviction, which can increase the grace
period from what is set on the Pod. This bug will be fixed in a future release.
Default: 0
evictionMinimumReclaim
evictionMinimumReclaim is a map of signal names to quantities that defines minimum reclaims,
which describe the minimum amount of a given resource the kubelet will reclaim when
performing a pod eviction while that resource is under pressure.
{"imagefs.available": "2Gi"}
.
podsPerCore
int32
podsPerCore is the maximum number of pods per core. Cannot exceed maxPods.
If 0, there is no limit on the number of Pods.
Default: 0
enableControllerAttachDetach
bool
enableControllerAttachDetach enables the Attach/Detach controller to
manage attachment/detachment of volumes scheduled to this node, and
disables kubelet from executing any att

ach/detach operations.
Note: attaching/detaching CSI volumes is not supported by the kubelet,
so this option needs to be true for that use case.
protectKernelDefaults
bool
protectKernelDefaults, if true, causes the Kubelet to error if kernel
flags are not as it expects. Otherwise the Kubelet will attempt to modify
kernel flags to match its expectation.
makeIPTablesUtilChains
bool
makeIPTablesUtilChains, if true, causes the Kubelet ensures a set of iptables rules
are present on host.
These rules will serve as utility rules for various components, e.g. kube-proxy.
The rules will be created based on iptablesMasqueradeBit and iptablesDropBit.
iptablesMasqueradeBit
int32
iptablesMasqueradeBit is the bit of the iptables fwmark space to mark for SNAT.
Values must be within the range [0, 31]. Must be different from other mark bits.
Warning: Please match the value of the corresponding parameter in kube-proxy.
TODO: clean up IPTablesMasqueradeBit in kube-proxy.
Default: 14
iptablesDropBit
int32
iptablesDropBit is the bit of the iptables fwmark space to mark for dropping packets.
Default: 15
featureGates is a map of feature names to bools that enable or disable experimental
features. This field modifies piecemeal the built-in default values from
"k8s.io/kubernetes/pkg/features/kube_features.go".
failSwapOn
bool
failSwapOn tells the Kubelet to fail to start if swap is enabled on the node.
memorySwap
MemorySwapConfiguration
memorySwap configures swap memory available to container workloads.
string
containerLogMaxSize is a quantity defining the maximum size of the container log
file before it is rotated. For example: "5Mi" or "256Ki".
Default: "10Mi"
int32
containerLogMaxFiles specifies the maximum number of container log files that can
be present for a container.
Default: 5
configMapAndSecretChangeDetectionStrategy
ResourceChangeDetectionStrategy
configMapAndSecretChangeDetectionStrategy is a mode in which ConfigMap and Secret
managers are running. Valid values include:
Get
: kubelet fetches necessary objects directly from the API server;
Cache
: kubelet uses TTL cache for object fetched from the API server;
Watch
: kubelet uses watches to observe changes to objects that are in its interest.
Default: "Watch"
systemReserved
systemReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G)
pairs that describe resources reserved for non-kubernetes components.
Currently only cpu and memory are supported.
See http://kubernetes.io/docs/user-guide/compute-resources for more detail.
kubeReserved
kubeReserved is a set of ResourceName=ResourceQuantity (e.g. cpu=200m,memory=150G) pairs
that describe resources reserved for kubernetes system components.
Currently cpu, memory and local storage for root file system are supported.
See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
reservedSystemCPUs
string
The reservedSystemCPUs option specifies the CPU list reserved for the host
level system threads and kubernetes related threads. This provide a "static"
CPU list rather than the "dynamic" list by systemReserved and kubeReserved.
This option does not support systemReservedCgroup or kubeReservedCgroup.
string
showHiddenMetricsForVersion is the previous version for which you want to show
hidden metrics.
Only the previous minor version is meaningful, other values will not be allowed.
The format is
&lt;major&gt;.&lt;minor&gt;
, e.g.:
1.16
.
The purpose of this format is make sure you have the opportunity to notice
if the next release hides additional metrics, rather than being surprised
when they are permanently removed in the release after that.
systemReservedCgr

oup
string
systemReservedCgroup helps the kubelet identify absolute name of top level CGroup used
to enforce
compute resource reservation for OS system daemons.
Refer to
doc for more information.
kubeReservedCgroup
string
kubeReservedCgroup helps the kubelet identify absolute name of top level CGroup used
to enforce
KubeReserved
compute resource reservation for Kubernetes node system daemons.
Refer to
enforceNodeAllocatable
[]string
This flag specifies the various Node Allocatable enforcements that Kubelet needs to perform.
This flag accepts a list of options. Acceptable options are
none
,
pods
,
and
.
If
none
is specified, no other options may be specified.
When
is in the list, systemReservedCgroup must be specified.
When
is in the list, kubeReservedCgroup must be specified.
This field is supported only when
is set to true.
Refer to
Default: ["pods"]
[]string
A comma separated whitelist of unsafe sysctls or sysctl patterns (ending in
*
).
Unsafe sysctl groups are
,
,
,
,
and
net.*
. For example: "
kernel.msg*,net.ipv4.route.min_pmtu
"
Default: []
volumePluginDir
string
volumePluginDir is the full path of the directory in which to search
for additional third party volume plugins.
Default: "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/"
string
providerID, if set, sets the unique ID of the instance that an external
provider (i.e. cloudprovider) can use to identify a specific node.
kernelMemcgNotification
bool
kernelMemcgNotification, if set, instructs the kubelet to integrate with the
kernel memcg notification for determining if memory eviction thresholds are
exceeded rather than polling.
logging
logging specifies the options of logging.
Refer to
Logs Options
Default:
Format: text
enableSystemLogHandler
bool
enableSystemLogHandler enables system logs via web interface host:port/logs/
shutdownGracePeriod
shutdownGracePeriod specifies the total duration that the node should delay the
shutdown and total grace period for pod termination during a node shutdown.
Default: "0s"
shutdownGracePeriodCriticalPods
shutdownGracePeriodCriticalPods specifies the duration used to terminate critical
pods during a node shutdown. This should be less than shutdownGracePeriod.
For example, if shutdownGracePeriod=30s, and shutdownGracePeriodCriticalPods=10s,
during a node shutdown the first 20 seconds would be reserved for gracefully
terminating normal pods, and the last 10 seconds would be reserved for terminating
critical pods.
shutdownGracePeriodByPodPriority
[]ShutdownGracePeriodByPodPriority
shutdownGracePeriodByPodPriority specifies the shutdown grace period for Pods based
on their associated priority class value.
When a shutdown request is received, the Kubelet will initiate shutdown on all pods
running on the node with a grace period that depends on the priority of the pod,
and then wait for all pods to exit.
Each entry in the array represents the graceful shutdown time a pod with a priority
class value that lies in the range of that value and the next higher entry in the
list when the node is shutting down.
For example, to allow critical pods 10s to shutdown, priority&gt;=10000 pods 20s to
shutdown, and all remaining pods 30s to shutdown.
shutdownGracePeriodByPodPriority:
priority: 2000000000
shutdownGracePeriodSeconds: 10
priority: 10000
shutdownGracePeriodSeconds: 20
priority: 0
shutdownGracePeriodSeconds: 30
The time the Kubelet will wait before exiting will at most be the maximum of all
shutdownGracePeriodSeconds for each priority class range represented on the node.
When all pods have exited or reached th

eir grace periods, the Kubelet will release
the shutdown inhibit lock.
Requires the GracefulNodeShutdown feature gate to be enabled.
This configuration must be empty if either ShutdownGracePeriod or ShutdownGracePeriodCriticalPods is set.
reservedMemory
[]MemoryReservation
reservedMemory specifies a comma-separated list of memory reservations for NUMA nodes.
The parameter makes sense only in the context of the memory manager feature.
The memory manager will not allocate reserved memory for container workloads.
For example, if you have a NUMA0 with 10Gi of memory and the reservedMemory was
specified to reserve 1Gi of memory at NUMA0, the memory manager will assume that
only 9Gi is available for allocation.
You can specify a different amount of NUMA node and memory types.
You can omit this parameter at all, but you should be aware that the amount of
reserved memory from all NUMA nodes should be equal to the amount of memory specified
by the
node allocatable
.
If at least one node allocatable parameter has a non-zero value, you will need
to specify at least one NUMA node.
Also, avoid specifying:
Duplicates, the same NUMA node, and memory type, but with a different value.
zero limits for any memory type.
NUMAs nodes IDs that do not exist under the machine.
memory types except for memory and hugepages-
enableProfilingHandler
bool
enableProfilingHandler enables profiling via web interface host:port/debug/pprof/
enableDebugFlagsHandler
bool
enableDebugFlagsHandler enables flags endpoint via web interface host:port/debug/flags/v
seccompDefault
bool
SeccompDefault enables the use of
as the default seccomp profile for all workloads.
This requires the corresponding SeccompDefault feature gate to be enabled as well.
memoryThrottlingFactor
float64
MemoryThrottlingFactor specifies the factor multiplied by the memory limit or node allocatable memory
when setting the cgroupv2 memory.high value to enforce MemoryQoS.
Decreasing this factor will set lower high limit for container cgroups and put heavier reclaim pressure
while increasing will put less reclaim pressure.
See https://kep.k8s.io/2570 for more details.
Default: 0.8
registerWithTaints
registerWithTaints are an array of taints to add to a node object when
the kubelet registers itself. This only takes effect when registerNode
is true and upon the initial registration of the node.
registerNode
bool
registerNode enables automatic registration with the apiserver.
tracing
Tracing specifies the versioned configuration for OpenTelemetry tracing clients.
See https://kep.k8s.io/2832 for more details.
localStorageCapacityIsolation
bool
LocalStorageCapacityIsolation enables local ephemeral storage isolation feature. The default setting is true.
This feature allows users to set request/limit for container's ephemeral storage and manage it in a similar way
as cpu and memory. It also allows setting sizeLimit for emptyDir volume, which will trigger pod eviction if disk
usage from the volume exceeds the limit.
This feature depends on the capability of detecting correct root file system disk usage. For certain systems,
such as kind rootless, if this capability cannot be supported, the feature LocalStorageCapacityIsolation should be
disabled. Once disabled, user should not set request/limit for container's ephemeral storage, or sizeLimit for emptyDir.
SerializedNodeConfigSource allows us to serialize v1.NodeConfigSource.
This type is used internally by the Kubelet for tracking checkpointed dynamic configs.
It exists in the kubeletconfig API group because it is classified as a versioned input to the Kubelet.
Field
string
kind
string
source
core/v1.NodeConfigSource
source is the source that we are serializing.
C

redentialProviderConfig
Field
name
string
[]string
'.
gcr.io
.
string
credentialprovider.kubelet.k8s.io/v1beta1
args
[]string
env
Field
name
string
value
string
KubeletAnonymousAuthentication
Field
enabled
bool
enabled allows anonymous requests to the kubelet server.
Requests that are not rejected by another authentication method are treated as
anonymous requests.
Anonymous requests have a username of
.
Field
x509
KubeletX509Authentication
x509 contains settings related to x509 client certificate authentication.
webhook
KubeletWebhookAuthentication
webhook contains settings related to webhook bearer token authentication.
anonymous
anonymous contains settings related to anonymous authentication.
Field
mode
KubeletAuthorizationMode
mode is the authorization mode to apply to requests to the kubelet server.
Valid values are
and
Webhook
.
Webhook mode uses the SubjectAccessReview API to determine authorization.
webhook
KubeletWebhookAuthorization
webhook contains settings related to Webhook authorization.
(Alias of
string
)
Field

enabled
bool
enabled allows bearer token authentication backed by the
tokenreviews.authentication.k8s.io API.
cacheTTL
cacheTTL enables caching of authentication results
Field
cacheAuthorizedTTL
cacheAuthorizedTTL is the duration to cache 'authorized' responses from the
webhook authorizer.
cacheUnauthorizedTTL
cacheUnauthorizedTTL is the duration to cache 'unauthorized' responses from
the webhook authorizer.
Field
clientCAFile
string
clientCAFile is the path to a PEM-encoded certificate bundle. If set, any request
presenting a client certificate signed by one of the authorities in the bundle
is authenticated with a username corresponding to the CommonName,
and groups corresponding to the Organization in the client certificate.
MemoryReservation
MemoryReservation specifies the memory reservation of different types for each NUMA node
Field
numaNode
int32
limits
core/v1.ResourceList
Field
swapBehavior
string
swapBehavior configures swap memory available to container workloads. May be one of
"", "LimitedSwap": workload combined memory and swap usage cannot exceed pod memory limit
"UnlimitedSwap": workloads can use unlimited swap, up to the allocatable limit.
(Alias of
string
)
ResourceChangeDetectionStrategy denotes a mode in which internal
managers (secret, configmap) are discovering object changes.
ShutdownGracePeriodByPodPriority
ShutdownGracePeriodByPodPriority specifies the shutdown grace period for Pods based on their associated priority class value
Field
priority
int32
priority is the priority value associated with the shutdown grace period
shutdownGracePeriodSeconds
int64
shutdownGracePeriodSeconds is the shutdown grace period in seconds
Field
json
Field
bool
Field
format
string
text
verbosity
vmodule
Only supp

orted for "text" log format.
options
Field
endpoint
string
int32
(Alias of
)
(Alias of
uint32
)
13.20 - Kubelet CredentialProvider (v1)
Field
string
kind
string
image
string
Field
string
kind
string
auth
map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1.AuthConfig
'.
Both contain the same number of domain parts and each

 part matches.
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
13.21 - Kubelet CredentialProvider (v1alpha1)
Field
string
kind
string
image
string
Field
string
kind
string
auth
'.
Both contain the same number of domain 

parts and each part matches.
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
13.22 - Kubelet CredentialProvider (v1beta1)
Field
string
kind
string
image
string
Field
string
kind
string
auth
map[string]k8s.io/kubelet/pkg/apis/credentialprovider/v1beta1.AuthConfig
'.
Both contain the same number 

of domain parts and each part matches.
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
13.23 - WebhookAdmission Configuration (v1)
Field
string
kind
string
string
14 - Scheduling
14.1 - Scheduler Configuration
You can customize the behavior of the
by writing a configuration
file and passing its path as a command line argument.
A scheduling Profile allows you to configure the different stages of scheduling
in the
.
Each stage is exposed in an extension point. Plugins provide scheduling behaviors
by implementing one or more of these extension points.
You can specify scheduling profiles by running
kube-scheduler --config &lt;filename&gt;
,
using the
KubeSchedulerConfiguration (
v1beta3
or
v1
)
struct.
A minimal configuration looks as follows:
:
kind
:
:
:
/etc/srv/kubernetes/kube-scheduler/kubeconfig
Note:
v1beta2
is deprecated in v1.25 and will be removed in v1.26. Please migrate KubeSchedulerConfiguration to
v1beta3
or
v1
before upgrading Kubernetes to v1.25.
Profiles
in the
.
Each stage is exposed in an
extension point
.
Plugins
provide scheduling behaviors by implementing one
or more of these extension points.
You can configure a single instance of
to run
multiple profiles
.
Extension points
Scheduling happens in a series of stages that are exposed through the following
extension points:
queueSort
: These plugins provide an ordering function that is used to
sort pending Pods in the scheduling queue. Exactly one queue sort plugin
may be enabled at a time.
preFilter
: These plugins are used to pre-process or check information
about a Pod or the cluster before filtering. They can mark a pod as
unschedulable.
filter
: These plugins are the equivalent of Predicates in a scheduling
Policy and are used to filter out nodes that can not run the Pod. Filters
are called in the configured order. A pod is marked as unschedulable if no
nodes pass all the filters.
: These plugins are called in their configured order when no
feasible nodes were found for the pod. If any
plugin marks the
Pod
schedulable
, the remaining plugins are not called.
preScore
: This is an informational extension point that can be used
for doing pre-scoring work.
score

: These plugins provide a score to each node that has passed the
filtering phase. The scheduler will then select the node with the highest
weighted scores sum.
reserve
: This is an informational extension point that notifies plugins
when resources have been reserved for a given Pod. Plugins also implement an
Unreserve
call that gets called in the case of failure during or after
Reserve
.
permit
: These plugins can prevent or delay the binding of a Pod.
preBind
: These plugins perform any work required before a Pod is bound.
bind
: The plugins bind a Pod to a Node.
bind
plugins are called in order
and once one has done the binding, the remaining plugins are skipped. At
least one bind plugin is required.
postBind
: This is an informational extension point that is called after
a Pod has been bound.
: This is a config-only field that allows plugins to be enabled
or disabled for all of their applicable extension points simultaneously.
For each extension point, you could disable specific
default plugins
or enable your own. For example:
:
kind
:
profiles
:
-
plugins
:
score
:
disabled
:
-
name
:
enabled
:
-
name
:
MyCustomPluginA
weight
:
2
-
name
:
MyCustomPluginB
weight
:
1
You can use
*
as name in the disabled array to disable all default plugins
for that extension point. This can also be used to rearrange plugins order, if
desired.
Scheduling plugins
The following plugins, enabled by default, implement one or more of these
ImageLocality
: Favors nodes that already have the container images that the
Pod runs.
Extension points:
score
.
TaintToleration
: Implements
.
Implements extension points:
filter
,
preScore
,
score
.
NodeName
: Checks if a Pod spec node name matches the current node.
filter
.
NodePorts
: Checks if a node has free ports for the requested Pod ports.
preFilter
,
filter
.
node selectors
and
node affinity
.
filter
,
score
.
Pod topology spread
.
preFilter
,
filter
,
preScore
,
score
.
NodeUnschedulable
: Filters out nodes that have
.spec.unschedulable
set to
true.
filter
.
NodeResourcesFit
: Checks if the node has all the resources that the Pod is
requesting. The score can use one of three strategies:
LeastAllocated
(default),
MostAllocated
and
RequestedToCapacityRatio
.
preFilter
,
filter
,
score
.
NodeResourcesBalancedAllocation
: Favors nodes that would obtain a more
balanced resource usage if the Pod is scheduled there.
score
.
VolumeBinding
: Checks if the node has or if it can bind the requested
volumes
.
preFilter
,
filter
,
reserve
,
preBind
,
score
.
Note:
score
extension point is enabled when
feature is
enabled. It prioritizes the smallest PVs that can fit the requested volume
size.
VolumeRestrictions
: Checks that volumes mounted in the node satisfy
restrictions that are specific to the volume provider.
filter
.
VolumeZone
: Checks that volumes requested satisfy any zone requirements they
might have.
filter
.
NodeVolumeLimits
: Checks that CSI volume limits can be satisfied for the
node.
filter
.
EBSLimits
: Checks that AWS EBS volume limits can be satisfied for the node.
filter
.
GCEPDLimits
: Checks that GCP-PD volume limits can be satisfied for the node.
filter
.
AzureDiskLimits
: Checks that Azure disk volume limits can be satisfied for
the node.
filter
.
InterPodAffinity
inter-Pod affinity and anti-affinity
.
preFilter
,
filter
,
preScore
,
score
.
PrioritySort
: Provides the default priority based sorting.
queueSort
.
DefaultBinder
: Provides the default binding mechanism.
Extension poin

ts:
bind
.
DefaultPreemption
: Provides the default preemption mechanism.
.
You can also enable the following plugins, through the component config APIs,
that are not enabled by default:
CinderLimits
: Checks that
OpenStack Cinder
volume limits can be satisfied for the node.
filter
.
Multiple profiles
to run more than one profile.
Each profile has an associated scheduler name and can have a different set of
plugins configured in its
extension points
.
With the following sample configuration, the scheduler will run with two
profiles: one with the default plugins and one with all scoring plugins
disabled.
:
kind
:
profiles
:
-
:
-
:
no
-scoring-scheduler
plugins
:
preScore
:
disabled
:
-
name
:
'*'
score
:
disabled
:
-
name
:
'*'
Pods that want to be scheduled according to a specific profile can include
the corresponding scheduler name in its
.spec.schedulerName
.
By default, one profile with the scheduler name
is created.
This profile includes the default plugins described above. When declaring more
than one profile, a unique scheduler name for each of them is required.
If a Pod doesn't specify a scheduler name, kube-apiserver will set it to
. Therefore, a profile with this scheduler name should exist
to get those pods scheduled.
Note:
Pod's scheduling events have
as the ReportingController.
Events for leader election use the scheduler name of the first profile in the
list.
Note:
All profiles must use the same plugin in the
queueSort
extension point and have
the same configuration parameters (if applicable). This is because the scheduler
only has one pending pods queue.
Plugins that apply to multiple extension points
Starting from
, there is an additional field in the
profile config,
, which allows for easily enabling or disabling a plugin
across several extension points. The intent of
config is to simplify the
configuration needed for users and administrators when using custom profiles.
Consider a plugin,
MyPlugin
, which implements the
preScore
,
score
,
preFilter
,
and
filter
extension points. To enable
MyPlugin
for all its available extension
points, the profile config looks like:
:
kind
:
profiles
:
-
:
multipoint-scheduler
plugins
:
:
enabled
:
-
name
:
MyPlugin
This would equate to manually enabling
MyPlugin
for all of its extension
points, like so:
:
kind
:
profiles
:
-
:
non-multipoint-scheduler
plugins
:
preScore
:
enabled
:
-
name
:
MyPlugin
score
:
enabled
:
-
name
:
MyPlugin
preFilter
:
enabled
:
-
name
:
MyPlugin
filter
:
enabled
:
-
name
:
MyPlugin
One benefit of using
here is that if
MyPlugin
implements another
extension point in the future, the
config will automatically enable it
for the new extension.
Specific extension points can be excluded from
MultiPoint
expansion using
the
disabled
field for that extension point. This works with disabling default
plugins, non-default plugins, or with the wildcard (
'*'
) to disable all plugins.
An example of this, disabling
Score
and
PreScore
, would be:
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
'MyPlugin'
preScore
:
disabled
:
-
name
:
'*'
score
:
disabled
:
-
name
:
'*'
, all
are enabled internally through
.
However, individual extension points are still available to allow flexible
reconfiguration of the default values (such as ordering and Score weights). For
exampl

e, consider two Score plugins
DefaultScore1
and
DefaultScore2
, each with
a weight of
1
. They can be reordered with different weights like so:
:
kind
:
profiles
:
-
:
plugins
:
score
:
enabled
:
-
name
:
'DefaultScore2'
weight
:
5
In this example, it's unnecessary to specify the plugins in
because they are default plugins. And the only plugin specified in
Score
is
.
This is because plugins set through specific extension points will always take precedence
over
plugins. So, this snippet essentially re-orders the two plugins
without needing to specify both of them.
The general hierarchy for precedence when configuring
plugins is as follows:
Specific extension points run first, and their settings override whatever is set elsewhere
Plugins manually configured through
and their settings
Default plugins and their default settings
To demonstrate the above hierarchy, the following example is based on these plugins:
Plugin
Extension Points
DefaultQueueSort
QueueSort
CustomQueueSort
QueueSort
DefaultPlugin1
Score
,
Filter
DefaultPlugin2
Score
CustomPlugin1
Score
,
Filter
CustomPlugin2
Score
,
Filter
A valid sample configuration for these plugins would be:
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
'CustomQueueSort'
-
name
:
'CustomPlugin1'
weight
:
3
-
name
:
'CustomPlugin2'
disabled
:
-
name
:
'DefaultQueueSort'
filter
:
disabled
:
-
name
:
'DefaultPlugin1'
score
:
enabled
:
-
name
:
'DefaultPlugin2'
Note that there is no error for re-declaring a
plugin in a specific
extension point. The re-declaration is ignored (and logged), as specific extension points
take precedence.
Besides keeping most of the config in one spot, this sample does a few things:
Enables the custom
queueSort
plugin and disables the default one
Enables
and
, which will run first for all of their extension points
Disables
, but only for
filter
Reorders
to run first in
score
(even before the custom plugins)
In versions of the config before
v1beta3
, without
, the above snippet would equate to this:
:
kind
:
profiles
:
-
:
plugins
:
# Disable the default QueueSort plugin
queueSort
:
enabled
:
-
name
:
disabled
:
-
name
:
# Enable custom Filter plugins
filter
:
enabled
:
-
name
:
-
name
:
-
name
:
disabled
:
-
name
:
# Enable and reorder custom score plugins
score
:
enabled
:
-
name
:
weight
:
1
-
name
:
weight
:
3
While this is a complicated example, it demonstrates the flexibility of
config
as well as its seamless integration with the existing methods for configuring extension points.
Scheduler configuration migrations
v1beta1 → v1beta2
v1beta2 → v1beta3
v1beta3 → v1
With the v1beta2 configuration version, you can use a new score extension for the
plugin.
The new extension combines the functionalities of the
NodeResourcesLeastAllocated
,
NodeResourcesMostAllocated
and
plugins.
For example, if you previously used the
plugin, you
would instead use
(enabled by default) and add a
with a
scoreStrategy
that is similar to:
:
kind
:
profiles
:
-
:
-
args
:
:
resources
:
-
name
:
cpu
weight
:
1
type
:
name
:
The scheduler plugin
NodeLabel
is deprecated; instead, use the
plugin (enabled by default) to achieve sim

ilar behavior.
ServiceAffinity
plugin (enabled by default) to achieve similar behavior.
NodePreferAvoidPods
is deprecated; instead, use
node taints
to achieve similar behavior.
A plugin enabled in a v1beta2 configuration file takes precedence over the default configuration for that plugin.
Invalid
host
or
port
configured for scheduler healthz and metrics bind address will cause validation failure.
Three plugins' weight are increased by default:
from 1 to 2
from 1 to 2
from 1 to 3
SelectorSpread
is removed, instead, use the
plugin (enabled by default)
Read the
kube-scheduler reference
Read the
reference
Read the
reference
Read the
reference
14.2 - Scheduling Policies
In Kubernetes versions before v1.23, a scheduling policy can be used to specify the
predicates
and
process. For example, you can set a scheduling policy by
running
kube-scheduler --policy-config-file &lt;filename&gt;
or
kube-scheduler --policy-configmap &lt;ConfigMap&gt;
.
This scheduling policy is not supported since Kubernetes v1.23. Associated flags
policy-config-file
,
policy-configmap
,
policy-configmap-namespace
and
use-legacy-policy-config
are also not supported. Instead, use the
kube-scheduler Configuration
Read the
kube-scheduler configuration reference (v1)
15 - Other Tools
Kubernetes contains several tools to help you work with the Kubernetes system.
crictl
crictl
is a command-line
interface for inspecting and debugging
CRI
-compatible
container runtimes.
Dashboard
Dashboard
, the web-based user interface of Kubernetes, allows you to deploy containerized applications
to a Kubernetes cluster, troubleshoot them, and manage the cluster and its
resources itself.
Helm
Helm
is a tool for managing packages of pre-configured
Kubernetes resources. These packages are known as
Helm charts
.
Use Helm to:
Find and use popular software packaged as Kubernetes charts
Share your own applications as Kubernetes charts
Create reproducible builds of your Kubernetes applications
Intelligently manage your Kubernetes manifest files
Manage releases of Helm packages
Kompose
Kompose
is a tool to help Docker Compose users move to Kubernetes.
Use Kompose to:
Translate a Docker Compose file into Kubernetes objects
Go from local Docker development to managing your application via Kubernetes
Convert v1 or v2 Docker Compose
yaml
files or
Distributed Application Bundles
Kui
Kui
is a GUI tool that takes your normal
kubectl
command line requests and responds with graphics.
Kui takes the normal
kubectl
command line requests and responds with graphics. Instead
of ASCII tables, Kui provides a GUI rendering with tables that you can sort.
Kui lets you:
Directly click on long, auto-generated resource names instead of copying and pasting
Type in
kubectl
commands and see them execute, even sometimes faster than
kubectl
itself
Query a
Job
and see its execution rendered
as a waterfall diagram
Click through resources in your cluster using a tabbed UI
Minikube
minikube
is a tool that
runs a single-node Kubernetes cluster locally on your workstation for
development and testing purposes.
15.1 - Mapping from dockercli to crictl
Note:
Note:
This page i

s deprecated and will be removed in Kubernetes 1.27.
crictl
is a command-line interface for
CRI
-compatible container runtimes.
crictl
cri-tools
This page provides a reference for mapping common commands for the
docker
command-line tool into the equivalent commands for
crictl
.
Mapping from docker CLI to crictl
The exact versions for the mapping table are for
docker
CLI v1.40 and
crictl
v1.19.0. This list is not exhaustive. For example, it doesn't include
experimental
docker
CLI commands.
Note:
The output format of
crictl
is similar to
docker
CLI, despite some missing
columns for some CLI. Make sure to check output for the specific command if your
command output is being parsed programmatically.
Retrieve debugging information
mapping from docker cli to crictl - retrieve debugging information
docker cli
crictl
Unsupported Features
attach
attach
Attach to a running container
--detach-keys
,
--sig-proxy
exec
exec
Run a command in a running container
--privileged
,
--user
,
images
images
info
info
Display system-wide information
inspect
inspect
,
inspecti
Return low-level information on a container, image or task
logs
logs
Fetch the logs of a container
--details
ps
ps
stats
stats
Display a live stream of container(s) resource usage statistics
Column: NET/BLOCK I/O, PIDs
version
version
Show the runtime (Docker, ContainerD, or others) version information
Perform Changes
mapping from docker cli to crictl - perform changes
docker cli
crictl
create
create
Create a new container
kill
stop
(timeout = 0)
Kill one or more running container
--signal
pull
pull
Pull an image or a repository from a registry
--all-tags
,
--disable-content-trust
rm
rm
Remove one or more containers
rmi
rmi
Remove one or more images
run
run
Run a command in a new container
start
start
Start one or more stopped containers
stop
stop
Stop one or more running containers
update
update
Update configuration of one or more containers
--restart
,
--blkio-weight
and some other resource limit not supported by CRI.
Supported only in crictl
mapping from docker cli to crictl - supported only in crictl
crictl
imagefsinfo
Return image filesystem info
inspectp
Display the status of one or more pods
Forward local port to a pod
pods
List pods
runp
Run a new pod
rmp
Remove one or more pods
stopp
Stop one or more running pods
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Custom Resour

ces
Tasks
Debug Pods
Debug Servic

es
Auditing
Run Jobs
TLS
Tutorials
Securi

ty
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Audit Configu

ration (v1)
Tasks
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Extend Kubernetes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

p kubectl on macOS
Debug Pods
Get a Shell to a Running C

ontainer
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's

 Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configuration (

v1)
Tasks
Understand advanced ways to adapt your Kubernetes cluster to the needs of your work environment.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Troubleshooting Applications | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Up kubectl on Linux
Debug Pods
Debug Ru

nning Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the N

amespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube

-apiserver Encryption Configuration (v1)
Tasks
Debug Pods
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Using CoreDNS for Service Discovery | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

v1alpha1)
Tasks
minikube
.
CoreDNS
CNCF
.
kube-dns
For
kubeadm
kubeadm
using
kube-dns
kubeadm
kube-dns
.
If you are upgrading your cluster using the
kubeadm
tool,
kubeadm
.
CoreDNS
kube-dns does by modifying the Cor

eDNS configuration ("Corefile").
for the
.
Feedback
Yes
No
.
or
.
Last modified March 08, 2022 at 9:09 PM PST:
Improve coredns page (15f643e5e8)
Home
Blog
Training
Partners
Community
CC BY 4.0

PriorityClass | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
kind
metadata
(
)
value
(string)
(boolean)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
timeo

utSeconds
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Mapping PodSecurityPolicies to Pod Security Standards | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Reference
.
Baseline
and
.
under the
.spec
Type
:
false
below.
Baseline
: subset of
CHOWN
FOWNER
FSETID
KILL
MKNOD
SETFCAP
SETGID
SETPCAP
SETUID
only
Baseline
ALL
volumes
Baseline
hostPath
*
: subset of
configMap
csi
emptyDir
ephemeral
projected
secret
:
false
hostPorts
hostPID
:
false
hostIPC
:
false
seLinux
:
is
MustRunAs
options
user
is unset (
""
role
is unset (
""
type
level
runAsUser
Baseline
: Anything
:
rule
is
No opinion
No opinion
fsGroup
No opinion
Mutating &am

p; Validating
No opinion
Mutating
false
Baseline
:
false
No opinion
:
Mutating
No opinion
No opinion
The
Type
Mutating
No opinion
Baseline
:
:
Mutating
No opinion
Baseline
:
:
Feedback
Yes
No
.
or
.
Last modified May 05, 2022 at 11:10 AM PST:
Scrub PSP docs for 1.25 (b167938367)
Home
Blog
Training
Partners
Community
CC BY 4.0

ValidatingAdmissionPolicy v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extendin

g Kubernetes
Tasks
Migrate from PodSecurityPolicy to the

 Built-In PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using Micr

oProfile, ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Client Authentication (v1beta1

)
Reference
kind
metadata
(
)
.
spec
Validations contain CEL expressions which is used to apply the validation. A minimum of one validation is required for a policy definition. Required.
ref
The
,
kind
,
and
dot
dash
' - '

/' escapes to '
slash
Examples:
namespace
&gt; 0"}
'set':
X + Y
X
Y
'map':
X + Y
X
Y
X
and
Y
Y
with
Required.
(string)
(string)
(string)
all
Constraints. However, in order to prevent clusters from being put into an unstable state that cannot be recovered from via the API ValidatingAdmissionPolicy cannot match ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding. Required.
' is all ver

sions. If '
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
ObjectSelector decides whether to run the validation based on if the object has matching labels. objectSelector is evaluated against both the oldObject 

and newObject that would be sent to the cel validation, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
any
Rule.
(string)
(string)
(string)
(string)
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info

:
items
([]
)
kind
(string)
metadata
(
ListMeta
)
(string)
kind
(string)
metadata
(
)
.
spec
spec.matchResources.excludeResou

rceRules.resourceNames
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
ObjectSelector decides whether to run the validation based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the cel validation, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. Default to th

e empty LabelSelector, which matches everything.
any
Rule.
(string)
ParamRef specifies the parameter resource used to configure the admission control policy. It should point to a resource of the type specified in ParamKind of the bound ValidatingAdmissionPolicy. If the policy specifies a ParamKind and the resource referred to by ParamRef does not exist, this binding is considered mis-configured and the FailurePolicy of the ValidatingAdmissionPolicy applied.
(string)
(string)
(string)
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
fieldS

elector
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Validat

ingAdmissionPolicyBinding
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure a Pod to Use a ConfigMap | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Ta

sks
Debug Pods
Determine the Reason for Pod Fail

ure
Auditing
Run Jobs
TLS
Tutorials
Security
App

ly Pod Security Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

rver Configuration (v1alpha1)
Tasks
minikube
.
kubectl
Use the
,
files
, or
:
.
You can use
or
You can use

=
and
in the

Data
====
----
lives=3
----
The
and
data
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"516"
uid
:
data
:
:
|
lives=3
:
|
You can use
=

Data
====
----
lives=3
=
=

Data
====
----
lives=3
----
When
kubectl
data
.
#

   There is no special handling of quotation marks (i.e. they will be part of the ConfigMap value)).
enemies
=
aliens
lives
=
3
allowed
=
"true"
\
=
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"809965"
uid
:
data
:
allowed
:
'"true"'
enemies
:
aliens
lives
:
"3"
kubectl
\
=
\
=
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"810136"
uid
:
data
:
allowed
:
'"true"'
color
:
purple
enemies
:
aliens
how
:
lives
:
"3"
textmode
:
"true"
data
argument:
=
=
where
=
=
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"530"
uid
:
05f8da22-d671-11e5-8cd0-68f728db1985
data
:
:
|
lives=3
You can use
with the
=
=
=
=
charm
data
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"651"
uid
:
data
:
:
very
:
charm
kubectl
supports
the Apiserver. The gener

ators
cat
files:
EOF
1
37s
{
:
"v1"
,
"data"
:
{
:
Data
====
----
lives=3
cat
files:
EOF
and
,
as
cat
literals:
EOF
=
=
very
Assign the
value defined in the ConfigMap to the
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
:
Never
.
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
---
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
log

_level
:
INFO
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
LOG_LEVEL
valueFrom
:
:
name
:
key
:
log_level
:
Never
and
.
Note:
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
:
charm
Use
envFrom
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
envFrom
:
-
:
name
:
:
Never
and
.
command
and
args
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
,
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
valueFrom
:
:
name
:
key
:
:
Never
very charm
Add ConfigMap data to a Volume
data
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
:
charm
volumes
).
The
command
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
]
:
-
name
:
config-v

olume
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
:
Never
Caution:
Note:
Use the
path
volume at
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
items
:
-
key
:
path
:
keys
:
Never
very
Caution:
basis. The
Secrets
Secrets
Note:
/etc
data
.
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
data
:
:
hello
:
world
:
|-
ConfigMap
optional
optional
optional
.
If you use
envFrom
LASTSEEN FIRSTSEEN COUNT NAME          KIND  SUBOBJECT  TYPE      REASON

                            SOURCE                MESSAGE
Namespace
optional
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
a-config
key
:
akey
optional
:
true
:
Never
a-config
a-config
a key named
akey
akey
in the
a-config
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
no
-config
optional
:
true
:
Never
ConfigMap. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as
Note:
subPath
.
Feedback
Yes
No
.
or
.
Last modified July 26, 2022 at 9:48 AM PST:
updates from Shannon feedback (ad33b0c107)
Use ConfigMap-de

fined environment variables in Pod commands
Home
Blog
Training
Partners
Community
CC BY 4.0

IP Masquerade Agent User Guide | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

all and Set Up kubectl on macOS
Debug Pods
Get a Shell t

o a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict 

a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Co

nfiguration (v1)
Tasks
.
minikube
.
The
CIDR
range.
Key Terms
Link Local
The ip-masq-agent configures iptables rules to handle masquerading node/pod IP addresses when sending traffic to destinations outside the cluster node's IP and the Cluster IP range. This essentially hides pod IP addresses behind the cluster node's IP address. In some environments, traffic to "external" addresses must come from a known machine address. For example, in Google Cloud, any traffic to the internet must come from a VM's IP. When containers are used, as in Google Kubernetes Engine, the Pod IP will be rejected for egress. To avoid this, we must hide the Pod IP behind the VM's own IP address - gen

erally known as "masquerade". By default, the agent is configured to treat the three private IP ranges specified by
RFC 1918
CIDR
CIDR
Traffic to 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16) ranges will NOT be masqueraded. Any other traffic (assumed to be internet) will be masqueraded. An example of a local destination from a pod could be its Node's IP address as well as another node's address or one of the IP addresses in Cluster's IP range. Any other traffic will be masqueraded by default. The below entries show the default set of rules that are applied by the ip-masq-agent:
DaemonSet
=
true
here
ConfigMap
ConfigMap
Note:
:
:
-
10.0.0.0
/8
:
60s
=
=
iptable

s -t nat -L IP-MASQ-AGENT
:
-
10.0.0.0
/8
:
60s
:
true
Feedback
Yes
No
.
or
.
Last modified March 25, 2022 at 1:59 PM PST:
Fix the ip-masq-agent page (3b21d5bc69)
Key Terms
Home
Blog
Training
Partners
Community
CC BY 4.0

Role | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up k

ubectl on Windows
Debug Pods
Resou

rce metrics pipeline
Auditing
Run Jobs
TLS
Tutorials
Security
Rest

rict a Container's Syscalls with seccomp
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Reference
Role
Role
Role
kind
: Role
metadata
(
)
rules
RoleList
kind
: RoleList
metadata
(
ListMeta
)
items
([]
Role
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
fi

eldSelector
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
create
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
202 (
Role
update
name
(
in path
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
patch
PATCH /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Was 

this page helpful?
Yes
No
.
or
.
Role
RoleList
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

SelfSubjectReview v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Com

Reference
kind
metadata
(
)
status
(
)
userInfo
(string)
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
Feedback
Yes
No

.
or
.
create
Home
Blog
Training
Partners
Community
CC BY 4.0

Connecting Applications with Services | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tutorials
Services
The Kubernetes model for connecting containers
Now that you have a continuously running, replicated application you can expose it on a network.
Kubernetes assumes that pods can communicate with other pods, regardless of which host they land on. Kubernetes gives every pod its own cluster-private IP address, so you do not need to explicitly create links between pods or map container ports to host ports. This means that containers within a Pod can all reach each other's ports on localhost, and all pods in a cluster can see each other without NAT. The rest of this document elaborates on how you can run reliable services on such a networking model.
This tutorial uses a simple nginx web server to demonstrate the concept.
Exposing pods to the cluster
We did this in a previous example, but let's do it once again and focus on the networking perspective.
Create an nginx Pod, and note that it has a container port specification:
service/networking/run-my-nginx.yaml
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
selector
:
:
run
:
my-nginx
replicas
:
2
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
name
:
my-nginx
image
:
nginx
ports
:
-
:
80
This makes it accessible from any node in your cluster. Check the nodes the Pod is running on:
kubectl apply -f ./run-my-nginx.yaml
run
=
my-nginx -o wide
NAME                        READY     STATUS    RESTARTS   AGE       IP            NODE
my-nginx-3800858182-jr4a2   1/1       Running   0          13s       10.244.3.4    kubernetes-minion-905m
my-nginx-3800858182-kna2y   1/1       Running   0          13s       10.244.2.5    kubernetes-minion-ljyd
Check your pods' IPs:
run
=
my-nginx -o custom-columns
=
POD_IP:.status.podIPs
POD_IP
[
map
[
ip:10.244.3.4
]]
[
map
[
ip:10.244.2.5
]]
You should be able to ssh into any node in your cluster and use a tool such as
curl
to make queries against both IPs. Note that the containers are
not
using port 80 on the node, nor are there any special NAT rules to route traffic to the pod. This means you can run multiple nginx pods on the same node all using the same
, and access them from any other pod or node in your cluster using the assigned IP address for the Service. If you want to arrange for a specific port on the host Node to be forwarded to backing Pods, you can - but the networking model should mean that you do not need to do so.
You can read more about

 the
Kubernetes Networking Model
if you're curious.
Creating a Service
So we have pods running nginx in a flat, cluster wide, address space. In theory, you could talk to these pods directly, but what happens when a node dies? The pods die with it, and the Deployment will create new ones, with different IPs. This is the problem a Service solves.
A Kubernetes Service is an abstraction which defines a logical set of Pods running somewhere in your cluster, that all provide the same functionality. When created, each Service is assigned a unique IP address (also called clusterIP). This address is tied to the lifespan of the Service, and will not change while the Service is alive. Pods can be configured to talk to the Service, and know that communication to the Service will be automatically load-balanced out to some pod that is a member of the Service.
You can create a Service for your 2 nginx replicas with
:
kubectl expose deployment/my-nginx
service/my-nginx exposed
This is equivalent to
kubectl apply -f
the following yaml:
service/networking/nginx-svc.yaml
:
v1
kind
:
Service
metadata
:
name
:
my-nginx
labels
:
run
:
my-nginx
spec
:
ports
:
-
port
:
80
protocol
:
TCP
selector
:
run
:
my-nginx
This specification will create a Service which targets TCP port 80 on any Pod
with the
run: my-nginx
label, and expose it on an abstracted Service port
(
: is the port the container accepts traffic on,
port
: is the
abstracted Service port, which can be any port other pods use to access the
Service).
View
Service
API object to see the list of supported fields in service definition.
Check your Service:
kubectl get svc my-nginx
NAME       TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
my-nginx   ClusterIP   10.0.162.149   &lt;none&gt;        80/TCP    21s
As mentioned previously, a Service is backed by a group of Pods. These Pods are
exposed through
.
The Service's selector will be evaluated continuously and the results will be POSTed
to an EndpointSlice that is connected to the Service using a
labels
.
When a Pod dies, it is automatically removed from the EndpointSlices that contain it
as an endpoint. New Pods that match the Service's selector will automatically get added
to an EndpointSlice for that Service.
Check the endpoints, and note that the IPs are the same as the Pods created in
the first step:
kubectl describe svc my-nginx
Name:                my-nginx
Namespace:           default
Labels:              run=my-nginx
Annotations:         &lt;none&gt;
Selector:            run=my-nginx
Type:                ClusterIP
IP:                  10.0.162.149
Port:                &lt;unset&gt; 80/TCP
Endpoints:           10.244.2.5:80,10.244.3.4:80
Session Affinity:    None
Events:              &lt;none&gt;
kubectl get endpointslices -l kubernetes.io/service-name
=
my-nginx
NAME             ADDRESSTYPE   PORTS   ENDPOINTS               AGE
my-nginx-7vzhx   IPv4          80      10.244.2.5,10.244.3.4   21s
You should now be able to curl the nginx Service on
&lt;CLUSTER-IP&gt;:&lt;PORT&gt;
from
any node in your cluster. Note that the Service IP is completely virtual, it
never hits the wire. If you're curious about how this works you can read more
about the
service proxy
.
Accessing the Service
Kubernetes supports 2 primary modes of finding a Service - environment variables
and DNS. The former works out of the box while the latter requires the
CoreDNS cluster addon
.
Note:
If the service environment variables are not desired (because possible clashing with expected program ones,
too many variables to process, only using DNS, etc) you can disable this mode by setting the
flag to
false
on the
pod spec
.
Environment Variables
When a Pod runs on a Node, the kubelet adds a set of environment variables for
each active Service. This introduces an ordering problem. To see why, inspect
the environment of your running nginx Pods (your Pod name will be different):
kubectl
exec
my-nginx

-3800858182-jr4a2 -- printenv | grep SERVICE
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
Note there's no mention of your Service. This is because you created the replicas
before the Service. Another disadvantage of doing this is that the scheduler might
put both Pods on the same machine, which will take your entire Service down if
it dies. We can do this the right way by killing the 2 Pods and waiting for the
Deployment to recreate them. This time around the Service exists
before
the
replicas. This will give you scheduler-level Service spreading of your Pods
(provided all your nodes have equal capacity), as well as the right environment
variables:
kubectl scale deployment my-nginx --replicas
=
0; kubectl scale deployment my-nginx --replicas
=
2;
run
=
NAME                        READY     STATUS    RESTARTS   AGE     IP            NODE
my-nginx-3800858182-e9ihh   1/1       Running   0          5s      10.244.2.7    kubernetes-minion-ljyd
my-nginx-3800858182-j4rm4   1/1       Running   0          5s      10.244.3.8    kubernetes-minion-905m
You may notice that the pods have different names, since they are killed and recreated.
kubectl
exec
my-nginx-3800858182-e9ihh -- printenv | grep SERVICE
MY_NGINX_SERVICE_HOST=10.0.162.149
MY_NGINX_SERVICE_PORT=80
DNS
Kubernetes offers a DNS cluster addon Service that automatically assigns dns names to other Services. You can check if it's running on your cluster:
kubectl get services kube-dns --namespace
=
NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)         AGE
kube-dns   ClusterIP   10.0.0.10    &lt;none&gt;        53/UDP,53/TCP   8m
The rest of this section will assume you have a Service with a long lived IP
(my-nginx), and a DNS server that has assigned a name to that IP. Here we use the CoreDNS cluster addon (application name
kube-dns
), so you can talk to the Service from any pod in your cluster using standard methods (e.g.
gethostbyname()
). If CoreDNS isn't running, you can enable it referring to the
CoreDNS README
or
. Let's run another curl application to test this:
kubectl run curl --image
=
radial/busyboxplus:curl -i --tty
Waiting for pod default/curl-131556218-9fnch to be running, status is Pending, pod ready: false
Hit enter for command prompt
Then, hit enter and run
nslookup my-nginx
:
[
root@curl-131556218-9fnch:/
]
$ nslookup my-nginx
Name:      my-nginx
Address 1: 10.0.162.149
Securing the Service
Till now we have only accessed the nginx server from within the cluster. Before exposing the Service to the internet, you want to make sure the communication channel is secure. For this, you will need:
Self signed certificates for https (unless you already have an identity certificate)
An nginx server configured to use the certificates
A
secret
that makes the certificates accessible to pods
You can acquire all these from the
nginx https example
. This requires having go and make tools installed. If you don't want to install those, then follow the manual steps later. In short:
make keys
KEY
=
/tmp/nginx.key
CERT
=
/tmp/nginx.crt
kubectl create secret tls nginxsecret --key /tmp/nginx.key --cert /tmp/nginx.crt
secret/nginxsecret created
kubectl get secrets
NAME                  TYPE                                  DATA      AGE
nginxsecret           kubernetes.io/tls                     2         1m
And also the configmap:
kubectl create configmap nginxconfigmap --from-file
=
default.conf
configmap/nginxconfigmap created
kubectl get configmaps
NAME             DATA   AGE
nginxconfigmap   1      114s
Following are the manual steps to follow in case you run into problems running make (on windows for example):
# Create a public private key pair
openssl req -x509 -nodes -days
365
-newkey rsa:2048 -keyout /d/tmp/nginx.key 

xcGFWb2RjL0d5aGVycjFDZzE2MmJaSjJDV2RsZkI0VEdtUjZZdmxTZEFOOFRwUWhFbUtKCnFBLzVzdHdxNWd0WGVLOVJmMWxXK29xNThRNTBxMmk1NVdUTThoSDZhTjlaMTltZ0FGdE5VdGNqQUx2dFYxdEYKWSs4WFJkSHJaRnBIWll2NWkwVW1VbGc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K"
Now create the secrets using the file:
kubectl apply -f nginxsecrets.yaml
Now modify your nginx replicas to start an https server using the certificate in the secret, and the Service, to expose both ports (80 and 443):
service/networking/nginx-secure-app.yaml
:
v1
kind
:
Service
metadata
:
name
:
my-nginx
labels
:
run
:
my-nginx
spec
:
type
:
NodePort
ports
:
-
port
:
8080
:
80
protocol
:
TCP
name
:
http
-
port
:
443
protocol
:
TCP
name
:
https
selector
:
run
:
my-nginx
---
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
selector
:
:
run
:
my-nginx
replicas
:
1
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
volumes
:
-
name
:
secret
:
:
nginxsecret
-
name
:
configmap-volume
configMap
:
name
:
nginxconfigmap
:
-
name
:
nginxhttps
image
:
bprashanth/nginxhttps:1.0
ports
:
-
:
443
-
:
80
:
-
mountPath
:
/etc/nginx/ssl
name
:
-
mountPath
:
/etc/nginx/conf.d
name
:
Noteworthy points about the nginx-secure-app manifest:
It contains both Deployment and Service specification in the same file.
The
nginx server
serves HTTP traffic on port 80 and HTTPS traffic on 443, and nginx Service
exposes both ports.
Each container has access to the keys through a volume mounted at
.
This is set up
before
the nginx server is started.
kubectl delete deployments,svc my-nginx; kubectl create -f ./nginx-secure-app.yaml
At this point you can reach the nginx server from any node.
run
=
=
POD_IP
[
map
[
ip:10.244.3.5
]]
node $ curl -k https://10.244.3.5
...
&lt;h1&gt;Welcome to nginx!&lt;/h1&gt;
Note how we supplied the
-k
parameter to curl in the last step, this is because we don't know anything about the pods running nginx at certificate generation time,
so we have to tell curl to ignore the CName mismatch. By creating a Service we linked the CName used in the certificate with the actual DNS name used by pods during Service lookup.
Let's test this from a pod (the same secret is being reused for simplicity, the pod only needs nginx.crt to access the Service):
service/networking/curlpod.yaml
:
apps/v1
kind
:
metadata
:
name
:
curl-deployment
spec
:
selector
:
:
app
:
curlpod
replicas
:
1
template
:
metadata
:
labels
:
app
:
curlpod
spec
:
volumes
:
-
name
:
secret
:
:
:
-
name
:
curlpod
command
:
- sh
- -c
- while true; do sleep 1; done
image
:
radial/busyboxplus:curl
:
-
mountPath
:
name
:
kubectl apply -f ./curlpod.yaml
app
=
curlpod
NAME                               READY     STATUS    RESTARTS   AGE
curl-deployment-1515033274-1410r   1/1       Running   0          1m
kubectl
exec
curl-deployment-1515033274-1410r -- curl https://my-nginx --cacert /etc/nginx/ssl/tls.crt
...
&lt;title&gt;Welcome to nginx!&lt;/title&gt;
...
Exposing the Service
For some parts of your applications you may want to expose a Service onto an
external IP address. Kubernetes supports two ways of doing this: NodePorts and
LoadBalancers. The Service created in the last section already used
NodePort
,
so your nginx HTTPS replica is ready to serve traffic on the internet if your
node has a public IP.
kubectl get svc my-nginx -o yaml | grep nodePort -C
5
uid: 07191fb3-f61a-11e5-8ae5-42010af00002
spec:
clusterIP: 10.0.162.149
ports:
- name: http
nodePort:
31704
port:
8080
targetPort:
80
- na

me: https
nodePort:
32453
port:
443
443
selector:
kubectl get nodes -o yaml | grep ExternalIP -C
1
- address: 104.197.41.11
type: ExternalIP
allocatable:
--
- address: 23.251.152.56
...
$ curl https://&lt;EXTERNAL-IP&gt;:&lt;NODE-PORT&gt; -k
...
Let's now recreate the Service to use a cloud load balancer. Change the
Type
of
my-nginx
Service from
NodePort
to
:
kubectl edit svc my-nginx
NAME       TYPE           CLUSTER-IP     EXTERNAL-IP        PORT(S)               AGE
my-nginx   LoadBalancer   10.0.162.149     xx.xxx.xxx.xxx     8080:30163/TCP        21s
curl https://&lt;EXTERNAL-IP&gt; -k
...
The IP address in the
EXTERNAL-IP
column is the one that is available on the public internet. The
CLUSTER-IP
is only available inside your
cluster/private cloud network.
Note that on AWS, type
creates an ELB, which uses a (long)
hostname, not an IP. It's too long to fit in the standard
kubectl get svc
output, in fact, so you'll need to do
kubectl describe service my-nginx
to
see it. You'll see something like this:
...
LoadBalancer Ingress:   a320587ffd19711e5a37606cf4a74574-1142138393.us-east-1.elb.amazonaws.com
...
Using a Service to Access an Application in a Cluster
Connecting a Front End to a Back End Using a Service
Creating an External Load Balancer
Feedback
Yes
No
.
or
.
Last modified December 08, 2021 at 6:50 PM PST:
Move "Connecting Applications with Services" to tutorials section (ce46f1ca74)
DNS
Home
Blog
Training
Partners
Community
CC BY 4.0

Authentication Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

(v1)
Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Labels and Selectors | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Debug Pods
Determine the Reason for Pod F

ailure
Auditing
Run Jobs
TLS
Tutorials
Security

Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-ap

iserver Configuration (v1alpha1)
Concepts
Overview
Labels
:
{
"labels"
: {
"key1"
:
"value1"
,
"key2"
:
"value2"
}
}
.
Service deployments and batch processing pipelines are often multi-dimensional entities (e.g., multiple partitions or deployments, multiple release tracks, multiple tiers, multiple micro-services per tier). Management often requires cross-cutting operations, which breaks encapsulation of strictly hierarchical representations, especially rigid hierarchies determined by the infrastructure rather than by users.
,
,
,
,
,
,
,
Labels
/
-
_
), dots (
.
.
/
).
,
kube-co

,
,
kubectl
The
and
k8s.io/
reserved
),
-
_
), dots (
.
and
app: nginx
:
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Unlike
Via a
and
set-based
.
AND
(
them.
Note:
Caution:
OR
(
||
Equality-
or
=
,
==
,
!=
equality
.
tier
frontend
tier
key.
excluding
frontend
the label "
".
:
v1
kind
:
Pod
metadata
:
name
:
cuda-test
spec
:
:
-
name
:
cuda-test
image
:
resources
:
limits
:
:
1
:
:
Set-based
Set-based
in
,
notin
and
exists
partition
or
qa
.
tier
frontend
and
backend
tier
key.
partition
partition
AND
partition
env

ironment
qa
.
The
set-based
!=
and
notin
.
Set-based
.
API
set-based
apiserver
with
kubectl
and using
=
=
frontend
or using
set-based
set-based
OR
exists
operator:
services
and
,
pods
.
service
json
or
yaml
:
{
:
"redis"
,
}
or
selector
:
component
:
redis
json
or
yaml
or
.
Job
,
,
, and
DaemonSet
,
support
set-based
selector
:
:
component
:
redis
:
- {
:
[cache]}
- {
:
[dev]}
is a map of
in the
, whose
key
operator
values
and
Find
See
Feedback
Yes
No
.
or
.
Last modified November 11, 2022 at 7:07 PM PST:
Updating labels.md (fab0fc76a5)
Set-based
A

PI
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up ku

bectl on Windows
Debug Pods
Resour

ce metrics pipeline
Auditing
Run Jobs
TLS
Tutorials
Security
Restr

ict a Container's Syscalls with seccomp
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Requirements:
You need a machine that is running Linux or macOS.
You need to have these tools installed:
Python
v3.7.x
Git
Golang
version 1.13+
Pip
used to install PyYAML
PyYAML
v5.1.2
make
gcc compiler/linker
Docker
(Required only for
kubectl
command reference)
Your
PATH
environment variable must include the required build tools, such as the
Go
binary and
python
.
You need to know how to create a pull request to a GitHub repository.
This involves creating your own fork of the repository. For more
information, see
Work from a local clone
.
Feedback
Yes
No
.
or
.
Last modified July 17, 2020 at 12:07 PM PST:
Tidy up: fix bad links in contribution guide (139da7d731)
Home
Blog
Training
Partners
Community
CC BY 4.0

Namespaces Walkthrough | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

ubectl on macOS
Debug Pods
Get a Shell to a Running Cont

ainer
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Ac

cess to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
A scope for
Names
.
minikube
.
You have an
.
Pods
,
Services
, and
.
and
.
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
name
:
developm

ent
Create the
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
name
:
:
v1
clusters
:
-
cluster
:
:
REDACTED
server
:
name
:
contexts
:
-
context
:
cluster
:
user
:
name
:
:
kind
:
Config
:
{}
users
:
-
name
:
user
:
:
REDACTED
:
REDACTED
token
:
-
name
:
user
:
password
:
username
:
admin
=
\
--cluster
=
\
--user
=
=
\
--cluster
=
\
--user
=
:
v1
clusters
:
-
cluster
:
:
REDACTED
server
:
name
:
contexts
:
-
context
:
cluster
:
user
:
name
:
-
context
:
cluster
:
namespace
:
user
:
name
:
dev
-
context
:
cluster
:
namespace
:
user
:
name
:
prod
:
kind
:
Config
:
{}
users
:
-
name
:
user
:
:
REDACTED
:
REDACTED
token
:
-
name
:
user
:
password
:
username
:
admin
dev
At this point, all requests we make to the Kubernetes cluster from the command line are scoped to the
:
apps/v1
kind
:
metadata
:
labels
:
app
:
snowflake
name
:
snowflake
spec
:
replicas
:
2
selector
:
:
app
:
snowflake
template
:
metadata
:
labels
:
app
:
snowflake
spec
:
:
-
image
:
registry.k8s.io/serve_hos

tname
:
Always
name
:
snowflake
snowflake
app
=
snowflake
The
=
=
5
app
=
cattle
Feedback
Yes
No
.
or
.
Last modified October 26, 2022 at 10:11 AM PST:
updated json manifest to yaml (18ae4051f7)
Home
Blog
Training
Partners
Community
CC BY 4.0

Resource metrics pipeline | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
The
(HPA) and
(VPA)
command.
Note:
.
S[
]
Server]
end
L[API
server]
W[HPA]
end
top]
enabled
cAdvisor
kubelet
and
/stats
kubelet
/stats
endpoint.
the
command. Metrics Server is a reference

 implementation of the Metrics API.
: Kubernetes API supporting access to CPU and memory used for
Note:
minikube
jq
for easier
reading:
| jq
'.'
curl
:
{
"kind"
:
,
:
,
: {
"name"
:
,
:
,
:
},
:
,
"window"
:
"30s"
,
"usage"
: {
"cpu"
:
,
"memory"
:
}
}
jq
| jq
'.'
curl
:
{
"kind"
:
,
:
,
: {
"name"
:
,
:
,
:
,
:
},
:
,
"window"
:
"30s"
,
: [
{
"name"
:
,
"usage"
: {
"cpu"
:
,
"memory"
:
"22244Ki"
}
}
]
}
for the
API.
,
the
and the
.
Note:
CPU
.
Memory

.
The metrics-server fetches resource metrics from the kubelets and exposes them in the Kubernetes
using the
command.
kubelet
API
.
.
Feedback
Yes
No
.
or
.
CPU
Memory
Home
Blog
Training
Partners
Community
CC BY 4.0

Suggesting content improvements | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

If you notice an issue with Kubernetes documentation or have an idea for new content, then open an issue. All you need is a
GitHub account
and a web browser.
In most cases, new work on Kubernetes documentation begins with an issue in GitHub. Kubernetes contributors
then review, categorize and tag issues as needed. Next, you or another member
of the Kubernetes community open a pull request with changes to resolve the issue.
Opening an issue
If you want to suggest improvements to existing content or notice an error, then open an issue.
Click the
link on the right sidebar. This redirects you
to a GitHub issue page pre-populated with some headers.
Describe the issue or suggestion for improvement. Provide as many details as you can.
Click
Submit new issue
.
After submitting, check in on your issue occasionally or turn on GitHub notifications.
Reviewers and other community members might ask questions before
they can take action on your issue.
Suggesting new content
If you have an idea for new content, but you aren't sure where it should go, you can
still file an issue. Either:
Choose an existing page in the section you think the content belongs in and click
.
Go to
GitHub
and file the issue directly.
How to file great issues
Keep the following in mind when filing an issue:
Provide a clear issue description. Describe what specifically is missing, out of date,
wrong, or needs improvement.
Explain the specific impact the issue has on users.
Limit the scope of a given issue to a reasonable unit of work. For problems
with a large scope, break them down into smaller issues. For example, "Fix the security docs"
is too broad, but "Add details to the 'Restricting network access' topic" is specific enough
to be actionable.
Search the existing issues to see if there's anything related or similar to the
new issue.
If the new issue relates to another issue or pull request, refer to it
either by its full URL or by the issue or pull request number prefixed
with a
#
character. For example,
Introduced by #987654
.
Follow the
Code of Conduct
. Respect your
fellow contributors. For example, "The docs are terrible" is not
helpful or polite feedback.
Feedback
Yes
No
.
or
.
Last modified June 18, 202

2 at 2:35 PM PST:
Tweak localization page (700433da48)
Home
Blog
Training
Partners
Community
CC BY 4.0

Specifying a Disruption Budget for your Application | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Debug Pods
Determine the Reason for Pod Fa

ilure
Auditing
Run Jobs
TLS
Tutorials
Security
A

pply Pod Security Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-api

server Configuration (v1alpha1)
Tasks
This page shows how to limit the number of concurrent disruptions
that your application experiences, allowing for higher availability
while permitting the cluster administrator to manage the clusters
nodes.
.
You are the owner of an application running on a Kubernetes cluster that requires
high availability.
You should know how to deploy
Replicated Stateless Applications
and/or
Replicated Stateful Applications
.
You should have read about
Pod Disruptions
.
You should confirm with your cluster owner or service provider that they respect
Pod Disruption Budgets.
Protecting an Application with a PodDisruptionBudget
Identify what application you want to protect with a PodDisruptionBudget (PDB).
Think about how your application reacts to disruptions.
Create a PDB definition as a YAML file.
Create the PDB object from the YAML file.
Identify an Application to Protect
The most common use case when you want to protect an application
specified by one of the built-in Kubernetes controllers:
In this case, make a note of the controller's
.spec.selector
; the same
selector goes into the PDBs
.
From version 1.15 PDBs support custom controllers where the
scale subresource
You can also use PDBs with pods which are not controlled by one of the above
controllers, or arbitrary groups of pods, but there are some restrictions,
described in
Arbitrary Controllers and Selectors
.
Think about how your application reacts to disruptions
Decide how many instances can be down at the same time for a short period
due to a voluntary disruption.
Stateless frontends:
Concern: don't reduce serving capacity by more than 10%.
Solution: use PDB with minAvailable 90% for example.
Single-instance Stateful Application:
Concern: do not terminate this application without talking to me.
Possible Solution 1: Do not use a PDB and tolerate occasional downtime.
Possible Solution 2: Set PDB with maxUnavailable=0. Have an understanding
(outside of Kubernetes) that the cluster operator needs to consult you before
termination. When the cluster operator contacts you, prepare for downtime,
and then delete the PDB to indicate readiness for disruption. Recreate afterwards.
Multiple-i

nstance Stateful application such as Consul, ZooKeeper, or etcd:
Concern: Do not reduce number of instances below quorum, otherwise writes fail.
Possible Solution 1: set maxUnavailable to 1 (works with varying scale of application).
Possible Solution 2: set minAvailable to quorum-size (e.g. 3 when scale is 5). (Allows more disruptions at once).
Restartable Batch Job:
Concern: Job needs to complete in case of voluntary disruption.
Possible solution: Do not create a PDB. The Job controller will create a replacement pod.
Rounding logic when specifying percentages
Values for
or
can be expressed as integers or as a percentage.
When you specify an integer, it represents a number of Pods. For instance, if you set
to 10, then 10
Pods must always be available, even during a disruption.
When you specify a percentage by setting the value to a string representation of a percentage (eg.
"50%"
), it represents a percentage of
total Pods. For instance, if you set
to
"50%"
, then only 50% of the Pods can be unavailable during a
disruption.
When you specify the value as a percentage, it may not map to an exact number of Pods. For example, if you have 7 Pods and
you set
to
"50%"
, it's not immediately obvious whether that means 3 Pods or 4 Pods must be available.
Kubernetes rounds up to the nearest integer, so in this case, 4 Pods must be available. You can examine the
code
that controls this behavior.
Specifying a PodDisruptionBudget
A
has three fields:
A label selector
to specify the set of
pods to which it applies. This field is required.
.spec.minAvailable
which is a description of the number of pods from that
set that must still be available after the eviction, even in the absence
of the evicted pod.
can be either an absolute number or a percentage.
.spec.maxUnavailable
(available in Kubernetes 1.7 and higher) which is a description
of the number of pods from that set that can be unavailable after the eviction.
It can be either an absolute number or a percentage.
Note:
The behavior for an empty selector differs between the policy/v1beta1 and policy/v1 APIs for
PodDisruptionBudgets. For policy/v1beta1 an empty selector matches zero pods, while
for policy/v1 an empty selector matches every pod in the namespace.
You can specify only one of
and
in a single
.
can only be used to control the eviction of pods
that have an associated controller managing them. In the examples below, "desired replicas"
is the
scale
of the controller managing the pods being selected by the
.
Example 1: With a
of 5, evictions are allowed as long as they leave behind
5 or more
healthy
pods among those selected by the PodDisruptionBudget's
selector
.
Example 2: With a
of 30%, evictions are allowed as long as at least 30%
of the number of desired replicas are healthy.
Example 3: With a
of 5, evictions are allowed as long as there are at most 5
unhealthy replicas among the total number of desired replicas.
Example 4: With a
of 30%, evictions are allowed as long as no more than 30%
of the desired replicas are unhealthy.
In typical usage, a single budget would be used for a collection of pods managed by
a controller—for example, the pods in a single ReplicaSet or StatefulSet.
Note:
A disruption budget does not truly guarantee that the specified
number/percentage of pods will always be up. For example, a node that hosts a
pod from the collection may fail when the collection is at the minimum size
specified in the budget, thus bringing the number of available pods from the
collection below the specified size. The budget can only protect against
voluntary evictions, not all causes of unavailability.
If you set
to 0% or 0, or you set
to 100% or the number of replicas,
you are requiring zero voluntar

y evictions. When you set zero voluntary evictions for a workload
object such as ReplicaSet, then you cannot successfully drain a Node running one of those Pods.
If you try to drain a Node where an unevictable Pod is running, the drain never completes. This is permitted as per the
semantics of
.
You can find examples of pod disruption budgets defined below. They match pods with the label
app: zookeeper
.
Example PDB Using minAvailable:
policy/zookeeper-pod-disruption-budget-minavailable.yaml
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
:
2
selector
:
:
app
:
zookeeper
Example PDB Using maxUnavailable:
policy/zookeeper-pod-disruption-budget-maxunavailable.yaml
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
:
1
selector
:
:
app
:
zookeeper
For example, if the above
zk-pdb
object selects the pods of a StatefulSet of size 3, both
specifications have the exact same meaning. The use of
is recommended as it
automatically responds to changes in the number of replicas of the corresponding controller.
Create the PDB object
You can create or update the PDB object using kubectl.
kubectl apply -f mypdb.yaml
Check the status of the PDB
Use kubectl to check that your PDB is created.
Assuming you don't actually have pods matching
in your namespace,
then you'll see something like this:
kubectl get poddisruptionbudgets
NAME     MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
zk-pdb   2               N/A               0                     7s
If there are matching pods (say, 3), then you would see something like this:
zk-pdb   2               N/A               1                     7s
The non-zero value for
ALLOWED DISRUPTIONS
means that the disruption controller has seen the pods,
counted the matching pods, and updated the status of the PDB.
You can get more information about the status of a PDB with this command:
kubectl get poddisruptionbudgets zk-pdb -o yaml
:
policy/v1
kind
:
metadata
:
:
…
:
"2020-03-04T04:22:56Z"
:
1
name
:
zk-pdb
…
status
:
:
3
:
2
:
1
:
3
:
1
Healthiness of a Pod
The current implementation considers healthy pods, as pods that have
.status.conditions
item with
type="Ready"
and
status="True"
.
These pods are tracked via
.status.currentHealthy
field in the PDB status.
Note:
In order to use this behavior, you must enable the
on the
API server
.
PodDisruptionBudget guarding an application ensures that
number of pods
does not fall below the number specified in
.status.desiredHealthy
by disallowing eviction of healthy pods.
By using
.spec.unhealthyPodEvictionPolicy
, you can also define the criteria when unhealthy pods
should be considered for eviction. The default behavior when no policy is specified corresponds
to the
IfHealthyBudget
policy.
Policies:
Running pods (
.status.phase="Running"
), but not yet healthy can be evicted only if the guarded application is not
disrupted (
is at least equal to
).
This policy ensures that running pods of an already disrupted application have the best chance to become healthy.
This has negative implications for draining nodes, which can be blocked by misbehaving applications that are guarded by a PDB.
More specifically applications with pods in
CrashLoopBackOff
state (due to a bug or misconfiguration),
or pods that are just failing to report the
Ready
condition.
), but not yet healthy are considered disrupted and

 can be evicted
regardless of whether the criteria in a PDB is met.
This means prospective running pods of a disrupted application might not get a chance to become healthy.
By using this policy, cluster managers can easily evict misbehaving applications that are guarded by a PDB.
Ready
Note:
Pods in
Pending
,
Succeeded
or
Failed
phase are always considered for eviction.
You can skip this section if you only use PDBs with the built-in
application controllers (Deployment, ReplicationController, ReplicaSet, and StatefulSet),
with the PDB selector matching the controller's selector.
You can use a PDB with pods controlled by another type of controller, by an
"operator", or bare pods, but with these restrictions:
only
can be used, not
.
only an integer value can be used with
, not a percentage.
You can use a selector which selects a subset or superset of the pods belonging to a built-in
controller. The eviction API will disallow eviction of any pod covered by multiple PDBs,
so most users will want to avoid overlapping selectors. One reasonable use of overlapping
PDBs is when pods are being transitioned from one PDB to another.
Feedback
Yes
No
.
or
.
Last modified November 22, 2022 at 2:39 PM PST:
Add documentation for Unhealthy Pod Eviction Policy for PDBs (b61f763cf0)
Home
Blog
Training
Partners
Community
CC BY 4.0

Romana for NetworkPolicy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

Tasks
.
Follow the
.
.
Feedback
Yes
No
.
or
.
Last modified October 22, 2020 at 3:01 PM PST:
Fix links in the tasks section (740eb340d4)
Home
Blog
Training
Partners
Community
CC BY 4.0

Manage HugePages | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Tasks
Configure and manage huge pages as a schedulable resource in a cluster.
Kubernetes supports the allocation and consumption of pre-allocated huge pages
by applications in a Pod. This page describes how users can consume huge pages.
Kubernetes nodes must pre-allocate huge pages in order for the node to report
its huge page capacity. A node can pre-allocate huge pages for multiple
sizes.
The nodes will automatically discover and report all huge page resources as
schedulable resources.
API
Huge pages can be consumed via container level resource requirements using the
resource name
, where
is the most compact binary
notation using integer values supported on a particular node. For example, if a
node supports 2048KiB and 1048576KiB page sizes, it will expose a schedulable
resources
and
. Unlike CPU or memory, huge pages
do not support overcommit. Note that when requesting hugepage resources, either
memory or CPU resources must be requested as well.
A pod may consume multiple huge page sizes in a single pod spec. In this case it
must use
medium: HugePages-&lt;hugepagesize&gt;
notation for all volume mounts.
:
v1
kind
:
Pod
metadata
:
name
:
huge-pages-example
spec
:
:
-
name
:
example
image
:
fedora:latest
command
:
- sleep
- inf
:
-
mountPath
:
/hugepages-2Mi
name
:
hugepage-2mi
-
mountPath
:
/hugepages-1Gi
name
:
hugepage-1gi
resources
:
limits
:
:
100Mi
:
2Gi
memory
:
100Mi
requests
:
memory
:
100Mi
volumes
:
-
name
:
emptyDir
:
medium
:
HugePages-2Mi
-
name
:
emptyDir
:
medium
:
HugePages-1Gi
A pod may use
medium: HugePages
only if it requests huge pages of one size.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
example
image
:
command
:
- sleep
- inf
:
-
mountPath
:
/hugepages
name
:
hugepage
resources
:
limits
:
:
100Mi
memory
:
100Mi
requests
:
memory
:
100Mi
volumes
:
-
name
:
hugepage
emptyDir
:
medium
:
HugePages
Huge page requests must equal the limits. This is the default if limits are
specified, but requests are not.
Huge pages are isolated at a container scope, so each container has own
limit on their cgroup sandbox as requested in a container spec.
EmptyDir volumes backed by huge pages may not consume more huge page memory
than the pod request.
Applications that consume huge pages via
shmget()
with
SHM_HUGETLB
mu

st
run with a supplemental group that matches
proc/sys/vm/hugetlb_shm_group
.
Huge page usage in a namespace is controllable via ResourceQuota similar
to other compute resources like
cpu
or
memory
using the
token.
Feedback
Yes
No
.
or
.
Last modified April 13, 2021 at 11:11 AM PST:
Graduate multiple sizes huge pages to GA (7a0f9bc01e)
API
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm config | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and S

et Up kubectl on macOS
Debug Pods
Get a Shell to a Runni

ng Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Contain

er's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configurati

on (v1)
Reference
Kubeadm
During
in the
,
and
.
You can use
uses for
and
.
Note:
init
and
join
or
.
.
You can use
and
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Options inherit

ed from parent commands
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help

Synopsis
Options
-h, --help
Feedback
Yes
No
.
or
.
Last modified October 27, 2021 at 12:32 AM PST:
kubeadm-config.md: update the page to reflect current state (586b359b05)
Home
Blog
Training
Partners
Community
CC BY 4.0

Roles and responsibilities | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 kubectl on Linux
Debug Pods
Debug Runn

ing Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Nam

espace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-a

piserver Encryption Configuration (v1)
Anyone can contribute to Kubernetes. As your contributions to SIG Docs grow,
you can apply for different levels of membership in the community.
These roles allow you to take on more responsibility within the community.
Each role requires more time and commitment. The roles are:
Anyone: regular contributors to the Kubernetes documentation
Members: can assign and triage issues and provide non-binding review on pull requests
Reviewers: can lead reviews on documentation pull requests and can vouch for a change's quality
Approvers: can lead reviews on documentation and merge changes
Anyone
Anyone with a GitHub account can contribute to Kubernetes. SIG Docs welcomes all new contributors!
Anyone can:
Open an issue in any
repository, including
kubernetes/website
Give non-binding feedback on a pull request
Contribute to a localization
Suggest improvements on
Slack
or the
SIG docs mailing list
.
After
signing the CLA
, anyone can also:
Open a pull request to improve existing content, add new content, or write a blog post or case study
Create diagrams, graphics assets, and embeddable screencasts and videos
contributing new content
.
Members
A member is someone who has submitted multiple pull requests to
. Members are a part of the
Kubernetes GitHub organization
.
Members can:
Do everything listed under
Anyone
Use the
/lgtm
comment to add the LGTM (looks good to me) label to a pull request
Note:
Using
/lgtm
triggers automation. If you want to provide non-binding
approval, commenting "LGTM" works too!
Use the
/hold
comment to block merging for a pull request
Use the
/assign
comment to assign a reviewer to a pull request
Provide non-binding review on pull requests
Use automation to triage and categorize issues
Document new features
Becoming a member
After submitting at least 5 substantial pull requests and meeting the other
:
Find two
reviewers
or
approvers
to
sponsor
your
membership.
Ask for sponsorship in the
#sig-docs channel on Slack
or on the
SIG Docs mailing list
.
Note:
Don't send a direct email or Slack direct message to an individual
SIG Docs member. You must request sponsorship before submitting your application.
Open a GitHub issue in the
kubernetes/org
repository. Use the
Organization Membership Request
issue template.
Let your sponsors know about the GitHub issue. You can either:
Mention their GitHub username in an issue (
@&lt;GitHub-username&gt;
)
Send them the issue link u

sing Slack or email.
Sponsors will approve your request with a
+1
vote. Once your sponsors
approve the request, a Kubernetes GitHub admin adds you as a member.
Congratulations!
If your membership request is not accepted you will receive feedback.
After addressing the feedback, apply again.
Accept the invitation to the Kubernetes GitHub organization in your email account.
Note:
GitHub sends the invitation to the default email address in your account.
Reviewers
Reviewers are responsible for reviewing open pull requests. Unlike member
feedback, the PR author must address reviewer feedback. Reviewers are members of the
@kubernetes/sig-docs-{language}-reviews
GitHub team.
Reviewers can:
Anyone
and
Members
Review pull requests and provide binding feedback
Note:
To provide non-binding feedback, prefix your comments with a phrase like "Optionally: ".
Edit user-facing strings in code
Improve code comments
You can be a SIG Docs reviewer, or a reviewer for docs in a specific subject area.
Assigning reviewers to pull requests
Automation assigns reviewers to all pull requests. You can request a
review from a specific person by commenting:
/assign [@_github_handle]
.
If the assigned reviewer has not commented on the PR, another reviewer can
step in. You can also assign technical reviewers as needed.
Using
/lgtm
LGTM stands for "Looks good to me" and indicates that a pull request is
technically accurate and ready to merge. All PRs need a
/lgtm
comment from a
reviewer and a
/approve
comment from an approver to merge.
A
/lgtm
comment from reviewer is binding and triggers automation that adds the
lgtm
label.
Becoming a reviewer
When you meet the
,
you can become a SIG Docs reviewer. Reviewers in other SIGs must apply
separately for reviewer status in SIG Docs.
To apply:
Open a pull request that adds your GitHub username to a section of the
OWNERS_ALIASES
file
in the
Note:
If you aren't sure where to add yourself, add yourself to
sig-docs-en-reviews
.
Assign the PR to one or more SIG-Docs approvers (usernames listed under
sig-docs-{language}-owners
).
If approved, a SIG Docs lead adds you to the appropriate GitHub team. Once added,
K8s-ci-robot
assigns and suggests you as a reviewer on new pull requests.
Approvers
Approvers review and approve pull requests for merging. Approvers are members of the
@kubernetes/sig-docs-{language}-owners
GitHub teams.
Approvers can do the following:
Everything listed under
Anyone
,
Members
and
Reviewers
Publish contributor content by approving and merging pull requests using the
/approve
comment
Propose improvements to the style guide
Propose improvements to docs tests
Propose improvements to the Kubernetes website or other tooling
If the PR already has a
/lgtm
, or if the approver also comments with
/lgtm
, the PR merges automatically. A SIG Docs approver should only leave a
/lgtm
on a change that doesn't need additional technical review.
Approving pull requests
Approvers and SIG Docs leads are the only ones who can merge pull requests
into the website repository. This comes with certain responsibilities.
Approvers can use the
/approve
command, which merges PRs into the repo.
Warning:
A careless merge can break the site, so be sure that when you merge something, you mean it.
Make sure that proposed changes meet the
documentation content guide
.
If you ever have a question, or you're not sure about something, feel free
to call for additional review.
Verify that Netlify tests pass before you
/approve
a PR.
Visit the Netlify page preview for a PR to make sure things look good before approving.
Participate in the
PR Wrangler rotation schedule
for weekly rotations. SIG Docs expects all approvers to participate in this
rotation. See
.
Becoming an approver
,
you can become a SIG Docs approver. Approvers in other SIGs must apply
separately for approver status in SIG Docs.
To apply:
Open a p

ull request adding yourself to a section of the
file in the
Note:
sig-docs-en-owners
.
Assign the PR to one or more current SIG Docs approvers.
@k8s-ci-robot
Read about
PR wrangling
, a role all approvers take on rotation.
Feedback
Yes
No
.
or
.
Last modified September 28, 2022 at 12:20 PM PST:
Reword in sig-docs contributor roles (35621ef457)
Anyone
Members
Reviewers
Using
/lgtm
Approvers
Home
Blog
Training
Partners
Community
CC BY 4.0

Explore Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Viewing Pods and Nodes
2
Pod
Summary:
Pods
Nodes
Nodes
Node
Kubelet, a process responsible for communication between the Kubernetes control plane and the Node; it manages the Pods and the containers running on a machine.
In Module
2
Now that we know more about our cluster components and the command line, let's explore our a

pplication.
›
2 - Interactive Tutorial - Exploring Your App
Home
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes Issues and Security | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
1 - Kubernetes Issue Tracker
.
.
Official
2 - Kubernetes Security and Disclosure Information
Join the
.
.
3 - Official CVE Feed
JSON Feed
of
published

 security issues. You can access it by executing the following command:
CVE ID
#113757
#113756
#112513
#112192
#104980
#102106
#101435
#100096
#97076
#95624
#95623
#95622
#95621
#93032
#92914
#92315
#91542
#91507
#89535
#89378
#89377
#87773
#85867
#85233
#83253
#81114
#81023
#80984
#80983
#78308
#76797
#76676
#74534
#71411
#65750
#61297
#60814
#60813
#47611
#43459
Bucket which is writable only by a small number of trusted memb

ers of the
Home
Blog
Training
Partners
Community
CC BY 4.0

Use a User Namespace With a Pod | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
host.
HIGH
or
CRITICAL
minikube
.
new
pod.spec
hostUsers
field of
.spec
to
false
:
v1
kind
:
Pod
metadata
:
name
:
userns
spec
:
hostUsers
:
false
:
-
name
:
shell
comm

and
:
[
"sleep"
,
]
image
:
debian
:
replacing
$pid
Feedback
Yes
No
.
or
.
Last modified September 25, 2022 at 12:08 AM PST:
Fix Markdown Error in user-namespaces.md (09fafef4f4)
Home
Blog
Training
Partners
Community
CC BY 4.0

Page content types | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

The Kubernetes documentation follows several types of page content:
Concept
Task
Tutorial
Reference
Content sections
Each page content type contains a number of sections defined by
Markdown comments and HTML headings. You can add content headings to
your page with the
heading
shortcode. The comments and headings help
maintain the structure of the page content types.
Examples of Markdown comments defining page content sections:
&lt;!-- overview --&gt;
&lt;!-- body --&gt;
To create common headings in your content pages, use the
heading
shortcode with
a heading string.
Examples of heading strings:
whatsnext
prerequisites
objectives
cleanup
synopsis
seealso
options
For example, to create a
whatsnext
heading, add the heading shortcode with the "whatsnext" string:
## {{% heading "whatsnext" %}}
You can declare a
heading as follows:
## {{% heading "prerequisites" %}}
The
heading
shortcode expects one string parameter.
The heading string parameter matches the prefix of a variable in the
i18n/&lt;lang&gt;.toml
files.
i18n/en.toml
:
[whatsnext_heading]
other =
"What's next"
i18n/ko.toml
:
other =
"다음 내용"
Content types
Each content type informally defines its expected page structure.
Create page content with the suggested page sections.
Concept
A concept page explains some aspect of Kubernetes. For example, a concept
page might describe the Kubernetes Deployment object and explain the role it
plays as an application once it is deployed, scaled, and updated. Typically, concept
pages don't include sequences of steps, but instead provide links to tasks or
tutorials.
To write a new concept page, create a Markdown file in a subdirectory of the
/content/en/docs/concepts
directory, with the following characteristics:
Concept pages are divided into three sections:
Page section
overview
body
whatsnext
The
overview
and
body
sections appear as comments in the concept page.
You can add the
whatsnext
section to your page with the
heading
shortcode.
Fill each section with content. Follow these guidelines:
Organize content with H2 and H3 headings.
For
overview
, set the topic's context with a single paragraph.
For
body
, explain the concept.
For
whatsnext
, provide a bulleted list of topics (5 maximum) to learn more about the concept.
is a published example of a concept page.
Task
A task page shows how to do a single thing, typically by giving a short
sequence of steps. Task pages have minimal explanation, but often provide links
to conceptual topics that provide rela

ted background and knowledge.
To write a new task page, create a Markdown file in a subdirectory of the
/content/en/docs/tasks
overview
steps
discussion
whatsnext
The
overview
,
steps
, and
sections appear as comments in the task page.
and
whatsnext
sections to your page
with the
heading
Within each section, write your content. Use the following guidelines:
Use a minimum of H2 headings (with two leading
#
characters). The sections
themselves are titled automatically by the template.
For
overview
, use a paragraph to set context for the entire topic.
For
, use bullet lists when possible. Start adding additional
prerequisites below the
include
. The default prerequisites include a running Kubernetes cluster.
For
steps
, use numbered lists.
For discussion, use normal content to expand upon the information covered
in
steps
.
For
whatsnext
, give a bullet list of up to 5 topics the reader might be
interested in reading next.
An example of a published task topic is
Using an HTTP proxy to access the Kubernetes API
.
Tutorial
A tutorial page shows how to accomplish a goal that is larger than a single
task. Typically a tutorial page has several sections, each of which has a
sequence of steps. For example, a tutorial might provide a walkthrough of a
code sample that illustrates a certain feature of Kubernetes. Tutorials can
include surface-level explanations, but should link to related concept topics
for deep explanations.
To write a new tutorial page, create a Markdown file in a subdirectory of the
/content/en/docs/tutorials
overview
lessoncontent
cleanup
whatsnext
The
overview
,
, and
sections appear as comments in the tutorial page.
,
cleanup
, and
whatsnext
with the
heading
#
For
overview
For
, use bullet lists when possible. Add additional
prerequisites below the ones included by default.
For
, use bullet lists.
For
, use a mix of numbered lists and narrative content as
appropriate.
For
cleanup
, use numbered lists to describe the steps to clean up the
state of the cluster after finishing the task.
For
whatsnext
An example of a published tutorial topic is
Running a Stateless Application Using a Deployment
.
Reference
A component tool reference page shows the description and flag options output for
a Kubernetes component tool. Each page generates from scripts using the component tool commands.
A tool reference page has several possible sections:
synopsis
options
options from parent commands
examples
seealso
Examples of published tool reference pages are:
kubectl
Learn about the
content organization
Feedback
Yes
No
.
or
.
Last modified October 25, 2022 at 1:09 AM PST:
Update page weights for pages in content/en/docs/contribute (35ebbd1e41)
Concept
Task
Tutorial
Reference
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 

The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Participating in SIG Docs | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
SIG Docs is one of the
special interest groups
within the Kubernetes project, focused on writing, updating, and maintaining
the documentation for Kubernetes as a whole. See
SIG Docs from the community github repo
for more information about the SIG.
SIG Docs welcomes content and reviews from all contributors. Anyone can open a
pull request (PR), and anyone is welcome to file issues about content or comment
on pull requests in progress.
You can also become a
member
,
reviewer
, or
approver
.
These roles require greater access and entail certain responsibilities for
approving and committing changes. See
community-membership
for more information on how membership works within the Kubernetes community.
The rest of this document outlines some unique ways these roles function within
SIG Docs, which is responsible for maintaining one of the most public-facing
aspects of Kubernetes -- the Kubernetes website and documentation.
SIG Docs chairperson
Each SIG, including SIG Docs, selects one or more SIG members to act as
chairpersons. These are points of contact between SIG Docs and other parts of
the Kubernetes organization. They require extensive knowledge of the structure
of the Kubernetes project as a whole and how SIG Docs works within it. See
Leadership
for the current list of chairpersons.
SIG Docs teams and automation
Automation in SIG Docs relies on two different mechanisms:
GitHub teams and OWNERS files.
GitHub teams
There are two categories of SIG Docs
teams
on GitHub:
@sig-docs-{language}-owners
are approvers and leads
@sig-docs-{language}-reviews
are reviewers
Each can be referenced with their
@name
in GitHub comments to communicate with
everyone in that group.
Sometimes Prow and GitHub teams overlap without matching exactly. For
assignment of issues, pull requests, and to support PR approvals, the
automation uses information from
OWNERS
files.
OWNERS files and front-matter
The Kubernetes project uses an automation tool called prow for automation
related to GitHub issues and pull requests. The
Kubernetes website repository
uses
two
prow plugins
:
blunderbuss
approve
These two plugins use the
OWNERS
and
files in the top level of the
GitHub repository to control
how prow works within the repository.
An OWNERS file contains a list of people who are SIG Docs reviewers and
approvers. OWNERS files can also exist in subdirectories, and can override who
can act as a reviewer or approver of files in that subdirectory and its
descendants. For more information about OWNERS files in general, see
OWNERS
.
In addition, an individual Markdown file can list reviewers and approvers in its
front-matter, either by listing individual GitHub usernames or GitHub groups.
The combination of OWNERS files and front-matter in Markdown files determines
the advice PR owners get from automated systems about who to ask for technical
and editorial review of their PR.
How merging works
When a pull request is merged to the branch used to publish content, that content is published to
http://kubernetes.io
. To ensure that
the quality of our published content is high, we limit merging pull requests to
SIG Docs approvers. Here's how it works.
When a pull request has both the
lgtm
and
approve
labels, has no
hold
labels, and all tests are passing, the pull request merges automatically.
Kubernetes organization members and SIG Docs approvers can add comments to
prevent automatic merging of a given pull request (by adding a
/hold
comment
or withholding a
/lgtm
comment).
Any Kubernetes 

member can add the
lgtm
label by adding a
/lgtm
comment.
Only SIG Docs approvers can merge a pull request
by adding an
/approve
comment. Some approvers also perform additional
specific roles, such as
PR Wrangler
or
.
For more information about contributing to the Kubernetes documentation, see:
Reviewing content
Documentation style guide
1 - Roles and responsibilities
Anyone
Slack
or the
.
After
.
Members
.
Anyone
Use the
/lgtm
Note:
Using
/lgtm
Use the
/hold
Use the
/assign
:
Find two
reviewers
or
approvers
to
sponsor
your
or on the
.
Note:
)
Send them the issue link using Slack or email.
+1
Note:
Reviewers
Anyone
and
Members
Note:
Automation assigns reviewers 

to all pull requests. You can request a
.
Using
/lgtm
/lgtm
/approve
A
/lgtm
lgtm
label.
,
To apply:
file
in the
Note:
.
).
Approvers
Anyone
,
Members
and
Reviewers
/approve
comment
/lgtm
/lgtm
/lgtm
/approve
Warning:
.
/approve
a PR.
.
,
To apply:
Open a pull request adding yourself to a section of the
file in the
Note:
.
Read about
2 - PR wranglers
SIG Docs
approvers
take week-long shifts
managing pull requests
for the repository.
This section covers the duties of a PR wrangler. For more information on giving good reviews, see
.
Duties
Each day in a week-long shift as PR Wrangler:
Triage and tag incoming issues daily. See
Triage and categorize issues
for guidelines on how SIG Docs uses metadata.
Review
open pull requests
for quality and adherence to the
Style
and
Content
guides.
Start with the smallest PRs (
size/XS
) first, and end with the largest (
size/XXL
). Review as many PRs as you can.
Make sure PR 

contributors sign the
CLA
.
Use
this
script to remind contributors that haven't signed the CLA to do so.
Provide feedback on changes and ask for technical reviews from members of other SIGs.
Provide inline suggestions on the PR for the proposed content changes.
If you need to verify content, comment on the PR and request more details.
Assign relevant
sig/
label(s).
If needed, assign reviewers from the
reviewers:
block in the file's front matter.
You can also tag a
SIG
for a review by commenting
@kubernetes/&lt;sig&gt;-pr-reviews
on the PR.
Use the
/approve
comment to approve a PR for merging. Merge the PR when ready.
PRs should have a
/lgtm
comment from another member before merging.
Consider accepting technically accurate content that doesn't meet the
style guidelines
. As you approve the change,
open a new issue to address the style concern. You can usually write these style fix
issues as
good first issues
.
Using style fixups as good first issues is a good way to ensure a supply of easier tasks
to help onboard new contributors.
Helpful GitHub queries for wranglers
The following queries are helpful when wrangling.
After working through these queries, the remaining list of PRs to review is usually small.
These queries exclude localization PRs. All queries are against the main branch except the last one.
No CLA, not eligible to merge
:
Remind the contributor to sign the CLA. If both the bot and a human have reminded them, close
the PR and remind them that they can open it after signing the CLA.
Do not review PRs whose authors have not signed the CLA!
Needs LGTM
:
Lists PRs that need an LGTM from a member. If the PR needs technical review, loop in one of the reviewers suggested by the bot. If the content needs work, add suggestions and feedback in-line.
Has LGTM, needs docs approval
:
Lists PRs that need an
/approve
comment to merge.
Quick Wins
: Lists PRs against the main branch with no clear blockers. (change "XS" in the size label as you work through the PRs [XS, S, M, L, XL, XXL]).
Not against the primary branch
: If the PR is against a
dev-
branch, it's for an upcoming release. Assign the
docs release manager
using:
/assign @&lt;manager's_github-username&gt;
. If the PR is against an old branch, help the author figure out whether it's targeted against the best branch.
Helpful Prow commands for wranglers
# add English label
/language en

# add squash label to PR if more than one commit
/label tide/merge-method-squash

# retitle a PR via Prow (such as a work-in-progress [WIP] or better detail of PR)
/retitle [WIP] &lt;TITLE&gt;
When to close Pull Requests
Reviews and approvals are one tool to keep our PR queue short and current. Another tool is closure.
Close PRs where:
The author hasn't signed the CLA for two weeks.
Authors can reopen the PR after signing the CLA. This is a low-risk way to make sure nothing gets merged without a signed CLA.
The author has not responded to comments or feedback in 2 or more weeks.
Don't be afraid to close pull requests. Contributors can easily reopen and resume works in progress. Often a closure notice is what spurs an author to resume and finish their contribution.
To close a pull request, leave a
/close
comment on the PR.
Note:
The
k8s-triage-robot
bot marks issues as stale after 90 days of inactivity. After 30 more days it marks issues as rotten and closes them. PR wranglers should close issues after 14-30 days of inactivity.
PR Wrangler shadow program
In late 2021, SIG Docs introduced the PR Wrangler Shadow Program. The program was introduced to help new contributors understand the PR wrangling process.
Become a shadow
If you are interested in shadowing as a PR wrangler, please visit the
PR Wranglers Wiki page
to see the PR wrangling schedule for this year and sign up.
Kubernetes org members can edit the
and sign up to shadow an existing PR Wrangler for a week.
Others can reach out on the
#sig-docs Slack channel
for requesting to shadow an assigned PR Wrangler for a

 specific week. Feel free to reach out to Brad Topol (
@bradtopol
) or one of the
SIG Docs co-chairs/leads
.
Once you've signed up to shadow a PR Wrangler, introduce yourself to the PR Wrangler on the
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Scale Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Running Multiple Instances of Your App
Scale an app using kubectl.
Scaling an application
In the previous modules we created a
, and then exposed it publicly via a
Service
. The Deployment created only one Pod for running our application. When traffic increases, we will need to scale the application to keep up with user demand.
Scaling
is accomplished by changing the number of replicas in a Deployment
Summary:
Scaling a Deployment
You can create from the start a Deployment with multiple instances using the --replicas parameter for the kubectl create deployment command
Scaling overview
Previous
Next
Scaling out a Deployment will ensure new Pods are created and scheduled to Nodes with available resources. Scaling will increase the number of Pods to the new desired state. Kubernetes also supports
of Pods, but it is outside of the scope of this tutorial. Scaling to zero is also possible, and it will terminate all Pods of the specified Deployment.
Running multiple instances of an application will require a way to distribute the traffic to all of them. Services have an integrated load-balancer that will distribute network traffic to all Pods of an exposed Deployment. Services will monitor continuously the running Pods using endpoints, to ensure the traffic is sent only to available Pods.
Scaling is accomplished by changing the number of replicas in a Deployment.
Once you have multiple instances of an Application running, you would be able to do Rolling updates without downtime. We'll cover that in the next module. Now, let's go to the online terminal and scale our application.
›
2 - Interactive Tutorial - Scaling Your App
&lt; Return to Module 4
Home
Continue to Module 6 &gt;
Home
Blog
Training
Partners
Community
CC BY 4.0

Performing a Rolling Update | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Debug Pod

s
Auditing
Run Jobs
TLS
Tutorials
Configuring Redis using a Co

nfigMap
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

 Audit Configuration (v1)
Tutorials
Summary:
Previous
Next
›
Feedback
Yes
No
.
or
.
Print entire

 section
Home
Blog
Training
Partners
Community
CC BY 4.0

Authenticating with Bootstrap Tokens | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

l Tools
Debug Pods
Debug 

Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Sec

urity Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Confi

guration (v1alpha1)
Reference
to support
kubeadm
kubeadm
system.
(
.
here
.
:
v1
kind
:
Secret
metadata
:
name
:
namespace
:
type
:
:
:
"The default bootstrap token generated 

by 'kubeadm init'."
token-id
:
07401b
:
:
:
"true"
:
"true"
:
be
The
true
The
kubeadm
in the
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
data
:
:
:
|
clusters:
- cluster:
name: ""
users: []
The
=
HS256
must
is used.
Warning:
Feedback
Yes
No
.
or
.
Last modified June 10, 2022 at 10:49 PM PST:
Fix links for k/design-proposals-archive (34155fed0b)
Bootstrap Tokens Overv

iew
Home
Blog
Training
Partners
Community
CC BY 4.0

NetworkPolicy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
kind
metadata
(
)
spec
(
)
status
(
)
(
ingress
List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic source is the pod's local node, OR if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are iso

lated by default)
(IPBlock)
(
)
(
)
(int32)
(string)
egress
List of egress rules to be applied to the selected pods. Outgoing traffic is allowed if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic ma

tches at least one egress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy limits all outgoing traffic (and serves solely to ensure that the pods it selects are isolated by default). This field is beta-level in 1.8
egress.to
(IPBlock)
(
)
(
)
(int32)
(string)
The protocol (TCP, UDP, or SCTP) which traffic must match. If not specified, this field defaults 

to TCP.
type
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit

pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(

in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Dynamic Admission Control | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

t Up kubectl on macOS
Debug Pods
Get a Shell to a Runnin

g Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Containe

r's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configuratio

n (v1)
Reference
,
and
.
Note:
Here
as an
See the
See the
field
empty
,
how to
.
the
.
The te

st also creates a
service
code
.
or
.
See the
:
kind
:
metadata
:
name
:
webhooks
:
-
name
:
rules
:
-
apiGroups
:
[
""
]
:
[
"v1"
]
:
[
"CREATE"
]
resources
:
[
"pods"
]
scope
:
:
service
:
namespace
:
name
:
"example-service"
caBundle
:
:
[
"v1"
]
:
None
:
5
Note:
The
scope
Note:
When using
.
Note:
timeout
rules
, the
.
flag.
:
kind
:
plugins
:
-
name
:
:
:
kind
:
:
-
name
:
:
:
kind
:
:
:
kind
:
plugins
:
-
name
:
:
:
kind
:
:
-
name
:
:
:
kind
:
:

, see the
.
See the
:
v1
kind
:
Config
users
:
#
#   user: ...
#
#   user: ...
#   user: ...
#
#   user: ...
#
#   user: ...
#
#   user: ...
#
-
name
:
user
:
:
:
-
name
:
user
:
password
:
username
:
-
name
:
'*'
user
:
token
:
Request
,
with an
API group
with the
:
kind
:
webhooks
:
-
name
:
:
[
"v1"
,
"v1beta1"
]
.
object
scale
apps/v1
:
:
kind
:
request
:
uid
:
kind
:
group
:
version
:
v1
kind
:
Scale
resource
:
group
:
apps
version
:
v1
resource
:
:
scale
:
group
:
version
:
v1
kind
:
Scale
# Fully-qualifi

ed group/version/kind of the resource being modified in the original request to the API server.
:
group
:
apps
version
:
v1
resource
:
:
scale
name
:
namespace
:
operation
:
UPDATE
userInfo
:
# Username of the authenticated user making the request to the API server
username
:
admin
uid
:
groups
:
extra
:
some-key
:
object
:
:
kind
:
Scale
oldObject
:
:
kind
:
Scale
options
:
:
kind
:
dryRun
:
False
Response
,
with the
response
response
uid
allowed
true
or
false
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
}
}
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
false
}
}
using the
status
See the
status
type.
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
false
,
"status"
: {
"code"
:
403
,
"message"
:
}
}
}
patch
and
patchType
patchType
is
JSONPatch
.
See
JSON patch
For

, the
patch
would be
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
,
:
,
"patch"
:
}
}
in HTTP
Warning
Don't include a "Warning:" prefix in the message
Caution:
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
,
: [
,
]
}
}
or
or a
.
"CREATE"
,
"UPDATE"
,
"DELETE"
,
"CONNECT"
,
or
"*"
apiGroups
""
"*"
"*"
resources
"*"
"*/*"
"pods/*"
scope
"Cluster"
,
, and
"*"
.
"*"
.
"Cluster"
"*"
,
groups
,
versions
,
resources
, and
scope
rules
Match
CREATE
or
UPDATE
apps/v1
and
and
:
:
kind
:
...
webhooks
:
-
name
:
rules
:
-
:
[
"CREATE"
,
"UPDATE"
]
apiGroups
:
[
"apps"
]
apiVersi

ons
:
[
"v1"
,
"v1beta1"
]
resources
:
[
,
]
scope
:
...
:
kind
:
webhooks
:
-
name
:
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
"*"
status
:
kind
:
webhooks
:
-
name
:
rules
:
-
:
[
"UPDATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
]
scope
:
"*"
oldObject
newObject
or a
object)
CREATE
foo: bar
:
:
kind
:
webhooks
:
-
name
:
:
:
foo
:
bar
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
"*"
See
.
The
CREATE
:
kind
:
webhooks
:
-
name
:
:
:
-
key
:
runlevel
operator
:
NotIn
values
:
[
"0"
,
"1"
]
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
CREATE
:
kind
:
webhooks
:
-
name
:
:
:
-
key
:
operator
:
In
values
:
[
"prod"
,
"staging"
]
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
See
(like
),
),
The
rules
ar

e used to match incoming requests.
Exact
or
.
Exact
rules
,
apps/v1
could use
:
means the
apps/v1
)
other versions of that resource that are still served.
rule
apps/v1
apps/v1
object:
:
kind
:
webhooks
:
-
name
:
:
rules
:
-
:
[
"CREATE"
,
"UPDATE"
,
"DELETE"
]
apiGroups
:
[
"apps"
]
:
[
"v1"
]
resources
:
[
]
scope
:
The
.
URL
url
(
).
The
host
service
(e.g.,
host
localhost
or
127.0.0.1
as a
host
is
#...
?...
:
kind
:
webhooks
:
-
name
:
:
url
:
The
service
service
instead of
url
.
at the subpath "/my-path", and to verify the 

TLS connection against the ServerName
:
kind
:
webhooks
:
-
name
:
:
caBundle
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
Note:
the out-of-band data modified by the admission webhook to reflect reality.
dryRun
,
None
is
dryRun
-aware).
requests:
:
kind
:
webhooks
:
-
name
:
:
Timeouts
.
:
kind
:
webhooks
:
-
name
:
:
2
(see
container
to a
pod
Never
or
IfNeeded
Never
.
Never
IfNeeded
The number of add

itional invocations is not guaranteed to be exactly one.
:
kind
:
webhooks
:
-
name
:
:
IfNeeded
in an object could already exist in the user-provided object, but it is essential for webhooks that opt into reinvocation.
Ignore
or
Fail
.
Ignore
Fail
:
kind
:
webhooks
:
-
name
:
:
Fail
Fail
.
auditing
on each
At
Metadata
{
"kind":
"Event"
,
,
{
false
}"
...
}
...
}
{
,
,
false
}
# the audit event rec

orded
{
"kind":
"Event"
,
,
{
true
}"
...
}
...
}
{
,
,
true
}
At
Request
mutating webhook chain, and responded with a JSON patch which got applied to the request object.
{
"kind":
"Event"
,
,
{
...
}
...
}
{
,
,
,
"patch":
[
{
"op":
"add"
,
"path":
,
"value":
"yes"
}
]
}
/metrics
name
operation
CREATE
,
UPDATE
,
DELETE
and
CONNECT
.
type
admit
and
.
webhook's
is set to
Fail
.
no_error
apiserver_admission_webhook

_rejection_count{error_type="calling_webhook_error",name="invalid-admission-response-webhook.example.com",operation="CREATE",rejection_code="0",type="validating"} 1
For a
CREATE
of the
For a
CREATE
For a
CREATE
For a
CREATE
).
For a
CREATE
/
UPDATE
"env"
set,
For a
CREATE
to
Timeouts
supports.
CREATE
must
CREATE
object.
A webhook running inside the cluster might cause deadlocks for its own deployment if it is c

onfigured
CREATE
pod (e.g.
"env"
label.
"env"
.
be set to
None
dryRun
set to
true
, and the
set to
. See
The
kube-dns
.
.
Feedback
Yes
No
.
or
.
Last modified July 01, 2022 at 12:33 AM PST:
Tweak extensible admission controllers page (a6ec7d8017)
Request
Response
Timeouts
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-scheduler Configuration (v1beta2) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

tl on Linux
Debug Pods
Debug Running Po

ds
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespace

 Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserv

er Encryption Configuration (v1)
Reference
Field
string
kind
string
int32
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
string
HealthzBindAddress is the IP address and port for the health check server t

o serve on.
string
int32
int64
int64
profiles
extenders
Field
string
kind
string
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
PodTopologySpreadArgs holds arguments used to config

ure the PodTopologySpread plugin.
Field
string
kind
string
.
Field
string
kind
string
int64
shape
Extender
Field
urlPrefix
string
string
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool
ManagedResources i

s a list of extended resources that are managed by
ignorable
bool
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
Field
string
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
Field
name
string
args
PluginSet
Plugins

Field
enabled
[]Plugin
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSet
PluginSet
preScore
PluginSet
score
PluginSet
reserve
PluginSet
permit
PluginSet
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
(Alias of
string
)
Field
shape
Field
name
string
weight
int64
Field
type
resources
The default resource set includes "cpu"

 and "memory" with an equal weight.
(Alias of
string
)
Field
int32
score
int32
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
enabled.
string
string
string
Feedback
Yes
No
Thanks for the feedback. If you have a speci

fic, answerable question about how to use Kubernetes, ask it on
.
or
.
Extender
Plugin
PluginSet
Plugins
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure DNS for a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Kubernetes offers a DNS cluster addon, which most of the supported environments enable by default. In Kubernetes version 1.11 and later, CoreDNS is recommended and is installed by default with kubeadm.
For more information on how to configure CoreDNS for a Kubernetes cluster, see the
. An example demonstrating how to use Kubernetes DNS with kube-dns, see the
Kubernetes DNS sample plugin
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure a Pod to Use a Volume for Storage | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kub

ernetes API Aggregation Layer
Tasks
Debug Pods
Debu

g a StatefulSet
Auditing
Run Jobs
TLS
Tutorials
Security
Appl

y Pod Security Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Audit Configuration 

(v1)
Tasks
Volume
minikube
.
emptyDir
:
v1
kind
:
Pod
metadata
:
name
:
redis
spec
:
:
-
name
:
redis
image
:
redis
:
-
name
:
mountPath
:
volumes
:
-
name
:
emptyDir
:
{}
the Pod:
kubectl
exec
cd
echo
root        15  0.0  0.0  1

7500  2072 ?        R+   00:48   0:00 ps aux
kill
where
of
Always
.
kubectl
exec
test-file
cd
test-file
See
Volume
.
See
Pod
.
emptyDir
Volumes
Feedback
Yes
No
.
or
.
Last modified November 14, 2022 at 8:41 PM PST:
Fix code mark in configure-volume-storage.md (cd4371d8d1)
Home
Blog
Training
Partners
Community
CC BY 4.0

Common Definitions | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
ListMeta
4:
5:
6:
7:
8:
9:
Patch
10:
Quantity
11:
12:
Status
13:
1 - DeleteOptions
(string)
dryRun
(int64)
kind
(string)
(boolean)
(string)
(string)
(string)
2 - LabelSelector
key
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists a

nd DoesNotExist.
3 - ListMeta
continue
(string)
(int64)
remainingItemCount is the number of subsequent items in the list which are not included in this list response. If the list request contained label or field selectors, then the number of remaining items is unknown and the field will be left unset and omitted during serialization. If the list is complete (either because it is not chunking or because this is the last chunk), then there are no more remaining items and this field will be left unset and omitted during serialization. Servers older than v1.15 do not set this field. The intended use of the remainingItemCount is
(string)
selfLink
(string)
4 - LocalObjectReference
name
(string)
5 - NodeSelectorRequirement
key
operator
values
An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during

 a strategic merge patch.
6 - ObjectFieldSelector
fieldPath
(string)
7 - ObjectMeta
name
(string)
(string)
namespace
(string)
Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.
labels
System
ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields 

is always in the version that the workflow used when modifying the object.
(string)
(string)
', where
', where
', where
', where
(string)
(string)
(string)
Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always corresponds to the version of the main resource.
(Time)
uid
(boolean)
http

s://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion
(boolean)
Read-only
(Time)
(int64)
(Time)
DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.
(int64)
(string)
selfLink
(string)
uid
(string)

8 - ObjectReference
(string)
fieldPath
(string)
kind
(string)
name
(string)
namespace
(string)
(string)
uid
(string)
9 - Patch
10 - Quantity

Examples:

11 - ResourceFieldSelector
resource
(string)
divisor
(
Quantity
)
12 - Status
(string)
code
(int32)
details
Extended data associated with the reason. Each reason may define its own extended details. This field is optional and the data returned is not guaranteed to conform to any schema except that defined by the reason type.
(string)
Examples:
(string)
(string)
(string)
(string)
(string)
(int32)
(string)
UID of the resourc

e. (when there is a single resource which can be described). More info:
kind
(string)
message
(string)
metadata
(
ListMeta
)
reason
(string)
status
(string)
13 - TypedLocalObjectReference
kind
name
apiGroup
(string)
Home
Blog
Training
Partners
Community
CC BY 4.0

Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes A

PI Aggregation Layer
Tasks
Debug Pods
Debug a State

fulSet
Auditing
Run Jobs
TLS
Tutorials
Security

urity Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube

-apiserver Configuration (v1)
Tasks
audit
and
warn
.
concepts.
or use a
Fine-grained control over policy definition
.
Validating Admissi

on Webhook
.
labels on
.
0
policies
Caution:
,
.spec.runtimeClass
Caution:
MustRunAs
strategy they may be

 mutating! Removing these could result in
ALL
MustRunAs
RunAsAny
for
false
profile.
PSP_NAME
=
jsonpath
=
$PSP_NAME
and
:
use
on all PSPs
use
them.
and
to how one might approach this on a new cluster.
- Using the
PSP to a Pod Sec

urity Standard level. If your PSPs aren't based on the Pod Security Standards, you
-o
jsonpath
=
| tr
" "
"\n"
| sort -u
,
Caution:
existing
=
=
$LEVEL
existing
level.
=
$LEVEL
=
$LEVEL
that your processes and/or policies for creating new namespaces are updated to ensure that an
You can also stat

ically configure the Pod Security admission controller to set a default enforce,
.
contain
.
Feedback
Yes
No
.
or
.
Last modified November 05, 2022 at 6:29 PM PST:
Add advice about PSP migration (fc34399bdd)
Home
Blog
Training
Partners
Community
CC BY 4.0

HorizontalPodAutoscaler Walkthrough | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security
Apply 

Pod Security Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

r Configuration (v1alpha1)
Tasks
A
(HPA for short)
automatically updates a workload resource (such as
a
or
), with the
aim of automatically scaling the workload to match demand.
Horizontal scaling means that the response to increased load is to deploy more
Pods
.
This is different from
vertical
scaling, which for Kubernetes would mean
assigning more resources (for example: memory or CPU) to the Pods that are already
running for the workload.
If the load decreases, and the number of Pods is above the configured minimum,
the HorizontalPodAutoscaler instructs the workload resource (the Deployment, StatefulSet,
or other similar resource) to scale back down.
This document walks you through an example of enabling HorizontalPodAutoscaler to
automatically manage scale for an example web app. This example workload is Apache
httpd running some PHP code.
minikube
Your Kubernetes server must be at or later than version 1.23.
.
If you're running an older
release of Kubernetes, refer to the version of the documentation for that release (see
available documentation versions
).
To follow this walkthrough, you also need to use a cluster that has a
deployed and configured.
The Kubernetes Metrics Server collects resource metrics from
the
kubelets
in your cluster, and exposes those metrics
,
using an
to add
new kinds of resource that represent metric readings.
To learn how to deploy the Metrics Server, see the
metrics-server documentation
.
Run and expose php-apache server
To demonstrate a HorizontalPodAutoscaler, you will first start a Deployment that runs a container using the
hpa-example
image, and expose it as a
Service
using the following manifest:
application/php-apache.yaml
:
apps/v1
kind
:
metadata
:
name
:
php-apache
spec
:
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
-
name
:
image
:
registry.k8s.io/hpa-ex

ample
ports
:
-
:
80
resources
:
limits
:
cpu
:
500m
requests
:
cpu
:
200m
---
:
v1
kind
:
Service
metadata
:
name
:
labels
:
run
:
spec
:
ports
:
-
port
:
80
selector
:
run
:
To do so, run the following command:
kubectl apply -f https://k8s.io/examples/application/php-apache.yaml
deployment.apps/php-apache created
service/php-apache created
Create the HorizontalPodAutoscaler
Now that the server is running, create the autoscaler using
kubectl
. There is
subcommand,
part of
kubectl
, that helps you do this.
You will shortly run a command that creates a HorizontalPodAutoscaler that maintains
between 1 and 10 replicas of the Pods controlled by the php-apache Deployment that
you created in the first step of these instructions.
Roughly speaking, the HPA
will increase and decrease
the number of replicas (by updating the Deployment) to maintain an average CPU utilization across all Pods of 50%.
The Deployment then updates the ReplicaSet - this is part of how all Deployments work in Kubernetes -
and then the ReplicaSet either adds or removes Pods based on the change to its
.spec
.
Since each pod requests 200 milli-cores by
, this means an average CPU usage of 100 milli-cores.
See
Algorithm details
on the algorithm.
Create the HorizontalPodAutoscaler:
kubectl autoscale deployment php-apache --cpu-percent
=
50
--min
=
1
--max
=
10
horizontalpodautoscaler.autoscaling/php-apache autoscaled
You can check the current status of the newly-made HorizontalPodAutoscaler, by running:
# You can use "hpa" or "horizontalpodautoscaler"; either name works OK.
kubectl get hpa
NAME         REFERENCE                     TARGET    MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache/scale   0% / 50%  1         10        1          18s
(if you see other HorizontalPodAutoscalers with different names, that means they already existed,
and isn't usually a problem).
Please note that the current CPU consumption is 0% as there are no clients sending requests to the server
(the
TARGET
column shows the average across all the Pods controlled by the corresponding deployment).
Increase the load
Next, see how the autoscaler reacts to increased load.
To do this, you'll start a different Pod to act as a client. The container within the client Pod
runs in an infinite loop, sending queries to the php-apache service.
# Run this in a separate terminal
# so that the load generation continues and you can carry on with the rest of the steps
kubectl run -i --tty load-generator --rm --image
=
=
Never -- /bin/sh -c
"while sleep 0.01; do wget -q -O- http://php-apache; done"
Now run:
# type Ctrl+C to end the watch when you're ready
kubectl get hpa php-apache --watch
Within a minute or so, you should see the higher CPU load; for example:
NAME         REFERENCE                     TARGET      MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache/scale   305% / 50%  1         10        1          3m
and then, more replicas. For example:
php-apache   Deployment/php-apache/scale   305% / 50%  1         10        7          3m
Here, CPU consumption has increased to 305% of the request.
As a result, the Deployment was resized to 7 replicas:
kubectl get deployment php-apache
You should see the replica count matching the figure from the HorizontalPodAutoscaler
php-apache   7/7      7           7           19m
Note:
It may take a few minutes to stabilize the number of replicas. Since the amount
of load is not controlled in any way it may happen that the final number of replicas
will differ from this example.
Stop generating load
To finish the example, stop sending the load.
In the terminal where you created the Pod that runs a
busybox
image, terminate
the load 

generation by typing
&lt;Ctrl&gt; + C
.
Then verify the result state (after a minute or so):
NAME         REFERENCE                     TARGET       MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache/scale   0% / 50%     1         10        1          11m
and the Deployment also shows that it has scaled down:
php-apache   1/1     1            1           27m
Once CPU utilization dropped to 0, the HPA automatically scaled the number of replicas back down to 1.
Autoscaling the replicas may take a few minutes.
Autoscaling on multiple metrics and custom metrics
You can introduce additional metrics to use when autoscaling the
by making use of the
API version.
First, get the YAML of your HorizontalPodAutoscaler in the
form:
kubectl get hpa php-apache -o yaml &gt; /tmp/hpa-v2.yaml
Open the
/tmp/hpa-v2.yaml
file in an editor, and you should see YAML which looks like this:
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
metrics
:
-
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
Utilization
averageUtilization
:
50
status
:
:
1
:
&lt;some-time&gt;
:
1
:
1
:
-
type
:
Resource
resource
:
name
:
cpu
current
:
:
0
averageValue
:
0
field has been replaced with an array called
metrics
.
The CPU utilization metric is a
resource metric
, since it is represented as a percentage of a resource
specified on pod containers. Notice that you can specify other resource metrics besides CPU. By default,
the only other supported resource metric is memory. These resources do not change names from cluster
to cluster, and should always be available, as long as the
API is available.
You can also specify resource metrics in terms of direct values, instead of as percentages of the
requested value, by using a
target.type
of
AverageValue
instead of
, and
setting the corresponding
target.averageValue
target.averageUtilization
.
There are two other types of metrics, both of which are considered
custom metrics
: pod metrics and
object metrics. These metrics may have names which are cluster specific, and require a more
advanced cluster monitoring setup.
The first of these alternative metric types is
pod metrics
. These metrics describe Pods, and
are averaged together across Pods and compared with a target value to determine the replica count.
They work much like resource metrics, except that they
only
support a
target
type of
.
Pod metrics are specified using a metric block like this:
type
:
Pods
pods
:
metric
:
name
:
packets-per-second
target
:
type
:
:
1k
The second alternative metric type is
object metrics
. These metrics describe a different
object in the same namespace, instead of describing Pods. The metrics are not necessarily
fetched from the object; they only describe it. Object metrics support
target
types of
both
Value
and
. With
Value
, the target is compared directly to the returned
metric from the API. With
, the value returned from the custom metrics API is divided
by the number of Pods before being compared to the target. The following example is the YAML
representation of the
requests-per-second
metric.
type
:
Object
object
:
metric
:
name
:
describedObject
:
:
kind
:
Ingress
name
:
main-route
target
:
type
:
Value
value
:
2k
If you provide multiple such metric blocks, the HorizontalPodAutoscaler will consider each metric in tu

rn.
The HorizontalPodAutoscaler will calculate proposed replica counts for each metric, and then choose the
one with the highest replica count.
For example, if you had your monitoring system collecting metrics about network traffic,
you could update the definition above using
to look like this:
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
metrics
:
-
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
50
-
type
:
Pods
pods
:
metric
:
name
:
target
:
type
:
:
1k
-
type
:
Object
object
:
metric
:
name
:
:
:
kind
:
Ingress
name
:
target
:
type
:
Value
value
:
10k
status
:
:
1
:
:
1
:
1
:
-
type
:
Resource
resource
:
name
:
cpu
current
:
:
0
:
0
-
type
:
Object
object
:
metric
:
name
:
:
:
kind
:
Ingress
name
:
current
:
value
:
10k
Then, your HorizontalPodAutoscaler would attempt to ensure that each pod was consuming roughly
50% of its requested CPU, serving 1000 packets per second, and that all pods behind the main-route
Ingress were serving a total of 10000 requests per second.
Autoscaling on more specific metrics
Many metrics pipelines allow you to describe metrics either by name or by a set of additional
descriptors called
labels
. For all non-resource metric types (pod, object, and external,
described below), you can specify an additional label selector which is passed to your metric
pipeline. For instance, if you collect a metric
http_requests
with the
verb
label, you can specify the following metric block to scale only on GET requests:
type
:
Object
object
:
metric
:
name
:
selector
:
{
:
{
verb
:
GET}}
This selector uses the same syntax as the full Kubernetes label selectors. The monitoring pipeline
determines how to collapse multiple series into a single value, if the name and selector
match multiple series. The selector is additive, and cannot select metrics
that describe objects that are
not
the target object (the target pods in the case of the
Pods
type, and the described object in the case of the
Object
type).
Autoscaling on metrics not related to Kubernetes objects
Applications running on Kubernetes may need to autoscale based on metrics that don't have an obvious
relationship to any object in the Kubernetes cluster, such as metrics describing a hosted service with
no direct correlation to Kubernetes namespaces. In Kubernetes 1.10 and later, you can address this use case
with
external metrics
.
Using external metrics requires knowledge of your monitoring system; the setup is
similar to that required when using custom metrics. External metrics allow you to autoscale your cluster
based on any metric available in your monitoring system. Provide a
metric
block with a
name
and
selector
, as above, and use the
External
metric type instead of
Object
.
If multiple time series are matched by the
metricSelector
,
the sum of their values is used by the HorizontalPodAutoscaler.
External metrics support both the
Value
and
target types, which function exactly the same
as when you use the
Object
type.
For example if your application processes tasks from a hosted queue service, you could add the following
section to your HorizontalPodAutoscaler manifest to specify that you need one worker per 30 outstanding tasks.
-
type
:
External
external
:
metric
:
name
:
queue_messages_ready
selector
:
:
queue
:
"worker_tasks"
target
:
type
:
:
30
When possible, it's preferable to use the custom metric target types instead of external metric

s, since it's
easier for cluster administrators to secure the custom metrics API. The external metrics API potentially allows
access to any metric, so cluster administrators should take care when exposing it.
Appendix: Horizontal Pod Autoscaler Status Conditions
When using the
form of the HorizontalPodAutoscaler, you will be able to see
status conditions
set by Kubernetes on the HorizontalPodAutoscaler. These status conditions indicate
whether or not the HorizontalPodAutoscaler is able to scale, and whether or not it is currently restricted
in any way.
The conditions appear in the
field. To see the conditions affecting a HorizontalPodAutoscaler,
we can use
kubectl describe hpa
:
kubectl describe hpa cm-test
Name:                           cm-test
Namespace:                      prom
Labels:                         &lt;none&gt;
Annotations:                    &lt;none&gt;
CreationTimestamp:              Fri, 16 Jun 2017 18:09:22 +0000
Reference:                      ReplicationController/cm-test
Metrics:                        ( current / target )
  "http_requests" on pods:      66m / 500m
Min replicas:                   1
Max replicas:                   4
ReplicationController pods:     1 current / 1 desired
  Type                  Status  Reason                  Message
  ----                  ------  ------                  -------
  AbleToScale           True    ReadyForNewScale        the last scale time was sufficiently old as to warrant a new scale
  ScalingActive         True    ValidMetricFound        the HPA was able to successfully calculate a replica count from pods metric http_requests
  ScalingLimited        False   DesiredWithinRange      the desired replica count is within the acceptable range
Events:
For this HorizontalPodAutoscaler, you can see several conditions in a healthy state. The first,
AbleToScale
, indicates whether or not the HPA is able to fetch and update scales, as well as
whether or not any backoff-related conditions would prevent scaling. The second,
ScalingActive
,
indicates whether or not the HPA is enabled (i.e. the replica count of the target is not zero) and
is able to calculate desired scales. When it is
False
, it generally indicates problems with
fetching metrics. Finally, the last condition,
ScalingLimited
, indicates that the desired scale
was capped by the maximum or minimum of the HorizontalPodAutoscaler. This is an indication that
you may wish to raise or lower the minimum or maximum replica count constraints on your
HorizontalPodAutoscaler.
Quantities
All metrics in the HorizontalPodAutoscaler and metrics APIs are specified using
a special whole-number notation known in Kubernetes as a
quantity
. For example,
the quantity
10500m
would be written as
10.5
in decimal notation. The metrics APIs
will return whole numbers without a suffix when possible, and will generally return
quantities in milli-units otherwise. This means you might see your metric value fluctuate
between
1
and
1500m
, or
1
and
1.5
when written in decimal notation.
Other possible scenarios
Creating the autoscaler declaratively
Instead of using
command to create a HorizontalPodAutoscaler imperatively we
can use the following manifest to create it declaratively:
application/hpa/php-apache.yaml
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
:
50
Then, create the autoscaler by executing the following command:
kubectl create -f https://k8s.io/examples/application/hpa/php-apache.yaml
horizontalpodautoscaler.autoscaling/php-apache created
Feedback
Yes
No
.
or
suggest an imp

rovement
.
Last modified August 06, 2022 at 4:01 AM PST:
Remove unnecessary instructions and use registry.k8s.io (aa27eb3a1e)
Home
Blog
Training
Partners
Community
CC BY 4.0

MutatingWebhookConfiguration | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

and Set Up kubectl on macOS
Debug Pods
Get a Shell to a 

Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Co

ntainer's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Config

uration (v1)
Reference
kind
metadata
(
)
.
webhooks
name
([]byte)
caBundle
service
service
or
url
service
.
name
namespace
(string)
path
(int32)
If specified, the port on the service that hosting webhook. 

Default to 443 for backward compatibility.
port
(string)
url
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
https://kub

ernetes.io/docs/concepts/overview/working-with-objects/labels/
(
)
(string)
must
any
(string)
scope specifies the scope of this rule. Valid values are "Clu

ster", "Namespaced", and "
(int32)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
bo

dy
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Docs smoke test page | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

This page serves two purposes:
Demonstrate how the Kubernetes documentation uses Markdown
Provide a "smoke test" document we can use to test HTML, CSS, and template
changes that affect the overall documentation.
Heading levels
The above heading is an H2. The page title renders as an H1. The following
sections show H3-H6.
H3
This is in an H3 section.
H4
This is in an H4 section.
H5
This is in an H5 section.
H6
This is in an H6 section.
Inline elements
Inline elements show up within the text of paragraph, list item, admonition, or
other block-level element.
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis
nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu
fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in
culpa qui officia deserunt mollit anim id est laborum.
Inline text styles
bold
italic
bold italic
strikethrough
underline
underline italic
underline bold
underline bold italic
monospace text
monospace bold
Lists
Markdown doesn't have strict rules about how to process lists. When we moved
from Jekyll to Hugo, we broke some lists. To fix them, keep the following in
mind:
Make sure you indent sub-list items
2 spaces
.
To end a list and start another, you need a HTML comment block on a new line
between the lists, flush with the left-hand border. The first list won't end
otherwise, no matter how many blank lines you put between it and the second.
Bullet lists
This is a list item
This is another list item in the same list
You can mix
-
and
*
To make a sub-item, indent two spaces.
This is a sub-sub-item. Indent two more spaces.
Another sub-item.
This is a new list. With Hugo, you need to use a HTML comment to separate two
consecutive lists.
The HTML comment needs to be at the left margin.
Bullet lists can have paragraphs or block elements within them.
Indent the content to be the same as the first line of the bullet point.
This paragraph and the code block line up with the first
B
in
Bullet
above.
ls -l
And a sub-list after some block-level content
A bullet list item can contain a numbered list.
Numbered sub-list item 1
Numbered sub-list item 2
Numbered lists
This is another list item in the same list. The number you use in Markdown
does not necessarily correlate to the number in the final output. By
convention, we keep them in sync.
Note:
For single-digit numbered lists, using two s

paces after the period makes
interior block-level content line up better along tab-stops.
This is a new list. With Hugo, you need to use a HTML comment to separate
two consecutive lists.
Numbered lists can have paragraphs or block elements within them.
Indent the content to be the same as the first line of the bullet
point.
This paragraph and the code block line up with the
N
in
Numbered
above.
ls -l
And a sub-list after some block-level content. This is at the same
"level" as the paragraph and code block above, despite being indented
more.
Tab lists
Tab lists can be used to conditionally display content, e.g., when multiple
options must be documented that require distinct instructions or context.
Choose one...
Formatting tab lists
Nested headers
Please select an option.
Tabs may also nest formatting styles.
Ordered
(Or unordered)
Lists
echo
'Tab lists may contain code blocks!'
Header within a tab list
Nested header tags may also be included.
Warning:
Headers within tab lists will not appear in the Table of Contents.
Checklists
Checklists are technically bullet lists, but the bullets are suppressed by CSS.
This is a checklist item
This is a selected checklist item
Code blocks
You can create code blocks two different ways by surrounding the code block with
three back-tick characters on lines before and after the code block.
Only use
back-ticks (code fences) for code blocks.
This allows you to specify the
language of the enclosed code, which enables syntax highlighting. It is also more
predictable than using indentation.
this is a code block created by back-ticks
The back-tick method has some advantages.
It works nearly every time
It is more compact when viewing the source code.
It allows you to specify what language the code block is in, for syntax
highlighting.
It has a definite ending. Sometimes, the indentation method breaks with
languages where spacing is significant, like Python or YAML.
To specify the language for the code block, put it directly after the first
grouping of back-ticks:
ls -l
Common languages used in Kubernetes documentation code blocks include:
bash
/
shell
(both work the same)
go
json
yaml
xml
none
(disables syntax highlighting for the block)
Code blocks containing Hugo shortcodes
To show raw Hugo shortcodes as in the above example and prevent Hugo
from interpreting them, use C-style comments directly after the
&lt;
and before
the
&gt;
characters. The following example illustrates this (view the Markdown
source for this page).
{{&lt; codenew file="pods/storage/gce-volume.yaml" &gt;}}
Links
To format a link, put the link text inside square brackets, followed by the
link target in parentheses.
Link to Kubernetes.io
or
Relative link to Kubernetes.io
You can also use HTML, but it is not preferred.
Images
To format an image, use similar syntax to
links
, but add a leading
!
character. The square brackets contain the image's alt text. Try to always use
alt text so that people using screen readers can get some benefit from the
image.
To specify extended attributes, such as width, title, caption, etc, use the
figure shortcode
,
which is preferred to using a HTML
&lt;img&gt;
tag. Also, if you need the image to
also be a hyperlink, use the
link
attribute, rather than wrapping the whole
figure in Markdown link syntax as shown below.
Pencil icon
Image used to illustrate the figure shortcode
Even if you choose not to use the figure shortcode, an image can also be a link. This
time the pencil icon links to the Kubernetes website. Outer square brackets enclose
the entire image tag, and the link target is in the parentheses at the end.
You can also use HTML for images, but it is not preferred.
Tables
Simple tables have one row per line, and columns are separated by
|
characters. The header is separated from the body by cells containing nothing
but at least three
-
characters. For ease of maintenance, try to keep all the
cell separators even, even if y

ou heed to use extra space.
Heading cell 1
Heading cell 2
Body cell 1
Body cell 2
The header is optional. Any text separated by
|
will render as a table.
Markdown tables have a hard time with block-level elements within cells, such as
list items, code blocks, or multiple paragraphs. For complex or very wide
tables, use HTML instead.
Body cell 1
Body cell 2
Visualizations with Mermaid
You can use
Mermaid JS
visualizations.
The Mermaid JS version is specified in
/layouts/partials/head.html
{{&lt; mermaid &gt;}}
graph TD;
  A--&gt;B;
  A--&gt;C;
  B--&gt;D;
  C--&gt;D;
{{&lt;/ mermaid &gt;}}
Produces:
graph TD;
A--&gt;B;
A--&gt;C;
B--&gt;D;
C--&gt;D;
enabled
    Alice -&gt;&gt; Bob: Hello Bob, how are you?
    Bob--&gt;&gt;John: How about you John?
    Bob--x Alice: I am good thanks!
    Bob-x John: I am good thanks!
    Note right of John: Bob thinks a long&lt;br/&gt;long time, so long&lt;br/&gt;that the text does&lt;br/&gt;not fit on a row.

    Bob--&gt;Alice: Checking with John...
    Alice-&gt;John: Yes... John, how are you?
Produces:
Alice -&gt;&gt; Bob: Hello Bob, how are you?
Bob--&gt;&gt;John: How about you John?
Bob--x Alice: I am good thanks!
Bob-x John: I am good thanks!
Note right of John: Bob thinks a long
long time, so long
that the text does
not fit on a row.
Bob--&gt;Alice: Checking with John...
Alice-&gt;John: Yes... John, how are you?
enabled
More
examples
from the official docs.
Sidebars and Admonitions
Sidebars and admonitions provide ways to add visual importance to text. Use
them sparingly.
Sidebars
A sidebar offsets text visually, but without the visual prominence of
admonitions
.
This is a sidebar.
You can have paragraphs and block-level elements within a sidebar.
You can even have code blocks.
sudo dmesg
Admonitions
Admonitions (notes, warnings, etc) use Hugo shortcodes.
Note:
Notes catch the reader's attention without a sense of urgency.
You can have multiple paragraphs and block-level elements inside an admonition.
| Or | a | table |
Caution:
The reader should proceed with caution.
Warning:
Warnings point out something that could cause harm if ignored.
Includes
To add shortcodes to includes.
Note:
minikube
Katacoda Embedded Live Environment
Launch Terminal
Feedback
Yes
No
.
or
.
Last modified June 17, 2021 at 7:49 PM PST:
Update docs for primary branch rename to "main" (da31234d5d)
H3
Lists
Tab lists
Links
Images
Tables
Sidebars
Includes
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubelet Checkpoint API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
root
user. It
post
CRI
.
tar
tar
namespace
(
in path
Namespace
pod
(
in path
Pod
container
(
in path
Container
timeout
(
in query
): integer
CRI

Response
200: OK
namespace
,
pod
or
container
Feedback
Yes
No
.
or
.
Last modified September 09, 2022 at 12:20 PM PST:
correct CheckpointContainer to ContainerCheckpoint (410a78bb63)
post
Home
Blog
Training
Partners
Community
CC BY 4.0

Set Kubelet parameters via a config file | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

ubectl on Linux
Debug Pods
Debug Runnin

g Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Names

pace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-api

server Encryption Configuration (v1)
Tasks
struct.
:
kind
:
address
:
port
:
20250
:
false
:
:
"200Mi"
Note:
Note:
.
See
--config
If
--config
is provided and t

he values are not specified via the command line, the
.
Feedback
Yes
No
.
or
.
Last modified December 27, 2022 at 5:44 AM PST:
fixed typo in the docs (65be431615)
Home
Blog
Training
Partners
Community
CC BY 4.0

Assign Memory Resources to Containers and Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Custom Resourc

es
Tasks
Debug Pods
Debug Service

s
Auditing
Run Jobs
TLS
Tutorials
Securit

y
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Audit Configur

ation (v1)
Tasks
request
limit
to a
minikube
.
enable
API (
.
NAME
field
.
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
requests
:
memory
:
"100Mi"
limits
:
memory
:
"200Mi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"150M"
,
,
"1"
]
The
args
The
arguments tell the 

Container to attempt to allocate 150 MiB of memory.
=
=
=
=
...
resources
:
requests
:
memory
:
100Mi
limits
:
memory
:
200Mi
...
Run
=
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
requests
:
memory
:
"50Mi"
limits
:
memory
:
"100Mi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"250M"
,
,
"1"
]
In the
args
=
=
1
24s
=
=
lastState
:
:
:
exitCode
:
137
:
reason
:
OOMKilled
startedAt
:
null
=
kubectl describe pod memory-demo-2 --namespace=mem-example
... Normal  Created   Created container with id 66a3a20aa7980e61be4922780bf9d24d1a1d8b7395c09861225b0eba1

b1f8511
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
requests
:
memory
:
"1000Gi"
limits
:
memory
:
"1000Gi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"150M"
,
,
"1"
]
=
=
=
Events:
=
Nodes. By keeping a Pod's memory request low, you give the Pod a good chance of being
The Pod can have bursts of activ

ity where it makes use of memory that happens to be available.
Clean up
Feedback
Yes
No
.
or
.
Last modified May 28, 2022 at 8:08 PM PST:
Remove data-lang for command outputs (50b9fb29b0)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Writing a new topic | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

et Up kubectl on Linux
Debug Pods
Debug

 Running Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at th

e Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
k

ube-apiserver Encryption Configuration (v1)
This page shows how to create a new topic for the Kubernetes docs.
Create a fork of the Kubernetes documentation repository as described in
Open a PR
.
Choosing a page type
As you prepare to write a new topic, think about the page type that would fit your content the best:
Guidelines for choosing a page type
Type
Concept
A concept page explains some aspect of Kubernetes. For example, a concept page might describe the Kubernetes Deployment object and explain the role it plays as an application while it is deployed, scaled, and updated. Typically, concept pages don't include sequences of steps, but instead provide links to tasks or tutorials. For an example of a concept topic, see
Nodes
.
Task
A task page shows how to do a single thing. The idea is to give readers a sequence of steps that they can actually do as they read the page. A task page can be short or long, provided it stays focused on one area. In a task page, it is OK to blend brief explanations with the steps to be performed, but if you need to provide a lengthy explanation, you should do that in a concept topic. Related task and concept topics should link to each other. For an example of a short task page, see
. For an example of a longer task page, see
Configure Liveness and Readiness Probes
Tutorial
A tutorial page shows how to accomplish a goal that ties together several Kubernetes features. A tutorial might provide several sequences of steps that readers can actually do as they read the page. Or it might provide explanations of related pieces of code. For example, a tutorial could provide a walkthrough of a code sample. A tutorial can include brief explanations of the Kubernetes features that are being tied together, but should link to related concept topics for deep explanations of individual features.
Creating a new page
Use a
for each new page
that you write. The docs site provides templates or
Hugo archetypes
to create
new content pages. To create a new type of page, run
hugo new
with the path to the file
you want to create. For example:
hugo new docs/concepts/my-first-concept.md
Choosing a title and filename
Choose a title that has the keywords you want search engines to find.
Create a filename that uses the words in your title separated by hyphens.
For example, the topic with title
Using an HTTP Proxy to Access the Kubernetes API
has filename
http-proxy-access-api.md
. You don't need to put
"ku

bernetes" in the filename, because "kubernetes" is already in the
URL for the topic, for example:
/docs/tasks/extend-kubernetes/http-proxy-access-api/
Adding the topic title to the front matter
In your topic, put a
title
front matter
.
The front matter is the YAML block that is between the
triple-dashed lines at the top of the page. Here's an example:
---
title: Using an HTTP Proxy to Access the Kubernetes API
---
Choosing a directory
Depending on your page type, put your new file in a subdirectory of one of these:
/content/en/docs/tasks/
/content/en/docs/tutorials/
/content/en/docs/concepts/
You can put your file in an existing subdirectory, or you can create a new
subdirectory.
Placing your topic in the table of contents
The table of contents is built dynamically using the directory structure of the
documentation source. The top-level directories under
/content/en/docs/
create
top-level navigation, and subdirectories each have entries in the table of
contents.
Each subdirectory has a file
_index.md
, which represents the "home" page for
a given subdirectory's content. The
_index.md
does not need a template. It
can contain overview content about the topics in the subdirectory.
Other files in a directory are sorted alphabetically by default. This is almost
never the best order. To control the relative sorting of topics in a
subdirectory, set the
weight:
front-matter key to an integer. Typically, we
use multiples of 10, to account for adding topics later. For instance, a topic
with weight
10
will come before one with weight
20
.
Embedding code in your topic
If you want to include some code in your topic, you can embed the code in your
file directly using the markdown code block syntax. This is recommended for the
following cases (not an exhaustive list):
The code shows the output from a command such as
kubectl get deploy mydeployment -o json | jq '.status'
.
The code is not generic enough for users to try out. As an example, you can
embed the YAML
file for creating a Pod which depends on a specific
FlexVolume
implementation.
The code is an incomplete example because its purpose is to highlight a
portion of a larger file. For example, when describing ways to
customize a
,
you can provide a short snippet directly in your topic file.
The code is not meant for users to try out due to other reasons. For example,
when describing how a new attribute should be added to a resource using the
command, you can provide a short example that includes only
the attribute to add.
Including code from another file
Another way to include code in your topic is to create a new, complete sample
file (or group of sample files) and then reference the sample from your topic.
Use this method to include sample YAML files when the sample is generic and
reusable, and you want the reader to try it out themselves.
When adding a new standalone sample file, such as a YAML file, place the code in
one of the
&lt;LANG&gt;/examples/
subdirectories where
&lt;LANG&gt;
is the language for
the topic. In your topic file, use the
codenew
shortcode:
{{&lt; codenew file="&lt;RELPATH&gt;/my-example-yaml&gt;" &gt;}}
where
&lt;RELPATH&gt;
is the path to the file to include, relative to the
examples
directory. The following Hugo shortcode references a YAML
file located at
/content/en/examples/pods/storage/gce-volume.yaml
.
Note:
&lt;
and before
the
&gt;
characters. View the code for this page for an example.
Showing how to create an API object from a configuration file
If you need to demonstrate how to create an API object based on a
configuration file, place the configuration file in one of the subdirectories
under
&lt;LANG&gt;/examples
.
In your topic, show this command:
kubectl create -f https://k8s.io/examples/pods/storage/gce-volume.yaml
Note:
When

 adding new YAML files to the
directory, make
sure the file is also included into the
&lt;LANG&gt;/examples_test.go
file. The
Travis CI for the Website automatically runs this test case when PRs are
submitted to ensure all examples pass the tests.
For an example of a topic that uses this technique, see
Running a Single-Instance Stateful Application
.
Adding images to a topic
Put image files in the
/images
directory. The preferred
image format is SVG.
using page content types
.
creating a pull request
.
Feedback
Yes
No
.
or
.
Last modified October 30, 2022 at 8:07 PM PST:
Update content/en/docs/contribute/style/write-new-topic.md (c098f10604)
Home
Blog
Training
Partners
Community
CC BY 4.0

ResourceFieldSelector | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

ectl on macOS
Debug Pods
Get a Shell to a Running Contai

ner
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Acce

ss to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
k

ube-proxy Configuration (v1alpha1)
Reference
resource
(string)
divisor
(
Quantity
)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Job | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

ectl on Windows
Debug Pods
Resourc

e metrics pipeline
Auditing
Run Jobs
TLS
Tutorials
Security
Restri

ct a Container's Syscalls with seccomp
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
k

ube-scheduler Configuration (v1)
Reference
Job
Job
Job
: batch/v1
kind
: Job
metadata
(
)
spec
(
JobSpec
)
status
(
JobStatus
)
JobSpec
Replicas
template
(
(int32)
Lifecycle
(int32)
(string)
Indexed
.
Indexed
Indexed

.
(int32)
Specifies the number of retries before marking this job failed. Defaults to 6
(int64)
(int32)
suspend
(boolean)
Selector
selector
(
)
(boolean)
Specifies the action taken on a pod failure when th

e requirements are satisfied. Possible values are:
Count: indicates that the pod is handled in the default way - the
(string)
JobStatus
startTime
(Time)
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are pr

ovided for many of the factory methods that the time package offers.
(Time)
Represents time when the job was completed. It is not guaranteed to be set in happens-before order across separate operations. It is represented in RFC3339 form and is in UTC. The completion time is only set when the job finishes successfully.
active
(int32)
failed
(int32)
succeeded
(int32)
(string)
type
(Time)
(Time)
(string)
(string)
counter.
Beta level
ready
(int32)
JobList
: batch/v1
kind
: JobList
metadata
(
ListMeta
)
https://git.k8s.io/community/contribu

tors/devel/sig-architecture/api-conventions.md#metadata
items
([]
Job
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
create
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
202 (
Job
update
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch

name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Job
JobSpec
Replicas
Lifecycle
Selector
JobStatus
Beta level
JobList
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Webhook Mode | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

all and Set Up kubectl on Windows
Debug Pods
Troubleshootin

g Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resources wi

th AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configur

Reference
Webhook
Mode
Webhook
flag.
:
v1
kind
:
Config
clusters
:
-
name
:
cluster
:
:
server
:
users
:
-
name
:
user
:
:
:
:
webhook
contexts
:
-
context
:
cluster
:
user
:
name
:
webhook
enable the
).
{
:
,
"kind"
:
,
"spec"
: {
: {
:
,
"verb"
:
"get"
,
"group"
:
,
:
"pods"
},
"user"
:
"jane"
,
"group"
: [
"group1"
,
"group2"

]
}
}
status
field of
spec
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
true
}
}
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"reason"
:
}
}
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"denied"
:
true
,
"reason"
:
}
}
{
:
,
"kind"
:
,
"spec"
: {
: {
"path"
:
"/debug"
,
"verb"
:
"get"
},
"user"
:
"jane"
,
"group"
: [
"group1"
,
"group2"
]
}
}
/api
,
/apis
,
/metrics
,
/logs
,
/debug
,
/healthz
,
/livez
,
,
/readyz
, and
/version.
/api
,
/api/*
,
/apis
,
/apis/*
,
and
/version
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Using a Service to Expose Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Device

 Plugins
Tasks
T

roubleshooting Applications
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configuring a Java Micr

oservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-ap

iserver Admission (v1)
Tutorials
Learn about a Service in Kubernetes
Understand how labels and LabelSelector objects relate to a Service
Expose an application outside a Kubernetes cluster using a Service
Overview of Kubernetes Services
Pods
are mortal. Pods have a
lifecycle
. When a worker node dies, the Pods running on the Node are also lost. A
might then dynamically drive the cluster back to the desired state via the creation of new Pods to keep your application running. As another example, consider an image-processing backend with 3 replicas. Those replicas are exchangeable; the front-end system should not care about backend replicas or even if a Pod is lost and recreated. That said, each Pod in a Kubernetes cluster has a unique IP address, even Pods on the same Node, so there needs to be a way of automatically reconciling changes among Pods so that your applications continue to function.
A Service in Kubernetes is an abstraction which defines a logical set of Pods and a policy by which to access them. Services enable a loose coupling between dependent Pods. A Service is defined using YAML
(preferred)
or JSON, like all Kubernetes objects. The set of Pods targeted by a Service is usually determined by a
(see below for why you might want a Service without including a
selector
in the spec).
Although each Pod has a unique IP address, those IPs are not exposed outside the cluster without a Service. Services allow your applications to receive traffic. Services can be exposed in different ways by specifying a
type
in the ServiceSpec:
ClusterIP
(default) - Exposes the Service on an internal IP in the cluster. This type makes the Service only reachable from within the cluster.
NodePort
- Exposes the Service on the same port of each selected Node in the cluster using NAT. Makes a Service accessible from outside the cluster using
&lt;NodeIP&gt;:&lt;NodePort&gt;
. Superset of ClusterIP.
- Creates an external load balancer in the current cloud (if supported) and assigns a fixed, external IP to the Service. Superset of NodePort.
- Maps the Service to the contents of the
field (e.g.
foo.bar.example.com
), by returning a
CNAME
record with its value. No proxying of any kind is set up. This type requires v1.7 or

 higher of
kube-dns
, or CoreDNS version 0.0.8 or higher.
More information about the different types of Services can be found in the
tutorial. Also see
.
Additionally, note that there are some use cases with Services that involve not defining a
selector
in the spec. A Service created without
selector
will also not create the corresponding Endpoints object. This allows users to manually map a Service to specific endpoints. Another possibility why there may be no selector is you are strictly using
type: ExternalName
.
Summary
Exposing Pods to external traffic
Load balancing traffic across multiple Pods
Using labels
A Kubernetes Service is an abstraction layer which defines a logical set of Pods and enables external traffic exposure, load balancing and service discovery for those Pods.
Services and Labels
A Service routes traffic across a set of Pods. Services are the abstraction that allows pods to die and replicate in Kubernetes without impacting your application. Discovery and routing among dependent Pods (such as the frontend and backend components in an application) are handled by Kubernetes Services.
Services match a set of Pods using
, a grouping primitive that allows logical operation on objects in Kubernetes. Labels are key/value pairs attached to objects and can be used in any number of ways:
Designate objects for development, test, and production
Embed version tags
Classify an object using tags
Labels can be attached to objects at creation time or later on. They can be modified at any time. Let's expose our application now using a Service and apply some labels.
›
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Share a Cluster with Namespaces | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

and Set Up kubectl on Linux
Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards 

at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v1alph

a1)
Tasks
Have an
.
Pods
,
Services
, and
.
default
This namespace is created automatically and is readable by all users (including those not authenticated). This namespace is mostly reserved for cluster usage, in case that some resources should be visible and readable publicly throughout the whole cluster. The public aspect of this namespace is only a convention, not a requirement.

Namespace
to define
Hard
Namespace
a
Namespace
.
See
Active
Namespace
Note:
kube-
:
v1
kind
:
Namespace
metadata
:
name
:
Then run:
kubect

l create namespace &lt;insert-namespace-name-here&gt;
DNS label
.
design doc
.
Warning:
state.
and
.
Create the
kubectl create -f https://k8s.io/examples/admin/namespace-prod.json
=
=
=
2
snowflake
=
app
=
=
The
=
=

=
=
=
5
-n
=
=
app
=
cattle -n
=
A scope for
Names
.
As a cluster operator, I want to limit the amount of resources each community can consume in order
Service
DNS entry
.
.
See
.
Feedback
Yes
No
.
or
.
Last modified August 09, 2022 at 6:12 AM PST:
Update docs to reference new container image registry (a180cd6853)
Home
Blog
Training
Partners
Community
Case 

LocalObjectReference | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
name
(string)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Generating Reference Documentation for the Kubernetes API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggre

gation Layer
Tasks
Debug Pods
D

etermine the Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security St

andards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

er Configuration (v1)
This page shows how to update the Kubernetes API reference documentation.
The Kubernetes API reference documentation is built from the
Kubernetes OpenAPI spec
using the
kubernetes-sigs/reference-docs
generation code.
If you find bugs in the generated documentation, you need to
fix them upstream
.
If you need only to regenerate the reference documentation from the
OpenAPI
spec, continue reading this page.
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
Setting up the local repositories
Create a local workspace and set your
GOPATH
.
mkdir -p
$HOME
/&lt;workspace&gt;
export
GOPATH
=
$HOME
Get a local clone of the following repositories:
go get -u github.com/kubernetes-sigs/reference-docs
go get -u github.com/go-openapi/loads
go get -u github.com/go-openapi/spec
If you don't already have the kubernetes/website repository, get it now:
git clone https://github.com/&lt;your-username&gt;/website
$GOPATH
/src/github.com/&lt;your-username&gt;/website
Get a clone of the kubernetes/kubernetes repository as k8s.io/kubernetes:
git clone https://github.com/kubernetes/kubernetes
$GOPATH
/src/k8s.io/kubernetes
The base directory of your clone of the
kubernetes/kubernetes
repository is
$GOPATH/src/k8s.io/kubernetes.
The remaining steps refer to your base directory as
&lt;k8s-base&gt;
.
$GOPATH/src/github.com/&lt;your username&gt;/website.
&lt;web-base&gt;
.
$GOPATH/src/github.com/kubernetes-sigs/reference-docs.
&lt;rdocs-base&gt;
.
Generating the API reference docs
This section shows how to generate the
published Kubernetes API reference documen

tation
.
Setting build variables
Set
K8S_ROOT
to
.
Set
K8S_WEBROOT
to
.
Set
K8S_RELEASE
to the version of the docs you want to build.
For example, if you want to build docs for Kubernetes 1.17.0, set
to 1.17.0.
export
=
${
GOPATH
}
export
K8S_ROOT
=
${
GOPATH
}
export
=
1.17.0
Creating versioned directory and fetching Open API spec
The
updateapispec
build target creates the versioned build directory.
After the directory is created, the Open API spec is fetched from the
repository. These steps ensure that the version
of the configuration files and Kubernetes Open API spec match the release version.
The versioned directory name follows the pattern of
v&lt;major&gt;_&lt;minor&gt;
.
In the
directory, run the following build target:
cd
make updateapispec
Building the API reference docs
The
copyapi
target builds the API reference and
copies the generated files to directories in
.
Run the following command in
:
cd
make copyapi
Verify that these two files have been generated:
[
-e
"&lt;rdocs-base&gt;/gen-apidocs/build/index.html"
]
echo
"index.html built"
||
echo
"no index.html"
[
-e
"&lt;rdocs-base&gt;/gen-apidocs/build/navData.js"
]
echo
"navData.js built"
||
echo
"no navData.js"
Go to the base of your local
, and
view which files have been modified:
cd
git status
static/docs/reference/generated/kubernetes-api/v1.26/css/bootstrap.min.css
static/docs/reference/generated/kubernetes-api/v1.26/css/font-awesome.min.css
static/docs/reference/generated/kubernetes-api/v1.26/css/stylesheet.css
static/docs/reference/generated/kubernetes-api/v1.26/fonts/FontAwesome.otf
static/docs/reference/generated/kubernetes-api/v1.26/fonts/fontawesome-webfont.eot
static/docs/reference/generated/kubernetes-api/v1.26/fonts/fontawesome-webfont.svg
static/docs/reference/generated/kubernetes-api/v1.26/fonts/fontawesome-webfont.ttf
static/docs/reference/generated/kubernetes-api/v1.26/fonts/fontawesome-webfont.woff
static/docs/reference/generated/kubernetes-api/v1.26/fonts/fontawesome-webfont.woff2
static/docs/reference/generated/kubernetes-api/v1.26/index.html
static/docs/reference/generated/kubernetes-api/v1.26/js/jquery.scrollTo.min.js
static/docs/reference/generated/kubernetes-api/v1.26/js/navData.js
static/docs/reference/generated/kubernetes-api/v1.26/js/scroll.js
Updating the API reference index pages
When generating reference documentation for a new release, update the file,
&lt;web-base&gt;/content/en/docs/reference/kubernetes-api/api-index.md
with the new
version number.
Open
for editing,
and update the API reference version number. For example:
---
title: v1.17
---

[Kubernetes API v1.17](/docs/reference/generated/kubernetes-api/v1.17/)
Open
&lt;web-base&gt;/content/en/docs/reference/_index.md
for editing, and add a
new link for the latest API reference. Remove the oldest API reference version.
There should be five links to the most recent API references.
Locally test the API reference
Publish a local version of the API reference.
Verify the
local preview
.
cd
git submodule update --init --recursive --depth
1
# if not already done
make container-serve
Commit the changes
In
run
git add
and
git commit
to commit the change.
Submit your changes as a
pull request
to the
Monitor your pull request, and respond to reviewer comments as needed. Continue
to monitor your pull request until it has been merged.
Generating Reference Docs for Kubernetes Components and Tools
Feedback
Was

 this page helpful?
Yes
No
.
or
.
Last modified May 13, 2021 at 10:46 AM PST:
Fixed variable substitution typo (faa06b802a)
Home
Blog
Training
Partners
Community
CC BY 4.0

API Access Control | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
read
.
including
and
1 - Authenticating
accounts.
stored as
Secrets
as
kubectl
kubelets
or
.
or
.
Extra fields: a map of strings to li

st of strings which holds additional information authorizers may find useful.
.
The
can be accomplished using an
or the
.
openssl
See
Note:
.
kubeadm
. It is
intentionally limited to discourage 

users from using these tokens past
kubeadm
)
cluster.
Please see
kubeadm
.
different files. If unspecified, --tls-private-key-file is used.
field of a
PodSpec
.
Note:
:
apps/v1
kind
:
metadata
:
name
:
namespace
:
default
spec
:
replicas
:
3
template
:
metadata
:
# ...
spec
:
:
:
-
name
:
nginx
image
:
above
,
and
.
Warning:
ID Token
.
id_token
(not the
)
above
kube -&gt;&gt; api: 

4. Authorization: Bearer...
enabled
,
id_token
and a
When using
kubectl
, use your
id_token
with the
--token
kubectl
sends your
id_token
The API server will make sure the JWT signature is valid by checking against the certificate named in the configuration
id_token
kubectl
kubectl
id_token
The
id_token
id_token
.
Parameter
Example
Required
https://
Yes
Yes
sub
email
or
name
email
sub
No
system:
oidc:
email
where
. The value
-
oidc:
No
groups
No
system:
oidc:
and
.
oidc:
No
No
/etc/kubernet

es/ssl/kc-ca.pem
No
RS512
No
azp
others
).
dex
,
Keycloak
,
UAA
, or
.
For an identity provider to work with Kubernetes it must:
Support
CA
flag set to
TRUE
UAA
Dex
oidc
id_token
id_token
,
,
client_id
, and
id_token
\
=
oidc
\
=
=(
issuer url
)
\
=
client-id
=(
)
\
=
=(
)
\
=
=(
)
\
=
=(
)
\
=
id-token
=(
)
\
=
oidc
\
=
=
\
=
client-id
=
\
=
=
\
=
=
=
\
=
=
\
=
id-token
=
eyJraWQiOiJDTj1vaWRjaWRwLnRyZW1vbG8ubGFuLCBPVT1EZW1vLCBPPVRybWVvbG8gU2VjdXJpdHksIEw9QXJsaW5ndG9uLCBTVD1WaXJnaW5pYSwgQz1VUy1DTj1rdWJlLWNhLTEyMDIxNDc5MjEwMzYwNzMyMTUyIiwiYWxnIjoiUlMyNTYifQ.eyJpc3MiOiJodHRwczovL29pZGNpZHAudHJlbW9sby5sYW46ODQ0My9hdXRoL2lkcC9PaWRjSWRQIiwiYXVkIjoia3ViZXJuZXRlcyIsImV4cCI6MTQ4MzU0OTUxMSwianRpIjoiMm96US15TXdFcHV4WDlHZUhQdy1hZyIsImlhdCI6MTQ4MzU0OTQ1MSwibmJmIjoxNDgzNTQ5MzMxLCJzdWIiOiI0YWViMzdiYS1iNjQ1LTQ4ZmQtYWIzMC0xYTAxZWU0MWUyMTgifQ.w6p4J_6qQ1HzTG9nrEOrubxIMb9K5hzcMPxc9IxPx2K4xO9l-oFiUw93daH3m5pluP6K7eOE6txBuRVfEcpJSwlelsOsW8gb8VJcnzMS9EnZpeA0tW_p-mnkFc3VcfyXuhe5R3G7aa5d8uHv70yJ9Y3-UhjiN9EhpMdf

PAoEB9fYKKkJRzF7utTTIPGrSaSU6d2pcpfYKaxIwePzEkT4DfcQthoZdy9ucNvvLoi1DIC-UocFD8HLs8LYKEqSxQvOcvnThbObJ9af71EwmuE21fO5KzMW20KtAeget1gnldOosPtz1G5EwvaQ401-RPQzPGMVBld0_zMCAwZttJ4knw
users
:
-
name
:
mmosley
user
:
:
config
:
client-id
:
:
id-token
:
:
:
:
name
:
oidc
Once your
id_token
expires,
kubectl
id_token
using your
and
and
id_token
in your
.
--token
Option
The
kubectl
--token
id_token
=
or
v1beta1
.
clusters
users
:
v1
kind
:
Config
clusters
:
-
name
:
cluster
:
:
server
:
users
:
-
name
:
user
:
:
:
:
webhook
contexts
:
-
context
:
cluster
:
user
:
name-of-api-s

erver
name
:
webhook
above
,
and
must
Note:
token reviews, the API server must be started with
.
{
,
"kind":
,
"spec":
{
"token":
,
[
,
]
}
}
{
,
"kind":
,
"spec":
{
"token":
,
[
,
]
}
}
status
spec
{
,
"kind":
,
"status":
{
true
,
"user":
{
# Required
,
# Optional
"uid":
"42"
,
"groups":
[
,
"qa"
],
"extra":
{
[
,
]
}
},
[
]
}
}
{
,
"kind":
,
"status":
{
true
,
"user":
{
# Required
,
# Optional
"uid":
"42"
,
"groups":
[
,
"qa"
],
# This

 should not contain confidential data, as it can be recorded in logs
"extra":
{
[
,
]
}
},
[
]
}
}
{
,
"kind":
,
"status":
{
false
,
"error":
}
}
{
,
"kind":
,
"status":
{
false
,
"error":
}
}
.
Note:
.
GET
/
HTTP
/
1.1
:
fido
:
dogs
:
:
:
openid
:
profile
name
:
fido
groups
:
- dogs
extra
:
:
scopes
:
- openid
- profile
not
treated as anonymo

us requests, and given a username of
.
error.
user or the
*
user or
*
and
.
Note:
.
Note:
When using
kubectl
set the
--as
header.
Set the
--as
and
flag:
=
=
Note:
kubectl
set user and group impersonation hea

ders:
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"users"
,
"groups"
,
]
verbs
:
[
]
apiGroup
.
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
,
"uids"
]
verbs
:
[
]
of
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"users"
]
verbs
:
[
]
:
[
]
-
apiGroups
:
[
""
]
resources
:
[
"groups"
]
verbs
:
[
]
:
[
,
"admins"
]
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
]
:
[
"view"
,
]
-
apiGroups
:
[
]
resources
:
[
"uids"
]
verbs
:
[
]
:
[
]
Note:
and a
,
not a
Role
and
.
kubectl
and
kubelet
kubectl
command.
to submit a
:
v1
kind
:
Config
users
:
-
name
:
my-user
user
:
exec
:
command
:
# API version to use when decoding the ExecCredentials resource. Requi

red.
#
#
:
env
:
-
name
:
"FOO"
value
:
"bar"
args
:
-
"arg1"
-
"arg2"
:
|
...
:
true
:
Never
clusters
:
-
name
:
cluster
:
server
:
:
:
-
name
:
extension
:
arbitrary
:
config
this
:
you
:
[
"can"
,
"put"
,
,
"here"
]
contexts
:
-
name
:
context
:
cluster
:
user
:
my-user
:
:
v1
kind
:
Config
users
:
-
name
:
my-user
user
:
exec
:
command
:
#
#
:
env
:
-
name
:
"FOO"
value
:
"bar"
args
:
-
"arg1"
-
"arg2"
:
|
...
:
true
# or "Always" (this exec plugin requires standard input to f

unction). Optional.
:
Never
clusters
:
-
name
:
cluster
:
server
:
:
:
-
name
:
extension
:
arbitrary
:
config
this
:
you
:
[
"can"
,
"put"
,
,
"here"
]
contexts
:
-
name
:
context
:
cluster
:
user
:
my-user
:
,
the binary
/home/jane/bin/example-client-go-exec-plugin
-
name
:
my-user
user
:
exec
:
command
:
:
:
Never
object to
stdout
.
status
.
stdin
stdin
stdin
stdin
stdin
(see table
.
Value
Meaning
Never
Always
{
:
,
"kind"
:
,
"status"
: {
"token"
:
}
}
{
:
,
"kind"
:
,
"status"
: {
"token"
:
}
}
and
client.authentication.k8s.io

/v1beta1
{
:
,
"kind"
:
,
"status"
: {
:
,
:
}
}
{
:
,
"kind"
:
,
"status"
: {
:
,
:
}
}
RFC 3339
{
:
,
"kind"
:
,
"status"
: {
"token"
:
,
:
}
}
{
:
,
"kind"
:
,
"status"
: {
"token"
:
,
:
}
}
on the
user.exec
.
{
:
,
"kind"
:
,
"spec"
: {
"cluster"
: {
"server"
:
,
:
"LS0t..."
,
"config"
: {
:
"config"
,
"this"
:
,
"you"
: [
"can"
,
"put"
,
,
"here"
]
}
},
:
true
}
}
{
:
,
"kind"
:
,
"spec"
: {
"cluster"
: {
"server"
:
,
:
"LS0t..."
,
"config"
: {
:
"config"
,
"this"
:
,
"you"
: [
"can"
,
"put"
,
,
"here"
]
}
},
:
true
}
}
):
{
:
,
"kind"
:
}
{
:
,
"kind"
:
,
"status"
: {
: {
"name"
:
,
"uid"
:
,
"groups"
: [
"viewers"
,
"editors"
,
],
"extra"
: {
: [
]
}
}
}
}
kubectl alpha auth whoam

i
JSON
YAML
{
:
,
"kind"
:
,
"status"
: {
: {
:
,
"uid"
:
"b79dbf30-0c6a-11ed-861d-0242ac120002"
,
"groups"
: [
,
,
],
"extra"
: {
"skills"
: [
"reading"
,
],
: [
"math"
,
"sports"
]
}
}
}
}
:
kind
:
status
:
userInfo
:
username
:
jane.doe
uid
:
groups
:
- students
- teachers
extra
:
skills
:
- reading
- learning
subjects
:
- math
- sports
or
.
Note:
userInfo
including
.
Note:
the
API group
enabled.
Read the
Read the
2 - Authenticating with Bootstrap Tokens
to support
kubeadm
kubeadm
system.
(
.
. Additional gro

ups may be specified in the
here
.
:
v1
kind
:
Secret
metadata
:
name
:
namespace
:
type
:
:
:
token-id
:
07401b
:
:
:
"true"
:
"true"
:
be
The
true
The
kubeadm
in the
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
data
:
:
:
|
clusters:
- cluster:
name: ""
users: []
The
=
HS256
must
is used.
Warning:
token. When using ConfigMap signing it's

 discouraged to share the same token with
3 - Certificate Signing Requests
X.509
X.509
The CertificateSigningRequest resource type allows a client to ask for an X.509 certificate
the
field.
Note that
.
600
by a
.
the
Once the
Signers
: wh

ether it is fixed by the signer, configurable by the admin, determined by the CSR
field, etc
field.
field. In
The PKCS#10 signing request format does not have a standard mechanism to specify a
use the
(the
and
is
used.
Note:
The
:
.
The
,
.
of the
.
".
.
of the

.
".
.
of the
and later)
as
.
.
of the
Note:
Note:
The
).
Verbs:
create
,
get
,
list
,
watch
, group:
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
resources
:
verbs
:
- create
- get
- list
- watch
Verbs:
get
,
list
,
watch
, group:
Verbs:
update
, group:
Verbs:
approve
, group:
signers
or
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
resources
:
verbs
:
- get
- list
- watch
-
apiGroups
:
resources
:
verbs
:
- update
-
apiGroups
:
resources
:
- signers
resourc

eNames
:
verbs
:
- approve
Verbs:
get
,
list
,
watch
, group:
Verbs:
update
, group:
Verbs:
sign
, group:
signers
or
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
resources
:
verbs
:
- get
- list
- watch
-
apiGroups
:
resources
:
verbs
:
- update
-
apiGroups
:
resources
:
- signers
:
verbs
:
- sign
RBAC
2048
cat
metadata:
spec:
usages:
EOF
usages
has to be '
'
864000
3600
request
.
k

ubectl get csr myuser -o
jsonpath
=
=
=
get --verb
=
=
=
=
pods
=
=
myuser
=
=
myuser.crt --embed-certs
=
true
=
=
myuser
myuser
:
kubectl
and
commands.
approval
operator
Approved
or
Denied
For
Approved
CSRs:
:
kind
:
...
status
:
:
-
:
:
message
:
reason
:
type
:
Approved
For
Denied
CSRs:
:
kind
:
...
status
:
:
-
:
:
message
:
reason
:
type
:
Denied
field.
Signing
,
Note:
Note:
The
status
All PEM blocks must have the "CERTIFICATE" label, con

tain no headers,
.
sTCsV4OUfyHhx2BBLUo4g4SptHFySTBwlpRWBnSjZPOhmN74JcpTLB4J5f4iEeA7
.
:
kind
:
...
status
:
:
Read
signer
approver
RFC 5280
RFC 2986
4 - Admission Controllers Reference
An
,
mutating
not
to read (
get
,
watch
or
list
) objects.
list
both.
canonic

al example of why this is necessary. Any such side-effect needs a
and the
=
Note:
=
Note:
The
and
the
API.
because
Always
required.
approve
See
for more
This admission controller observes upda

tes to the
sign
See
for more
of
of
.
Ingress
objects that do not request any specific
Ingress
Ingress
See the
Ingress
See
the taints
and
and
or
.
and
Service
field
. This
on
existing
Service
Service
objects.
of it.
EventRateLimi

t
:
:
kind
:
plugins
:
-
name
:
path
:
...
Server
Namespace
User
:
kind
:
limits
:
-
type
:
Namespace
qps
:
50
burst
:
100
cacheSize
:
2000
-
type
:
User
qps
:
10
burst
:
50
See the
:
:
allowTTL
:
50
denyTTL
:
50
:
500
:
true
:
:
kind
:
plugins
:
-
name
:
path
:
...
:
kind
:
plugins
:
-
name
:
:
:
:
allowTTL
:
50
denyTTL
:
50
:
500
:
true
cluster
user
field
clusters
:
-
name
:
cluster
:
:
server
:
users
:
-
name
:
user
:
:
:
.
Note:
The webhook API objects are subject to the same versioning compatibility rul

es
group (
).
{
:
,
"kind"
:
,
"spec"
:{
:[
{
"image"
:
},
{
"image"
:
}
],
:{
:
},
:
}
}
status
spec
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
true
}
}
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"reason"
:
}
}
API
.
in
.
object in a
Namespace
Pods in the
default
See the
and the
must
Future changes to c

ontrol loops for built-in resources or third-party resources
Namespace
itself.
Namespace
Namespace
default
,
,
.
A
Namespace
Node
and
Pod
.
Node
Pod
Node
The
Node
or
k8s.io/
Prevents
prefix.
Node
Allows
or
k8s.io
delete
update
owner
to
true
.
apiVersio

n
:
kind
:
metadata
:
name
:
:
:
resturl
:
restuser
:
""
:
""
:
""
:
true
.
may need to add the labels manually to prevent pods from mounting volumes from
the
.
:
:
:
:
:
:
kind
:
plugins
:
-
name
:
path
:
...
to assign
:
v1
kind
:
Namespace
metadata
:
:
:
name
:
If the
Namespace
,
Note:
.
See the
The PodTolerationRestriction admission controller verifies any conflict between tolerations o

f a
:
v1
kind
:
Namespace
metadata
:
name
:
:
:
:
Priority
object in a
Namespace
. If you are
using
See the
and the
.spec
,
See also
task.
,
See
.
objects.
The
or
taints
Nodes as
NotReady
and
validatingadmissionpol

icy
and
must
flag.
(shown
here
),
flag (
).
5 - Dynamic Admission Control
,
and
.
Note:
Here
as an
See the
See the
field
empty
,
how to
.
the
.
service
code
.
You may also deploy your webhooks outside of the cluster. 

You will need to update
or
.
See the
:
kind
:
metadata
:
name
:
webhooks
:
-
name
:
rules
:
-
apiGroups
:
[
""
]
:
[
"v1"
]
:
[
"CREATE"
]
resources
:
[
"pods"
]
scope
:
:
service
:
namespace
:
name
:
caBundle
:
:
[
"v1"
]
:
None
:
5
Note:
The
scope
Note:
When using
.
Note:
timeout
rules
, the
.
flag.
:
kind
:
plugins
:
-
name
:
:
:
kind
:
:
-
name
:
:
:
kind
:
:
:
kind
:
plugins
:
-
name
:
:
:
kind
:
:
-
name
:
:
:
kind
:
:
, see the
.
See the
In the kub

eConfig file, provide the credentials:
:
v1
kind
:
Config
users
:
#
#   user: ...
#
#   user: ...
#   user: ...
#
#   user: ...
#
#   user: ...
#
#   user: ...
#
-
name
:
user
:
:
:
-
name
:
user
:
password
:
username
:
-
name
:
'*'
user
:
token
:
Request
,
with an
API group
with the
:
kind
:
webhooks
:
-
name
:
:
[
"v1"
,
"v1beta1"
]
.
object
scale
apps/v1
:
:
kind
:
request
:
uid
:
kind
:
group
:
version
:
v1
kind
:
Scale
resource
:
group
:
apps
version
:
v1
resource
:
:
scale
:
group
:
version
:
v1
kind
:
Scale
# This only differs from `resource` if 

the webhook specified `matchPolicy: Equivalent` and the
:
group
:
apps
version
:
v1
resource
:
:
scale
name
:
namespace
:
operation
:
UPDATE
userInfo
:
username
:
admin
uid
:
groups
:
extra
:
some-key
:
object
:
:
kind
:
Scale
oldObject
:
:
kind
:
Scale
options
:
:
kind
:
dryRun
:
False
Response
,
with the
response
response
uid
allowed
true
or
false
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
}
}
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
false
}
}
using the
status
See the
status
type.
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
false
,
"status"
: {
"code"
:
403
,
"message"
:
}
}
}
patch
and
patchType
patchType
is
JSONPatch
.
See
JSON patch
For
, the
patch
spe

c.replicas
would be
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
,
:
,
"patch"
:
}
}
in HTTP
Warning
Caution:
{
:
,
"kind"
:
,
: {
"uid"
:
,
"allowed"
:
true
,
: [
,
]
}
}
or
or a
.
"CREATE"
,
"UPDATE"
,
"DELETE"
,
"CONNECT"
,
or
"*"
apiGroups
""
"*"
"*"
resources
"*"
"*/*"
"pods/*"
scope
"Cluster"
,
, and
"*"
.
"*"
.
"Cluster"
"*"
,
groups
,
versions
,
resources
, and
scope
rules
Match
CREATE
or
UPDATE
apps/v1
and
and
:
:
kind
:
...
webhooks
:
-
name
:
rules
:
-
:
[
"CREATE"
,
"UPDATE"
]
apiGroups
:
[
"apps"
]
:
[
"v1"
,
"v1beta1"
]
resources
:
[
,
]
scope
:
...
Match create requests for all resourc

es (but not subresources) in all API groups and versions:
:
kind
:
webhooks
:
-
name
:
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
"*"
status
:
kind
:
webhooks
:
-
name
:
rules
:
-
:
[
"UPDATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
]
scope
:
"*"
oldObject
newObject
or a
object)
CREATE
foo: bar
:
:
kind
:
webhooks
:
-
name
:
:
:
foo
:
bar
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
"*"
See
.
The
CREATE
:
kind
:
webhooks
:
-
name
:
:
:
-
key
:
runlevel
operator
:
NotIn
values
:
[
"0"
,
"1"
]
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
CREATE
:
kind
:
webhooks
:
-
name
:
:
:
-
key
:
operator
:
In
values
:
[
"prod"
,
"staging"
]
rules
:
-
:
[
"CREATE"
]
apiGroups
:
[
"*"
]
:
[
"*"
]
resources
:
[
"*"
]
scope
:
See
(like
),
),
The
rules
Exact
or
.
Exact
means a request should be intercepted only if it exac

tly matches a specified rule.
rules
,
apps/v1
could use
:
means the
apps/v1
)
rule
apps/v1
apps/v1
object:
:
kind
:
webhooks
:
-
name
:
:
rules
:
-
:
[
"CREATE"
,
"UPDATE"
,
"DELETE"
]
apiGroups
:
[
"apps"
]
:
[
"v1"
]
resources
:
[
]
scope
:
The
.
URL
url
(
).
The
host
service
(e.g.,
host
localhost
or
127.0.0.1
as a
host
is
#...
?...
:
kind
:
webhooks
:
-
name
:
:
url
:
The
service
service
instead of
url
.
:
admissionregistra

tion.k8s.io/v1
kind
:
webhooks
:
-
name
:
:
caBundle
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
Note:
dryRun
,
None
is
dryRun
-aware).
requests:
:
kind
:
webhooks
:
-
name
:
:
Timeouts
.
:
kind
:
webhooks
:
-
name
:
:
2
(see
container
to a
pod
Never
or
IfNeeded
Never
.
Never
IfNeeded
If additional invocations result in further modifications to the object, webho

oks are not
:
kind
:
webhooks
:
-
name
:
:
IfNeeded
Ignore
or
Fail
.
Ignore
Fail
:
kind
:
webhooks
:
-
name
:
:
Fail
Fail
.
auditing
on each
At
Metadata
{
"kind":
"Event"
,
,
{
false
}"
...
}
...
}
{
,
,
false
}
{
"kind":
"Event"
,
,
{
"mutation.webhook.admission.k8s.io/round_0_index_0": "{\"c

onfiguration\":\"my-mutating-webhook-configuration.example.com\",\"webhook\":\"my-webhook-always-mutate.example.com\",\"mutated\":
true
}"
...
}
...
}
{
,
,
true
}
At
Request
{
"kind":
"Event"
,
,
{
...
}
...
}
{
,
,
,
"patch":
[
{
"op":
"add"
,
"path":
,
"value":
"yes"
}
]
}
/metrics
name
operation
CREATE
,
UPDATE
,
DELETE
and
CONNECT
.
type
admit
and
.
webhook's
is set to
Fail
.
no_error
apiserver_admission_webhook_rejection_count{error_type="calling_webhook_error",name="invalid-admission-response-webhook.example.com",operation="CREATE",rejection_

code="0",type="validating"} 1
For a
CREATE
of the
For a
CREATE
For a
CREATE
For a
CREATE
).
For a
CREATE
/
UPDATE
"env"
set,
For a
CREATE
to
Timeouts
supports.
CREATE
must
CREATE
object.
CREAT

E
pod (e.g.
"env"
label.
"env"
.
be set to
None
dryRun
set to
true
, and the
set to
. See
The
kube-dns
.
.
6 - Managing Service Accounts
A
.
minikube
examplens
.
A configuration bundle for a complex system may include definition of various se

rvice
)
adds a
to Pods,
...
-
name
:
projected
:
sources
:
-
:
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
A
A
configMap
A
Note:
are
using the
API,
Note:
using
.
the
using the
flag. The private key is used to sig

n generated service account tokens.
using the
called an
.
default
.
default
Pod's
false
:
volume
.
;
, then the
.
You use the
.
kubectl
, see
.
...
-
name
:
projected
:
:
420
sources
:
-
:
:
3607
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
A
A
configMap
A
Create additional

 API tokens
Caution:
mechanism
:
v1
kind
:
Secret
type
:
metadata
:
name
:
:
:

Data
====
token:          ...
examplens
:
v1
kind
:
metadata
:
:
:
|
{
:
"v1"
,
"kind"
:
,
:{
:{},
"name"
:
,
:
}}
:
name
:
namespace
:
examplens
:
"777"
selfLink
:
uid
:
secrets
:
-
name
:
:
v1
kind
:
metadata
:
:
:
|
{
:
"v1"
,
"kind"
:
,
:{
:{},
"name"
:
,
:
}}
:
name
:
namespace
:
examplens
:
"1026"
selfLink
:
uid
:
secrets
:
-
name
:
Clean up
examplens
.
tok

en Secrets.
the
using the
using the
.
7 - Authorization Overview
Learn more about Kubernetes authorization, including details about creating
see
.
user
- The
user
group
extra
API
/api
or
/healthz
.
get
,
list
,
create
,
update
,
patch
,
watch
,
delete
, and
.
get
,
post
,
put
, and
delete
Resource
get
,
update
,
patch
, and
delete
Namespace
API group
- The
API Group
core
API group
.
or
GET
/api
or
/healthz
would use
get
collection of resource

s:
HTTP verb
POST
create
GET, HEAD
PUT
update
PATCH
patch
DELETE
Caution:
The
get
,
list
and
watch
list
on
secrets
data
RBAC
bind
and
escalate
verbs on
roles
and
API group.
verb on
users
,
groups
, and
in the
API group.
Node
.
ABAC
ABAC Mode
.
RBAC
RBAC Mode
.
Webhook
.
kubectl
auth can-i
yes
no
no
dev-sa
dev
target
:
\
\
yes
- Like
- A review which returns the set of actions a user can 

perform within a namespace. Useful for users to quickly summarize their own access, or for UIs to hide/show actions.
spec:
EOF
is:
:
kind
:
metadata
:
:
null
spec
:
:
group
:
apps
resource
:
namespace
:
dev
verb
:
create
status
:
allowed
:
true
denied
:
false
You must include a flag in your policy to indicate which authorization module
Caution:
Caution:
in
.
.
8 - Using RBAC Authorization
API group
API server
with the
RBAC
;
=
Role
,
,
and
. You can

,
kubectl
Caution:
An RBAC
Role
or
namespace
;
and ClusterRole) because a Kubernetes object always has to be either namespaced or not namespaced;
pods
:
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"watch"
,
"list"
]
nodes
)
/healthz
)
secrets
bound
):
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"secrets"
]
verbs
:
[
"get"
,
"watch"
,
"list"
]
.
subjects
.
:
kind
:
metadata
:
name
:
read-pods
namespace
:
default
subjects
:
-
kind
:
User
name
:
jane
apiGroup
:
roleRef
:
kind
:
Role

name
:
apiGroup
:
:
# This role binding allows "dave" to read secrets in the "development" namespace.
kind
:
metadata
:
name
:
#
namespace
:
subjects
:
-
kind
:
User
name
:
dave
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
manager
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
roleRef
roleRef
Making
roleRef
update
roleRef
The
See
pods
pods
log
is a
pods
/
) to
pods
and
log
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"pods"
,
]
verb

s
:
[
"get"
,
"list"
]
list.
get
or
update
a
ConfigMap
named
:
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
]
:
[
]
verbs
:
[
"update"
,
"get"
]
Note:
create
or
For
create
, this limitation is because the name of the new object may not be known at authorization time.
list
or
watch
list
or
watch
resources
and
verbs
*
For
*
apiGroups
and
role).
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
"*"
]
verbs
:
[
"*"
]
Caution:
The
You can
aggregate
set. The
selector
rules
Caution:
rules
field of an
objects
.
:
kind
:
metadata
:
name
:
:
:
-
:
:
"true"
rules
:
[]
.
:
kind
:
metadata
:
name
:
labels
:
:
"true"
rules
:
-
apiGroups
:
[
""
]
resources
:
[
,
,
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
The
use ClusterRole aggregation

. This lets you,
:
kind
:
metadata
:
name
:
labels
:
:
"true"
:
"true"
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
---
kind
:
:
metadata
:
name
:
labels
:
:
"true"
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
the
rules
section.
"pods"
API Group
:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
"apps"
rules
:
-
apiGroups
:
[
"apps"
]
#
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
"batch"
API group:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
-
apiGroups
:
[
"batch"
]
#
resources
:
[
"jobs"
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
]
:
[
]
verbs
:
[
"get"
]
"nodes"
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"nodes"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
/healthz
and
rules
:
-
:
[
,
]
verbs
:
[
"get"
,
"post"
]
.
Caution:
The prefix
system:
is reserved for Kubernetes system use, so you shoul

d ensure
system:
by
accident.
system:
with
.
Note:
show the
subjects
section.
:
subjects
:
-
kind
:
User
name
:
apiGroup
:
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
name
:
default
namespace
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
-
kind
:
Group
name
:
apiGroup
:
system:
.
Caution:
that have a
system:
prefix.
false
.
kubectl
run:
Note:
via
group
Allows a user read-only access to basic information about themselves. Prior to v1.14, this role was also bound t

o
group
and
groups
system:
admin
,
edit
,
view
).
User-facing ClusterRoles use
admin
,
edit
, or
view
metadata
:
labels
:
:
"true"
:
"true"
:
"true"
group
admin
None
.
If used in a
.
edit
None
.
view
None
user
scheduler
user
user
.
None
.
and
The
sys

tem:node
user
None
None
Heapster
None
kube-dns
namespace
kube-dns
None
None
.
None
None
.
group
/healthz
,
/livez
,
/readyz
,
/livez/*
,
/readyz/*
), and
/metrics
runs
.
escalate
verb on the
roles
or
API group.
user-1
containing that permission. To allow a user to create/update role

s:
Role
or
escalate
verb on
roles
or
API group
or
if you have been authorized to perform the
bind
user-1
bind
user-1
admin
,
edit
, and
view
:
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"create"
]
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"bind"
]
:
[
"admin"
,
"edit"
,
"view"
]
---
:
kind
:
metadata
:
name
:
namespace
:
roleRef
:
apiGroup
:
kind
:
name
:
subjects
:
-
apiGroup
:
kind
:
User
name
:
user-1
get
,
watch
and
list
on pods:
=
get --verb
=
=
=
pods
=
=
=
=
=
=
=
=
=
=
=
get
,
watch
and
list
on pods:
=
=
pods
=
=
=
=
Create a Clu

sterRole named "foo" with apiGroups specified:
=
=
=
=
"foo"
--verb
=
=
/logs/*
=
Within the namespace "acme", grant the permissions in the "admin" ClusterRole to a user named "bob":
=
=
=
acme
=
=
=
acme
=
=
=
acme
=
=
root
=
=
=
=
Examples:
namespace
This

 requires the application to specify a
, etc.).
\
=
view
\
=
\
=
Note:
.
For example, grant read-only permission within "my-namespace" to the "default" service account:
\
=
view
\
=
\
=
Many
add-ons
run as the
Caution:
\
=
\
=
\
=
view
\
--group
=
\
=
\
=
view
\
--group
=
Warning:
\
=
\
--group
=
,
CVE
includes
:
kind
:
metadata
:
:
:
|-
intended to prevent/isolate ac

cess to those backends.
labels
:
:
"true"
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
]
verbs
:
[
"create"
,
"delete"
,
,
"patch"
,
"update"
]
namespace
the
:
allowed by
either
(
or
--v=5
RBAC
).
Warning:
ALL
\
=
\
--user
=
admin
\
--user
=
kubelet
\
--group
=
9 - Using ABAC Authorization
To enable
ABAC
and
. There
kind
spec
user
user
group
group
Resource-matching pr

operties:
apiGroup
Ex:
apps
,
Wildcard:
*
namespace
Ex:
Wildcard:
*
resource
Ex:
pods
,
Wildcard:
*
Ex:
/version
or
/apis
Wildcard:
*
/foo/*
/foo/
.
readonly
Note:
(e.g. empty string, 0, false). However, unset should be preferred for
"*"
.
.
"*"
.
Kubectl
/api
and
/apis
.
examples
below):
/api
,
/api/*
,
/apis
, and
/apis/*
/version
.
Examples
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"alice"
,
:
"*"
,
:
"*"
,
:
"*"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"kubelet"
,
:
"*"
,
:
"pods"
,
:
true
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"kubelet"
,
:
"*"
,
:
"events"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"bob"
,
:
,
:
"pods"
,
:
true
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"group"
:
,
:
true
,
:
"*"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"group"
:
,
:
true
,
:
"*"
}}
Every service account has a corresponding ABAC username, and that service account's username is generated according to the naming con

vention:
{
:
,
"kind"
:
"Policy"
,
"spec"
:{
"user"
:
,
:
"*"
,
:
"*"
,
:
"*"
}}
10 - Using Node Authorization
Overview
services
endpoints
nodes
pods
events
.
.
must
hostname
hostname
and the
option.
.
.
group
Node
group,
format.
Node
The
RBAC
Node
to limit node access to the API, that c

an be done non-disruptively:
Enable the
Node
) and the
Node
NODE DENY
Delete the
In 1.6, the
.
Node
and
RBAC
11 - Mapping PodSecurityPolicies to Pod Security Standards
.
Baseline
and
.
under the
.spec
Type
:
false
below.
Baseline
: subset of
CHOWN
FOWNER
FSETID
KILL
MKNOD
SETFCAP
SETGID
SETPCAP
SETUID
only
Baseline
ALL
volumes
Baseline
hostPath
*
: subset of
configMap
csi
emptyDir
ephemeral
projected
secret
:
false
hostPorts
hostPID
:
false
hostIPC
:
false
seLinux
:
is
MustRunAs
options
user
is unset (
""
role
is unset (
""
type
level
runAsUser
Baseline
: Anything
:
rule
is
No opinion
No opinion
fsGroup
No opinion
No opinion
Mutating
false
Baseline
:
false
No opinion

 (volumes takes precedence)
No opinion
:
Mutating
No opinion
No opinion
The
Type
Mutating
No opinion
Baseline
:
:
Mutating
No opinion
Baseline
:
:
12 - Webhook Mode
Webhook
Mode
Webhook
flag.
:
v1
kind
:
Config
clusters
:
-
name
:
cluster
:
:
server
:
users
:
-
name
:
user
:
:
:
:
webhook
contexts
:
-
context
:
cluster
:
user
:
name
:
webhook
enable the
).
{
:
,
"kind"
:
,
"spec"
: {
: {
:
,
"verb"
:
"get"
,
"group"
:
,
:
"pods"
},
"user"
:
"jane"
,
"group"
: [
"group1"
,
"group2"
]
}
}
status
field of
the request and re

spec
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
true
}
}
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"reason"
:
}
}
{
:
,
"kind"
:
,
"status"
: {
"allowed"
:
false
,
"denied"
:
true
,
"reason"
:
}
}
{
:
,
"kind"
:
,
"spec"
: {
: {
"path"
:
"/debug"
,
"verb"
:
"get"
},
"user"
:
"jane"
,
"group"
: [
"group1"
,
"group2"
]
}
}
/api
,
/apis
,
/metrics
,
/logs
,
/debug
,
/healthz
,
/livez
,
,
/readyz
, and
/version.
/api
,
/api/*
,
/apis
,
/apis/*
,
and
/version
.
13 - Kubelet authentication/authorization
Overview
.
flag
and
flags
see the
ensure the
and
flags
client certificate auth is enabl

ed, but only some of the client certificates signed by the configured CA should be allowed to use the kubelet API
ensure the
and the
flags
HTTP verb
POST
create
GET, HEAD
get
PUT
update
PATCH
patch
DELETE
delete
resource
/stats/*
nodes
stats
nodes
metrics
/logs/*
nodes
log
/spec/*
nodes
spec
all others
nodes
proxy
Node
and
14 - TLS bootstrapping
found
here
.
file
file
a cluster.
In the bootstrap initialization process, the following occu

rs:
not
have a
file
file
kubectl
Certificate is issued to the kubelet
kubelet
(key).
group
.
should ensure tokens are bound to a Role Based A

ccess Control (RBAC) policy
here
From the kubelet's perspective, one token is like another and has no special meaning.
type
,
namespace
and
name
here
.
head -c
16
' '
.
Add the
here
for
to create a
.
.
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
cfssl
Since these signed certificates will, in turn, be used

 by the kubelet to authenticate as a regular kubelet to kube-apiserver, it is important that the CA
=
=
=
=
The validity duration of signed certificates can be configured with flag:
Approval
.
.
to the
:
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
to the
that
:
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
The
API
to
A path to a
should be in a path available to the kubelet, for exa

mple
.
:
v1
kind
:
Config
clusters
:
-
cluster
:
:
server
:
name
:
bootstrap
contexts
:
-
context
:
cluster
:
bootstrap
user
:
name
:
bootstrap
:
bootstrap
:
{}
users
:
-
name
:
user
:
token
:
server
token
The format of the token does not matter, as long as it matches what kube-apiserver expects. In the above example, we used a bootstrap token.
any
is
a standard
kubectl
does not
is used
.
client
serving
and
flags
.
can
beta
field of
Enabling
both
and
of
the
or pass the following comman

d line argument to the kubelet (deprecated):
Note:
serving
security
reasons
. To use
and
contains
)
contains
,
and
only have IP and DNS subjectAltNames that belong to the requesting node,
in
to verify
)
manager.
and
.
15 - Validating Admission Policy
The
A
owner
Pods
owner
label must be set to

 something that ends in
").
At least a
If a
in
unset.
Ensure the
Validating Admission Policy is part of the cluster control-plane. You should write and deploy them
:
kind
:
metadata
:
name
:
spec
:
:
Fail
:
:
-
apiGroups
:
[
"apps"
]
:
[
"v1"
]
:
[
"CREATE"
,
"UPDATE"
]
resources
:
[
]
:
-
:
field.
:
kind
:
metadata
:
name
:
spec
:
:
:
:
:
:
test
:
kind
:
metadata
:
name
:
spec
:
:
Fail
paramKind
:
:
kind
:
:
:
-
apiGroups
:
[
"apps"
]
:
[
"v1"
]
:
[
"CREATE"
,
"UPDATE"
]
resources
:
[
]
:
-
:
reason
:
Invalid
The
e.g.
.
ConfigMap
The
field.
The validating admission policy au

thor is responsible for providing the ReplicaLimit parameter CRD.
:
kind
:
metadata
:
name
:
spec
:
:
paramRef
:
name
:
:
:
:
:
test
:
kind
:
metadata
:
name
:
:
3
This policy parameter resource limits deployments to a max of 3 replicas in all namespaces in the
:
kind
:
metadata
:
name
:
spec
:
:
paramRef
:
name
:
:
:
:
-
key
:
operator
:
NotIn,
values
:
[
"test"
]
:
kind
:
metadata
:
name
:
:
100
:
kind
:
metadata
:
name
:
spec
:
:
params
:
:
:
:
-
key
:
operator
:
Exists
:
has()
.
If
If
read
paramKind
in
and
paramRef
in
.
paramKind
read
ad

mission policy are handled. Allowed values are
Ignore
or
Fail
.
Ignore
Fail
:
:
kind
:
spec
:
...
:
Ignore
:
-
:
CEL expressions have access to the contents of the Admission request/response, organized into CEL
.
null if
ParamKind
is unset.
The
,
kind
,
and
__
__dot__
.
__dash__
-
__slash__
/
Note:
A
int
namespace
x-prop
redact__d
string
- 'set':
X + Y
X
Y
- 'map':
X + Y
X
Y
X
and
Y
Y
with
Purpose
Validate t

hat an object's name has the prefix of another field value
Read
HTTP response code, are used in the HTTP response to the client.
,
Forbidden
,
Invalid
,
.
If not set,
Home
Blog
Training
Partners
Community
CC BY 4.0

Handling retriable and non-retriable pod failures with Pod failure policy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggregati

on Layer
Tasks
Debug Pods
Deter

mine the Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

rds at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver C

Tasks
Run Jobs
This document shows you how to use the
Pod failure policy
,
in combination with the default
Pod backoff failure policy
,
to improve the control over the handling of container- or Pod-level failure
within a
Job
.
The definition of Pod failure policy may help you to:
better utilize the computational resources by avoiding unnecessary Pod retries.
avoid Job failures due to Pod disruptions (such
preemption
,
or
taint
-based eviction).
You should already be familiar with the basic use of
Job
.
minikube
.
Using Pod failure policy to avoid unnecessary Pod retries
With the following example, you can learn how to use Pod failure policy to
avoid unnecessary Pod restarts when a Pod failure indicates a non-retriable
software bug.
First, create a Job based on the config:
/controllers/job-pod-failure-policy-failjob.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
job-pod-failure-policy-failjob
spec
:
:
8
:
2
template
:
spec
:
:
Never
:
-
name
:
main
image
:
docker.io/library/bash:5
command
:
[
"bash"
]
args
:
- -c
- echo "Hello world! I'm going to exit with 42 to simulate a software bug." &amp;&amp; sleep 30 &amp;&amp; exit 42
:
6
:
rules
:
-
action
:
FailJob
onExitCodes
:
:
main
operator
:
In
values
:
[
42
]
by running:
kubectl create -f job-pod-failure-policy-failjob.yaml
After around 30s the entire Job should be terminated. Inspect the status of the Job by running:
jobs
-l job-name
=
job-pod-failure-policy-failjob -o yaml
In the Job status, see a job
Failed
condition with the field
reason
equal
PodFailurePolicy
. Additionally, the
message
field contains a
more detailed information about the Job termination, such as:
Container main for pod d

efault/job-pod-failure-policy-failjob-8ckj8 failed with exit code 42 matching FailJob rule at index 0
.
For comparison, if the Pod failure policy was disabled it would take 6 retries
of the Pod, taking at least 2 minutes.
Clean up
Delete the Job you created:
kubectl delete jobs/job-pod-failure-policy-failjob
The cluster automatically cleans up the Pods.
Using Pod failure policy to ignore Pod disruptions
ignore Pod disruptions from incrementing the Pod retry counter towards the
.spec.backoffLimit
limit.
Caution:
Timing is important for this example, so you may want to read the steps before
execution. In order to trigger a Pod disruption it is important to drain the
node while the Pod is running on it (within 90s since the Pod is scheduled).
Create a Job based on the config:
/controllers/job-pod-failure-policy-ignore.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
job-pod-failure-policy-ignore
spec
:
:
4
:
2
template
:
spec
:
:
Never
:
-
name
:
main
image
:
command
:
[
"bash"
]
args
:
- -c
- echo "Hello world! I'm going to exit with 0 (success)." &amp;&amp; sleep 90 &amp;&amp; exit 0
:
0
:
rules
:
-
action
:
Ignore
onPodConditions
:
-
type
:
DisruptionTarget
kubectl create -f job-pod-failure-policy-ignore.yaml
Run this command to check the
nodeName
the Pod is scheduled to:
nodeName
=
$(
kubectl get pods -l job-name
=
job-pod-failure-policy-ignore -o
jsonpath
=
'{.items[0].spec.nodeName}'
)
Drain the node to evict the Pod before it completes (within 90s):
kubectl drain nodes/
$nodeName
--ignore-daemonsets --grace-period
=
0
Inspect the
.status.failed
to check the counter for the Job is not incremented:
jobs
=
job-pod-failure-policy-ignore -o yaml
Uncordon the node:
kubectl uncordon nodes/
$nodeName
The Job resumes and succeeds.
For comparison, if the Pod failure policy was disabled the Pod disruption would
result in terminating the entire Job (as the
is set to 0).
kubectl delete jobs/job-pod-failure-policy-ignore
Alternatives
You could rely solely on the
,
by specifying the Job's
field. However, in many situations
it is problematic to find a balance between setting a low value for
to avoid unnecessary Pod retries, yet high enough to make sure the Job would
not be terminated by Pod disruptions.
Feedback
Yes
No
.
or
.
Last modified November 10, 2022 at 3:58 PM PST:
Update of the docs for Beta (163e53de42)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Debug Services | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

bectl on macOS
Debug Pods
Get a Shell to a Running Conta

iner
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Acc

ess to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
Note:
kubectl
exec
Setup
=
kubectl
Let's scale the deployment to 3 replicas.
=
3
YAML:
:
apps/v1
kind
:
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
:
app
:
hostnames
replicas
:
3
template
:
metadata
:
labels
:
app
:
hostnames
spec
:
:
-
name
:
hostnames
image
:
app
=
hostnames
app
=
hostnames
\
=
vi

a HTTP on port 9376, but if you are debugging your own app, you'll want to
for
do
wget -qO-
$ep
done
there.
=
80
=
9376
:
v1
kind
:
Service
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
app
:
hostnames
ports
:
-
name
:
default
protocol
:
TCP
port
:
80
:
9376
traffic to
name.

Name:      hostnames

Note:
Address

:        10.0.0.10#53

The
kubelet
with the
flag.
The
search
kubelet
with the
commands.
The
options
ndots

section
above).
for
i in
$(
seq
1
3
)
;
do
done
{
"kind"
:
"Service"
,
:
"v1"
,
: {
"name"
:
,
:
"default"
,
"uid"
:
,
:
"347189"
,
:
,
"labels"
: {
"app"
:
}
},
"spec"
: {
"ports"
: [
{
"name"
:
"default"
,
:
"TCP"
,
"port"
:
80
,
:
9376
,
:
0
}
],
: {
"app"
:
},
:
,
"type"
:
,
:
"None"
},
"status"
: {
: {}
}
}
?
Is the
protocol
kube

ctl get pods -l
app
=
hostnames
The
debug pods
.
ENDPOINTS
column is
the
, but the
the
above.
Note:
for
do
wget -qO-
$ep
done
cluster does not use kube-proxy, the following sections will not apply, and you
I1027 22:14:54.038209    5063 proxier.go:352] Setting end

points for "default/kubernetes:https" to [10.240.0.2:443]
required
conntrack
conntrack
on Ubuntu)
line
and
one
and one
chain with
IPVS mode
...
...
,
,
).
If this s

till fails, look at the
with the
-v
iptables
Kubelet
exposes a
flag
or
.
Confirm
is set to
or
.
is set to
match
hairpin
Kubelet
has
/sys
for
do
cat
$intf
done
1
1
1
1
, ensure
Kubelet
cbr0
bridge is
Seek help
If you get this far, something very strange is happening. Your Service is
Slack
or
Forum
or
GitHub
.
Visit the
Feedback
Yes
No
.
or
.
Last modified December 05, 2022 at 10:36 AM PST:
Remove references to kube-proxy userspace mode (37ee1e335c)
Setup
Edge cas

e: A Pod fails to reach itself via the Service IP
Seek help
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
7:
8:
Role
9:
1 - LocalSubjectAccessReview
kind
metadata
(
)
spec
(
status
(
)
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
2 - SelfSubjectAccessReview
kind
metadata
(
)
spec
(
status
(
)
(string)
(string)
(string)
(string)

(string)
(string)
(string)
(string)
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
3 - SelfSubjectRulesReview
kind
metadata
(
)
spec
(
status
s are allowed, but only as the full, final step in t

he path. "
Resources is a list of resources this rule applies to. "
"
(string)
namespace
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
4 - SubjectAccessReview
kind
metadata
(
)
spec
(
status
(
)
extra
groups
(string)
(string)
(string)
Group is the API Group of th

e Resource. "*" means all.
(string)
(string)
(string)
(string)
(string)
(string)
uid
(string)
user
(string)
User is the user you're testing for. If you specify "User" but not "Groups", then is it interpreted as "What if User were not a member of any groups
allowed
denied
(boolean)
(string)
reason
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5 - SelfSubjectReview v1alpha1
kind
metadata
(
)
status
(
)
userInfo
(string)
(string)

create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
6 - ClusterRole
kind
metadata
(
)
([]
)
rules
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolea

n
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7 - ClusterRoleBinding
kind
metadata
(
)
roleRef
subjects
Kind of object being referenced. Values defined by this API group are "User", "Group", and "ServiceAccount". If the Authorizer does not r

ecognized the kind value, the Authorizer should report an error.
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
fieldSe

lector
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8 - Role
Role
kind
: Role
metadata
(
)
rules
RoleList
kind
: RoleList
metadata
(
ListMeta
)
items
([]
Role
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): intege

r
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
create
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
202 (
Role
update
name
(
in path
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
9 - RoleBinding
kind
metadata
(
)
roleRef
subjects
Subjects holds references to the

 objects the role applies to.
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/rbac.authorizatio

n.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
DELETE /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

Working with Kubernetes Objects | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Cus

tom Resources
Tasks
Debug Pods
De

bug Services
Auditing
Run Jobs
TLS
Tutorials
Configuring Redis using a Config

Map
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Aud

it Configuration (v1)
Concepts
Overview
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Debugging DNS Resolution | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
minikube
addon
kube-dns.
.
:
v1
kind
:
Pod
metadata
:
name
:
dnsutils
namespace
:
default
spec
:
:
-
name
:
dnsutils
image
:
command
:
- sleep
-
:
:
Always
Note:
default
.
nslookup
kubectl
exec

If the
nslookup
(See
and
kubectl
exec
Errors such as the following indicate a problem with the CoreDNS (or kube-dns)
kubectl
exec

nslookup: can

't resolve 'kubernetes.default'
or
kubectl
exec

Use the
=
=
kube-dns
...
...
Note:
k8s-app
is
kube-dns
Use the
=
=
kube-dns
.:53
command.
=
...
...
Note:
kube-dns
for
command.
=
log
ConfigMap
named
coredns
Then add
log
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
log
errors
health
upstream
}
cache 30
loop
reload
}
.:53
172.17.0.18:41675 - [07/Sep/2018:15:29:11 +0000] 59925 "A IN kubernetes.default.svc.cluster.local. udp 54 false 512" NOERROR qr,aa,rd,ra 106 0.000066649s
service
and
endpoint
related resources to properly resolve service

 names.
:
...
  verbs:
  - list
  - watch
...
kubectl
exec
kubectl
exec
.
flag
(With
, this is
).
dnsmasq
flag.
See
.
Read
Feedback
Yes
No
.
or
.
Last modified October 18, 2022 at 10:45 PM PST:
Add DNS search domain list limits and remove resolved DNS issue (a2b4c34eca)
Home
Blog

Training
Partners
Community
CC BY 4.0

Using ABAC Authorization | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

(v1)
Reference
To enable
ABAC
and
. There
kind
spec
user
user
group
group
apiGroup
Ex:
apps
,
Wildcard:
*
namespace
Ex:
Wildcard:
*
resource
Ex:
pods
,
Wildcard:
*
Ex:
/version
or
/apis
Wildcard:
*
/foo/*
/foo/
.
readonly
Note:
"*"
will match any value of the corresponding attri

bute.
.
.
"*"
.
Kubectl
/api
and
/apis
.
examples
below):
/api
,
/api/*
,
/apis
, and
/apis/*
/version
.
Examples
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"alice"
,
:
"*"
,
:
"*"
,
:
"*"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"kubelet"
,
:
"*"
,
:
"pods"
,
:
true
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"kubelet"
,
:
"*"
,
:
"events"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"bob"
,
:
,
:
"pods"
,
:
true
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"group"
:
,
:
true
,
:
"*"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"group"
:
,
:
true
,
:
"*"
}}
{
:
,
"kind"
:
"Policy"
,
"spec"
:{
"user"
:
,
:
"*"
,
:
"*"
,
:
"*"
}}
Feedback
Yes
No
.
or
.
Last modified August 22, 2022 at 5:43 PM PST:
Fix typos on /access-authn-authz (61a5b7b69d)
Kubectl
Examples
Home
Blog
Training
Partners
Community
© 2023 The Kubernetes Authors | Documentat

ion Distributed under
CC BY 4.0

Inject Data Into Applications | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Specify configuration and other data for the Pods that run your workload.
1:
2:
3:
4:
5:
6:
1 - Define a Command and Arguments for a Container
This page shows how to define commands and arguments when you run a container
in a
Pod
.
minikube
.
Define a command and arguments when you create a Pod
When you create a Pod, you can define a command and arguments for the
containers that run in the Pod. To define a command, include the
command
field in the configuration file. To define arguments for the command, include
the
args
field in the configuration file. The command and arguments that
you define cannot be changed after the Pod is created.
The command and arguments that you define in the configuration file
override the default command and arguments provided by the container image.
If you define args, but do not define a command, the default command is used
with your new arguments.
Note:
The
command
field corresponds to
entrypoint
in some container runtimes.
file for the Pod defines a command and two arguments:
pods/commands.yaml
:
v1
kind
:
Pod
metadata
:
name
:
command-demo
labels
:
purpose
:
demonstrate-command
spec
:
:
-
name
:
command-demo-container
image
:
debian
command
:
[
"printenv"
]
args
:
[
"HOSTNAME"
,
"KUBERNETES_PORT"
]
:
OnFailure
kubectl apply -f https://k8s.io/examples/pods/commands.yaml
The output shows that the container that ran in the command-demo Pod has
completed.
To see the output of the command that ran in the container, view the logs
from the Pod:
kubectl logs command-demo
The output shows the values of the HOSTNAME and KUBERNETES_PORT environment
tcp://10.3.240.1:443
Use environment variables to define arguments
In the preceding example, you defined the arguments directly by
providing strings. As an alternative to providing strings directly,
you can define arguments by using environment variables:
env
:
-
name
:
MESSAGE
value
:
command
:
[
]
args
:
[
"$(MESSAGE)"
]
This means you can define an argument for a Pod using any of
the techniques available for defining environment variables, including
and
Secrets
.
Note:
The environment variable appears in parentheses,
"$(VAR)"
. This is
required for the variable to be expanded in the
command
or
args
field.
Run a command in a shell
In some cases, you need your command to run in a shell. For example, your
command might consist of several commands piped together, or it might be a shell
script. To run your command in a shell, wrap it like this:
command:
[
"/bin/sh"
]
args:
[
"-c"
,
"while true; do echo hello; sleep 1

0;done"
]
configuring pods and containers
.
running commands in a container
.
See
Container
.
2 - Define Dependent Environment Variables
minikube
value
of
env
Pod:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
args
:
command
:
- sh
- -c
image
:
env
:
-
name
:
value
:
"80"
-
name
:
value
:
-
name
:
value
:
-
name
:
PROTOCOL
value
:
"https"
-
name
:
value
:
-
name
:
value
:
and skip dependent references of
.
case.
env
The
$
, ie:
.
.
See
.
3 - Define Environment Variables for a Container
This page shows how to define environment variables for a container
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes tha

t are not acting as control plane hosts. If you do not already have a
minikube
Define an environment variable for a container
When you create a Pod, you can set environment variables for the containers
that run in the Pod. To set environment variables, include the
env
or
envFrom
field in the configuration file.
file for the Pod defines an environment variable with name
DEMO_GREETING
and
value
"Hello from the environment"
. Here is the configuration manifest for the
Pod:
pods/inject/envars.yaml
:
v1
kind
:
Pod
metadata
:
name
:
envar-demo
labels
:
purpose
:
demonstrate-envars
spec
:
:
-
name
:
envar-demo-container
image
:
env
:
-
name
:
value
:
-
name
:
DEMO_FAREWELL
value
:
"Such a sweet sorrow"
kubectl apply -f https://k8s.io/examples/pods/inject/envars.yaml
purpose
=
envar-demo      1/1       Running   0          9s
List the Pod's container environment variables:
kubectl
exec
envar-demo -- printenv
NODE_VERSION=4.4.2
EXAMPLE_SERVICE_PORT_8080_TCP_ADDR=10.3.245.237
HOSTNAME=envar-demo
...
DEMO_GREETING=Hello from the environment
DEMO_FAREWELL=Such a sweet sorrow
Note:
The environment variables set using the
env
or
envFrom
field
override any environment variables specified in the container image.
Note:
Environment variables may reference each other, however ordering is important.
Variables making use of others defined in the same context must come later in
the list. Similarly, avoid circular references.
Using environment variables inside of your config
Environment variables that you define in a Pod's configuration can be used
elsewhere in the configuration, for example in commands and arguments that
you set for the Pod's containers.
In the example configuration below, the
GREETING
,
HONORIFIC
, and
NAME
environment variables are set to
Warm greetings to
,
The Most Honorable
, and
, respectively. Those environment variables
are then used in the CLI arguments passed to the
env-print-demo
:
v1
kind
:
Pod
metadata
:
name
:
print-greeting
spec
:
:
-
name
:
image
:
bash
env
:
-
name
:
GREETING
value
:
"Warm greetings to"
-
name
:
HONORIFIC
value
:
"The Most Honorable"
-
name
:
NAME
value
:
"Kubernetes"
command
:
[
"echo"
]
args
:
[
"$(GREETING) $(HONORIFIC) $(NAME)"
]
Upon creation, the command
echo Warm greetings to The Most Honorable Kubernetes
is run on the container.
.
using secrets as environment variables
.
See
.
4 - Expose Pod Information to Containers Through Environment Variables
This page shows how a Pod can use environment variables to expose information
about itself to containers running in the Pod, using the
.
You can use environment variables to expose Pod fields, container fields, or both.
In Kubernetes, there are two ways to expose Pod and container fields to a running container:
, as explained in this task
Volume files
Together, these two ways of exposing Pod and container fields are called the
downward API.
minikube
Use Pod fields as values for 

In this part of exercise, you create a Pod that has one container, and you
project Pod-level fields into the running container as environment variables.
pods/inject/dapi-envars-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
dapi-envars-fieldref
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
- while true; do
echo -en '\n';
printenv MY_NODE_NAME MY_POD_NAME MY_POD_NAMESPACE;
printenv MY_POD_IP MY_POD_SERVICE_ACCOUNT;
sleep 10;
done;
env
:
-
name
:
MY_NODE_NAME
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
MY_POD_NAME
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
MY_POD_NAMESPACE
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
MY_POD_IP
valueFrom
:
fieldRef
:
fieldPath
:
status.podIP
-
name
:
MY_POD_SERVICE_ACCOUNT
valueFrom
:
fieldRef
:
fieldPath
:
:
Never
In that manifest, you can see five environment variables. The
env
field is an array of
environment variable definitions.
The first element in the array specifies that the
variable gets its value from the Pod's
field. Similarly, the
other environment variables get their names from Pod fields.
Note:
The fields in this example are Pod fields. They are not fields of the
container in the Pod.
kubectl apply -f https://k8s.io/examples/pods/inject/dapi-envars-pod.yaml
Verify that the container in the Pod is running:
# If the new Pod isn't yet healthy, rerun this command a few times.
View the container's logs:
kubectl logs dapi-envars-fieldref
The output shows the values of selected environment variables:
minikube
default
172.17.0.4
default
To see why these values are in the log, look at the
command
and
args
fields
in the configuration file. When the container starts, it writes the values of
five environment variables to stdout. It repeats this every ten seconds.
Next, get a shell into the container that is running in your Pod:
kubectl
exec
-it dapi-envars-fieldref -- sh
In your shell, view the environment variables:
# Run this in a shell inside the container
printenv
The output shows that certain environment variables have been assigned the
values of Pod fields:
MY_POD_SERVICE_ACCOUNT=default
...
MY_POD_NAMESPACE=default
MY_POD_IP=172.17.0.4
...
MY_NODE_NAME=minikube
...
MY_POD_NAME=dapi-envars-fieldref
Use container fields as values for environment variables
In the preceding exercise, you used information from Pod-level fields as the values
for environment variables.
In this next exercise, you are going to pass fields that are part of the Pod
definition, but taken from the specific
container
rather than from the Pod overall.
Here is a manifest for another Pod that again has just one container:
pods/inject/dapi-envars-container.yaml
:
v1
kind
:
Pod
metadata
:
name
:
dapi-envars-resourcefieldref
spec
:
:
-
name
:
image
:
registry.k8s.io/busybox:1.24
command
:
[
"sh"
,
"-c"
]
args
:
printenv MY_CPU_REQUEST MY_CPU_LIMIT;
printenv MY_MEM_REQUEST MY_MEM_LIMIT;
sleep 10;
done;
resources
:
requests
:
memory
:
"32Mi"
cpu
:
"125m"
limits
:
memory
:
"64Mi"
cpu
:
"250m"
env
:
-
name
:
MY_CPU_REQUEST
valueFrom
:
:
:
resource
:
-
name
:
MY_CPU_LIMIT
valueFrom
:
:
:
resource
:
-
name
:
MY_MEM_REQUEST
valueFrom
:
:
:
resource
:
-
name
:
MY_MEM_LIMIT
valueFrom
:
:
:
resource
:
:
Never
In this manifest, you can see four environment variables. The
env
variable gets its value from the
field

 of a container named
. Similarly, the other environment variables get their values
from fields that are specific to this container.
kubectl apply -f https://k8s.io/examples/pods/inject/dapi-envars-container.yaml
kubectl logs dapi-envars-resourcefieldref
1
1
33554432
67108864
Read
Defining Environment Variables for a Container
Read the
spec
API definition for Pod. This includes the definition of Container (part of Pod).
Read the list of
available fields
that you
can expose using the downward API.
Read about Pods, containers and environment variables in the legacy API reference:
PodSpec
Container
EnvVar
5 - Expose Pod Information to Containers Through Files
This page shows how a Pod can use a
volume
,
to expose information about itself to containers running in the Pod.
A
volume can expose Pod fields and container fields.
Volume files, as explained in this task
.
minikube
Store Pod fields
project Pod-level fields into the running container as files.
Here is the manifest for the Pod:
pods/inject/dapi-volume.yaml
:
v1
kind
:
Pod
metadata
:
name
:
kubernetes-downwardapi-volume-example
labels
:
zone
:
us-est-coast
cluster
:
test-cluster1
rack
:
rack-22
:
build
:
two
builder
:
john-doe
spec
:
:
-
name
:
client-container
image
:
command
:
[
"sh"
,
"-c"
]
args
:
if [[ -e /etc/podinfo/labels ]]; then
echo -en '\n\n'; cat /etc/podinfo/labels; fi;
if [[ -e /etc/podinfo/annotations ]]; then
echo -en '\n\n'; cat /etc/podinfo/annotations; fi;
sleep 5;
done;
:
-
name
:
podinfo
mountPath
:
/etc/podinfo
volumes
:
-
name
:
podinfo
:
items
:
-
path
:
"labels"
fieldRef
:
fieldPath
:
-
path
:
fieldRef
:
fieldPath
:
metadata.annotations
In the manifest, you can see that the Pod has a
Volume,
and the container mounts the volume at
.
Look at the
items
array under
. Each element of the array
defines a
volume.
The first element specifies that the value of the Pod's
field should be stored in a file named
labels
.
The second element specifies that the value of the Pod's
.
Note:
The fields in this example are Pod fields. They are not
fields of the container in the Pod.
kubectl apply -f https://k8s.io/examples/pods/inject/dapi-volume.yaml
kubectl logs kubernetes-downwardapi-volume-example
The output shows the contents of the
labels
file and the
file:
cluster="test-cluster1"
rack="rack-22"
zone="us-est-coast"

build="two"
builder="john-doe"
Get a shell into the container that is running in your Pod:
kubectl
exec
-it kubernetes-downwardapi-volume-example -- sh
In your shell, view the
labels
file:
/# cat /etc/podinfo/labels
The output shows that all of the Pod's labels 

have been written
to the
labels
file:
cluster
=
"test-cluster1"
rack
=
"rack-22"
zone
=
"us-est-coast"
Similarly, view the
file:
/# cat /etc/podinfo/annotations
View the files in the
/# ls -laR /etc/podinfo
In the output, you can see that the
labels
and
files
are in a temporary subdirectory: in this example,
..2982_06_02_21_47_53.299460680
. In the
..data
is
a symbolic link to the temporary subdirectory. Also in the
labels
and
are symbolic links.
drwxr-xr-x  ... Feb 6 21:47 ..2982_06_02_21_47_53.299460680
lrwxrwxrwx  ... Feb 6 21:47 ..data -&gt; ..2982_06_02_21_47_53.299460680
lrwxrwxrwx  ... Feb 6 21:47 annotations -&gt; ..data/annotations
lrwxrwxrwx  ... Feb 6 21:47 labels -&gt; ..data/labels

/etc/..2982_06_02_21_47_53.299460680:
total 8
-rw-r--r--  ... Feb  6 21:47 annotations
-rw-r--r--  ... Feb  6 21:47 labels
Using symbolic links enables dynamic atomic refresh of the metadata; updates are
written to a new temporary directory, and the
..data
symlink is updated
atomically using
rename(2)
.
Note:
A container using Downward API as a
subPath
volume mount will not
receive Downward API updates.
Exit the shell:
/#
exit
Store container fields
The preceding exercise, you made Pod-level fields accessible using the
container
rather than from the Pod overall. Here is a manifest for a Pod that again has
just one container:
pods/inject/dapi-volume-resources.yaml
:
v1
kind
:
Pod
metadata
:
name
:
kubernetes-downwardapi-volume-example-2
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
if [[ -e /etc/podinfo/cpu_limit ]]; then
echo -en '\n'; cat /etc/podinfo/cpu_limit; fi;
if [[ -e /etc/podinfo/cpu_request ]]; then
echo -en '\n'; cat /etc/podinfo/cpu_request; fi;
if [[ -e /etc/podinfo/mem_limit ]]; then
echo -en '\n'; cat /etc/podinfo/mem_limit; fi;
if [[ -e /etc/podinfo/mem_request ]]; then
echo -en '\n'; cat /etc/podinfo/mem_request; fi;
sleep 5;
done;
resources
:
requests
:
memory
:
"32Mi"
cpu
:
"125m"
limits
:
memory
:
"64Mi"
cpu
:
"250m"
:
-
name
:
podinfo
mountPath
:
volumes
:
-
name
:
podinfo
:
items
:
-
path
:
"cpu_limit"
:
:
resource
:
divisor
:
1m
-
path
:
"cpu_request"
:
:
resource
:
divisor
:
1m
-
path
:
"mem_limit"
:
:
resource
:
divisor
:
1Mi
-
path
:
"mem_request"
:
:
resource
:
divisor
:
1Mi
volume
,
and that the single container in that Pod mounts the volume at
.
Look at the
items
defines a file in the downward API volume.
The first element specifies that in the container named
,
the value of the
field in the format specified by
1m
should be
published as a file named
cpu_limit
. The
divisor
field is optional and has the
default value of
1
. A divisor of 1 means cores for
cpu
resources, or
bytes for
memory
resources.
kubectl apply -f https://k8s.io/examples/pods/inject/dapi-volume-resources.yaml
kubectl
exec
-it kubernetes-downwardapi-volume-example-2 -- sh
cpu_limit
file:
cat /etc/podinfo/cpu_limit
You can use similar commands to view the
cpu_request
,
mem_limit
and
mem_request
files.
You can project keys to specific paths and specific permissions on a per-fil

e
basis. For more information, see
Secrets
.
Read the
spec
that you
Read about volumes in the legacy API reference:
Check the
Volume
API definition which defines a generic volume in a Pod for containers to access.
Check the
DownwardAPIVolumeSource
API definition which defines a volume that contains Downward API information.
Check the
API definition which contains references to object or resource fields for
populating a file in the Downward API volume.
Check the
API definition which specifies the container resources and their output format.
6 - Distribute Credentials Securely Using Secrets
minikube
my-app
echo
-n
'my-app'
| base64
echo
-n
| base64
bXktYXBw
,
.
Caution:
:
v1
kind
:
Secret
metadata
:
name
:
data
:
username
:
bXktYXBw
password
:
Output:
Output:

Data
====
=
=
Here is a configuration file you can use to create a Pod:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
:
-
name
:
mountPath
:
volumes
:
-
name
:
secret
:
:
Output:
kubectl
exec
T

he secret data is exposed to the Container through a Volume mounted under
.
username
and
password
files:
echo
"
$(
)
"
echo
"
$(
)
"
my-app
=
=
Assign the
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
env
:
-
name
:
valueFrom
:
:
name
:
key
:
kubectl
exec
=
=
=
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
valueFrom
:
:
name
:
db-user
key
:
kubectl
exec
Note:
=
username
=
'my-app'
=
password
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
envFrom
:
-
secretRef
:
name
:
username
and
password
kubectl
exec
Secret
Volume
Pod
Secrets
.
Volumes
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foun

dation, please see our

Debugging Kubernetes Nodes With Kubectl | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Installing Addon

s
Tasks
Migrate from PodSecurityPol

icy to the Built-In PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config 

using MicroProfile, ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Client Authenticatio

n (v1beta1)
Tasks
node
command.
minikube
.
Use the
=
ubuntu
ip
,
ifconfig
,
nc
,
ping
, and
ps
mtr
,
tcpdump
, and
curl
Note:
/host
in the Pod.
kubelet
L

ogs from
/host
.
the pod isn't privileged. This means that reading some process information might fail
will fail.
Note:
The
Check
Feedback
Yes
No
.
or
.
Last modified January 03, 2023 at 12:49 PM PST:
Add Documentation for Kubectl debug node command (#35300) (ea458f2009)
Home
Blog
Training
Partners
Community
CC BY 4.0

Troubleshooting Clusters | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
the
Ready
state.
and

Lease:
  ----                 ------    -----------------                 ------------------                ------              -------
Capacity:
  cpu:                2
  pods:               110
  cpu:                2
  pods:               110
Events:
...
:
v1
kind
:
Node
metadata
:
:
:
:
"0"
:
"true"
:
labels
:
:
amd64
:
linux
:
amd64
:
:
linux
name
:
:
"4026"
uid
:
spec
:
{}
status
:
addresses
:
-
address
:
type
:
-
address
:
type
:
Hostname
:
cpu
:
"2"
:
:
"0"
memory
:
1922788

Ki
pods
:
"110"
capacity
:
cpu
:
"2"
:
:
"0"
memory
:
2025188Ki
pods
:
"110"
:
-
:
:
message
:
reason
:
WeaveIsUp
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"True"
type
:
Ready
:
:
Port
:
10250
nodeInfo
:
:
amd64
bootID
:
22333234
:
:
5.13.0-28
-generic
:
v1.23.3
:
v1.23.3
machineID
:
:
linux
osImage
:
:
- logs from
Results
Results
Results
Results
Results
cra

shing kubelet cannot start new pods on the node
Results
etc.
Action: Use IaaS provider's automatic VM restarting feature for IaaS VMs
Use
to
Use
crictl
to
Use
to
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

PodTemplate | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

Reference
: v1
kind
: PodTemplate
metadata
(
)
template
(
)
metadata
(
)
spec
(
PodSpec
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query

): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
list
list or watch objects of kind PodTempl

ate
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Extend Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
1 - CustomResourceDefinition
kind
metadata
(
)
spec
(
status
(
)
group
).
names
kind
.
(string)
kind
List".
(string)
kind
.
scope
Cluster
and
.
versions
versions is the list of all API versions of the defined custom resource. Version names are used to compute the order in which served versions are listed in API discovery. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1

, v11alpha2, foo1, foo10.
if
served
is true.
(string)
(string)
(int32)
(boolean)
(string)
(
)
/scale
.spec
/scal

e
.status
/scale
(string)
.status
or
.spec
/scale
/status
status
/status
status
.status
None
Webhook
strategy
is set to
Webhook
.
Webhook
.
([]byte)
If the webhook is running 

within the cluster, then you should use
service
.
(string)
(int32)
port
(string)
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
(boolean)
to true in
. See
.
$ref
(string)
$schema
(string)
allOf
([]
)
anyOf
([]
)
default
(JSON)
)
(string)
enum
([]JSON)
example
(JSON)
(boolean)
(boolean)
(string)
(string)
format
(string)
format is an OpenAPI v3 format string. U

nknown formats are ignored. The following formats are validated:
id
(string)
items
maxItems
(int64)
maxLength
(int64)
(int64)
maximum
(double)
minItems
(int64)
minLength
(int64)
(int64)
minimum
(double)
(double)
not
(
)
nullable
(boolean)
oneOf
([]
)
pattern
(string)
)
)
required
title
(string)
type
(string)
(boolean)
(boolean)
x-kubernetes-embedded-resource defines that the value is an embedded Kubernetes runtime.Object, with TypeMeta and ObjectMeta. The type must be object. It is allowed to further restrict the embedded object. kind, apiVersion and metadata are validated automatically. x-kubernetes-preserve-unknown-fields is allowed to be true, but does not have to be if the object is fully specified (up to kind, apiVersion, metadata).
(boolean)
anyOf:
allOf:
anyOf:
map
The properties specified mu

st either be required or have a default value, to ensure those properties are present for all list items.
(string)
atomic
set
:
atomic
or an
atomic
.
map
:
(string)
granular
:
atomic
(boolean)
rule
self
self[i]
self
The
,
kind
,
and
Unknown data preserved in custom resources via x-kubernetes-preserve-unkn

own-fields is not accessible in CEL expressions. This includes: - Unknown field values that are preserved by object schemas with x-kubernetes-preserve-unknown-fields. - Object properties where the property schema is of an "unknown type". An "unknown type" is recursively defined as:
dot
dash
slash
Examples:
namespace
&gt; 0"}
'set':
X + Y
X
Y
'map':
X + Y
X
Y
X
and
Y
Y
with
(string)
kind
.
(string)
kind
List".
(string)
kind
.
conditions indicate state for particular as

pects of a CustomResourceDefinition
(Time)
(string)
(string)
items
([]
(string)
kind
(string)
metadata
(
ListMeta
)
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CustomResourceDefini

tion
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - MutatingWebhookConfiguration
: adm

issionregistration.k8s.io/v1
kind
metadata
(
)
.
webhooks
name
([]byte)
caBundle
service
service
or
url
service
.
name
namespace
(string)
path
(int32)
port
(string)
url
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
SideEffects states whether this webhook has side effects. Acceptable values are: None, NoneOnDryRun (webhooks created via v1beta1 may also specify Some or Unknown). Webhooks with side effects MUST implement a reconciliation system, since a request may be rejected by a future step in the admission chain and the side effects therefore need to be undone. Requests with the dryRun att

ribute will be auto-rejected if they match a webhook with sideEffects == Unknown or Some.
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
(string)
must
Defaults to "Ne

ver".
any
(string)
(int32)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string

(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3 - ValidatingWebhookConfiguration
kind
metadata
(
)
.
webhooks
name
ValidatingWebhook describes an admission webhook

 and the resources and operations it applies to.
([]byte)
caBundle
service
service
or
url
service
.
name
namespace
(string)
path
(int32)
port
(string)
url
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
(string)
(string)
matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equ

ivalent".
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
any
Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If '*' is present, the length of the slice must be one. 

Required.
(string)
(int32)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create a ValidatingWebhookConfiguration
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/admission

registration.k8s.io/v1/validatingwebhookconfigurations/{name}
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
4 - ValidatingAdmissionPolicy v1alpha1
kind
metadata
(
)
.
spec
ref
The
,
kind
,
and
' 

- '.' escapes to '
dot
dash
slash
Examples:
namespace
&gt; 0"}
'set':
X + Y
X
Y
'map':
X + Y
X
Y
X
and
Y
Y
with
Required.
(string)
(string)
(string)
all
APIVersions is th

e API versions the resources belong to. '
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
ObjectSelector decides whether to run the validation based on if the object has matching labels. o

bjectSelector is evaluated against both the oldObject and newObject that would be sent to the cel validation, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
any
Rule.
(string)
(string)
(string)
(string)
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest interna

l value, and may reject unrecognized values. More info:
items
([]
)
kind
(string)
metadata
(
ListMeta
)
(string)
kind
(string)
metadata
(
)
.
spec
Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If '*' is present, the length of the slice m

ust be one. Required.
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
ObjectSelector decides whether to run the validation based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the cel validation, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the 

admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
any
Rule.
(string)
(string)
(string)
(string)
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): str

ing
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

Deploy and Access the Kubernetes Dashboard | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggregat

ion Layer
Tasks
Debug Pods
Dete

rmine the Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

ards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver 

Tasks
Deploy the web UI (Kubernetes Dashboard) and access it.
Dashboard is a web-based Kubernetes user interface.
You can use Dashboard to deploy containerized applications to a Kubernetes cluster,
troubleshoot your containerized application, and manage the cluster resources.
You can use Dashboard to get an overview of applications running on your cluster,
as well as for creating or modifying individual Kubernetes resources
(such as Deployments, Jobs, DaemonSets, etc).
For example, you can scale a Deployment, initiate a rolling update, restart a pod
or deploy new applications using a deploy wizard.
Dashboard also provides information on the state of Kubernetes resources in your cluster and on any errors that may have occurred.
Deploying the Dashboard UI
The Dashboard UI is not deployed by default. To deploy it, run the following command:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.7.0/aio/deploy/recommended.yaml
Accessing the Dashboard UI
To protect your cluster data, Dashboard deploys with a minimal RBAC configuration by default.
Currently, Dashboard only supports logging in with a Bearer Token.
To create a token for this demo, you can follow our guide on
creating a sample user
.
Warning:
The sample user created in the tutorial will have administrative privileges and is for educational purposes only.
Command line proxy
You can enable access to the Dashboard using the
kubectl
by running the following command:
Kubectl will make Dashboard available at
http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/
.
The UI can
only
be accessed from the machine where the command is executed. See
kubectl proxy --help
for more options.
Note:
The kubeconfig authentication method does
not
support external identity providers
or X.509 certificate-based authentication.
Welcome view
When you access Dashboard on an empty cluster, you'll see the welcome page.
This page contains a link to this document as well as a button to deploy your first application.
In addition, you can view which system applications are running by default in the
namespace
of your cluster, for example the Dashboard itself.
Deploying containerized applications
Dashboard lets you create and deploy a containerized application as a Deployment and optional Service with a simple wiza

rd.
You can either manually specify application details, or upload a YAML or JSON
manifest
file containing application configuration.
Click the
CREATE
button in the upper right corner of any page to begin.
Specifying application details
The deploy wizard expects that you provide the following information:
App name
(mandatory): Name for your application.
A
label
with the name will be
added to the Deployment and Service, if any, that will be deployed.
The application name must be unique within the selected Kubernetes
namespace
.
It must start with a lowercase character, and end with a lowercase character or a number,
and contain only lowercase letters, numbers and dashes (-). It is limited to 24 characters.
Leading and trailing spaces are ignored.
Container image
(mandatory):
The URL of a public Docker
on any registry,
or a private image (commonly hosted on the Google Container Registry or Docker Hub).
The container image specification must end with a colon.
Number of pods
(mandatory): The target number of Pods you want your application to be deployed in.
The value must be a positive integer.
A
will be created to
maintain the desired number of Pods across your cluster.
Service
(optional): For some parts of your application (e.g. frontends) you may want to expose a
Service
onto an external,
maybe public IP address outside of your cluster (external Service).
Note:
For external Services, you may need to open up one or more ports to do so.
Other Services that are only visible from inside the cluster are called internal Services.
Irrespective of the Service type, if you choose to create a Service and your container listens
on a port (incoming), you need to specify two ports.
The Service will be created mapping the port (incoming) to the target port seen by the container.
This Service will route to your deployed Pods. Supported protocols are TCP and UDP.
The internal DNS name for this Service will be the value you specified as application name above.
If needed, you can expand the
Advanced options
section where you can specify more settings:
: The text you enter here will be added as an
to the Deployment and displayed in the application's details.
Labels
: Default
labels
to be used
for your application are application name and version.
You can specify additional labels to be applied to the Deployment, Service (if any), and Pods,
such as release, environment, tier, partition, and release track.
Example:
release=1.0
tier=frontend
environment=pod
track=stable
Namespace
: Kubernetes supports multiple virtual clusters backed by the same physical cluster.
These virtual clusters are called
.
They let you partition resources into logically named groups.
Dashboard offers all available namespaces in a dropdown list, and allows you to create a new namespace.
The namespace name may contain a maximum of 63 alphanumeric characters and dashes (-) but can not contain capital letters.
Namespace names should not consist of only numbers.
If the name is set as a number, such as 10, the pod will be put in the default namespace.
In case the creation of the namespace is successful, it is selected by default.
If the creation fails, the first namespace is selected.
Image Pull Secret
:
In case the specified Docker container image is private, it may require
pull secret
credentials.
Dashboard offers all available secrets in a dropdown list, and allows you to create a new secret.
The secret name must follow the DNS domain name syntax, for example
new.image-pull.secret
.
The content of a secret must be base64-encoded and specified in a
.dockercfg
file.
The secret name may consist of a maximum of 253 characters.
In case the creation of the image pull secret is successful, it is selected by default. If the creation fails, no secret is applied.
CPU requirement (cores)
and
Memory requirement (MiB)
:
You can specify the minimum
resource limits
for the container. By default, Pods run with unbounded CPU and memo

ry limits.
Run command
and
Run command arguments
:
By default, your containers run the specified Docker image's default
entrypoint command
.
You can use the command options and arguments to override the default.
Run as privileged
: This setting determines whether processes in
privileged containers
are equivalent to processes running as root on the host.
Privileged containers can make use of capabilities like manipulating the network stack and accessing devices.
: Kubernetes exposes Services through
.
You can compose environment variable or pass arguments to your commands using the values of environment variables.
They can be used in applications to find a Service.
Values can reference other variables using the
syntax.
Uploading a YAML or JSON file
Kubernetes supports declarative configuration.
In this style, all configuration is stored in manifests (YAML or JSON configuration files).
The manifests use Kubernetes
API
resource schemas.
As an alternative to specifying application details in the deploy wizard,
you can define your application in one or more manifests, and upload the files using Dashboard.
Using Dashboard
Following sections describe views of the Kubernetes Dashboard UI; what they provide and how can they be used.
Navigation
When there are Kubernetes objects defined in the cluster, Dashboard shows them in the initial view.
By default only objects from the
default
namespace are shown and
this can be changed using the namespace selector located in the navigation menu.
Dashboard shows most Kubernetes object kinds and groups them in a few menu categories.
Admin overview
For cluster and namespace administrators, Dashboard lists Nodes, Namespaces and PersistentVolumes and has detail views for them.
Node list view contains CPU and memory usage metrics aggregated across all Nodes.
The details view shows the metrics for a Node, its specification, status,
allocated resources, events and pods running on the node.
Workloads
Shows all applications running in the selected namespace.
The view lists applications by workload kind (for example: Deployments, ReplicaSets, StatefulSets).
Each workload kind can be viewed separately.
The lists summarize actionable information about the workloads,
such as the number of ready pods for a ReplicaSet or current memory usage for a Pod.
Detail views for workloads show status and specification information and
surface relationships between objects.
For example, Pods that ReplicaSet is controlling or new ReplicaSets and HorizontalPodAutoscalers for Deployments.
Services
Shows Kubernetes resources that allow for exposing services to external world and
discovering them within a cluster.
For that reason, Service and Ingress views show Pods targeted by them,
internal endpoints for cluster connections and external endpoints for external users.
Storage
Storage view shows PersistentVolumeClaim resources which are used by applications for storing data.
ConfigMaps and Secrets
Shows all Kubernetes resources that are used for live configuration of applications running in clusters.
The view allows for editing and managing config objects and displays secrets hidden by default.
Logs viewer
Pod lists and detail pages link to a logs viewer that is built into Dashboard.
The viewer allows for drilling down logs from containers belonging to a single Pod.
Kubernetes Dashboard project page
.
Feedback
Yes
No
.
or
.
Last modified December 26, 2022 at 2:06 AM PST:
Update web-ui-dashboard.md (1af5764704)
Spec

ifying application details
Home
Blog
Training
Partners
Community
CC BY 4.0

ObjectReference | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on macO

S
Debug Pods
Troubles

hooting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resour

ces with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Co

Reference
(string)
fieldPath
(string)
kind
(string)
name
(string)
namespace
(string)
(string)
uid
(string)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Trademark Usage

Scheduler Configuration | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
in the
.
,
using the
v1beta3
or
v1
)
struct.
:
kind
:
:
:
Note:
v1beta2
v1beta3
or
v1
Profiles
in the
.
.
Plugins
to run
.
queueSort
preFilter
filter
Pod
preScore
score
reserve

Unreserve
Reserve
.
permit
preBind
bind
bind
postBind
:
kind
:
profiles
:
-
plugins
:
score
:
disabled
:
-
name
:
enabled
:
-
name
:
weight
:
2
-
name
:
weight
:
1
You can use
*
desired.
Pod runs.
score
.
.
filter
,
preScore
,
score
.
NodeName
filter
.
NodePorts
preFilter
,
filter
.
and
.
filter
,
score
.
.
preFilter
,
filter
,
preScore
,
score
.
set to
true.
filter
.
and
.
preFilter
,
filter
,
score
.
score
.
volumes
.
preFilter
,
filter
,
reserve
,
preBind
,
score
.
Note:
score
feature is
size.
filter
.
filter
.
node.
filter
.
EBSLimits
filter
.
filter
.
the node.
filter
.
.
preFilter
,
filter
,
preScore
,
score
.
queueSort
.
bind
.
.
: Che

cks that
filter
.
.
disabled.
:
kind
:
profiles
:
-
:
-
:
no
plugins
:
preScore
:
disabled
:
-
name
:
'*'
score
:
disabled
:
-
name
:
'*'
.
Note:
list.
Note:
queueSort
MyPlugin
preScore
,
score
,
preFilter
,
and
filter
MyPlugin
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
MyPlugin
MyPlugin
:
kind
:
profiles
:
-
:
plugins
:
preScore
:
enabled
:
-
name
:
MyPlugin
score
:
enabled
:
-
name
:
MyPlugin
preFilter
:
enabled
:
-
name
:
MyPlugin
filter
:
enabled
:
-
name
:
MyPlugin
MyPlugin
the
disabled
'*'
Score
and
PreScore
, would be:
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
preScore
:
disabled
:
-
name
:
'*'
score
:
disabled
:
-
name
:
'*'
, all
.
example, consider two Score plugins
and
, each with
a weight of
1
:
kind
:
profiles
:
-

:
plugins
:
score
:
enabled
:
-
name
:
weight
:
5
Score
is
.
over
Plugin
QueueSort
QueueSort
Score
,
Filter
Score
Score
,
Filter
Score
,
Filter
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
-
name
:
weight
:
3
-
name
:
disabled
:
-
name
:
filter
:
disabled
:
-
name
:
score
:
enabled
:
-
name
:
queueSort
Enables
and
Disables
filter
Reorders
score
v1beta3
, without
:
kind
:
profiles
:
-
:
plugins
:
queueSort
:
enabled
:
-
name
:
disabled
:
-
name
:
filter
:
enabled
:
-
name
:
-
name
:
-
name
:
disabled
:
-
name
:
score
:
enabled
:
-
name
:
weight
:
1
-
name
:
weight
:
3
config
plugin.
,
and
plugins.
with a
:
kind
:
profiles
:
-
:
-
args
:
:
resources
:
-
name
:
cpu
weight
:
1
type
:
name
:
NodeLabel
node ta

ints
Invalid
host
or
port
from 1 to 2
from 1 to 2
from 1 to 3
Read the
Read the
reference
Read the
reference
Read the
reference
Feedback
Yes
No
.
or
.
Last modified August 16, 2022 at 9:30 PM PST:
address comments (ab283be3f5)
Profiles
Home
Blog
Training
Partners
Community
CC BY 4.0

Event Rate Limit Configuration (v1alpha1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Field
string
kind
string
limits
[]Limit
Limit
Field
type
LimitType
qps
int32
burst
int32
cacheSize
int32
LimitType
(Alias of
string
)
Limit
Feedback
Yes
No
.
Open an issue in th

e GitHub repo if you want to
or
.
Last modified June 03, 2022 at 5:35 PM PST:
Fix the package name for eventratelimit config API (55fd6c705c)
Limit
LimitType
Home
Blog
Training
Partners
Community
CC BY 4.0

Use Cascading Deletion in a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
.
minikube
app
=
=
yaml
:
v1
...
:
-
:
apps/v1
:
true
:
true
kind
:
name
:
uid
:
...
kubectl
.
kubectl
or the
=
=
8080
Use
curl
\
-d
\
-H
finalizer
like this:
    ]
    ...
.
Use either
kubectl
or the Kuber

netes API to delete the Deployment,
.
kubectl
--cascade
flag or the
argument.
=
=
8080
Use
curl
\
-d
\
-H
...
}
orphan
kubectl
.
=
orphan
=
8080
Use
curl
\
-d
\
-H
orphan
in the
  "orphan"
],
...
app
=
nginx
.
.
Feedback
Yes
No
.
or
.
Last modified September 15, 2022 at 11:05 AM PST:
Drop doc prior to v1.20 (bf9bce3107)
Home
Blog
Training
Partners
Community
CC BY 4.0

Run Jobs | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Run Jobs
Run Jobs using parallel processing.
1:
2:
3:
4:
5:
6:
7:
1 - Running Automated Tasks with a CronJob
You can use a
CronJob
to run
Jobs
on a time-based schedule.
These automated jobs run like
Cron
tasks on a Linux or UNIX system.
Cron jobs are useful for creating periodic and recurring tasks, like running backups or sending emails.
Cron jobs can also schedule individual tasks for a specific time, such as if you want to schedule a job for a low activity period.
Cron jobs have limitations and idiosyncrasies.
For example, in certain circumstances, a single cron job can create multiple jobs.
Therefore, jobs should be idempotent.
For more limitations, see
CronJobs
.
minikube
Creating a CronJob
Cron jobs require a config file.
Here is a manifest for a CronJob that runs a simple demonstration task every minute:
application/job/cronjob.yaml
:
batch/v1
kind
:
CronJob
metadata
:
name
:
hello
spec
:
schedule
:
"* * * * *"
:
spec
:
template
:
spec
:
:
-
name
:
hello
image
:
:
command
:
- /bin/sh
- -c
- date; echo Hello from the Kubernetes cluster
:
OnFailure
Run the example CronJob by using this command:
kubectl create -f https://k8s.io/examples/application/job/cronjob.yaml
cronjob.batch/hello created
After creating the cron job, get its status using this command:
kubectl get cronjob hello
NAME    SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE
hello   */1 * * * *   False     0        &lt;none&gt;          10s
As you can see from the results of the command, the cron job has not scheduled or run any jobs yet.
Watch for the job to be created in around one minute:
jobs
--watch
NAME               COMPLETIONS   DURATION   AGE
hello-4111706356   0/1                      0s
hello-4111706356   0/1           0s         0s
hello-4111706356   1/1           5s         5s
Now you've seen one running job scheduled by the "hello" cron job.
You can stop watching the job and view the cron job again to see that it scheduled the job:
hello   */1 * * * *   False     0        50s             75s
You should see that the cron job
hello
successfully scheduled a job at the time specified in
LAST SCHEDULE
. There are currently 0 active jobs, meaning that the job has completed or failed.
Now, find the pods that the last scheduled job created and view the standard output of one of the pods.
Note:
The job name is different from the pod name.
# Replace "hello-4111706356" with the job name in your system
pods
=
$(
=
job-name
=
hello-4111706356 --output
=
jsonpath
={
.items
[
*
]
}
)
Show th

e pod log:
$pods
Fri Feb 22 11:02:09 UTC 2019
Hello from the Kubernetes cluster
Deleting a CronJob
When you don't need a cron job any more, delete it with
kubectl delete cronjob &lt;cronjob name&gt;
:
kubectl delete cronjob hello
Deleting the cron job removes all the jobs and pods it created and stops it from creating additional jobs.
You can read more about removing jobs in
.
Writing a CronJob Spec
As with all other Kubernetes objects, a CronJob must have
,
kind
, and
metadata
fields.
For more information about working with Kubernetes objects and their
manifests
, see the
managing resources
,
and
using kubectl to manage resources
documents.
Each manifest for a CronJob also needs a
.spec
section.
Note:
If you modify a CronJob, the changes you make will apply to new jobs that start to run after your modification
is complete. Jobs (and their Pods) that have already started continue to run without changes.
That is, the CronJob does
not
update existing jobs, even if those remain running.
Schedule
The
.spec.schedule
is a required field of the
.spec
.
It takes a
Cron
format string, such as
0 * * * *
or
@hourly
,
as schedule time of its jobs to be created and executed.
The format also includes extended "Vixie cron" step values. As explained in the
FreeBSD manual
:
Step values can be used in conjunction with ranges. Following a range
with
/&lt;number&gt;
specifies skips of the number's value through the
range. For example,
0-23/2
can be used in the hours field to specify
command execution every other hour (the alternative in the V7 standard is
0,2,4,6,8,10,12,14,16,18,20,22
). Steps are also permitted after an
asterisk, so if you want to say "every two hours", just use
*/2
.
Note:
A question mark (
?
) in the schedule has the same meaning as an asterisk
*
, that is,
it stands for any of available value for a given field.
Job Template
The
.spec.jobTemplate
is the template for the job, and it is required.
It has exactly the same schema as a
Job
, except that
it is nested and does not have an
or
kind
.
For information about writing a job
.spec
, see
Writing a Job Spec
.
Starting Deadline
The
.spec.startingDeadlineSeconds
field is optional.
It stands for the deadline in seconds for starting the job if it misses its scheduled time for any reason.
After the deadline, the cron job does not start the job.
Jobs that do not meet their deadline in this way count as failed jobs.
If this field is not specified, the jobs have no deadline.
If the
field is set (not null), the CronJob
controller measures the time between when a job is expected to be created and
now. If the difference is higher than that limit, it will skip this execution.
For example, if it is set to
200
, it allows a job to be created for up to 200
seconds after the actual schedule.
Concurrency Policy
The
.spec.concurrencyPolicy
field is also optional.
It specifies how to treat concurrent executions of a job that is created by this cron job.
The spec may specify only one of the following concurrency policies:
Allow
(default): The cron job allows concurrently running jobs
Forbid
: The cron job does not allow concurrent runs; if it is time for a new job run and the
previous job run hasn't finished yet, the cron job skips the new job run
Replace
: If it is time for a new job run and the previous job run hasn't finished yet, the
cron job replaces the currently running job run with a new job run
Note that concurrency policy only applies to the jobs created by the same cron job.
If there are multiple cron jobs, their respective jobs are always allowed to run concurrently.
Suspend
The
.spec.suspend
If it is set to
true
, all subsequent executions are suspended.
This setting does not apply to already started executions.
Caution:
Executions that are suspended during their scheduled time count as missed jobs.
When
changes fr

om
true
to
false
on an existing cron job without a
starting deadline
, the missed jobs are scheduled immediately.
Jobs History Limits
The
.spec.successfulJobsHistoryLimit
and
.spec.failedJobsHistoryLimit
fields are optional.
These fields specify how many completed and failed jobs should be kept.
By default, they are set to 3 and 1 respectively. Setting a limit to
0
corresponds to keeping
none of the corresponding kind of jobs after they finish.
2 - Coarse Parallel Processing Using a Work Queue
In this example, we will run a Kubernetes Job with multiple parallel
In this example, as each pod is created, it picks up one unit of work
from a task queue, completes it, deletes it from the queue, and exits.
Start a message queue service.
In this example, we use RabbitMQ, but you could use another
one. In practice you would set up a message queue service once and reuse it for many jobs.
Create a queue, and fill it with messages.
Each message represents one task to be done. In
this example, a message is an integer that we will do a lengthy computation on.
Start a Job that works on tasks from the queue
. The Job starts several pods. Each pod takes
one task from the message queue, processes it, and repeats until the end of the queue is reached.
Be familiar with the basic,
Job
.
minikube
Starting a message queue service
This example uses RabbitMQ, however, you can adapt the example to use another AMQP-type message service.
In practice you could set up a message queue service once in a
cluster and reuse it for many jobs, as well as for long-running services.
Start RabbitMQ as follows:
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.3/examples/celery-rabbitmq/rabbitmq-service.yaml
service "rabbitmq-service" created
kubectl create -f https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.3/examples/celery-rabbitmq/rabbitmq-controller.yaml
replicationcontroller "rabbitmq-controller" created
We will only use the rabbitmq part from the
celery-rabbitmq example
.
Testing the message queue service
Now, we can experiment with accessing the message queue. We will
create a temporary interactive pod, install some tools on it,
and experiment with queues.
First create a temporary interactive Pod.
# Create a temporary interactive container
kubectl run -i --tty temp --image ubuntu:18.04
Waiting for pod default/temp-loe07 to be running, status is Pending, pod ready: false
... [ previous line repeats several times .. hit return when it stops ] ...
Note that your pod name and command prompt will be different.
Next install the
amqp-tools
so we can work with message queues.
# Install some tools
root@temp-loe07:/# apt-get update
....
[
lots of output
]
....
root@temp-loe07:/# apt-get install -y curl ca-certificates amqp-tools python dnsutils
....
[
]
....
Later, we will make a docker image that includes these packages.
Next, we will check that we can discover the rabbitmq service:
# Note the rabbitmq-service has a DNS name, provided by Kubernetes:

root@temp-loe07:/# nslookup rabbitmq-service
Server:        10.0.0.10
Address:    10.0.0.10#53

Name:    rabbitmq-service.default.svc.cluster.local
Address: 10.0.147.152

# Your address will vary.
If Kube-DNS is not set up correctly, the previous step may not work for you.
You can also find the service IP in an env var:
# env | grep RABBIT | grep HOST
RABBITMQ_SERVICE_SERVICE_HOST=10.0.147.152
Next we will verify we can create a queue, and

 publish and consume messages.
# In the next line, rabbitmq-service is the hostname where the rabbitmq-service
# can be reached.  5672 is the standard port for rabbitmq.
root@temp-loe07:/#
export
BROKER_URL
=
amqp://guest:guest@rabbitmq-service:5672
# If you could not resolve "rabbitmq-service" in the previous step,
# then use this command instead:
# root@temp-loe07:/# BROKER_URL=amqp://guest:guest@$RABBITMQ_SERVICE_SERVICE_HOST:5672
# Now create a queue:
root@temp-loe07:/# /usr/bin/amqp-declare-queue --url
=
$BROKER_URL
-q foo -d
foo
# Publish one message to it:
root@temp-loe07:/# /usr/bin/amqp-publish --url
=
-r foo -p -b Hello
# And get it back.
root@temp-loe07:/# /usr/bin/amqp-consume --url
=
-q foo -c
1
cat
echo
Hello
In the last command, the
amqp-consume
tool takes one message (
-c 1
)
from the queue, and passes that message to the standard input of an arbitrary command. In this case, the program
cat
prints out the characters read from standard input, and the echo adds a carriage
return so the example is readable.
Filling the Queue with tasks
Now let's fill the queue with some "tasks". In our example, our tasks are strings to be
printed.
In a practice, the content of the messages might be:
names of files to that need to be processed
extra flags to the program
ranges of keys in a database table
configuration parameters to a simulation
frame numbers of a scene to be rendered
In practice, if there is large data that is needed in a read-only mode by all pods
of the Job, you will typically put that in a shared file system like NFS and mount
that readonly on all the pods, or the program in the pod will natively read data from
a cluster file system like HDFS.
For our example, we will create the queue and fill it using the amqp command line tools.
In practice, you might write a program to fill the queue using an amqp client library.
/usr/bin/amqp-declare-queue --url
=
-q job1  -d
job1
for
f in apple banana cherry date fig grape lemon melon
do
/usr/bin/amqp-publish --url
=
-r job1 -p -b
$f
done
So, we filled the queue with 8 messages.
Create an Image
Now we are ready to create an image that we will run as a job.
We will use the
utility to read the message
from the queue and run our actual program. Here is a very simple
example program:
application/job/rabbitmq/worker.py
#!/usr/bin/env python
# Just prints standard out and sleeps for 10 seconds.
import
sys
import
time
print
(
"Processing "
+
sys
.
stdin
.
readlines()[
0
])
time
.
sleep(
10
)
Give the script execution permission:
chmod +x worker.py
Now, build an image. If you are working in the source
tree, then change directory to
examples/job/work-queue-1
.
Otherwise, make a temporary directory, change to it,
download the
Dockerfile
,
and
worker.py
. In either case,
build the image with this command:
docker build -t job-wq-1 .
For the
Docker Hub
, tag your app image with
your username and push to the Hub with the below commands. Replace
&lt;username&gt;
with your Hub username.
docker tag job-wq-1 &lt;username&gt;/job-wq-1
docker push &lt;username&gt;/job-wq-1
If you are using
Google Container
Registry
, tag
your app image with your project ID, and push to GCR. Replace
&lt;project&gt;
with your project ID.
docker tag job-wq-1 gcr.io/&lt;project&gt;/job-wq-1
gcloud docker -- push gcr.io/&lt;project&gt;/job-wq-1
Defining a Job
Here is a job definition. You'll need to make a copy of the Job and edit the
image to match the name you used, and call it
./job.yaml
.
application/job/rabbitmq/job.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
job-wq-1
spec
:
:
8
:
2
template
:
metadata
:
name
:
job-wq-1
spec
:
:
-
name
:
c
image
:
gcr.io/&lt;project&gt;/job-wq-1
env
:
-
name
:
value
:
-
name
:
QUEUE
value
:
job1
:
OnFailure
In this example, each pod works on one item from the queue and then ex

its.
So, the completion count of the Job corresponds to the number of work items
done. So we set,
.spec.completions: 8
for the example, since we put 8 items in the queue.
So, now run the Job:
kubectl apply -f ./job.yaml
kubectl
wait
--for
=
condition
=
complete
--timeout
=
300s job/job-wq-1
Next, check on the Job:
kubectl describe jobs/job-wq-1
Name:             job-wq-1
Namespace:        default
Selector:         controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
Labels:           controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
                  job-name=job-wq-1
Annotations:      &lt;none&gt;
Parallelism:      2
Completions:      8
Start Time:       Wed, 06 Sep 2017 16:42:02 +0800
Pods Statuses:    0 Running / 8 Succeeded / 0 Failed
  Labels:       controller-uid=41d75705-92df-11e7-b85e-fa163ee3c11f
                job-name=job-wq-1
   c:
    Image:      gcr.io/causal-jigsaw-637/job-wq-1
    Port:
    Environment:
      BROKER_URL:       amqp://guest:guest@rabbitmq-service:5672
      QUEUE:            job1
    Mounts:             &lt;none&gt;
  Volumes:              &lt;none&gt;
Events:
  FirstSeen  LastSeen   Count    From    SubobjectPath    Type      Reason              Message
  ─────────  ────────   ─────    ────    ─────────────    ──────    ──────              ───────
  27s        27s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-hcobb
  27s        27s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-weytj
  27s        27s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-qaam5
  27s        27s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-b67sr
  26s        26s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-xe5hj
  15s        15s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-w2zqe
  14s        14s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-d6ppa
  14s        14s        1        {job }                   Normal    SuccessfulCreate    Created pod: job-wq-1-p17e0
All the pods for that Job succeeded. Yay.
This approach has the advantage that you
do not need to modify your "worker" program to be aware that there is a work queue.
It does require that you run a message queue service.
If running a queue service is inconvenient, you may
want to consider one of the other
job patterns
.
This approach creates a pod for every work item. If your work items only take a few seconds,
though, creating a Pod for every work item may add a lot of overhead. Consider another
example
, that executes multiple work items per Pod.
In this example, we use the
from the queue and run our actual program. This has the advantage that you
do not need to modify your program to be aware of the queue.
A
different example
, shows how to
communicate with the work queue using a client library.
Caveats
If the number of completions is set to less than the number of items in the queue, then
not all items will be processed.
If the number of completions is set to more than the number of items in the queue,
then the Job will not appear to be completed, even though all items in the queue
have been processed. It will start additional pods which will block waiting
for a message.
There is an unlikely race with this pattern. If the container is killed in between the time
that the message is acknowledged by the amqp-consume command and the time that the container
exits with success, or if the node crashes before the kubelet is able to post the success of the pod
back to the api-server, then the Job will not appear to be complete, even though all it

ems
in the queue have been processed.
3 - Fine Parallel Processing Using a Work Queue
worker processes in a given pod.
from a task queue, processes it, and repeats until the end of the queue is reached.
Start a storage service to hold the work queue.
In this example, we use Redis to store
our work items. In the previous example, we used RabbitMQ. In this example, we use Redis and
a custom work-queue client library because AMQP does not provide a good way for clients to
detect when a finite-length work queue is empty. In practice you would set up a store such
as Redis once and reuse it for the work queues of many jobs, and other things.
minikube
Job
.
Starting Redis
For this example, for simplicity, we will start a single instance of Redis.
See the
Redis Example
for an example
of deploying Redis scalably and redundantly.
You could also download the following files directly:
redis-pod.yaml
redis-service.yaml
job.yaml
rediswq.py
worker.py
printed.
Start a temporary interactive pod for running the Redis CLI.
kubectl run -i --tty temp --image redis --command
"/bin/sh"
Waiting
for
pod default/redis2-c7h78 to be running, status is Pending, pod ready:
false
Hit enter
for
command
prompt
Now hit enter, start the redis CLI, and create a list with some work items in it.
# redis-cli -h redis
redis:6379&gt; rpush job2 "apple"
(integer) 1
redis:6379&gt; rpush job2 "banana"
(integer) 2
redis:6379&gt; rpush job2 "cherry"
(integer) 3
redis:6379&gt; rpush job2 "date"
(integer) 4
redis:6379&gt; rpush job2 "fig"
(integer) 5
redis:6379&gt; rpush job2 "grape"
(integer) 6
redis:6379&gt; rpush job2 "lemon"
(integer) 7
redis:6379&gt; rpush job2 "melon"
(integer) 8
redis:6379&gt; rpush job2 "orange"
(integer) 9
redis:6379&gt; lrange job2 0 -1
1) "apple"
2) "banana"
3) "cherry"
4) "date"
5) "fig"
6) "grape"
7) "lemon"
8) "melon"
9) "orange"
So, the list with key
job2
will be our work queue.
Note: if you do not have Kube DNS setup correctly, you may need to change
the first step of the above block to
redis-cli -h $REDIS_SERVICE_HOST
.
Now we are ready to create an image that we will run.
We will use a python worker program with a redis client to read
the messages from the message queue.
A simple Redis work queue client library is provided,
called rediswq.py (
Download
).
The "worker" program in each Pod of the Job uses the work queue
client library to get work. Here it is:
application/job/redis/worker.py
import
time
import
rediswq
host
=
"redis"
# Uncomment next two lines if you do not have Kube-DNS working.
# import os
# host = os.getenv("REDIS_SERVICE_HOST")
q
=
rediswq
.
RedisWQ(name
=
"job2"
, host
=
host)
print
(
"Worker with sessionID: "
+
q
.
sessionID())
print
(
"Initial queue state: empty="
+
str
(q
.
empty()))
while
not
q
.
empty():
item
=
q
.
lease(lease_secs
=
10
, block
=
True
, timeout

=
2
)
if
item
is
not
None
:
itemstr
=
item
.
decode(
"utf-8"
)
print
(
"Working on "
+
itemstr)
time
.
sleep(
10
)
# Put your actual work here instead of sleep.
q
.
complete(item)
else
:
print
(
"Waiting for work"
)
print
(
"Queue empty, exiting"
)
You could also download
worker.py
,
, and
files, then build
the image:
docker build -t job-wq-2 .
Push the image
For the
Docker Hub
docker tag job-wq-2 &lt;username&gt;/job-wq-2
docker push &lt;username&gt;/job-wq-2
You need to push to a public repository or
configure your cluster to be able to access
your private repository
.
Registry
, tag
docker tag job-wq-2 gcr.io/&lt;project&gt;/job-wq-2
gcloud docker -- push gcr.io/&lt;project&gt;/job-wq-2
Here is the job definition:
application/job/redis/job.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
job-wq-2
spec
:
:
2
template
:
metadata
:
name
:
job-wq-2
spec
:
:
-
name
:
c
image
:
gcr.io/myproject/job-wq-2
:
OnFailure
Be sure to edit the job template to
change
gcr.io/myproject
to your own path.
In this example, each pod works on several items from the queue and then exits when there are no more items.
Since the workers themselves detect when the workqueue is empty, and the Job controller does not
know about the workqueue, it relies on the workers to signal when they are done working.
The workers signal that the queue is empty by exiting with success. So, as soon as any worker
exits with success, the controller knows the work is done, and the Pods will exit soon.
So, we set the completion count of the Job to 1. The job controller will wait for the other pods to complete
too.
Now wait a bit, then check on the job.
kubectl describe jobs/job-wq-2
Name:             job-wq-2
Selector:         controller-uid
=
b1c7e4e3-92e1-11e7-b85e-fa163ee3c11f
Labels:           controller-uid
=
job-name
=
job-wq-2
Parallelism:
2
Completions:      &lt;unset&gt;
Start Time:       Mon,
11
Jan
2016
17:07:59 -0800
Pods Statuses:
1
Running /
0
Succeeded /
0
Failed
Labels:       controller-uid
=
job-name
=
job-wq-2
c:
Image:              gcr.io/exampleproject/job-wq-2
Port:
Environment:        &lt;none&gt;
Mounts:             &lt;none&gt;
Volumes:              &lt;none&gt;
Events:
FirstSeen    LastSeen    Count    From            SubobjectPath    Type        Reason            Message
---------    --------    -----    ----            -------------    --------    ------            -------
33s          33s
1
{
job-controller
}
Normal      SuccessfulCreate  Created pod: job-wq-2-lglf8
kubectl
wait
--for
=
condition
=
complete
--timeout
=
300s job/job-wq-2
kubectl logs pods/job-wq-2-7r7b2
Worker with sessionID: bbd72d0a-9e5c-4dd6-abf6-416cc267991f
Initial queue state: empty=False
Working on banana
Working on date
Working on lemon
As you can see, one of our pods worked on several work units.
If running a queue service or modifying your containers to use a work queue is inconvenient, you may
.
If you have a continuous stream of background processing work to run, then
consider running your background workers with a
instead,
and consider running a background processing library such as
https://github.com/resque/resque
.
4 - Indexed Job for Parallel Processing with Static Work Assignment
In this ex

ample, you will run a Kubernetes Job that uses multiple parallel
.
Start an
Indexed
.
Job
.
minikube
.
Read the
index.
rev
rev
busybox
As this is only an example, each Pod only does a tiny piece of work (reversing a short
the
Indexed
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
5
:
3
:
Indexed
template
:
spec
:
:
Never
:
-
name
:
'input'
image
:
command
:
-
"bash"
-
"-c"
- |
:
-
mountPath
:
/input
name
:
input
:
-
name
:
'worker'
image
:
command
:
-
"rev"
-
:
-
mountPath
:
/input
name
:
input
volumes
:
-
name
:
input
emptyDir
:
{}
.
API
from a
.
Alternatively

, you can directly
,
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
5
:
3
:
Indexed
template
:
spec
:
:
Never
:
-
name
:
'worker'
image
:
command
:
-
"rev"
-
:
-
mountPath
:
/input
name
:
input
volumes
:
-
name
:
input
:
items
:
-
path
:
fieldRef
:
fieldPath
:
Because
kubectl
wait
--for
=
condition
=
complete
--timeout
=
   input:
    Command:
      bash
      -c
      echo ${items[$JOB_COMPLETION_INDEX]} &gt; /input/data.txt

    Mounts:
   worker:
    Command:
      rev
    Mounts:
  Volumes:
   input:
    Medium:
Events:
xuq
5 - Job with Pod-to-Pod Communication
In this example, you will run a Job in
Indexed completion mode
configured such that
the pods created by the Job can communicate with each other using pod hostnames rather than pod IP addresses.
Pods within a Job might need to communicate among themselves. The user workload running in each pod could query the Kubernetes API server
to learn the IPs of the other Pods, but it's much simpler to rely on Kubernetes' built-in D

NS resolution.
Jobs in Indexed completion mode automatically set the pods' hostname to be in the format of
${jobName}-${completionIndex}
. You can use this format to deterministically build
pod hostnames and enable pod communication
without
needing to create a client connection to
the Kubernetes control plane to obtain pod hostnames/IPs via API requests.
This configuration is useful
for use cases where pod networking is required but you don't want to depend on a network
connection with the Kubernetes API server.
Job
.
minikube
.
Note:
If you are using MiniKube or a similar tool, you may need to take
extra steps
to ensure you have DNS.
Starting a Job with Pod-to-Pod Communication
To enable pod-to-pod communication using pod hostnames in a Job, you must do the following:
Set up a
headless service
with a valid label selector for the pods created by your Job. The headless service must be in the same namespace as
the Job. One easy way to do this is to use the
job-name: &lt;your-job-name&gt;
selector, since the
job-name
label will be automatically added by Kubernetes. This configuration will trigger the DNS system to create records of the hostnames of
the pods running your Job.
Configure the headless service as subdomain service for the Job pods by including the following value in your Job template spec:
subdomain
:
&lt;headless-svc-name&gt;
Example
Below is a working example of a Job with pod-to-pod communication via pod hostnames enabled.
The Job is completed only after all pods successfully ping each other using hostnames.
Note:
In the Bash script executed on each pod in the example below, the pod hostnames can be prefixed
by the namespace as well if the pod needs to be reached from outside the namespace.
:
v1
kind
:
Service
metadata
:
name
:
headless-svc
spec
:
clusterIP
:
None
# clusterIP must be None to create a headless service
selector
:
job-name
:
example-job
# must match Job name
---
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
3
:
3
:
Indexed
template
:
spec
:
subdomain
:
# has to match Service name
:
Never
:
-
name
:
example-workload
image
:
bash:latest
command
:
- bash
- -c
- |
for i in 0 1 2
do
gotStatus="-1"
wantStatus="0"
while [ $gotStatus -ne $wantStatus ]
do
ping -c 1 example-job-${i}.headless-svc &gt; /dev/null 2&gt;&amp;1
gotStatus=$?
if [ $gotStatus -ne $wantStatus ]; then
echo "Failed to ping pod example-job-${i}.headless-svc, retrying in 1 second..."
sleep 1
fi
done
echo "Successfully pinged pod: example-job-${i}.headless-svc"
done
After applying the example above, reach each other over the network
using:
&lt;pod-hostname&gt;.&lt;headless-service-name&gt;
. You should see output similar to the following:
kubectl logs example-job-0-qws42
Failed to ping pod example-job-0.headless-svc, retrying in 1 second...
Successfully pinged pod: example-job-0.headless-svc
Successfully pinged pod: example-job-1.headless-svc
Successfully pinged pod: example-job-2.headless-svc
Note:
Keep in mind that the
name format used
in this example would not work with DNS policy set to
None
or
Default
.
You can learn more about pod DNS policies
here
.
6 - Parallel Processing using Expansions
This task demonstrates running multiple
Jobs
based on a common template. You can use this approach to process batches of wor

k in
parallel.
For this example there are only three items:
apple
,
banana
, and
cherry
.
The sample Jobs process each item by printing a string then pausing.
See
using Jobs in real workloads
to learn about how
this pattern fits more realistic use cases.
You should be familiar with the basic,
Job
.
minikube
For basic templating you need the command-line utility
sed
.
To follow the advanced templating example, you need a working installation of
Python
, and the Jinja2 template
library for Python.
Once you have Python set up, you can install Jinja2 by running:
pip install --user jinja2
Create Jobs based on a template
First, download the following template of a Job to a file called
job-tmpl.yaml
.
Here's what you'll download:
application/job/job-tmpl.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
process-item-$ITEM
labels
:
jobgroup
:
jobexample
spec
:
template
:
metadata
:
name
:
labels
:
jobgroup
:
spec
:
:
-
name
:
c
image
:
command
:
[
"sh"
,
"-c"
,
"echo Processing item $ITEM &amp;&amp; sleep 5"
]
:
Never
# Use curl to download job-tmpl.yaml
curl -L -s -O https://k8s.io/examples/application/job/job-tmpl.yaml
The file you downloaded is not yet a valid Kubernetes
manifest
.
Instead that template is a YAML representation of a Job object with some placeholders
that need to be filled in before it can be used. The
$ITEM
syntax is not meaningful to Kubernetes.
Create manifests from the template
The following shell snippet uses
sed
to replace the string
$ITEM
with the loop
variable, writing into a temporary directory named
jobs
. Run this now:
# Expand the template into multiple files, one for each item to be processed.
mkdir ./jobs
for
i in apple banana cherry
do
cat job-tmpl.yaml | sed
"s/\$ITEM/
$i
/"
&gt; ./jobs/job-
$i
.yaml
done
Check if it worked:
ls jobs/
job-apple.yaml
job-banana.yaml
job-cherry.yaml
You could use any type of template language (for example: Jinja2; ERB), or
write a program to generate the Job manifests.
Create Jobs from the manifests
Next, create all the Jobs with one kubectl command:
kubectl create -f ./jobs
job.batch/process-item-apple created
job.batch/process-item-banana created
job.batch/process-item-cherry created
Now, check on the jobs:
jobs
-l
jobgroup
=
NAME                  COMPLETIONS   DURATION   AGE
process-item-apple    1/1           14s        22s
process-item-banana   1/1           12s        21s
process-item-cherry   1/1           12s        20s
Using the
-l
option to kubectl selects only the Jobs that are part
of this group of jobs (there might be other unrelated jobs in the system).
You can check on the Pods as well using the same
:
jobgroup
=
process-item-apple-kixwv    0/1       Completed   0          4m
process-item-banana-wrsf7   0/1       Completed   0          4m
process-item-cherry-dnfu9   0/1       Completed   0          4m
We can use this single command to check on the output of all jobs at once:
jobgroup
=
Processing item apple
Processing item banana
Processing item cherry
Clean up
# Remove the Jobs you created
# Your cluster automatically cleans up their Pods
kubectl delete job -l
jobgroup
=
Use advanced template parameters
In the
first example
, each instance of the templat

e had one
parameter, and that parameter was also used in the Job's name. However,
names
are restricted
to contain only certain characters.
This slightly more complex example uses the
Jinja template language
to generate manifests
and then objects from those manifests, with a multiple parameters for each Job.
For this part of the task, you are going to use a one-line Python script to
convert the template to a set of manifests.
First, copy and paste the following template of a Job object, into a file called
job.yaml.jinja2
:
{% set params = [{ "name": "apple", "url": "http://dbpedia.org/resource/Apple", },
                  { "name": "banana", "url": "http://dbpedia.org/resource/Banana", },
                  { "name": "cherry", "url": "http://dbpedia.org/resource/Cherry" }]
%}
{% for p in params %}
{% set name = p["name"] %}
{% set url = p["url"] %}
---
kind: Job
metadata:
  name: jobexample-{{ name }}
  labels:
    jobgroup: jobexample
spec:
  template:
    metadata:
      name: jobexample
      labels:
        jobgroup: jobexample
    spec:
      containers:
      - name: c
        image: busybox:1.28
        command: ["sh", "-c", "echo Processing URL {{ url }} &amp;&amp; sleep 5"]
      restartPolicy: Never
{% endfor %}
The above template defines two parameters for each Job object using a list of
python dicts (lines 1-4). A
for
loop emits one Job manifest for each
set of parameters (remaining lines).
This example relies on a feature of YAML. One YAML file can contain multiple
documents (Kubernetes manifests, in this case), separated by
---
on a line
by itself.
You can pipe the output directly to
kubectl
to create the Jobs.
Next, use this one-line Python program to expand the template:
alias
render_template
=
'python -c "from jinja2 import Template; import sys; print(Template(sys.stdin.read()).render());"'
Use
to convert the parameters and template into a single
YAML file containing Kubernetes manifests:
# This requires the alias you defined earlier
cat job.yaml.jinja2 | render_template &gt; jobs.yaml
You can view
jobs.yaml
to verify that the
script worked
correctly.
Once you are happy that
is working how you intend,
you can pipe its output into
kubectl
:
cat job.yaml.jinja2 | render_template | kubectl apply -f -
Kubernetes accepts and runs the Jobs you created.
Clean up
jobgroup
=
Using Jobs in real workloads
In a real use case, each Job performs some substantial computation, such as rendering a frame
of a movie, or processing a range of rows in a database. If you were rendering a movie
you would set
$ITEM
to the frame number. If you were processing rows from a database
table, you would set
$ITEM
to represent the range of database rows to process.
In the task, you ran a command to collect the output from Pods by fetching
their logs. In a real use case, each Pod for a Job writes its output to
durable storage before completing. You can use a PersistentVolume for each Job,
or an external storage service. For example, if you are rendering frames for a movie,
use HTTP to
PUT
the rendered frame data to a URL, using a different URL for each
frame.
Labels on Jobs and Pods
After you create a Job, Kubernetes automatically adds additional
labels
that
distinguish one Job's pods from another Job's pods.
In this example, each Job and its Pod template have a label:
jobgroup=jobexample
.
Kubernetes itself pays no attention to labels named
jobgroup
. Setting a label
for all the Jobs you create from a template makes it convenient to operate on all
those Jobs at once.
In the
you used a template to
create several Jobs. The template ensures that each Pod also gets the same label, so
you can check on all Pods for these templated Jobs with a single command.
Note:
The label key
jobgroup
is not special or reserved.
You can pick your own labelling scheme.
Ther

e are
that you can use if you wish.
If you plan to create a large number of Job objects, you may find that:
Even using labels, managing so many Jobs is cumbersome.
If you create many Jobs in a batch, you might place high load
on the Kubernetes control plane. Alternatively, the Kubernetes API
server could rate limit you, temporarily rejecting your requests with a 429 status.
You are limited by a
on Jobs: the API server permanently rejects some of your requests
when you create a great deal of work in one batch.
There are other
that you can use to process large amounts of work without creating very many Job
objects.
You could also consider writing your own
to manage Job objects automatically.
7 - Handling retriable and non-retriable pod failures with Pod failure policy
,
,
within a
Job
.
,
or
taint
Job
.
minikube
.
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
8
:
2
template
:
spec
:
:
Never
:
-
name
:
main
image
:
command
:
[
"bash"
]
args
:
- -c
:
6
:
rules
:
-
action
:
FailJob
:
:
main
operator
:
In
values
:
[
42
]
jobs
=
Failed
reason
equal
message
Container main for pod default/job-pod-failure-policy-failjob-8ckj8 failed with exit code 42 matching FailJob rule at index 0
.
Clean up
limit.
Caution:
node while the Pod is running on it (within 90s since 

the Pod is scheduled).
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
4
:
2
template
:
spec
:
:
Never
:
-
name
:
main
image
:
command
:
[
"bash"
]
args
:
- -c
:
0
:
rules
:
-
action
:
Ignore
:
-
type
:
nodeName
nodeName
=
$(
=
jsonpath
=
)
$nodeName
=
0
jobs
=
$nodeName
,
Home
Blog
Training
Partners
Community
CC BY 4.0

Scheduling Policies | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
and
running
or
.
,
,
and
Read the
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Update API Objects in Place Using kubectl patch | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

l and Set Up kubectl on Linux
Debug Pods
Debug Init Container

s
Auditing
Run Jobs
TLS
Tutorials
Security

s at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v1al

pha1)
Tasks
This task shows how to use
to update an API object in place. The exercises
in this task demonstrate a strategic merge patch and a JSON merge patch.
minikube
.
Use a strategic merge patch to update a Deployment
Here's the configuration file for a Deployment that has two replicas. Each replica
is a Pod that has one container:
application/deployment-patch.yaml
:
apps/v1
kind
:
metadata
:
name
:
patch-demo
spec
:
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
patch-demo-ctr
image
:
nginx
:
-
effect
:
key
:
dedicated
value
:
test-team
kubectl apply -f https://k8s.io/examples/application/deployment-patch.yaml
View the Pods associated with your Deployment:
The output shows that the Deployment has two Pods. The
1/1
indicates that
each Pod has one container:
patch-demo-28633765-670qr   1/1       Running   0          23s
patch-demo-28633765-j5qs3   1/1       Running   0          23s
Make a note of the names of the running Pods. Later, you will see that these Pods
get terminated and replaced by new ones.
At this point, each Pod has one Container that runs the nginx image. Now suppose
you want each Pod to have two containers: one that runs nginx and one that runs redis.
patch-file.yaml
that has this content:
spec
:
template
:
spec
:
:
-
name
:
patch-demo-ctr-2
image
:
redis
Patch your Deployment:
kubectl patch deployment patch-demo --patch-file patch-file.yaml
View the patched Deployment:
kubectl get deployment patch-demo --output yaml
The output shows that the PodSpec in the Deployment has two Containers:
:
-
image
:
redis
:
Always
name
:
...

-
image
:
nginx
:
Always
name
:
...
View the Pods associated with your patched Deployment:
The output shows that the running Pods have different names from the Pods that
were running previously. The Deployment terminated the old Pods and created two
new Pods that comply with the updated Deployment spec. The
2/2
each Pod has two Containers:
patch-demo-1081991389-2wrn5   2/2       Running   0          1m
patch-demo-1081991389-jmg7b   2/2       Running   0          1m
Take a closer look at one of the patch-demo Pods:
kubectl get pod &lt;your-pod-name&gt; --output yaml
The output shows that the Pod has two Containers: one running nginx and one running redis:
- image: redis
  ...
  ...
Notes on the strategic merge patch
The patch you did in the preceding exercise is called a
strategic merge patch
.
Notice that the patch did not replace the
list. Instead it added a new
Container to the list. In other words, the list in the patch was merged with the
existing list. This is not always what happens when you use a strategic merge patch on a list.
In some cases, the list is replaced, not merged.
With a strategic merge patch, a list is either replaced or merged depending on its
patch strategy. The patch strategy is specified by the value of the
patchStrategy
key
in a field tag in the Kubernetes source code. For example, the
field of
PodSpec
struct has a
of
merge
:
type
PodSpec
struct
{
...
Containers []Container
`json:"containers" patchStrategy:"merge" patchMergeKey:"name" ...`
...
}
You can also see the patch strategy in the
OpenApi spec
:
"io.k8s.api.core.v1.PodSpec":
{
...,
"containers":
{
"description":
"List of containers belonging to the pod.  ...."
},
"x-kubernetes-patch-merge-key":
"name"
,
"x-kubernetes-patch-strategy":
"merge"
}
And you can see the patch strategy in the
Kubernetes API documentation
.
patch-file-tolerations.yaml
spec
:
template
:
spec
:
:
-
effect
:
key
:
disktype
value
:
ssd
kubectl patch deployment patch-demo --patch-file patch-file-tolerations.yaml
The output shows that the PodSpec in the Deployment has only one Toleration:
:
-
effect
:
key
:
disktype
value
:
ssd
list in the PodSpec was replaced, not merged. This is because
the Tolerations field of PodSpec does not have a
key in its field tag. So the
strategic merge patch uses the default patch strategy, which is
replace
.
type
PodSpec
struct
{
...
Tolerations []Toleration
`json:"tolerations,omitempty" protobuf:"bytes,22,opt,name=tolerations"`
...
}
Use a JSON merge patch to update a Deployment
A strategic merge patch is different from a
JSON merge patch
.
With a JSON merge patch, if you
want to update a list, you have to specify the entire new list. And the new list completely
replaces the existing list.
The
command has a
type
parameter that you can set to one of these values:
Parameter value
Merge type
json
JSON Patch, RFC 6902
merge
JSON Merge Patch, RFC 7386
strategic
Strategic merge patch
For a comparison of JSON patch and JSON merge patch, see
JSON Patch and JSON Merge Patch
.
The default value for the
type
parameter is
strategic
. So in the preceding exercise, you
did a strategic merge patch.
Next, do a JSON merge patch on your same Deployment. Create a file named
patch-file-2.yaml
spec
:
template
:
spec
:
:
-
name
:
patch-demo-ctr-3
image
:
In your patch command, set
type
to
merge
:
kubectl patch deployment patch-demo --type merge --patch-file patch-file-2.yaml
The
lis

t that you specified in the patch has only one Container.
The output shows that your list of one Container replaced the existing
list.
spec:
- image: gcr.io/google-samples/node-hello:1.0
...
name: patch-demo-ctr-3
In the output, you can see that the existing Pods were terminated, and new Pods
were created. The
1/1
indicates that each new Pod is running only one Container.
patch-demo-1307768864-69308   1/1       Running
0
1m
patch-demo-1307768864-c86dc   1/1       Running
0
1m
Use strategic merge patch to update a Deployment using the retainKeys strategy
Here's the configuration file for a Deployment that uses the
strategy:
application/deployment-retainkeys.yaml
:
apps/v1
kind
:
metadata
:
name
:
retainkeys-demo
spec
:
selector
:
:
app
:
nginx
strategy
:
rollingUpdate
:
maxSurge
:
30
%
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
retainkeys-demo-ctr
image
:
nginx
Create the deployment:
kubectl apply -f https://k8s.io/examples/application/deployment-retainkeys.yaml
At this point, the deployment is created and is using the
strategy.
patch-file-no-retainkeys.yaml
spec
:
strategy
:
type
:
Recreate
kubectl patch deployment retainkeys-demo --type strategic --patch-file patch-file-no-retainkeys.yaml
In the output, you can see that it is not possible to set
type
as
Recreate
when a value is defined for
spec.strategy.rollingUpdate
:
The Deployment "retainkeys-demo" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy `type` is 'Recreate'
The way to remove the value for
when updating the value for
type
is to use the
retainKeys
strategy for the strategic merge.
Create another file named
patch-file-retainkeys.yaml
spec
:
strategy
:
$retainKeys
:
- type
type
:
Recreate
With this patch, we indicate that we want to retain only the
type
key of the
strategy
object. Thus, the
will be removed during the patch operation.
Patch your Deployment again with this new patch:
kubectl patch deployment retainkeys-demo --type strategic --patch-file patch-file-retainkeys.yaml
Examine the content of the Deployment:
kubectl get deployment retainkeys-demo --output yaml
The output shows that the strategy object in the Deployment does not contain the
key anymore:
spec
:
strategy
:
type
:
Recreate
template
:
Notes on the strategic merge patch using the retainKeys strategy
strategic merge patch with retainKeys strategy
. This method introduces a new directive
that has the following strategies:
It contains a list of strings.
All fields needing to be preserved must be present in the
list.
The fields that are present will be merged with live object.
All of the missing fields will be cleared when patching.
All fields in the
list must be a superset or the same as the fields present in the patch.
The
strategy does not work for all objects. It only works when the value of the
key in a field tag in the Kubernetes source code contains
. For example, the
Strategy
of
:
type
struct
{
...
// +patchStrategy=retainKeys
Strategy DeploymentStrategy
`json:"strategy,omitempty" patchStrategy:"retainKeys" ...`
...
}
You can also see the
strategy in the
:
"io.k8s.api.apps.v1.DeploymentSpec":
{
...,
"strategy":
{
"$ref":
"#/definitions/io.k8s.api.apps.v1.DeploymentStrategy"
,
"The deployment strategy to use to replace existing pods with new ones."
,
"retainKeys"
},
....
}
And you can see the
Kubernet

es API documentation
.
Alternate forms of the kubectl patch command
The
command takes YAML or JSON. It can take the patch as a file or
directly on the command line.
patch-file.json
{
"spec"
: {
"template"
: {
"spec"
: {
: [
{
"name"
:
"patch-demo-ctr-2"
,
"image"
:
"redis"
}
]
}
}
}
}
The following commands are equivalent:
kubectl patch deployment patch-demo --patch
'spec:\n template:\n  spec:\n   containers:\n   - name: patch-demo-ctr-2\n     image: redis'
kubectl patch deployment patch-demo --patch-file patch-file.json
'{"spec": {"template": {"spec": {"containers": [{"name": "patch-demo-ctr-2","image": "redis"}]}}}}'
Update an object's replica count using
with
The flag
--subresource=[subresource-name]
is used with kubectl commands like get, patch,
edit and replace to fetch and update
status
and
scale
subresources of the resources
(applicable for kubectl version v1.24 or more). This flag is used with all the API resources
(built-in and CRs) that have
status
or
scale
subresource. Deployment is one of the
examples which supports these subresources.
Here's a manifest for a Deployment that has two replicas:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
app
=
nginx
In the output, you can see that Deployment has two Pods. For example:
nginx-deployment-7fb96c846b-22567   1/1     Running   0          47s
nginx-deployment-7fb96c846b-mlgns   1/1     Running   0          47s
Now, patch that Deployment with
flag:
=
'scale'
--type
=
'merge'
-p
'{"spec":{"replicas":3}}'
scale.autoscaling/nginx-deployment patched
app
=
nginx
In the output, you can see one new pod is created, so now you have 3 running pods.
nginx-deployment-7fb96c846b-22567   1/1     Running   0          107s
nginx-deployment-7fb96c846b-lxfr2   1/1     Running   0          14s
nginx-deployment-7fb96c846b-mlgns   1/1     Running   0          107s
kubectl get deployment nginx-deployment -o yaml
...
spec
:
replicas
:
3
...
status
:
...
:
3
:
3
replicas
:
3
Note:
If you run
and specify
flag for resource that doesn't support that
particular subresource, the API server returns a 404 Not Found error.
Summary
In this exercise, you used
to change the live configuration
of a Deployment object. You did not change the configuration file that you originally used to
create the Deployment object. Other commands for updating API objects include
,
,
,
,
and
.
Note:
Strategic merge patch is not supported for custom resources.
Feedback
Yes
No
.
O

pen an issue in the GitHub repo if you want to
or
.
Last modified December 09, 2022 at 8:07 PM PST:
Small grammar fix (231f3ed94e)
with
Summary
Home
Blog
Training
Partners
Community
CC BY 4.0

Scale a StatefulSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
This task shows how to scale a StatefulSet. Scaling a StatefulSet refers to increasing or decreasing the number of replicas.
StatefulSets are only available in Kubernetes version 1.5 or later.
To check your version of Kubernetes, run
.
Not all stateful applications scale nicely. If you are unsure about whether to scale your StatefulSets, see
StatefulSet concepts
or
StatefulSet tutorial
for further information.
You should perform scaling only when you are confident that your stateful application
cluster is completely healthy.
Scaling StatefulSets
Use kubectl to scale StatefulSets
First, find the StatefulSet you want to scale.
kubectl get statefulsets &lt;stateful-set-name&gt;
Change the number of replicas of your StatefulSet:
kubectl scale statefulsets &lt;stateful-set-name&gt; --replicas
=
&lt;new-replicas&gt;
Make in-place updates on your StatefulSets
Alternatively, you can do
in-place updates
on your StatefulSets.
If your StatefulSet was initially created with
,
update
.spec.replicas
of the StatefulSet manifests, and then do a
:
kubectl apply -f &lt;stateful-set-file-updated&gt;
Otherwise, edit that field with
:
kubectl edit statefulsets &lt;stateful-set-name&gt;
Or use
:
kubectl patch statefulsets &lt;stateful-set-name&gt; -p
'{"spec":{"replicas":&lt;new-replicas&gt;}}'
Scaling down does not work right
You cannot scale down a StatefulSet when any of the stateful Pods it manages is unhealthy. Scaling down only takes place
after those stateful Pods become running and ready.
If spec.replicas &gt; 1, Kubernetes cannot determine the reason for an unhealthy Pod. It might be the result of a permanent fault or of a transient fault. A transient fault can be caused by a restart required by upgrading or maintenance.
If the Pod is unhealthy due to a permanent fault, scaling
without correcting the fault may lead to a state where the StatefulSet membership
drops below a certain minimum number of replicas that are needed to function
correctly. This may cause your StatefulSet to become unavailable.
If the Pod is unhealthy due to a transient fault and the Pod might become available again,
the transient error may interfere with your scale-up or scale-down operation. Some distributed
databases have issues when nodes join and leave at the same time. It is better
to reason about scaling operations at the application level in these cases, and
perform scaling only when you are sure that your stateful ap

plication cluster is
completely healthy.
deleting a StatefulSet
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Create a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
cluster
and create a simple cluster using Minikube.
1 - Using Minikube to Create a Cluster
The
Nodes
Summary:
Minikube
etcd
When you deploy applications on Kubernetes, you tell the control plane to start the application containers. The control plane schedules the containers to run on the cluster's nodes.
Now that you know what Kubernetes is, let's go 

to the online tutorial and start our first cluster!
›
2 - Interactive Tutorial - Creating a Cluster
Home
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm reset | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 (v1)
Reference
Kubeadm
or
.
Synopsis
Options
--dry-run
-h, --help
the
or
reset
the
and
""
--prefix
See the
to bootstrap a Kubernetes control-plane 

node
Feedback
Yes
No
.
or
.
Last modified January 06, 2022 at 2:50 PM PST:
kubeadm: remove ClusterStatus references (4030f7fab9)
Home
Blog
Training
Partners
Community
CC BY 4.0

FlowSchema v1beta3 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, Storage, a

nd Networking Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission 

Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials

Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate Limit Configuration

 (v1alpha1)
Reference
kind
metadata
(
)
metadata
spec
(
)
spec
status
(
)
status
name
nil
type
(int32)
is used to choose among the FlowSchemas that match a given request. The chosen FlowSchema is among those with the numerically lowest (which we take to be logically highest) MatchingPrec

edence. Each MatchingPrecedence value must be ranged in [1,10000]. Note that if the precedence is not specified, it will be set to 1000 as default.
rules
rules
kind
group
https://github.com/kubernetes/apiserver/blob/master/pkg/authentication/user/user.go
name
namespace
user
name
"
verbs
is a slice of ResourcePolicyRules that identify matching requests according 

to their verb and the target resource. At least one of
and
apiGroups
resources
verbs
(boolean)
field must contain a non-empty list.
is true.
type
(Time)
(string)
message
(string)
reason
(string)
status
(string)
type
kind
metadata
(
ListMeta
)
metadata
items
([]
items
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of 

the specified FlowSchema
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
fieldS

elector
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Configuration APIs | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
1 - Client Authentication (v1)
Field
string
kind
string
spec
status
Cluster
Field
server
string
string
bool
[]byte
proxy-url
string
cluster.
bool
config
clusters:
cluster:
...
specified with 

the cluster info. Using this field to store secret data is not
Field
cluster
Cluster
bool
Field
token
string
string
string
2 - Client Authentication (v1beta1)
Field
string
kind
string
spec
status
Cluster
Field
server
string
string
bool
[]byte
proxy-url
string
cluster.
bool
config
clusters:
cluster:
...

Field
cluster
Cluster
bool
Field
token
string
string
string
3 - Event Rate Limit Configuration (v1alpha1)
Field
string
kind
string
limits
[]Limit
Limit
Field
type
LimitType
qps
int32
burst
int32
cacheSize
int32
LimitType
(Alias of
string
)
Limit
4 - Image Policy API (v1alpha1)
Field
string
imagepolicy.k

8s.io/v1alpha1
kind
string
metadata
metadata
field.
spec
status
Field
image
string
Field
.
namespace
string
Field
allowed
bool
reason
string
5 - kube-apiserver Admission (v1)
Field
string
kind
string
request
response
Field
uid
kind
resource
string
and
,
with
kind: {group:"apps", version:"v1", kind:"Deploy

ment"}
and
and
,
with
and
string
name
string
namespace
string
operation
Operation
userInfo
object
oldObject
dryRun
bool
options
e.g.
or
.
Field
uid
allowed
bool
status
patch
[]byte
patchType
PatchType
the admission web

hook to add additional context to the audit log for this request.
warnings
[]string
Operation
(Alias of
string
)
PatchType
(Alias of
string
)
6 - kube-apiserver Audit Configuration (v1)
Event
EventList
Policy
Event
EventList
Field
string
kind
string
Event
level
Level
auditID
stage
Stage
string
verb
string
user
sourceIPs
[]string
userAgent
string
Note that the UserAgent is provided by the client, and must not be trusted.
objectRef
EventList
Field
De

scription
string
kind
string
EventList
metadata
items
[]Event
Policy
Field
string
kind
string
Policy
metadata
metadata
field.
rules
[]Stage
bool
Field
string
kind
string
metadata
items
[]Policy
Field
group
string
resources
[]string
'
' matches all resources and their subresources.
'pods/
[]string
Level
(Alias of
string
)
Event
Event
Field
resource
string
namespace
string
name
string
uid
apiGroup
string
string
string
string
Policy
Field
level
Level
users
[]string
[]string
verbs
[]string
The verbs that match this rul

e.
resources
[]string
[]string
Examples:
"/healthz
[]Stage
bool
Stage
(Alias of
string
)
Event
Policy
7 - kube-apiserver Configuration (v1)
Field
string
kind
string
plugins
Field
name
string
path
string
8 - kube-apiserver Configuration (v1alpha1)
Field
string
kind
string
plugins
Field
string
kind
string
Field
string
kind
string
endpoint
string
int32
AdmissionCon

figuration
Field
name
string
path
string
Field
transport
Transport
Field
name
string
(Alias of
string
)
Transport
Field
url
string
tlsConfig
TLSConfig
TLSConfig
Field
caBundle
string
clientKey
string
string
Transport
Field
tcp
uds
Transport
Field
udsName
string
9 - kube-apiserver Encryption Configuration (v1)
Field

string
kind
string
resources
Field
keys
[]Key
Field
string
name
string
cachesize
int32
endpoint
string
timeout
Key
Field
name
string
secret
string
Field
aesgcm
aescbc
secretbox
identity
kms
Field
resources
[]string
providers
Field
keys
[]Key
10 - kube-proxy Configuration (v1alpha1)
Field
string
kind
string
string
string
string
[

Required]
bool
bool
string
string
iptables
ipvs
int32
mode
ProxyMode
portRange
string
conntrack
than 0.
[]string
winkernel
string
LocalMode
Field
string
string
Field
int32
min
int32
min is the minimum value of

 connect-tracking records to allocate,
Field
int32
bool
bool
'2h22m').
Field
'2h22m').
scheduler
string
[]string
strictARP
bool
Field
string
sourceVip
string
enableDSR
bool
with DSR
string
bool
Windows
LocalMode
(Alias of
string
)
ProxyMode
(Alias of
string
)
If the proxy mode is unspecified, the b

est-available proxy mode will be used (currently this
is
iptables
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
enabled.
string
string
string
11 - kube-scheduler Configuration (v1)
Field
string
kind
string
int32
MinCandidateNodesPercentage is the min

imum number of candidates to
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
int32
int64
int64
profiles
extenders
Field
string
kind
string
specified in the PodSpec. That

 is, Nodes need to satisfy AddedAffinity
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
Field
string
kind
string
.
Field
string
kind
string
int64
shape
Extender
Field
urlPrefix
string
string
Verb for the filter call, empty if not supported. This verb is appended to the URLPrefix when issuing the filter

 call to extender.
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool
ignorable
bool
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
Field
string
percen

tageOfNodesToScore
int32
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
Field
name
string
args
PluginSet
Plugins
Field
enabled
[]Plugin
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSe

t
PluginSet
preScore
PluginSet
score
PluginSet
reserve
PluginSet
permit
PluginSet
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
set in both
and
including
and
(Alias of
string
)
Field
shape
Field
name
string
weight
int64
Field
type
resources
(Alias of
string
)
RequestedToCapacit

yRatioParam
Field
int32
score
int32
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
enabled.
string
string
string
12 - kube-scheduler Configuration (v1beta2)
Field
string
kind
string
int32
shortlist when dry running preemption as a

 percentage of number of nodes.
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
string
string
int32
int64
int64
profiles
extenders
NodeAffinityArgs holds argu

ments to configure the NodeAffinity plugin.
Field
string
kind
string
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
Field
string
kind
string
.
Field
string
kind
string
int64
shape
Extender
i

t is assumed that the extender chose not to provide that extension.
Field
urlPrefix
string
string
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool
ignorable
bool
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
KubeSch

edulerConfiguration
Field
string
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
Field
name
string
args
PluginSet
Plugins
Field
enabled
[]Plugin
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSet
PluginSet
preScore
PluginSet
score
PluginSet
Score is a list of plugins that should be invoked when ranking nodes that have pa

ssed the filtering phase.
reserve
PluginSet
permit
PluginSet
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
(Alias of
string
)
Field
shape
Field
name
string
weight
int64
Field
type
resources
(Alias of
string
)
Field
int32
score
int32
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
LeaderElectionConfiguration define

s the configuration of leader election
Field
bool
enabled.
string
string
string
13 - kube-scheduler Configuration (v1beta3)
Field
string
kind
string
int32
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
DebuggingConf

iguration holds configuration for Debugging related features
int32
int64
int64
profiles
extenders
Field
string
kind
string
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
Field
string
kind
string

.
Field
string
kind
string
int64
shape
Extender
Field
urlPrefix
string
string
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool
one of the extended resources in this list. If empty or unspe

cified,
ignorable
bool
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
Field
string
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
Field
name
string
args
PluginSet
Plugins
Field
enabled
[]Plugin
Enabled specifies plugins that should be enabled in ad

dition to default plugins.
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSet
PluginSet
preScore
PluginSet
score
PluginSet
reserve
PluginSet
permit
PluginSet
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
set in both
and
including
and
(Alias of
string
)
PodTopologySpreadConstraint

sDefaulting defines how to set default constraints
Field
shape
Field
name
string
weight
int64
Field
type
resources
(Alias of
string
)
Field
int32
score
int32
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
enabled.
than or equal to the lease duration. This is only appl

icable if leader
string
string
string
14 - kubeadm Configuration (v1beta2)
Overview
Basics
--config
---
).
:
kind
:
:
kind
:
:
kind
:
:
kind
:
:
kind
:
init
or
join
--config
between InitConfiguration and ClusterConfi

guration is mandatory.
:
kind
:
:
...
:
...
:
kind
:
:
...
etcd
:
...
apiServer
:
extraArgs
:
...
:
...
...
:
kind
:
...
:
kind
:
...
run.
:
kind
:
:
-
token
:
:
ttl
:
"24h"
-
token
:
:
usages
:
- signing
groups
:
:
name
:
criSocket
:
taints
:
-
key
:
value
:
effect
:
:
v
:
4
:
:
:
bindPort
:
6443
:
---
:
kind
:
etcd
:
local
:
:
imageTag
:
"3.2.24"
dataDir
:
extraArgs
:
:
:
-
:
-
:
:
podSubnet
:
dnsDomain
:
:
"v1.12.0"
co

ntrolPlaneEndpoint
:
apiServer
:
extraArgs
:
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
certSANs
:
-
-
:
4m0s
:
extraArgs
:
"20"
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
scheduler
:
extraArgs
:
address
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
:
:
:
false
:
---
:
kind
:
---
:
kind
:
--config
:
kind
:
...
Field
string
kind
string
etcd
Etcd
etcd
string
string
In case the
+
bindPort
the
bindPort
is used.
apiServer
APIServer
apiServer
scheduler
scheduler
dns
DNS
dns
string
string
If empty,
ci/
)
will be used for all the other image

s.
bool
hyperkube
string
Field
string
kind
string
Field
string
kind
string
string
phase.
Field
string
kind
string
string
discovery
Discovery
discovery
Field
string
bindPort
int32
bindPort
APIServer
Field
Contr

olPlaneComponent
certSANs
[]string
certSANs
to appear.
Field
token
token
string
ttl
ttl
expires
and
ttl
expires
expires
ttl
.
expires
and
ttl
usages
[]string
usages
groups
[]string
groups
Discovery
Field
token
string
token
string
[]string
bool
Field
-
string
-
string
APIServer
Field
extraArgs
extraArgs
DNS
Field
type
type
ImageMeta
ImageMeta
ImageMeta
(Alias of
string
)
DNS
Discovery
Appea

rs in:
Field
and
file
file
file
and
file
string
If
,
If
file
must be set
timeout
timeout
Etcd
Field
local
LocalEtcd
local
local
and
external
external
external
local
and
external
Etcd
Field
endpoints
[]string
endpoints
caFile
string
caFile
certFile
string
certFile
keyFile
string
keyFile
Discovery
Field
string
Field
name
string
name
hostPath
string
hostPath
mountPath
string
mountPath
readOnly
bool
readOnly
pathType
pathType
ImageMeta
DNS
LocalEtcd
Field
string
imageTag
string
imageTag
Field
string
LocalEtcd
Etcd
LocalEtcd describes 

that kubeadm should run an etcd cluster locally.
Field
ImageMeta
ImageMeta
ImageMeta
dataDir
string
dataDir
extraArgs
extraArgs
[]string
[]string
Field
string
podSubnet
string
podSubnet
dnsDomain
string
dnsDomain
Field
name
string
name
is the
in this
or
criSocket
string
criSocket
taints
taints
taints: []
[]string
15 - kubeadm Configuration (v1beta3)
Overview
Add "InitConfiguration.NodeRegistration.ImagePullPolicy" and "JoinConfiguration.NodeRegistration.ImagePullPo

licy"
Basics
--config
---
).
:
kind
:
:
kind
:
:
kind
:
:
kind
:
:
kind
:
init
or
join
--config
:
kind
:
:
...
:
...
:
kind
:
:
...
etcd
:
...
apiServer
:
extraArgs
:
...
:
...
...
that holds configuration for the networking 

topology of the cluster; use it e.g. to customize
etcd
:
kind
:
...
:
kind
:
...
run.
:
kind
:
:
-
token
:
:
ttl
:
"24h"
-
token
:
:
usages
:
- signing
groups
:
:
name
:
criSocket
:
taints
:
-
key
:
value
:
effect
:
:
v
:
4
:
:
:
:
bindPort
:
6443
:
:
---
:
kind
:
etcd
:
local
:
:
imageTag
:
"3.2.24"
dataDir
:
extraArgs
:
:
:
-
:
-
:
:
podSubnet
:
dnsDomain
:
:
"v1.21.0"
:
apiServer
:
extraArgs
:
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
certSANs
:
-
-
:
4m0s
:
extraArgs
:
"20"
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
scheduler
:
extraArgs
:
address
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
:
:
:
---
:
kind
:
---
:
kind
:
Kube

adm join configuration types
with the
--config
:
kind
:
...
Field
token
token
string
ttl
ttl
24h
.
expires
and
ttl
expires
expires
ttl
.
expires
and
ttl
usages
[]string
usages
groups
[]string
groups
Field
-
string
-
string
Field
string
kind
string
etcd
Etcd
etcd
string
string
In case the
+
bindPort
the
bindPort
is used.
could
apiServer
APIServer
apiServer
scheduler
scheduler
dns
DNS
dns
string
string
sets the

 container registry to pull images from.
If empty,
ci/
)
string
runs.
ConfigMap
Field
string
kind
string
string
uploaded in a Secret in the cluster during the
phase.
[]string
command.
patches
Patches
patches
.
Field
string
kind
string
string
discovery
Discovery
discovery
[]string
command.
The flag
patches
Patches
patches
.
Field
string
bindPort
int32
bindPort
APIServer
Cl

usterConfiguration
Field
certSANs
[]string
certSANs
Discovery
Field
token
string
token
string
[]string
bool
via
APIServer
Field
extraArgs
extraArgs
is an extra set of host volumes, mounted to the control plane component.
DNS
Field
ImageMeta
ImageMeta
ImageMeta
imageMeta
Discovery
Field
and
file
file
file
and
file
string
If
, but
file
must be set
timeout
timeout
Etcd
Field
local
LocalEtcd
local
local
and
external
external
external
local
and
external
Etcd
Field
endpoints
[]string
endpoints
caFile
string
caFile
ce

rtFile
string
certFile
keyFile
string
keyFile
Discovery
Field
string
Field
name
string
name
hostPath
string
hostPath
mountPath
string
mountPath
hostPath
readOnly
bool
readOnly
pathType
pathType
hostPath
.
ImageMeta
DNS
LocalEtcd
Field
string
imageTag
string
imageTag
Field
string
LocalEtcd
Etcd
Field
ImageMeta
ImageMeta
ImageMeta
dataDir
string
dataDir
extraArgs
extraArgs
[]string
[]string
Field
string
podSubnet
string
podSubnet
dnsDomain
string
dnsDomain
Field
name
string
name
is the
or
criSocket
string
criSocket

taints
taints
i.e.
taints: []
[]string
Patches
Field
directory
string
directory
kubectl.
The default "patchtype" is "strategic". "extension" must be either "json" or "yaml".
16 - kubeconfig (v1)
Config
AuthInfo
Field
string
[]byte
string
[]byte
token
string
tokenFile
string
as
string
as-uid
string
as-groups
[]string
username
string
password
string
exec
Extensions holds additional information. This is useful for extenders so that reads and writes don't clobber unk

nown fields
AuthInfo
Field
name
string
config
Cluster
Field
server
string
string
bool
string
[]byte
proxy-url
string
bool
Context
Field
cluster
string
user
string
namespace
string
AuthInfo
Field
command
string
args
[]string
env
string
string
bool
to false. Package k8s.io/client-go/tools/auth/exec provides helper 

methods for
details.
Field
name
string
value
string
(Alias of
string
)
Config
Field
name
string
user
AuthInfo
Config
Field
name
string
cluster
Cluster
Config
Field
name
string
context
Context
Config
AuthInfo
Cluster
Context
Field
name
string
extension
Config
Field
colors
bool
17 - Kubelet Configuration (v1)
Field
string
kind
string
providers
Field
name
string
[]string
to provide credentials. Images are expected to contain the registr

y domain
'.
gcr.io
.
string
args
[]string
env
Field
name
string
value
string
Field
json
Field
bool
Field
format
string
text
verbosity
vmodule
options
TracingCo

Field
endpoint
string
int32
(Alias of
)
(Alias of
uint32
)
18 - Kubelet Configuration (v1alpha1)
Field
string
kind
string
providers
Field
name
string
[]string
*.k8s.io
or
k8s.*.io
k8s.*
.
*.io
*.k8s.io
.
gcr.io
string
credentialprovider.

kubelet.k8s.io/v1alpha1
args
[]string
env
Field
name
string
value
string
19 - Kubelet Configuration (v1beta1)
Field
string
kind
string
providers
Field
string
kind
string
bool
string
new data.
string
address
string
port
int32
int32
string
string
[]string
string
bool
rotateCertificates enables client certificate rotation. The 

Kubelet will request a
bool
Defaults:
webhook:
Defaults:
webhook:
int32
Default: 5
int32
int32
Default: 5
int32
bool
bool
enableContentionProfiling enables lock contention profiling, if enableDebuggingHandlers is true.
int32
string
int32
string
[]string
nodeStatusReportFrequency's default valu

e will be set to
int32
int32
int32
string
string
string
cgroupRoot is the root cgroup to use for pods. This is handled by the
bool
string
string
string
string
none
string
that topology manage

r requests and hint providers generate. Valid values include:
container
pod
string
maxPods
int32
podCIDR
string
int64
string
resolvConf is the resolver configuration file used as the basis
runOnce
bool
bool
.
int32
int64
string
int32
Default: 5
int32
bool
not
{"memory.available

": "300Mi"}
.
Default:
.
.
int32
Default: 0
.
int32
Default: 0
bool
disables kubelet from executing any attach/detach operations.
bool
bool
int32
int32
bool
string
int32
containerLogMaxFiles spe

cifies the maximum number of container log files that can
Default: 5
Get
Cache
Watch
string
string
, e.g.:
1.16
.
systemReservedCgroup
string
to enforce
Refer to
string
to enforce
Refer to
[]string
none
,
pods
,
and
.
If
none
When
When
Refer to
[]string
*
).
,
,
,
,
and
net.*
"
string
string
bool
exceeded rather than p

olling.
logging
Refer to
Default:
bool
When all pods have exited or reached their grace periods, the Kubelet will release
by the
.
bool
bool
bool
float64
MemoryThrottlingFactor specifies the factor multiplied by the memor

y limit or node allocatable memory
bool
tracing
bool
Field
string
kind
string
source
Field
name
string
[]string
'.
gcr.io
.
if a cache duration is not provided in the plugin response. This field is re

quired.
string
args
[]string
env
Field
name
string
value
string
Field
enabled
bool
.
Field
x509
webhook
anonymous
Field
mode
and
Webhook
.
webhook
(Alias of
string
)
Field
enabled
bool
cacheTTL
Field
Field
string
Field
numaNode
int32
limits
Field
string
(Alias of
string
)
ShutdownGracePeriodByPodPriority specifies the shutdown grace period for Pods based o

n their associated priority class value
Field
priority
int32
int64
Field
json
Field
bool
Field
format
string
text
verbosity
vmodule
options
Field
endpoint
string
int32
(Alias of
)
(Alias of
uint32
)
20 - Kubelet CredentialProvider (v1)
Field
string
kind
string
image
string
t

o extract any information required to fetch credentials.
Field
string
kind
string
auth
'.
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
21 - Kubelet CredentialProvider (v1alpha1)
Field
string
kind
string
image
string
to

 extract any information required to fetch credentials.
Field
string
kind
string
auth
'.
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
22 - Kubelet CredentialProvider (v1beta1)
Field
string
kind
string
image
string
credential provider plugin request. Plugins may optionally parse the

 image
Field
string
kind
string
auth
'.
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
23 - WebhookAdmission Configuration (v1)
Field
string
kind
string
string
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. Fo

r a list of trademarks of The Linux Foundation, please see our

Migrate Replicated Control Plane To Use Cloud Controller Manager | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator 

pattern
Tasks
Debug Pods
Determine the Reason fo

r Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Cluster

 Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
component
.
and the
Leader Migration can be enabled by setting
on
or
and
It is assumed that the control plane is running Kubernetes version N and to be upgraded to version N + 1. Although it is possible to migrate within the same version, ideally the migration should be performed as part of an upgrade so that changes of configuration can be aligned to each release. The exact versions of N and N + 1 depend on each cloud

 provider. For example, if a cloud provider builds a
flag and
and
.
and
and
.
role.
runs
route
,
service
, and
kind
:
:
:
:
-
name
:
route
component
:
-
name
:
service
component
:
-
name
:
component
:
component
to
*
kind
:
:
:
:
-
name
:
route
component
:
*
-
name
:
service
component
:
*
-
name
:
component
:
*
--leader-migration-config=/etc

/leadermigration.conf
Restart
component
to
kind
:
:
:
:
-
name
:
route
component
:
-
name
:
service
component
:
-
name
:
component
:
and
.
flag set to
external
for
.
.
and
and
of version N + 1, Leader Migration has finished its job and can be safely disabled to save one Lease resource. It is safe to re-enable Leader Migration for the rollback in the future.
and
.
.
For
and
, if there are no flags that enable any in-tree cloud provider or change ownership of controllers, the default configuration can be used to avoid manual cre

ation of the configuration file.
nodeipam
kind
:
:
:
:
-
name
:
route
component
:
*
-
name
:
service
component
:
*
-
name
:
component
:
*
-
name
:
nodeipam
-
component
:
*
Read the
Feedback
Yes
No
.
or
.
Last modified June 30, 2022 at 12:08 PM PST:
Use relative links for k8s.io (34721abcac)
Home
Blog
Training
Partners
Community
CC BY 4.0

Articles on dockershim Removal and on Using CRI-compatible Runtimes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Reference
,
(
)
. (
)
"
Mirantis:
Tripwire:
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyrig

ht © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Binding | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

lpha1)
Reference
Binding
Binding
Binding
: v1
kind
: Binding
metadata
(
)
target
(
create
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
create
name
(
in path
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
Feedback
Yes
No
.
or
.
Binding
create
create
Home
Blog
Training
Partners
Community
CC BY 4.0

Declarative Management of Kubernetes Objects Using Kustomize | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Kustomize
is a standalone tool
to customize Kubernetes objects
through a
kustomization file
.
Since 1.14, Kubectl also
supports the management of Kubernetes objects using a kustomization file.
To view Resources found in a directory containing a kustomization file, run the following command:
kubectl kustomize &lt;kustomization_directory&gt;
To apply those Resources, run
with
--kustomize
or
-k
flag:
kubectl apply -k &lt;kustomization_directory&gt;
Install
kubectl
.
minikube
.
Overview of Kustomize
Kustomize is a tool for customizing Kubernetes configurations. It has the following features to manage application configuration files:
generating resources from other sources
setting cross-cutting fields for resources
composing and customizing collections of resources
Generating Resources
ConfigMaps and Secrets hold configuration or sensitive data that are used by other Kubernetes objects, such as Pods. The source of truth of ConfigMaps or Secrets are usually external to a cluster, such as a
.properties
file or an SSH keyfile.
Kustomize has
secretGenerator
and
configMapGenerator
, which generate Secret and ConfigMap from files or literals.
To generate a ConfigMap from a file, add an entry to the
files
list in
. Here is an example of generating a ConfigMap with a data item from a
file:
# Create a application.properties file
cat
&lt;&lt;EOF &gt;application.properties
FOO=Bar
EOF
cat
- name: example-configmap-1
files:
- application.properties
EOF
The generated ConfigMap can be examined with the following command:
kubectl kustomize ./
The generated ConfigMap is:
:
v1
data
:
application.properties
:
|
FOO=Bar
kind
:
ConfigMap
metadata
:
name
:
example-configmap-1-8mbdf7882g
To generate a ConfigMap from an env file, add an entry to the
envs
list in
c

onfigMapGenerator
.env
file:
# Create a .env file
cat
&lt;&lt;EOF &gt;.env
FOO=Bar
EOF
cat
envs:
- .env
EOF
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
name
:
example-configmap-1-42cfbf598f
Note:
Each variable in the
.env
file becomes a separate key in the ConfigMap that you generate. This is different from the previous example which embeds a file named
(and all its entries) as the value for a single key.
ConfigMaps can also be generated from literal key-value pairs. To generate a ConfigMap from a literal key-value pair, add an entry to the
literals
list in configMapGenerator. Here is an example of generating a ConfigMap with a data item from a key-value pair:
cat
- name: example-configmap-2
literals:
- FOO=Bar
EOF
The generated ConfigMap can be checked by the following command:
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
name
:
example-configmap-2-g2hdhfc6tk
To use a generated ConfigMap in a Deployment, reference it by the name of the configMapGenerator. Kustomize will automatically replace this name with the generated name.
This is an example deployment that uses a generated ConfigMap:
cat &lt;&lt;EOF &gt;application.properties
FOO=Bar
EOF
cat &lt;&lt;EOF &gt;deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
my-app
labels
:
app
:
my-app
spec
:
selector
:
:
app
:
my-app
template
:
metadata
:
labels
:
app
:
my-app
spec
:
:
-
name
:
app
image
:
my-app
:
-
name
:
config
mountPath
:
/config
volumes
:
-
name
:
config
configMap
:
name
:
example-configmap-1
EOF
cat &lt;&lt;EOF &gt;./kustomization.yaml
resources
:
- deployment.yaml
:
-
name
:
files
:
EOF
Generate the ConfigMap and Deployment:
The generated Deployment will refer to the generated ConfigMap by name:
:
v1
data
:
:
|
FOO=Bar
kind
:
ConfigMap
metadata
:
name
:
example-configmap-1-g4hk9g2ff8
---
:
apps/v1
kind
:
metadata
:
labels
:
app
:
my-app
name
:
my-app
spec
:
selector
:
:
app
:
my-app
template
:
metadata
:
labels
:
app
:
my-app
spec
:
:
-
image
:
my-app
name
:
app
:
-
mountPath
:
/config
name
:
config
volumes
:
-
configMap
:
name
:
name
:
config
You can generate Secrets from files or literal key-value pairs. To generate a Secret from a file, add an entry to the
files
list in
. Here is an example of generating a Secret with a data item from a file:
# Create a password.txt file
cat
&lt;&lt;EOF &gt;./password.txt
username=admin
password=secret
EOF
cat
secretGenerator:
- name: example-secret-1
files:
- password.txt
EOF
The generated Secret is as follows:
:
v1
data
:
password.txt
:
dXNlcm5hbWU9YWRtaW4KcGFzc3dvcmQ9c2VjcmV0Cg==
kind
:
Secret
metadata
:
name
:
example-secret-1-t2kt65hgtb
type
:
Opaque
To generate a Secret from a literal key-value pair, add an entry to
literals
list in
. Here is an example of generating a Secret with a data item from a key-value pair:
cat
- name: example-secret-2
literals:
- username=admin
- password=secret
EOF
:
v1
data
:
password
:
c2VjcmV0
username
:
YWRtaW4=
kind
:
Secret
metadata
:
name
:
example-secret-2-t52t6g96d8
type
:
Opaque
Like ConfigMaps, generated Secrets can be used in Deployments by referring to the name of the secret

Generator:
cat
EOF
cat
&lt;&lt;EOF &gt;deployment.yaml
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
selector:
matchLabels:
template:
metadata:
labels:
spec:
- name: app
image: my-app
volumeMounts:
- name: password
mountPath: /secrets
volumes:
secret:
secretName: example-secret-1
EOF
cat
files:
EOF
generatorOptions
The generated ConfigMaps and Secrets have a content hash suffix appended. This ensures that a new ConfigMap or Secret is generated when the contents are changed. To disable the behavior of appending a suffix, one can use
. Besides that, it is also possible to specify cross-cutting options for generated ConfigMaps and Secrets.
cat
- name: example-configmap-3
literals:
- FOO=Bar
generatorOptions:
disableNameSuffixHash: true
labels:
type: generated
note: generated
EOF
Run
to view the generated ConfigMap:
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
:
note
:
generated
labels
:
type
:
generated
name
:
example-configmap-3
Setting cross-cutting fields
It is quite common to set cross-cutting fields for all Kubernetes resources in a project.
Some use cases for setting cross-cutting fields:
setting the same namespace for all Resources
adding the same name prefix or suffix
adding the same set of labels
adding the same set of annotations
# Create a deployment.yaml
cat
&lt;&lt;EOF &gt;./deployment.yaml
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
selector:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
- name: nginx
EOF
cat
namespace: my-namespace
namePrefix: dev-
nameSuffix: "-001"
commonLabels:
app: bingo
commonAnnotations:
oncallPager: 800-555-1212
EOF
Run
to view those fields are all set in the Deployment Resource:
:
apps/v1
kind
:
metadata
:
:
oncallPager
:
800-555-1212
labels
:
app
:
bingo
name
:
dev-nginx-deployment-001
namespace
:
spec
:
selector
:
:
app
:
bingo
template
:
metadata
:
:
:
labels
:
app
:
bingo
spec
:
:
-
image
:
nginx
name
:
nginx
Composing and Customizing Resources
It is common to compose a set of Resources in a project and manage them inside
the same file or directory.
Kustomize offers composing Resources from different files and applying patches or other customization to them.
Composing
Kustomize supports composition of different resources. The
resources
field, in the
file, defines the list of resources to include in a configuration. Set the path to a resource's configuration file in the
resources
list.
Here is an example of an NGINX application comprised of a Deployment and a Service:
# Create a deployment.yaml file
cat
&lt;&lt;EOF &gt; deployment.yaml
metadata:
name: my-nginx
spec:
selector:
replicas: 2
template:
metadata:
labels:
spec:
- name: my-nginx
ports:
EOF
# Create a service.yaml file
cat
&lt;&lt;EOF &gt; service.yaml
kind: Service
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
# Create a kustomization.yaml composing them
cat
- service.yaml
EOF
The Resources from
contain both the Deployment and the Service objects.
Customizing
Patches can be used to 

apply different customizations to Resources. Kustomize supports different patching
mechanisms through
patchesStrategicMerge
and
patchesJson6902
.
is a list of file paths. Each file should be resolved to a
. The names inside the patches must match Resource names that are already loaded. Small patches that do one thing are recommended. For example, create one patch for increasing the deployment replica number and another patch for setting the memory limit.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
# Create a patch increase_replicas.yaml
cat
&lt;&lt;EOF &gt; increase_replicas.yaml
metadata:
spec:
EOF
# Create another patch set_memory.yaml
cat
&lt;&lt;EOF &gt; set_memory.yaml
metadata:
spec:
template:
spec:
limits:
EOF
cat
patchesStrategicMerge:
- increase_replicas.yaml
- set_memory.yaml
EOF
Run
to view the Deployment:
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
3
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
nginx
name
:
my-nginx
ports
:
-
:
80
resources
:
limits
:
memory
:
512Mi
Not all Resources or fields support strategic merge patches. To support modifying arbitrary fields in arbitrary Resources,
Kustomize offers applying
JSON patch
through
.
To find the correct Resource for a Json patch, the group, version, kind and name of that Resource need to be
specified in
. For example, increasing the replica number of a Deployment object can also be done
through
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
# Create a json patch
cat
&lt;&lt;EOF &gt; patch.yaml
- op: replace
path: /spec/replicas
value: 3
EOF
# Create a kustomization.yaml
cat
patchesJson6902:
- target:
version: v1
path: patch.yaml
EOF
Run
to see the
replicas
field is updated:
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
3
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
nginx
name
:
my-nginx
ports
:
-
:
80
In addition to patches, Kustomize also offers customizing container images or injecting field values from other objects into containers
without creating patches. For example, you can change the image used inside containers by specifying the new image in
images
field in
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
images:
newName: my.image.registry/nginx
newTag: 1.4.0
EOF
Run
to see that the image being used is updated:
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
2
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx

spec
:
:
-
image
:
my.image.registry/nginx:1.4.0
name
:
my-nginx
ports
:
-
:
80
Sometimes, the application running in a Pod may need to use configuration values from other objects. For example,
a Pod from a Deployment object need to read the corresponding Service name from Env or as a command argument.
Since the Service name may change as
namePrefix
or
nameSuffix
is added in the
file. It is
not recommended to hard code the Service name in the command argument. For this usage, Kustomize can inject the Service name into containers through
vars
.
# Create a deployment.yaml file (quoting the here doc delimiter)
cat
&lt;&lt;'EOF' &gt; deployment.yaml
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
command: ["start", "--host", "$(MY_SERVICE_NAME)"]
EOF
cat
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
cat
vars:
- name: MY_SERVICE_NAME
objref:
EOF
Run
to see that the Service name injected into containers is
dev-my-nginx-001
:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
2
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
command
:
- start
- --host
- dev-my-nginx-001
image
:
nginx
name
:
my-nginx
Bases and Overlays
Kustomize has the concepts of
bases
and
overlays
. A
base
is a directory with a
, which contains a
set of resources and associated customization. A base could be either a local directory or a directory from a remote repo,
as long as a
is present inside. An
overlay
that refers to other
kustomization directories as its
bases
. A
base
has no knowledge of an overlay and can be used in multiple overlays.
An overlay may have multiple bases and it composes all resources
from bases and may also have customization on top of them.
Here is an example of a base:
# Create a directory to hold the base
mkdir base
# Create a base/deployment.yaml
cat
&lt;&lt;EOF &gt; base/deployment.yaml
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
EOF
# Create a base/service.yaml file
cat
&lt;&lt;EOF &gt; base/service.yaml
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
# Create a base/kustomization.yaml
cat
&lt;&lt;EOF &gt; base/kustomization.yaml
EOF
This base can be used in multiple overlays. You can add different
or other cross-cutting fields
in different overlays. Here are two overlays using the same base.
mkdir dev
cat
&lt;&lt;EOF &gt; dev/kustomization.yaml
bases:
- ../base
EOF
mkdir prod
cat
&lt;&lt;EOF &gt; prod/kustomization.yaml
bases:
- ../base
namePrefix: prod-
EOF
How to apply/view/delete objects using Kustomize
Use
or
-k
in
kubectl
commands to recognize Resources managed by
.
Note that
-k
should point to a kustomization directory, such as
kubectl apply -k &lt;kustomization directory&gt;/
Given the following
,
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
- name: my

-nginx
ports:
EOF
cat
app: my-nginx
EOF
Run the following command to apply the Deployment object
dev-my-nginx
:
&gt; kubectl apply -k ./
deployment.apps/dev-my-nginx created
Run one of the following commands to view the Deployment object
:
kubectl get -k ./
kubectl describe -k ./
Run the following command to compare the Deployment object
against the state that the cluster would be in if the manifest was applied:
kubectl diff -k ./
Run the following command to delete the Deployment object
:
&gt; kubectl delete -k ./
"dev-my-nginx"
deleted
Kustomize Feature List
Field
Type
Explanation
namespace
string
add namespace to all resources
string
value of this field is prepended to the names of all resources
string
value of this field is appended to the names of all resources
commonLabels
labels to add to all resources and selectors
commonAnnotations
annotations to add to all resources
resources
[]string
each entry in this list must resolve to an existing resource configuration file
[]
ConfigMapArgs
Each entry in this list generates a ConfigMap
[]
SecretArgs
Each entry in this list generates a Secret
GeneratorOptions
Modify behaviors of all ConfigMap and Secret generator
bases
[]string
Each entry in this list should resolve to a directory containing a kustomization.yaml file
[]string
Each entry in this list should resolve a strategic merge patch of a Kubernetes object
[]
Patch
Each entry in this list should resolve to a Kubernetes object and a Json Patch
vars
[]
Var
Each entry is to capture text from one resource's field
images
[]
Image
Each entry is to modify the name, tags and/or digest for one image without creating patches
configurations
[]string
Each entry in this list should resolve to a file containing
Kustomize transformer configurations
crds
[]string
Each entry in this list should resolve to an OpenAPI definition file for Kubernetes types
Kustomize
Feedback
Yes
No
.
or
.
Last modified July 28, 2022 at 5:49 PM PST:
Revert "Document the environment variable substitution feature of configMapGenerator" (39fb094c52)
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm token | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 (v1)
Reference
Kubeadm
.
Synopsis
Options
-h, --help
--dry-run
Synopsis
Token ID of

 the form "[a-z0-9]{6}" to delete.
Options
-h, --help
--dry-run
Synopsis
Options
-h, --help
--dry-run
Synopsis
Options
-h, --help
--dry-run
Feedback
Yes
No
.
or
.
Last modified August 09, 2020 at 3:41 PM PST:
add content_type param, kubeadm pages (1e0c500571)
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm init | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Kubeadm
Synopsis
  /local                       Generate the static Pod manifest file for a local, si

ngle-node local etcd instance
Options
--dry-run
-h, --help
The token to use for 

establishing bidirectional trust between nodes and control-plane nodes. The format is [a-z0-9]{6}.[a-z0-9]{16} - e.g. abcdef.0123456789abcdef
.
(
.
run there.
token via
--token
docs.
and
See
Warning:
command.
. The list
--help
--help
=
local
--config
=
# you can now modify the contro

l plane and etcd manifest files
=
=
.
Caution:
--config
contain a
---\n
Mixing
--config
command.
the
command.
.
flag for
a
using
--config
.
Passing
.
Feature
Default
Alpha
Beta
GA
false
1.19
-
-
false
1.22
-
-
true
1.22
1.23
1.25
Note:
true
for
,
,
and
etcd
ConfigMap
true
.
false
or
(during
,
,
kube-

proxy reference
IPVS
--config
and
fields.
. If the
ci/latest
)
is used.
.
To provide
.
and
imageTag
to
using
.
,
where
, and/or
imageTag
to
.
topic.
to
and
to
.
=
Note:
with
--config
.
and
.
The
kubeadm
kubelet
by
systemd
.
.
By default kubeadm attempts to detect your container

 runtime. For more details on this detection,
see the
.
kubeadm
flag.
.
in the
,
.
--token
and
.
at
.
phases
or
Feedback
Yes
No
.
or
.
Last modified December 06, 2022 at 8:32 AM PST:
updated kubeadm certificate page misleading example (#38258) (4a8cf5c2e1)
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Certificate Rotation for the Kubelet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
TLS
This page shows how to enable and configure certificate rotation for the kubelet.
Kubernetes version 1.8.0 or later is required
Overview
The kubelet uses certificates for authenticating to the Kubernetes API. By
default, these certificates are issued with one year expiration so that they do
not need to be renewed too frequently.
Kubernetes contains
kubelet certificate
rotation
,
that will automatically generate a new key and request a new certificate from
the Kubernetes API as the current certificate approaches expiration. Once the
new certificate is available, it will be used for authenticating connections to
Enabling client certificate rotation
The
kubelet
that controls
if the kubelet will automatically request a new certificate as the expiration of
the certificate currently in use approaches.
The
(
--experimental-cluster-signing-duration
prior to 1.19)
that controls how long certificates will be issued for.
Understanding the certificate rotation configuration
When a kubelet starts up, if it is configured to bootstrap (using the
flag), it will use its initial certificate to connect
to the Kubernetes API and issue a certificate signing request. You can view the
status of certificate signing requests using:
Initially a certificate signing request from the kubelet on a node will have a
status of
Pending
. If the certificate signing requests meets specific
criteria, it will be auto approved by the controller manager, then it will have
a status of
Approved
. Next, the controller manager will sign a certificate,
issued for the duration specified by the
parameter, and the signed certificate
will be attached to the certificate signing request.
The kubelet will retrieve the signed certificate from the Kubernetes API and
write that to disk, in the location specified by
. Then the kubelet
will use the new certificate to connect to the Kubernetes API.
As the expiration of the signed certificate approaches, the kubelet will
automatically issue a new certificate signing request, using the Kubernetes API.
This can happen at any point between 30% and 10% of the time remaining on the
certificate. Again, the controller manager wi

ll automatically approve the certificate
request and attach a signed certificate to the certificate signing request. The
kubelet will retrieve the new signed certificate from the Kubernetes API and
write that to disk. Then it will update the connections it has to the
Kubernetes API to reconnect using the new certificate.
Feedback
Yes
No
.
or
.
Overview
Home
Blog
Training
Partners
Community
CC BY 4.0

Opening a pull request | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

 and Set Up kubectl on macOS
Debug Pods
Get a Shell to a

 Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a C

ontainer's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Confi

guration (v1)
Note:
Code developers
: If you are documenting a new feature for an
upcoming Kubernetes release, see
Document a new feature
.
To contribute new content pages or improve existing content pages, open a pull request (PR).
Make sure you follow all the requirements in the
section.
If your change is small, or you're unfamiliar with git, read
Changes using GitHub
to learn how to edit a page.
If your changes are large, read
Work from a local fork
to learn how to make
changes locally on your computer.
If you're less experienced with git workflows, here's an easier method of
opening a pull request. Figure 1 outlines the steps and the details follow.
A([fa:fa-user New
Contributor]) --- id1[(K8s/Website
GitHub)]
subgraph tasks[Changes using GitHub]
0[ ] -.-
1[1. Edit this page] --&gt; 2[2. Use GitHub markdown
editor to make changes]
2 --&gt; 3[3. fill in Propose file change]
end
subgraph tasks2[ ]
4[4. select Propose file change] --&gt; 5[5. select Create pull request] --&gt; 6[6. fill in Open a pull request]
6 --&gt; 7[7. select Create pull request]
end
id1 --&gt; tasks --&gt; tasks2
class A,1,2,3,4,5,6,7 grey
class 0 spacewhite
class tasks,tasks2 white
class id1 k8s
enabled
Figure 1. Steps for opening a PR using GitHub.
On the page where you see the issue, select the
option in the right-hand side navigation panel.
Make your changes in the GitHub markdown editor.
Below the editor, fill in the
Propose file change
form.
In the first field, give your commit message a title.
In the second field, provide a description.
Note:
Do not use any
GitHub Keywords
in your commit message. You can add those to the pull request description later.
Select
.
Select
Create pull request
.
The
Open a pull request
screen appears. Fill in the form:
The
Subject
field of the pull request defaults to the commit summary.
You can change it if needed.
The
Body
contains your extended commit message, if you have one,
and some template text. Add the
details the template text asks for, then delete the extra template text.
Leave the
Allow edits from maintainers
chec

kbox selected.
Note:
PR descriptions are a great way to help reviewers understand your change.
Opening a PR
.
Select
.
Addressing feedback in GitHub
Before merging a pull request, Kubernetes community members review and
approve it. The
k8s-ci-robot
suggests reviewers based on the nearest
owner mentioned in the pages. If you have someone specific in mind,
leave a comment with their GitHub username in it.
If a reviewer asks you to make changes:
Go to the
tab.
Select the pencil (edit) icon on any files changed by the pull request.
Make the changes requested.
Commit the changes.
If you are waiting on a reviewer, reach out once every 7 days. You can also post a message in the
#sig-docs
Slack channel.
When your review is complete, a reviewer merges your PR and your changes go live a few minutes later.
If you're more experienced with git, or if your changes are larger than a few lines,
work from a local fork.
Make sure you have
git
installed
on your computer. You can also use a git UI application.
Figure 2 shows the steps to follow when you work from a local fork. The details for each step follow.
1[Fork the K8s/website
repository] --&gt; 2[Create local clone
and set upstream]
subgraph changes[Your changes]
S[ ] -.-
3[Create a branch
example: my_new_branch] --&gt; 3a[Make changes using
text editor] --&gt; 4["Preview your changes
locally using Hugo
(localhost:1313)
or build container image"]
end
subgraph changes2[Commit / Push]
T[ ] -.-
5[Commit your changes] --&gt; 6[Push commit to
origin/my_new_branch]
end
2 --&gt; changes --&gt; changes2
class 1,2,3,3a,4,5,6 grey
class changes,changes2 white
enabled
Figure 2. Working from a local fork to make your changes.
Fork the kubernetes/website repository
Navigate to the
Select
Fork
.
Create a local clone and set the upstream
In a terminal window, clone your fork and update the
Docsy Hugo theme
:
git clone git@github.com/&lt;github_username&gt;/website
cd
website
1
Navigate to the new
website
directory. Set the
repository as the
upstream
remote:
cd
website
git remote add upstream https://github.com/kubernetes/website.git
Confirm your
origin
and
upstream
repositories:
git remote -v
Output is similar to:
origin	git@github.com:&lt;github_username&gt;/website.git (fetch)
origin	git@github.com:&lt;github_username&gt;/website.git (push)
upstream	https://github.com/kubernetes/website.git (fetch)
upstream	https://github.com/kubernetes/website.git (push)
Fetch commits from your fork's
origin/main
and
's
upstream/main
:
git fetch origin
git fetch upstream
This makes sure your local repository is up to date before you start making changes.
Note:
This workflow is different than the
Kubernetes Community GitHub Workflow
.
You do not need to merge your local copy of
main
with
before pushing updates
to your fork.
Create a branch
Decide which branch base to your work on:
For improvements to existing content, use
.
For new content about existing features, use
.
For localized content, use the localization's conventions. For more information, see
localizing Kubernetes documentation
.
For new features in an upcoming Kubernetes release, use the feature branch. For more
documenting for a release
.
For long-running efforts that multiple SIG Docs contributors collaborate on,
like content reorganization, use a specific feature branch created for that effort.

If you need help choosing a branch, ask in the
#sig-docs
Create a new branch based on the branch identified in step 1. This example assumes the base
branch is
:
git checkout -b &lt;my_new_branch&gt; upstream/main
Make your changes using a text editor.
At any time, use the
git status
command to see what files you've changed.
Commit your changes
When you are ready to submit a pull request, commit your changes.
In your local repository, check which files you need to commit:
git status
On branch &lt;my_new_branch&gt;
Your branch is up to date with 'origin/&lt;my_new_branch&gt;'.

Changes not staged for commit:
(use "git add &lt;file&gt;..." to update what will be committed)
(use "git checkout -- &lt;file&gt;..." to discard changes in working directory)

modified:   content/en/docs/contribute/new-content/contributing-content.md

no changes added to commit (use "git add" and/or "git commit -a")
Add the files listed under
Changes not staged for commit
to the commit:
git add &lt;your_file_name&gt;
Repeat this for each file.
After adding all the files, create a commit:
git commit -m
"Your commit message"
Note:
in your commit message. You can add those to the pull request
description later.
Push your local branch and its new commit to your remote fork:
git push origin &lt;my_new_branch&gt;
Preview your changes locally
It's a good idea to preview your changes locally before pushing them or opening a pull request.
A preview lets you catch build errors or markdown formatting problems.
You can either build the website's container image or run Hugo locally. Building the container
image is slower but displays
Hugo shortcodes
, which can
be useful for debugging.
Hugo in a container
Hugo on the command line
Note:
The commands below use Docker as default container engine. Set the
CONTAINER_ENGINE
variable to override this behaviour.
Build the container image locally
You only need this step if you are testing a change to the Hugo tool itself
# Run this in a terminal (if required)
make container-image
Start Hugo in a container:
# Run this in a terminal
In a web browser, navigate to
https://localhost:1313
. Hugo watches the
changes and rebuilds the site as needed.
To stop the local Hugo instance, go back to the terminal and type
Ctrl+C
,
or close the terminal window.
Alternately, install and use the
hugo
command on your computer:
Hugo
version specified in
website/netlify.toml
.
If you have not updated your website repository, the
website/themes/docsy
directory is empty.
The site cannot build without a local copy of the theme. To update the website theme, run:
1
In a terminal, go to your Kubernetes website repository and start the Hugo server:
cd
&lt;path_to_your_repo&gt;/website
hugo server --buildFuture
Ctrl+C
,
Open a pull request from your fork to kubernetes/website
Figure 3 shows the steps to open a PR from your fork to the K8s/website. The details follow.
subgraph first[ ]
1[1. Go to K8s/website repository] --&gt; 2[2. Select New Pull Request]
2 --&gt; 3[3. Select compare across forks]
3 --&gt; 4[4. Select your fork from
head repository drop-down menu]
end
subgraph second [ ]
5[5. Select your branch from
the compare drop-down menu] --&gt; 6[6. Select Create Pull Request]
6 --&gt; 7[7. Add a description
to your PR]
7 --&gt; 8[8. Select Create pull request]
end
first --&gt; second
class 1,2,3,4,5,6,7,8 grey
class first,second white
enabled
to

 view this content
Figure 3. Steps to open a PR from your fork to the K8s/website.
In a web browser, go to the
Select
New Pull Request
.
Select
compare across forks
.
From the
head repository
drop-down menu, select your fork.
From the
compare
drop-down menu, select your branch.
Select
Create Pull Request
.
Add a description for your pull request:
Title
(50 characters or less): Summarize the intent of the change.
: Describe the change in more detail.
If there is a related GitHub issue, include
Fixes #12345
or
Closes #12345
in the
description. GitHub's automation closes the mentioned issue after merging the PR if used.
If there are other related PRs, link those as well.
If you want advice on something specific, include any questions you'd like reviewers to
think about in your description.
Select the
button.
Congratulations! Your pull request is available in
Pull requests
.
After opening a PR, GitHub runs automated tests and tries to deploy a preview using
Netlify
.
If the Netlify build fails, select
Details
If the Netlify build succeeds, select
Details
opens a staged version of the Kubernetes
website with your changes applied. This is how reviewers check your changes.
GitHub also automatically assigns labels to a PR, to help reviewers. You can add them too, if
needed. For more information, see
Adding and removing issue labels
.
Addressing feedback locally
After making your changes, amend your previous commit:
git commit -a --amend
-a
: commits all changes
--amend
: amends the previous commit, rather than creating a new one
Update your commit message if needed.
Use
to push your changes and re-run the Netlify tests.
Note:
If you use
instead of amending, you must
squash your commits
before merging.
Changes from reviewers
Sometimes reviewers commit to your pull request. Before making any other changes, fetch those commits.
Fetch commits from your remote fork and rebase your working branch:
git rebase origin/&lt;your-branch-name&gt;
After rebasing, force-push new changes to your fork:
git push --force-with-lease origin &lt;your-branch-name&gt;
Merge conflicts and rebasing
Note:
Git Branching - Basic Branching and Merging
,
Advanced Merging
, or ask in the
#sig-docs
Slack channel for help.
If another contributor commits changes to the same file in another PR, it can create a merge
conflict. You must resolve all merge conflicts in your PR.
Update your fork and rebase your local branch:
Then force-push the changes to your fork:
Fetch changes from
's
and rebase your branch:
git rebase upstream/main
Inspect the results of the rebase:
git status
This results in a number of files marked as conflicted.
Open each conflicted file and look for the conflict markers:
&gt;&gt;&gt;
,
&lt;&lt;&lt;
, and
===
.
Resolve the conflict and delete the conflict marker.
Note:
How conflicts are presented
.
Add the files to the changeset:
git add &lt;filename&gt;
Continue the rebase:
git rebase --continue
Repeat steps 2 to 5 as needed.
After applying all commits, the
git status
command shows that the rebase is complete.
Force-push the branch to your fork:
The pull request no longer shows any conflicts.
Squashing commits
Note:
Git Tools - Rewriting History
,
or ask in the
#sig-docs
If your PR has multiple commits, you must squash them into a single commit before merging your PR.
You can check the number of commits on your PR's
Commits
tab or by running the
git log
command locally.
Note:
This topic assumes
vim
as the command line text editor.
Start an interactive rebase:
git rebase -

i HEAD~&lt;number_of_commits_in_branch&gt;
Squashing commits is a form of rebasing. The
-i
switch tells git you want to rebase interactively.
HEAD~&lt;number_of_commits_in_branch
indicates how many commits to look at for the rebase.
pick d875112ca Original commit
pick 4fa167b80 Address feedback 1
pick 7d54e15ee Address feedback 2

# Rebase 3d18sf680..7d54e15ee onto 3d183f680 (3 commands)

...

# These lines can be re-ordered; they are executed from top to bottom.
The first section of the output lists the commits in the rebase. The second section lists the
options for each commit. Changing the word
pick
changes the status of the commit once the rebase
is complete.
For the purposes of rebasing, focus on
squash
and
pick
.
Note:
Interactive Mode
.
Start editing the file.
Change the original text:
To:
squash 4fa167b80 Address feedback 1
squash 7d54e15ee Address feedback 2
This squashes commits
4fa167b80 Address feedback 1
and
7d54e15ee Address feedback 2
into
d875112ca Original commit
, leaving only
as a part of the timeline.
Save and exit your file.
Push your squashed commit:
git push --force-with-lease origin &lt;branch_name&gt;
Contribute to other repos
The
contains 50+ repositories. Many of these
repositories contain documentation: user-facing help text, error messages, API references or code
comments.
If you see text you'd like to improve, use GitHub to search all repositories in the Kubernetes
organization. This can help you figure out where to submit your issue or PR.
Each repository has its own processes and procedures. Before you file an issue or submit a PR,
read that repository's
README.md
,
CONTRIBUTING.md
, and
code-of-conduct.md
, if they exist.
Most repositories use issue and PR templates. Have a look through some open issues and PRs to get
a feel for that team's processes. Make sure to fill out the templates with as much detail as
possible when you file issues or PRs.
Read
Reviewing
to learn more about the review process.
Feedback
Yes
No
.
or
.
Last modified December 26, 2022 at 11:48 PM PST:
Improved the steps for opening a PR using GitHub (b6499ef3f7)
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes Metrics Reference | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Migrate from PodSecurityPolicy to the Built-In Po

dSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfile, Co

nfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate

 Limit Configuration (v1alpha1)
Reference
Name
Type
Help
Labels
STABLE
Histogram
name
operation
rejected
type
None
STABLE
Histogram
operation
rejected
type
None
STABLE
Histogram
name
operation
rejected
type
None
STABLE
Gauge
None
STABLE
Gauge
component
group
resource
scope
verb
version
None
STABLE
Histogram
component
dry_run
group
resource
scope
verb
version
None
STABLE
Counter
code
component
dry_run
group
resource
scope
verb
version
None
STABLE
Gauge
group
resou

rce
version
None
STABLE
Histogram
component
group
resource
scope
verb
version
None
STABLE
Gauge
resource
None
STABLE
Counter
zone
None
STABLE
Histogram
profile
status
None
STABLE
Gauge
queue
None
STABLE
Histogram
None
None
STABLE
Histogram
attempts
None
STABLE
Counter
None
None
STABLE
Histogram
None
None
STABLE
Counter
event
queue
None
STABLE
Counter
profile
result
None
STABLE
Histogram
profile
result
None
Name
Type
Help
Labels
ALPHA
Counter
Counter of OpenAPI v2 spec regeneration count broken down by causing APIService name and reason.
reason
None
ALPHA
Gauge
reason
None
ALPHA
Custom
name
None
ALPHA
Counter
name
reason
None
ALPHA
Counter
crd
reason
None
ALPHA
Counter
crd
group
reason
version
None
ALPHA
Summary
operation
rejected
type
None
ALPHA
Counter
name
type
None
ALPHA
Counter
name
operation
type
Non

e
ALPHA
Counter
code
name
operation
rejected
type
None
ALPHA
Counter
plugin
None
ALPHA
Counter
None
None
ALPHA
Counter
level
None
ALPHA
Counter
None
None
ALPHA
Counter
index
None
ALPHA
Counter
None
ALPHA
Counter
index
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Histogram
Distribution of the remaining lifetime on the certificate used to authenticate a request.
None
None
ALPHA
Histogram
crd_name
succeeded
None
ALPHA
Gauge
None
ALPHA
Histogram
code
None
ALPHA
Counter
code
None
ALPHA
Histogram
code
None
ALPHA
Counter
code
None
ALPHA
Histogram
protocol
transport
None
ALPHA
Counter
protocol
stage
transport
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
ALPHA
Gauge
None
ALPHA
Gauge
Number of requests cur

rently pending in queues of the API Priority and Fairness subsystem
None
ALPHA
Gauge
None
ALPHA
Gauge
None
ALPHA
Counter
None
ALPHA
Counter
success
None
ALPHA
Gauge
None
ALPHA
Gauge
bound
None
ALPHA
Gauge
bound
None
ALPHA
phase
None
ALPHA
ALPHA
phase
None
ALPHA
Counter
reason
None
ALPHA
Gauge
Concurrency (number of seats) occupied by the currently executing (initial stage for a WATCH, any stage otherwise) requests in the API Priority and Fairness subsystem
None
ALPHA
Gauge
None
ALPHA
Counter
None
ALPHA
Histogram
type
None
ALPHA
Histogram
None
ALPHA
Histogram
execute
None
ALPHA
Histogram
None
ALPHA
Histogram
None
ALPHA
Counter
resource
None
ALPHA
Counter
None
None
apiserver_kube_aggregator_x50

9_missing_san_total
ALPHA
Counter
None
None
ALPHA
Counter
group
resource
scope
verb
version
None
ALPHA
Histogram
resource
verb
None
ALPHA
Histogram
filter
None
ALPHA
Counter
source
status
None
ALPHA
Histogram
component
group
resource
scope
verb
version
None
ALPHA
Counter
code
component
group
resource
scope
verb
version
None
ALPHA
Histogram
code_path
None
ALPHA
Counter
resource
verb
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
None
ALPHA
Gauge
Total size of the storage database file physically allocated in bytes.
endpoint
None
ALPHA
Counter
None
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Histogram
None
ALPHA
Counter
status
None
ALPHA
Counter
resource
None
ALPHA
Counter
None
None
ALPHA
Histogram
params
policy
state
None
ALPHA
Counter
params
policy
p

olicy_binding
state
None
ALPHA
Counter
state
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Histogram
group
kind
version
None
ALPHA
Counter
group
kind
version
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Custom
state
None
ALPHA
Counter
username
None
ALPHA
Counter
result
None
ALPHA
Histogram
result
None
ALPHA
Gauge
status
None
ALPHA
Counter
status
None
ALPHA
Histogram
status
None
ALPHA
Counter
status
None
ALPHA
Histogram
request
None
ALPHA
Counter
request
None
ALPHA
Counter
None
ALPHA
Histogram
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Histogram
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Histogram
region
request
version
zone
None
ALPHA
Counter
region
request
version
zone
None
ALPHA
Histogram
request
None
ALPHA
Counter
request
None
ALPHA
Histogram
operation
None
ALPHA
Counter
operation
None
ALPHA
Custom

hostname
version
build
None
ALPHA
Custom
container
pod
namespace
None
ALPHA
Custom
container
pod
namespace
None
ALPHA
Custom
container
pod
namespace
None
ALPHA
Histogram
None
None
ALPHA
Histogram
migrated
None
ALPHA
Counter
operation
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
topology
None
ALPHA
Gauge
None
None
ALPHA
Counter
result
None
ALPHA
Histogram
None
None
ALPHA
Counter
operation
None
endpoint_slice_mirroring_controller_desired_endpoint_slices
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Gauge
resource
None
ALPHA
Histogram
None
None
ALPHA
Histogram
operation
type
None
ALPHA
Gauge
None
ALPHA
Histogram
enabled
None
ALPHA
Counter
None
None
ALPHA
Counter
Counter of total Token() requests to

 the alternate token source
None
None
ALPHA
Counter
None
None
ALPHA
Counter
reason
result
None
ALPHA
Counter
result
None
ALPHA
Histogram
action
result
None
ALPHA
Counter
action
result
None
ALPHA
Counter
action
None
ALPHA
Counter
event
None
ALPHA
Gauge
cidr
None
ALPHA
Counter
cidr
scope
None
ALPHA
Counter
cidr
scope
None
ALPHA
Gauge
cidr
None
ALPHA
Counter
None
None
ALPHA
Counter
Total number of requests for pods/logs sliced by usage type: enforce_tls, skip_tls_allowed, skip_tls_denied
usage
None
ALPHA
Custom
namespace
pod
node
scheduler
priority
resource
unit
None
ALPHA
Custom
namespace
pod
node
scheduler
priority
resource
unit
None
ALPHA
Counter
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
ALPHA
Custom
uid
namespace
pod
container
None
ALPHA
Histogram
None
None
ALPHA
Counter
The

 number of cpu core allocations which required pinning failed.
None
None
ALPHA
Counter
None
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
method
path
None
ALPHA
Histogram
method
path
None
ALPHA
Counter
method
path
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
None
ALPHA
Gauge
Current number of ephemeral containers in pods managed by this kubelet. Ephemeral containers will be ignored if disabled by the EphemeralContainers feature gate, and this number will be 0.
None
None
ALPHA
Gauge
node
None
ALPHA
Counter
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Histogram
None
None
ALPHA
Histogram
Duration in seconds to sync a pod status update. Measures time from detection of a change to pod status until the API is successfully updated for that pod, even if multiple inteve

ning changes to pod status occur.
None
None
ALPHA
Histogram
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Gauge
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
None
ALPHA
Counter
code
None
ALPHA
Counter
None
ALPHA
Counter
code
None
ALPHA
Counter
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Histogram
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Counter
None
None
kubeproxy_sync_proxy_rules_ipta

bles_restore_failures_total
ALPHA
Counter
None
None
ALPHA
Gauge
table
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
ALPHA
Gauge
None
None
ALPHA
Counter
None
None
ALPHA
Gauge
compiler
major
minor
platform
None
ALPHA
Gauge
name
stage
None
ALPHA
Gauge
name
type
None
ALPHA
Counter
name
status
type
None
ALPHA
Gauge
name
None
ALPHA
Histogram
operation
None
ALPHA
Counter
Number of Node evictions that happened since current instance of NodeController started, This metric is replaced by node_collector_evictions_total.
zone
None
ALPHA
Gauge
zone
None
ALPHA
Gauge
zone
None
ALPHA
Gauge
zone
None
ALPHA
Custom
None
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Gauge
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Gauge
None
ALPHA
Custom
None
None
ALPHA
Gauge
feature
None
ALPHA
Custom
state
None
ALPHA
Custom
pod
namespace
None
ALPHA

Custom
pod
namespace
None
ALPHA
Counter
fatal
resource
None
ALPHA
Counter
decision
mode
resource
None
ALPHA
Counter
resource
None
ALPHA
Histogram
container
namespace
pod
None
ALPHA
Counter
container
namespace
pod
pod_uid
result
None
ALPHA
Custom
None
ALPHA
Custom
namespace
None
ALPHA
Custom
None
ALPHA
Custom
None
ALPHA
Custom
namespace
None
ALPHA
Histogram
None
None
ALPHA
Counter
Number of calls to an exec plugin, partitioned by the type of event encountered (no_error, plugin_execution_error, plugin_not_found_error, client_internal_error) and an optional exit code. The exit code will be set to 0 if and only if the plugin call was successful.
code
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
host
verb
None
ALPHA
Histogram
host
verb
None
ALPHA
Histogram
host
verb
None
ALPHA
Counter
code
host
method
None
ALPHA
Histogram
host
verb
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Histogram
code
None
r

oot_ca_cert_publisher_sync_total
ALPHA
Counter
code
None
ALPHA
Gauge
manager
name
None
ALPHA
Histogram
profile
result
None
ALPHA
Gauge
operation
None
ALPHA
Histogram
result
None
ALPHA
Histogram
plugin
status
None
ALPHA
Gauge
type
None
ALPHA
Gauge
work
None
ALPHA
Histogram
None
None
ALPHA
Gauge
plugin
profile
None
ALPHA
Counter
operation
None
ALPHA
Counter
operation
None
ALPHA
Custom
None
None
ALPHA
Histogram
None
None
service_controller_update_loadbalancer_host_latency_seconds
ALPHA
Histogram
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Custom
node
None
ALPHA
Histogram
migrated
status
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
Number of errors when a Pod defines different SELinux contexts for its containers that use the same volume. They are not errors yet, but they will become real

 errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Custom
state
None
ALPHA
Counter
None
ALPHA
Histogram
None
ALPHA
Gauge
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
name
None
ALPHA
Gauge
name
None
ALPHA
Gauge
name
None
ALPHA
Histogram
How long in seconds an item stays in workqueue before being requested.
name
None
ALPHA
Counter
name
None
ALPHA
Gauge
name
None
ALPHA
Histogram
name
None
Feedback
Yes
No
.
or
.
Last modified November 16, 2022 at 10:21 AM PST:
Revert "update documentation for metrics now that code freeze is in effect" (51267b884c)
Home
Blog
Training
Partners
Community
CC BY 4.0

Enable Or Disable A Kubernetes API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

kubectl on Linux
Debug Pods
Debug Runni

ng Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Name

space Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-ap

iserver Encryption Configuration (v1)
Tasks
.
as a
The
api/all
.
to the
.
Read the
for the
Feedback
Yes
No
.
or
.
Last modified December 16, 2022 at 3:21 PM PST:
Fixed typo (294557aaeb)
Home
Blog
Training
Partners
Community
CC BY 4.0

Apply Pod Security Standards at the Cluster Level | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

Tutorials
Security
Note
.
baseline
.
KinD
kubectl
enforce
,
audit
, and
warn
.

Use
=
server 

--overwrite ns --all
\
=
Baseline
=
\
=
baseline
=
\
=
baseline
and
latest
version:
baseline
enforce
mode.
warn
and
audit
mode.
The
baseline
plugins:
    kind: PodSecurityC

onfiguration
    defaults:
EOF
Note:
v1beta1
.
v1alpha1
.
nodes:
  - |
            pathType: "DirectoryOrCreate"
EOF
Note:
add
/tmp
.

kind: Pod
metadata:
spec:
      ports:
EOF
Clean up
Run
and
created.
Run a
Set kubec

tl context to this new cluster
Feedback
Yes
No
.
or
.
Last modified October 25, 2022 at 5:32 PM PST:
Fix: Grammar (aa08044428)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Using RBAC Authorization | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
API group
API server
with the
RBAC
;
=
Role
,
,
and
. You can
,
kubectl
Caution:
An RBAC
Role
or
namespace
;
pods
:
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"watch"
,
"list"
]

nodes
)
/healthz
)
secrets
bound
):
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"secrets"
]
verbs
:
[
"get"
,
"watch"
,
"list"
]
.
subjects
.
:
kind
:
metadata
:
name
:
read-pods
namespace
:
default
subjects
:
-
kind
:
User
name
:
jane
apiGroup
:
roleRef
:
kind
:
Role
name
:
apiGroup
:
:
kind
:
metadata
:
name
:
#
namespace
:
subjects
:
-
kind
:
User
name
:
dave
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
manager
apiGroup
:
rbac.authoriza

tion.k8s.io
roleRef
:
kind
:
name
:
apiGroup
:
roleRef
roleRef
Making
roleRef
update
roleRef
The
See
pods
pods
log
is a
pods
/
) to
pods
and
log
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"pods"
,
]
verbs
:
[
"get"
,
"list"
]
list.
get
or
update
a
ConfigMap
named
:
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
]
:
[
]
verbs
:
[
"update"
,
"get"
]
Note:
create
or
For
create
list
or
watch
list
or
watch
resources
and
verbs
*
For
*
apiGroups
and
role).
:
kind
:
Role
metadata
:
namespace
:
default
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
"*"
]
verbs
:
[
"*"
]
Cautio

n:
The
You can
aggregate
set. The
selector
rules
Caution:
rules
field of an
objects
.
:
kind
:
metadata
:
name
:
:
:
-
:
:
"true"
rules
:
[]
.
:
kind
:
metadata
:
name
:
labels
:
:
"true"
rules
:
-
apiGroups
:
[
""
]
resources
:
[
,
,
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
The
:
kind
:
metadata
:
name
:
labels
:
:
"true"
:
"true"
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
---
kind
:
:
metadata
:
name
:
labels
:
:
"true"
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
the
rules
section.
"pods"
API Group
:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
"apps"
rules
:
-
apiGroups
:
[
"apps"
]
#
# at the HTTP level, th

e name of the resource for accessing Deployment
resources
:
[
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
"batch"
API group:
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"pods"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
-
apiGroups
:
[
"batch"
]
#
resources
:
[
"jobs"
]
verbs
:
[
"get"
,
"list"
,
"watch"
,
"create"
,
"update"
,
"patch"
,
"delete"
]
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
]
:
[
]
verbs
:
[
"get"
]
"nodes"
rules
:
-
apiGroups
:
[
""
]
#
resources
:
[
"nodes"
]
verbs
:
[
"get"
,
"list"
,
"watch"
]
/healthz
and
rules
:
-
:
[
,
]
verbs
:
[
"get"
,
"post"
]
.
Caution:
The prefix
system:
system:
by
accident.
system:
with
.
Note:
show the
subjects
section.
:
subjects
:
-
kind
:
User
name
:
apiGroup
:
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
name
:
default
namespace
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
subjects
:
-
kind
:
Group
name
:
apiG

roup
:
-
kind
:
Group
name
:
apiGroup
:
system:
.
Caution:
that have a
system:
prefix.
false
.
kubectl
run:
Note:
via
group
group
and
groups
system:
admin
,
edit
,
view
).
admin
,
edit
, or
view
metadata
:
labels
:
:
"true"
:
"true"
:
"true"
group
admin
None
.
If used in a
, allows read/write access to most resources in a name

space,
.
edit
None
.
view
None
user
scheduler
user
user
.
None
.
and
The
user
None
None
Heapster
None
kube-dns
namespace
kube-dns
None
None
.
None
None
.
group
/healthz
,
/livez
,
/readyz
,
/livez/*
,
/readyz/*
), and
/metrics
runs
Corresponding roles exist for each built-in controller, 

prefixed with
.
escalate
verb on the
roles
or
API group.
user-1
Role
or
escalate
verb on
roles
or
API group
or
bind
user-1
bind
user-1
admin
,
edit
, and
view
:
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"create"
]
-
apiGroups
:
[
"rbac.authorization.k

8s.io"
]
resources
:
[
]
verbs
:
[
"bind"
]
:
[
"admin"
,
"edit"
,
"view"
]
---
:
kind
:
metadata
:
name
:
namespace
:
roleRef
:
apiGroup
:
kind
:
name
:
subjects
:
-
apiGroup
:
kind
:
User
name
:
user-1
get
,
watch
and
list
on pods:
=
get --verb
=
=
=
pods
=
=
=
=
=
=
=
=
=
=
=
get
,
watch
and
list
on pods:
=
=
pods
=
=
=
=
=
=
=
=
"foo"
--verb
=
=
/logs/*
=
=
=
=
acme
=
=
=
acme
=
=
=
acme
=
=
root
Across the entire cluster, grant th

e permissions in the "system:node-proxier" ClusterRole to a user named "system:kube-proxy":
=
=
=
=
Examples:
namespace
, etc.).
\
=
view
\
=
\
=
Note:
.
\
=
view
\
=
\
=
Many
add-ons
run as the
Caution:
\
=
\
=
kubectl create rolebinding serviceaccou

nts-view
\
=
view
\
--group
=
\
=
\
=
view
\
--group
=
Warning:
\
=
\
--group
=
,
CVE
includes
:
kind
:
metadata
:
:
:
|-
labels
:
:
"true"
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
]
verbs
:
[
"create"
,
"delete"
,
,
"patch"
,
"update"
]
namespace
the
:
allowed by
either
(
or
--v=5
RBAC
).
You can use that information to determine which roles need to be granted to

 which users, groups, or service accounts.
Warning:
ALL
\
=
\
--user
=
admin
\
--user
=
kubelet
\
--group
=
Feedback
Yes
No
.
or
.
Last modified October 19, 2022 at 7:15 PM PST:
feat: Add caution note about rules field override in aggregated clusterroles (35c3321b02)
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-controller-manager | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
Synopsis
current state towards the desired state. Examples of controllers that ship with
Options
kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io. This is optional. If empty, all requests not skipped b

y authorization are forbidden.
The max length of duration signed certificates will be given. Individual CSRs may request shorter certs by setting spec.expirationSeconds.
File

name containing a PEM-encoded RSA or ECDSA private key used to sign certificates for the kubernetes.io/legacy-unknown signer. If specified, --cluster-signing-{cert,key}-file must not be set.
The number of resource quotas that are allowed to sync concurrently. Larger number = more responsive quota management, but more CPU (and network) load
All controllers: attachdetach, bootstrapsigner, cloud-node-lifecycle, clusterrole-aggregation, cronjob, csrapproving, csrcleaner, csrsigning, daemonset, deployment, disruption, endpoint, endpointslice, endpointslicemirroring, ephemeral-volume, garbagecollector, horizontalpodautoscaling, job, namespace, nodeipam, nodelifecycle, persistentvolume-binder, persistentvolume-expander, podgc, pv-protection, pvc-protection, replicaset, replicationcontroller, resourcequota, root-ca-cert-pub

lisher, route, service, serviceaccount, serviceaccount-token, statefulset, tokencleaner, ttl, ttl-after-finished
The length of endpoint slice updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated

To

pologyManagerPolicyBetaOptions=true|false (BETA - default=false)
-h, --help
Path to the config file for 

controller leader migration, or empty to use the value that reflects default configuration of the controller manager. The config file should be of type LeaderMigrationConfiguration, group controllermanager.config.k8s.io, version v1alpha1.
--namespace-sync-period duration     Default: 5m0s
The 

minimum ActiveDeadlineSeconds to use for an NFS Recycler pod
The period for syncing quota usage status in the system
File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified b

y --cert-dir.
-v, --v int
Feedback
Yes
No
.
or
.
Synopsis
Options
Home
Blog
Training
Partners
Community
CC BY 4.0

Considerations for large clusters | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

pha1)
A cluster is a set of
nodes
(physical
or virtual machines) running Kubernetes agents, managed by the
.
Kubernetes v1.26 supports clusters with up to 5000 nodes. More specifically,
Kubernetes is designed to accommodate configurations that meet
all
of the following criteria:
No more than 110 pods per node
No more than 5000 nodes
No more than 150000 total pods
No more than 300000 total containers
You can scale your cluster by adding or removing nodes. The way you do this depends
on how your cluster is deployed.
Cloud provider resource quotas
To avoid running into cloud provider quota issues, when creating a cluster with many nodes,
consider:
Requesting a quota increase for cloud resources such as:
Computer instances
CPUs
Storage volumes
In-use IP addresses
Packet filtering rule sets
Number of load balancers
Network subnets
Log streams
Gating the cluster scaling actions to bring up new nodes in batches, with a pause
between batches, because some cloud providers rate limit the creation of new instances.
Control plane components
For a large cluster, you need a control plane with sufficient compute and other
Typically you would run one or two control plane instances per failure zone,
scaling those instances vertically first and then scaling horizontally after reaching
the point of falling returns to (vertical) scale.
You should run at least one instance per failure zone to provide fault-tolerance. Kubernetes
nodes do not automatically steer traffic towards control-plane endpoints that are in the
same failure zone; however, your cloud provider might have its own mechanisms to do this.
For example, using a managed load balancer, you configure the load balancer to send traffic
that originates from the kubelet and Pods in failure zone
A
, and direct that traffic only
to the control plane hosts that are also in zone
A
. If a single control-plane host or
endpoint failure zone
A
goes offline, that means that all the control-plane traffic for
nodes in zone
A
is now being sent between zones. Running multiple control plane hosts in
each zone makes that outcome less likely.
etcd storage
To improve performance of large clusters, you can store Event objects in a separate
dedicated etcd instance.
When creating a cluster, you can (using custom tooling):
start and configure additional etcd instance
configure the
API server
to use it for storing events
See
and
Set up a High Availabil

ity etcd cluster with kubeadm
for details on configuring and managing etcd for a large cluster.
Addon resources
help to minimize the impact of memory leaks and other ways that pods and containers can
impact on other components. These resource limits apply to
addon
resources just as they apply to application workloads.
For example, you can set CPU and memory limits for a logging component:
...
:
-
name
:
fluentd-cloud-logging
image
:
fluent/fluentd-kubernetes-daemonset:v1
resources
:
limits
:
cpu
:
100m
memory
:
200Mi
Addons' default limits are typically based on data collected from experience running
each addon on small or medium Kubernetes clusters. When running on large
clusters, addons often consume more of some resources than their default limits.
If a large cluster is deployed without adjusting these values, the addon(s)
may continuously get killed because they keep hitting the memory limit.
Alternatively, the addon may run but with poor performance due to CPU time
slice restrictions.
To avoid running into cluster addon resource issues, when creating a cluster with
many nodes, consider the following:
Some addons scale vertically - there is one replica of the addon for the cluster
or serving a whole failure zone. For these addons, increase requests and limits
as you scale out your cluster.
Many addons scale horizontally - you add capacity by running more pods - but with
a very large cluster you may also need to raise CPU or memory limits slightly.
The VerticalPodAutoscaler can run in
recommender
mode to provide suggested
figures for requests and limits.
Some addons run as one copy per node, controlled by a
DaemonSet
: for example, a node-level log aggregator. Similar to
the case with horizontally-scaled addons, you may also need to raise CPU or memory
limits slightly.
is a custom resource that you can deploy into your cluster
to help you manage resource requests and limits for pods.
Vertical Pod Autoscaler
and how you can use it to scale cluster
components, including cluster-critical addons.
The
cluster autoscaler
integrates with a number of cloud providers to help you run the right number of
nodes for the level of resource demand in your cluster.
The
addon resizer
helps you in resizing the addons automatically as your cluster's scale changes.
Feedback
Yes
No
.
or
.
Last modified December 01, 2022 at 1:06 PM PST:
Update content/en/docs/setup/best-practices/cluster-large.md (aa98b4556f)
Home
Blog
Training
Partners
Community
CC BY 4.0

Translate a Docker Compose File to Kubernetes Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator 

Tasks
.
minikube
.
.
# Linux
curl -L https://github.com/kubernetes/kompose/releases/download/v1.26.0/kompose-linux-amd64 -o kompose
# macOS
# Windows
tarball
.
go get
EPEL
EPEL
.
If you have
EPEL
Homebrew
:
file.
version
:
"2"
services
:
:
image
:

ports
:
-
"6379"
:
image
:
ports
:
-
"6379"
:
frontend
:
image
:
ports
:
-
"80:80"
:
labels
:
:
kubectl
, run
and then
.
minikube
No events.
.
User Guide
CLI
Labels
Restart
example
ls
db-svc.yaml         docker-voting.yml          redis-svc.yaml     result-svc.yaml        vote-svc.yam

l           worker-svc.yaml
ls
OpenShift
example
and
kompose --provider openshift --file buildconfig/docker-compose.yml convert
Note:
.
kompose
and
Services
-j
objects,
, or
Helm
charts.
created
created
created
created
The
created
"web-svc.yaml

"
created
created
created
The
flag:
.
created
created
created
created
The
Helm
run:
Labels
kompose
version
:
"2"
services
:
nginx
:
image
:
nginx
:
foobar
build
:
./foobar
cap_add
:
- ALL
:
foobar
labels
:
:
nodeport
version
:
"2"
services
:
web
:
image
:
ports
:
-
links
:
- redis
labels
:
:
redis
:
image
:
redis:3.0
ports
:
-
"6379"
Key
Value
Note:
The
ports
kompose
will fail.
Restart
restart
construct of docker-compose to define that. Follow table below to see what happens on the
restart
value.
restart
Pod
""
Always
always
Always
Pod
OnFailure
no
Pod
Never
Note:
or
.
pival
pi
.
version
:
'2'
services
:
pival
:
image
:
perl
command
:
[
"perl"
,
,
"-wle"
,
]
restart
:
_
-
_
files.
Kompose supports Docker Compose versions: 1, 2 and 3. We have limited support on versions 2.1 

and 3.2 due to their experimental nature.
Feedback
Yes
No
.
or
.
Last modified September 05, 2022 at 9:16 PM PST:
Fix typos and layout: translate-compose-kubernetes.md (610a11cfdd)
User Guide
example
OpenShift
example
Labels
Restart
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm version | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 Set Up kubectl on macOS
Debug Pods
Get a Shell to a Run

ning Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Conta

iner's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configura

tion (v1)
Reference
Kubeadm
Synopsis
Options
-h, --help
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Communicate Between Containers in the Same Pod Using a Shared Volume | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubern

etes API Aggregation Layer
Tasks
Debug Pods
Debug a

 StatefulSet
Auditing
Run Jobs
TLS
Tutorials
Security
Apply P

od Security Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Audit Configuration (v1

)
Tasks
This page shows how to use a Volume to communicate between two Containers running
in the same Pod. See also how to allow processes to communicate by
sharing process namespace
between containers.
minikube
.
Creating a Pod that runs two Containers
In this exercise, you create a Pod that runs two Containers. The two containers
share a Volume that they can use to communicate. Here is the configuration file
pods/two-container-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
two-containers
spec
:
:
Never
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
nginx-container
image
:
nginx
:
-
name
:
mountPath
:
-
name
:
debian-container
image
:
debian
:
-
name
:
mountPath
:
/pod-data
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
"echo Hello from the debian container &gt; /pod-data/index.html"
]
In the configuration file, you can see that the Pod has a Volume named
.
The first container listed in the configuration file runs an nginx server. The
mount path for the shared Volume is
.
The second container is based on the debian image, and has a mount path of
/pod-data
. The second container runs the following command and then terminates.
echo Hello from the debian container &gt; /pod-data/index.html
Notice that the second container writes the
file in the root
directory of the nginx server.
Create the Pod and the two Containers:
kubectl apply -f https://k8s.io/examples/pods/two-container-pod.yaml
View information about the Pod and the Containers:
kubectl get pod two-containers --output=yaml
Here is a portion of the output:
kind: Pod
metadata:
  ...
  name: two-containers
  namespace: default
  ...
spec:
  ...
  containerStatuses:

  - containerID: docker://c1d8abd1 ...
    i

mage: debian
    ...
    lastState:
      terminated:
        ...
    name: debian-container
    ...

  - containerID: docker://96c1ff2c5bb ...
    image: nginx
    ...
    name: nginx-container
    ...
    state:
      running:
    ...
You can see that the debian Container has terminated, and the nginx Container
is still running.
Get a shell to nginx Container:
kubectl exec -it two-containers -c nginx-container -- /bin/bash
In your shell, verify that nginx is running:
root@two-containers:/# apt-get update
root@two-containers:/# apt-get install curl procps
root@two-containers:/# ps aux
USER       PID  ...  STAT START   TIME COMMAND
root         1  ...  Ss   21:12   0:00 nginx: master process nginx -g daemon off;
Recall that the debian Container created the
file in the nginx root
directory. Use
curl
to send a GET request to the nginx server:
root@two-containers:/# curl localhost
The output shows that nginx serves a web page written by the debian container:
Hello from the debian container
The primary reason that Pods can have multiple containers is to support
helper applications that assist a primary application. Typical examples of
helper applications are data pullers, data pushers, and proxies.
Helper and primary applications often need to communicate with each other.
Typically this is done through a shared filesystem, as shown in this exercise,
or through the loopback network interface, localhost. An example of this pattern is a
web server along with a helper program that polls a Git repository for new updates.
The Volume in this exercise provides a way for Containers to communicate during
the life of the Pod. If the Pod is deleted and recreated, any data stored in
the shared Volume is lost.
patterns for composite containers
.
composite containers for modular architecture
.
See
Configuring a Pod to Use a Volume for Storage
.
See
Configure a Pod to share process namespace between containers in a Pod
See
Volume
.
See
Pod
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

IngressClass | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Reference
kind
metadata
(
)
spec
(
)
(string)
IngressClassParametersReference identifies an API object. This can be used to specify a cluster or namespace-scoped resource.
(string)
(string)
(string)
IngressClassList is a collection of IngressCla

sses.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
Thanks for the feedback. If you have a specific, answerable question about how to use Kubernetes, ask it o

n
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

kubectl | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

Reference
kubectl
kubectl
Synopsis
Options
-h, --help
If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
log to standard 

error instead of files
See Also
kubectl cp
- Diff live version against would-be applied version
- Print the list of flags inherited 

by all commands
Feedback
Yes
No
.
or
.
Last modified November 10, 2022 at 5:22 PM PST:
Events beta docs update (232983a4a2)
Synopsis
Options
See Also
Home
Blog
Training
Partners
Community
CC BY 4.0

PodScheduling v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, St

orage, and Networking Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodSecurity Ad

mission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfile, ConfigMaps and

 Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate Limit Confi

Reference
kind
metadata
(
)
spec
(
status
(
)
The size of this field is limited to 128. This is large enough for many clusters. Larger clusters may need more attempts to find a node that suits all pending resources. This may get increased in the future, but not reduced.
(string)
(string)
PodSc

hedulingList
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
HTTP 

Request
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0
Trademark U

Official CVE Feed | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
JSON Feed
of
CVE ID
#113757
#113756
#112513
#112192
#104980
#102106
#101435
#100096
#97076
#95624
#95623
#95622
#95621
#93032
#92914
#92315
#91542
#91507
#89535
#89378
#89377
#87773
#85867
#85233
#83253
Bearer token

s are revealed in logs
#81114
#81023
#80984
#80983
#78308
#76797
#76676
#74534
#71411
#65750
#61297
#60814
#60813
#47611
#43459
Bucket which is writable only by a small number of trusted members of the
Feedback
Yes
No
.
or
.
Last modified August 26, 2022 at 11:58 AM PST:
Update official-cve-feed.md (ec2a59efda)
Home
Blog
Training
Partners
Community
CC BY 4.0

Validating Admission Policy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
C

ompute, Storage, and Networking Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodS

ecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfile, Conf

igMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate L

imit Configuration (v1alpha1)
Reference
The
A
owner
Pods
owner
").
At least a
If a
in
unset.
Ensure the
:
kind
:
metadata
:
name
:
spec
:
failur

ePolicy
:
Fail
:
:
-
apiGroups
:
[
"apps"
]
:
[
"v1"
]
:
[
"CREATE"
,
"UPDATE"
]
resources
:
[
]
:
-
:
field.
:
kind
:
metadata
:
name
:
spec
:
:
:
:
:
:
test
:
kind
:
metadata
:
name
:
spec
:
:
Fail
paramKind
:
:
kind
:
:
:
-
apiGroups
:
[
"apps"
]
:
[
"v1"
]
:
[
"CREATE"
,
"UPDATE"
]
resources
:
[
]
:
-
:
reason
:
Invalid
The
e.g.
.
ConfigMap
The
field.
:
kind
:
metadata
:
name
:
spec
:
:
paramRef
:
name
:
:
:
:
:
test
:
kind
:
metadata
:
name
:
:
3
:
kind
:
metadata
:
name
:
"replical

imit-binding-nontest"
spec
:
:
paramRef
:
name
:
:
:
:
-
key
:
operator
:
NotIn,
values
:
[
"test"
]
:
kind
:
metadata
:
name
:
:
100
:
kind
:
metadata
:
name
:
spec
:
:
params
:
:
:
:
-
key
:
operator
:
Exists
:
has()
.
If
If
read
paramKind
in
and
paramRef
in
.
paramKind
read
Ignore
or
Fail
.
Ignore
Fail
:
:
kind
:
spec
:
...
:
Ignore
:
-
:
.
'params' - Parameter resource referred

 to by the policy binding being evaluated. The value is
null if
ParamKind
is unset.
The
,
kind
,
and
__
__dot__
.
__dash__
-
__slash__
/
Note:
A
int
namespace
x-prop
redact__d
string
- 'set':
X + Y
X
Y
- 'map':
X + Y
X
Y
X
and
Y
Y
with
Purpose
Read
,
Forbidden
,
Invalid
,
.
If not set,
Feedback
Yes
No
.

or
.
Last modified December 17, 2022 at 4:35 PM PST:
Reformat the validating-admission-policy reference page (37955a816b)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, Storage, and Networking Extension

s
Tasks
Monitoring, 

Logging, and Debugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial -

 Configuring a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Image Polic

y API (v1alpha1)
Tutorials
Feedback
Yes
No
.
or
.
Last modified May 15, 2018 at 3:29 PM PST:
Fix up Tutorials landing page and Tutorials left nav. (#8557) (21fd86ed4b)
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kubectl
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

CSIDriver | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

Reference
CSIDriver
CSIDriver
CSIDriver
kind
metadata
(
)
spec
(
(boolean)
attachRequired indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume() method), and that the Kubernetes attach detach controller should call the attach volume interface which checks the volumeattachment status and waits until the volume is attached before proceeding to mounting. The CSI external-attacher coordinates with CSI volume driver and updates the volumeattachment status when the attach operation is complete. If the CSIDriverRegistry feature gate is enabled and the value is specified to false, the attach operation will be skipped. Otherwise the attach operation will be called.
(string)
(boolean)
If set to true, podInfoOnMount indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during moun

t operations. If set to false, pod information will not be passed on mount. Default is false. The CSI driver specifies podInfoOnMount as part of driver deployment. If true, Kubelet will pass pod information as VolumeContext in the CSI NodePublishVolume() calls. The CSI driver is responsible for parsing and validating the information passed in as VolumeContext. The following VolumeConext will be passed if podInfoOnMount is set to true. This list might grow, but the prefix will be used. "csi.storage.k8s.io/pod.name": pod.Name "csi.storage.k8s.io/pod.namespace": pod.Namespace "csi.storage.k8s.io/pod.uid": string(pod.UID) "csi.storage.k8s.io/ephemeral": "true" if the volume is an ephemeral inline volume
(boolean)
(boolean)
-o context
(boolean)
This field was immutable in Kubernetes &lt;= 1.22 and now is mutable.
},
...
}
TokenRequest contains parameters of a 

service account token.
(int64)
kind
metadata
(
ListMeta
)
items
([]
CSIDriver
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
202 (
CSIDriver
update
name
(
in path
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
gracePeriod

Seconds
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
CSIDriver
): OK
202 (
CSIDriver
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
CSIDriver
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Diagram Guide | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

This guide shows you how to create, edit and share diagrams using the Mermaid
JavaScript library. Mermaid.js allows you to generate diagrams using a simple
markdown-like syntax inside Markdown files. You can also use Mermaid to
generate
.svg
or
.png
image files that you can add to your documentation.
The target audience for this guide is anybody wishing to learn about Mermaid
and/or how to create and add diagrams to Kubernetes documentation.
Figure 1 outlines the topics covered in this section.
subgraph m[Mermaid.js]
S[ ]-.-
C[build
diagrams
with markdown] --&gt;
D[on-line
live editor]
end
A[Why are diagrams
useful?] --&gt; m
m --&gt; N[3 x methods
for creating
diagrams]
N --&gt; T[Examples]
T --&gt; X[Styling
and
captions]
X --&gt; V[Tips]
class A,C,D,N,X,m,T,V box
%% you can hyperlink Mermaid diagram nodes to a URL using click statements
click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank
click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIG

ojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank
click V "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank
enabled
Figure 1. Topics covered in this section.
All you need to begin working with Mermaid is the following:
Basic understanding of markdown.
Using the Mermaid live editor.
Using
.
Using the
Hugo {{&lt; figure &gt;}} shortcode
.
Performing
Hugo local previews
.
Familiar with the
process.
Note:
You can click on each diagram in this section to view the code and rendered
diagram in the Mermaid live editor.
Why you should use diagrams in documentation
Diagrams improve documentation clarity and comprehension. There are advantages for both the user and the contributor.
The user benefits include:
Friendly landing spot
. A detailed text-only greeting page could
intimidate users, in particular, first-time Kubernetes users.
Faster grasp of concepts
. A diagram can help users understand the key
points of a complex topic. Your diagram can serve as a visual learning guide
to dive into the topic details.
Better retention
. For some, it is easier to recall pictures rather than text.
The contributor benefits include:
Assist in developing the structure and content
of your contribution. For
example, you can start with a simple diagram covering the high-level points
and then dive into details.
Expand and grow the user community
. Easily consumed documentation
augmented with diagrams attracts new users who might previously have been
reluctant to engage due to perceived complexities.
You should consider your target audience. In addition to experienced K8s
users, you will have many who are new to Kubernetes. Even a simple diagram can
assist new users in absorbing Kubernetes concepts. They become emboldened and
more confident to further explore Kubernetes and the documentation.
Mermaid
Mermaid
is an open source
JavaScript library that allows you to create, edit and easily share diagrams
using a simple, markdown-like syntax configured inline in Markdown files.
The following lists features of Mermaid:
Simple code syntax.
Includes a web-based tool allowing you to code and preview your diagrams.
Supports multiple formats including flowchart, state and sequence.
Easy collaboration with colleagues by sharing a per-diagram URL.
Broad selection of shapes, lines, themes and styling.
The following lists advantages of using Mermaid:
No need for separate, non-Mermaid diagram tools.
Adheres to existing PR workflow. You can think of Mermaid code as just
Markdown text included in your PR.
Simple tool builds simple diagrams. You don't want to get bogged down
(re)crafting an overly complex and detailed picture. Keep it simple!
Mermaid provides a simple, open and transparent method for the SIG communities
to add, edit and collaborate on diagrams for new or e

xisting documentation.
Note:
You can still use Mermaid to create/edit diagrams even if it's not supported
in your environment. This method is called
Mermaid+SVG
and is explained
below.
Live editor
The
Mermaid live editor
is
a web-based tool that enables you to create, edit and review diagrams.
The following lists live editor functions:
Displays Mermaid code and rendered diagram.
Generates a URL for each saved diagram. The URL is displayed in the URL
field of your browser. You can share the URL with colleagues who can access
and modify the diagram.
Option to download
.svg
or
.png
files.
Note:
The live editor is the easiest and fastest way to create and edit Mermaid diagrams.
Methods for creating diagrams
Figure 2 outlines the three methods to generate and add diagrams.
graph TB
A[Contributor]
B[Inline
Mermaid code
added to .md file]
C[Mermaid+SVG
Add mermaid-generated
svg file to .md file]
D[External tool
Add external-tool-
generated svg file
to .md file]
A --&gt; B
A --&gt; C
A --&gt; D
class A,B,C,D box
click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
click B "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpm

YWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
enabled
Figure 2. Methods to create diagrams.
Inline
Figure 3 outlines the steps to follow for adding a diagram using the Inline
method.
graph LR
A[1. Use live editor
to create/edit
diagram] --&gt;
B[2. Store diagram
URL somewhere] --&gt;
C[3. Copy Mermaid code
to page markdown file] --&gt;
D[4. Add caption]
click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
click B "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJcbiAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
enabled
Figure 3. Inline Method steps.
The following lists the steps you should follow for adding a diagram using the Inline method:
Create your diagram using the live editor.
Store the diagram URL somewhere for later access.
Copy the mermaid code to the location in your
.md
file where you want the diagram to appear.
Add a caption below the diagram using Markdown text.
A Hugo build runs the Mermaid code and turns it into a diagram.
Note:
You may find keeping track of diagram URLs is cumbersome. If so, make a note
in the
.md
file that the Mermaid code is self-documenting. Contributors can
copy the Mermaid code to and from the live editor for diagram edits.
Here is a sample code snippet contained in an
.md
file:
---
title: My PR
---
Figure 17 shows a simple A to B process.
some markdown text
...
{{&lt; mermaid &gt;}} 
    graph TB
    A --&gt; B
{{&lt; /mermaid &gt;}}

Figure 17. A t

o B
more text
Note:
You must include the Hugo Mermaid shortcode
tags at the start and end of the Mermaid code block. You should add a diagram
caption below the diagram.
For more details on diagram captions, see
How to use captions
.
The following lists advantages of the Inline method:
Live editor tool.
Easy to copy Mermaid code to and from the live editor and your
.md
file.
No need for separate
.svg
image file handling.
Content text, diagram code and diagram caption contained in the same
.md
file.
local
and Netlify previews to verify the diagram is properly rendered.
Caution:
The Mermaid live editor feature set may not support the K8s/website Mermaid feature set.
You might see a syntax error or a blank screen after the Hugo build.
If that is the case, consider using the Mermaid+SVG method.
Figure 4 outlines the steps to follow for adding a diagram using the Mermaid+SVG method.
diagram]
URL somewhere]
C[3. Generate .svg file
and download to
images/ folder]
subgraph w[ ]
D[4. Use figure shortcode
to reference .svg
file in page
.md file] --&gt;
E[5. Add caption]
end
A --&gt; B
B --&gt; C
C --&gt; w
class A,B,C,D,E,w box
click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click B "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dX

G4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click E "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
enabled
Figure 4. Mermaid+SVG method steps.
The following lists the steps you should follow for adding a diagram using the Mermaid+SVG method:
Generate an
.svg
image file for the diagram and download it to the appropriate
images/
folder.
Use the
{{&lt; figure &gt;}}
shortcode to reference the diagram in the
.md
file.
Add a caption using the
shortcode's
caption
For example, use the live editor to create a diagram called
boxnet
.
Store the diagram URL somewhere for later access. Generate and download a
boxnet.svg
file to the appropriate
../images/
folder.
Use the
shortcode in your PR's
.md
file to reference
the
.svg
image file and add a caption.
{
{&lt;
figure
src=
"/static/images/boxnet.svg"
alt=
"Boxnet figure"
class=
"diagram-large"
caption=
"Figure 14. Boxnet caption"
&gt;
}
}
.
Note:
The
shortcode is the preferred method for adding
.svg
image files
to your documentation. You can also use the standard markdown image syntax like so:
![my boxnet diagram](static/images/boxnet.svg)
.
And you will need to add a caption below the diagram.
You should add the live editor URL as a comment block in the
.svg
image file using a text editor.
For example, you would include the following at the beginning of the
.svg
image file:
&lt;!-- To view or edit the mermaid code, use the following URL: --&gt;
&lt;!-- https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb ... &lt;remainder of the URL&gt; --&gt;
The following lists advantages of the Mermaid+SVG method:
Live editor tool supports the most current Mermaid feature set.
Employ existing K8s/website methods for handling
.svg
image files.
Environment doesn't require Mermaid support.
Be sure to check that your diagram renders properly using the
local
and Netlify previews.
External tool
Figure 5 outlines the steps to follow for adding a diagram using the External Tool method.
First, use your external tool to create the diagram and save it as an
.svg
or
.png
image file. After that, use the same steps as the
method for adding
.svg
A[1. Use external
tool to create/edit
diagram]
B[2. If possible, save
diagram coordinates
for contributor
access]
C[3. Generate .svg
or.png file
appropriate
images/ fol

wZGF0ZURpYWdyYW0iOmZhbHNlfQ"
click E "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0byBjcmVhdGUvZWRpdDxicj5kaWFncmFtXVxuICAgIEJbMi4gSWYgcG9zc2libGUsIHNhdmU8YnI-ZGlhZ3JhbSBjb29yZGluYXRlczxicj5mb3IgY29udHJpYnV0b3I8YnI-YWNjZXNzXVxuICAgIENbMy4gR2VuZXJhdGUgLnN2ZyA8YnI-b3IucG5nIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSBzdmcgb3I8YnI-cG5nIGZpbGUgaW48YnI-cGFnZSAubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbiAgICBBIC0tPiBCXG4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ"
enabled
Figure 5. External Tool method steps
The following lists the steps you should follow for adding a diagram using the External Tool method:
Use your external tool to create a diagram.
Save the diagram coordinates for contributor access. For example, your tool
may offer a link to the diagram image, or you could place the source code
file, such as an
.xml
file, in a public repository for later contributor access.
Generate and save the diagram as an
.svg
or
.png
image file.
Download this file to the appropriate
folder.
Use the
.md
file.
caption
Here is the
shortcode for the
images/apple.svg
diagram:
{{&lt; figure src="/static/images/apple.svg" alt="red-apple-figure" class="diagram-large" caption="Figure 9. A Big Red Apple" &gt;}}
If your external drawing tool permits:
You can incorporate multiple
.svg
or
.png
logos, icons and images into your diagram.
However, make sure you observe copyright and follow the Kubernetes documentation
guidelines
on the use of third party content.
You should save the diagram source coordinates for later contributor access.
For example, your tool may offer a link to the diagram image, or you could
place the source code file, such as an
.xml
file, somewhere for contributor access.
For more information on K8s and CNCF logos and images, check out
CNCF Artwork
.
The following lists advantages of the External Tool method:
Contributor familiarity with external tool.
Diagrams require more detail than what Mermaid can offer.
Don't forget to check that your diagram renders correctly using the
local
Examples
This section shows several examples of Mermaid diagrams.
Note:
The code block examples omit the Hugo Mermaid
shortcode tags. This allows you to copy the code block into the live editor
to experiment on your own.
Note that the live editor doesn't recognize Hugo shortcodes.
Example 1 - Pod topology spread constraints
Figure 6 shows the diagram appearing in the
page.
graph TB
subgraph "zoneB"
n3(Node3)
n4(Node4)
end
subgraph "zoneA"
n1(Node1)
n2(Node2)
end
classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
class n1,n2,n3,n4 k8s;
class zoneA,zoneB cluster;
click n3 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZm

lsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click n4 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click n1 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click n2 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
enabled
Figure 6. Pod Topology Spread Constraints.
Code block:
graph TB
   subgraph "zoneB"
       n3(Node3)
       n4(Node4)
   end
   subgraph "zoneA"
       n1(Node1)
       n2(Node2)
   end
 
   classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
   classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
   classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
   class n1,n2,n3,n4 k8s;
   class zoneA,zoneB cluster;
Example 2 - Ingress
Figure 7 shows the diagram appearing in the
What is Ingress
page.
graph LR;
client([client])-. Ingress-managed
load balancer .-&gt;ingress[Ingress];
ingress--&gt;|routing rule|service[Service];
subgraph cluster
ingress;
service--&gt;pod1[Pod];
service--&gt;pod2[Pod];
end
class ingress,service,pod1,pod2 k8s;
class client plain;
class cluster cluster;
click client "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58c

mZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6ZmFsc2V9" _blank
enabled
Figure 7. Ingress
graph LR;
 client([client])-. Ingress-managed &lt;br&gt; load balancer .-&gt;ingress[Ingress];
 ingress--&gt;|routing rule|service[Service];
 subgraph cluster
 ingress;
 service--&gt;pod1[Pod];
 service--&gt;pod2[Pod];
 end
 classDef plain fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
 classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff;
 classDef cluster fill:#fff,stroke:#bbb,stroke-width:2px,color:#326ce5;
 class ingress,service,pod1,pod2 k8s;
 class client plain;
 class cluster cluster;
Example 3 - K8s system flow
Figure 8 depicts a Mermaid sequence diagram showing the system flow between
K8s components to start a container.
Figure 8. K8s system flow diagram
%%{init:{"theme":"neutral"}}%%
    actor me
    participant apiSrv as control plane&lt;br&gt;&lt;br&gt;api-server
    participant etcd as control plane&lt;br&gt;&lt;br&gt;etcd datastore
    participant cntrlMgr as control plane&lt;br&gt;&lt;br&gt;controller&lt;br&gt;manager
    participant sched as control plane&lt;br&gt;&lt;br&gt;scheduler
    participant kubelet as node&lt;br&gt;&lt;br&gt;kubelet
    participant container as node&lt;br&gt;&lt;br&gt;container&lt;br&gt;runtime
    me-&gt;&gt;apiSrv: 1. kubectl create -f pod.yaml
    apiSrv--&gt;&gt;etcd: 2. save new state
    cntrlMgr-&gt;&gt;apiSrv: 3. check for changes
    sched-&gt;&gt;apiSrv: 4. watch for unassigned pods(s)
    apiSrv-&gt;&gt;sched: 5. notify about pod w nodename=" "
    sched-&gt;&gt;apiSrv: 6. assign pod to node
    apiSrv--&gt;&gt;etcd: 7. save new state
    kubelet-&gt;&gt;apiSrv: 8. look for newly assigned pod(s)
    apiSrv-&gt;&gt;kubelet: 9. bind pod to node
    kubelet-&gt;&gt;container: 10. start container
    kubelet-&gt;&gt;apiSrv: 11. update pod status
    apiSrv--&gt;&gt;etcd: 12. save new state
How to style diagrams
You can style one or more diagram elements using well-known CSS nomenclature.
You accomplish this using two types of statements in the Mermaid code.
classDef
defines a class of style attributes.
class
defines one or more elements to apply the class to.
In the code for
figure 7
,
you can see examples of both.
classDef k8s fill:#326ce5,stroke:#fff,stroke-width:4px,color:#fff; // defines style for the k8s class
class ingress,service,pod1,pod2 k8s; // k8s class is applied to elements ingress, service, pod1 and pod2.
You can include one or multiple
classDef
and
class
statements in your diagram.
You can also use the official K8s
#326ce5
hex color code for K8s components in your diagram.
For more information on styling and classes, see
Mermaid Styling and classes docs
.
A caption is a brief description of a diagram. A title or a short description
of the diagram are examples of captions. Captions aren't meant to replace
explanatory text you have in your documentation. Rather, they serve as a
"context link" between that text and your diagram.
The combination of some text and a diagram tied together with a caption help
provide a concise representation of the information you wish to convey to the
user.
Without captions, you are asking the user to scan the text above or below the
diagram to figure out a meaning. This can be frustrating for the user.
Figure 9 lays out the three components for proper captioning: diagram, diagram
caption and the diagram referral.
flowchart
A[Diagram
Inline Mermaid or
SVG image files]
B[Diagram Caption
Add Figure Number. and
Caption Text]
C[Diagram Referral
Referenence Figure Number
in text]
classDef box fill:#f

ff,stroke:#000,stroke-width:1px,color:#000;
class A,B,C box
click A "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE51bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjpmYWxzZX0" _blank
click B "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE51bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjpmYWxzZX0" _blank
click C "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE51bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjpmYWxzZX0" _blank
enabled
Figure 9. Caption Components.
Note:
You should always add a caption to each diagram in your documentation.
Diagram
The
and
External Tool
methods generate
.svg
Here is the
shortcode for the diagram defined in an
.svg
image file saved to
/images/docs/components-of-kubernetes.svg
:
{{&lt; figure src="/images/docs/components-of-kubernetes.svg" alt="Kubernetes pod running inside a cluster" class="diagram-large" caption="Figure 4. Kubernetes Architecture Components &gt;}}
You should pass the
src
,
alt
,
class
and
caption
values into the
shortcode. You can adjust the size of the diagram using
diagram-large
,
diagram-medium
and
diagram-small
classes.
Note:
Diagrams created using the
Inline
method don't use the
shortcode. The Mermaid code defines how the diagram will render on your page.
See
for more information on the different methods for creating diagrams.
Diagram Caption
Next, add a diagram caption.
If you define your diagram in an
.svg
image file, then you should use the
caption
{{&lt; figure src="/images/docs/components-of-kubernetes.svg" alt="Kubernetes pod running inside a cluster" class="diagram-large" caption="Figure 4. Kubernetes Architecture Components" &gt;}}
If you define your diagram using inline Mermaid code, then you should use Markdown text.
Figure 4. Kubernetes Architecture Components
The following lists several items to consider when adding diagram captions:
Use the
shortcode to add a diagram caption for
and
diagrams.
Use simple Markdown text to add a diagram caption for the
Inline
method.
Prepend your diagram caption with
Figure NUMBER.
. You must use
Figure
and the number must be unique for each diagram in your documentation page.
Add a period after the number.
Add your diagram caption text after the
on the same line.
You must puncuate the caption with a period. Keep the capti

on text short.
Position your diagram caption
BELOW
your diagram.
Diagram Referral
Finally, you can add a diagram referral. This is used inside your text and
should precede the diagram itself. It allows a user to connect your text with
the associated diagram. The
Figure NUMBER
in your referral and caption must
match.
You should avoid using spatial references such as
..the image below..
or
..the following figure ..
Here is an example of a diagram referral:
Figure 10 depicts the components of the Kubernetes architecture.
The control plane ...
Diagram referrals are optional and there are cases where they might not be
suitable. If you are not sure, add a diagram referral to your text to see if
it looks and sounds okay. When in doubt, use a diagram referral.
Complete picture
Figure 10 shows the Kubernetes Architecture diagram that includes the diagram,
diagram caption and diagram referral. The
shortcode
renders the diagram, adds the caption and includes the optional
link
parameter so you can hyperlink the diagram. The diagram referral is contained
in this paragraph.
Here is the
shortcode for this diagram:
{{&lt; figure src="/images/docs/components-of-kubernetes.svg" alt="Kubernetes pod running inside a cluster" class="diagram-large" caption="Figure 10. Kubernetes Architecture." link="https://kubernetes.io/docs/concepts/overview/components/" &gt;}}
Figure 10. Kubernetes Architecture.
Tips
Always use the live editor to create/edit your diagram.
Always use Hugo local and Netlify previews to check out how the diagram
appears in the documentation.
Include diagram source pointers such as a URL, source code location, or
indicate the code is self-documenting.
Always use diagram captions.
Very helpful to include the diagram
.svg
or
.png
image and/or Mermaid
source code in issues and PRs.
With the
and
methods, use
.svg
because they stay sharp when you zoom in on the diagram.
Best practice for
.svg
files is to load it into an SVG editing tool and use the
"Convert text to paths" function.
This ensures that the diagram renders the same on all systems, regardless of font
availability and font rendering support.
No Mermaid support for additional icons or artwork.
Hugo Mermaid shortcodes don't work in the live editor.
Any time you modify a diagram in the live editor, you
must
save it
to generate a new URL for the diagram.
Click on the diagrams in this section to view the code and diagram rendering
in the live editor.
Look over the source code of this page,
diagram-guide.md
, for more examples.
Mermaid docs
for explanations and examples.
Most important,
Keep Diagrams Simple
.
This will save time for you and fellow contributors, and allow for easier reading
by new and experienced users.
Feedback
Yes
No
.
or
.
Mermaid
Inline
Examples
Tips
Home
Blog
Training
Partners
Community
CC BY 4.0

Safely Drain a Node | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
node
,
.
concept,
and have
for
.
Use
You can use
node before you perform maintenance on the node (e.g. kernel upgrade,
to
Note:
By default
the
When
The
different nodes in parallel, in different terminals or i

n the
,
healthy
;
healthy
(such as
.
.
Feedback
Yes
No
.
or
.
Use
Home
Blog
Training
Partners
Community
CC BY 4.0

Fine Parallel Processing Using a Work Queue | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

nstall and Set Up kubectl on Linux
Debug Pods
Debug Init Cont

ainers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Sta

ndards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration 

Tasks
Run Jobs
minikube
Job
.
See the
job.yaml
worker.py
printed.
"/bin/sh"
Waiting
for
false
Hit enter
for
command
prompt
redis:637

9&gt; rpush job2 "cherry"
1) "apple"
4) "date"
5) "fig"
6) "grape"
7) "lemon"
8) "melon"
job2
.
Download
).
import
time
import
rediswq
host
=
"redis"
# import os
q
=
rediswq
.
=
"job2"
, host
=
host)
print
(
+
q
.
print
(
+
str
(q
.
empty()))
while
not
q
.
empty():
item
=
q
.
=
10
, block
=
True
, timeout
=
2
)
if
item
is
not
None
:
itemstr
=
item
.
decode(
"utf-8"
)
print
(
+
itemstr)
time
.
sleep(
10
)
q
.
else
:
print
(
)
print
(
)
worker.py
,
, and
the image:
For the
Docker Hub
.
Registry
, tag
:
batch/v1
kind
:
Job
metadata
:
name
:
job-wq-2
spec
:
:
2
template
:
metadata
:
name
:
job-wq-2
spec
:
:
-
name
:
c
image
:
:
OnFailure
change
too.
=
=
job-name
=
job-wq-2
2
11
Jan
2016
1
Running /
0
0
Failed
=
b1c7e4e3-92e1-11e7-b85e

-fa163ee3c11f
job-name
=
job-wq-2
c:
Port:
Events:
33s          33s
1
{
}
kubectl
wait
--for
=
condition
=
complete
--timeout
=
.
instead,
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Interactive Tutorial - Deploying an App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extending Kuberne

tes
Tasks
Migrate from PodSecurityPolicy to the Built-In

 PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfile,

 ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event R

ate Limit Configuration (v1alpha1)
Tutorials
A Pod is the basic execution unit of a Kubernetes application. Each Pod represents a part of a workload that is running on your cluster.
Learn more about Pods
.
&lt; Return to Module 1
Home
Continue to Module 3 &gt;
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

TLS | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
TLS
Understand how to protect traffic within your cluster using Transport Layer Security (TLS).
1:
2:
3:
1 - Configure Certificate Rotation for the Kubelet
Overview
rotation
,
The
kubelet
The
(
status of
Pending
a status of
Approved
certificate. Again, the controller manager will automatically approve the certificate
2 - Manage TLS Certificates in a Cluster
Kubernetes provides a
API, which lets you provision TLS
certificates signed by a Certificate Authority (CA) that you control. These CA
and certificates can be used by your workloads to establish trust.
API uses a protocol that is similar to the
ACME
draft
.
Note:
Certificates created using the
API are signed by a
dedicated CA
. It is possible to configure your cluster to use the cluster root
CA for this purpose, but you should never rely on this. Do not assume 

that
these certificates will validate against the cluster root CA.
minikube
You need the
cfssl
tool. You can download
cfssl
from
https://github.com/cloudflare/cfssl/releases
.
Some steps in this page use the
jq
tool. If you don't have
jq
, you can
install it via your operating system's software sources, or fetch it from
https://stedolan.github.io/jq/
.
Trusting TLS in a cluster
Trusting the
custom CA
from an application running as a pod usually requires
some extra application configuration. You will need to add the CA certificate
bundle to the list of CA certificates that the TLS client or server trusts. For
example, you would do this with a golang TLS config by parsing the certificate
chain and adding the parsed certificates to the
RootCAs
tls.Config
struct.
Note:
Even though the custom CA certificate may be included in the filesystem (in the
ConfigMap
),
you should not use that certificate authority for any purpose other than to verify internal
Kubernetes endpoints. An example of an internal Kubernetes endpoint is the
Service named
in the default namespace.
If you want to use a custom certificate authority for your workloads, you should generate
that CA separately, and distribute its CA certificate using a
ConfigMap
that your pods
have access to read.
Requesting a certificate
The following section demonstrates how to create a TLS certificate for a
Kubernetes service accessed through DNS.
Note:
This tutorial uses CFSSL: Cloudflare's PKI and TLS toolkit
click here
to know more.
Create a certificate signing request
Generate a private key and certificate signing request (or CSR) by running
the following command:
cat
&lt;&lt;EOF | cfssl genkey - | cfssljson -bare server
{
"hosts": [
"my-svc.my-namespace.svc.cluster.local",
"my-pod.my-namespace.pod.cluster.local",
"192.0.2.24",
"10.0.34.2"
],
"CN": "my-pod.my-namespace.pod.cluster.local",
"key": {
"algo": "ecdsa",
"size": 256
}
}
EOF
Where
192.0.2.24
is the service's cluster IP,
my-svc.my-namespace.svc.cluster.local
is the service's DNS name,
10.0.34.2
is the pod's IP and
my-pod.my-namespace.pod.cluster.local
is the pod's DNS name. You should see the output similar to:
2022/02/01 11:45:32 [INFO] generate received request
2022/02/01 11:45:32 [INFO] received CSR
2022/02/01 11:45:32 [INFO] generating key: ecdsa-256
2022/02/01 11:45:32 [INFO] encoded CSR
This command generates two files; it generates
server.csr
containing the PEM
encoded
PKCS#10
certification request,
and
containing the PEM encoded key to the certificate that
is still to be created.
Create a CertificateSigningRequest object to send to the Kubernetes API
Generate a CSR manifest (in YAML), and send it to the API server. You can do that by
running the following command:
cat
metadata:
name: my-svc.my-namespace
spec:
request: $(cat server.csr | base64 | tr -d '\n')
signerName: example.com/serving
usages:
- digital signature
- key encipherment
- server auth
EOF
file created in step 1 is base64 encoded
and stashed in the
.spec.request
field. You are also requesting a
certificate with the "digital signature", "key encipherment", and "server
auth" key usages, signed by an example
example.com/serving
signer.
A specific
must be requested.
View documentation for
supported signer names
The CSR should now be visible from the API in a Pending state. You can see
it by running:
kubectl describe csr my-svc.my-namespace

Name:                   my-svc.my-namespace
Labels:                 &lt;none&gt;
Annotations:            &lt;none&gt;
CreationTimestamp:      Tue, 01 Feb 2022 11:49:15 -0500
Requesting User:        yourname@example.com
Signer:                 example.com/serving
Status:                 Pending
Subject:
        Common Name:    my-pod.my-namespace.pod.cluster.local
        Serial Number:
Subject Alternative Names:
        DNS Names:      my-pod.my-namespace.pod.cluster.local
                        my-svc.my-namespace.svc.cluster.local
        IP Addresses:   192.0.2.24
                        10.0.34.2
Events: &lt;none&gt;
Get the CertificateSigningRequest approved
Approving the
certificate signing request
is either done by an automated approval process or on a one off basis by a cluster
administrator. If you're authorized to approve a certificate request, you can do that
manually using
kubectl
; for example:
kubectl certificate approve my-svc.my-namespace
certificatesigningrequest.certificates.k8s.io/my-svc.my-namespace approved
You should now see the following:
NAME                  AGE   SIGNERNAME            REQUESTOR              REQUESTEDDURATION   CONDITION
my-svc.my-namespace   10m   example.com/serving   yourname@example.com   &lt;none&gt;              Approved
This means the certificate request has been approved and is waiting for the
requested signer to sign it.
Sign the CertificateSigningRequest
Next, you'll play the part of a certificate signer, issue the certificate, and upload it to the API.
A signer would typically watch the CertificateSigningRequest API for objects with its
,
check that they have been approved, sign certificates for those requests,
and update the API object status with the issued certificate.
Create a Certificate Authority
You need an authority to provide the digital signature on the new certificate.
First, create a signing certificate by running the following:
cat
&lt;&lt;EOF | cfssl gencert -initca - | cfssljson -bare ca
{
"CN": "My Example Signer",
"key": {
"algo": "rsa",
"size": 2048
}
}
EOF
You should see output similar to:
2022/02/01 11:50:39 [INFO] generating a new CA key and certificate from CSR
2022/02/01 11:50:39 [INFO] generate received request
2022/02/01 11:50:39 [INFO] received CSR
2022/02/01 11:50:39 [INFO] generating key: rsa-2048
2022/02/01 11:50:39 [INFO] encoded CSR
2022/02/01 11:50:39 [INFO] signed certificate with serial number 263983151013686720899716354349605500797834580472
This produces a certificate authority key file (
ca.pem
).
Issue a certificate
tls/server-signing-config.json
{
"signing"
: {
"default"
: {
"usages"
: [
"digital signature"
,
,
],
"expiry"
:
"876000h"
,
"ca_constraint"
: {
"is_ca"
:
false
}
}
}
}
Use a
server-signing-config.json
signing configuration and the certificate authority key file
and certificate to sign the certificate request:
kubectl get csr my-svc.my-namespace -o
jsonpath
=
'{.spec.request}'
|
\
base64 --decode |
\
cfssl sign -ca ca.pem -ca-key ca-key.pem -config server-signing-config.json - |
\
cfssljson -bare ca-signed-server
You should see the output similar to:
2022/02/01 11:52:26 [INFO] signed certificate with serial number 576048928624926584381415936700914530534472870337
This produces a signed serving certificate file,
ca-signed-server.pem
.
Upload the signed certificate
Finally, populate the signed certificate in the API object's status:
kubectl get csr my-svc.my-namespace -o json |
\
jq
'.status.certificate = "'
$(
base64 ca-signed-server.pem | tr -d
'\n'
)
'"'
|
\
kubectl replace --raw /apis/certificates.k8s.io/v1/certificatesigningrequests/my-svc.my-namespace/status -f -
Note:
This uses the command line tool
jq
to populate the base64-encoded
content in the
.status.certificate
field.
If you do not have
jq
, you can also save the JSON output to a file, populate this field manually, and
upload the resulting file.
Once the CSR is approved and th

e signed certificate is uploaded, run:
my-svc.my-namespace   20m   example.com/serving   yourname@example.com   &lt;none&gt;              Approved,Issued
Download the certificate and use it
Now, as the requesting user, you can download the issued certificate
and save it to a
server.crt
file by running the following:
jsonpath
=
\
| base64 --decode &gt; server.crt
Now you can populate
and
in a
Secret
that you could later mount into a Pod (for example, to use with a webserver
that serves HTTPS).
kubectl create secret tls server --cert server.crt --key server-key.pem
secret/server created
Finally, you can populate
ca.pem
into a
ConfigMap
and use it as the trust root to verify the serving certificate:
kubectl create configmap example-serving-ca --from-file ca.crt
=
ca.pem
configmap/example-serving-ca created
Approving CertificateSigningRequests
and
commands. However if you intend
to make heavy usage of this API, you might consider writing an automated
certificates controller.
Caution:
The ability to approve CSRs decides who trusts whom within your environment. The
ability to approve CSRs should not be granted broadly or lightly.
You should make sure that you confidently understand both the verification requirements
that fall on the approver
and
the repercussions of issuing a specific certificate
before you grant the
approve
permission.
Whether a machine or a human using kubectl as above, the role of the
approver
is
to verify that the CSR satisfies two requirements:
The subject of the CSR controls the private key used to sign the CSR. This
addresses the threat of a third party masquerading as an authorized subject.
In the above example, this step would be to verify that the pod controls the
private key used to generate the CSR.
The subject of the CSR is authorized to act in the requested context. This
addresses the threat of an undesired subject joining the cluster. In the
above example, this step would be to verify that the pod is allowed to
participate in the requested service.
If and only if these two requirements are met, the approver should approve
the CSR and otherwise should deny the CSR.
For more information on certificate approval and access control, read
the
reference page.
Configuring your cluster to provide signing
This page assumes that a signer is set up to serve the certificates API. The
Kubernetes controller manager provides a default implementation of a signer. To
enable it, pass the
and
parameters to the controller manager with paths to
your Certificate Authority's keypair.
3 - Manual Rotation of CA Certificates
This page shows how to manually rotate the certificate authority (CA) certificates.
minikube
For more information about authentication in Kubernetes, see
.
For more information about best practices for CA certificates, see
Single root CA
.
Rotate the CA certificates manually
Caution:
Make sure to back up your certificate directory along with configuration files and any other necessary files.
This approach assumes operation of the Kubernetes control plane in a HA configuration with multiple API ser

vers.
Graceful termination of the API server is also assumed so clients can cleanly disconnect from one API server and
reconnect to another.
Configurations with a single API server will experience unavailability while the API server is being restarted.
Distribute the new CA certificates and private keys (for example:
ca.crt
,
ca.key
,
,
and
) to all your control plane nodes in the Kubernetes certificates directory.
Update the
flag for the
to include
both old and new CA, then restart the kube-controller-manager.
Any
created after this point will get
Secrets that include both old and new CAs.
Note:
The files specified by the kube-controller-manager flags
and
cannot be CA bundles. If these flags and
point to the same
ca.crt
file which is now a
bundle (includes both old and new CA) you will face an error. To workaround this problem you can copy the new CA
to a separate file and make the flags
and
point to the copy.
Once
ca.crt
is no longer a bundle you can restore the problem flags to point to
ca.crt
and delete the copy.
Issue 1350
for kubeadm tracks an bug with the
kube-controller-manager being unable to accept a CA bundle.
Wait for the controller manager to update
ca.crt
in the service account Secrets to include both old and new CA certificates.
If any Pods are started before new CA is used by API servers, the new Pods get this update and will trust both
old and new CAs.
Restart all pods using in-cluster configurations (for example: kube-proxy, CoreDNS, etc) so they can use the
updated certificate authority data from Secrets that link to ServiceAccounts.
Make sure CoreDNS, kube-proxy and other Pods using in-cluster configurations are working as expected.
Append the both old and new CA to the file against
and
--kubelet-certificate-authority
flag in the
flag in the
Update certificates for user accounts by replacing the content of
and
For information about creating certificates for individual user accounts, see
Configure certificates for user accounts
.
Additionally, update the
section in the kubeconfig files,
respectively with Base64-encoded old and new certificate authority data
Update the
to include
both old and new CA, then restart the cloud-controller-manager.
Note:
If your cluster does not have a cloud-controller-manager, you can skip this step.
Follow the steps below in a rolling fashion.
Restart any other
aggregated API servers
or
webhook handlers to trust the new CA certificates.
Restart the kubelet by update the file against
in kubelet configuration and
in
to use both the old and new CA on all nodes.
If your kubelet is not using client certificate rotation, update
and
in
on all nodes along with the kubelet client certificate file
usually found in
.
Restart API servers with the certificates (
,
and
) signed by new CA.
You can use the existing private keys or new private keys.
If you changed the private keys then update these in the Kubernetes certificates directory as well.
Since the Pods in your cluster trust both old and new CAs, there will be a momentarily disconnection
after which pods' Kubernetes clients reconnect to the new API server.
The new API server uses a certificate signed by the new CA.
Restart the
to use and
trust the new CAs.
Make sure control plane components logs no TLS errors.
Note:
To generate certificates and private keys for your 

cluster using the
openssl
command line tool,
see
Certificates (
openssl
)
.
cfssl
.
Annotate any DaemonSets and Deployments to trigger pod replacement in a safer rolling fashion.
for
namespace in
$(
kubectl get namespace -o
jsonpath
=
'{.items[*].metadata.name}'
)
;
do
for
name in
$(
kubectl get deployments -n
$namespace
-o
jsonpath
=
)
;
do
kubectl patch deployment -n
${
namespace
}
${
name
}
-p
'{"spec":{"template":{"metadata":{"annotations":{"ca-rotation": "1"}}}}}'
;
done
for
name in
$(
kubectl get daemonset -n
-o
jsonpath
=
)
;
do
kubectl patch daemonset -n
${
namespace
}
${
name
}
-p
;
done
done
Note:
To limit the number of concurrent disruptions that your application experiences,
see
configure pod disruption budget
.
Depending on how you use StatefulSets you may also need to perform similar rolling replacement.
If your cluster is using bootstrap tokens to join nodes, update the ConfigMap
in the
namespace with new CA.
base64_encoded_ca
=
"
$(
base64 -w0 /etc/kubernetes/pki/ca.crt
)
"
kubectl get cm/cluster-info --namespace kube-public -o yaml |
\
/bin/sed
"s/\(certificate-authority-data:\).*/\1
${
}
/"
|
\
Verify the cluster functionality.
Check the logs from control plane components, along with the kubelet and the kube-proxy.
Ensure those components are not reporting any TLS errors; see
looking at the logs
Validate logs from any aggregated api servers and pods using in-cluster config.
Once the cluster functionality is successfully verified:
Update all service account tokens to include new CA certificate only.
All pods using an in-cluster kubeconfig will eventually need to be restarted to pick up the new Secret,
so that no Pods are relying on the old cluster CA.
Restart the control plane components by removing the old CA from the kubeconfig files and the files against
,
flags resp.
On each node, restart the kubelet by removing the old CA from file against the
flag
and from the kubelet kubeconfig file. You should carry this out as a rolling update.
If your cluster lets you make this change, you can also roll it out by replacing nodes rather than
reconfiguring them.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator pat

tern
Tasks
Debug Pods
Determine the Reason for P

od Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Cluster Le

vel
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kub

e-apiserver Configuration (v1alpha1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Using Source IP | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

l and Set Up kubectl on macOS
Debug Pods
Get a Shell to 

a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a 

Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Conf

iguration (v1)
Tutorials
Services
Applications running in a Kubernetes cluster find and communicate with each
other, and the outside world, through the Service abstraction. This document
explains what happens to the source IP of packets sent to different types
of Services, and how you can toggle this behavior according to your needs.
Terminology
This document makes use of the following terms:
NAT
network address translation
Source NAT
replacing the source IP on a packet; in this page, that usually means replacing with the IP address of a node.
Destination NAT
replacing the destination IP on a packet; in this page, that usually means replacing with the IP address of a
Pod
VIP
a virtual IP address, such as the one assigned to every
Service
in Kubernetes
a network daemon that orchestrates Service VIP management on every node
minikube
The examples use a small nginx webserver that echoes back the source
IP of requests it receives through an HTTP header. You can create it as follows:
kubectl create deployment source-ip-app --image
=
registry.k8s.io/echoserver:1.4
deployment.apps/source-ip-app created
Expose a simple application through various types of Services
Understand how each Service type handles source IP NAT
Understand the tradeoffs involved in preserving source IP
Source IP for Services with
Type=ClusterIP
Packets sent to ClusterIP from within the cluster are never source NAT'd if
you're running kube-proxy in
iptables mode
,
(the default). You can query the kube-proxy mode by fetching
http://localhost:10249/proxyMode
on the node where kube-proxy is running.
NAME                           STATUS     ROLES    AGE     VERSION
kubernetes-node-6jst   Ready      &lt;none&gt;   2h      v1.13.0
kubernetes-node-cx31   Ready      &lt;none&gt;   2h      v1.13.0
kubernetes-node-jj1t   Ready      &lt;none&gt;   2h      v1.13.0
Get the proxy mode on one of the nodes (kube-proxy listens on port 10249):
# Run this in a shell on the node you want to query.
curl http://localhost:10249/proxyMode
iptables
You can test source IP preservati

on by creating a Service over the source IP app:
kubectl expose deployment source-ip-app --name
=
clusterip --port
=
80
=
8080
service/clusterip exposed
kubectl get svc clusterip
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
clusterip    ClusterIP   10.0.170.92   &lt;none&gt;        80/TCP    51s
And hitting the
ClusterIP
from a pod in the same cluster:
kubectl run busybox -it --image
=
=
Never --rm
Waiting for pod default/busybox to be running, status is Pending, pod ready: false
You can then run a command inside that Pod:
# Run this inside the terminal from "kubectl run"
ip addr
1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
3: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1460 qdisc noqueue
    link/ether 0a:58:0a:f4:03:08 brd ff:ff:ff:ff:ff:ff
    inet 10.244.3.8/24 scope global eth0
    inet6 fe80::188a:84ff:feb0:26a5/64 scope link
…then use
wget
to query the local webserver
# Replace "10.0.170.92" with the IPv4 address of the Service named "clusterip"
wget -qO - 10.0.170.92
CLIENT VALUES:
client_address=10.244.3.8
command=GET
...
The
client_address
is always the client pod's IP address, whether the client pod and server pod are in the same node or in different nodes.
Type=NodePort
Packets sent to Services with
are source NAT'd by default. You can test this by creating a
NodePort
Service:
=
nodeport --port
=
80
=
8080
--type
=
NodePort
service/nodeport exposed
NODEPORT
=
$(
kubectl get -o
jsonpath
=
"{.spec.ports[0].nodePort}"
services nodeport
)
NODES
=
$(
jsonpath
=
'{ $.items[*].status.addresses[?(@.type=="InternalIP")].address }'
)
If you're running on a cloud provider, you may need to open up a firewall-rule
for the
nodes:nodeport
reported above.
Now you can try reaching the Service from outside the cluster through the node
port allocated above.
for
node in
$NODES
;
do
curl -s
$node
:
$NODEPORT
| grep -i client_address;
done
client_address=10.180.1.1
client_address=10.240.0.5
client_address=10.240.0.3
Note that these are not the correct client IPs, they're cluster internal IPs. This is what happens:
Client sends packet to
node2:nodePort
node2
replaces the source IP address (SNAT) in the packet with its own IP address
node2
replaces the destination IP on the packet with the pod IP
packet is routed to node 1, and then to the endpoint
the pod's reply is routed back to node2
the pod's reply is sent back to the client
Visually:
Figure. Source IP Type=NodePort using SNAT
To avoid this, Kubernetes has a feature to
preserve the client source IP
.
If you set
service.spec.externalTrafficPolicy
to the value
Local
,
kube-proxy only proxies proxy requests to local endpoints, and does not
forward traffic to other nodes. This approach preserves the original
source IP address. If there are no local endpoints, packets sent to the
node are dropped, so you can rely on the correct source-ip in any packet
processing rules you might apply a packet that make it through to the
endpoint.
Set the
field as follows:
kubectl patch svc nodeport -p
'{"spec":{"externalTrafficPolicy":"Local"}}'
service/nodeport patched
Now, re-run the test:
for
node in
$NODES
;
do
curl --connect-timeout
1
-s
$node
:
$NODEPORT
done
client_address=198.51.100.79
Note that you only got one reply, with the

right
client IP, from the one node on which the endpoint pod
This is what happens:
client sends packet to
, which doesn't have any endpoints
packet is dropped
node1:nodePort
, which
does
have endpoints
node1 routes packet to endpoint with the correct source IP
Visually:
Figure. Source IP Type=NodePort preserves client source IP address
Type=LoadBalancer
are source NAT'd by default, because all schedulable Kubernetes nodes in the
Ready
state are eligible for load-balanced traffic. So if packets arrive
at a node without an endpoint, the system proxies it to a node
with
an
endpoint, replacing the source IP on the packet with the IP of the node (as
described in the previous section).
You can test this by exposing the source-ip-app through a load balancer:
=
loadbalancer --port
=
80
=
8080
--type
=
service/loadbalancer exposed
Print out the IP addresses of the Service:
kubectl get svc loadbalancer
NAME           TYPE           CLUSTER-IP    EXTERNAL-IP       PORT(S)   AGE
loadbalancer   LoadBalancer   10.0.65.118   203.0.113.140     80/TCP    5m
Next, send a request to this Service's external-ip:
curl 203.0.113.140
...
However, if you're running on Google Kubernetes Engine/GCE, setting the same
field to
Local
forces nodes
without
Service endpoints to remove
themselves from the list of nodes eligible for loadbalanced traffic by
deliberately failing health checks.
Visually:
You can test this by setting the annotation:
kubectl patch svc loadbalancer -p
You should immediately see the
service.spec.healthCheckNodePort
field allocated
by Kubernetes:
kubectl get svc loadbalancer -o yaml | grep -i healthCheckNodePort
:
32122
The
field points to a port on every node
serving the health check at
/healthz
. You can test this:
kubectl get pod -o wide -l
app
=
source-ip-app
NAME                            READY     STATUS    RESTARTS   AGE       IP             NODE
source-ip-app-826191075-qehz4   1/1       Running   0          20h       10.180.1.136   kubernetes-node-6jst
Use
curl
to fetch the
/healthz
endpoint on various nodes:
# Run this locally on a node you choose
curl localhost:32122/healthz
1 Service Endpoints found
On a different node you might get a different result:
No Service Endpoints Found
A controller running on the
is
responsible for allocating the cloud load balancer. The same controller also
allocates HTTP health checks pointing to this port/path on each node. Wait
about 10 seconds for the 2 nodes without endpoints to fail health checks,
then use
curl
to query the IPv4 address of the load balancer:
...
Cross-platform support
Only some cloud providers offer support for source IP preservation through
Services with
.
The cloud provider you're running on might fulfill the request for a loadbalancer
in a few different ways:
With a proxy that terminates the client connection and opens a new connection
to your nodes/endpoints. In such cases the source IP will always be that of the
cloud LB, not that of the client.
With a packet forwarder, such that requests from the client sent to the
loadbalancer VIP end up at the node with the source IP of the client, not
an intermediate proxy.
Load balancers in the first category must use an agreed upon
protocol between the loadbalancer and backend to communicate the true client IP
such as the HTTP
Forwarde

d
or
X-FORWARDED-FOR
headers, or the
proxy protocol
.
Load balancers in the second category can leverage the feature described above
by creating an HTTP health check pointing at the port stored in
the
field on the Service.
Delete the Services:
kubectl delete svc -l
app
=
Delete the Deployment, ReplicaSet and Pod:
kubectl delete deployment source-ip-app
connecting applications via services
Read how to
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Declarative Management of Kubernetes Objects Using Configuration Files | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Kubernetes objects can be created, updated, and deleted by storing multiple
object configuration files in a directory and using
to
recursively create and update those objects as needed. This method
retains writes made to live objects without merging the changes
back into the object configuration files.
also gives you a
preview of what changes
apply
will make.
Install
kubectl
.
minikube
.
The
kubectl
See
Overview
Declarative object configuration requires a firm understanding of
the Kubernetes object definitions and configuration. Read and complete
the following documents if you have not already:
Following are definitions for terms used in this document:
object configuration file / configuration file
: A file that defines the
configuration for a Kubernetes object. This topic shows how to pass configuration
files to
. Configuration files are typically stored in source control, such as Git.
live object configuration / live configuration
: The live configuration
values of an object, as observed by the Kubernetes cluster. These are kept in the Kubernetes
cluster storage, typically etcd.
declarative configuration writer / declarative writer
: A person or software component
that makes updates to a live object. The live writers referred to in this topic make changes
to object configuration files and run
to write the changes.
Use
to create all objects, 

except those that already exist,
defined by configuration files in a specified directory:
kubectl apply -f &lt;directory&gt;/
This sets the
kubectl.kubernetes.io/last-applied-configuration: '{...}'
annotation on each object. The annotation contains the contents of the object
configuration file that was used to create the object.
Note:
Add the
-R
flag to recursively process directories.
Here's an example of an object configuration file:
application/simple_deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Run
to print the object that will be created:
kubectl diff -f https://k8s.io/examples/application/simple_deployment.yaml
Note:
diff
uses
server-side dry-run
,
which needs to be enabled on
.
Since
diff
performs a server-side apply request in dry-run mode,
it requires granting
PATCH
,
CREATE
, and
UPDATE
permissions.
See
Dry-Run Authorization
Create the object using
:
kubectl apply -f https://k8s.io/examples/application/simple_deployment.yaml
Print the live configuration using
:
kubectl get -f https://k8s.io/examples/application/simple_deployment.yaml -o yaml
was written to the live configuration, and it matches the configuration file:
kind
:
metadata
:
:
# ...
# This is the json representation of simple_deployment.yaml
# It was written by kubectl apply when the object was created
:
|
{"apiVersion":"apps/v1","kind":"Deployment",
"metadata":{"annotations":{},"name":"nginx-deployment","namespace":"default"},
"spec":{"minReadySeconds":5,"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.14.2","name":"nginx",
"ports":[{"containerPort":80}]}]}}}}
# ...
spec
:
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
to update all objects defined in a directory, even
if those objects already exist. This approach accomplishes the following:
Sets fields that appear in the configuration file in the live configuration.
Clears fields removed from the configuration file in the live configuration.
kubectl diff -f &lt;directory&gt;/
Note:
Add the
-R
Here's an example configuration file:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
Note:
For purposes of illustration, the preceding command refers to a single
configuration file instead of a directory.
:
kind
:
metadata
:
:
# ...
:
|
"metadata":{"annotations":{},"name":"nginx-deployme

nt","namespace":"default"},
# ...
spec
:
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
Directly update the
replicas
field in the live configuration by using
.
This does not use
:
kubectl scale deployment/nginx-deployment --replicas
=
2
:
replicas
field has been set to 2, and the
annotation does not contain a
replicas
field:
:
apps/v1
kind
:
metadata
:
:
# ...
# note that the annotation does not contain replicas
# because it was not updated through apply
:
|
# ...
spec
:
replicas
:
2
# written by scale
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
Update the
simple_deployment.yaml
configuration file to change the image from
to
, and delete the
field:
application/update_deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
# update the image
ports
:
-
:
80
Apply the changes made to the configuration file:
kubectl diff -f https://k8s.io/examples/application/update_deployment.yaml
kubectl apply -f https://k8s.io/examples/application/update_deployment.yaml
:
kubectl get -f https://k8s.io/examples/application/update_deployment.yaml -o yaml
The output shows the following changes to the live configuration:
The
replicas
field retains the value of 2 set by
.
This is possible because it is omitted from the configuration file.
The
image
field has been updated to
from
.
The
annotation has been updated with the new image.
The
field has been cleared.
The
annotation no longer contains the
field.
:
apps/v1
kind
:
metadata
:
:
# ...
# The annotation contains the updated image to nginx 1.16.1,
# but does not contain the updated replicas to 2
:
|
"spec":{"selector":{"matchLabels":{"app":nginx}},"template":{"metadata":{"labels":{"app":"nginx"}},
"spec":{"containers":[{"image":"nginx:1.16.1","name":"nginx",
# ...
spec
:
replicas
:
2
# Set by `kubectl scale`.  Ignored by `kubectl apply`.
# minReadySeconds cleared by `kubectl apply`
# ...
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# Set by `kubectl apply`
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
Warning:
Mixing
with the imperative object configuration commands
create
and
r

eplace
is not supported. This is because
create
and
replace
do not retain the
that
uses to compute updates.
There are two approaches to delete objects managed by
.
Recommended:
kubectl delete -f &lt;filename&gt;
Manually deleting objects using the imperative command is the recommended
approach, as it is more explicit about what is being deleted, and less likely
to result in the user deleting something unintentionally:
Alternative:
kubectl apply -f &lt;directory/&gt; --prune -l your=label
Only use this if you know what you are doing.
Warning:
kubectl apply --prune
is in alpha, and backwards incompatible
changes might be introduced in subsequent releases.
Warning:
You must be careful when using this command, so that you
do not delete objects unintentionally.
As an alternative to
to identify objects to be deleted after their
configuration files have been removed from the directory. Apply with
--prune
queries the API server for all objects matching a set of labels, and attempts
to match the returned live object configurations against the object
configuration files. If an object matches the query, and it does not have a
configuration file in the directory, and it has a
annotation,
it is deleted.
kubectl apply -f &lt;directory/&gt; --prune -l &lt;labels&gt;
Warning:
Apply with prune should only be run against the root directory
containing the object configuration files. Running against sub-directories
can cause objects to be unintentionally deleted if they are returned
by the label selector query specified with
-l &lt;labels&gt;
and
do not appear in the subdirectory.
You can use
with
-o yaml
to view the configuration of a live object:
kubectl get -f &lt;filename|url&gt; -o yaml
How apply calculates differences and merges changes
Caution:
A
patch
is an update operation that is scoped to specific fields of an object
instead of the entire object. This enables updating only a specific set of fields
on an object without reading the object first.
When
updates the live configuration for an object,
it does so by sending a patch request to the API server. The
patch defines updates scoped to specific fields of the live object
configuration. The
command calculates this patch request
using the configuration file, the live configuration, and the
annotation stored in the live configuration.
Merge patch calculation
The
command writes the contents of the configuration file to the
annotation. This
is used to identify fields that have been removed from the configuration
file and need to be cleared from the live configuration. Here are the steps used
to calculate which fields should be deleted or set:
Calculate the fields to delete. These are the fields present in
and missing from the configuration file.
Calculate the fields to add or set. These are the fields present in the configuration file whose values don't match the live configuration.
Here's an example. Suppose this is the configuration file for a Deployment object:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Also, suppose this is the live configuration for the same Deployment object:
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
"metadata":{"annotations

":{},"name":"nginx-deployment","namespace":"default"},
# ...
spec
:
replicas
:
2
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
Here are the merge calculations that would be performed by
:
Calculate the fields to delete by reading values from
and comparing them to values in the
configuration file.
Clear fields explicitly set to null in the local object configuration file
regardless of whether they appear in the
.
In this example,
appears in the
annotation, but does not appear in the configuration file.
Action:
Clear
from the live configuration.
Calculate the fields to set by reading values from the configuration
file and comparing them to values in the live configuration. In this example,
image
in the configuration file does not match
the value in the live configuration.
Action:
Set the value of
image
in the live configuration.
Set the
annotation to match the value
of the configuration file.
Merge the results from 1, 2, 3 into a single patch request to the API server.
Here is the live configuration that is the result of the merge:
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
selector
:
:
# ...
app
:
nginx
replicas
:
2
# ...
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
How different types of fields are merged
How a particular field in a configuration file is merged with
the live configuration depends on the
type of the field. There are several types of fields:
primitive
: A field of type string, integer, or boolean.
image
and
replicas
are primitive fields.
Action:
Replace.
map
, also called
object
: A field of type map or a complex type that contains subfields. For example,
labels
,
,
spec
and
metadata
are all maps.
Action:
Merge elements or subfields.
list
: A field containing a list of items that can be either primitive types or maps.
,
ports
, and
args
are lists.
Action:
Varies.
When
updates a map or list field, it typically does
not replace the entire field, but instead updates the individual subelements.
For instance, when merging the
spec
on a Deployment, the entire
spec
is
not replaced. Instead the subfields of
spec
, such as
replicas
, are compared
and merged.
Merging changes to primitive fields
Primitive fields are replaced or cleared.
Note:
-
is used for "not applicable" because the value is not used.
Field in object configuration file
Field in live object configuration
Field in last-applied-configuration
Action
Yes
Yes
-
Set live to configuration file value.
Yes
No
-
Set live to local configuration.
No
-
Yes
Clear from live configuration.
No
-
No
Do nothing. Keep live value.
Merging changes to map fields
Fields that represent ma

ps are merged by comparing each of the subfields or elements of the map:
Note:
-
Key in object configuration file
Key in live object configuration
Action
Yes
Yes
-
Compare sub fields values.
Yes
No
-
No
-
Yes
Delete from live configuration.
No
-
No
Merging changes for fields of type list
Merging changes to a list uses one of three strategies:
Replace the list if all its elements are primitives.
Merge individual elements in a list of complex elements.
Merge a list of primitive elements.
The choice of strategy is made on a per-field basis.
Replace the list if all its elements are primitives
Treat the list the same as a primitive field. Replace or delete the
entire list. This preserves ordering.
Example:
Use
to update the
args
field of a Container in a Pod. This sets
args
in the live configuration to the value in the configuration file.
Any
args
elements that had previously been added to the live configuration are lost.
The order of the
args
elements defined in the configuration file is
retained in the live configuration.
# last-applied-configuration value
args
:
[
"a"
,
"b"
]
# configuration file value
args
:
[
"a"
,
"c"
]
# live configuration
args
:
[
"a"
,
"b"
,
"d"
]
# result after merge
args
:
[
"a"
,
"c"
]
Explanation:
The merge used the configuration file value as the new list value.
Merge individual elements of a list of complex elements:
Treat the list as a map, and treat a specific field of each element as a key.
Add, delete, or update individual elements. This does not preserve ordering.
This merge strategy uses a special tag on each field called a
. The
is defined for each field in the Kubernetes source code:
types.go
When merging a list of maps, the field specified as the
for a given element
is used like a map key for that element.
Example:
Use
field of a PodSpec.
This merges the list as though it was a map where each element is keyed
by
name
.
:
-
name
:
nginx
image
:
nginx:1.16
-
name: nginx-helper-a # key
:
nginx-helper-a; will be deleted in result
image
:
helper:1.3
-
name: nginx-helper-b # key
:
nginx-helper-b; will be retained
image
:
:
-
name
:
nginx
image
:
-
name
:
nginx-helper-b
image
:
-
name: nginx-helper-c # key
:
nginx-helper-c; will be added in result
image
:
:
-
name
:
nginx
image
:
-
name
:
nginx-helper-a
image
:
-
name
:
image
:
args
:
[
"run"
]
# Field will be retained
-
name: nginx-helper-d # key
:
nginx-helper-d; will be retained
image
:
:
-
name
:
nginx
image
:
# Element nginx-helper-a was deleted
-
name
:
image
:
args
:
[
"run"
]
# Field was retained
-
name
:
nginx-helper-c
# Element was added
image
:
-
name
:
nginx-helper-d
# Element was ignored
image
:
The container named "nginx-helper-a" was deleted because no container
named "nginx-helper-a" appeared in the configuration file.
The container named "nginx-helper-b" retained the changes to
args
was able to identify
that "nginx-helper-b" in the live configuration was the same
"nginx-helper-b" as in the configuration file, even though their fields
had different values (no
args
in the configuration file). This is
because the
field value (name) was identical in both.
The container named "nginx-helper-c" was added because no container
with that name appeared in the live configuration, but one with
that name appeared in the configuration file.
The container named "nginx-helper-d" was retained because
no element with t

hat name appeared in the last-applied-configuration.
Merge a list of primitive elements
As of Kubernetes 1.5, merging lists of primitive elements is not supported.
Note:
Which of the above strategies is chosen for a given field is controlled by
the
tag in
types.go
If no
is specified for a field of type list, then
the list is replaced.
Default field values
The API server sets certain fields to default values in the live configuration if they are
not specified when the object is created.
Here's a configuration file for a Deployment. The file does not specify
strategy
:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
:
The output shows that the API server set several fields to default values in the live
configuration. These fields were not specified in the configuration file.
:
apps/v1
kind
:
# ...
spec
:
selector
:
:
app
:
nginx
:
5
replicas
:
1
# defaulted by apiserver
strategy
:
:
# defaulted by apiserver - derived from strategy.type
maxSurge
:
1
:
1
type
:
template
:
metadata
:
:
null
labels
:
app
:
nginx
spec
:
:
-
image
:
:
name
:
nginx
ports
:
-
:
80
protocol
:
TCP
resources
:
{}
:
dnsPolicy
:
:
Always
:
{}
:
30
# ...
In a patch request, defaulted fields are not re-defaulted unless they are explicitly cleared
as part of a patch request. This can cause unexpected behavior for
fields that are defaulted based
on the values of other fields. When the other fields are later changed,
the values defaulted from them will not be updated unless they are
explicitly cleared.
For this reason, it is recommended that certain fields defaulted
by the server are explicitly defined in the configuration file, even
if the desired values match the server defaults. This makes it
easier to recognize conflicting values that will not be re-defaulted
by the server.
Example:
# last-applied-configuration
spec
:
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
# configuration file
spec
:
strategy
:
type
:
Recreate
# updated value
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
spec
:
strategy
:
type
:
# defaulted value
:
# defaulted value derived from type
maxSurge
:
1
:
1
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
# result after merge - ERROR!
spec
:
strategy
:
type: Recreate # updated value
:
incompatible with rollingUpdate
rollingUpdate: # defaulted value
:
incompatible with "type: Recreate"
maxSurge
:
1
:
1
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
The user creates a Deployment without defining
.
The server defaults
to
and defaults the
values.
Th

e user changes
to
Recreate
. The
values remain at their defaulted values, though the server expects them to be cleared.
If the
values had been defined initially in the configuration file,
it would have been more clear that they needed to be deleted.
Apply fails because
is not cleared. The
strategy.rollingupdate
field cannot be defined with a
of
Recreate
.
Recommendation: These fields should be explicitly defined in the object configuration file:
Selectors and PodTemplate labels on workloads, such as Deployment, StatefulSet, Job, DaemonSet,
ReplicaSet, and ReplicationController
Deployment rollout strategy
How to clear server-defaulted fields or fields set by other writers
Fields that do not appear in the configuration file can be cleared by
setting their values to
null
and then applying the configuration file.
For fields defaulted by the server, this triggers re-defaulting
the values.
How to change ownership of a field between the configuration file and direct imperative writers
These are the only methods you should use to change an individual object field:
Use
.
Write directly to the live configuration without modifying the configuration file:
for example, use
.
Changing the owner from a direct imperative writer to a configuration file
Add the field to the configuration file. For the field, discontinue direct updates to
the live configuration that do not go through
.
Changing the owner from a configuration file to a direct imperative writer
As of Kubernetes 1.5, changing ownership of a field from a configuration file to
an imperative writer requires manual steps:
Remove the field from the configuration file.
Remove the field from the
annotation on the live object.
Changing management methods
Kubernetes objects should be managed using only one method at a time.
Switching from one method to another is possible, but is a manual process.
Note:
It is OK to use imperative deletion with declarative management.
Migrating from imperative command management to declarative object configuration
Migrating from imperative command management to declarative object
configuration involves several manual steps:
Export the live object to a local configuration file:
kubectl get &lt;kind&gt;/&lt;name&gt; -o yaml &gt; &lt;kind&gt;_&lt;name&gt;.yaml
Manually remove the
status
field from the configuration file.
Note:
This step is optional, as
does not update the status field
even if it is present in the configuration file.
Set the
annotation on the object:
kubectl replace --save-config -f &lt;kind&gt;_&lt;name&gt;.yaml
Change processes to use
for managing the object exclusively.
Migrating from imperative object configuration to declarative object configuration
Set the
Defining controller selectors and PodTemplate labels
Warning:
Updating selectors on controllers is strongly discouraged.
The recommended approach is to define a single, immutable PodTemplate label
used only by the controller selector with no other semantic meaning.
Example:
selector
:
:
controller-selector
:
"apps/v1/deployment/nginx"
template
:
metadata
:
labels
:
:
Feedback
Yes
No
.
r

eport a problem
or
.
Last modified January 04, 2023 at 9:34 AM PST:
[en] update declarative-config.md (3ec0af5808)
Overview
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

 and Set Up kubectl on Linux
Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security

 at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v1alp

ha1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

1)
Reference
kind
metadata
(
)
spec
(
)
.
status
(
)
(string)
(int32)
behavior
behavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respecti

vely). If not set, the default HPAScalingRules for scale up and scale down are used.
HorizontalPodAutoscalerBehavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively).
(string)
(int32)
(string)
(int32)
Stab

ilizationWindowSeconds is the number of seconds for which past recommendations should be considered while scaling up or scaling down. StabilizationWindowSeconds must be greater than or equal to zero and less than or equal to 3600 (one hour). If not set, use the default values: - For scale up: 0 (i.e. no stabilization is done). - For scale down: 300 (i.e. the stabilization window is 300 seconds long).
metrics
type
(int32)
(
Quantity
)
(
Quantity
)
ExternalMetricSource indicates how to scale on a metric not associated with any Kubernetes object (for example length of qu

eue in cloud messaging service, or QPS from loadbalancer running outside of cluster).
(
)
(int32)
(
Quantity
)
(
Quantity
)
(string)
(
)
(int32)
(
Quantity
)
(
Quantity
)
pods refers to a metric describing ea

ch pod in the current scale target (for example, transactions-processed-per-second). The values will be averaged together before being compared to the target value.
(
)
(int32)
(
Quantity
)
(
Quantity
)
(int32)
(
Quantity
)
(
Quantity
)
type
Map: unique values on key type will be ke

pt during a merge
conditions is the set of conditions required for this autoscaler to scale its target, and indicates whether or not those conditions are met.
(Time)
(string)
(string)
(int32)
(
Quantity
)
(
Quantity
)
(int32)
currentAverageUtilization is the current value of the average of the resource metric across all relevan

t pods, represented as a percentage of the requested value of the resource for the pods.
(
Quantity
)
(
Quantity
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
(string)
(
)
(int32)
currentAverageUtilization is the current value of the average of the resource metric across all relevant pods, represented as a percenta

ge of the requested value of the resource for the pods.
(
Quantity
)
(
Quantity
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
(int32)
(Time)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
): strin

g, required
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/

autoscaling/v2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Submitting blog posts and case studies | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Submitting blog posts and case studies
Anyone can write a blog post and submit it for review.
Case studies require extensive review before they're approved.
The Kubernetes Blog
The Kubernetes blog is used by the project to communicate new features, community reports, and any
news that might be relevant to the Kubernetes community. This includes end users and developers.
Most of the blog's content is about things happening in the core project, but we encourage you to
submit about things happening elsewhere in the ecosystem too!
Submit a Post
Blog posts should not be commercial in nature and should consist of original content that applies
broadly to the Kubernetes community. Appropriate blog content includes:
New Kubernetes capabilities
Kubernetes projects updates
Updates from Special Interest Groups
Tutorials and walkthroughs
Thought leadership around Kubernetes
Kubernetes Partner OSS integration
Original content only
Unsuitable content includes:
Vendor product pitches
Partner updates without an integration and customer story
Syndicated posts (language translations ok)
To submit a blog post, follow these steps:
if you have not yet done so.
Have a look at the Markdown format for existing blog posts in the
website repository
.
Write out your blog post in a text editor of your choice.
On the same link from step 2, click the Create new file button. Paste your content into the editor.
Name the file to match the proposed title of the blog post, but don’t put the date in the file name.
The blog reviewers will work with you on the final file name and the date the blog will be published.
When you save the file, GitHub will walk you through the pull request process.
A blog post reviewer will review your submission and work with you on feedback and final details.
When the blog post is approved, the blog will be scheduled for publication.
Guidelines and expectations
Blog posts should not be vendor pitches.
Articles must contain content that applies broadly to the Kubernetes community. For example, a
submission should focus on upstream Kubernetes as opposed to vendor-specific configurations.
Check the
for
what is typically allowed on Kubernetes properties.
Links should primarily be to the official Kubernetes documentation. When using external
references, links should be diverse - For example a submission shouldn't contain only links
back to a single company's blog.
Sometimes this is a delicate balance

. The
blog team
is there to give guidance on whether a post is appropriate for the Kubernetes blog, so don't
hesitate to reach out.
Blog posts are not published on specific dates.
Articles are reviewed by community volunteers. We'll try our best to accommodate specific
timing, but we make no guarantees.
Many core parts of the Kubernetes projects submit blog posts during release windows, delaying
publication times. Consider submitting during a quieter period of the release cycle.
If you are looking for greater coordination on post release dates, coordinating with
CNCF marketing
is a more appropriate choice than submitting a blog post.
Sometimes reviews can get backed up. If you feel your review isn't getting the attention it needs,
you can reach out to the blog team on the
#sig-docs-blog
Slack channel
to ask in real time.
Blog posts should be relevant to Kubernetes users.
Topics related to participation in or results of Kubernetes SIGs activities are always on
topic (see the work in the
Contributor Comms Team
for support on these posts).
The components of Kubernetes are purposely modular, so tools that use existing integration
points like CNI and CSI are on topic.
Posts about other CNCF projects may or may not be on topic. We recommend asking the blog team
before submitting a draft.
Many CNCF projects have their own blog. These are often a better choice for posts. There are
times of major feature or milestone for a CNCF project that users would be interested in
reading on the Kubernetes blog.
Blog posts about contributing to the Kubernetes project should be in the
Kubernetes Contributors site
Blog posts should be original content
The official blog is not for repurposing existing content from a third party as new content.
The
license
for the blog allows
commercial use of the content for commercial purposes, but not the other way around.
Blog posts should aim to be future proof
Given the development velocity of the project, we want evergreen content that won't require
updates to stay accurate for the reader.
It can be a better choice to add a tutorial or update official documentation than to write a
high level overview as a blog post.
Consider concentrating the long technical content as a call to action of the blog post, and
focus on the problem space or why readers should care.
Technical Considerations for submitting a blog post
Submissions need to be in Markdown format to be used by the
Hugo
generator
for the blog. There are
many resources available
on how to use
this technology stack.
We recognize that this requirement makes the process more difficult for less-familiar folks to
submit, and we're constantly looking at solutions to lower this bar. If you have ideas on how to
lower the barrier, please volunteer to help out.
The SIG Docs
blog subproject
manages the review process for blog posts. For more information, see
Submit a post
.
To submit a blog post follow these directions:
with a new blog post.
New blog posts go under the
content/en/blog/_posts
Ensure that your blog post follows the correct naming conventions and the following frontmatter
(metadata) information:
The Markdown file name must follow the format
YYYY-MM-DD-Your-Title-Here.md
2020-02-07-Deploying-External-OpenStack-Cloud-Provider-With-Kubeadm.md
.
Do
not
include dots in the filename. A name like
2020-01-01-whats-new-in-1.19.md
causes
failures during a build.
The front matter must include the following:
---
layout
:
blog
title
:
"Your Title Here"
date
:
YYYY-MM-DD
slug
:
text-for-URL-link-here-no-spaces
---
The first or initial commit message should be a short summary of the work being done and
should stand alone as a description of the blog post. Please note that subsequent edits to
your blog will be squashed into this main commit, so it should be as useful as possible.
Examples of a good commit message:
Add blog post on the foo kubernetes feature
blog: foobar announcement
Examples of bad commit message:
Add blog p

ost
.
initial commit
draft post
The blog team will then review your PR and give you comments on things you might need to fix.
After that the bot will merge your PR and your blog post will be published.
If the content of the blog post contains only content that is not expected to require updates
to stay accurate for the reader, it can be marked as evergreen and exempted from the automatic
warning about outdated content added to blog posts older than one year.
To mark a blog post as evergreen, add this to the front matter:
evergreen
:
true
Examples of content that should not be marked evergreen:
Tutorials
that only apply to specific releases or versions and not all future versions
References to pre-GA APIs or features
Submit a case study
Case studies highlight how organizations are using Kubernetes to solve real-world problems. The
Kubernetes marketing team and members of the
CNCF
collaborate with you on all case studies.
Have a look at the source for the
existing case studies
.
case study guidelines
and submit your request as outlined in the guidelines.
Feedback
Yes
No
.
or
.
Last modified October 19, 2022 at 12:59 PM PST:
Update content/en/docs/contribute/new-content/blogs-case-studies.md (75ef3b4a42)
Home
Blog
Training
Partners
Community
CC BY 4.0

Audit Annotations | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Event
.
Note:
Event
.
Event API
(API group
).
Example:
Value
must
be one of
user
,
namespace
, or
Example:
Value
must
be
,
,
Standard
must
be
latest
See
Example:
level
violated from the PodSecurity enforcement.
See
Example:
See
Auditing
Example:
decision
See
Auditing
Example:
This annotation indic

ates a webhook or aggregated API server
.
.
Example:
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Customizing DNS Service | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Pod(s)
minikube
Your Kubernetes server must be at or later than version v1.12.
.
using the
.
Note:
kube-dns
in the
field.
the legacy
kube-dns
kube-dns
flag.
.
.
If a Pod's
dnsPolicy
is set to
default
But see
.
CoreDNS
.
The CoreDNS server can be configured by ma

intaining a
Corefile
,
ConfigMap
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
errors
health {
}
ready
ttl 30
}
cache 30
loop
reload
}
plugins
errors
health
lameduck
ready
ttl
The
kube-dns
.
The
in the
format
forward
cache
loop
reload
: Allows automatic reload of a changed Corefile.
using the
.
Example
Consul
    errors
    cache 30
}
point the
forward
Corefile
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
errors
health
}
cache 30
loop
reload
}
errors
cache 30
}
Note:
Read
Feedback
Yes
No
.
or
suggest an improvemen

t
.
Last modified December 19, 2022 at 5:07 PM PST:
Update dns-custom-nameservers.md (0c12cb7206)
CoreDNS
Home
Blog
Training
Partners
Community
CC BY 4.0

Create an External Load Balancer | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

pha1)
Tasks
This page shows how to create an external load balancer.
Service
, you have
the option of automatically creating a cloud load balancer. This provides an
externally-accessible IP address that sends traffic to the correct port on your cluster
nodes,
provided your cluster runs in a supported environment and is configured with
the correct cloud load balancer provider package
.
Ingress
in place of Service.
For more information, check the
Ingress
minikube
Your cluster must be running in a cloud or other environment that already has support
for configuring external load balancers.
Create a Service
Create a Service from a manifest
To create an external load balancer, add the following line to your
Service manifest:
type
:
Your manifest might then look like:
:
v1
kind
:
Service
metadata
:
name
:
example-service
spec
:
selector
:
app
:
example
ports
:
-
port
:
8765
:
9376
type
:
Create a Service using kubectl
You can alternatively create the service with the
command and
its
--type=LoadBalancer
flag:
kubectl expose deployment example --port
=
8765
=
9376
\
--name
=
example-service --type
=
This command creates a new Service using the same selectors as the referenced
resource (in the case of the example above, a
named
example
).
For more information, including optional flags, refer to the
reference
.
Finding your IP address
You can find the IP address created for your service by getting the service
information through
kubectl
:
kubectl describe services example-service
which should produce output similar to:
Name:                     example-service
Namespace:                default
Labels:                   app=example
Annotations:              &lt;none&gt;
Selector:                 app=example
Type:                     LoadBalancer
IP Families:              &lt;none&gt;
IP:                       10.3.22.96
IPs:                      10.3.22.96
LoadBalancer In

gress:     192.0.2.89
Port:                     &lt;unset&gt;  8765/TCP
TargetPort:               9376/TCP
NodePort:                 &lt;unset&gt;  30593/TCP
Endpoints:                172.17.0.3:9376
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   &lt;none&gt;
The load balancer's IP address is listed next to
.
Note:
If you are running your service on Minikube, you can find the assigned IP address and port with:
minikube service example-service --url
Preserving the client source IP
By default, the source IP seen in the target container is
not the original
source IP
of the client. To enable preservation of the client IP, the following
fields can be configured in the
.spec
of the Service:
- denotes if this Service desires to route
external traffic to node-local or cluster-wide endpoints. There are two available
options:
Cluster
(default) and
Local
.
Cluster
obscures the client source
IP and may cause a second hop to another node, but should have good overall
load-spreading.
Local
preserves the client source IP and avoids a second hop
for LoadBalancer and NodePort type Services, but risks potentially imbalanced
traffic spreading.
.spec.healthCheckNodePort
- specifies the health check node port
(numeric port number) for the service. If you don't specify
, the service controller allocates a port from your
cluster's NodePort range.
You can configure that range by setting an API server command line option,
--service-node-port-range
. The Service will use the user-specified
value if you specify it, provided that the
Service
type
is set to LoadBalancer and
is set
to
Local
.
Setting
to Local in the Service manifest
activates this feature. For example:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
example
ports
:
-
port
:
8765
:
9376
:
Local
type
:
Caveats and limitations when preserving source IPs
Load balancing services from some cloud providers do not let you configure different weights for each target.
With each target weighted equally in terms of sending traffic to Nodes, external
traffic is not equally load balanced across different Pods. The external load balancer
is unaware of the number of Pods on each node that are used as a target.
Where
NumServicePods &lt;&lt; _NumNodes
or
NumServicePods &gt;&gt; NumNodes
, a fairly close-to-equal
distribution will be seen, even without weights.
Internal pod to pod traffic should behave similar to ClusterIP services, with equal probability across all pods.
Garbage collecting load balancers
In usual case, the correlating load balancer resources in cloud provider should
be cleaned up soon after a LoadBalancer type Service is deleted. But it is known
that there are various corner cases where cloud resources are orphaned after the
associated Service is deleted. Finalizer Protection for Service LoadBalancers was
introduced to prevent this from happening. By using finalizers, a Service resource
will never be deleted until the correlating load balancer resources are also deleted.
Specifically, if a Service has
type
LoadBalancer, the service controller will attach
a finalizer named
service.kubernetes.io/load-balancer-cleanup
.
The finalizer will only be removed after the load balancer resource is cleaned up.
This prevents dangling load balancer resources even in corner cases such as the
service controller crashing.
External load balancer providers
It is important to note that the datapath for this functionality is provided by a load balancer external to the Kubernetes cluster.
When the Service
type
is set to LoadBalancer, Kubernetes provides functionality equivalent to
type
equals ClusterIP to pods
within the cluster and extends it by programming the (external to Kubernetes) load balancer with entries

 for the nodes
hosting the relevant Kubernetes pods. The Kubernetes control plane automates the creation of the external load balancer,
health checks (if needed), and packet filtering rules (if needed). Once the cloud provider allocates an IP address for the load
balancer, the control plane looks up that external IP address and populates it into the Service object.
Follow the
tutorial
Read about
Service
Read about
Ingress
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

ResourceClaim v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Reference
ResourceClaim describes which resources are needed by a resource consumer. Its status tracks whether the resource has been allocated and what the resulting attributes are.
kind
metadata
(
)
spec
(
status
(
)
(string)
(string)
Allocation is set b

y the resource driver once a resource has been allocated successfully. If this is not specified, the resource is not yet allocated.
([]
)
([]
)
(string)
(boolean)
(boolean)
The driver then must deallocate this claim and reset the field together with clearing the Allocation field.
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path

namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query

): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Config and Storage Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
ConfigMap
2:
Secret
3:
Volume
4:
5:
6:
7:
8:
CSIDriver
9:
CSINode
10:
1 - ConfigMap
ConfigMap
: v1
kind
metadata
(
)
data
immutable
(boolean)
: v1
kind
metadata
(
ListMeta
)
More info:
items
([]
ConfigMap
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
fieldManag

er
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
202 (
ConfigMap
update
name
(
in path
namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - Secret
Secret
: v1
kind
: Secret
metadata
(
)
data
immutable
(boolean)
type
(string)
: v1
kind
metadata
(

ListMeta
)
items
([]
Secret
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
202 (
Secret
update
name
(
in path
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue

dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3 - Volume
Volume
name
(boolean)
configMap
(string)
(boolean)
(int32)
([]
KeyToPath
)
secret
(string)
se

cretName is the name of the secret in the pod's namespace to use. More info:
(boolean)
(int32)
([]
KeyToPath
)
(int32)
([]
)
projected
(int32)
(string)
(boolean)
([]
KeyToPath
)
items if unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be pre

sent. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
([]
)
(string)
(boolean)
([]
KeyToPath
)
(string)
(int64)
emptyDir
(string)
(
Quantity
)
sizeLimit is the total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which me

ans that the limit is undefined. More info:
hostPath
(string)
(string)
(int32)
(boolean)
azureDisk
(string)
(string)
(string)
(boolean)
azureFile
(string), requir

ed
(boolean)
cephfs
(string)
(boolean)
(string)
(
)
(string)
cinder
(string)
(boolean)
(
)
csi
(string)
(
)
(boolean)
ephemeral
ephemeral represents a volume t

hat is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed.
where
not
(
(
)
fc
fc.fsType
(string)
fc.lun
(int32)
(boolean)
fc.wwids
driv

er is the name of the driver to use for this volume.
(string)
(boolean)
(
)
flocker
(string)
(string)
(string)
(int32)
(boolean)
iscsi
iscsi.iqn
iscsi.lun
(boolean)
(

boolean)
(string)
(string)
(string)
(boolean)
(
)
nfs
nfs.path
(boolean)
(string)
(string)
(boolean)
quobyte
(string)
(boolean)
(string)
tenant owning the given Quobyte volume in the Backend Use

d with dynamically provisioned Quobyte volumes, value is set by the plugin
(string)
rbd
rbd.image
(string)
(string)
rbd.pool
(string)
(boolean)
(
)
rbd.user
(string)
scaleIO
(
(string)
(string)
(boolean)
(boolean)
(string)
(string)
(string)
storageos
(string)
(boolean)
(
)
secretRef specifies the secret to use for obtaining the

 StorageOS API credentials. If not specified, default values will be attempted.
(string)
(string)
(string)
(string)
(string)
gitRepo
(string)
(string)
path
fieldRef
(
)
mode
(int32)
(
)
KeyToPath
key
path
mode
(int32)
mode is Optional: mode bits used to set permissions on this f

ile. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
4 - PersistentVolumeClaim
: v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
)
resources
Quantity
)
Quantity
)
(string)
(string)
(string)
Beta level
(
)
dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new vol

ume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.
**
(string)
(string)
Quantity
)
capacity
Quantity
)
type
conditions is the current Condition of persistent volume claim. If underlying persis

tent volume is being resized then the Condition will be set to 'ResizeStarted'.
(Time)
(Time)
(string)
(string)
phase
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean

watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pr

etty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5 - PersistentVolume
: v1
kind
metadata
(
)
spec
(
)
status
(
)
capacity
Quantity
)
claimRef
(
)
([]
)
([]
)
(string)
(string)
storageClassName is the name of StorageClass to which this

 persistent volume belongs. Empty value means that this volume does not belong to any StorageClass.
(string)
Local
hostPath
(string)
local
(string)
(string)
(int32)
(boolean)
azureDisk
(string)
(string)
fsType is Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4

" if unspecified.
(string)
(boolean)
azureFile
(boolean)
(string)
cephfs
(string)
(boolean)
(string)
(string)
(string)
(string)
cinder
(string)
(boolean)
SecretReference represents a Secret Reference. It has enough information to retrieve secret in 

any namespace
(string)
(string)
csi
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
nodeStageSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeStageVolume and NodeStageVolum

e and NodeUnstageVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
(string)
(string)
(boolean)
fc
fc.fsType
(string)
fc.lun
(int32)
(boolean)
fc.wwids
(string)
(boolean)
(string)
(string)
flocker
(string)
(string)
gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin. More in

fo:
(string)
(int32)
(boolean)
iscsi
iscsi.iqn
iscsi.lun
(boolean)
(boolean)
(string)
(string)
(string)
(boolean)
(string)
(string)
nfs
Represents an NFS mount that 

lasts the lifetime of a pod. NFS volumes do not support ownership management or SELinux relabeling.
nfs.path
(boolean)
(string)
(string)
(boolean)
quobyte
(string)
(boolean)
(string)
(string)
rbd
rbd.image
(string)
(string)
rbd.pool
(string)
https://examples.k8s.io/volumes/rbd/README.md#how-to-use-

it
(boolean)
(string)
(string)
rbd.user
(string)
scaleIO
(string)
(string)
(string)
(string)
(boolean)
(boolean)
(string)
(string)
(string)
storageos
(string)
(boolean)
(
)
(string)
(string)
volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the Pod's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within Stora

geOS. Namespaces that do not pre-exist within StorageOS will be created.
(string)
(string)
(string)
message
(string)
phase
(string)
reason
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
name

 of the PersistentVolume
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
6 - StorageClass
kind
metadata
(
)
(boolean)
allowedTopo

logies.matchLabelExpressions
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
): s

tring, required
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7 - VolumeAttachment
kind
metadata
(
)
spec
(
status
(
)
attacher
nodeName
source
(
)
(string)
attached
(string)
(Time)
attachmentMe

tadata
(string)
(Time)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the 

specified VolumeAttachment
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8 - CSIDriver
CSIDriver
kind
metadata
(
)
spec
(
(boolean)
attachRequired indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume() method), and that the Kubernetes attach detach controller should call the attach volume interface which checks the volumeattachment status and waits until the volume is attached before proceeding to mounting. The CSI external-attacher coordinates with CSI volume driver and updates the volumeattachment status when the attach operation is complete. If the CSIDriverRegistry feature gate is enabled and the value is specified to fa

lse, the attach operation will be skipped. Otherwise the attach operation will be called.
(string)
(boolean)
(boolean)
(boolean)
-o context
(boolean)
This field was immutable in Kuberne

tes &lt;= 1.22 and now is mutable.
},
...
}
(int64)
kind
metadata
(
ListMeta
)
items
([]
CSIDriver
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
202 (
CSIDriver
update
name
(
in path
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in quer

y
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
CSIDriver
): OK
202 (
CSIDriver
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
9 - CSINode
CSINode
kind
: CSINode
metadata
(
)
spec
(
drivers
name
VolumeNodeResources is a set of r

esource limits for scheduling of volumes.
(int32)
kind
metadata
(
ListMeta
)
items
([]
CSINode
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
202 (
CSINode
update
name
(
in path
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
CSINode
): OK
202

 (
CSINode
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
10 - CSIStorageCapacity
kind
metadata
(
)
More info:
capacity
(
Quantity
)
(
Quantity
)
(
)
NodeTopology defines wh

ich nodes have access to the storage for which capacity was reported. If not set, the storage is not accessible from any node in the cluster. If empty, the storage is accessible from all nodes. This field is immutable.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in que

ry
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

Viewing Site Analytics | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Insta

ll and Set Up kubectl on Linux
Debug Pods
Debug Init Containe

rs
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standar

ds at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v1a

lpha1)
This page contains information about the kubernetes.io analytics dashboard.
View the dashboard
.
This dashboard is built using Google Data Studio and shows information collected on kubernetes.io using Google Analytics.
Using the dashboard
By default, the dashboard shows all collected analytics for the past 30 days. Use the date selector to see data from a different date range. Other filtering options allow you to view data based on user location, the device used to access the site, the translation of the docs used, and more.
If you notice an issue with this dashboard, or would like to request any improvements, please
open an issue
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

PKI certificates and requirements | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

pha1)
Kubernetes requires PKI certificates for authentication over TLS.
If you install Kubernetes with
kubeadm
, the certificates
that your cluster requires are automatically generated.
You can also generate your own certificates -- for example, to keep your private keys more secure
by not storing them on the API server.
This page explains the certificates that your cluster requires.
How certificates are used by your cluster
Kubernetes requires PKI for the following operations:
Client certificates for the kubelet to authenticate to the API server
Kubelet
server certificates
for the API server to talk to the kubelets
Server certificate for the API server endpoint
Client certificates for administrators of the cluster to authenticate to the API server
Client certificates for the API server to talk to the kubelets
Client certificate for the API server to talk to etcd
Client certificate/kubeconfig for the controller manager to talk to the API server
Client certificate/kubeconfig for the scheduler to talk to the API server.
Client and server certificates for the
front-proxy
Note:
certificates are required only if you run kube-proxy to support
an extension API server
.
etcd also implements mutual TLS to authenticate clients and peers.
Where certificates are stored
If you install Kubernetes with kubeadm, most certificates are stored in
.
All paths in this documentation are relative to that directory, with the exception of user account
certificates which kubeadm places in
.
Configure certificates manually
If you don't want kubeadm to generate the required certificates, you can create them using a
single root CA or by providing all certificates. See
for details on creating your own certificate authority. See
for more on managing certificates.
You can create a single root CA, controlled by an administrator. This root CA can then create
multiple intermediate CAs, and delegate all further creation to Kubernetes itself.
Required CAs:
path
Default CN
ca.crt,key
kubernetes-ca
Kubernetes general CA
etcd/ca.crt,key
etcd-ca
For all etcd-related functions
front-proxy-ca.crt,key
kubernetes-front-proxy-ca
For the
front-end proxy
On top of the above CAs, it is also necessary to get a public/private key pair for service account
management,
sa.key
and
sa.pub
.
The following example illustrates the CA key and certificate files shown

 in the previous table:
/etc/kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key
All certificates
If you don't wish to copy the CA private keys to your cluster, you can generate all certificates yourself.
Required certificates:
Default CN
Parent CA
O (in Subject)
kind
hosts (SAN)
kube-etcd
etcd-ca
server, client
&lt;hostname&gt;
,
&lt;Host_IP&gt;
,
localhost
,
127.0.0.1
kube-etcd-peer
etcd-ca
,
,
localhost
,
127.0.0.1
kube-etcd-healthcheck-client
etcd-ca
client
kube-apiserver-etcd-client
etcd-ca
client
server
,
,
&lt;advertise_IP&gt;
,
[1]
kube-apiserver-kubelet-client
client
client
[1]: any other IP or DNS name you contact your cluster on (as used by
kubeadm
the load balancer stable IP and/or DNS name,
,
,
,
,
)
where
kind
maps to one or more of the x509 key usage, which is also documented in the
.spec.usages
of a
type:
kind
Key usage
server
digital signature, key encipherment, server auth
client
digital signature, key encipherment, client auth
Note:
Hosts/SAN listed above are the recommended ones for getting a working cluster; if required by a
specific setup, it is possible to add additional SANs on all the server certificates.
Note:
For kubeadm users only:
The scenario where you are copying to your cluster CA certificates without private keys is
referred as external CA in the kubeadm documentation.
If you are comparing the above list with a kubeadm generated PKI, please be aware that
kube-etcd
,
and
certificates are not generated
in case of external etcd.
Certificate paths
Certificates should be placed in a recommended path (as used by
kubeadm
).
Paths should be specified using the given argument regardless of location.
Default CN
recommended key path
recommended cert path
command
key argument
cert argument
etcd-ca
etcd/ca.key
etcd/ca.crt
--etcd-cafile
apiserver-etcd-client.key
apiserver-etcd-client.crt
--etcd-keyfile
--etcd-certfile
ca.key
ca.crt
ca.key
ca.crt
--client-ca-file, --root-ca-file, --cluster-signing-cert-file
etcd-ca
etcd
--trusted-ca-file, --peer-trusted-ca-file
kube-etcd
etcd/server.key
etcd/server.crt
etcd
--key-file
--cert-file
etcd/peer.key
etcd/peer.crt
etcd
--peer-key-file
--peer-cert-file
etcd-ca
etcdctl
--cacert
etcd/healthcheck-client.key
etcd/healthcheck-client.crt
etcdctl
--key
--cert
Same considerations apply for the service account key pair:
private key path
public key path
command
argument
sa.key
sa.pub
The following example illustrates the file paths
from the previous tables
you need to provide if you are generating all of your own keys and certificates:
/etc/

kubernetes/pki/etcd/ca.key
/etc/kubernetes/pki/apiserver-etcd-client.key
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver.key
/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver-kubelet-client.key
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/kubernetes/pki/front-proxy-client.key
/etc/kubernetes/pki/front-proxy-client.crt
/etc/kubernetes/pki/etcd/peer.key
/etc/kubernetes/pki/etcd/peer.crt
/etc/kubernetes/pki/etcd/healthcheck-client.key
/etc/kubernetes/pki/etcd/healthcheck-client.crt
/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub
You must manually configure these administrator account and service accounts:
filename
credential name
Default CN
default-admin
default-auth
(see note)
default-controller-manager
Note:
for
must
match precisely the value of the node name
provided by the kubelet as it registers with the apiserver. For further details, read the
.
For each config, generate an x509 cert/key pair with the given CN and O.
Run
kubectl
as follows for each config:
KUBECONFIG=&lt;filename&gt; kubectl config set-cluster default-cluster --server=https://&lt;host ip&gt;:6443 --certificate-authority &lt;path-to-kubernetes-ca&gt; --embed-certs
KUBECONFIG=&lt;filename&gt; kubectl config set-credentials &lt;credential-name&gt; --client-key &lt;path-to-key&gt;.pem --client-certificate &lt;path-to-cert&gt;.pem --embed-certs
KUBECONFIG=&lt;filename&gt; kubectl config set-context default-system --cluster default-cluster --user &lt;credential-name&gt;
KUBECONFIG=&lt;filename&gt; kubectl config use-context default-system
These files are used as follows:
filename
command
comment
kubectl
Configures administrator user for the cluster
kubelet
One required for each node in the cluster.
Must be added to manifest in
manifests/kube-controller-manager.yaml
manifests/kube-scheduler.yaml
The following files illustrate full paths to the files listed in the previous table:
/etc/kubernetes/kubelet.conf
Feedback
Yes
No
.
or
.
Last modified December 17, 2022 at 5:21 PM PST:
Normalize the markdown for the certificates.md page (227cb354a9)
Home
Blog
Training
Partners
Community
CC BY 4.0

Concepts | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Concepts
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Device Pl

ugins
Tasks
Trou

bleshooting Applications
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configuring a Java Microse

rvice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

rver Admission (v1)
Concepts
The Concepts section helps you learn about the parts of the Kubernetes system and the abstractions Kubernetes uses to represent your
cluster
, and helps you obtain a deeper understanding of how Kubernetes works.
Overview
The architectural concepts behind Kubernetes.
Technology for packaging an application along with its runtime dependencies.
Workloads
Understand Pods, the smallest deployable compute object in Kubernetes, and the higher-level abstractions that help you to run them.
Concepts and resources behind networking in Kubernetes.
Storage
Ways to provide both long-term and temporary storage to Pods in your cluster.
Resources that Kubernetes provides for configuring Pods.
Security
Concepts for keeping your cloud-native workload secure.
Policies
Policies you can configure that apply to groups of resources.
In Kubernetes, scheduling refers to making sure that Pods are matched to Nodes so that the kubelet can run them. Preemption is the process of terminating Pods with lower Priority so that Pods with higher Priority can schedule on Nodes. Eviction is the process of proactively terminating one or more Pods on resource-starved Nodes.
Different ways to change the behavior of your Kubernetes cluster.
Feedback
Yes
No
.
or
.
Last modified June 22, 2020 at 11:01 PM PST:
Add descriptions to Concept sections (3ff7312cff)
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and use

s trademarks. For a list of trademarks of The Linux Foundation, please see our

Validate node setup | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Node Conformance Test
Node conformance test
is a containerized test framework that provides a system
verification and functionality test for a node. The test validates whether the
node meets the minimum requirements for Kubernetes; a node that passes the test
is qualified to join a Kubernetes cluster.
Node Prerequisite
To run node conformance test, a node must satisfy the same prerequisites as a
standard Kubernetes node. At a minimum, the node should have the following
daemons installed:
Container Runtime (Docker)
Kubelet
Running Node Conformance Test
To run the node conformance test, perform the following steps:
Work out the value of the
option for the kubelet; for example:
--kubeconfig=/var/lib/kubelet/config.yaml
.
Because the test framework starts a local control plane to test the kubelet,
use
as the URL of the API server.
There are some other kubelet command line parameters you may want to use:
: If you are using
--cloud-provider=gce
, you should
remove the flag to run the test.
Run the node conformance test with command:
# $CONFIG_DIR is the pod manifest path of your Kubelet.
# $LOG_DIR is the test output path.
sudo docker run -it --rm --privileged --net
=
host
\
-v /:/rootfs -v
$CONFIG_DIR
:
-v
$LOG_DIR
:/var/result
\
registry.k8s.io/node-test:0.2
Running Node Conformance Test for Other Architectures
Kubernetes also provides node conformance test docker images for other
architectures:
Arch
Image
amd64
node-test-amd64
arm
node-test-arm
arm64
node-test-arm64
Running Selected Test
To run specific tests, overwrite the environment variable
FOCUS
with the
regular expression of tests you want to run.
=
host
\
-v /:/rootfs:ro -v
:
-v
$LOG_DIR
\
-e
FOCUS
=
MirrorPod
\
# Only run MirrorPod test
To skip specific tests, overwrite the environment variable
SKIP
with the
regular expression of tests you want to skip.
=
host
\
:
-v
$LOG_DIR
\
-e
SKIP
=
MirrorPod
\
# Run all conformance tests but skip MirrorPod test
Node conformance test is a containerized version of
node e2e test
.
By default, it runs all conformance tests.
Theoretically, you can run any node e2e test if you configure the container and
mount required volumes properly. But
it is strongly recommended to only run conformance

test
, because it requires much more complex configuration to run non-conformance test.
Caveats
The test leaves some docker images on the node, including the node conformance
test image and images of containers used in the functionality
test.
The test leaves dead containers on the node. These containers are created
during the functionality test.
Feedback
Yes
No
.
or
.
Caveats
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes API Concepts | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
GET).
watches
.
online,
A
pods
,
,
services
)
All resource types have a concrete representation (their object schema) which is called a
kind
resource
object
objects
:
virtual
in
to the
eviction
).
name
Within a
namespace
API verbs
and DELETE. Kubernetes also uses its own verbs, which are often written lowercase to dis

tinguish
list
of
a
get
?watch
watch
and not a
get
(see
create
or
update
update
patch
; the
patch
is PATCH.
) or to a
/api
instead of
/apis
Examples:
.
see the
watch
. Clients
can send a
list
or a
get
watch
request.
use that
watch
watch
create
,
delete
,
and
update
to the
watch
watch
If a client
watch
watch
from
get
/
list

---
200 OK
{
}
(such as
create
,
delete
,
apply
or
update
test
(served as
---
200 OK

{
}
{
}
...
watch
410 Gone
get
or
list
watch
from the
list
-then-
watch
Reflector
package.)
BOOKMARK
to a given
BOOKMARK
---
200 OK
{
}
...
{
}
BOOKMARK
events by setting the
watch
BOOKMARK
, the
list
chunks
limit
and
continue
continue
list
metadata
field. A client should specify the maximum

 results they
limit
limit
continue
continue
chunk
continue
Like a
watch
continue
410 Gone
limit
---
200 OK
{
...
},
}
---
200 OK
{
...
},
}
---
200 OK
{
...
},
}
10245
list
continue
watch
operation
is the number of subsequent items in the collection that are not
list
selectors
If the
list
list
is
a
List
appended.
list
has
kind
set to
{
"kind":
,
"v1"
,
{
"2947301"
},
"items":
[
{
{
"name":
,
"default"
,
...
{
"name":
,
,
...
PodList
,
,
and
NodeList
kubectl
, re

present the Kubernetes collection
kubectl
multiple
list
kubectl
kind: List
:
v1
kind
:
List
metadata
:
:
""
selfLink
:
""
items
:
-
:
v1
kind
:
Service
metadata
:
:
labels
:
component
:
apiserver
provider
:
name
:
namespace
:
default
...
-
:
v1
kind
:
Service
metadata
:
:
:
"9153"
:
"true"
:
labels
:
k8s-app
:
kube-dns
:
"true"
:
CoreDNS
name
:
kube-dns
namespace
:
Note:
kind
named
List
.
kind: List
kind: List
kubectl
an
Accept
with a
GET
type.
---
200 OK
{
...
...
]
}
For API resource types that do not have a custom Table definition known to the control
name
and
fields.
---
200 OK
...
{
...
{
...
},
{
...
}
]
}
Accept
Accept
GET
the server for 

a
PUT
or
POST
---
200 OK
in JSON.
---
200 OK
{
...
}
.
Accept
// contentType is the serialization method used to serialize 'raw'. Unspecified means application/vnd.kubernetes.protobuf and is usually
}
}
Note:
that does
When you
delete
removal
{
"kind":
,
"v1"
,
{
{
,
},
nil,
}
}
delete
Once the
Order is
not
.
The
compo

nent responsible for a finalizer later in the list, resulting in a deadlock.
get
,
create
,
apply
,
update
,
patch
,
delete
and
proxy
list
and
watch
PUT
request).
CRDs
fields via
).
POST
,
PUT
, and
PATCH
), field
validation are
Ignore
,
Warn
, and
Strict
.
Note:
Ignore
Warn
Warning:
.
Strict
detected.
kubectl
Warn
The
kubectl
flag to set the level of field 

validation.
ignore
,
warn
,
and
strict
true
strict
) and
false
ignore
,
Dry-run
POST
,
PUT
,
PATCH
, and
DELETE
dry run
dryRun
?dryRun
or
All
PATCH
field to
None
.
Note:
If a webhook actually does have side effects, then the
DryRun
:
name
: if
is set,
name
/
UID
:
For the
Service
patch
to perfo

rm that
patch
RBAC
rules
:
-
apiGroups
:
[
"apps"
]
resources
:
[
]
verbs
:
[
"patch"
]
See
.
Apply
and
Update
.
apply
.
See
watch
list
- The
- The
list
The
get
,
list
, and
watch
list
requests.
. If you set
get
and
list
For
get
and
list
are:
get:
Any
list:
parameter
on
list
and
, the
.
on a
list
and
a known
and
list
and
:
unset
Any
unset
Any
Exact
unset
Invalid
Invalid
Exact
Invalid
Invalid
Exact
Inva

lid
Any
Invalid
Any
Note:
get
and
list
Any
may be
list
of any
Exact
is
list
list
continue
tokens
list
list
.
Note:
When you
list
metadata
well as
When using
newer
.
When using
and
limit
matches
limit
set.
watch
For
watch
watch:
watch
Caution:
Start a
watch
watch
watch
watch
all resource instances that exi

st at the starting resource version. All following
watch
Start a
watch
the
watch
watch
Start a
watch
watch
410 (Gone)
410 (Gone)
410 (Gone)
410 Gone
list
or
get
watch
Feedback
Yes
No
.
or
.
Last modified December 26, 2022 at 11:58 AM PST:
Fixed spelling mistake (7549365878)
API verbs
Dry-run
get
and
list
watch
Home
Blog
Training
Partners
Community
CC BY 4.0

Hello Minikube | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Network

 Plugins
Tasks
Monitoring, Logging, 

and Debugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configur

ing a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Image Policy API (v1

alpha1)
Tutorials
This tutorial shows you how to run a sample app
on Kubernetes using minikube and Katacoda.
Katacoda provides a free, in-browser Kubernetes environment.
Note:
You can also follow this tutorial if you've installed minikube locally.
See
minikube start
for installation instructions.
Deploy a sample application to minikube.
Run the app.
View application logs.
This tutorial provides a container image that uses NGINX to echo back all the requests.
Create a minikube cluster
Click
Note:
If you installed minikube locally, run
. Before you run
minikube dashboard
, you should open a new terminal, start
there, and then switch back to the main terminal.
Open the Kubernetes dashboard in a browser:
Katacoda environment only: At the top of the terminal pane, click the plus sign, and then click
Select port to view on Host 1
.
Katacoda environment only: Type
30000
, and then click
Display Port
.
Note:
The
dashboard
command enables the dashboard add-on and opens the proxy in the default web browser.
You can create Kubernetes resources on the dashboard such as Deployment and Service.
If you are running in an environment as root, see
Open Dashboard with URL
.
By default, the dashboard is only accessible from within the internal Kubernetes virtual network.
The
dashboard
command creates a temporary proxy to make the dashboard accessible from outside the Kubernetes virtual network.
To stop the proxy, run
Ctrl+C
to exit the process.
After the command exits, the dashboard remains running in the Kubernetes cluster.
You can run the
dashboard
command again to create another proxy to access the dashboard.
If you don't want to open a web browser, run the dashboard command with the
--url
flag to emit a URL:
minikube dashboard --url
Create a Deployment
Pod
is a group of one or more Containers,
tied together for the purposes of administration and networking. The Pod in this
tutorial has only one Container. A Kubernetes
checks on the health of your
Pod and restarts the Pod's Container if it terminates. Deployments are the
recommended way to manage the creation and scaling of Pods.
Use the
command to create a Deployment that manages a Pod. The
Pod runs a Container based on the provided Docker image.
k

ubectl create deployment hello-node --image
=
registry.k8s.io/e2e-test-images/agnhost:2.39 -- /agnhost netexec --http-port
=
8080
View the Deployment:
kubectl get deployments
hello-node   1/1     1            1           1m
View the Pod:
hello-node-5f76cf6ccf-br9b5   1/1       Running   0          1m
View cluster events:
View the
kubectl
Note:
kubectl
commands, see the
.
By default, the Pod is only accessible by its internal IP address within the
Kubernetes cluster. To make the
hello-node
Container accessible from outside the
Kubernetes virtual network, you have to expose the Pod as a
Service
.
Expose the Pod to the public internet using the
command:
kubectl expose deployment hello-node --type
=
LoadBalancer --port
=
8080
The
flag indicates that you want to expose your Service
outside of the cluster.
The application code inside the image
registry.k8s.io/echoserver
only listens on TCP port 8080. If you used
to expose a different port, clients could not connect to that other port.
View the Service you created:
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
hello-node   LoadBalancer   10.108.144.78   &lt;pending&gt;     8080:30369/TCP   21s
kubernetes   ClusterIP      10.96.0.1       &lt;none&gt;        443/TCP          23m
On cloud providers that support load balancers,
an external IP address would be provisioned to access the Service. On minikube,
the
type makes the Service accessible through the
minikube service
command.
minikube service hello-node
Katacoda environment only: Click the plus sign, and then click
.
Katacoda environment only: Note the 5-digit port number displayed opposite to
8080
in services output. This port number is randomly generated and it can be different for you. Type your number in the port number text box, then click Display Port. Using the example from earlier, you would type
30369
.
This opens up a browser window that serves your app and shows the app's response.
Enable addons
The minikube tool includes a set of built-in
addons
that can be enabled, disabled and opened in the local Kubernetes environment.
List the currently supported addons:
minikube addons list
addon-manager: enabled
dashboard: enabled
default-storageclass: enabled
efk: disabled
freshpod: disabled
gvisor: disabled
helm-tiller: disabled
ingress: disabled
ingress-dns: disabled
logviewer: disabled
metrics-server: disabled
nvidia-driver-installer: disabled
nvidia-gpu-device-plugin: disabled
registry: disabled
registry-creds: disabled
storage-provisioner: enabled
storage-provisioner-gluster: disabled
Enable an addon, for example,
:
enable
The 'metrics-server' addon is enabled
View the Pod and Service you created:
kubectl get pod,svc -n kube-system
NAME                                        READY     STATUS    RESTARTS   AGE
pod/coredns-5644d7b6d9-mh9ll                1/1       Running   0          34m
pod/coredns-5644d7b6d9-pqd2t                1/1       Running   0          34m
pod/metrics-server-67fb648c5                1/1       Running   0          26s
pod/etcd-minikube                           1/1       Running   0          34m
pod/influxdb-grafana-b29w8                  2/2       Running   0          26s
pod/kube-addon-manager-minikube             1/1       Running   0          34m
pod/kube-apiserver-minikube                 1/1       Running   0          34m
pod/kube-controller-manager-minikube        1/1      

 Running   0          34m
pod/kube-proxy-rnlps                        1/1       Running   0          34m
pod/kube-scheduler-minikube                 1/1       Running   0          34m
pod/storage-provisioner                     1/1       Running   0          34m

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/metrics-server         ClusterIP   10.96.241.45    &lt;none&gt;        80/TCP              26s
service/kube-dns               ClusterIP   10.96.0.10      &lt;none&gt;        53/UDP,53/TCP       34m
service/monitoring-grafana     NodePort    10.99.24.54     &lt;none&gt;        80:30002/TCP        26s
service/monitoring-influxdb    ClusterIP   10.111.169.94   &lt;none&gt;        8083/TCP,8086/TCP   26s
Disable
:
minikube addons disable metrics-server
metrics-server was successfully disabled
Clean up
Now you can clean up the resources you created in your cluster:
kubectl delete service hello-node
kubectl delete deployment hello-node
Optionally, stop the Minikube virtual machine (VM):
minikube stop
Optionally, delete the Minikube VM:
minikube delete
Deployment objects
.
Deploying applications
.
Service objects
.
Feedback
Yes
No
.
or
.
Last modified October 19, 2022 at 3:40 PM PST:
Update echoserver image in tutorial for multiarch (25d48ec80a)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Expose Your App Publicly | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Using a Service to Expose Your App
Pods
lifecycle
selector
type
ClusterIP
NodePort
CNAME
record with its value. No proxying of any kind is set up. This type requires v1.7 or higher of
kube-dns
.
selector
selector
.
Summary
A Service routes traffic across a set of Pods. Services are the abstraction that allows pods to die and replicate in Kubernetes without impacting your application. 

Discovery and routing among dependent Pods (such as the frontend and backend components in an application) are handled by Kubernetes Services.
›
2 - Interactive Tutorial - Exposing Your App
&lt; Return to Module 3
Home
Continue to Module 5 &gt;
Home
Blog
Training
Partners
Community
CC BY 4.0

StatefulSet Basics | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tutorials
This tutorial provides an introduction to managing applications with
.
It demonstrates how to create, delete, scale, and update the Pods of StatefulSets.
Before you begin this tutorial, you should familiarize yourself with the
following Kubernetes concepts:
Pods
Cluster DNS
Headless Services
PersistentVolume Provisioning
The
kubectl
command line tool
Note:
This tutorial assumes that your cluster is configured to dynamically provision
PersistentVolumes. If your cluster is not configured to do so, you
will have to manually provision two 1 GiB volumes prior to starting this
tutorial.
StatefulSets are intended to be used with stateful applications and distributed
systems. However, the administration of stateful applications and
distributed systems on Kubernetes is a broad, complex topic. In order to
demonstrate the basic features of a StatefulSet, and not to conflate the former
topic with the latter, you will deploy a simple web application using a StatefulSet.
After this tutorial, you will be familiar with the following.
How to create a StatefulSet
How a StatefulSet manages its Pods
How to delete a StatefulSet
How to scale a StatefulSet
How to update a StatefulSet's Pods
Creating a StatefulSet
Begin by creating a StatefulSet using the example below. It is similar to the
example presented in the
concept.
It creates a
headless Service
,
nginx
, to publish the IP addresses of Pods in the StatefulSet,
web
.
application/web/web.yaml
:
v1
kind
:
Service
metadata
:
name
:
nginx
labels
:
app
:
nginx
spec
:
ports
:
-
port
:
80
name
:
web
clusterIP
:
None
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
web
spec
:
:
"nginx"
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
registry.k8s.io/nginx-slim:0.8
ports
:
-
:
80
name
:
web
:
-
name
:
www
mountPath
:
:
-
metadata
:
name
:
www
spec
:
:
[
"ReadWriteOnce"
]
resources
:
requests
:
storage
:
1Gi
Download the example above, and save it to a file named
web.yaml
You will need to use two terminal windows. In the first terminal, use
to watch the creation
of the StatefulSet's Pods.
kubectl get pods -w -l
app
=
nginx
In the second terminal, use
to create the
headless Service and StatefulSet defined in

web.yaml
.
kubectl apply -f web.yaml
service/nginx created
statefulset.apps/web created
The command above creates two Pods, each running an
NGINX
webserver. Get the
nginx
Service...
kubectl get service nginx
NAME      TYPE         CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
nginx     ClusterIP    None         &lt;none&gt;        80/TCP    12s
...then get the
web
StatefulSet, to verify that both were created successfully:
kubectl get statefulset web
NAME      DESIRED   CURRENT   AGE
web       2         1         20s
Ordered Pod Creation
For a StatefulSet with
n
replicas, when Pods are being deployed, they are
created sequentially, ordered from
{0..n-1}
. Examine the output of the
command in the first terminal. Eventually, the output will
look like the example below.
app
=
nginx
web-0     0/1       Pending   0          0s
web-0     0/1       Pending   0         0s
web-0     0/1       ContainerCreating   0         0s
web-0     1/1       Running   0         19s
web-1     0/1       Pending   0         0s
web-1     0/1       ContainerCreating   0         0s
web-1     1/1       Running   0         18s
web-1
Pod is not launched until the
web-0
Pod is
Running
(see
Pod Phase
)
and
Ready
(see
type
in
Pod Conditions
).
Note:
To configure the integer ordinal assigned to each Pod in a StatefulSet, see
.
Pods in a StatefulSet
Pods in a StatefulSet have a unique ordinal index and a stable network identity.
Examining the Pod's Ordinal Index
Get the StatefulSet's Pods:
app
=
nginx
web-0     1/1       Running   0          1m
web-1     1/1       Running   0          1m
As mentioned in the
concept, the Pods in a StatefulSet have a sticky, unique identity. This identity
is based on a unique ordinal index that is assigned to each Pod by the
.
The Pods' names take the form
&lt;statefulset name&gt;-&lt;ordinal index&gt;
.
Since the
web
StatefulSet has two replicas, it creates two Pods,
web-0
and
web-1
.
Using Stable Network Identities
Each Pod has a stable hostname based on its ordinal index. Use
to execute the
hostname
command in each Pod:
for
i in
0
1;
do
kubectl
exec
"web-
$i
"
-- sh -c
'hostname'
;
done
web-0
web-1
Use
to execute
a container that provides the
nslookup
command from the
dnsutils
package.
Using
nslookup
on the Pods' hostnames, you can examine their in-cluster DNS
addresses:
kubectl run -i --tty --image busybox:1.28 dns-test --restart
=
Never --rm
which starts a new shell. In that new shell, run:
# Run this in the dns-test container shell
nslookup web-0.nginx

Name:      web-0.nginx
Address 1: 10.244.1.6

nslookup web-1.nginx

Name:      web-1.nginx
Address 1: 10.244.2.6
(and now exit the container shell:
exit
)
The CNAME of the headless service points to SRV records (one for each Pod that
is Running and Ready). The SRV records point to A record entries that
contain the Pods' IP addresses.
In one terminal, watch the StatefulSet's Pods:
kubectl get pod -w -l
app
=
nginx
In a second terminal, use
to delete all
the Pods in the StatefulSet:
kubectl delete pod -l
app
=
nginx
pod "web-0" deleted
pod "web-1" deleted
Wait for the StatefulSet to restart them, and for both Pods to transition to
Running and Ready:
app
=
nginx
NAME      READY     STATUS              RESTARTS   AGE
web-0     0/1       ContainerCreating   0          0s
web-0     1/1       Running   0          2s
web-1     0/1       ContainerCreating   0         

0s
web-1     1/1       Running   0         34s
Use
and
to view the Pods' hostnames and in-cluster
DNS entries. First, view the Pods' hostnames:
for
i in
0
1;
do
kubectl
exec
web-
$i
-- sh -c
;
done
web-0
web-1
then, run:
kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm /bin/sh
which starts a new shell.
In that new shell, run:

Address 1: 10.244.1.7

Address 1: 10.244.2.8
exit
)
The Pods' ordinals, hostnames, SRV records, and A record names have not changed,
but the IP addresses associated with the Pods may have changed. In the cluster
used for this tutorial, they have. This is why it is important not to configure
other applications to connect to Pods in a StatefulSet by IP address.
If you need to find and connect to the active members of a StatefulSet, you
should query the CNAME of the headless Service
(
nginx.default.svc.cluster.local
). The SRV records associated with the
CNAME will contain only the Pods in the StatefulSet that are Running and
Ready.
If your application already implements connection logic that tests for
liveness and readiness, you can use the SRV records of the Pods (
web-0.nginx.default.svc.cluster.local
,
web-1.nginx.default.svc.cluster.local
), as they are stable, and your
application will be able to discover the Pods' addresses when they transition
to Running and Ready.
Writing to Stable Storage
Get the PersistentVolumeClaims for
web-0
and
web-1
:
kubectl get pvc -l
app
=
nginx
NAME        STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE
www-web-0   Bound     pvc-15c268c7-b507-11e6-932f-42010a800002   1Gi        RWO           48s
www-web-1   Bound     pvc-15c79307-b507-11e6-932f-42010a800002   1Gi        RWO           48s
The StatefulSet controller created two
that are bound to two
.
As the cluster used in this tutorial is configured to dynamically provision PersistentVolumes,
the PersistentVolumes were created and bound automatically.
The NGINX webserver, by default, serves an index file from
/usr/share/nginx/html/index.html
. The
StatefulSet's
spec
ensures that the
directory is
backed by a PersistentVolume.
Write the Pods' hostnames to their
files and verify that the NGINX
webservers serve the hostnames:
for
i in
0
1;
do
kubectl
exec
"web-
$i
"
-- sh -c
'echo "$(hostname)" &gt; /usr/share/nginx/html/index.html'
;
done
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
-- curl http://localhost/;
done
web-0
web-1
Note:
If you instead see
403 Forbidden
responses for the above curl command,
you will need to fix the permissions of the directory mounted by the
(due to a
bug when using hostPath volumes
),
for i in 0 1; do kubectl exec web-$i -- chmod 755 /usr/share/nginx/html; done
before retrying the
curl
command above.
app
=
nginx
In a second terminal, delete all of the StatefulSet's Pods:
app
=
nginx
Examine the output of the
command in the first terminal, and wait
for all of the Pods to transition to Running and Ready.
app
=
nginx

Verify the web servers continue to serve their hostnames:
for i in 0 1; do kubectl exec -i -t "web-$i" -- curl http://localhost/; done
web-0
web-1
Even though
web-0
and
web-1
were rescheduled, they continue to serve their
hostnames because the PersistentVolumes associated with their
PersistentVolumeClaims are remounted to their
. No matter what
node
web-0
and
web-1
are scheduled on, their PersistentVolumes will be
mounted to the appropriate mount points.
Scaling a StatefulSet
Scaling a StatefulSet refers to increasing or decreasing the number of replicas.
This is accomplished by updating the
replicas
field. You can use either
or
to scale a StatefulSet.
Scaling Up
In one terminal window, watch the Pods in the StatefulSet:
app
=
nginx
In another terminal window, use
to scale the number of replicas
to 5:
kubectl scale sts web --replicas
=
5
statefulset.apps/web scaled
for the three additional Pods to transition to Running and Ready.
app
=
nginx
web-0     1/1       Running   0          2h
web-1     1/1       Running   0          2h
web-2     0/1       Pending   0          0s
web-2     0/1       Pending   0         0s
web-2     0/1       ContainerCreating   0         0s
web-2     1/1       Running   0         19s
web-3     0/1       Pending   0         0s
web-3     0/1       ContainerCreating   0         0s
web-3     1/1       Running   0         18s
web-4     0/1       Pending   0         0s
web-4     0/1       ContainerCreating   0         0s
web-4     1/1       Running   0         19s
The StatefulSet controller scaled the number of replicas. As with
StatefulSet creation
, the StatefulSet controller
created each Pod sequentially with respect to its ordinal index, and it
waited for each Pod's predecessor to be Running and Ready before launching the
subsequent Pod.
Scaling Down
app
=
nginx
In another terminal, use
to scale the StatefulSet back down to
three replicas:
kubectl patch sts web -p
statefulset.apps/web patched
Wait for
web-4
and
web-3
to transition to Terminating.
app
=
nginx
web-0     1/1       Running             0          3h
web-1     1/1       Running             0          3h
web-2     1/1       Running             0          55s
web-3     1/1       Running             0          36s
web-4     0/1       ContainerCreating   0          18s
web-4     1/1       Running   0          19s
web-4     1/1       Terminating   0         24s
web-3     1/1       Terminating   0         42s
Ordered Pod Termination
The controller deleted one Pod at a time, in reverse order with respect to its
ordinal index, and it waited for each to be completely shutdown before
deleting the next.
Get the StatefulSet's PersistentVolumeClaims:
app
=
nginx
www-web-0   Bound     pvc-15c268c7-b507-11e6-932f-42010a800002   1Gi        RWO           13h
www-web-1   Bound     pvc-15c79307-b507-11e6-932f-42010a800002   1Gi        RWO           13h
www-web-2   Bound     pvc-e1125b27-b508-11e6-932f-42010a800002   1Gi        RWO           13h
www-web-3   Bound     pvc-e1176df6-b508-11e6-932f-42010a800002   1Gi        RWO           13h
www-web-4   Bound     pvc-e11bb5f8-b508-11e6-932f-42010a800002   1Gi        RWO           13h
There a

re still five PersistentVolumeClaims and five PersistentVolumes.
When exploring a Pod's
stable storage
, we saw that the PersistentVolumes mounted to the Pods of a StatefulSet are not deleted when the StatefulSet's Pods are deleted. This is still true when Pod deletion is caused by scaling the StatefulSet down.
Updating StatefulSets
In Kubernetes 1.7 and later, the StatefulSet controller supports automated updates. The
strategy used is determined by the
spec.updateStrategy
StatefulSet API Object. This feature can be used to upgrade the container
images, resource requests and/or limits, labels, and annotations of the Pods in a
StatefulSet. There are two valid update strategies,
and
OnDelete
.
update strategy is the default for StatefulSets.
Rolling Update
The
update strategy will update all Pods in a StatefulSet, in
reverse ordinal order, while respecting the StatefulSet guarantees.
Patch the
web
StatefulSet to apply the
update strategy:
kubectl patch statefulset web -p
'{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'
In one terminal window, patch the
web
StatefulSet to change the container
image again:
kubectl patch statefulset web --type
=
'json'
-p
=
'[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"gcr.io/google_containers/nginx-slim:0.8"}]'
In another terminal, watch the Pods in the StatefulSet:
kubectl get pod -l
app
=
nginx -w
web-0     1/1       Running   0          7m
web-1     1/1       Running   0          7m
web-2     1/1       Running   0          8m
web-2     1/1       Terminating   0         8m
web-2     0/1       Terminating   0         8m
web-1     1/1       Terminating   0         8m
web-1     0/1       Terminating   0         8m
web-1     1/1       Running   0         6s
web-0     1/1       Terminating   0         7m
web-0     0/1       Terminating   0         7m
web-0     1/1       Running   0         10s
The Pods in the StatefulSet are updated in reverse ordinal order. The
StatefulSet controller terminates each Pod, and waits for it to transition to Running and
Ready prior to updating the next Pod. Note that, even though the StatefulSet
controller will not proceed to update the next Pod until its ordinal successor
is Running and Ready, it will restore any Pod that fails during the update to
its current version.
Pods that have already received the update will be restored to the updated version,
and Pods that have not yet received the update will be restored to the previous
version. In this way, the controller attempts to continue to keep the application
healthy and the update consistent in the presence of intermittent failures.
Get the Pods to view their container images:
for
p in
0
1
2;
do
"web-
$p
"
--template
'{{range $i, $c := .spec.containers}}{{$c.image}}{{end}}'
; echo;
done
registry

.k8s.io/nginx-slim:0.8
All the Pods in the StatefulSet are now running the previous container image.
Note:
kubectl rollout status sts/&lt;name&gt;
to view
the status of a rolling update to a StatefulSet
Staging an Update
You can stage an update to a StatefulSet by using the
partition
parameter of
the
update strategy. A staged update will keep all of the Pods
in the StatefulSet at the current version while allowing mutations to the
.spec.template
.
Patch the
web
StatefulSet to add a partition to the
field:
'{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":3}}}}'
Patch the StatefulSet again to change the container's image:
=
'json'
-p
=
'[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"registry.k8s.io/nginx-slim:0.7"}]'
Delete a Pod in the StatefulSet:
kubectl delete pod web-2
pod "web-2" deleted
Wait for the Pod to be Running and Ready.
app
=
nginx -w
web-0     1/1       Running             0          4m
web-1     1/1       Running             0          4m
web-2     0/1       ContainerCreating   0          11s
web-2     1/1       Running   0         18s
Get the Pod's container image:
kubectl get pod web-2 --template
Notice that, even though the update strategy is
the StatefulSet
restored the Pod with its original container. This is because the
ordinal of the Pod is less than the
partition
specified by the
.
Rolling Out a Canary
You can roll out a canary to test a modification by decrementing the
partition
you specified
above
.
Patch the StatefulSet to decrement the partition:
'{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":2}}}}'
Wait for
web-2
to be Running and Ready.
app
=
nginx -w
Get the Pod's container:
registry.k8s.io/nginx-slim:0.7
When you changed the
partition
, the StatefulSet controller automatically
updated the
web-2
Pod because the Pod's ordinal was greater than or equal to
the
partition
.
Delete the
web-1
Pod:
kubectl delete pod web-1
Wait for the
web-1
Pod to be Running and Ready.
app
=
nginx -w
NAME      READY     STATUS        RESTARTS   AGE
web-0     1/1       Running       0          6m
web-1     0/1       Terminating   0          6m
web-2     1/1       Running       0          2m
web-1     0/1       Terminating   0         6m
Get the
web-1
Pod's container image:
kubectl get pod web-1 --template
web-1
was restored to its original configuration because the Pod's ordinal
was less than the partition. When a partition is specified, all Pods with an
ordinal that is greater than or equal to the partition will be updated when the
is updated. If a Pod that has an ordinal less
than the partition is deleted or otherwise terminated, it will be restored to
its original configuratio

n.
Phased Roll Outs
You can perform a phased roll out (e.g. a linear, geometric, or exponential
roll out) using a partitioned rolling update in a similar manner to how you
rolled out a
canary
. To perform a phased roll out, set
the
partition
to the ordinal at which you want the controller to pause the
update.
The partition is currently set to
2
. Set the partition to
0
:
'{"spec":{"updateStrategy":{"type":"RollingUpdate","rollingUpdate":{"partition":0}}}}'
Wait for all of the Pods in the StatefulSet to become Running and Ready.
app
=
nginx -w
web-0     1/1       Running             0          3m
web-1     0/1       ContainerCreating   0          11s
web-2     1/1       Running             0          2m
web-0     1/1       Terminating   0         3m
web-0     0/1       Terminating   0         3m
web-0     1/1       Running   0         3s
Get the container image details for the Pods in the StatefulSet:
for
p in
0
1
2;
do
"web-
$p
"
; echo;
done
By moving the
partition
to
0
, you allowed the StatefulSet to
continue the update process.
On Delete
The
OnDelete
update strategy implements the legacy (1.6 and prior) behavior,
When you select this update strategy, the StatefulSet controller will not
automatically update Pods when a modification is made to the StatefulSet's
field. This strategy can be selected by setting the
.spec.template.updateStrategy.type
to
OnDelete
.
Deleting StatefulSets
StatefulSet supports both Non-Cascading and Cascading deletion. In a
Non-Cascading Delete, the StatefulSet's Pods are not deleted when the StatefulSet is deleted. In a Cascading Delete, both the StatefulSet and its Pods are
deleted.
Non-Cascading Delete
In one terminal window, watch the Pods in the StatefulSet.
kubectl get pods -w -l app=nginx
Use
to delete the
StatefulSet. Make sure to supply the
--cascade=orphan
parameter to the
command. This parameter tells Kubernetes to only delete the StatefulSet, and to
not delete any of its Pods.
kubectl delete statefulset web --cascade
=
orphan
statefulset.apps "web" deleted
Get the Pods, to examine their status:
app
=
nginx
web-0     1/1       Running   0          6m
web-2     1/1       Running   0          5m
web
has been deleted, all of the Pods are still Running and Ready.
Delete
web-0
:
kubectl delete pod web-0
app
=
nginx
web-1     1/1       Running   0          10m
web-2     1/1       Running   0          7m
As the
web
StatefulSet has been deleted,
web-0
has not been relaunched.
In one terminal, watch the StatefulSet's Pods.
app
=
nginx
In a second terminal, recreate the StatefulSet. Note that, unless
you deleted the
nginx
Service (which you should not have), you will see
an error indicating that the Service already exists.
service/nginx unchanged
Ignore the error. It only indicates that an attempt was made to create the
nginx
headless Service even though that Service already exists.
command running in the first terminal.
kubectl

 get pods -w -l
app
=
nginx
web-1     1/1       Running   0          16m
web-2     1/1       Running   0          2m
web-0     1/1       Running   0         18s
web-2     1/1       Terminating   0         3m
web-2     0/1       Terminating   0         3m
When the
web
StatefulSet was recreated, it first relaunched
web-0
.
Since
web-1
was already Running and Ready, when
web-0
transitioned to
Running and Ready, it adopted this Pod. Since you recreated the StatefulSet
with
replicas
equal to 2, once
web-0
had been recreated, and once
web-1
had been determined to already be Running and Ready,
web-2
was
terminated.
Let's take another look at the contents of the
file served by the
Pods' webservers:
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
Even though you deleted both the StatefulSet and the
web-0
Pod, it still
serves the hostname originally entered into its
file. This is
because the StatefulSet never deletes the PersistentVolumes associated with a
Pod. When you recreated the StatefulSet and it relaunched
web-0
, its original
PersistentVolume was remounted.
Cascading Delete
app
=
nginx
In another terminal, delete the StatefulSet again. This time, omit the
kubectl delete statefulset web
command running in the first terminal,
and wait for all of the Pods to transition to Terminating.
app
=
nginx
web-0     1/1       Running   0          11m
web-1     1/1       Running   0          27m
web-0     1/1       Terminating   0          12m
web-1     1/1       Terminating   0         29m
web-0     0/1       Terminating   0         12m
web-1     0/1       Terminating   0         29m
As you saw in the
section, the Pods
are terminated one at a time, with respect to the reverse order of their ordinal
indices. Before terminating a Pod, the StatefulSet controller waits for
the Pod's successor to be completely terminated.
Note:
Although a cascading delete removes a StatefulSet together with its Pods,
the cascade does not delete the headless Service associated with the StatefulSet.
You must delete the
nginx
Service manually.
kubectl delete service nginx
service "nginx" deleted
Recreate the StatefulSet and headless Service one more time:
When all of the StatefulSet's Pods transition to Running and Ready, retrieve
the contents of their
files:
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
Even though you completely deleted the StatefulSet, and all of its Pods, the
Pods are recreated with their PersistentVolumes mounted, and
web-0
and
web-1
continue to serve their hostnames.
Finally, delete the
nginx
...and the
web
StatefulSet:
statefulset "web" deleted
Pod Management Policy
For some distributed systems, the StatefulSet ordering guarantees are
unnecessary and/or undesirable. These systems require only uniqueness and
identity. To address this, in Kubernetes 1.7, we introduced
.spec.podManagementPolicy
to the StatefulSet API Object.
OrderedReady Pod Managem

ent
pod management is the default for StatefulSets. It tells the
StatefulSet controller to respect the ordering guarantees demonstrated
above.
Parallel Pod Management
Parallel
pod management tells the StatefulSet controller to launch or
terminate all Pods in parallel, and not to wait for Pods to become Running
and Ready or completely terminated prior to launching or terminating another
Pod. This option only affects the behavior for scaling operations. Updates are not affected.
application/web/web-parallel.yaml
:
v1
kind
:
Service
metadata
:
name
:
nginx
labels
:
app
:
nginx
spec
:
ports
:
-
port
:
80
name
:
web
clusterIP
:
None
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
web
spec
:
:
"nginx"
:
"Parallel"
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
name
:
web
:
-
name
:
www
mountPath
:
:
-
metadata
:
name
:
www
spec
:
:
[
]
resources
:
requests
:
storage
:
1Gi
web-parallel.yaml
This manifest is identical to the one you downloaded above except that the
of the
web
StatefulSet is set to
Parallel
.
In one terminal, watch the Pods in the StatefulSet.
app
=
nginx -w
In another terminal, create the StatefulSet and Service in the manifest:
kubectl apply -f web-parallel.yaml
command that you executed in the first terminal.
app
=
nginx -w
web-1     1/1       Running   0         10s
The StatefulSet controller launched both
web-0
and
web-1
at the same time.
Keep the second terminal open, and, in another terminal window scale the
kubectl scale statefulset/web --replicas
=
4
Examine the output of the terminal where the
command is running.
web-3     0/1       Pending   0         7s
web-3     0/1       ContainerCreating   0         7s
web-2     1/1       Running   0         10s
web-3     1/1       Running   0         26s
The StatefulSet launched two new Pods, and it did not wait for
the first to become Running and Ready prior to launching the second.
You should have two terminals open, ready for you to run
kubectl
commands as
part of cleanup.
kubectl delete sts web
# sts is an abbreviation for statefulset
You can watch
to see those Pods being deleted.
app
=
nginx -w
web-3     1/1       Terminating   0         9m
web-2     1/1       Terminating   0         9m
web-1     1/1       Terminating   0         44m
web-0     1/1       Terminating   0         44m
web-0     0/1       Terminating   0         44m
web-3     0/1       Terminating   0         9m
web-2     0/1       Terminating   0         9m
web-1     0/1       Terminating   0         44m
web-0     0/1  

     Terminating   0         44m
During deletion, a StatefulSet removes all Pods concurrently; it does not wait for
a Pod's ordinal successor to terminate prior to deleting that Pod.
Close the terminal where the
command is running and delete the
nginx
Service:
kubectl delete svc nginx
Delete the persistent storage media for the PersistentVolumes used in this tutorial.
kubectl get pvc
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-web-0   Bound    pvc-2bf00408-d366-4a12-bad0-1869c65d0bee   1Gi        RWO            standard       25m
www-web-1   Bound    pvc-ba3bfe9c-413e-4b95-a2c0-3ea8a54dbab4   1Gi        RWO            standard       24m
www-web-2   Bound    pvc-cba6cfa6-3a47-486b-a138-db5930207eaf   1Gi        RWO            standard       15m
www-web-3   Bound    pvc-0c04d7f0-787a-4977-8da3-d9d3a6d8d752   1Gi        RWO            standard       15m
www-web-4   Bound    pvc-b2c73489-e70b-4a4e-9ec1-9eab439aa43e   1Gi        RWO            standard       14m
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS   REASON   AGE
pvc-0c04d7f0-787a-4977-8da3-d9d3a6d8d752   1Gi        RWO            Delete           Bound    default/www-web-3   standard                15m
pvc-2bf00408-d366-4a12-bad0-1869c65d0bee   1Gi        RWO            Delete           Bound    default/www-web-0   standard                25m
pvc-b2c73489-e70b-4a4e-9ec1-9eab439aa43e   1Gi        RWO            Delete           Bound    default/www-web-4   standard                14m
pvc-ba3bfe9c-413e-4b95-a2c0-3ea8a54dbab4   1Gi        RWO            Delete           Bound    default/www-web-1   standard                24m
pvc-cba6cfa6-3a47-486b-a138-db5930207eaf   1Gi        RWO            Delete           Bound    default/www-web-2   standard                15m
kubectl delete pvc www-web-0 www-web-1 www-web-2 www-web-3 www-web-4
persistentvolumeclaim "www-web-0" deleted
persistentvolumeclaim "www-web-1" deleted
persistentvolumeclaim "www-web-2" deleted
persistentvolumeclaim "www-web-3" deleted
persistentvolumeclaim "www-web-4" deleted
Note:
You also need to delete the persistent storage media for the PersistentVolumes
used in this tutorial.
Follow the necessary steps, based on your environment, storage configuration,
and provisioning method, to ensure that all storage is reclaimed.
Feedback
Yes
No
.
or
.
Last modified November 28, 2022 at 3:58 AM PST:
Rename docs section for StatefulSetStartOrdinal to 'Start ordinal' (6474d77d4f)
Scaling Up
On Delete
OrderedReady Pod Management
Home
Blog
Training
Partners
Community
CC BY 4.0
ICP 

Auditing | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Auditing
Auditing
auditing
stage
chain.
Panic
Note:
Event
API group
.
None
Metadata
Request
- log event metadata, request and response bodies.
using the
flag.

 If the flag is omitted, no events are logged.
rules
field
must
:
kind
:
Policy
:
-
rules
:
-
level
:
resources
:
-
group
:
""
resources
:
[
"pods"
]
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
,
]
-
level
:
None
resources
:
-
group
:
""
resources
:
[
]
:
[
]
-
level
:
None
users
:
[
]
verbs
:
[
"watch"
]
resources
:
-
group
:
""
resources
:
[
,
]
-
level
:
None
:
[
]
:
-
"/api*"
-
-
level
:
Request
resources
:
-
group
:
""
resources
:
[
]
:
[
]
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
"secrets"
,
]
-
level
:
Request
resources
:
-
group
:
""
-
group
:
-
level
:
Metadata
:
-
Metadata
level:
:
kind
:
Policy
rules
:
-
level
:
Metadata
Policy
API group
.
Note:
request to
:
[
{
"op"
:
"replace"
,
"path"
:
,
"value"
:
0
},
{
"op"
:
"remove"
,
"path"
:
}
]
JSONlines
format.
flags:
specifies the log file path that log backend uses to write
-
--audit-log-maxag

e
hostPath
=
\
=
...
:
-
mountPath
:
name
:
audit
readOnly
:
true
-
mountPath
:
name
:
audit-log
readOnly
:
false
hostPath
:
...
volumes
:
-
name
:
audit
hostPath
:
path
:
type
:
File
-
name
:
audit-log
hostPath
:
path
:
type
:
.
webhook
with
log
in the flag
webhook
log
webhook
log
.
batch
blocking
batch
mode:
on
and
setting them manually. You can look at the following Prometheus metrics exposed by kube-apiserver

webhook
and
log
or
.
Event
and the
Policy
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Assign Pods to Nodes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Insta

ll Tools
Debug Pods
Debug

 Init Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Se

curity Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Conf

Tasks
minikube
.
List the
nodes
=
worker0
=
worker1
=
worker2
disktype
=
ssd
where
label:
=
=
worker0
=
worker1
=
worker2
worker0
node has a
label.
a
label.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
labels
:
env
:
test
spec
:
:
-
name
:
nginx
image
:
nginx
:
:
disktype
:
ssd
=
wide
NAME 

    READY     STATUS    RESTARTS   AGE    IP           NODE
0
You can also schedule a pod to one specific node via setting
nodeName
.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
nodeName
:
foo-node
:
-
name
:
nginx
image
:
nginx
:
foo-node
only.
.
nodes
.
Feedback
Yes
No
.
or
.
Last modified June 18, 2022 at 10:06 PM PST:
Fix typo in assign-pods-nodes.md (f1f9a50f36)
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure a Pod to Use a Projected Volume for Storage | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Ope

rator pattern
Tasks
Debug Pods
Determine the Rea

son for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the C

luster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration

 (v1)
Tasks
projected
secret
,
configMap
,
,
and
Note:
minikube
.
Secrets
projected
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
args
:
- sleep
-
"86400"
:
-
name
:
mountPath
:
"/projected-volume"
readOnly
:
true
volumes
:
-
name
:
projected
:
sources
:
-
secret
:
name
:
user
-
secret
:
name
:
pass
echo
-n
"admin"
echo
-n
=
=
the Pod:
kubectl
exec
Clean up
kubectl delete secr

et user pass
projected
volumes.
Read the
Feedback
Yes
No
.
or
.
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Advertise Extended Resources for a Node | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Tasks
minikube
.
To advertise a new extended resource on a Node, send an HTTP PATCH request to
[
{
"op"
:
"add"
,
"path"
:
,
"value"
:
"4"
}
]
Replace
\
\
--data
\
Note:
~1
Capacity
:
cpu
:
2
memory
:
2049008Ki
examp

le.com/dongle
:
4
.
Capacity
:
...
:
8
Capacity
:
...
:
800Gi
Clean up

[
  {
  }
]
Replace
\
\
--data
\
Verify that the dongle advertisement has been removed:
Feedback
Yes
No
.
or
.
Last modified February 11, 2021 at 3:51 PM PST:
clean up use of word: just (3ff5ec1eff)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Glossary | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, Storage, and Networking Extens

ions
Tasks
Monitorin

g, Logging, and Debugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutoria

l - Configuring a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Image Po

licy API (v1alpha1)
Reference
Glossary
Glossary
This glossary is intended to be a comprehensive, standardized list of Kubernetes terminology. It includes technical terms that are specific to Kubernetes, as well as more general terms that provide useful context.
Filter terms according to their tags
.
The inner components of Kubernetes.
Related to Kubernetes open-source development.
A resource type that Kubernetes supports by default.
Supported customizations of Kubernetes.
Relevant for a first-time user of Kubernetes.
How Kubernetes components talk to each other (and to programs outside the cluster).
Starting and maintaining Kubernetes.
Keeping Kubernetes applications safe and secure.
How Kubernetes applications handle persistent data.
Software that makes Kubernetes easier or better to use.
Represents a common type of Kubernetes user.
Applications running on Kubernetes.
Architecture
Community
Core Object
Extension
Fundamental
Operation
Security
Storage
Tool
User Type
Workload
Select all
Deselect all
[+]
indicators below to get a longer explanation for any particular term.
Add-ons
LINK
Resources that extend the functionality of Kubernetes.
[+]
Installing addons
explains more about using add-ons with your cluster, and lists some popular add-ons.
LINK
A piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object.
[+]
Admission controllers are configurable for the Kubernetes API server and may be "validating", "mutating", or
both. Any admission controller may reject the request. Mutating controllers may modify the objects they admit;
validating controllers may not.
Admission controllers in the Kubernetes documentation
Affinity
LINK
affinity
is a set of rules that give hints to the scheduler about where to place pods.
[+]
There are two kinds of affinity:
pod-to-pod affinity
The rules are defined using the Kubernetes
labels
,
and
selectors
pods
,
and they can be either required or preferred, depending on how strictly you want the scheduler to enforce them.
Aggregation Layer
LINK
The aggregation layer lets you install additional Kubernetes-style APIs in your cluster.
[+]
When you've configured the
Kubernetes API Server
to
support additional APIs
, you can add
objects to "claim" a URL path in the Kubernetes API.
Annotation
LINK
A key-value pair that is us

ed to attach arbitrary non-identifying metadata to objects.
[+]
The metadata in an annotation can be small or large, structured or unstructured, and can include characters not permitted by
labels
. Clients such as tools and libraries can retrieve this metadata.
API Group
LINK
A set of related paths in Kubernetes API.
[+]
You can enable or disable each API group by changing the configuration of your API server. You can also disable or enable paths to specific resources. API group makes it easier to extend the Kubernetes API. The API group is specified in a REST path and in the
field of a serialized object.
Read
API Group
API server
LINK
Also known as:
[+]
.
You can run several instances of kube-apiserver and balance traffic between those instances.
LINK
API-initiated eviction is the process by which you use the
Eviction API
to create an
Eviction
object that triggers graceful pod termination.
[+]
You can request eviction either by directly calling the Eviction API
using a client of the kube-apiserver, like the
command.
When an
Eviction
object is created, the API server terminates the Pod.
API-initiated evictions respect your configured
PodDisruptionBudgets
and
.
API-initiated eviction is not the same as
node-pressure eviction
.
See
App Container
LINK
Application containers (or app containers) are the
in a
pod
that are started after any
init containers
have completed.
[+]
An init container lets you separate initialization details that are important for the overall
workload
, and that don't need to keep running
once the application container has started.
If a pod doesn't have any init containers configured, all the containers in that pod are app containers.
Application Architect
LINK
A person responsible for the high-level design of an application.
[+]
An architect ensures that an app's implementation allows it to interact with its surrounding components in a scalable, maintainable way. Surrounding components include databases, logging infrastructure, and other microservices.
Application Developer
LINK
A person who writes an application that runs in a Kubernetes cluster.
[+]
An application developer focuses on one part of an application. The scale of their focus may vary significantly in size.
Applications
LINK
The layer where various containerized applications run.
[+]
Approver
LINK
A person who can review and approve Kubernetes code contributions.
[+]
While code review is focused on code quality and correctness, approval is focused on the holistic acceptance of a contribution. Holistic acceptance includes backwards/forwards compatibility, adhering to API and flag conventions, subtle performance and correctness issues, interactions with other parts of the system, and others. Approver status is scoped to a part of the codebase. Approvers were previously referred to as maintainers.
cAdvisor
LINK
cAdvisor (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running
.
[+]
It is a running daemon that collects, aggregates, processes, and exports information about running containers. Specifically, for each container it keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics. This data is exported by container and machine-wide.
Certificate
LINK
A cryptographically secure file used to validate access to the Kubernetes cluster.
[+]
Cert

ificates enable applications within a Kubernetes cluster to access the Kubernetes API securely. Certificates validate that clients are allowed to access the API.
cgroup (control group)
LINK
A group of Linux processes with optional resource isolation, accounting and limits.
[+]
cgroup is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network) for a collection of processes.
CIDR
LINK
CIDR (Classless Inter-Domain Routing) is a notation for describing blocks of IP addresses and is used heavily in various networking configurations.
[+]
In the context of Kubernetes, each
Node
is assigned a range of IP addresses through the start address and a subnet mask using CIDR. This allows Nodes to assign each
Pod
a unique IP address. Although originally a concept for IPv4, CIDR has also been expanded to include IPv6.
CLA (Contributor License Agreement)
LINK
Terms under which a
contributor
grants a license to an open source project for their contributions.
[+]
CLAs help resolve legal disputes involving contributed material and intellectual property (IP).
LINK
component
[+]
Cloud Native Computing Foundation (CNCF)
LINK
The Cloud Native Computing Foundation (CNCF) builds sustainable ecosystems and
fosters a community around
projects
that
orchestrate containers as part of a microservices architecture.
Kubernetes is a CNCF project.
[+]
The CNCF is a sub-foundation of
the Linux Foundation
.
Its mission is to make cloud native computing ubiquitous.
Cloud Provider
LINK
Cloud Service Provider
A business or other organization that offers a cloud computing platform.
[+]
Cloud providers, sometimes called Cloud Service Providers (CSPs), offer
cloud computing platforms or services.
Many cloud providers offer managed infrastructure (also called
Infrastructure as a Service or IaaS).
With managed infrastructure the cloud provider is responsible for
servers, storage, and networking while you manage layers on top of that
such as running a Kubernetes cluster.
You can also find Kubernetes as a managed service; sometimes called
Platform as a Service, or PaaS. With managed Kubernetes, your
cloud provider is responsible for the Kubernetes control plane as well
as the
nodes
and the
infrastructure they rely on: networking, storage, and possibly other
elements such as load balancers.
Cluster
LINK
A set of worker machines, called
nodes
,
[+]
Pods
that are
Cluster Architect
LINK
A person who designs infrastructure that involves one or more Kubernetes clusters.
[+]
Cluster architects are concerned with best practices for distributed systems, for example: high availability and security.
Cluster Infrastructure
LINK
The infrastructure layer provides and maintains VMs, networking, security groups and others.
[+]
Cluster Operations
LINK
The work involved in managing a Kubernetes cluster: managing
day-to-day operations, and co-ordinating upgrades.
[+]
Examples of cluster operati

ons work include: deploying new Nodes to
scale the cluster; performing software upgrades; implementing security
controls; adding or removing storage; configuring cluster networking;
managing cluster-wide observability; and responding to events.
Cluster Operator
LINK
A person who configures, controls, and monitors clusters.
[+]
Their primary responsibility is keeping a cluster up and running, which may involve periodic maintenance activities or upgrades.
Note:
Cluster operators are different from the
that extends the Kubernetes API.
Code Contributor
LINK
A person who develops and contributes code to the Kubernetes open source codebase.
[+]
They are also an active
community member
who participates in one or more
Special Interest Groups (SIGs)
.
ConfigMap
LINK
An API object used to store non-confidential data in key-value pairs.
Pods
can consume ConfigMaps as
environment variables, command-line arguments, or as configuration files in a
volume
.
[+]
A ConfigMap allows you to decouple environment-specific configuration from your
container images
, so that your applications are easily portable.
Container
LINK
A lightweight and portable executable image that contains software and all of its dependencies.
[+]
Containers decouple applications from underlying host infrastructure to make deployment easier in different cloud or OS environments, and for easier scaling.
The applications that run inside containers are called containerized applications. The process of bundling these applications and their dependencies into a container image is called containerization.
Container Environment Variables
LINK
Container environment variables are name=value pairs that provide useful information into containers running in a
pod
[+]
Container environment variables provide information that is required by the running containerized applications along with information about important resources to the
. For example, file system details, information about the container itself, and other cluster resources such as service endpoints.
LINK
The lifecycle hooks expose events in the
Container
management lifecycle and let the user run code when the events occur.
[+]
Two hooks are exposed to Containers: PostStart which executes immediately after a container is created and PreStop which is blocking and is called immediately before a container is terminated.
Container network interface (CNI)
LINK
Container network interface (CNI) plugins are a type of Network plugin that adheres to the appc/CNI specification.
[+]
For information on Kubernetes and CNI, see
.
LINK
[+]
,
CRI-O
,
.
LINK
The main protocol for the communication between the kubelet and Container Runtime.
[+]
The Kubernetes Container Runtime Interface (CRI) defines the main
gRPC
protocol for the communication between the
cluster components
kubelet
and
.
Container runtime interface (CRI)
LINK
The container runtime interface (CRI) is an API for container runtimes
to integrate with kubelet on a node.
[+]
CRI
API and specifications.
LINK
The Container Storage Interface (CSI) defines a standard interface to expose storage systems to containers.
[+]
CSI allows vendors to create custom storage plugins for Kubernetes without adding them to the Kubernetes repository (out-of-tree plugins). To use a CSI driver from a storage provider, you must first
deploy it to your cluster
. You will then be able to create a
Storage Class
that uses that CSI driver.
CSI in the Kubernetes documentation
List of available CSI drivers
LINK
A container runtime with an emphasis on simplicity, robustness and portability

[+]
containerd is a
container
runtime
that runs as a daemon on Linux or Windows. containerd takes care of fetching and
storing container images, executing containers, providing network access, and more.
Contributor
LINK
Someone who donates code, documentation, or their time to help the Kubernetes project or community.
[+]
Contributions include pull requests (PRs), issues, feedback,
special interest groups (SIG)
participation, or organizing community events.
LINK
The container orchestration layer that exposes the API and interfaces to define, deploy, and manage the lifecycle of containers.
[+]
This layer is composed by many different components, such as (but not restricted to):
etcd
API Server
Scheduler
Controller Manager
These components can be run as traditional operating system services (daemons) or as containers. The hosts running these components were historically called
masters
.
LINK
In Kubernetes, controllers are control loops that watch the state of your
cluster
, then make or request
changes where needed.
Each controller tries to move the current cluster state closer to the desired
state.
[+]
Controllers watch the shared state of your cluster through the
apiserver
(part of the
).
Some controllers also run inside the control plane, providing control loops that
are core to Kubernetes' operations. For example: the deployment controller, the
daemonset controller, the namespace controller, and the persistent volume
controller (and others) all run within the
.
CRI-O
LINK
A tool that lets you use OCI container runtimes with Kubernetes CRI.
[+]
CRI-O is an implementation of the
to enable using
container
runtimes that are compatible with the Open Container Initiative (OCI)
runtime spec
.
Deploying CRI-O allows Kubernetes to use any OCI-compliant runtime as the container
runtime for running
Pods
, and to fetch
OCI container images from remote registries.
CronJob
LINK
Manages a
Job
that runs on a periodic schedule.
[+]
Similar to a line in a
crontab
file, a CronJob object specifies a schedule using the
cron
format.
LINK
Custom code that defines a resource to add to your Kubernetes API server without building a complete custom server.
[+]
Custom Resource Definitions let you extend the Kubernetes API for your environment if the publicly supported API resources can't meet your needs.
DaemonSet
LINK
Ensures a copy of a
Pod
is running across a set of nodes in a
cluster
.
[+]
Used to deploy system daemons such as log collectors and monitoring agents that typically must run on every
Node
.
Data Plane
LINK
The layer that provides capacity such as CPU, memory, network, and storage so that the containers can run and connect to a network.
[+]
LINK
An API object that manages a replicated application, typically by running Pods with no local state.
[+]
Each replica is represented by a
Pod
, and the Pods are distributed among the
nodes
of a cluster.
For workloads that do require local state, consider using a
.
Developer (disambiguation)
LINK
May refer to:
,
, or
Platform Developer
.
[+]
This overloaded term may have different meanings depending on the context
Device Plugin
LINK
Device plugins run on worker
Nodes
and provide
Pods
with access to resources,
such as local hardware, that require vendor-specific initialization or setup
steps.
[+]
Device plugins advertise resources to the
kubelet
, so that workload
Pods can access hardware features that relate to the Node where that Pod is running.
You can deploy a device plugin as a
DaemonSet
,
or install the device plugin software directly on each target Node.
See
Disruption
LINK
Disruptions are events that lead to one or more

Pods
going out of service.
A disruption has consequences for workload resources, such as
, that rely on the affected
Pods.
[+]
If you, as cluster operator, destroy a Pod that belongs to an application,
Kubernetes terms that a
voluntary disruption
. If a Pod goes offline
because of a Node failure, or an outage affecting a wider failure zone,
Kubernetes terms that an
involuntary disruption
.
See
Docker
LINK
Docker (specifically, Docker Engine) is a software technology providing operating-system-level virtualization also known as
.
[+]
Docker uses the resource isolation features of the Linux kernel such as cgroups and kernel namespaces, and a union-capable file system such as OverlayFS and others to allow independent containers to run within a single Linux instance, avoiding the overhead of starting and maintaining virtual machines (VMs).
Dockershim
LINK
The dockershim is a component of Kubernetes version 1.23 and earlier. It allows the kubelet
to communicate with
Docker Engine
.
[+]
Starting with version 1.24, dockershim has been removed from Kubernetes. For more information, see
Dockershim FAQ
.
Downstream (disambiguation)
LINK
May refer to: code in the Kubernetes ecosystem that depends upon the core Kubernetes codebase or a forked repo.
[+]
In the
Kubernetes Community
: Conversations often use
downstream
to mean the ecosystem, code, or third-party tools that rely on the core Kubernetes codebase. For example, a new feature in Kubernetes may be adopted by applications
to improve their functionality.
In
GitHub
or
git
: The convention is to refer to a forked repo as
, whereas the source repo is considered
upstream
.
LINK
Kubernetes' mechanism to expose Pod and container field values to code running in a container.
[+]
It is sometimes useful for a container to have information about itself, without
needing to make changes to the container code that directly couple it to Kubernetes.
The Kubernetes downward API allows containers to consume information about themselves
or their context in a Kubernetes cluster. Applications in containers can have
access to that information, without the application needing to act as a client of
There are two ways to expose Pod and container fields to a running container:
using
using
a
volume
.
LINK
Allows users to request automatic creation of storage
Volumes
.
[+]
Dynamic provisioning eliminates the need for cluster administrators to pre-provision storage. Instead, it automatically provisions storage by user request. Dynamic volume provisioning is based on an API object,
, referring to a
Volume Plugin
that provisions a
Volume
and the set of parameters to pass to the Volume Plugin.
Endpoints
LINK
Endpoints track the IP addresses of Pods with matching
selectors
.
[+]
Endpoints can be configured manually for
Services
without selectors specified.
The
resource provides a scalable and extensible alternative to Endpoints.
LINK
A way to group network endpoints together with Kubernetes resources.
[+]
A scalable and extensible way to group network endpoints together. These can be
used by
to
establish network routes on each
node
.
Ephemeral Container
LINK
A
Container
type that you can temporarily run inside a
Pod
.
[+]
If you want to investigate a Pod that's running with problems, you can add an ephemeral container to that Pod and carry out diagnostics. Ephemeral containers have no resource or scheduling guarantees, and you should not use them to run any part of the workload itself.
Ephemeral containers are not supported by
.
etcd
LINK
[+]
If your Kubernetes cluster uses etcd as its backing

 store, make sure you have a
back up
plan
.
Event
LINK
Each Event is a report of an event somewhere in the
cluster
.
It generally denotes some state change in the system.
[+]
Events have a limited retention time and triggers and messages may evolve with time.
Event consumers should not rely on the timing of an event with a given reason reflecting a consistent underlying trigger,
or the continued existence of events with that reason.
Events should be treated as informative, best-effort, supplemental data.
auditing
generates a different kind of
Event record (API group
).
Eviction
LINK
Eviction is the process of terminating one or more Pods on Nodes.
[+]
There are two kinds of eviction:
Node-pressure eviction
Extensions
LINK
Extensions are software components that extend and deeply integrate with Kubernetes to support new types of hardware.
[+]
Many cluster administrators use a hosted or distribution instance of Kubernetes. These clusters
come with extensions pre-installed. As a result, most Kubernetes users will not need to install
and even fewer users will need to author new ones.
Finalizer
LINK
[+]
When you tell Kubernetes to delete an object that has finalizers specified for
,
202
LINK
FlexVolume is a deprecated interface for creating out-of-tree volume plugins. The
Container Storage Interface
is a newer interface that addresses several problems with FlexVolume.
[+]
FlexVolumes enable users to write their own drivers and add support for their volumes in Kubernetes. FlexVolume driver binaries and dependencies must be installed on host machines. This requires root access. The Storage SIG suggests implementing a
CSI
driver if possible since it addresses the limitations with FlexVolumes.
FlexVolume in the Kubernetes documentation
More information on FlexVolumes
Volume Plugin FAQ for Storage Vendors
LINK
Garbage collection is a collective term for the various mechanisms Kubernetes uses to clean up
cluster resources.
[+]
Kubernetes uses garbage collection to clean up resources like
unused containers and images
,
failed Pods
,
objects owned by the targeted resource
,
completed Jobs
, and resources
that have expired or failed.
Helm Chart
LINK
A package of pre-configured Kubernetes resources that can be managed with the Helm tool.
[+]
Charts provide a reproducible way of creating and sharing Kubernetes applications.
A single chart can be used to deploy something simple, like a memcached Pod, or something complex, like a full web app stack with HTTP servers, databases, caches, and so on.
LINK
HPA
An API resource that automatically scales the number of
Pod
replicas based on targeted CPU utilization or custom metric targets.
[+]
HPA is typically used with
ReplicationControllers
,
, or
ReplicaSets
. It cannot be applied to objects that cannot be scaled, for example
DaemonSets
.
HostAliases
LINK
A 

HostAliases is a mapping between the IP address and hostname to be injected into a
Pod
's hosts file.
[+]
is an optional list of hostnames and IP addresses that will be injected into the Pod's hosts file if specified. This is only valid for non-hostNetwork Pods.
Image
LINK
Stored instance of a
Container
that holds a set of software needed to run an application.
[+]
A way of packaging software that allows it to be stored in a container registry, pulled to a local system, and run as an application. Meta data is included in the image that can indicate what executable to run, who built it, and other information.
Ingress
LINK
An API object that manages external access to the services in a cluster, typically HTTP.
[+]
Ingress may provide load balancing, SSL termination and name-based virtual hosting.
Init Container
LINK
One or more initialization
that must run to completion before any app containers run.
[+]
Initialization (init) containers are like regular app containers, with one difference: init containers must run to completion before any app containers can start. Init containers run in series: each init container must run to completion before the next init container begins.
Istio
LINK
An open platform (not Kubernetes-specific) that provides a uniform way to integrate microservices, manage traffic flow, enforce policies, and aggregate telemetry data.
[+]
Adding Istio does not require changing application code. It is a layer of infrastructure between a service and the network, which when combined with service deployments, is commonly referred to as a service mesh. Istio's control plane abstracts away the underlying cluster management platform, which may be Kubernetes, Mesosphere, etc.
Job
LINK
A finite or batch task that runs to completion.
[+]
Creates one or more
Pod
objects and ensures that a specified number of them successfully terminate. As Pods successfully complete, the Job tracks the successful completions.
kOps (Kubernetes Operations)
LINK
kOps
will not only help you create, destroy, upgrade and maintain production-grade, highly available, Kubernetes cluster, but it will also provision the necessary cloud infrastructure.
[+]
Note:
AWS (Amazon Web Services) is currently officially supported, with DigitalOcean, GCE and OpenStack in beta support, and Azure in alpha.
kOps
is an automated provisioning system:
Fully automated installation
Uses DNS to identify clusters
Self-healing: everything runs in Auto-Scaling Groups
Multiple OS support (Amazon Linux, Debian, Flatcar, RHEL, Rocky and Ubuntu)
High-Availability support
Can directly provision, or generate terraform manifests
LINK
[+]
LINK
node
Service
concept.
[+]
LINK
Pods
node
to run on.
[+]
Kubeadm
LINK
A tool for quickly installing Kubernetes and setting up a secure cluster.
[+]
You can use kubeadm to install both the control plane and the
worker node
components.
Kubectl
LINK
kubectl
Command line tool for communicating with a Kubernetes c

luster's
,
[+]
You can use
kubectl
to create, inspect, update, and delete Kubernetes objects.
Kubelet
LINK
node
Pod
.
[+]
LINK
The application that serves Kubernetes functionality through a RESTful interface and stores the state of the cluster.
[+]
kubectl
Label
LINK
Tags objects with identifying attributes that are meaningful and relevant to users.
[+]
Labels are key/value pairs that are attached to objects such as
Pods
. They are used to organize and to select subsets of objects.
LINK
Provides constraints to limit resource consumption per
or
Pods
in a namespace.
[+]
LimitRange limits the quantity of objects that can be created by type,
as well as the amount of compute resources that may be requested/consumed by individual
or
Pods
Logging
LINK
Logs are the list of events that are logged by
cluster
or application.
[+]
Application and systems logs can help you understand what is happening inside your cluster. The logs are particularly useful for debugging problems and monitoring cluster activity.
Managed Service
LINK
A software offering maintained by a third-party provider.
[+]
Some examples of Managed Services are AWS EC2, Azure SQL Database, and
GCP Pub/Sub, but they can be any software offering that can be used by an application.
Manifest
LINK
Specification of a Kubernetes API object in JSON or YAML format.
[+]
A manifest specifies the desired state of an object that Kubernetes will maintain when you apply the manifest. Each configuration file can contain multiple manifests.
Master
LINK
Legacy term, used as synonym for
nodes
hosting the
.
[+]
The term is still being used by some provisioning tools, such as
kubeadm
, and managed services, to
label
nodes
with
kubernetes.io/role
and control placement of
pods
.
Member
LINK
A continuously active
in the K8s community.
[+]
Members can have issues and PRs assigned to them and participate in
special interest groups (SIGs)
through GitHub teams. Pre-submit tests are automatically run for members' PRs. A member is expected to remain an active contributor to the community.
Minikube
LINK
A tool for running Kubernetes locally.
[+]
Minikube runs a single-node cluster inside a VM on your computer.
You can use Minikube to
try Kubernetes in a learning environment
.
Mirror Pod
LINK
A
pod
object that a kubelet uses
to represent a
static pod
[+]
When the kubelet finds a static pod in its configuration, it automatically tries to
create a Pod object on the Kubernetes API server for it. This means that the pod
will be visible on the API server, but cannot be controlled from there.
(For example, removing a mirror pod will not stop the kubelet daemon from running it).
Name
LINK
.
[+]
Namespace
LINK
An abstraction used by Kubernetes to support isolation of groups of resources within a single
cluster
.
[+]
Namespaces are used to organize objects in a cluster and provide a way to divide cluster resources. Names of resources need to be unique within a names

pace, but not across namespaces. Namespace-based scoping is applicable only for namespaced objects
.
Network Policy
LINK
A specification of how groups of Pods are allowed to communicate with each other and with other network endpoints.
[+]
Network Policies help you declaratively configure which Pods are allowed to connect to each other, which namespaces are allowed to communicate, and more specifically which port numbers to enforce each policy on.
resources use labels to select Pods and define rules which specify what traffic is allowed to the selected Pods. Network Policies are implemented by a supported network plugin provided by a network provider. Be aware that creating a network resource without a controller to implement it will have no effect.
Node
LINK
A node is a worker machine in Kubernetes.
[+]
A worker node may be a VM or physical machine, depending on the cluster. It has local daemons or services necessary to run
Pods
and is managed by the control plane. The daemons on a node include
kubelet
,
, and a container runtime implementing the
CRI
such as
Docker
.
In early Kubernetes versions, Nodes were called "Minions".
LINK
kubelet eviction
Node-pressure eviction is the process by which the
kubelet
proactively terminates
pods to reclaim resources on nodes.
[+]
The kubelet monitors resources like CPU, memory, disk space, and filesystem
inodes on your cluster's nodes. When one or more of these resources reach
specific consumption levels, the kubelet can proactively fail one or more pods
on the node to reclaim resources and prevent starvation.
Node-pressure eviction is not the same as
.
Object
LINK
An entity in the Kubernetes system. The Kubernetes API uses these entities to represent the state
of your cluster.
[+]
A Kubernetes object is typically a “record of intent”—once you create the object, the Kubernetes
works constantly to ensure
that the item it represents actually exists.
By creating an object, you're effectively telling the Kubernetes system what you want that part of
your cluster's workload to look like; this is your cluster's desired state.
LINK
The
operator pattern
is a system
design that links a
to one or more custom
[+]
You can extend Kubernetes by adding controllers to your cluster, beyond the built-in
controllers that come as part of Kubernetes itself.
If a running application acts as a controller and has API access to carry out tasks
against a custom resource that's defined in the control plane, that's an example of
the Operator pattern.
Persistent Volume
LINK
An API object that represents a piece of storage in the cluster. Available as a general, pluggable resource that persists beyond the lifecycle of any individual
Pod
.
[+]
PersistentVolumes (PVs) provide an API that abstracts details of how storage is provided from how it is consumed.
PVs are used directly in scenarios where storage can be created ahead of time (static provisioning).
For scenarios that require on-demand storage (dynamic provisioning), PersistentVolumeClaims (PVCs) are used instead.
Persistent Volume Claim
LINK
Claims storage resources defined in a
so that it can be mounted as a volume in a
container
.
[+]
Specifies the amount of storage, how the storage will be accessed (read-only, read-write and/or exclusive) and how it is reclaimed (retained, recycled or deleted). Details of the storage itself are described in the PersistentVolume object.
LINK
A person who customizes the Kubernetes platform to fit the needs of their project.
[+]
A platform developer may, for example, use
or
Extend the Kubernetes API with the aggregation layer
to add functionality to their instance of Kubernetes, specifically for their application.
Some Platform

 Developers are also
contributors
and
develop extensions which are contributed to the Kubernetes community.
Others develop closed-source commercial or site-specific extensions.
Pod
LINK
The smallest and simplest Kubernetes object. A Pod represents a set of running
on your cluster.
[+]
A Pod is typically set up to run a single primary container. It can also run optional sidecar containers that add supplementary features like logging. Pods are commonly managed by a
.
Pod Disruption
LINK
Pod disruption
is the process by which
Pods on Nodes are terminated either voluntarily or involuntarily.
[+]
Voluntary disruptions are started intentionally by application owners or cluster
administrators. Involuntary disruptions are unintentional and can be triggered by
unavoidable issues like Nodes running out of resources, or by accidental deletions.
Pod Disruption Budget
LINK
PDB
A
allows an
application owner to create an object for a replicated application, that ensures
a certain number or percentage of Pods with an assigned label will not be voluntarily
evicted at any point in time.
[+]
Involuntary disruptions cannot be prevented by PDBs; however they
do count against the budget.
LINK
The sequence of states through which a Pod passes during its lifetime.
[+]
The
is defined by the states or phases of a Pod. There are five possible Pod phases: Pending, Running, Succeeded, Failed, and Unknown. A high-level description of the Pod state is summarized in the
PodStatus
phase
field.
LINK
Pod Priority indicates the importance of a
Pod
relative to other Pods.
[+]
gives the ability to set scheduling priority of a Pod to be higher and lower than other Pods — an important feature for production clusters workload.
Pod Security Policy
LINK
Enables fine-grained authorization of
Pod
creation and updates.
[+]
A cluster-level resource that controls security sensitive aspects of the Pod specification. The
objects define a set of conditions that a Pod must run with in order to be accepted into the system, as well as defaults for the related fields. Pod Security Policy control is implemented as an optional admission controller.
PodSecurityPolicy was deprecated as of Kubernetes v1.21, and removed in v1.25.
As an alternative, use
or a 3rd party admission plugin.
Preemption
LINK
Preemption logic in Kubernetes helps a pending
Pod
to find a suitable
Node
by evicting low priority Pods existing on that Node.
[+]
If a Pod cannot be scheduled, the scheduler tries to
preempt
lower priority Pods to make scheduling of the pending Pod possible.
Proxy
LINK
In computing, a proxy is a server that acts as an intermediary for a remote
service.
[+]
A client interacts with the proxy; the proxy copies the client's data to the
actual server; the actual server replies to the proxy; the proxy sends the
actual server's reply to the client.
is a
network proxy that runs on each node in your cluster, implementing part of
the Kubernetes
Service
concept.
You can run kube-proxy as a plain userland proxy service. If your operating
system supports it, you can instead run kube-proxy in a hybrid mode that
achieves the same overall effect using less system resources.
QoS Class
LINK
QoS Class (Quality of Service Class) provides a way for Kubernetes to classify Pods within the cluster into several classes and make decisions about scheduling and eviction.
[+]
QoS Class of a Pod is set at creation time based on its compute resources requests and limits settings. QoS classes are used to make decisions about Pods scheduling and eviction.
Kubernetes can assign one of the following QoS classes to a Pod:
,
Burstable
or
.
Quantity
LINK
A whole-number representation of small or large numbers using
SI
suffixes.
[+]
Quantities are representations of small or large numbers using a compact,
whole-number notation with SI suffixes

. Fractional numbers are represented
using milli units, while large numbers can be represented using kilo,
mega, or giga units.
For instance, the number
1.5
is represented as
1500m
, while the number
1000
can be represented as
1k
, and
1000000
as
1M
. You can also specify
binary-notation
suffixes; the number 2048 can be written as
2Ki
.
The accepted decimal (power-of-10) units are
m
(milli),
k
(kilo,
intentionally lowercase),
M
(mega),
G
(giga),
T
(tera),
P
(peta),
E
(exa).
The accepted binary (power-of-2) units are
Ki
(kibi),
Mi
(mebi),
Gi
(gibi),
Ti
(tebi),
Pi
(pebi),
Ei
(exbi).
RBAC (Role-Based Access Control)
LINK
Manages authorization decisions, allowing admins to dynamically configure access policies through the
.
[+]
RBAC utilizes
roles
, which contain permission rules, and
role bindings
, which grant the permissions defined in a role to a set of users.
LINK
A ReplicaSet (aims to) maintain a set of replica Pods running at any given time.
[+]
Workload objects such as
make use of ReplicaSets
to ensure that the configured number of
Pods
are
running in your cluster, based on the spec of that ReplicaSet.
LINK
A workload resource that manages a replicated application, ensuring that
a specific number of instances of a
Pod
are running.
[+]
The control plane ensures that the defined number of Pods are running, even if some
Pods fail, if you delete Pods manually, or if too many are started by mistake.
Note:
ReplicationController is deprecated. See
, which is similar.
LINK
Provides constraints that limit aggregate resource consumption per
Namespace
.
[+]
Limits the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that project.
Reviewer
LINK
A person who reviews code for quality and correctness on some part of the project.
[+]
Reviewers are knowledgeable about both the codebase and software engineering principles. Reviewer status is scoped to a part of the codebase.
Secret
LINK
Stores sensitive information, such as passwords, OAuth tokens, and SSH keys.
[+]
Secrets give you more control over how sensitive information is used and reduces
the risk of accidental exposure. Secret values are encoded as base64 strings and
are stored unencrypted by default, but can be configured to be
encrypted at rest
.
A
Pod
can reference the Secret in
a variety of ways, such as in a volume mount or as an environment variable.
Secrets are designed for confidential data and
are
designed for non-confidential data.
LINK
The
field defines privilege and access control settings for
a
Pod
or
container
.
[+]
In a
, you can define: the user that processes run as,
the group that processes run as, and privilege settings.
You can also configure security policies (for example: SELinux, AppArmor or seccomp).
The
PodSpec.securityContext
setting applies to all containers in a Pod.
Selector
LINK
Allows users to filter a list of resources based on
labels
.
[+]
Selectors are applied when querying lists of resources to filter them by labels.
Service
LINK
An abstract way to expose an application running on a set of
Pods
as a network service.
[+]
The set of Pods targeted by a Service is (usually) determined by a
selector
. If more Pods are added or removed, the set of Pods matching the selector will change. The Service makes sure that network traffic can be directed to the current set of Pods for the workload.
Service Catalog
LINK
A former extension API that enabled applications running in Kubernetes clusters to easily use external managed software offerings, such as a datastore service offered by a cloud provider.
[+]
It provided a way to list, provision, and bind with external
Managed Services
without needing detailed knowledge about how those services would be created or managed.
LINK
Provides an identity for proce

sses that run in a
Pod
.
[+]
When processes inside Pods access the cluster, they are authenticated by the API server as a particular service account, for example,
default
. When you create a Pod, if you do not specify a service account, it is automatically assigned the default service account in the same
Namespace
.
Shuffle-sharding
LINK
A technique for assigning requests to queues that provides better isolation than hashing modulo the number of queues.
[+]
We are often concerned with insulating different flows of requests
from each other, so that a high-intensity flow does not crowd out low-intensity flows.
A simple way to put requests into queues is to hash some
characteristics of the request, modulo the number of queues, to get
the index of the queue to use. The hash function uses as input
characteristics of the request that align with flows. For example, in
the Internet this is often the 5-tuple of source and destination
address, protocol, and source and destination port.
That simple hash-based scheme has the property that any high-intensity flow
will crowd out all the low-intensity flows that hash to the same queue.
Providing good insulation for a large number of flows requires a large
number of queues, which is problematic. Shuffle-sharding is a more
nimble technique that can do a better job of insulating the low-intensity
flows from the high-intensity flows. The terminology of shuffle-sharding uses
the metaphor of dealing a hand from a deck of cards; each queue is a
metaphorical card. The shuffle-sharding technique starts with hashing
the flow-identifying characteristics of the request, to produce a hash
value with dozens or more of bits. Then the hash value is used as a
source of entropy to shuffle the deck and deal a hand of cards
(queues). All the dealt queues are examined, and the request is put
into one of the examined queues with the shortest length. With a
modest hand size, it does not cost much to examine all the dealt cards
and a given low-intensity flow has a good chance to dodge the effects of a
given high-intensity flow. With a large hand size it is expensive to examine
the dealt queues and more difficult for the low-intensity flows to dodge the
collective effects of a set of high-intensity flows. Thus, the hand size
should be chosen judiciously.
SIG (special interest group)
LINK
Community members
who collectively manage an ongoing piece or aspect of the larger Kubernetes open source project.
[+]
Members within a SIG have a shared interest in advancing a specific area, such as architecture, API machinery, or documentation.
SIGs must follow the SIG
governance guidelines
, but can have their own contribution policy and channels of communication.
kubernetes/community
repo and the current list of
SIGs and Working Groups
.
LINK
Manages the deployment and scaling of a set of
Pods
,
and provides guarantees about the ordering and uniqueness
of these Pods.
[+]
Like a
, a StatefulSet manages Pods that are based on an identical container spec. Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.
If you want to use storage volumes to provide persistence for your workload, you can use a StatefulSet as part of the solution. Although individual Pods in a StatefulSet are susceptible to failure, the persistent Pod identifiers make it easier to match existing volumes to the new Pods that replace any that have failed.
Static Pod
LINK
A
pod
managed directly by the kubelet
daemon on a specific node,
[+]
without the API server observing it.
Static Pods do not support
.
LINK
A StorageClass provides a way for administrators to describe different available storage types.
[+]
StorageClasses can map to quality-of-service levels, backup policies, or to arbitrary policies

 determined by cluster administrators. Each StorageClass contains the fields
,
, and
, which are used when a
belonging to the class needs to be dynamically provisioned. Users can request a particular class using the name of a StorageClass object.
sysctl
LINK
sysctl
is a semi-standardized interface for reading or changing the
attributes of the running Unix kernel.
[+]
On Unix-like systems,
sysctl
is both the name of the tool that administrators
use to view and modify these settings, and also the system call that the tool
uses.
Container
runtimes and
network plugins may rely on
sysctl
values being set a certain way.
Taint
LINK
A core object consisting of three required properties: key, value, and effect. Taints prevent the scheduling of
Pods
on
nodes
or node groups.
[+]
Taints and
work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node. A node should only schedule a Pod with the matching tolerations for the configured taints.
Toleration
LINK
A core object consisting of three required properties: key, value, and effect. Tolerations enable the scheduling of pods on nodes or node groups that have matching
taints
.
[+]
Tolerations and
taints
work together to ensure that pods are not scheduled onto inappropriate nodes. One or more tolerations are applied to a
pod
. A toleration indicates that the
pod
is allowed (but not required) to be scheduled on nodes or node groups with matching
taints
.
UID
LINK
[+]
Upstream (disambiguation)
LINK
May refer to: core Kubernetes or the source repo from which a repo was forked.
[+]
In the
upstream
to mean the core Kubernetes codebase, which the general ecosystem, other code, or third-party tools rely upon. For example,
community members
may suggest that a feature is moved upstream so that it is in the core codebase instead of in a plugin or third-party tool.
In
GitHub
or
git
: The convention is to refer to a source repo as
upstream
, whereas the forked repo is considered
.
LINK
A kernel feature to emulate root. Used for "rootless containers".
[+]
User namespaces are a Linux kernel feature that allows a non-root user to
emulate superuser ("root") privileges,
for example in order to run containers without being a superuser outside the container.
User namespace is effective for mitigating damage of potential container break-out attacks.
In the context of user namespaces, the namespace is a Linux kernel feature, and not a
namespace
in the Kubernetes sense
of the term.
Volume
LINK
A directory containing data, accessible to the
in a
Pod
.
[+]
A Kubernetes volume lives as long as the Pod that encloses it. Consequently, a volume outlives any containers that run within the Pod, and data in the volume is preserved across container restarts.
See
storage
LINK
A Volume Plugin enables integration of storage within a
Pod
.
[+]
A Volume Plugin lets you attach and mount storage volumes for use by a
Pod
. Volume plugins can be
in tree
or
out of tree
.
In tree
plugins are part of the Kubernetes code repository and follow its release cycle.
Out of tree
plugins are developed independently.
WG (working group)
LINK
Facilitates the discussion and/or implementation of a short-lived, narrow, or decoupled project for a committee,
SIG
, or cross-SIG effort.
[+]
Working groups are a way of organizing people to accomplish a discrete task.
SIGs and working groups
.
Workload
LINK
A workload is an application running on Kubernetes.
[+]
Various core objects that represent di

fferent types or parts of a workload
include the DaemonSet, Deployment, Job, ReplicaSet, and StatefulSet objects.
For example, a workload that has a web server and a database might run the
database in one
and the web server
in a
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Ports and Protocols | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Protocol
Direction
Port Range
Purpose
Used By
TCP
Inbound
6443
All
TCP
Inbound
2379-2380
TCP
Inbound
10250
TCP
Inbound
10259
Self
TCP
Inbound
10257
Self
Protocol
Direction
Port Range
Purpose
Used By
TCP
Inbound
10250
TCP
Inbound
All
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Perform a Rolling Update on a DaemonSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install a

nd Set Up kubectl on Linux
Debug Pods
D

ebug Running Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards a

t the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v1alpha

1)
Tasks
This page shows how to perform a rolling update on a DaemonSet.
minikube
DaemonSet Update Strategy
DaemonSet has two update strategy types:
OnDelete
: With
OnDelete
update strategy, after you update a DaemonSet template, new
DaemonSet pods will
only
be created when you manually delete old DaemonSet
pods. This is the same behavior of DaemonSet in Kubernetes version 1.5 or
before.
: This is the default update strategy.
With
update strategy, after you update a
DaemonSet template, old DaemonSet pods will be killed, and new DaemonSet pods
will be created automatically, in a controlled fashion. At most one pod of
the DaemonSet will be running on each node during the whole update process.
To enable the rolling update feature of a DaemonSet, you must set its
.spec.updateStrategy.type
to
.
You may want to set
.spec.updateStrategy.rollingUpdate.maxUnavailable
(default to 1),
.spec.minReadySeconds
(default to 0) and
.spec.updateStrategy.rollingUpdate.maxSurge
(defaults to 0) as well.
Creating a DaemonSet with
update strategy
This YAML file specifies a DaemonSet with an update strategy as 'RollingUpdate'
controllers/fluentd-daemonset.yaml
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
fluentd-elasticsearch
namespace
:
labels
:
k8s-app
:
fluentd-logging
spec
:
selector
:
:
name
:
:
type
:
:
:
1
template
:
metadata
:
labels
:
name
:
spec
:
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
-
name
:
image
:
quay.io/fluentd_elasticsearch/fluentd:v2.5.2
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
varlib

dockercontainers
mountPath
:
/var/lib/docker/containers
readOnly
:
true
:
30
volumes
:
-
name
:
varlog
hostPath
:
path
:
/var/log
-
name
:
varlibdockercontainers
hostPath
:
path
:
After verifying the update strategy of the DaemonSet manifest, create the DaemonSet:
kubectl create -f https://k8s.io/examples/controllers/fluentd-daemonset.yaml
Alternatively, use
to create the same DaemonSet if you plan to
update the DaemonSet with
.
kubectl apply -f https://k8s.io/examples/controllers/fluentd-daemonset.yaml
Checking DaemonSet
Check the update strategy of your DaemonSet, and make sure it's set to
:
kubectl get ds/fluentd-elasticsearch -o go-template
=
'{{.spec.updateStrategy.type}}{{"\n"}}'
-n kube-system
If you haven't created the DaemonSet in the system, check your DaemonSet
manifest with the following command instead:
kubectl apply -f https://k8s.io/examples/controllers/fluentd-daemonset.yaml --dry-run
=
client -o go-template
=
The output from both commands should be:
If the output isn't
, go back and modify the DaemonSet object or
manifest accordingly.
Updating a DaemonSet template
Any updates to a
DaemonSet
will trigger a rolling
update. Let's update the DaemonSet by applying a new YAML file. This can be done with several different
kubectl
commands.
controllers/fluentd-daemonset-update.yaml
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
spec
:
selector
:
:
name
:
:
type
:
:
:
1
template
:
metadata
:
labels
:
name
:
spec
:
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
-
name
:
image
:
resources
:
limits
:
memory
:
200Mi
requests
:
cpu
:
100m
memory
:
200Mi
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
mountPath
:
readOnly
:
true
:
30
volumes
:
-
name
:
varlog
hostPath
:
path
:
/var/log
-
name
:
hostPath
:
path
:
Declarative commands
If you update DaemonSets using
,
use
:
kubectl apply -f https://k8s.io/examples/controllers/fluentd-daemonset-update.yaml
imperative commands
,
use
:
kubectl edit ds/fluentd-elasticsearch -n kube-system
Updating only the container image
If you only need to update the container image in the DaemonSet template, i.e.
.spec.template.spec.containers[*].image
, use
:
kubectl
set
image ds/fluentd-elasticsearch fluentd-elasticsearch
=
quay.io/fluentd_elasticsearch/fluentd:v2.6.0 -n kube-system
Watching the rolling update status
Finally, watch the rollout status of the latest DaemonSet rolling update:
kubectl rollout status ds/fluentd-elasticsearch -n kube-system
When the rollout is complete, the output is similar to this:
daemonset
"fluentd-elasticsearch"
successfully rolled out
DaemonSet rolling update is stuck
Sometimes, a DaemonSet rolling update may be stuck. Here are some possible
causes:
Some nodes run out of resources
The rollout is stuck because new DaemonSet pods can't be scheduled on at least one
node. This is possible when the node is
running out of resources
.
When this happens, find the nodes that don't have the DaemonSet pods scheduled on
by comparing the ou

tput of
and the output of:
name
=
fluentd-elasticsearch -o wide -n kube-system
Once you've found those nodes, delete some non-DaemonSet pods from the node to
make room for new DaemonSet pods.
Note:
This will cause service disruption when deleted pods are not controlled by any controllers or pods are not
replicated. This does not respect
either.
Broken rollout
If the recent DaemonSet template update is broken, for example, the container is
crash looping, or the container image doesn't exist (often due to a typo),
DaemonSet rollout won't progress.
To fix this, update the DaemonSet template again. New rollout won't be
blocked by previous unhealthy rollouts.
Clock skew
If
is specified in the DaemonSet, clock skew between
master and nodes will make DaemonSet unable to detect the right rollout
progress.
Clean up
Delete DaemonSet from a namespace :
kubectl delete ds fluentd-elasticsearch -n kube-system
See
Performing a rollback on a DaemonSet
See
Creating a DaemonSet to adopt existing DaemonSet pods
Feedback
Yes
No
.
or
.
Last modified July 29, 2022 at 10:29 AM PST:
Promote DaemonSet MaxSurge to GA (02b0dea382)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

TokenRequest | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Reference
kind
metadata
(
)
spec
(
status
(
)
audiences
(string)
(string)
(string)
(string)
(int64)
token
create
POST /api/v1/namespaces/{namespace}/serviceaccounts/{name}/toke

n
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
Feedback
Yes
No
.
or
.
create
Home
Blog
Training
Partners
Community
CC BY 4.0

Stateful Applications | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator patte

rn
Tasks
Debug Pods
Determine the Reason for Pod

 Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Cluster Leve

l
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-

apiserver Configuration (v1alpha1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

kubectl Usage Conventions | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

alpha1)
Reference
kubectl
.
Using
kubectl
-o name
,
-o json
,
-o yaml
,
, or
.
get
,
patch
,
edit
and
replace
status
and
scale
status
For
:v1234
,
v1.2.3
,
:latest
).
flags.
You can use
.
Feedback
Yes
No
.
or
.
Using
kubectl
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademar

ks. For a list of trademarks of The Linux Foundation, please see our

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
2.1:
2.2:
3:
4:
5:
6:
7:
1 - Configure the Aggregation Layer
Configuring the
allows the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.
minikube
.
Note:
There are a few setup requirements for getting the aggregation layer working in your environment to support mutual TLS auth between the proxy and extension apiservers. Kubernetes and the kube-apiserver have multiple CAs, so make sure that the proxy is signed by the aggregation layer CA and not by something else, like the Kubernetes general CA.
Caution:
Reusing the same CA for different client types can negatively impact the cluster's ability to function. For more information, see
CA Reusage and Conflicts
.
Authentication Flow
Unlike Custom Resource Definitions (CRDs), the Aggregation API involves another server - your Extension apiserver - in addition to the standard Kubernetes apiserver. The Kubernetes apiserver will need to communicate with your extension apiserver, and your extension apiserver will need to communicate with the Kubernetes apiserver. In order for this communication to be secured, the Kubernetes apiserver uses x509 certificates to authenticate itself to the extension apiserver.
This section describes how the authentication and authorization flows work, and how to configure them.
The high-level flow is as follows:
Kubernetes apiserver: authenticate the requesting user and authorize their rights to the requested API path.
Kubernetes apiserver: proxy the request to the extension apiserver
Extension apiserver: authenticate the request from the Kubernetes apiserver
Extension apiserver: authorize the request from the original user
Extension apiserver: execute
The rest of this section describes these steps in detail.
The flow can be seen in the following diagram.
.
The source for the above swimlanes can be found in the source of this document.
Kubernetes Apiserver Authentication and Authorization
A request to an API path that is served by an extension apiserver begins the same way as all API requests: communication to the Kubernetes apiserver. This path already has been registered with the Kubernetes apiserver by the extension apiserver.
The user communicates with the Kubernetes apiserver, requesting access to the path. The Kubernetes apiserver uses standard authentication and authorization configured with the Kubernetes apiserver to authenticate the user and authorize access to the specific path.
For an overview of authenticating to a Kubernetes cluster, see
"Authenticating to a Cluster"
. For an overview of authorization of access to Kubernetes cluster resources, see
"Authorization Overview"
.
Everything to this point has been standard Kubernetes API requests, authentication

 and authorization.
The Kubernetes apiserver now is prepared to send the request to the extension apiserver.
Kubernetes Apiserver Proxies the Request
The Kubernetes apiserver now will send, or proxy, the request to the extension apiserver that registered to handle the request. In order to do so, it needs to know several things:
How should the Kubernetes apiserver authenticate to the extension apiserver, informing the extension apiserver that the request, which comes over the network, is coming from a valid Kubernetes apiserver?
How should the Kubernetes apiserver inform the extension apiserver of the username and group for which the original request was authenticated?
In order to provide for these two, you must configure the Kubernetes apiserver using several flags.
Kubernetes Apiserver Client Authentication
The Kubernetes apiserver connects to the extension apiserver over TLS, authenticating itself using a client certificate. You must provide the following to the Kubernetes apiserver upon startup, using the provided flags:
private key file via
signed client certificate file via
certificate of the CA that signed the client certificate file via
valid Common Name values (CNs) in the signed client certificate via
The Kubernetes apiserver will use the files indicated by
--proxy-client-*-file
to authenticate to the extension apiserver. In order for the request to be considered valid by a compliant extension apiserver, the following conditions must be met:
The connection must be made using a client certificate that is signed by the CA whose certificate is in
.
The connection must be made using a client certificate whose CN is one of those listed in
.
Note:
You can set this option to blank as
--requestheader-allowed-names=""
. This will indicate to an extension apiserver that
any
CN is acceptable.
When started with these options, the Kubernetes apiserver will:
Use them to authenticate to the extension apiserver.
Create a configmap in the
extension-apiserver-authentication
, in which it will place the CA certificate and the allowed CNs. These in turn can be retrieved by extension apiservers to validate requests.
Note that the same client certificate is used by the Kubernetes apiserver to authenticate against
all
extension apiservers. It does not create a client certificate per extension apiserver, but rather a single one to authenticate as the Kubernetes apiserver. This same one is reused for all extension apiserver requests.
Original Request Username and Group
When the Kubernetes apiserver proxies the request to the extension apiserver, it informs the extension apiserver of the username and group with which the original request successfully authenticated. It provides these in http headers of its proxied request. You must inform the Kubernetes apiserver of the names of the headers to be used.
the header in which to store the username via
the header in which to store the group via
the prefix to append to all extra headers via
These header names are also placed in the
configmap, so they can be retrieved and used by extension apiservers.
Extension Apiserver Authenticates the Request
The extension apiserver, upon receiving a proxied request from the Kubernetes apiserver, must validate that the request actually did come from a valid authenticating proxy, which role the Kubernetes apiserver is fulfilling. The extension apiserver validates it via:
Retrieve the following from the configmap in
, as described above:
Client CA certificate
List of allowed names (CNs)
Header names for username, group and extra info
Check that the TLS connection was authenticated using a client certificate which:
Was signed 

by the CA whose certificate matches the retrieved CA certificate.
Has a CN in the list of allowed CNs, unless the list is blank, in which case all CNs are allowed.
Extract the username and group from the appropriate headers
If the above passes, then the request is a valid proxied request from a legitimate authenticating proxy, in this case the Kubernetes apiserver.
Note that it is the responsibility of the extension apiserver implementation to provide the above. Many do it by default, leveraging the
k8s.io/apiserver/
package. Others may provide options to override it using command-line options.
In order to have permission to retrieve the configmap, an extension apiserver requires the appropriate role. There is a default role named
extension-apiserver-authentication-reader
in the
namespace which can be assigned.
Extension Apiserver Authorizes the Request
The extension apiserver now can validate that the user/group retrieved from the headers are authorized to execute the given request. It does so by sending a standard
request to the Kubernetes apiserver.
In order for the extension apiserver to be authorized itself to submit the
request to the Kubernetes apiserver, it needs the correct permissions. Kubernetes includes a default
named
that has the appropriate permissions. It can be granted to the extension apiserver's service account.
Extension Apiserver Executes
If the
passes, the extension apiserver executes the request.
Enable Kubernetes Apiserver flags
Enable the aggregation layer via the following
flags. They may have already been taken care of by your provider.
--requestheader-client-ca-file=&lt;path to aggregator CA cert&gt;
--proxy-client-cert-file=&lt;path to aggregator proxy cert&gt;
--proxy-client-key-file=&lt;path to aggregator proxy key&gt;
The Kubernetes apiserver has two client CA options:
Each of these functions independently and can conflict with each other, if not used correctly.
: When a request arrives to the Kubernetes apiserver, if this option is enabled, the Kubernetes apiserver checks the certificate of the request. If it is signed by one of the CA certificates in the file referenced by
, then the request is treated as a legitimate request, and the user is the value of the common name
CN=
, while the group is the organization
O=
. See the
documentation on TLS authentication
.
: When a request arrives to the Kubernetes apiserver, if this option is enabled, the Kubernetes apiserver checks the certificate of the request. If it is signed by one of the CA certificates in the file reference by
, then the request is treated as a potentially legitimate request. The Kubernetes apiserver then checks if the common name
CN=
is one of the names in the list provided by
. If the name is allowed, the request is approved; if it is not, the request is not.
If
both
and
are provided, then the request first checks the
CA and then the
. Normally, different CAs, either root CAs or intermediate CAs, are used for each of these options; regular client requests match against
, while aggregation requests match against
. However, if both use the
same
CA, then client requests that normally would pass via
will fail, because the CA will match the CA in
, but the common name
CN=
will
not
match one of the acceptable common names in
--requestheader-

allowed-names
. This can cause your kubelets and other control plane components, as well as end-users, to be unable to authenticate to the Kubernetes apiserver.
For this reason, use different CA certs for the
option - to authorize control plane components and end-users - and the
option - to authorize aggregation apiserver requests.
Warning:
Do
not
reuse a CA that is used in a different context unless you understand the risks and the mechanisms to protect the CA's usage.
If you are not running kube-proxy on a host running the API server, then you must make sure that the system is enabled with the following
flag:
--enable-aggregator-routing=true
Register APIService objects
You can dynamically configure what client requests are proxied to extension
apiserver. The following is an example registration:
:
kind
:
metadata
:
name
:
&lt;name of the registration object&gt;
spec
:
group
:
&lt;API group name this extension apiserver hosts&gt;
version
:
&lt;API version this extension apiserver hosts&gt;
:
&lt;priority this APIService for this group, see API documentation&gt;
:
&lt;prioritizes ordering of this version within a group, see API documentation&gt;
service
:
namespace
:
&lt;namespace of the extension apiserver service&gt;
name
:
&lt;name of the extension apiserver service&gt;
caBundle
:
&lt;pem encoded ca cert that signs the server cert used by the webhook&gt;
The name of an APIService object must be a valid
.
Contacting the extension apiserver
Once the Kubernetes apiserver has determined a request should be sent to an extension apiserver,
it needs to know how to contact it.
The
service
stanza is a reference to the service for an extension apiserver.
Here is an example of an extension apiserver that is configured to be called on port "1234",
and to verify the TLS connection against the ServerName
:
kind
:
...
spec
:
...
service
:
namespace
:
name
:
port
:
1234
caBundle
:
...
Set up an extension api-server
to work with the aggregation layer.
For a high level overview, see
Extending the Kubernetes API with the aggregation layer
.
Extend the Kubernetes API Using Custom Resource Definitions
.
2 - Use Custom Resources
2.1 - Extend the Kubernetes API with CustomResourceDefinitions
This page shows how to install a
custom resource
into the Kubernetes API by creating a
.
minikube
Your Kubernetes server must be at or later than version 1.16.
.
If you are using an older version of Kubernetes that is still supported, switch to
the documentation for that version to see advice that is relevant for your cluster.
Create a CustomResourceDefinition
When you create a new CustomResourceDefinition (CRD), the Kubernetes API Server
creates a new RESTful resource path for each version you specify. The custom
resource created from a CRD object can be either namespaced or cluster-scoped,
as specified in the CRD's
field. As with existing built-in
objects, deleting a namespace deletes all custom objects in that namespace.
CustomResourceDefinitions themselves are non-namespaced and are available t

o
all namespaces.
For example, if you save the following CustomResourceDefinition to
resourcedefinition.yaml
:
:
kind
:
metadata
:
name
:
crontabs.stable.example.com
spec
:
group
:
stable.example.com
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
and create it:
kubectl apply -f resourcedefinition.yaml
Then a new namespaced RESTful API endpoint is created at:
/apis/stable.example.com/v1/namespaces/*/crontabs/...
This endpoint URL can then be used to create and manage custom objects.
The
kind
of these objects will be
CronTab
from the spec of the
CustomResourceDefinition object you created above.
It might take a few seconds for the endpoint to be created.
You can watch the
Established
condition of your CustomResourceDefinition
to be true or watch the discovery information of the API server for your
resource to show up.
Create custom objects
After the CustomResourceDefinition object has been created, you can create
custom objects. Custom objects can contain custom fields. These fields can
contain arbitrary JSON.
In the following example, the
cronSpec
and
image
custom fields are set in a
custom object of kind
CronTab
. The kind
CronTab
comes from the spec of the
If you save the following YAML to
my-crontab.yaml
:
:
"stable.example.com/v1"
kind
:
CronTab
metadata
:
name
:
my-new-cron-object
spec
:
cronSpec
:
"* * * * */5"
image
:
my-awesome-cron-image
kubectl apply -f my-crontab.yaml
You can then manage your CronTab objects using kubectl. For example:
kubectl get crontab
Should print a list like this:
NAME                 AGE
my-new-cron-object   6s
Resource names are not case-sensitive when using kubectl, and you can use either
the singular or plural forms defined in the CRD, as well as any short names.
You can also view the raw YAML data:
kubectl get ct -o yaml
You should see that it contains the custom
cronSpec
and
image
fields
from the YAML you used to create it:
:
v1
items
:
-
:
stable.example.com/v1
kind
:
CronTab
metadata
:
:
:
|
{
:
,
"kind"
:
"CronTab"
,
:{
:{},
"name"
:
"my-new-cron-object"
,
:
"default"
},
"spec"
:{
"cronSpec"
:
"* * * * */5"
,
"image"
:
"my-awesome-cron-image"
}}
:
"2021-06-20T07:35:27Z"
:
1
name
:
namespace
:
default
:
"1326"
uid
:
9aab1d66-628e-41bb-a422-57b8b3b1f5a9
spec
:
cronSpec
:
'* * * * */5'
image
:
kind
:
List
metadata
:
:
""
selfLink
:
""
Delete a CustomResourceDefinition
When you delete a CustomResourceDefinition, the server will uninstall the RESTful API endpoint
and delete all custom objects stored in it.
kubectl delete -f resourcedefinition.yaml
kubectl get crontabs
Error from server (NotFound): Unable to list {"stable.example.com" "v1" "crontabs"}: the server could not
find th

e requested resource (get crontabs.stable.example.com)
If you later recreate the same CustomResourceDefinition, it will start out empty.
Specifying a structural schema
CustomResources store structured data in custom fields (alongside the built-in
fields
,
kind
and
metadata
, which the API server validates
implicitly). With
OpenAPI v3.0 validation
a schema can be
specified, which is validated during creation and updates, compare below for
details and limits of such a schema.
With
the definition of a structural schema is
mandatory for CustomResourceDefinitions. In the beta version of
CustomResourceDefinition, the structural schema was optional.
A structural schema is an
OpenAPI v3.0 validation schema
which:
specifies a non-empty type (via
type
in OpenAPI) for the root, for each specified field of an object node
(via
or
in OpenAPI) and for each item in an array node
(via
items
in OpenAPI), with the exception of:
a node with
x-kubernetes-int-or-string: true
a node with
for each field in an object and each item in an array which is specified within any of
allOf
,
anyOf
,
oneOf
or
not
, the schema also specifies the field/item outside of those logical junctors (compare example 1 and 2).
does not set
,
type
,
default
,
,
nullable
within an
allOf
,
anyOf
,
oneOf
or
not
, with the exception of the two pattern for
(see below).
if
metadata
is specified, then only restrictions on
and
are allowed.
Non-structural example 1:
allOf:
- properties:
    foo:
      ...
conflicts with rule 2. The following would be correct:
  foo:
    ...
allOf:
    foo:
      ...
Non-structural example 2:
allOf:
- items:
    properties:
      foo:
        ...
items:
  properties:
    foo:
      ...
allOf:
- items:
      foo:
        ...
Non-structural example 3:
  foo:
    pattern: "abc"
  metadata:
    type: object
      name:
        type: string
        pattern: "^a"
      finalizers:
        type: array
        items:
          type: string
          pattern: "my-finalizer"
anyOf:
    bar:
      type: integer
      minimum: 42
  required: ["bar"]
  description: "foo bar object"
is not a structural schema because of the following violations:
the type at the root is missing (rule 1).
the type of
foo
is missing (rule 1).
bar
inside of
anyOf
is not specified outside (rule 2).
bar
's
type
is within
anyOf
(rule 3).
the description is set within
anyOf
(rule 3).
might not be restricted (rule 4).
In contrast, the following, corresponding schema is structural:
type
:
object
:
"foo bar object"
:
foo
:
type
:
string
pattern
:
"abc"
bar
:
type
:
integer
metadata
:
type
:
object
:
name
:
type
:
string
pattern
:
"^a"
anyOf
:
-
:
bar
:
minimum
:
42
required
:
[
"bar"
]
Violations of the structural schema rules are reported in the
NonStructural
condition in the
CustomResourceDefinition.
Field pruning
CustomResourceDefinitions store validated resource data in the cluster's persistence store,
etcd
.
As with native Kubernetes resources such as
ConfigMap
,
if you specify a field that the API server does not recognize, the unknown field is
pruned
(removed) before being persisted.
CRDs converted from
to
might lack
structural schemas, and
spec.preserveUnknownFields
might be
true
.
For legacy CustomResourceDefinition objects created as
with
set to
true
, the following is also true:
Pruning is not enabled.
You can store arbitrary data.
For compatibility with
, update your custom
resource definitions to:
Use a structural OpenAPI schema.
Set
spec.p

reserveUnknownFields
to
false
.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
someRandomField
:
42
kubectl create --validate
=
false
-f my-crontab.yaml -o yaml
Your output is similar to:
:
kind
:
CronTab
metadata
:
:
2017-05-31T12:56:35Z
:
1
name
:
namespace
:
default
:
"285"
uid
:
9423255b-4600-11e7-af6a-28d2447dc82b
spec
:
cronSpec
:
'* * * * */5'
image
:
Notice that the field
was pruned.
This example turned off client-side validation to demonstrate the API server's behavior, by adding
the
--validate=false
command line option.
Because the
OpenAPI validation schemas are also published
to clients,
kubectl
also checks for unknown fields and rejects those objects well before they
would be sent to the API server.
Controlling pruning
By default, all unspecified fields for a custom resource, across all versions, are pruned. It is possible though to
opt-out of that for specifc sub-trees of fields by adding
in the
structural OpenAPI v3 validation schema
.
type
:
object
:
json
:
:
true
The field
json
can store any JSON value, without anything being pruned.
You can also partially specify the permitted JSON; for example:
type
:
object
:
json
:
:
true
type
:
object
:
this is arbitrary JSON
With this, only
object
type values are allowed.
Pruning is enabled again for each specified property (or
):
type
:
object
:
json
:
:
true
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
bar
:
type
:
string
With this, the value:
json
:
spec
:
foo
:
abc
bar
:
def
something
:
x
status
:
something
:
x
is pruned to:
json
:
spec
:
foo
:
abc
bar
:
def
status
:
something
:
x
This means that the
something
field in the specified
spec
object is pruned, but everything outside is not.
IntOrString
Nodes in a schema with
are excluded from rule 1, such that the
following is structural:
type
:
object
:
foo
:
:
true
Also those nodes are partially excluded from rule 3 in the sense that the following two patterns are allowed
(exactly those, without variations in order to additional fields):
anyOf:
  - type: integer
  - type: string
...
and
allOf:
  - anyOf:
      - type: integer
      - type: string
  - ... # zero or more
...
With one of those specification, both an integer and a string validate.
In
Validation Schema Publishing
,
is unfolded to one of the two patterns shown above.
RawExtension
RawExtensions (as in
runtime.RawExtension
)
holds complete Kubernetes objects, i.e. with
and
kind
fields.
It is possible to specify those embedded objects (both completely without constraints or partially specified)
by setting
x-kubernetes-embedded-resource: true
type
:
object
:
foo
:
:
true
:
true
Here, the field
foo
holds a complete object, e.g.:
foo:
  apiVersion: v1
  kind: Pod
  spec:
    ...
Because
is specified alongside, nothing is pruned.
The use of
is optional though.
With
, the
,
kind
and
metadata
are implicitly specified and validated.
Serving multiple versions of a CRD
See
Custom resource definition versioning
for more information about serving multiple versions of your
CustomResourceDefinition and migratin

g your objects from one version to another.
Advanced topics
allow controllers to implement asynchronous pre-delete hooks.
Custom objects support finalizers similar to built-in objects.
You can add a finalizer to a custom object like this:
:
kind
:
CronTab
metadata
:
:
- stable.example.com/finalizer
Identifiers of custom finalizers consist of a domain name, a forward slash and the name of
the finalizer. Any controller can add a finalizer to any object's list of finalizers.
The first delete request on an object with finalizers sets a value for the
field but does not delete it. Once this value is set,
entries in the
list can only be removed. While any finalizers remain it is also
impossible to force the deletion of an object.
When the
field is set, controllers watching the object execute any
finalizers they handle and remove the finalizer from the list after they are done. It is the
responsibility of each controller to remove its finalizer from the list.
metadata.deletionGracePeriodSeconds
controls the interval between polling updates.
Once the list of finalizers is empty, meaning all finalizers have been executed, the resource is
deleted by Kubernetes.
Custom resources are validated via
OpenAPI v3 schemas
,
by x-kubernetes-validations when the
Validation Rules feature
is enabled, and you
can add additional validation using
admission webhooks
.
Additionally, the following restrictions are applied to the schema:
These fields cannot be set:
,
,
,
discriminator
,
id
,
,
readOnly
,
writeOnly
,
xml
,
$ref
.
The field
cannot be set to
true
.
The field
false
.
The field
is mutually exclusive with
.
The
extension can be used to validate custom resources using
expressions when the
Validation rules
feature is enabled and the CustomResourceDefinition schema is a
.
structural schemas
section for other
restrictions and CustomResourceDefinition features.
The schema is defined in the CustomResourceDefinition. In the following example, the
CustomResourceDefinition applies the following validations on the custom object:
spec.cronSpec
must be a string and must be of the form described by the regular expression.
must be an integer and must have a minimum value of 1 and a maximum value of 10.
Save the CustomResourceDefinition to
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
# openAPIV3Schema is the schema for validating custom objects.
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
'^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
A request to create a custom object of kind CronTab is rejected if there are invalid values in its fields.
In the following example, the custom object contains fields with invalid values:
does not match the regular expression.
is greater than 10.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * *"
image
:
replicas
:
15
and attempt to create it:
then you get an error:
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stab

le.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
validation failure list:
spec.cronSpec in body should match '^(\d+|\*)(/\d+)?(\s+(\d+|\*)(/\d+)?){4}$'
spec.replicas in body should be less than or equal to 10
If the fields contain valid values, the object creation request is accepted.
Save the following YAML to
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
5
And create it:
crontab
created
Validation rules are in beta since 1.25 and the
is enabled by default to
validate custom resource based on
validation rules
. You can disable this feature by explicitly
setting the
feature gate to
false
, for the
component. This
feature is only available if the schema is a
.
Validation rules use the
to validate custom resource values. Validation rules are included in
CustomResourceDefinition schemas using the
The Rule is scoped to the location of the
extension in the schema.
And
self
variable in the CEL expression is bound to the scoped value.
All validation rules are scoped to the current object: no cross-object or stateful validation
rules are supported.
...
  openAPIV3Schema:
      spec:
        type: object
        x-kubernetes-validations:
          - rule: "self.minReplicas &lt;= self.replicas"
            message: "replicas should be greater than or equal to minReplicas."
          - rule: "self.replicas &lt;= self.maxReplicas"
            message: "replicas should be smaller than or equal to maxReplicas."
        properties:
          ...
          minReplicas:
            type: integer
          replicas:
          maxReplicas:
        required:
          - minReplicas
          - replicas
          - maxReplicas
will reject a request to create this custom resource:
:
kind
:
CronTab
metadata
:
name
:
spec
:
:
0
replicas
:
20
:
10
with the response:
The CronTab "my-new-cron-object" is invalid:
* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: replicas should be smaller than or equal to maxReplicas.
could have multiple rules.
The
rule
under
The
message
represents the message displayed when validation fails. If message is unset, the
above response would be:
* spec: Invalid value: map[string]interface {}{"maxReplicas":10, "minReplicas":0, "replicas":20}: failed rule: self.replicas &lt;= self.maxReplicas
Validation rules are compiled when CRDs are created/updated.
The request of CRDs create/update will fail if compilation of validation rules fail.
Compilation process includes type checking as well.
The compilation failure:
no_matching_overload
: this function has no overload for the types of the arguments.
For example, a rule like
self == true
against a field of integer type will get error:
Invalid value: apiextensions.ValidationRule{Rule:"self == true", Message:""}: compilation failed: ERROR: \&lt;input&gt;:1:6: found no matching overload for '_==_' applied

 to '(int, bool)'
no_such_field
: does not contain the desired field.
self.nonExistingField &gt; 0
against a non-existing field will return
the following error:
Invalid value: apiextensions.ValidationRule{Rule:"self.nonExistingField &gt; 0", Message:""}: compilation failed: ERROR: \&lt;input&gt;:1:5: undefined field 'nonExistingField'
invalid argument
: invalid argument to macros.
has(self)
will return error:
Invalid value: apiextensions.ValidationRule{Rule:"has(self)", Message:""}: compilation failed: ERROR: &lt;input&gt;:1:4: invalid argument to has() macro
Validation Rules Examples:
Rule
Purpose
self.minReplicas &lt;= self.replicas &amp;&amp; self.replicas &lt;= self.maxReplicas
'Available' in self.stateCounts
(size(self.list1) == 0) != (size(self.list2) == 0)
!('MY_KEY' in self.map1) || self['MY_KEY'].matches('^[a-zA-Z]*$')
self.envars.filter(e, e.name == 'MY_ENV').all(e, e.value.matches('^[a-zA-Z]*$')
has(self.expired) &amp;&amp; self.created + self.ttl &lt; self.expired
self.health.startsWith('ok')
self.widgets.exists(w, w.key == 'x' &amp;&amp; w.foo &lt; 10)
type(self) == string ? self == '100%' : self == 1000
self.metadata.name.startsWith(self.prefix)
self.set1.all(e, !(e in self.set2))
size(self.names) == size(self.details) &amp;&amp; self.names.all(n, n in self.details)
size(self.clusters.filter(c, c.name == self.primary)) == 1
Xref:
If the Rule is scoped to the root of a resource, it may make field selection into any fields
declared in the OpenAPIv3 schema of the CRD as well as
,
kind
,
and
. This includes selection of fields in both the
spec
and
status
in the
same expression:
...
    x-kubernetes-validations:
      - rule: "self.status.availableReplicas &gt;= self.spec.minReplicas"
        spec:
          type: object
          properties:
            minReplicas:
              type: integer
            ...
        status:
            availableReplicas:
If the Rule is scoped to an object with properties, the accessible properties of the object are field selectable
via
. Null valued fields are treated as
absent fields in CEL expressions.
...
      spec:
          - rule: "has(self.foo)"
          ...
          foo:
If the Rule is scoped to an object with additionalProperties (i.e. a map) the value of the map
are accessible via
and all
entries of the map are accessible via CEL macros and functions such as
.
...
      spec:
          - rule: "self['xyz'].foo &gt; 0"
        additionalProperties:

          ...
            foo:
If the Rule is scoped to an array, the elements of the array are accessible via
self[i]
and
also by macros and functions.
...
      ...
      foo:
          - rule: "size(self) == 1"
        items:
If the Rule is scoped to a scalar,
self
is bound to the scalar value.
...
      spec:
          ...
          foo:
            x-kubernetes-validations:
            - rule: "self &gt; 0"
Examples:
type of the field rule scoped to
Rule example
root object
self.status.actual &lt;= self.spec.maxDesired
map of objects
self.components['Widget'].priority &lt; 10
list of integers
self.values.all(value, value &gt;= 0 &amp;&amp; value &lt; 100)
string
The
,
kind
,
and
the root of the object and from any
annotated objects. No other
metadata properties are accessible.
Unknown data preserved in custom resources via
is not
accessible in CEL expressions. This includes:
Unknown field values that are preserved by object schemas with
.
Object properties where the property schema is of an "unknown type". An "unknown type" is
recursively defined as:
Accessible property names are escaped according to the following rules when accessed in the expression:
__
__dot__
.
__dash__
-
__slash__
/
Note: CEL RESERVED keyword needs to match the exact property name to be escaped (e.g. int in the word sprint would not be escaped).
namespace
x-prop
redact__d
string
Equality on arrays with
of
set
or
map
ignores element order,
i.e.,
[1, 2] == [2, 1]
. Concatenation on arrays with x-kubernetes-list-type use the semantics of
the list type:
set
:
X + Y
X
are preserved
and non-intersecting elements in
Y
map
:
X + Y
X
are preserved but
the values are overwritten by values in
Y
X
and
Y
intersect. Elements
in
Y
with non-intersecting keys are appended, retaining their partial order.
Here is the declarations type mapping between OpenAPIv3 and CEL type:
OpenAPIv3 type
CEL type
'object' with Properties
object / "message type"
'object' with AdditionalProperties
map
'object' with x-kubernetes-embedded-type
object / "message type", 'apiVersion', 'kind', 'metadata.name' and 'metadata.generateName' are implicitly included in schema
'object' with x-kubernetes-preserve-unknown-fields
object / "message type", unknown fields are NOT accessible in CEL expression
dynamic object that is either an int or a string,
type(value)
can be used to check the type
'array
list
'array' with x-kubernetes-list-type=map
list with map based Equality &amp; unique key guarantees
'array' with x-kubernetes-list-type=set
list with set based Equality &amp; unique entry guarantees
'boolean'
boolean
'number' (all formats)
double
'integer' (all formats)
int (64)
'null'
null_type
'string'
string
'str

ing' with format=byte (base64 encoded)
bytes
'string' with format=date
timestamp (google.protobuf.Timestamp)
'string' with format=datetime
'string' with format=duration
duration (google.protobuf.Duration)
xref:
CEL types
,
OpenAPI types
,
Kubernetes Structural Schemas
.
Validation functions
Functions available include:
CEL standard functions, defined in the
list of standard definitions
CEL standard
macros
CEL
extended string function library
CEL extension library
Transition rules
A rule that contains an expression referencing the identifier
oldSelf
is implicitly considered a
transition rule
. Transition rules allow schema authors to prevent certain transitions between two
otherwise valid states. For example:
type
:
string
enum
:
[
"low"
,
"medium"
,
"high"
]
:
-
rule
:
"!(self == 'high' &amp;&amp; oldSelf == 'low') &amp;&amp; !(self == 'low' &amp;&amp; oldSelf == 'high')"
message
:
cannot transition directly between 'low' and 'high'
Unlike other rules, transition rules apply only to operations meeting the following criteria:
The operation updates an existing object. Transition rules never apply to create operations.
Both an old and a new value exist. It remains possible to check if a value has been added or
removed by placing a transition rule on the parent node. Transition rules are never applied to
custom resource creation. When placed on an optional field, a transition rule will not apply to
update operations that set or unset the field.
The path to the schema node being validated by a transition rule must resolve to a node that is
comparable between the old object and the new object. For example, list items and their
descendants (
spec.foo[10].bar
) can't necessarily be correlated between an existing object and a
later update to the same object.
Errors will be generated on CRD writes if a schema node contains a transition rule that can never be
applied, e.g. "
path
: update rule
rule
cannot be set on schema because the schema or its parent
schema is not mergeable".
Transition rules are only allowed on
correlatable portions
of a schema.
A portion of the schema is correlatable if all
array
parent schemas are of type
x-kubernetes-list-type=map
;
any
set
or
atomic
array parent schemas make it impossible to unambiguously correlate a
self
with
oldSelf
.
Here are some examples for transition rules:
Transition rules examples
Use Case
Rule
Immutability
self.foo == oldSelf.foo
Prevent modification/removal once assigned
oldSelf != 'bar' || self == 'bar'
or
!has(oldSelf.field) || has(self.field)
Append-only set
self.all(element, element in oldSelf)
If previous value was X, new value can only be A or B, not Y or Z
oldSelf != 'X' || self in ['A', 'B']
Monotonic (non-decreasing) counters
self &gt;= oldSelf
Resource use by validation functions
When you create or update a CustomResourceDefinition that uses validation rules,
the API server checks the likely impact of running those validation rules. If a rule is
estimated to be prohibitively expensive to execute, the API server rejects the create
or update operation, and returns an error message.
A similar system is used at runtime that observes the actions the interpreter takes. If the interpreter executes
too many instructions, execution of the rule will be halted, and an error will result.
Each CustomResourceDefinition is also allowed a certain amount of resources to finish executing all of
its validation rules. If the sum total of its rules are estimated at creation time to go over that limit,
then a validation error will also occur.
You are unlikely to encounter issues with the resource budget for validation if you only
specify rules that always take the same amount of time regardless of how large their input is.
For example, a rule that asserts that
self.foo == 1
does not by itself have any
risk of rejection on validation resource budget groups.
But if
foo
is a string and you define a validation rule
self.foo.con

tains("someString")
, that rule takes
longer to execute depending on how long
foo
is.
Another example would be if
foo
were an array, and you specified a validation rule
self.foo.all(x, x &gt; 5)
.
The cost system always assumes the worst-case scenario if a limit on the length of
foo
is not
given, and this will happen for anything that can be iterated over (lists, maps, etc.).
Because of this, it is considered best practice to put a limit via
maxItems
,
, and
maxLength
for anything that will be processed in a validation rule in order to prevent validation
errors during cost estimation. For example, given this schema with one rule:
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
string
:
-
rule
:
"self.all(x, x.contains('a string'))"
then the API server rejects this rule on validation budget grounds with error:
spec.validation.openAPIV3Schema.properties[spec].properties[foo].x-kubernetes-validations[0].rule: Forbidden: 
CEL rule exceeded budget by more than 100x (try simplifying the rule, or adding maxItems, maxProperties, and 
maxLength where arrays, maps, and strings are used)
The rejection happens because
self.all
implies calling
contains()
on every string in
foo
,
which in turn will check the given string to see if it contains
'a string'
. Without limits, this
is a very expensive rule.
If you do not specify any validation limit, the estimated cost of this rule will exceed the
per-rule cost limit. But if you add limits in the appropriate places, the rule will be allowed:
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
maxLength
:
10
:
-
rule
:
The cost estimation system takes into account how many times the rule will be executed in addition to the
estimated cost of the rule itself. For instance, the following rule will have the same estimated cost as the
previous example (despite the rule now being defined on the individual array items):
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
:
-
rule
:
"self.contains('a string'))"
maxLength
:
10
If a list inside of a list has a validation rule that uses
self.all
, that is significantly more expensive
than a non-nested list with the same rule. A rule that would have been allowed on a non-nested list might need
lower limits set on both nested lists in order to be allowed. For example, even without having limits set,
the following rule is allowed:
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
integer
:
-
rule
:
"self.all(x, x == 5)"
But the same rule on the following schema (with a nested array added) produces a validation error:
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
array
items
:
type
:
integer
:
-
rule
:
This is because each item of
foo
is itself an array, and each subarray in turn calls
self.all
.
Avoid nested lists and maps if possible where validation rules are used.
Defaulting
Note:
To use defaulting, your CustomResourceDefinition must use API version
.
Defaulting allows to specify default values in the
OpenAPI v3 validation schema
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
default
:
"5 0 * * *"
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
default
:
1
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct

With this both
cronSpec
and
replicas
are defaulted:
:
kind
:
CronTab
metadata
:
name
:
spec
:
image
:
leads to
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"5 0 * * *"
image
:
replicas
:
1
Defaulting happens on the object
in the request to the API server using the request version defaults,
when reading from etcd using the storage version defaults,
after mutating admission plugins with non-empty patches using the admission webhook object version defaults.
Defaults applied when reading data from etcd are not automatically written back to etcd.
An update request via the API is required to persist those defaults back into etcd.
Default values must be pruned (with the exception of defaults for
metadata
fields) and must
validate against a provided schema.
Default values for
metadata
fields of
x-kubernetes-embedded-resources: true
nodes (or parts of
a default value covering
metadata
) are not pruned during CustomResourceDefinition creation, but
through the pruning step during handling of requests.
Defaulting and Nullable
Null values for fields that either don't specify the nullable flag, or give it a
false
value, will be pruned before defaulting happens. If a default is present, it will be
applied. When nullable is
true
, null values will be conserved and won't be defaulted.
For example, given the OpenAPI schema below:
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
nullable
:
false
default
:
"default"
bar
:
type
:
string
nullable
:
true
baz
:
type
:
string
creating an object with null values for
foo
and
bar
and
baz
spec
:
foo
:
null
bar
:
null
baz
:
null
leads to
spec
:
foo
:
"default"
bar
:
null
with
foo
pruned and defaulted because the field is non-nullable,
bar
maintaining the null
value due to
nullable: true
, and
baz
pruned because the field is non-nullable and has no
default.
Publish Validation Schema in OpenAPI v2
OpenAPI v3 validation schemas
which are
structural
and
enable pruning
are published
OpenAPI v2 spec
from Kubernetes API server.
The
kubectl
command-line tool consumes the published schema to perform
client-side validation (
and
), schema explanation (
)
on custom resources. The published schema can be consumed for other purposes as well, like client generation or documentation.
The OpenAPI v3 validation schema is converted to OpenAPI v2 schema, and
show up in
and
paths
fields in the
.
The following modifications are applied during the conversion to keep backwards compatibility with
kubectl in previous 1.13 version. These modifications prevent kubectl from being over-strict and rejecting
valid OpenAPI schemas that it doesn't understand. The conversion won't modify the validation schema defined in CRD,
and therefore won't affect
in the API server.
The following fields are removed as they aren't supported by OpenAPI v2
(in future versions OpenAPI v3 will be used without these restrictions)
The fields
allOf
,
anyOf
,
oneOf
and
not
are removed
If
is set, we drop
type
,
nullable
,
items
and
because OpenAPI v2 is
not able to express nullable. To avoid kubectl to reject good objects, this is necessary.
Additional printer columns
The kubectl tool relies on server-side output formatting. Your cluster's API server decides which
columns are shown by the
command. You can customize these columns for a
CustomResourceDefinition. The following example adds the
Spec
,
Replicas
, and
Age
columns.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
shor

tNames
:
- ct
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
:
-
name
:
Spec
type
:
string
:
The cron spec defining the interval a CronJob is run
jsonPath
:
.spec.cronSpec
-
name
:
Replicas
type
:
integer
:
The number of jobs launched by the CronJob
jsonPath
:
-
name
:
Age
type
:
date
jsonPath
:
Create the CustomResourceDefinition:
Create an instance using the
from the previous section.
Invoke the server-side printing:
kubectl get crontab my-new-cron-object
Notice the
NAME
,
SPEC
,
REPLICAS
, and
AGE
columns in the output:
NAME                 SPEC        REPLICAS   AGE
my-new-cron-object   * * * * *   1          7s
Note:
The
NAME
column is implicit and does not need to be defined in the CustomResourceDefinition.
Priority
Each column includes a
priority
field. Currently, the priority
differentiates between columns shown in standard view or wide view (using the
-o wide
flag).
Columns with priority
0
are shown in standard view.
Columns with priority greater than
0
are shown only in wide view.
Type
A column's
type
field can be any of the following (compare
OpenAPI v3 data types
):
integer
– non-floating-point numbers
number
– floating point numbers
string
– strings
boolean
–
true
or
false
date
– rendered differentially as time since this timestamp.
If the value inside a CustomResource does not match the type specified for the column,
the value is omitted. Use CustomResource validation to ensure that the value
types are correct.
Format
A column's
format
field can be any of the following:
int32
int64
float
double
byte
date
date-time
password
The column's
format
controls the style used when
kubectl
prints the value.
Custom resources support
/status
and
/scale
subresources.
The status and scale subresources can be optionally enabled by
defining them in the CustomResourceDefinition.
Status subresource
When the status subresource is enabled, the
/status
subresource for the custom resource is exposed.
The status and the spec stanzas are represented by the
.status
and
.spec
JSONPaths
respectively inside of a custom resource.
PUT
requests to the
/status
subresource take a custom resource object and ignore changes to
anything except the status stanza.
PUT
/status
subresource only validate the status stanza of the custom
resource.
PUT
/
POST
/
PATCH
requests to the custom resource ignore changes to the status stanza.
The
.metadata.generation
value is incremented for all changes, except for changes to
.metadata
or
.status
.
Only the following constructs are allowed at the root of the CRD OpenAPI validation schema:
example
format
items
maximum
maxItems
maxLength
minimum
minItems
minLength
pattern
required
title
type
Scale subresource
When the scale subresource is enabled, the
/scale
The
autoscaling/v1.Scale
object is sent as the payload for
/scale
.
To enable the scale subresource, the following fields are defined in the CustomResourceDefinition.
specReplicasPath
defines the JSONPath inside of a custom resource that corresponds to
scale.spec.replicas
.
It is a required value.
Only JSONPaths under
.spec
and with the dot notation are allowed.
If there is no value under the
in the custom resource,
the
/scale
statusReplicasPath
scale.status.replicas
.
.status
the status replica value in the

/scale
labelSelectorPath
Scale.Status.Selector
.
It is an optional value.
It must be set to work with HPA.
.status
or
.spec
the status selector value in the
/scale
The field pointed by this JSON path must be a string field (not a complex selector struct)
which contains a serialized label selector in string form.
In the following example, both status and scale subresources are enabled.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
status
:
type
:
object
:
replicas
:
type
:
integer
:
type
:
string
# subresources describes the subresources for custom resources.
:
# status enables the status subresource.
status
:
{}
# scale enables the scale subresource.
scale
:
# specReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Spec.Replicas.
:
# statusReplicasPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Replicas.
:
.status.replicas
# labelSelectorPath defines the JSONPath inside of a custom resource that corresponds to Scale.Status.Selector.
:
.status.labelSelector
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
After the CustomResourceDefinition object has been created, you can create custom objects.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
3
Then new namespaced RESTful API endpoints are created at:
/apis/stable.example.com/v1/namespaces/*/crontabs/status
and
/apis/stable.example.com/v1/namespaces/*/crontabs/scale
A custom resource can be scaled using the
command.
For example, the following command sets
of the
custom resource created above to 5:
=
5
crontabs/my-new-cron-object
crontabs
scaled
kubectl get crontabs my-new-cron-object -o
jsonpath
=
'{.spec.replicas}'
5
to protect custom
resources that have the scale subresource enabled.
Categories
Categories is a list of grouped resources the custom resource belongs to (eg.
all
).
You can use
kubectl get &lt;category-name&gt;
to list the resources belonging to the category.
The following example adds
all
in the list of categories in the CustomResourceDefinition
and illustrates how to output the custom resource using
.
Save the following CustomResourceDefinition to
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
# categories is a list of grouped resources the custom resource belongs to.
categories
:
- all
After the CustomResourceDefinition

 object has been created, you can create custom objects.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
You can specify the category when using
:
and it will include the custom resources of kind
CronTab
:
NAME                          AGE
crontabs/my-new-cron-object   3s
Read about
.
See
.
Serve
multiple versions
of a
2.2 - Versions in CustomResourceDefinitions
minikube
.
.
Overview
v1beta1
v1beta1
API.
v1
.
None
the
list with
. Also, set
field
to the selected conversion strategy. If using a conversion webhook, configure
See the
section.
Set
served
to
false
list. If
errors attempting to a

ccess the custom resource objects at the old version.
Ensure the
storage
is set to
true
.
list.
versions
Note:
In
version
versions
. The
version
versions
field.
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
None
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
-
name
:
v1
served
:
true
storage
:
false
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
# None

 conversion assumes the same schema for all versions and only sets the apiVersion
strategy
:
None
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
and
.
name
v
beta
or
alpha
like
v2
or
v2beta1
do not.
beta
or
alpha
beta
or
alpha
beta
, or
alpha
foo1
foo10
- v10
- v2
- v1
- v11beta2
- v10beta3
- v3beta1
- foo1
- foo10
, the
v1
v1beta1
v1
When API requests to a deprecated version of that resource are made, a warning message is returned in the API response as a header.
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
:
true
# This overrides the default warning returned to API c

lients making v1alpha1 API requests.
:
schema
:
...
-
name
:
v1beta1
served
:
true
:
true
schema
:
...
-
name
:
v1
served
:
true
storage
:
true
schema
:
...
:
kind
:
metadata
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
:
...
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
:
true
:
-
name
:
v1beta1
served
:
true
:
true
-
name
:
v1
served
:
true
storage
:
true
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1beta1
served
:
false
schema
:
...
-
name
:
v1
served
:
true
storage
:
true
schema
:
...
Note:
field
custom resource is requested in a different version than stored version.
server
changing the

 order of objects.
Note:
field
empty
,
how to
.
metadata
other than
labels
and
.
name
,
UID
and
namespace
.
named
in
default
.
Note:
The
None
spec
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
Webhook
webhook
:
# The webhook must respond with a ConversionReview object in the same version it received.
:
[
"v1"
,
"v1beta1"
]
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
kind
:
metadata
:
# name must match the spec fields

 below, and be in the form: &lt;plural&gt;.&lt;group&gt;
name
:
spec
:
group
:
:
false
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
Webhook
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
URL
url
(
).
The
host
service
(i.e.,
host
localhost
or
127.0.0.1
as a
host
is
Attempting to use a user or basic auth (for example "user:password@") is not allowed.
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
url
:
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
url
:
...
The
service
webh

ookClientConfig
service
instead of
url
.
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
caBundle
:
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
caBundle
:
...
Request
,
with a
API group
with the
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
[
"v1"
,
"v1beta1"
]
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
[
"v1"
,
"v1beta1"
]
...
If no
v1beta1
.
object
CronTab
objects to
:
{
,
"kind":
,
{
"uid":
,
,
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":

},
}
]
}
}
{
,
"kind":
,
{
"uid":
,
,
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
}
]
}
}
Response
,
with the
response
response
uid
result
, set to
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Success"
},
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
"host":
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
,
"port":
"2345"
}
]
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
},
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
"host":
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
,

"port":
"2345"
}
]
}
}
response
uid
result
, set to
Warning:
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
,
}
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
,
}
}
}
:
strategy
value
None
pruning
if
v1beta1
v1beta1
v1
v1
and
v1beta1
which is typically the case when promoting an API to stable in the
v1beta1
version
v1
field.
v1
. You now
v1beta1
v1
.
v1
since that
When de

precating versions and dropping support, select a storage upgrade
Option 1:
Run the
field.
Option 2:
v1beta1
to
v1
.
Set
v1
is now
.
v1
.
Remove
v1beta1
field.
Note:
The flag
status
and
scale
status
kubectl
tool
status
.
status
kubectl
:
=
'status'
--type
=
'merge'
-p
3 - Set up an Extension API Server
Setting up an extension API server to work with the aggregation layer allows the Kubernetes apiserver to be extended with additional APIs, which are not part of the core Kubernetes APIs.
minikube
.
You must
configure the aggregation layer
and enable the apiserver flags.
Set up an extension api-server to work with the aggregation layer
The following steps describe how to set up an extension-apiserver
at a high level
. These steps apply regardless if you're using YAML configs or using APIs. An attempt is made to specifically identify any differences between the two. For a concrete example of how they can be implemented using YAML configs, you can look at the
sample-apiserver
in the Kubernetes repo.
Alternatively, you can use an existing 3rd party solution, such as
apiserver-builder
, which should generate a skeleton and automate all of the following steps for you.
Make sure the APIService API is enabled (check
). It should be on by default, unless it's been deliberately turned off in your cluster.
You may need to make an RBAC rule allowing you to add APIService objects, or get your cluster administrator to make one. (Since API extensions affect the entire cluster, it is not recommended to do testing/development/debug of an API extension in a live cluster.)
Create the Kubernetes namespace you want to run your extension api-service in.
Create/get a CA cert to be used to sign the server cert the extension api-server uses for HTTPS.
Create a server cert/key for the api-server to use for HTTPS. This cert should be signed by the above CA. It should also have a CN of the Kube DNS name. This is derived from the Kubernetes service and be of the form
&lt;service name&gt;.&lt;service name namespace&gt;.svc
Create a Kubernetes secret with t

he server cert/key in your namespace.
Create a Kubernetes deployment for the extension api-server and make sure you are loading the secret as a volume. It should contain a reference to a working image of your extension api-server. The deployment should also be in your namespace.
Make sure that your extension-apiserver loads those certs from that volume and that they are used in the HTTPS handshake.
Create a Kubernetes service account in your namespace.
Create a Kubernetes cluster role for the operations you want to allow on your resources.
Create a Kubernetes cluster role binding from the service account in your namespace to the cluster role you created.
Create a Kubernetes cluster role binding from the service account in your namespace to the
cluster role to delegate auth decisions to the Kubernetes core API server.
Create a Kubernetes role binding from the service account in your namespace to the
role. This allows your extension api-server to access the
configmap.
Create a Kubernetes apiservice. The CA cert above should be base64 encoded, stripped of new lines and used as the spec.caBundle in the apiservice. This should not be namespaced. If using the
kube-aggregator API
, only pass in the PEM encoded CA bundle because the base 64 encoding is done for you.
Use kubectl to get your resource. When run, kubectl should return "No resources found.". This message
indicates that everything worked but you currently have no objects of that resource type created.
Walk through the steps to
configure the API aggregation layer
.
Extend the Kubernetes API using Custom Resource Definitions
.
4 - Configure Multiple Schedulers
Kubernetes ships with a default scheduler that is described
here
.
If the default scheduler does not suit your needs you can implement your own scheduler.
Moreover, you can even run multiple schedulers simultaneously alongside the default
scheduler and instruct Kubernetes what scheduler to use for each of your pods. Let's
learn how to run multiple schedulers in Kubernetes with an example.
A detailed description of how to implement a scheduler is outside the scope of this
document. Please refer to the kube-scheduler implementation in
pkg/scheduler
in the Kubernetes source directory for a canonical example.
minikube
.
Package the scheduler
Package your scheduler binary into a container image. For the purposes of this example,
you can use the default scheduler (kube-scheduler) as your second scheduler.
Clone the
Kubernetes source code from GitHub
and build the source.
git clone https://github.com/kubernetes/kubernetes.git
cd
make
Create a container image containing the kube-scheduler binary. Here is the
to build the image:
FROM
busybox
ADD
./_output/local/bin/linux/amd64/kube-scheduler /usr/local/bin/kube-scheduler
Save the file as
, build the image and push it to a registry. This example
pushes the image to
Google Container Registry (GCR)
.
For more details, please read the GCR
.
docker build -t gcr.io/my-gcp-project/my-kube-scheduler:1.0 .
gcloud docker -- push gcr.io/my-gcp-project/my-kube-scheduler:1.0
Define a Kubernetes Deployment for the scheduler
Now that you have your scheduler in a container image, create a pod
configuration for it and run it in your Kubernetes cluster. But instea

d of creating a pod
directly in the cluster, you can use a
for this example. A
manages a
Replica Set
which in turn manages the pods,
thereby making the scheduler resilient to failures. Here is the deployment
config. Save it as
my-scheduler.yaml
:
admin/sched/my-scheduler.yaml
:
v1
kind
:
metadata
:
name
:
my-scheduler
namespace
:
---
:
kind
:
metadata
:
name
:
my-scheduler-as-kube-scheduler
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
---
:
kind
:
metadata
:
name
:
my-scheduler-as-volume-scheduler
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
---
:
v1
kind
:
ConfigMap
metadata
:
name
:
my-scheduler-config
namespace
:
data
:
my-scheduler-config.yaml
:
|
apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: my-scheduler
leaderElection:
leaderElect: false
---
:
apps/v1
kind
:
metadata
:
labels
:
component
:
scheduler
tier
:
name
:
namespace
:
spec
:
selector
:
:
component
:
scheduler
tier
:
replicas
:
1
template
:
metadata
:
labels
:
component
:
scheduler
tier
:
version
:
second
spec
:
:
:
-
command
:
- /usr/local/bin/kube-scheduler
- --config=/etc/kubernetes/my-scheduler/my-scheduler-config.yaml
image
:
gcr.io/my-gcp-project/my-kube-scheduler:1.0
:
httpGet
:
path
:
/healthz
port
:
10259
scheme
:
HTTPS
:
15
name
:
kube-second-scheduler
:
httpGet
:
path
:
/healthz
port
:
10259
scheme
:
HTTPS
resources
:
requests
:
cpu
:
'0.1'
:
:
false
:
-
name
:
mountPath
:
/etc/kubernetes/my-scheduler
:
false
hostPID
:
false
volumes
:
-
name
:
configMap
:
name
:
In the above manifest, you use a
to customize the behavior of your scheduler implementation. This configuration has been passed to
the
during initialization with the
--config
option. The
ConfigMap stores the configuration file. The Pod of the
Deployment mounts the
ConfigMap as a volume.
In the aforementioned Scheduler Configuration, your scheduler implementation is represented via
a
.
Note:
To determine if a scheduler is responsible for scheduling a specific Pod, the
spec.schedulerName
field in a
PodTemplate or Pod manifest must match the
.
All schedulers running in the cluster must have unique names.
Also, note that you create a dedicated service account
and bind the ClusterRole
to it so that it can acquire the same privileges as
.
Please see the
kube-scheduler documentation
for
detailed description of other command line arguments and
Scheduler Configuration reference
for
detailed description of other customizable
Run the second scheduler in the cluster
In order to run your scheduler in a Kubernetes cluster, create the deployment
specified in the config above in a Kubernetes cluster:
kubectl create -f my-scheduler.yaml
Verify that the scheduler pod is running:
=
NAME                                           READY     STATUS    RESTARTS   AGE
....
my-scheduler-lnf4s-4744f                       1/1       Running   0          2m
...
You should see a "Running" my-scheduler pod, in addition to the default kube-schedule

r
pod in this list.
Enable leader election
To run multiple-scheduler with leader election enabled, you must do the following:
Update the following fields for the KubeSchedulerConfiguration in the
ConfigMap in your YAML file:
leaderElection.leaderElect
to
true
leaderElection.resourceNamespace
to
&lt;lock-object-namespace&gt;
leaderElection.resourceName
to
&lt;lock-object-name&gt;
Note:
The control plane creates the lock objects for you, but the namespace must already exist.
If RBAC is enabled on your cluster, you must update the
Add your scheduler name to the resourceNames of the rule applied for
endpoints
and
leases
resources, as in the following example:
kubectl edit clusterrole system:kube-scheduler
admin/sched/clusterrole.yaml
:
kind
:
metadata
:
:
:
"true"
labels
:
kubernetes.io/bootstrapping
:
rbac-defaults
name
:
rules
:
-
apiGroups
:
- coordination.k8s.io
resources
:
- leases
verbs
:
- create
-
apiGroups
:
:
- kube-scheduler
- my-scheduler
resources
:
- leases
verbs
:
- get
- update
-
apiGroups
:
-
""
:
resources
:
- endpoints
verbs
:
- delete
- get
- patch
- update
Specify schedulers for pods
Now that your second scheduler is running, create some pods, and direct them
to be scheduled by either the default scheduler or the one you deployed.
In order to schedule a given pod using a specific scheduler, specify the name of the
scheduler in that pod spec. Let's look at three examples.
Pod spec without any scheduler name
admin/sched/pod1.yaml
:
v1
kind
:
Pod
metadata
:
name
:
no
-annotation
labels
:
name
:
multischeduler-example
spec
:
:
-
name
:
pod-with-no-annotation-container
image
:
registry.k8s.io/pause:2.0
When no scheduler name is supplied, the pod is automatically scheduled using the
default-scheduler.
Save this file as
pod1.yaml
and submit it to the Kubernetes cluster.
kubectl create -f pod1.yaml
Pod spec with
admin/sched/pod2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
annotation-default-scheduler
labels
:
name
:
spec
:
:
:
-
name
:
pod-with-default-annotation-container
image
:
A scheduler is specified by supplying the scheduler name as a value to
. In this case, we supply the name of the
default scheduler which is
.
pod2.yaml
kubectl create -f pod2.yaml
admin/sched/pod3.yaml
:
v1
kind
:
Pod
metadata
:
name
:
annotation-second-scheduler
labels
:
name
:
spec
:
:
:
-
name
:
pod-with-second-annotation-container
image
:
In this case, we specify that this pod should be scheduled using the scheduler that we
deployed -
. Note that the value of
should match the name supplied for the scheduler
in the
field of the mapping
.
pod3.yaml
kubectl create -f pod3.yaml
Verify that all three pods are running.
Verifying that the pods were scheduled using the desired schedulers
In order to make it easier to work through these examples, we did not verify that the
pods were actually scheduled using the desired schedulers. We can verify that by
changing the order of pod and deployment config submissions above. If we submit all the
pod configs to a Kubernetes cluster before submitting the scheduler deployment config,
we see that the pod
remains in "Pending" state forever
while the other two pods get scheduled. Once we 

submit the scheduler deployment config
and our new scheduler starts running, the
pod gets
scheduled as well.
Alternatively, you can look at the "Scheduled" entries in the event logs to
verify that the pods were scheduled by the desired schedulers.
You can also use a
custom scheduler configuration
or a custom container image for the cluster's main scheduler by modifying its static pod manifest
on the relevant control plane nodes.
5 - Use an HTTP Proxy to Access the Kubernetes API
This page shows how to use an HTTP proxy to access the Kubernetes API.
minikube
.
If you do not already have an application running in your cluster, start
a Hello world application by entering this command:
kubectl create deployment node-hello --image
=
gcr.io/google-samples/node-hello:1.0 --port
=
8080
Using kubectl to start a proxy server
This command starts a proxy to the Kubernetes API server:
kubectl proxy --port=8080
Exploring the Kubernetes API
When the proxy server is running, you can explore the API using
curl
,
wget
,
or a browser.
Get the API versions:
The output should look similar to this:
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "10.0.2.15:8443"
    }
  ]
}
Get a list of pods:
curl http://localhost:8080/api/v1/namespaces/default/pods
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "33074"
  },
  "items": [
    {
      "metadata": {
        "name": "kubernetes-bootcamp-2321272333-ix8pt",
        "generateName": "kubernetes-bootcamp-2321272333-",
        "namespace": "default",
        "uid": "ba21457c-6b1d-11e6-85f7-1ef9f1dab92b",
        "resourceVersion": "33003",
        "creationTimestamp": "2016-08-25T23:43:30Z",
        "labels": {
          "pod-template-hash": "2321272333",
          "run": "kubernetes-bootcamp"
        },
        ...
}
.
6 - Use a SOCKS5 Proxy to Access the Kubernetes API
This page shows how to use a SOCKS5 proxy to access the API of a remote Kubernetes cluster.
This is useful when the cluster you want to access does not expose its API directly on the public internet.
minikube
Your Kubernetes server must be at or later than version v1.24.
.
You need SSH client software (the
ssh
tool), and an SSH service running on the remote server.
You must be able to log in to the SSH service on the remote server.
Task context
Note:
This example tunnels traffic using SSH, with the SSH client and server acting as a SOCKS proxy.
You can instead use any other kind of
SOCKS5
proxies.
Figure 1 represents what you're going to achieve in this task.
You have a client computer, referred to as local in the steps ahead, from where you're going to create requests to talk to the Kubernetes API.
The Kubernetes server/API is hosted on a remote server.
You will use SSH client and server software to creat

e a secure SOCKS5 tunnel between the local and
the remote server. The HTTPS traffic between the client and the Kubernetes API will flow over the SOCKS5
tunnel, which is itself tunnelled over SSH.
graph LR;
subgraph local[Local client machine]
client([client])-- local
traffic .-&gt; local_ssh[Local SSH
SOCKS5 proxy];
end
local_ssh[SSH
SOCKS5
proxy]-- SSH Tunnel --&gt;sshd
subgraph remote[Remote server]
sshd[SSH
server]-- local traffic --&gt;service1;
end
client([client])-. proxied HTTPs traffic
going through the proxy .-&gt;service1[Kubernetes API];
class ingress,service1,service2,pod1,pod2,pod3,pod4 k8s;
enabled
Figure 1. SOCKS5 tutorial components
Using ssh to create a SOCKS5 proxy
This command starts a SOCKS5 proxy between your client machine and the remote server.
The SOCKS5 proxy lets you connect to your cluster's API server.
# The SSH tunnel continues running in the foreground after you run this
ssh -D
1080
-q -N username@kubernetes-remote-server.example
-D 1080
: opens a SOCKS proxy on local port :1080.
-q
: quiet mode. Causes most warning and diagnostic messages to be suppressed.
-N
: Do not execute a remote command. Useful for just forwarding ports.
username@kubernetes-remote-server.example
: the remote SSH server where the Kubernetes cluster is running.
Client configuration
To explore the Kubernetes API you'll first need to instruct your clients to send their queries through
the SOCKS5 proxy we created earlier.
For command-line tools, set the
https_proxy
environment variable and pass it to commands that you run.
export
=
socks5h://localhost:1080
When you set the
variable, tools such as
curl
route HTTPS traffic through the proxy
you configured. For this to work, the tool must support SOCKS5 proxying.
Note:
In the URL https://localhost:6443/api,
localhost
does not refer to your local client computer.
Instead, it refers to the endpoint on the remote server known as
localhost
.
The
curl
tool sends the hostname from the HTTPS URL over SOCKS, and the remote server
resolves that locally (to an address that belongs to its loopback interface).
curl -k -v https://localhost:6443/api
To use the official Kubernetes client
kubectl
with a proxy, set the
proxy-url
element
for the relevant
cluster
entry within your
:
v1
clusters
:
-
cluster
:
:
LRMEMMW2
# shortened for readability
server
:
https://&lt;API_SERVER_IP_ADRESS&gt;:6443
# the "Kubernetes API" server, in other words the IP address of kubernetes-remote-server.example
proxy-url
:
socks5://localhost:1080
# the "SSH SOCKS5 proxy" in the diagram above (DNS resolution over socks is built-in)
name
:
default
contexts
:
-
context
:
cluster
:
default
user
:
default
name
:
default
:
default
kind
:
Config
:
{}
users
:
-
name
:
default
user
:
:
LS0tLS1CR==
:
LS0tLS1CRUdJT=
If the tunnel is operating and you use
kubectl
with a context that uses this cluster, you can interact with your cluster through that proxy. For example:
NAMESPACE     NAME                                     READY   STATUS      RESTARTS   AGE
kube-system   coredns-85cb69466-klwq8                  1/1     Running     0          5m46s
Clean up
Stop the ssh port-forwarding process by pressing
CTRL+C
on the terminal where it is running.
Type
unset https_proxy
in a terminal to stop forwarding http traffic through the proxy.
Further reading
OpenSSH remote login client
7 - Set up Konnectivity service
The Konnectivity service provides a TCP level proxy for the control plane to cluster
communication.
Before you b

egin
be configured to communicate with your cluster. It is recommended to run this
tutorial on a cluster with at least two nodes that are not acting as control
plane hosts. If you do not already have a cluster, you can create one by using
minikube
.
Configure the Konnectivity service
The following steps require an egress configuration, for example:
admin/konnectivity/egress-selector-configuration.yaml
:
apiserver.k8s.io/v1beta1
kind
:
:
# Since we want to control the egress traffic to the cluster, we use the
# "cluster" as the name. Other supported values are "etcd", and "controlplane".
-
name
:
cluster
:
# This controls the protocol between the API Server and the Konnectivity
# server. Supported values are "GRPC" and "HTTPConnect". There is no
# end user visible difference between the two modes. You need to set the
# Konnectivity server to work in the same mode.
:
GRPC
transport
:
# This controls what transport the API Server uses to communicate with the
# Konnectivity server. UDS is recommended if the Konnectivity server
# locates on the same machine as the API Server. You need to configure the
# Konnectivity server to listen on the same UDS socket.
# The other supported transport is "tcp". You will need to set up TLS
# config to secure the TCP transport.
uds
:
udsName
:
/etc/kubernetes/konnectivity-server/konnectivity-server.socket
You need to configure the API Server to use the Konnectivity service
and direct the network traffic to the cluster nodes:
Make sure that
Service Account Token Volume Projection
feature enabled in your cluster. It is enabled by default since Kubernetes v1.20.
Create an egress configuration file such as
.
Set the
--egress-selector-config-file
flag of the API Server to the path of
your API Server egress configuration file.
If you use UDS connection, add volumes config to the kube-apiserver:
spec
:
:
:
-
name
:
konnectivity-uds
mountPath
:
/etc/kubernetes/konnectivity-server
readOnly
:
false
volumes
:
-
name
:
hostPath
:
path
:
type
:
Generate or obtain a certificate and kubeconfig for konnectivity-server.
For example, you can use the OpenSSL command line tool to issue a X.509 certificate,
using the cluster CA certificate
from a control-plane host.
openssl req -subj
"/CN=system:konnectivity-server"
-new -newkey rsa:2048 -nodes -out konnectivity.csr -keyout konnectivity.key
openssl x509 -req -in konnectivity.csr -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key -CAcreateserial -out konnectivity.crt -days
375
-sha256
SERVER
=
$(
jsonpath
=
'{.clusters..server}'
)
kubectl --kubeconfig /etc/kubernetes/konnectivity-server.conf config set-credentials system:konnectivity-server --client-certificate konnectivity.crt --client-key konnectivity.key --embed-certs
=
true
kubectl --kubeconfig /etc/kubernetes/konnectivity-server.conf config set-cluster kubernetes --server
"
$SERVER
"
--certificate-authority /etc/kubernetes/pki/ca.crt --embed-certs
=
true
kubectl --kubeconfig /etc/kubernetes/konnectivity-server.conf config set-context system:konnectivity-server@kubernetes --cluster kubernetes --user system:konnectivity-server
kubectl --kubeconfig /etc/kubernetes/konnectivity-server.conf config use-context system:konnectivity-server@kubernetes
rm -f konnectivity.crt konnectivity.key konnectivity.csr
Next, you need to deploy the Konnectivity server and agents.
kubernetes-sigs/apiserver-network-proxy
is a reference implementation.
Deploy the Konnectivity server on your control plane node. The provided
konnectivity-server.yaml
manifest assumes
that the Kubernetes components are deployed as a
static Pod
in your cluster. If not, you can deploy the Konnecti

vity
server as a DaemonSet.
admin/konnectivity/konnectivity-server.yaml
:
v1
kind
:
Pod
metadata
:
name
:
konnectivity-server
namespace
:
spec
:
:
:
true
:
-
name
:
konnectivity-server-container
image
:
registry.k8s.io/kas-network-proxy/proxy-server:v0.0.32
command
:
[
"/proxy-server"
]
args
:
[
"--logtostderr=true"
,
# This needs to be consistent with the value set in egressSelectorConfiguration.
"--uds-name=/etc/kubernetes/konnectivity-server/konnectivity-server.socket"
,
# The following two lines assume the Konnectivity server is
# deployed on the same machine as the apiserver, and the certs and
# key of the API Server are at the specified location.
"--cluster-cert=/etc/kubernetes/pki/apiserver.crt"
,
"--cluster-key=/etc/kubernetes/pki/apiserver.key"
,
"--mode=grpc"
,
"--server-port=0"
,
"--agent-port=8132"
,
"--admin-port=8133"
,
"--health-port=8134"
,
"--agent-namespace=kube-system"
,
"--agent-service-account=konnectivity-agent"
,
"--kubeconfig=/etc/kubernetes/konnectivity-server.conf"
,
"--authentication-audience=system:konnectivity-server"
]
:
httpGet
:
scheme
:
HTTP
host
:
127.0.0.1
port
:
8134
path
:
/healthz
:
30
:
60
ports
:
-
name
:
agentport
:
8132
hostPort
:
8132
-
name
:
adminport
:
8133
hostPort
:
8133
-
name
:
healthport
:
8134
hostPort
:
8134
:
-
name
:
k8s-certs
mountPath
:
readOnly
:
true
-
name
:
mountPath
:
/etc/kubernetes/konnectivity-server.conf
readOnly
:
true
-
name
:
mountPath
:
readOnly
:
false
volumes
:
-
name
:
k8s-certs
hostPath
:
path
:
-
name
:
hostPath
:
path
:
type
:
FileOrCreate
-
name
:
hostPath
:
path
:
type
:
Then deploy the Konnectivity agents in your cluster:
admin/konnectivity/konnectivity-agent.yaml
:
apps/v1
# Alternatively, you can deploy the agents as Deployments. It is not necessary
# to have an agent on each node.
kind
:
DaemonSet
metadata
:
labels
:
addonmanager.kubernetes.io/mode
:
Reconcile
k8s-app
:
konnectivity-agent
namespace
:
name
:
spec
:
selector
:
:
k8s-app
:
template
:
metadata
:
labels
:
k8s-app
:
spec
:
:
:
-
key
:
operator
:
"Exists"
:
-
image
:
us.gcr.io/k8s-artifacts-prod/kas-network-proxy/proxy-agent:v0.0.16
name
:
command
:
[
"/proxy-agent"
]
args
:
[
,
"--ca-cert=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
,
# Since the konnectivity server runs with hostNetwork=true,
# this is the IP address of the master machine.
"--proxy-server-host=35.225.206.7"
,
"--proxy-server-port=8132"
,
"--admin-server-port=8133"
,
"--health-server-port=8134"
,
"--service-account-token-path=/var/run/secrets/tokens/konnectivity-agent-token"
]
:
-
mountPath
:
name
:
konnectivity-agent-token
:
httpGet
:
port
:
8134
path
:
/healthz
:
15
:
15
:
volumes
:
-
name
:
projected
:
sources
:
-
:
path
:
audience
:
system:konnectivity-server
Last, if RBAC is enabled in your cluster, create the relevant RBAC rules:
admin/konnectivity/konnectivity-rbac.yaml
:
kind
:
metadata
:
name
:
labels
:
:
"true"
:
Reconcile
roleRef
:
apiGroup
:
rbac.authorizatio

n.k8s.io
kind
:
name
:
subjects
:
-
apiGroup
:
kind
:
User
name
:
---
:
v1
kind
:
metadata
:
name
:
namespace
:
labels
:
:
"true"
:
Reconcile
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
Kubeadm
1.1:
1.1.1:
1.1.2:
1.1.3:
1.1.4:
1.1.5:
1.1.6:
1.1.7:
1.1.8:
1.1.9:
1.1.10:
1.1.11:
1.1.12:
1.1.13:
1.1.14:
1.1.15:
1.1.16:
1.1.17:
1.1.18:
1.1.19:
1.1.20:
1.1.21:
1.1.22:
1.1.23:
1.1.24:
1.1.25:
1.1.26:
1.1.27:
1.1.28:
1.1.29:
1.1.30:
1.1.31:
1.1.32:
1.1.33:
1.1.34:
1.1.35:
1.1.36:
1.1.37:
1.1.38:
1.1.39:
1.1.40:
1.1.41:
1.1.42:
1.1.43:
1.1.44:
1.1.45:
1.1.46:
1.1.47:
1.1.48:
1.1.49:
1.1.50:
1.1.51:
1.1.52:
1.1.53:
1.1.54:
1.1.55:
1.1.56:
1.1.57:
1.1.58:
1.1.59:
1.1.60:
1.1.61:
1.1.62:
1.1.63:
1.1.64:
1.1.65:
1.1.66:
1.1.67:
1.1.68:
1.1.69:
1.1.70:
1.1.71:
1.1.72:
1.1.73:
1.1.74:
1.1.75:
1.1.76:
1.1.77:
1.1.78:
1.1.79:
1.1.80:
1.1.81:
1.1.82:
1.1.83:
1.1.84:
1.1.85:
1.1.86:
1.1.87:
1.1.88:
1.1.89:
1.1.90:
1.1.91:
1.1.92:
1.1.93:
1.1.94:
1.1.95:
1.1.96:
1.1.97:
1.1.98:
1.1.99:
1.1.100:
1.1.101:
1.1.102:
1.1.103:
1.1.104:
1.1.105:
1.1.106:
1.1.107:
1.1.108:
1.1.109:
1.2:
1.3:
1.4:
1.5:
1.6:
1.7:
1.8:
1.9:
1.10:
1.11:
1.12:
1.13:
1.14:
1.15:
1.16:
1 - Kubeadm
and
.
or
1.1 - Kubeadm Generated
1.1.1 -
Synopsis
│ KUBEADM                                                  │
│                                                          │

│ control-plane# kubeadm init               

               │

Options
-h, --help
1.1.2 -
Synopsis
Options
-h, --help
1.1.3 -
Synopsis
Options
-h, --help
1.1.4 -
Synopsis
Options
-h, --help
1.1.5 -
Synopsis
Examples
Options
-h, --help
1.1.6 -
Synopsis
Options
-h, --help
1.1.7 -
Synopsis
Renew the certificate embedded in the k

ubeconfig file for the admin to use and for kubeadm itself.
Options
-h, --help
1.1.8 -
Synopsis
Options
-h, --help
1.1.9 -
Synopsis
Options
-h, --help
1.1.10 -
Synopsis
After renewal, in order to make changes effective, is required to restart control-plane components and eventually re-distribute the 

renewed certificate in case the file is used elsewhere.
Options
-h, --help
1.1.11 -
Synopsis
Options
-h, --help
1.1.12 -
Synopsis
Options
-h, --help
1.1.13 -
Synopsis
After renewal, in order to make changes effective, is required to restart control-plane components and eventually re-distribute the renewed 

certificate in case the file is used elsewhere.
Options
-h, --help
1.1.14 -
Synopsis
Options
-h, --help
1.1.15 -
Synopsis
Options
-h, --help
1.1.16 -
Synopsis
Options
--cert-dir string     Def

ault: "/etc/kubernetes/pki"
-h, --help
1.1.17 -
Synopsis
Options
-h, --help
1.1.18 -
Synopsis
Examples

Options
-h, --help
1.1.19 -
Synopsis
initialized your cluster using kubeadm v1.7.x or lower, you must use the 'config up

load' command to create this
Options
-h, --help
1.1.20 -
Synopsis
Options
-h, --help
1.1.21 -
Synopsis
Options
-h, --help
1.1.22 -
Synopsis
Options
-h, --help
1.1.23 -
Read an older version of the kubeadm configuration API types from a file, and output the similar config object for t

he newer version
Synopsis
Options
-h, --help
1.1.24 -
Synopsis
Options
-h, --help
1.1.25 -
Synopsis
Options
-h, --help
1.1.26 -
Synopsis
Options
-h, --help
The kubeconfig file to use when talking to the cluster. I

f the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.
1.1.27 -
Synopsis
Options
Optional extra Subject Alternativ

e Names (SANs) to use for the API Server serving certificate. Can be both IP addresses and DNS names.
--dry-run
-h, --help
1.1.28 -
Synopsis
Options
-h, --help
1.1.29 -
Synopsis
Options
-h, --help
1.1.30 -
Synopsis
Options
The IP address the API Server will advertise it's listening on. If no

t set the default network interface will be used.
--dry-run
-h, --help
1.1.31 -
Synopsis
Options
--dry-run
-h, --help
1.1.32 -
Synopsis
Options
--dry-run
-h, --help
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can 

be searched for an existing kubeconfig file.
1.1.33 -
Synopsis
Examples
Options
--dry-run
-h, --help
1.1.34 -
Synopsis
Options
-h, --help
1.1.35 -
Synopsis
Options
--dry-run
-h, --help
1.1.36 -
Synopsis
Options
--dry-run
-h, --help
Options i

1.1.37 -
Synopsis
Options
--dry-run
-h, --help
1.1.38 -
Synopsis
Options
--dry-run
-h, --help
1.1.39 -
Synopsis
Options
--dry-run
-h, --help
help for ca
1.1.40 -
Synopsis
Options
Pat

h to a kubeadm configuration file.
--dry-run
-h, --help
1.1.41 -
Synopsis
Options
--dry-run
-h, --help
1.1.42 -
Synopsis
Options
--dry-run
-h, --help
1.1.43 -
Synopsis
Options
--dry-run
-h, --help
1.1.44 -
Synopsis
Options
--dry-run
-h, --help
--rootf

s string
1.1.45 -
Synopsis
Options
--dry-run
-h, --help
1.1.46 -
Synopsis
Options
-h, --help
help for sa
1.1.47 -
Synopsis
Options
-h, --help
1.1.48 -
Synopsis
Examples
  
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].exte

nsion". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.1.49 -
Synopsis
Options
--dry-run
-h, --help
1.1.50 -
Synopsis
Options
--dry-run
-h, --help
Choose a container registry to

 pull control plane images from
1.1.51 -
Synopsis
Options
--dry-run
-h, --help
1.1.52 -
Synopsis
Options
-h, --help
1.1.53 -
Synopsis
Examples
  
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserv

er", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.1.54 -
Synopsis
Options
-h, --help
1.1.55 -
Synopsis
Options
--dry-run
-h, --help
1.1.56 -
Synopsis
Options
--dry-run
-h, --help
1.1.57 -
Synopsis
Options
--dry-run
-h, --help
--kubeconfig-dir string 

    Default: "/etc/kubernetes"
1.1.58 -
only
Synopsis
only
Options
--dry-run
-h, --help
1.1.59 -
Synopsis
Options
--dry-run
-h, --help
1.1.60 -
Synopsis
Examples
Options
-h, --help
1.1.61 -
Synopsis
Examples
Options
--dry-run
-h, --help
1.1.62 -
Synopsis
Enable kubelet client certificate rotati

on
Options
--dry-run
-h, --help
1.1.63 -
Synopsis
Examples
Options
--dry-run
-h, --help
1.1.64 -
Synopsis
Examples
  
Options
--dry-run
-h, --help
1.1.65 -
Synopsis
Examples
Options
--dry-run
-h, --help
1.1.66 -
Synopsis
Options
-h, --help
1.1.67 -
Synopsis
Options
Key used to encrypt the control-plane cert

ificates in the kubeadm-certs Secret.
--dry-run
-h, --help
1.1.68 -
Synopsis
Options
-h, --help
1.1.69 -
Synopsis
Options
--dry-run
-h, --help
1.1.70 -
Synopsis
Examples
Options
--dry-run
-h, --help
1.1.71 -
Synopsis
Examples
Options
--dry-run
-h, --help
1.1.72 -
Synopsis
token along with the IP address of t

he API server. The second is to
Options
For file-based discovery, a file or URL from which to load cluster 

--dry-run
-h, --help
1.1.73 -
Synopsis
Options
-h, --help
1.1.74 -
Synopsis
Examples
Options
-h, --help
1.1.75 -
Synopsis
Options
--dry-run
-h, --help
1.1.76 -
Synopsis
Options
If the node should host a new control plane in

stance, the IP address the API Server will advertise it's listening on. If not set the default network interface will be used.
--dry-run
-h, --help
1.1.77 -
Synopsis
Options
--dry-run
-h, --help
1.1.78 -
Synopsis
Options
-h, --help
1.1.79 -
Synopsis
Examples
Options
-h, --help
1.1.80 -
Synopsis
Options
For token-based discovery, validate that the root CA public key matches this h

ash (format: "&lt;type&gt;:&lt;value&gt;").
--dry-run
-h, --help
1.1.81 -
Synopsis
Options
--dry-run
-h, --help
1.1.82 -
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the

 patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.1.83 -
Synopsis
Options
--dry-run
-h, --help
1.1.84 -
Synopsis
Options
--dry-run
-h, --help
1.1.85 -
Synopsis
Options
For token-based discovery

, the token used to validate cluster information fetched from the API server.
--dry-run
-h, --help
1.1.86 -
Synopsis
Examples
Options
--dry-run
-h, --help
1.1.87 -
Synopsis
Options
-h, --help
1.1.88 -
Output a kubeconfig file for an additional u

ser
Synopsis
Examples
Options
-h, --help
1.1.89 -
Synopsis
Options
--dry-run
-h, --help
1.1.90 -
Synopsis
Options
-h, --help
1.1.91 -
Synopsis
Options
--dry-run
-h, --help
1.1.92 -
Synopsis
Options
--dry-run
-h, --help
Options inherite

d from parent commands
1.1.93 -
Synopsis
Options
--dry-run
-h, --help
1.1.94 -
Synopsis
Options
--dry-run
-h, --help
1.1.95 -
Synopsis
Options
-h, --help
--dry-run
--kubeconfig string     Default: "/etc/kubern

etes/admin.conf"
1.1.96 -
Synopsis
Options
-h, --help
--dry-run
1.1.97 -
Synopsis
Options
-h, --help
--dry-run
1.1.98 -
Synopsis
Options
-h, --help
--dry-run
1.1.99 -
Synopsis
Options
-h, --help
1.1.100 -
Synopsis
Options
--dry-run
Do no

t change any state, just output what actions would be performed.
-h, --help
-y, --yes
1.1.101 -
Synopsis
Options
-h, --help
1.1.102 -
Synopsis
Options
--dry-run
-h, --help
The kubeconfig file to use when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig f

ile.
1.1.103 -
Synopsis
Options
-h, --help
1.1.104 -
Synopsis
Options
--dry-run
-h, --help
1.1.105 -
Synopsis
Options
--dry-run
-h, --help
1.1.106 -
Synopsis
Options
-h, --help
A list of checks whose errors wil

l be shown as warnings. Example: 'IsPrivilegedUser,Swap'. Value 'all' ignores errors from all checks.
1.1.107 -
Synopsis
Options
-h, --help
1.1.108 -
Synopsis
Options
-h, --help
1.1.109 -
1.2 - kubeadm init
Synopsis
k

ubeconfig                   Generate all kubeconfig files necessary to establish the control plane and the admin kubeconfig file
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "

kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
.
(
.
run there.
token via
--token
docs.
and
See
Warning:
kube-dns usage with kubeadm is deprecated as of v1.1

8 and is removed in v1.21.
command.
. The list
--help
--help
=
local
--config
=
=
=
.
Caution:
--config
contain a
---\n
Mixing
--config
command.
the
command.
.
flag for
a
using
--config
.
Passing
.
Feature
Default
Alpha
Beta
GA
false
1.19
-
-
false
1.22
-
-
true
1.22
1.23
1.25
Note:
true
for
,
,
and
etcd
to run as non-root user

s.
ConfigMap
true
.
false
or
(during
,
,
IPVS
--config
and
fields.
. If the
ci/latest
)
is used.
.
To provide
.
and
imageTag
to
using
.
,
where
, and/or
imageTag
to
.
topic.
to
and
to
.
The following phase command can be used to re-upload the

 certificates after expiration:
=
Note:
with
--config
.
and
.
The
kubeadm
kubelet
by
systemd
.
.
see the
.
kubeadm
flag.
.
in the
,
.
--token
and
.
at
.
phases
or
1.3 - kubeadm join
Synopsis
local file or downloaded via an HTTPS U

RL. The forms are
Options
For token-based discovery, validate that the root CA public key matches this hash (format: 

"&lt;type&gt;:&lt;value&gt;").
--dry-run
-h, --help
process.
.
--help
command,
flag.
=
=
field in
.
as well as validating that the root CA public key matches the provi

ded hash and
RFC7469
's/^.* //'
Example
commands:
join
if the
.
command.
Example
command:
file.
Example
commands:
By default

, there is a CSR auto-approver enabled that basically approves any client certificate request
Using
ConfigMap
flag of the
Fetch the
jsonpath
=
:
v1
kind
:
Config
clusters
:
-
cluster
:
:
server
:
name
:
""
contexts
:
[]
:
""
:
{}
users
:
[]
Use the
.
but before
.
Caution:
--config
contain a
--config
command.
the
command.
.
.
or
.
1.4 - kubeadm upgrade
The steps for perf

orming an upgrade using kubeadm are outlined in
.
You can use
and
will also
.
Note:
and
--config
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
Force upgrading although some requirements might not be met. This also

 implies non-interactive mode.
-h, --help
-y, --yes
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The 

default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
1.5 - kubeadm config
During
in the
,
and
.
You can use
uses for
and
.
Note:
init
and
join
or
.
.
You can use
and
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
A comma-separated list for component config API objects to print the default values for. Available values: [Ku

beProxyConfiguration KubeletConfiguration]. If this flag is not set, no component configs will be printed.
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
Path to the CRI socket to connect. If empty kubeadm will try 

to auto-detect this value; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.
-h, --help
1.6 - kubeadm reset
or
.
Synopsis
Options
--dry-run
-h, --help
the
or
reset
the
and
""
--prefix
See the
1.7 - kubeadm token
.
such a token and also to creat

e and manage new ones.
Synopsis
Options
-h, --help
--dry-run
Synopsis
Options
-h, --help
--dry-run
Synopsis
Options
-h, --help
--dry-run
Synopsis
Options
If true, ignore any errors in templates when a field or map ke

y is missing in the template. Only applies to golang and jsonpath output formats.
-h, --help
--dry-run
1.8 - kubeadm version
Synopsis
Options
-h, --help
1.9 - kubeadm alpha
Caution:
.
or
1.10 - kubeadm certs
.
overview
Synopsis
Options
-h, --help
all
.
renew
all
apiserver
etcd-peer
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewals run unconditionally, regardless of certificate expiration date; extra attributes such as SANs will 

be based on the existing file/certificates, there is no need to resupply them.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewal by default t

ries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewal by default tries to

 use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, to generate a CSR request.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API fo

r certificate renewal, or as a last option, to generate a CSR request.
Options
-h, --help
to
and
Synopsis
Options
-h, --help
.
Synopsis
Options
-h, --help
Synopsis
Examples
Options
-h, --help
The path where to save the kub

econfig file.
or
1.11 - kubeadm init phase
,
preflight
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
certs
all
ca
apiserver
etcd-ca
etcd-peer
sa
Synopsis
Options
-h, --help
Synopsis
Options
Optional extra Subject Alternative Names (SANs) to use fo

r the API Server serving certificate. Can be both IP addresses and DNS names.
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
help for ca
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
--ku

bernetes-version string     Default: "stable-1"
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Default SANs are lo

calhost, 127.0.0.1, 127.0.0.1, ::1
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
-h, --help
help for sa
all
all
admin
kubelet
scheduler
Synopsis
Options
-h, --help
Synopsis
Options
The IP address the API Server will advertise it's

 listening on. If not set the default network interface will be used.
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
only
Synopsis
only
Options
--dry-run
-h, --help
Synopsis
Options
Path to a kubeadm configuration

 file.
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
all
apiserver
scheduler
Synopsis
Options
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
--kubernetes-ver

sion string     Default: "stable-1"
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
Don't appl

y any changes; just output what would be done.
-h, --help
Synopsis
Options
--dry-run
-h, --help
etcd
local
Synopsis
Options
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help

.
all
kubeadm
kubelet
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Upload certific

ates to kubeadm-certs
Synopsis
Options
--dry-run
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
all
phases.
Synopsis
Examples
Options
-h, --help
Synopsis
Examples
Opti

ons
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
all
addon
all
coredns
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
The kubeconfig file to use when talking to 

the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.
Synopsis
Options
--dry-run
-h, --help
v1beta3
or
1.12 - kubeadm join phase
,
phase
Synopsis
Options
-h, --help
preflight
Synopsis
Examples
Options
Create a new control plane instance 

on this node
--dry-run
-h, --help
all
certs
Synopsis
Examples
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches a

re applied first alpha-numerically.
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
For file-based discovery, a file or URL from which

 to load cluster information.
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd

", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
all
etcd
Synopsis
Examples
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Register the new control-plane node into the ClusterStatus maintained in the kubeadm-config ConfigMap (DEPRECAT

ED)
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
or
1.13 - kubeadm kubeconfig
see
.
overview
Synopsis
Options
-h, --help
user
Synopsis
Examples
Options
-h, --help
1.14 - kubeadm reset phase
,
phase
Synopsis
Options
-h, --help
preflight
Synopsis
Options
--dry-run
Reset the

 node without prompting for confirmation.
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
or
1.15 - kubeadm upgrade phase
phases.
apply
phase
preflight
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
The kubeconfig file to use

 when talking to the cluster. If the flag is not set, a set of standard locations can be searched for an existing kubeconfig file.
Synopsis
Options
--dry-run
-h, --help
or
1.16 - Implementation details
and
how
and
Secure
:
It should
not

es
etcd.yaml
(
ca.crt
,
ca.key
,
,
sa.pub
,
sa.key
,
,
The
.
The
option.
flag) is
[Error] if
[Error] if
[Error] if
conntrack
,
ip
,
iptables
,
mount
,
nsenter
ebtables
,
ethtool
,
socat
,
tc
,
touch
,
crictl
[Error] 

if Etcd.DataDir folder already exists and it is not empty
command
ca.crt
file and
ca.key
ca.crt
10.96.0.1
)
if
,
,
The
ca.crt
.
sa.key
sa.pub
saved into
using the
flag.
See also
ca.crt
ca.key
csrsigner
;
--dry-run
command
Be in the
module
have the CN
a client certificate with scheduler id

entity.
root
by default
.
CN.
ca.crt
,
--dry-run
command
namespace
and
labels
The
address
127.0.0.1
See
--dry-run
the
command
API server
The
and
The
(
,
,
);
with the
See
to
true
to
to:

and
using
gce
or
aws
objects
to
this makes
to
ca.crt
to
to
to
to
to
sa.pub
to
to
to
(
)
the users:
and
with the
and
to
true
to
ca.crt
to
ca.crt
""
to
ca.key
""
to
sa.key
Scheduler
listen on
and use
make a
hostPath
dataDir
--dry-run
command.
ok
.
ok
kub

eadm relies on the kubelet to pull the control plane images and run them properly as static Pods.
under
.
command.
and
The
.
,
--token
under
group
flag)
.
.
The role
.
Kubeadm ensures that certificate rotation is enabled

 for nodes, and that new certificate request
role
.
).
ConfigMap
is not
.
proxy
ca.crt
and
token
The
DNS
kube-dns
the
--config
here
.
The
coredns
kube-dns
kube-dns
.
, also
see
.
kubeadm
with the
option.
If
users for
)
in the output 

of
flag;
If
server
and
ca.crt
file and
)
during the
process
process
Home
Blog
Training
Partners
Community
CC BY 4.0

NodeSelectorRequirement | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
key
operator
values
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Documenting a feature for a release | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Each major Kubernetes release introduces new features that require documentation. New releases also bring updates to existing features and documentation (such as upgrading a feature from alpha to beta).
Generally, the SIG responsible for a feature submits draft documentation of the
feature as a pull request to the appropriate development branch of the
repository, and someone on the SIG Docs team provides
editorial feedback or edits the draft directly. This section covers the branching
conventions and process used during a release by both groups.
For documentation contributors
In general, documentation contributors don't write content from scratch for a release.
Instead, they work with the SIG creating a new feature to refine the draft documentation and make it release ready.
After you've chosen a feature to document or assist, ask about it in the
#sig-docs
Slack channel, in a weekly SIG Docs meeting, or directly on the PR filed by the
feature SIG. If you're given the go-ahead, you can edit into the PR using one of
the techniques described in
Commit into another person's PR
.
Find out about upcoming features
To find out about upcoming features, attend the weekly SIG Release meeting (see
the
community
page for upcoming meetings)
and monitor the release-specific documentation
in the
kubernetes/sig-release
repository. Each release has a sub-directory in the
/sig-release/tree/master/releases/
directory. The sub-directory contains a release schedule, a draft of the release
notes, and a document listing each person on the release team.
The release schedule contains links to all other documents, meetings,
meeting minutes, and milestones relating to the release. It also contains
information about the goals and timeline of the release, and any special
processes in place for this release. Near the bottom of the document, several
release-related terms are defined.
This document also contains a link to the
Feature tracking sheet
, which is
the official way to find out about all new features scheduled to go into the
release.
The release team document lists who is responsible for each release role. If
it's not clear who to talk to about a specific feature or question you have,
either attend the release meeting to ask your question, or contact the release
lead so that they can redirect you.
The release notes draft is a good place to find out about
specific features, changes, deprecations, and more about the release. The
content is not finalized

 until late in the release cycle, so use caution.
The feature tracking sheet
for a given Kubernetes release
lists each feature that is planned for a release.
Each line item includes the name of the feature, a link to the feature's main
GitHub issue, its stability level (Alpha, Beta, or Stable), the SIG and
individual responsible for implementing it, whether it
needs docs, a draft release note for the feature, and whether it has been
merged. Keep the following in mind:
Beta and Stable features are generally a higher documentation priority than
Alpha features.
It's hard to test (and therefore to document) a feature that hasn't been merged,
or is at least considered feature-complete in its PR.
Determining whether a feature needs documentation is a manual process. Even if
a feature is not marked as needing docs, you may need to document the feature.
For developers or other SIG members
This section is information for members of other Kubernetes SIGs documenting new features
for a release.
If you are a member of a SIG developing a new feature for Kubernetes, you need
to work with SIG Docs to be sure your feature is documented in time for the
release. Check the
feature tracking spreadsheet
or check in the
#sig-release
Kubernetes Slack channel to verify scheduling details and
deadlines.
Open a placeholder PR
Open a
draft
pull request against the
dev-1.27
branch in the
repository, with a small
commit that you will amend later. To create a draft pull request, use the
Create Pull Request drop-down and select
Create Draft Pull Request
,
then click
Draft Pull Request
.
Edit the pull request description to include links to
PR(s) and
kubernetes/enhancements
issue(s).
Leave a comment on the related
issue with a link to the PR to notify the docs person managing this release that
the feature docs are coming and should be tracked for the release.
If your feature does not need
any documentation changes, make sure the sig-release team knows this, by
mentioning it in the
Slack channel. If the feature does need
documentation but the PR is not created, the feature may be removed from the
milestone.
PR ready for review
When ready, populate your placeholder PR with feature documentation and change
the state of the PR from draft to
ready for review
. To mark a pull request
as ready for review, navigate to the merge box and click
Ready for review
.
Do your best to describe your feature and how to use it. If you need help structuring your documentation, ask in the
#sig-docs
When you complete your content, the documentation person assigned to your feature reviews it.
To ensure technical accuracy, the content may also require a technical review from corresponding SIG(s).
Use their suggestions to get the content to a release ready state.
If your feature is an Alpha or Beta feature and is behind a feature gate,
make sure you add it to
Alpha/Beta Feature gates
table as part of your pull request. With new feature gates, a description of
the feature gate is also required. If your feature is GA'ed or deprecated,
make sure to move it from the
Feature gates for Alpha/Feature
table
to
table with Alpha and Beta history intact.
If your feature needs documentation and the first draft
content is not received, the feature may be removed from the milestone.
All PRs reviewed and ready to merge
If your PR has not yet been merged into the
dev-1.27
branch by the release deadline, work with the
docs person managing the release to get it in by the deadline. If your feature needs
documentation and the docs are not ready, the feature may be removed from the
Feedback
Yes
No
.
or
suggest an improveme

nt
.
Last modified November 10, 2022 at 9:15 AM PST:
Fix a link in new-features.md (d7ceef3bae)
Home
Blog
Training
Partners
Community
CC BY 4.0

RuntimeClass | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Reference
kind
metadata
(
)
More info:
handler
overhead
Quantity
)
tolerations are 

appended (excluding duplicates) to pods running with this RuntimeClass during admission, effectively unioning the set of nodes tolerated by the pod and the RuntimeClass.
.
(string)
(string)
(string)
(string)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in que

ry
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Implementation details | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Kubeadm
and
how
and
Secure
:
It should
not
etcd.yaml
(
ca.crt
,
ca.key
,
,
by the API server to connect to the ku

belets securely
sa.pub
,
sa.key
,
,
The
.
The
option.
flag) is
[Error] if
[Error] if
[Error] if
conntrack
,
ip
,
iptables
,
mount
,
nsenter
ebtables
,
ethtool
,
socat
,
tc
,
touch
,
crictl
command
ca.crt
file and
ca.key
ca.crt
10.96.0.1
)

if
,
,
The
ca.crt
.
sa.key
sa.pub
saved into
using the
flag.
See also
ca.crt
ca.key
csrsigner
;
--dry-run
command
Be in the
module
have the CN
root
by default
.
CN.
ca.crt
,
all the required kubeconfig must be provided 

by the user as well, because kubeadm cannot
--dry-run
command
namespace
and
labels
The
address
127.0.0.1
See
--dry-run
the
command
API server
The
and
The
(
,
,
);
with the
See
to
true
to
to:
and
using
gce
or
aws
objects
to
this makes
--client-

ca-file
to
ca.crt
to
to
to
to
to
sa.pub
to
to
to
(
)
the users:
and
with the
and
to
true
to
ca.crt
to
ca.crt
""
to
ca.key
""
to
sa.key
Scheduler
listen on
and use
make a
hostPath
dataDir
--dry-run
command.
ok
.
ok
under
.
Labels the node as con

trol-plane with
command.
and
The
.
,
--token
under
group
flag)
.
.
The role
.
role
.
).
ConfigMap
is not
cluster-i

nfo
.
proxy
ca.crt
and
token
The
DNS
kube-dns
the
--config
here
.
The
coredns
kube-dns
kube-dns
.
, also
see
.
kubeadm
with the
option.
If
users for
)
flag;
If
disco

very file is a kubeconfig file with only
server
and
ca.crt
file and
)
during the
process
process
Feedback
Yes
No
.
or
.
Last modified November 08, 2022 at 11:43 AM PST:
Use the `registry.k8s.io` instead of `k8s.gcr.io` image registry (759e2c9cf5)
Home
Blog
Training
Partners
Community
CC BY 4.0

Assign CPU Resources to Containers and Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Debug Pods
Determine the 

Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

e Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

ion (v1)
Tasks
request
and a CPU
limit
to
minikube
.
Minikube
, run the
enable
API,
.
NAME
Create a
Namespace
field
.
:
v1
kind
:
Pod
metadata
:
name
:
cpu-demo
namespace
:
spec
:
:
-
name
:
image
:
resources
:
limits
:
cpu
:
"1"
requests
:
cpu
:
"0.5"
args
:
- -cpus
-
"2"
The
args
The
-cpus "2"
kubectl apply -f https://k8s.io/examples/pods/resource/cpu-request-

limit.yaml --namespace
=
=
=
=
resources
:
limits
:
cpu
:
"1"
requests
:
cpu
:
500m
Use
=
This example output shows that the Pod is using 974 milliCPU, which is
-cpu "2"
Note:
CPU units
CPU
1 AWS vCPU
1 GCP Core
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
limits
:
cpu
:
"100"
requests
:
cpu
:
"100"
args
:
- -cpus
-
"2"
=
=
=
Events:
  FailedScheduling      No nodes are a

vailable that match all of the following predicates:: Insufficient cpu (3).
=
the limit.
Clean up
Feedback
Yes
No
.
or
.
Last modified October 05, 2022 at 5:07 PM PST:
Make "Pod" uppercase for consistency in this file (af02f58676)
CPU units
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
1 - StatefulSet Basics
.
Pods
The
kubectl
Note:
tutorial.
concept.
,
nginx
web
.
:
v1
kind
:
Service
metadata
:
name
:
nginx
labels
:
app
:
nginx
spec
:
ports
:
-
port
:
80
name
:
web
clusterIP
:
None
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
web
spec
:
:
"nginx"
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
name
:
web
:
-
name
:
www
mountPath
:
:
-
metadata
:
name
:
www
spec
:
:
[
]
resources
:
requests
:
storage
:
1Gi
web.yaml
app
=
nginx
web.yaml
.
NGINX
nginx
web
web       2         1         20s
n
{0..n-1}
app
=
nginx
NAME      READY     STATUS    R

ESTARTS   AGE
web-1
web-0
Pod is
Running
(see
Pod Phase
)
and
Ready
(see
type
in
).
Note:
.
app
=
nginx
.
.
Since the
web
web-0
and
web-1
.
hostname
for
i in
0
1;
do
kubectl
exec
"web-
$i
"
-- sh -c
;
done
web-0
web-1
Use
to execute
nslookup
dnsutils
package.
Using
nslookup
=
Never --rm

exit
)
app
=
nginx
app
=
nginx
app
=
nginx
Use
and
for
i in
0
1;
do
kubectl
exec
web-
$i
-- sh -c
;
done
web-0
web-1
then, run:

exit
)
The Pods' ordinals, hostnames, SRV records, and A record

 names have not changed,
(
Ready.
,
web-0
and
web-1
:
app
=
nginx
.
. The
spec
for
i in
0
1;
do
kubectl
exec
"web-
$i
"
-- sh -c
;
done
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
Note:
(due to a
),
curl
app
=
nginx
app
=
nginx
app
=
nginx
web-0
web-1
web-0
and
web-1
node
web-0
and
web-1
replicas
or
Scaling Up
app
=
nginx

to 5:
=
5
app
=
nginx
app
=
nginx
Wait for
web-4
and
web-3
app
=
nginx
app
=
nginx
There are still five PersistentVolumeClaims and five PersistentVolumes.
and
OnDelete
.
The
update strategy will update

 all Pods in a StatefulSet, in
Patch the
web
web
=
'json'
-p
=
app
=
nginx -w
for
p in
0
1
2;
do
"web-
$p
"
; echo;
done
Note:
to view
partition
the
.
Patch the
web
field:
=
'json'
-p
=
'[{"op": "replace",

 "path": "/spec/template/spec/containers/0/image", "value":"registry.k8s.io/nginx-slim:0.7"}]'
app
=
nginx -w
partition
.
partition
above
.
Wait for
web-2
app
=
nginx -w
partition
web-2
the
partition
.
Delete the
web-1
Pod:
web-1
app
=
nginx -w
Get the
web-1
web-1
its original configuration.
canary
the
partition
update.
2
0
:
app
=
nginx -w
web-2     1/1       Running    

         0          2m
for
p in
0
1
2;
do
"web-
$p
"
; echo;
done
partition
to
0
On Delete
The
OnDelete
to
OnDelete
.
deleted.
Use
=
orphan
app
=
nginx
web
Delete
web-0
:
app
=
nginx
As the
web
web-0
app
=
nginx
nginx
nginx
app
=
nginx
When the
web
web-0
.
Since
web-1
web-0
with
replicas
web-0
had been recreated, a

nd once
web-1
web-2
was
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
web-0
web-0
app
=
nginx
app
=
nginx
Note:
nginx
files:
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
web-0
and
web-1
nginx
...and the
web
above.
Parallel
:
v1
kind
:
Service
metadata
:
name
:
nginx
labels
:
app
:
nginx
spec
:
ports
:
-
port
:
80
name
:
web
clusterIP
:
None
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
web
spec
:
:
"nginx"
:
replicas
:
2
selector
:
:
ap

p
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
name
:
web
:
-
name
:
www
mountPath
:
:
-
metadata
:
name
:
www
spec
:
:
[
]
resources
:
requests
:
storage
:
1Gi
of the
web
Parallel
.
app
=
nginx -w
app
=
nginx -w
web-0
and
web-1
=
4
kubectl
app
=
nginx -w
nginx
Service:
www-web-0   Bound    pvc-2bf00408-d366-4a12-bad0-1869c65d0bee   1Gi        RWO            standard       2

5m
Note:
2 - Example: Deploying WordPress and MySQL with Persistent Volumes
This tutorial shows you how to deploy a WordPress site and a MySQL database using Minikube. Both applications use PersistentVolumes and PersistentVolumeClaims to store data.
A
(PV) is a piece of storage in the cluster that has been manually provisioned by an administrator, or dynamically provisioned by Kubernetes using a
. A
(PVC) is a request for storage by a user that can be fulfilled by a PV. PersistentVolumes and PersistentVolumeClaims are independent from Pod lifecycles and preserve data through restarting, rescheduling, and even deleting Pods.
Warning:
This deployment is not suitable for production use cases, as it uses single instance WordPress and MySQL Pods. Consider using
WordPress Helm Chart
to deploy WordPress in production.
Note:
The files provided in this tutorial are using GA Deployment APIs and are specific to kubernetes version 1.9 and later. If you wish to use this tutorial with an earlier version of Kubernetes, please update the API version appropriately, or reference earlier versions of this tutorial.
Create PersistentVolumeClaims and PersistentVolumes
Create a
with
a Secret generator
MySQL resource configs
WordPress resource configs
Apply the kustomization directory by
kubectl apply -k ./
Clean up
minikube
.
kubectl
Download the following configuration files:
mysql-deployment.yaml
wordpress-deployment.yaml
MySQL and Wordpress each require a 

PersistentVolume to store data. Their PersistentVolumeClaims will be created at the deployment step.
Many cluster environments have a default StorageClass installed. When a StorageClass is not specified in the PersistentVolumeClaim, the cluster's default StorageClass is used instead.
When a PersistentVolumeClaim is created, a PersistentVolume is dynamically provisioned based on the StorageClass configuration.
Warning:
In local clusters, the default StorageClass uses the
hostPath
provisioner.
hostPath
volumes are only suitable for development and testing. With
hostPath
volumes, your data lives in
/tmp
on the node the Pod is scheduled onto and does not move between nodes. If a Pod dies and gets scheduled to another node in the cluster, or the node is rebooted, the data is lost.
Note:
If you are bringing up a cluster that needs to use the
hostPath
provisioner, the
flag must be set in the
Note:
If you have a Kubernetes cluster running on Google Kubernetes Engine, please follow
this guide
.
Create a kustomization.yaml
Add a Secret generator
A
Secret
is an object that stores a piece of sensitive data like a password or key. Since 1.14,
kubectl
supports the management of Kubernetes objects using a kustomization file. You can create a Secret by generators in
.
Add a Secret generator in
from the following command. You will need to replace
YOUR_PASSWORD
with the password you want to use.
cat
- name: mysql-pass
literals:
- password=YOUR_PASSWORD
EOF
Add resource configs for MySQL and WordPress
The following manifest describes a single-instance MySQL Deployment. The MySQL container mounts the PersistentVolume at /var/lib/mysql. The
MYSQL_ROOT_PASSWORD
environment variable sets the database password from the Secret.
application/wordpress/mysql-deployment.yaml
:
v1
kind
:
Service
metadata
:
name
:
wordpress-mysql
labels
:
app
:
wordpress
spec
:
ports
:
-
port
:
3306
selector
:
app
:
wordpress
tier
:
mysql
clusterIP
:
None
---
:
v1
kind
:
metadata
:
name
:
mysql-pv-claim
labels
:
app
:
wordpress
spec
:
:
resources
:
requests
:
storage
:
20Gi
---
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
selector
:
:
app
:
wordpress
tier
:
mysql
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
wordpress
tier
:
mysql
spec
:
:
-
image
:
mysql:5.6
name
:
mysql
env
:
-
name
:
valueFrom
:
:
name
:
mysql-pass
key
:
password
ports
:
-
:
3306
name
:
mysql
:
-
name
:
mysql-persistent-storage
mountPath
:
/var/lib/mysql
volumes
:
-
name
:
:
claimName
:
The following manifest describes a single-instance WordPress Deployment. The WordPress container mounts the
PersistentVolume at
/var/www/html
for website data files. The
WORDPRESS_DB_HOST
environment variable sets
the name of the MySQL Service defined above, and WordPress will access the database by Service. The
WORDPRESS_DB_PASSWORD
environment variable sets the database password from the Secret kustomize generated.
application/wordpress/wordpress-deployment.yaml
:
v1
kind
:
Service
metadata
:
name
:
wordpress
labels
:
app
:
wordpress
spec
:
ports
:
-
port
:
80
selector
:
app
:
wordpress
tier
:
frontend
type
:
---
:
v1
kind
:
metadata
:
name
:
wp-pv-claim
labels
:
app
:
wordpress
spec
:
:
resources
:
requests
:
storage
:
20Gi
---
:
apps/v1
kind
:
metadata
:
name
:
wordpress
labels
:
app
:
wordpress
spec
:
selector
:
:
app
:
wordpress
tier
:
frontend
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
wordpress
tier
:
frontend
spec
:
:
-
image

:
wordpress:4.8-apache
name
:
wordpress
env
:
-
name
:
value
:
-
name
:
valueFrom
:
:
name
:
key
:
password
ports
:
-
:
80
name
:
wordpress
:
-
name
:
wordpress-persistent-storage
mountPath
:
volumes
:
-
name
:
:
claimName
:
Download the MySQL deployment configuration file.
curl -LO https://k8s.io/examples/application/wordpress/mysql-deployment.yaml
Download the WordPress configuration file.
curl -LO https://k8s.io/examples/application/wordpress/wordpress-deployment.yaml
Add them to
file.
cat
&lt;&lt;EOF &gt;&gt;./kustomization.yaml
- mysql-deployment.yaml
- wordpress-deployment.yaml
EOF
Apply and Verify
The
contains all the resources for deploying a WordPress site and a
MySQL database. You can apply the directory by
Now you can verify that all objects exist.
Verify that the Secret exists by running the following command:
The response should be like this:
NAME                    TYPE                                  DATA   AGE
mysql-pass-c57bb4t7mf   Opaque
1
9s
Verify that a PersistentVolume got dynamically provisioned.
Note:
It can take up to a few minutes for the PVs to be provisioned and bound.
NAME             STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
mysql-pv-claim   Bound     pvc-8cbd7b2e-4044-11e9-b2bb-42010a800002   20Gi       RWO            standard           77s
wp-pv-claim      Bound     pvc-8cd0df54-4044-11e9-b2bb-42010a800002   20Gi       RWO            standard           77s
Verify that the Pod is running by running the following command:
Note:
It can take up to a few minutes for the Pod's Status to be
RUNNING
.
wordpress-mysql-1894417608-x5dzt   1/1       Running   0          40s
Verify that the Service is running by running the following command:
kubectl get services wordpress
NAME        TYPE            CLUSTER-IP   EXTERNAL-IP   PORT(S)        AGE
wordpress   LoadBalancer    10.0.0.89    &lt;pending&gt;     80:32406/TCP   4m
Note:
Minikube can only expose Services through
NodePort
. The EXTERNAL-IP is always pending.
Run the following command to get the IP Address for the WordPress Service:
minikube service wordpress --url
http://1.2.3.4:32406
Copy the IP address, and load the page in your browser to view your site.
You should see the WordPress set up page similar to the following screenshot.
Warning:
Do not leave your WordPress installation on this page. If another user finds it, they can set up a website on your instance and use it to serve malicious content.
Either install WordPress by creating a username and password or delete your instance.
Run the following command to delete your Secret, Deployments, Services and PersistentVolumeClaims:
kubectl delete -k ./
Introspection and Debugging
Jobs
Port Forwarding
Get a Shell to a Container
3 - Example: Deploying Cassandra with a StatefulSet
This tutorial shows you how to run
Apache Cassandra
on Kubernetes.
Cassandra, a database, needs persistent storage to provide data durability (application
state
).
In this example, a custom Cassandra seed provider lets the database discover new Cassandra instances as they join the Cassandra cluster.
make it easier to deploy stateful applications into your Kubernetes cluster.
For more information on the features used in this tutorial, see
.
Note:
Cassandra and Kubernetes both use the term
node
to mean a member of a cluster. In this
tutorial, the Pods th

at belong to the StatefulSet are Cassandra nodes and are members
of the Cassandra cluster (called a
ring
). When those Pods run in your Kubernetes cluster,
the Kubernetes control plane schedules those Pods onto Kubernetes
Nodes
.
When a Cassandra node starts, it uses a
seed list
to bootstrap discovery of other
nodes in the ring.
This tutorial deploys a custom Cassandra seed provider that lets the database discover
new Cassandra Pods as they appear inside your Kubernetes cluster.
Create and validate a Cassandra headless
Service
.
Use a
to create a Cassandra ring.
Validate the StatefulSet.
Modify the StatefulSet.
Delete the StatefulSet and its
Pods
.
minikube
To complete this tutorial, you should already have a basic familiarity with
Pods
,
Services
, and
.
Additional Minikube setup instructions
Caution:
Minikube
defaults to 2048MB of memory and 2 CPU.
Running Minikube with the default resource configuration results in insufficient resource
errors during this tutorial. To avoid these errors, start Minikube with the following settings:
minikube start --memory
5120
--cpus
=
4
Creating a headless Service for Cassandra
In Kubernetes, a
Service
describes a set of
Pods
that perform the same task.
The following Service is used for DNS lookups between Cassandra Pods and clients within your cluster:
application/cassandra/cassandra-service.yaml
:
v1
kind
:
Service
metadata
:
labels
:
app
:
cassandra
name
:
cassandra
spec
:
clusterIP
:
None
ports
:
-
port
:
9042
selector
:
app
:
cassandra
Create a Service to track all Cassandra StatefulSet members from the
cassandra-service.yaml
file:
kubectl apply -f https://k8s.io/examples/application/cassandra/cassandra-service.yaml
Validating (optional)
Get the Cassandra Service.
kubectl get svc cassandra
The response is
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
cassandra   ClusterIP   None         &lt;none&gt;        9042/TCP   45s
If you don't see a Service named
cassandra
, that means creation failed. Read
for help troubleshooting common issues.
Using a StatefulSet to create a Cassandra ring
The StatefulSet manifest, included below, creates a Cassandra ring that consists of three Pods.
Note:
This example uses the default provisioner for Minikube.
Please update the following StatefulSet for the cloud you are working with.
application/cassandra/cassandra-statefulset.yaml
:
apps/v1
kind
:
metadata
:
name
:
cassandra
labels
:
app
:
cassandra
spec
:
:
cassandra
replicas
:
3
selector
:
:
app
:
cassandra
template
:
metadata
:
labels
:
app
:
cassandra
spec
:
:
1800
:
-
name
:
cassandra
image
:
gcr.io/google-samples/cassandra:v13
:
Always
ports
:
-
:
7000
name
:
intra-node
-
:
7001
name
:
tls-intra-node
-
:
7199
name
:
jmx
-
:
9042
name
:
cql
resources
:
limits
:
cpu
:
"500m"
memory
:
1Gi
requests
:
cpu
:
"500m"
memory
:
1Gi
:
:
add
:
- IPC_LOCK
lifecycle
:
preStop
:
exec
:
command
:
- /bin/sh
- -c
- nodetool drain
env
:
-
name
:
MAX_HEAP_SIZE
value
:
512M
-
name
:
HEAP_NEWSIZE
value
:
100M
-
name
:
CASSANDRA_SEEDS
value
:
"cassandra-0.cassandra.default.svc.cluster.local"
-
name
:
CASSANDRA_CLUSTER_NAME
value
:
"K8Demo"
-
name
:
CASSANDRA_DC
value
:
"DC1-K8Demo"
-
name
:
CASSANDRA_RACK
value
:
"Rack1-K8Demo"
-
name
:
POD_IP
valueFrom
:
fieldRef
:
fieldPath
:
:
exec
:
command
:
- /bin/bash
- -c
- /ready-pro

be.sh
:
15
:
5
# These volume mounts are persistent. They are like inline claims,
# but not exactly because the names need to match exactly one of
# the stateful pod volumes.
:
-
name
:
cassandra-data
mountPath
:
/cassandra_data
# These are converted to volume claims by the controller
# and mounted at the paths mentioned above.
# do not use these in production until ssd GCEPersistentDisk or other ssd pd
:
-
metadata
:
name
:
spec
:
:
[
]
:
fast
resources
:
requests
:
storage
:
1Gi
---
kind
:
:
metadata
:
name
:
fast
:
k8s.io/minikube-hostpath
:
type
:
pd-ssd
Create the Cassandra StatefulSet from the
cassandra-statefulset.yaml
file:
# Use this if you are able to apply cassandra-statefulset.yaml unmodified
kubectl apply -f https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml
If you need to modify
to suit your cluster, download
https://k8s.io/examples/application/cassandra/cassandra-statefulset.yaml
and then apply
that manifest, from the folder you saved the modified version into:
# Use this if you needed to modify cassandra-statefulset.yaml locally
kubectl apply -f cassandra-statefulset.yaml
Validating the Cassandra StatefulSet
Get the Cassandra StatefulSet:
kubectl get statefulset cassandra
The response should be similar to:
NAME        DESIRED   CURRENT   AGE
cassandra   3         0         13s
The
resource deploys Pods sequentially.
Get the Pods to see the ordered creation status:
=
"app=cassandra"
NAME          READY     STATUS              RESTARTS   AGE
cassandra-0   1/1       Running
0
1m
cassandra-1   0/1       ContainerCreating
0
8s
It can take several minutes for all three Pods to deploy. Once they are deployed, the same command
returns output similar to:
NAME          READY     STATUS    RESTARTS   AGE
cassandra-0   1/1       Running   0          10m
cassandra-1   1/1       Running   0          9m
cassandra-2   1/1       Running   0          8m
Run the Cassandra
nodetool
inside the first Pod, to
display the status of the ring.
kubectl
exec
-it cassandra-0 -- nodetool status
The response should look something like:
Datacenter: DC1-K8Demo
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  172.17.0.5  83.57 KiB  32           74.0%             e2dd09e6-d9d3-477e-96c5-45094c08db0f  Rack1-K8Demo
UN  172.17.0.4  101.04 KiB  32           58.8%             f89d6835-3a42-4419-92b3-0e62cae1479c  Rack1-K8Demo
UN  172.17.0.6  84.74 KiB  32           67.1%             a6a1e8c2-3dc5-4417-b1a0-26507af2aaad  Rack1-K8Demo
Modifying the Cassandra StatefulSet
Use
to modify the size of a Cassandra StatefulSet.
kubectl edit statefulset cassandra
This command opens an editor in your terminal. The line you need to change is the
replicas
field.
The following sample is an excerpt of the StatefulSet file:
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
:
apps/v1
kind
:
metadata
:
:
2016-08-13T18:40:58Z
:
1
labels
:
app
:
cassandra
name
:
cassandra
namespace
:
default
:
"323"
uid
:
7a219483-6185-11e6-a910-42010a8a0fc0
spec
:
replicas
:
3
Change the number of replicas to 4, and then save the manifest.
The StatefulSet now scales to run with 4 Pods.
Get the Cassandra StatefulSet to verify your change:
cassandra   4         4         36m
Del

eting or scaling a StatefulSet down does not delete the volumes associated with the StatefulSet.
This setting is for your safety because your data is more valuable than automatically purging all related StatefulSet resources.
Warning:
Depending on the storage class and reclaim policy, deleting the
may cause the associated volumes
to also be deleted. Never assume you'll be able to access data if its volume claims are deleted.
Run the following commands (chained together into a single command) to delete everything in the Cassandra StatefulSet:
grace
=
$(
kubectl get pod cassandra-0 -o
=
jsonpath
=
'{.spec.terminationGracePeriodSeconds}'
)
\
kubectl delete statefulset -l
app
=
cassandra
\
echo
"Sleeping
${
grace
}
seconds"
1&gt;&amp;
2
\
sleep
$grace
\
kubectl delete persistentvolumeclaim -l
app
=
cassandra
Run the following command to delete the Service you set up for Cassandra:
app
=
cassandra
Cassandra container environment variables
The Pods in this tutorial use the
image from Google's
container registry
.
The Docker image above is based on
debian-base
and includes OpenJDK 8.
This image includes a standard Cassandra installation from the Apache Debian repo.
By using environment variables you can change values that are inserted into
cassandra.yaml
.
Environment variable
Default value
'Test Cluster'
CASSANDRA_NUM_TOKENS
32
CASSANDRA_RPC_ADDRESS
0.0.0.0
.
KubernetesSeedProvider
See more custom
Seed Provider Configurations
4 - Running ZooKeeper, A Distributed System Coordinator
This tutorial demonstrates running
Apache Zookeeper
on
Kubernetes using
,
,
and
.
Before starting this tutorial, you should be familiar with the following
Kubernetes concepts:
Pods
kubectl CLI
You must have a cluster with at least four nodes, and each node requires at least 2 CPUs and 4 GiB of memory. In this tutorial you will cordon and drain the cluster's nodes.
This means that the cluster will terminate and evict all Pods on its nodes, and the nodes will temporarily become unschedulable.
You should use a dedicated cluster for this tutorial, or you should ensure that the disruption you cause will not interfere with other tenants.
This tutorial assumes that you have configured your cluster to dynamically provision
will have to manually provision three 20 GiB volumes before starting this
tutorial.
After this tutorial, you will know the following.
How to deploy a ZooKeeper ensemble using StatefulSet.
How to consistently configure the ensemble.
How to spread the deployment of ZooKeeper servers in the ensemble.
How to use PodDisruptionBudgets to ensure service availability during planned maintenance.
ZooKeeper
Apache ZooKeeper
is a
distributed, open-source coordination service for distributed applications.
ZooKeeper allows you to read, write, and observe updates to data. Data are
organized in a file system like hierarchy and replicated to all ZooKeeper
servers in the ensemble (a set of ZooKeeper servers). All operations on data
are atomic and sequentially consistent. ZooKeeper ensures this by using the
Zab
consensus protocol to replicate a state machine across all servers in the ensemble.
The ensemble uses the Zab protocol to elect a leader, and the ensemble cannot write data until that election is complete. Once complete, the ensemble uses Zab to ensure that it replicates all writes to a quorum before it acknowledges and makes them visible to clients. Without respect to weighted quorums, a quorum is a majority component of the ensemble containing the current leader. For instan

ce, if the ensemble has three servers, a component that contains the leader and one other server constitutes a quorum. If the ensemble can not achieve a quorum, the ensemble cannot write data.
ZooKeeper servers keep their entire state machine in memory, and write every mutation to a durable WAL (Write Ahead Log) on storage media. When a server crashes, it can recover its previous state by replaying the WAL. To prevent the WAL from growing without bound, ZooKeeper servers will periodically snapshot them in memory state to storage media. These snapshots can be loaded directly into memory, and all WAL entries that preceded the snapshot may be discarded.
Creating a ZooKeeper ensemble
The manifest below contains a
Headless Service
,
a
Service
,
a
,
and a
.
application/zookeeper/zookeeper.yaml
:
v1
kind
:
Service
metadata
:
name
:
zk-hs
labels
:
app
:
zk
spec
:
ports
:
-
port
:
2888
name
:
server
-
port
:
3888
name
:
clusterIP
:
None
selector
:
app
:
zk
---
:
v1
kind
:
Service
metadata
:
name
:
zk-cs
labels
:
app
:
zk
spec
:
ports
:
-
port
:
2181
name
:
client
selector
:
app
:
zk
---
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
selector
:
:
app
:
zk
:
1
---
:
apps/v1
kind
:
metadata
:
name
:
zk
spec
:
selector
:
:
app
:
zk
:
zk-hs
replicas
:
3
:
type
:
:
template
:
metadata
:
labels
:
app
:
zk
spec
:
affinity
:
podAntiAffinity
:
:
-
:
:
-
key
:
"app"
operator
:
In
values
:
- zk
:
"kubernetes.io/hostname"
:
-
name
:
kubernetes-zookeeper
:
Always
image
:
"registry.k8s.io/kubernetes-zookeeper:1.0-3.4.10"
resources
:
requests
:
memory
:
"1Gi"
cpu
:
"0.5"
ports
:
-
:
2181
name
:
client
-
:
2888
name
:
server
-
:
3888
name
:
command
:
- sh
- -c
-
"start-zookeeper \
--servers=3 \
--data_dir=/var/lib/zookeeper/data \
--data_log_dir=/var/lib/zookeeper/data/log \
--conf_dir=/opt/zookeeper/conf \
--client_port=2181 \
--election_port=3888 \
--server_port=2888 \
--tick_time=2000 \
--init_limit=10 \
--sync_limit=5 \
--heap=512M \
--max_client_cnxns=60 \
--snap_retain_count=3 \
--purge_interval=12 \
--max_session_timeout=40000 \
--min_session_timeout=4000 \
--log_level=INFO"
:
exec
:
command
:
- sh
- -c
-
"zookeeper-ready 2181"
:
10
:
5
:
exec
:
command
:
- sh
- -c
-
:
10
:
5
:
-
name
:
datadir
mountPath
:
/var/lib/zookeeper
:
runAsUser
:
1000
fsGroup
:
1000
:
-
metadata
:
name
:
datadir
spec
:
:
[
]
resources
:
requests
:
storage
:
10Gi
Open a terminal, and use the
command to create the
manifest.
kubectl apply -f https://k8s.io/examples/application/zookeeper/zookeeper.yaml
This creates the
zk-hs
Headless Service, the
zk-cs
Service,
the
zk-pdb
PodDisruptionBudget, and the
zk
StatefulSet.
service/zk-hs created
service/zk-cs created
poddisruptionbudget.policy/zk-pdb created
statefulset.apps/zk created
Use
to watch the
StatefulSet controller create the StatefulSet's Pods.
app
=
zk
Once the
zk-2
Pod is Running and Ready, use
CTRL-C
to terminate kubectl.
zk-0      0/1       Pending   0          0s
zk-0      0/1       Pending   0         0s
zk-0      0/1       ContainerCreating   0         0s
zk-0      0/1       Running   0         19s
zk-0      1/1       Running   0         40s
zk-1      0/1       Pending   0         0s
zk-1      0/1       ContainerCreating   0         0s
zk-1      0/1       Running   0         

18s
zk-1      1/1       Running   0         40s
zk-2      0/1       Pending   0         0s
zk-2      0/1       ContainerCreating   0         0s
zk-2      0/1       Running   0         19s
zk-2      1/1       Running   0         40s
The StatefulSet controller creates three Pods, and each Pod has a container with
a
ZooKeeper
server.
Facilitating leader election
Because there is no terminating algorithm for electing a leader in an anonymous network, Zab requires explicit membership configuration to perform leader election. Each server in the ensemble needs to have a unique identifier, all servers need to know the global set of identifiers, and each identifier needs to be associated with a network address.
Use
to get the hostnames
of the Pods in the
zk
for
i in
0
1
2;
do
kubectl
exec
zk-
$i
-- hostname;
done
The StatefulSet controller provides each Pod with a unique hostname based on its ordinal index. The hostnames take the form of
. Because the
replicas
zk
3
, the Set's controller creates three Pods with their hostnames set to
zk-0
,
zk-1
, and
zk-2
.
zk-0
zk-1
zk-2
The servers in a ZooKeeper ensemble use natural numbers as unique identifiers, and store each server's identifier in a file called
myid
in the server's data directory.
To examine the contents of the
myid
file for each server use the following command.
for
i in
0
1
2;
do
echo
"myid zk-
$i
"
;kubectl
exec
zk-
$i
-- cat /var/lib/zookeeper/data/myid;
done
Because the identifiers are natural numbers and the ordinal indices are non-negative integers, you can generate an identifier by adding 1 to the ordinal.
myid zk-0
1
myid zk-1
2
myid zk-2
3
To get the Fully Qualified Domain Name (FQDN) of each Pod in the
zk
StatefulSet use the following command.
for
i in
0
1
2;
do
kubectl
exec
zk-
$i
-- hostname -f;
done
The
zk-hs
Service creates a domain for all of the Pods,
zk-hs.default.svc.cluster.local
.
zk-0.zk-hs.default.svc.cluster.local
zk-1.zk-hs.default.svc.cluster.local
zk-2.zk-hs.default.svc.cluster.local
The A records in
resolve the FQDNs to the Pods' IP addresses. If Kubernetes reschedules the Pods, it will update the A records with the Pods' new IP addresses, but the A records names will not change.
ZooKeeper stores its application configuration in a file named
zoo.cfg
. Use
to view the contents of the
zoo.cfg
file in the
zk-0
Pod.
kubectl
exec
zk-0 -- cat /opt/zookeeper/conf/zoo.cfg
In the
server.1
,
server.2
, and
server.3
properties at the bottom of
the file, the
1
,
2
, and
3
correspond to the identifiers in the
ZooKeeper servers'
myid
files. They are set to the FQDNs for the Pods in
the
zk
clientPort=2181
dataDir=/var/lib/zookeeper/data
dataLogDir=/var/lib/zookeeper/log
tickTime=2000
initLimit=10
syncLimit=2000
maxClientCnxns=60
minSessionTimeout= 4000
maxSessionTimeout= 40000
autopurge.snapRetainCount=3
autopurge.purgeInterval=0
server.1=zk-0.zk-hs.default.svc.cluster.local:2888:3888
server.2=zk-1.zk-hs.default.svc.cluster.local:2888:3888
server.3=zk-2.zk-hs.default.svc.cluster.local:2888:3888
Achieving consensus
Consensus protocols require that the identifiers of each participant be unique. No two participants in the Zab protocol should claim the same unique identifier. This is necessary to allow the processes in the system to agree on which processes have committed which data. If two Pods are launched with the same ordinal, two ZooKeeper servers would both identify themselves as the same server.
app
=
zk
zk-1      0/1       Pending   0       

  0s
zk-1      0/1       Running   0         18s
The A records for each Pod are entered when the Pod becomes Ready. Therefore,
the FQDNs of the ZooKeeper servers will resolve to a single endpoint, and that
endpoint will be the unique ZooKeeper server claiming the identity configured
in its
myid
file.
This ensures that the
servers
properties in the ZooKeepers'
zoo.cfg
files
represents a correctly configured ensemble.
When the servers use the Zab protocol to attempt to commit a value, they will either achieve consensus and commit the value (if leader election has succeeded and at least two of the Pods are Running and Ready), or they will fail to do so (if either of the conditions are not met). No state will arise where one server acknowledges a write on behalf of another.
Sanity testing the ensemble
The most basic sanity test is to write data to one ZooKeeper server and
to read the data from another.
The command below executes the
zkCli.sh
script to write
world
to the path
/hello
on the
zk-0
Pod in the ensemble.
kubectl
exec
zk-0 -- zkCli.sh create /hello world
WATCHER::

WatchedEvent state:SyncConnected type:None path:null
Created /hello
To get the data from the
zk-1
Pod use the following command.
kubectl
exec
zk-1 -- zkCli.sh get /hello
The data that you created on
zk-0
is available on all the servers in the
ensemble.
WATCHER::

world
cZxid = 0x100000002
ctime = Thu Dec 08 15:13:30 UTC 2016
mZxid = 0x100000002
mtime = Thu Dec 08 15:13:30 UTC 2016
pZxid = 0x100000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 5
numChildren = 0
Providing durable storage
ZooKeeper Basics
section,
ZooKeeper commits all entries to a durable WAL, and periodically writes snapshots
in memory state, to storage media. Using WALs to provide durability is a common
technique for applications that use consensus protocols to achieve a replicated
state machine.
Use the
command to delete the
zk
kubectl delete statefulset zk
statefulset.apps "zk" deleted
Watch the termination of the Pods in the StatefulSet.
app
=
zk
When
zk-0
if fully terminated, use
CTRL-C
zk-2      1/1       Terminating   0         9m
zk-0      1/1       Terminating   0         11m
zk-1      1/1       Terminating   0         10m
zk-2      0/1       Terminating   0         9m
zk-1      0/1       Terminating   0         10m
zk-0      0/1       Terminating   0         11m
Reapply the manifest in
zookeeper.yaml
.
zk
StatefulSet object, but the other API objects in the manifest are not modified because they already exist.
Watch the StatefulSet controller recreate the StatefulSet's Pods.
app
=
zk
Once the
zk-2
CTRL-C
zk-0      0/1       ContainerCreatin

g   0         0s
Use the command below to get the value you entered during the
sanity test
,
from the
zk-2
Pod.
kubectl
exec
zk-2 zkCli.sh get /hello
Even though you terminated and recreated all of the Pods in the
zk
StatefulSet, the ensemble still serves the original value.
WATCHER::

world
The
zk
spec
specifies a PersistentVolume provisioned for each Pod.
:
-
metadata
:
name
:
datadir
:
volume.alpha.kubernetes.io/storage-class
:
anything
spec
:
:
[
]
resources
:
requests
:
storage
:
20Gi
The
controller generates a
for each Pod in
the
.
Use the following command to get the
's
.
app
=
zk
When the
recreated its Pods, it remounts the Pods' PersistentVolumes.
NAME           STATUS    VOLUME                                     CAPACITY   ACCESSMODES   AGE
datadir-zk-0   Bound     pvc-bed742cd-bcb1-11e6-994f-42010a800002   20Gi       RWO           1h
datadir-zk-1   Bound     pvc-bedd27d2-bcb1-11e6-994f-42010a800002   20Gi       RWO           1h
datadir-zk-2   Bound     pvc-bee0817e-bcb1-11e6-994f-42010a800002   20Gi       RWO           1h
The
's container
template
mounts the PersistentVolumes in the ZooKeeper servers' data directories.
:
-
name
:
datadir
mountPath
:
When a Pod in the
zk
is (re)scheduled, it will always have the
same
mounted to the ZooKeeper server's data directory.
Even when the Pods are rescheduled, all the writes made to the ZooKeeper
servers' WALs, and all their snapshots, remain durable.
Ensuring consistent configuration
As noted in the
Facilitating Leader Election
and
Achieving Consensus
sections, the servers in a
ZooKeeper ensemble require consistent configuration to elect a leader
and form a quorum. They also require consistent configuration of the Zab protocol
in order for the protocol to work correctly over a network. In our example we
achieve consistent configuration by embedding the configuration directly into
the manifest.
Get the
zk
kubectl get sts zk -o yaml
…
command:
      - sh
      - -c
      - "start-zookeeper \
        --servers=3 \
        --data_dir=/var/lib/zookeeper/data \
        --data_log_dir=/var/lib/zookeeper/data/log \
        --conf_dir=/opt/zookeeper/conf \
        --client_port=2181 \
        --election_port=3888 \
        --server_port=2888 \
        --tick_time=2000 \
        --init_limit=10 \
        --sync_limit=5 \
        --heap=512M \
        --max_client_cnxns=60 \
        --snap_retain_count=3 \
        --purge_interval=12 \
        --max_session_timeout=40000 \
        --min_session_timeout=4000 \
        --log_level=INFO"
…
The command used to start the ZooKeeper servers passed the configuration as command line parameter. You can also use environment variables to pass configuration to the ensemble.
Configuring logging
One of the files generated by the
zkGenConfig.sh
script controls ZooKeeper's logging.
ZooKeeper uses
Log4j
, and, by d

efault,
it uses a time and size based rolling file appender for its logging configuration.
Use the command below to get the logging configuration from one of Pods in the
zk
.
kubectl
exec
zk-0 cat /usr/etc/zookeeper/log4j.properties
The logging configuration below will cause the ZooKeeper process to write all
of its logs to the standard output file stream.
zookeeper.root.logger=CONSOLE
zookeeper.console.threshold=INFO
log4j.rootLogger=${zookeeper.root.logger}
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.Threshold=${zookeeper.console.threshold}
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} [myid:%X{myid}] - %-5p [%t:%C{1}@%L] - %m%n
This is the simplest possible way to safely log inside the container.
Because the applications write logs to standard out, Kubernetes will handle log rotation for you.
Kubernetes also implements a sane retention policy that ensures application logs written to
standard out and standard error do not exhaust local storage media.
Use
to retrieve the last 20 log lines from one of the Pods.
kubectl logs zk-0 --tail
20
You can view application logs written to standard out or standard error using
and from the Kubernetes Dashboard.
2016-12-06 19:34:16,236 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52740
2016-12-06 19:34:16,237 [myid:1] - INFO  [Thread-1136:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52740 (no session established for client)
2016-12-06 19:34:26,155 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52749
2016-12-06 19:34:26,155 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52749
2016-12-06 19:34:26,156 [myid:1] - INFO  [Thread-1137:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52749 (no session established for client)
2016-12-06 19:34:26,222 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52750
2016-12-06 19:34:26,222 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52750
2016-12-06 19:34:26,226 [myid:1] - INFO  [Thread-1138:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52750 (no session established for client)
2016-12-06 19:34:36,151 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52760
2016-12-06 19:34:36,152 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52760
2016-12-06 19:34:36,152 [myid:1] - INFO  [Thread-1139:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52760 (no session established for client)
2016-12-06 19:34:36,230 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52761
2016-12-06 19:34:36,231 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52761
2016-12-06 19:34:36,231 [myid:1] - INFO  [Thread-1140:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52761 (no session established for client)
2016-12-06 19:34:46,149 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52767
2016-12-06 19:34:46,149 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52767
2016-12-06 19:34:46,149 [myid:1] - INFO  [Thread-1141:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52767 (no session establi

shed for client)
2016-12-06 19:34:46,230 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /127.0.0.1:52768
2016-12-06 19:34:46,230 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@827] - Processing ruok command from /127.0.0.1:52768
2016-12-06 19:34:46,230 [myid:1] - INFO  [Thread-1142:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52768 (no session established for client)
Kubernetes integrates with many logging solutions. You can choose a logging solution
that best fits your cluster and applications. For cluster-level logging and aggregation,
consider deploying a
sidecar container
to rotate and ship your logs.
Configuring a non-privileged user
The best practices to allow an application to run as a privileged
user inside of a container are a matter of debate. If your organization requires
that applications run as a non-privileged user you can use a
to control the user that
the entry point runs as.
The
zk
's Pod
template
contains a
.
:
runAsUser
:
1000
fsGroup
:
1000
In the Pods' containers, UID 1000 corresponds to the zookeeper user and GID 1000
corresponds to the zookeeper group.
Get the ZooKeeper process information from the
zk-0
Pod.
kubectl
exec
zk-0 -- ps -elf
As the
runAsUser
object is set to 1000,
instead of running as root, the ZooKeeper process runs as the zookeeper user.
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S zookeep+     1     0  0  80   0 -  1127 -      20:46 ?        00:00:00 sh -c zkGenConfig.sh &amp;&amp; zkServer.sh start-foreground
0 S zookeep+    27     1  0  80   0 - 1155556 -    20:46 ?        00:00:19 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg
By default, when the Pod's PersistentVolumes is mounted to the ZooKeeper server's data directory, it is only accessible by the root user. This configuration prevents the ZooKeeper process from writing to its WAL and storing its snapshots.
Use the command below to get the file permissions of the ZooKeeper data directory on the
zk-0
Pod.
kubectl
exec
-ti zk-0 -- ls -ld /var/lib/zookeeper/data
fsGroup
object is set to 1000, the ownership of the Pods' PersistentVolumes is set to the zookeeper group, and the ZooKeeper process is able to read and write its data.
drwxr-sr-x 3 zookeeper zookeeper 4096 Dec  5 20:45 /var/lib/zookeeper/data
Managing the ZooKeeper process
The
ZooKeeper documentation
mentions that "You will want to have a supervisory process that
manages each of your ZooKeeper server processes (JVM)." Utilizing a watchdog
(supervisory process) to restart failed processes in a distributed system is a
common pattern. When deploying an application in Kubernetes, rather than using
an external utility as a supervisory process, you should use Kubernetes as the
watchdog for your application.
Updating the ensemble
The
zk
is configured to use the
update strategy.
You can use
to update the number of
cpus
allocated to the servers.
kubectl patch sts zk --type
=
'json'
-p
=
'[{"op": "replace", "path": "/spec/template/spec/containers/0/resources/requests/cpu", "value":"0.3"}]'
statefulset.apps/zk patched
U

se
kubectl rollout status
to watch the status of the update.
kubectl rollout status sts/zk
waiting for statefulset rolling update to complete 0 pods at revision zk-5db4499664...
Waiting for 1 pods to be ready...
waiting for statefulset rolling update to complete 1 pods at revision zk-5db4499664...
waiting for statefulset rolling update to complete 2 pods at revision zk-5db4499664...
statefulset rolling update complete 3 pods at revision zk-5db4499664...
This terminates the Pods, one at a time, in reverse ordinal order, and recreates them with the new configuration. This ensures that quorum is maintained during a rolling update.
Use the
kubectl rollout history
command to view a history or previous configurations.
history
sts/zk
statefulsets "zk"
REVISION
1
2
Use the
kubectl rollout undo
command to roll back the modification.
kubectl rollout undo sts/zk
statefulset.apps/zk rolled back
Handling process failure
Restart Policies
control how
Kubernetes handles process failures for the entry point of the container in a Pod.
For Pods in a
, the only appropriate
RestartPolicy
is Always, and this
is the default value. For stateful applications you should
never
override
the default policy.
Use the following command to examine the process tree for the ZooKeeper server running in the
zk-0
Pod.
kubectl
exec
zk-0 -- ps -ef
The command used as the container's entry point has PID 1, and
the ZooKeeper process, a child of the entry point, has PID 27.
UID        PID  PPID  C STIME TTY          TIME CMD
zookeep+     1     0  0 15:03 ?        00:00:00 sh -c zkGenConfig.sh &amp;&amp; zkServer.sh start-foreground
zookeep+    27     1  0 15:03 ?        00:00:03 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg
In another terminal watch the Pods in the
zk
with the following command.
app
=
zk
In another terminal, terminate the ZooKeeper process in Pod
zk-0
kubectl
exec
zk-0 -- pkill java
The termination of the ZooKeeper process caused its parent process to terminate. Because the
of the container is Always, it restarted the parent process.
zk-0      1/1       Running   0          21m
zk-1      1/1       Running   0          20m
zk-2      1/1       Running   0          19m
zk-0      0/1       Error     0          29m
zk-0      0/1       Running   1         29m
zk-0      1/1       Running   1         29m
If your application uses a script (such as
zkServer.sh
) to launch the process
that implements the application's business logic, the script must terminate with the
child process. This ensures that Kubernetes will restart the application's
container when the process implementing the application's business logic fails.
Testing for liveness
Configuring your application to restart failed processes is not enough to
keep a distributed system healthy. There are scenarios where
a system's processes can be both alive and unresponsive, or otherwise
unhealthy. You should use liveness probes to notify Kubernetes

that your application's processes are unhealthy and it should restart them.
The Pod
template
for the
zk
specifies a liveness probe.
:
exec
:
command
:
- sh
- -c
-
:
15
:
5
The probe calls a bash script that uses the ZooKeeper
ruok
four letter
word to test the server's health.
OK=$(echo ruok | nc 127.0.0.1 $1)
if [ "$OK" == "imok" ]; then
    exit 0
else
    exit 1
fi
In one terminal window, use the following command to watch the Pods in the
zk
app
=
zk
In another window, using the following command to delete the
zookeeper-ready
script from the file system of Pod
zk-0
.
kubectl
exec
zk-0 -- rm /opt/zookeeper/bin/zookeeper-ready
When the liveness probe for the ZooKeeper process fails, Kubernetes will
automatically restart the process for you, ensuring that unhealthy processes in
the ensemble are restarted.
app
=
zk
zk-0      1/1       Running   0          1h
zk-1      1/1       Running   0          1h
zk-2      1/1       Running   0          1h
zk-0      0/1       Running   0          1h
zk-0      0/1       Running   1         1h
zk-0      1/1       Running   1         1h
Testing for readiness
Readiness is not the same as liveness. If a process is alive, it is scheduled
and healthy. If a process is ready, it is able to process input. Liveness is
a necessary, but not sufficient, condition for readiness. There are cases,
particularly during initialization and termination, when a process can be
alive but not ready.
If you specify a readiness probe, Kubernetes will ensure that your application's
processes will not receive network traffic until their readiness checks pass.
For a ZooKeeper server, liveness implies readiness. Therefore, the readiness
probe from the
manifest is identical to the liveness probe.
:
exec
:
command
:
- sh
- -c
-
:
15
:
5
Even though the liveness and readiness probes are identical, it is important
to specify both. This ensures that only healthy servers in the ZooKeeper
ensemble receive network traffic.
Tolerating Node failure
ZooKeeper needs a quorum of servers to successfully commit mutations
to data. For a three server ensemble, two servers must be healthy for
writes to succeed. In quorum based systems, members are deployed across failure
domains to ensure availability. To avoid an outage, due to the loss of an
individual machine, best practices preclude co-locating multiple instances of the
application on the same machine.
By default, Kubernetes may co-locate Pods in a
on the same node.
For the three server ensemble you created, if two servers are on the same node, and that node fails,
the clients of your ZooKeeper service will experience an outage until at least one of the Pods can be rescheduled.
You should always provision additional capacity to allow the processes of critical
systems to be rescheduled in the event of node failures. If you do so, then the
outage will only last until the Kubernetes scheduler reschedules one of the ZooKeeper
servers. However, if you want your service to tolerate node failures with no downtime,
you should set
.
Use the command below to get the nodes for Pods in the
zk
.
for
i in
0
1
2;
do
kubectl get pod zk-
$i
{{
.spec.nodeName
}}
;
echo
""
;
done
All of the Pods in the
zk
are deployed on different nodes.
kubernetes-node-cxpk
kubernetes-node-a5aq
kubernetes-node-2g2d
This is because the Pods in the
zk
have a
affinity
:
:
:
-
:
:
-
key
:
"app"
operator
:
In
values
:
- zk
:
The
field t

ells the
Kubernetes Scheduler that it should never co-locate two Pods which have
app
label
as
zk
in the domain defined by the
. The
indicates that the domain is an individual node. Using
different rules, labels, and selectors, you can extend this technique to spread
your ensemble across physical, network, and power failure domains.
Surviving maintenance
In this section you will cordon and drain nodes. If you are using this tutorial
on a shared cluster, be sure that this will not adversely affect other tenants.
The previous section showed you how to spread your Pods across nodes to survive
unplanned node failures, but you also need to plan for temporary node failures
that occur due to planned maintenance.
Use this command to get the nodes in your cluster.
This tutorial assumes a cluster with at least four nodes. If the cluster has more than four, use
to cordon all but four nodes. Constraining to four nodes will ensure Kubernetes encounters affinity and PodDisruptionBudget constraints when scheduling zookeeper Pods in the following maintenance simulation.
kubectl cordon &lt;node-name&gt;
Use this command to get the
zk-pdb
.
kubectl get pdb zk-pdb
The
max-unavailable
field indicates to Kubernetes that at most one Pod from
zk
can be unavailable at any time.
NAME      MIN-AVAILABLE   MAX-UNAVAILABLE   ALLOWED-DISRUPTIONS   AGE
zk-pdb    N/A             1                 1
In one terminal, use this command to watch the Pods in the
zk
.
app
=
zk
In another terminal, use this command to get the nodes that the Pods are currently scheduled on.
for
i in
0
1
2;
do
$i
{{
}}
;
echo
""
;
done
kubernetes-node-pb41
kubernetes-node-ixsl
kubernetes-node-i4c4
Use
to cordon and
drain the node on which the
zk-0
Pod is scheduled.
$(
kubectl get pod zk-0 --template
{{
}}
)
--ignore-daemonsets --force --delete-emptydir-data
node "kubernetes-node-pb41" cordoned

WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-pb41, kube-proxy-kubernetes-node-pb41; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-o5elz
pod "zk-0" deleted
node "kubernetes-node-pb41" drained
As there are four nodes in your cluster,
, succeeds and the
zk-0
is rescheduled to another node.
zk-0      1/1       Running   2          1h
zk-0      1/1       Terminating   2          2h
zk-0      0/1       Terminating   2         2h
zk-0      0/1       Running   0         51s
zk-0      1/1       Running   0         1m
Keep watching the
's Pods in the first terminal and drain the node on which
zk-1
is scheduled.
$(
kubectl get pod zk-1 --template
{{
}}
)
"kubernetes-node-ixsl" cordoned
WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-ixsl, kube-proxy-kubernetes-node-ixsl; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-voc74
pod "zk-1" deleted
node "kubernetes-node-ixsl" drained
The
zk-1
Pod cannot be scheduled because the
zk
contains a
rule preventing
co-location of the Pods, and as only two nodes are schedulable, the Pod will remain in a Pending state.
k

ubectl get pods -w -l
app
=
zk
zk-1      1/1       Terminating   0         2h
zk-1      0/1       Terminating   0         2h
Continue to watch the Pods of the StatefulSet, and drain the node on which
zk-2
$(
kubectl get pod zk-2 --template
{{
}}
)
node "kubernetes-node-i4c4" cordoned

WARNING: Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog
WARNING: Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-dyrog; Deleting pods not managed by ReplicationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-i4c4, kube-proxy-kubernetes-node-i4c4
There are pending pods when an error occurred: Cannot evict pod as it would violate the pod's disruption budget.
pod/zk-2
Use
CTRL-C
You cannot drain the third node because evicting
zk-2
would violate
zk-budget
. However, the node will remain cordoned.
Use
zkCli.sh
to retrieve the value you entered during the sanity test from
zk-0
.
kubectl
exec
zk-0 zkCli.sh get /hello
The service is still available because its
is respected.
world
cZxid = 0x200000002
ctime = Wed Dec 07 00:08:59 UTC 2016
mZxid = 0x200000002
mtime = Wed Dec 07 00:08:59 UTC 2016
pZxid = 0x200000002
Use
to uncordon the first node.
kubectl uncordon kubernetes-node-pb41
node "kubernetes-node-pb41" uncordoned
zk-1
is rescheduled on this node. Wait until
zk-1
is Running and Ready.
app
=
zk
zk-1      0/1       Pending   0         12m
zk-1      0/1       ContainerCreating   0         12m
zk-1      0/1       Running   0         13m
zk-1      1/1       Running   0         13m
Attempt to drain the node on which
zk-2
$(
{{
.spec.n

odeName
}}
)
node "kubernetes-node-i4c4" already cordoned
pod "heapster-v1.2.0-2604621511-wht1r" deleted
pod "zk-2" deleted
node "kubernetes-node-i4c4" drained
This time
succeeds.
Uncordon the second node to allow
zk-2
to be rescheduled.
kubectl uncordon kubernetes-node-ixsl
node "kubernetes-node-ixsl" uncordoned
You can use
in conjunction with
to ensure that your services remain available during maintenance.
If drain is used to cordon nodes and evict pods prior to taking the node offline for maintenance,
services that express a disruption budget will have that budget respected.
You should always allocate additional capacity for critical services so that their Pods can be immediately rescheduled.
Use
to uncordon all the nodes in your cluster.
You must delete the persistent storage media for the PersistentVolumes used in this tutorial.
Home
Blog
Training
Partners
Community
CC BY 4.0

Authorization Overview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
see
.
user
- The
user
group
extra
API
/api
or
/healthz
.
get
,
list
,
create
,
update
,
patch
,
watch
,
delete
, and
.
get
,
post
,
put
, and
delete
Resource
get
,
update
,
patch
, and
delete

Namespace
API group
- The
API Group
core
API group
.
or
GET
/api
or
/healthz
would use
get
HTTP verb
POST
create
GET, HEAD
PUT
update
PATCH
patch
DELETE
Caution:
The
get
,
list
and
watch
list
on
secrets
data
RBAC
bind
and
escalate
verbs on
roles
and
API group.
verb on
users
,
groups
, and
in the
API group.
Node
.
ABAC
ABAC Mode
.
RBAC
RBAC Mode
.
Webhook
.
kubectl
auth can-i
yes
no
no
dev-sa
dev
can list 

Pods in the Namespace
target
:
\
\
yes
- Like
spec:
EOF
is:
:
kind
:
metadata
:
:
null
spec
:
:
group
:
apps
resource
:
namespace
:
dev
verb
:
create
status
:
allowed
:
true
denied
:
false
Caution:
Caution:
in
.
Using Admission C

ontrollers
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Other Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
1 - ValidatingAdmissionPolicyBindingList v1alpha1
Home
Blog
Training
Partners
Community
CC BY 4.0

VolumeAttachment | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on ma

cOS
Debug Pods
Troubl

eshooting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Reso

urces with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy 

Configuration (v1alpha1)
Reference
kind
metadata
(
)
spec
(
status
(
)
attacher
nodeName
source
(
)
(string)
attached
(st

ring)
(Time)
attachmentMetadata
(string)
(Time)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
Par

ameters
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
partially update the specified VolumeAttachment
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved

. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Imperative Management of Kubernetes Objects Using Configuration Files | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Ku

 (v1)
Tasks
Kubernetes objects can be created, updated, and deleted by using the
kubectl
command-line tool along with an object configuration file written in YAML or JSON.
This document explains how to define and manage objects using configuration files.
Install
kubectl
.
minikube
.
The
kubectl
See
You can use
kubectl create -f
to create an object from a configuration file.
kubernetes API reference
kubectl create -f &lt;filename|url&gt;
Warning:
Updating objects with the
replace
command drops all
parts of the spec not specified in the configuration file. This
should not be used with objects whose specs are partially managed
by the cluster, such as Services of type
, where
the
field is managed independently from the configuration
file. Independently managed fields must be copied to the configuration
file to prevent
replace
from dropping them.
You can use
kubectl replace -f
to update a live object according to a
kubectl replace -f &lt;filename|url&gt;
You can use
kubectl delete -f
to delete an object that is described in a
kubectl delete -f &lt;filename|url&gt;
Note:
If configuration file has specified the
metadata
section instead of the
name
field, you cannot delete the object using
.
You will have to use other flags for deleting the object. For example:
kubectl delete &lt;type&gt; &lt;name&gt;
kubectl delete &lt;type&gt; -l &lt;label&gt;
You can us

e
kubectl get -f
to view information about an object that is
described in a configuration file.
The
-o yaml
flag specifies that the full object configuration is printed.
Use
kubectl get -h
The
create
,
replace
, and
delete
commands work well when each object's
configuration is fully defined and recorded in its configuration
file. However when a live object is updated, and the updates are not merged
into its configuration file, the updates will be lost the next time a
replace
is executed. This can happen if a controller, such as
a HorizontalPodAutoscaler, makes updates directly to a live object. Here's
an example:
You create an object from a configuration file.
Another source updates the object by changing some field.
You replace the object from the configuration file. Changes made by
the other source in step 2 are lost.
If you need to support multiple writers to the same object, you can use
to manage the object.
Creating and editing an object from a URL without saving the configuration
Suppose you have the URL of an object configuration file. You can use
to make changes to the configuration before the
object is created. This is particularly useful for tutorials and tasks
that point to a configuration file that could be modified by the reader.
kubectl create -f &lt;url&gt; --edit
Migrating from imperative commands to imperative object configuration
Migrating from imperative commands to imperative object configuration involves
several manual steps.
Export the live object to a local object configuration file:
Manually remove the status field from the object configuration file.
For subsequent object management, use
replace
exclusively.
kubectl replace -f &lt;kind&gt;_&lt;name&gt;.yaml
Warning:
Example label:
selector
:
:
:
template
:
metadata
:
labels
:
:
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Reference Documentation Quickstart | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

Reference Documentation Quickstart
update-imported-docs.py
script to generate
the Kubernetes reference documentation. The script automates
the build setup and generates the reference documentation for a release.
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
Getting the docs repository
Make sure your
website
fork is up-to-date with the
remote on
GitHub (
main
branch), and clone your
website
fork.
mkdir github.com
cd
github.com
git clone git@github.com:&lt;your_github_username&gt;/website.git
Determine the base directory of your clone. For example, if you followed the
preceding step to get the repository, your base directory is
github.com/website.
.
Note:
If you want to change the content of the component tools and API reference,
see the
contributing upstream guide
.
Overview of update-imported-docs
The
script is located in the
&lt;web-base&gt;/update-imported-docs/
The script builds the following references:
Component and tool reference pages
The
kubectl
The Kubernetes API reference
The
script generates the Kubernetes reference documentation
from the Kubernetes source code. The script creates a temporary directory
under
/tmp
on your machine and clones the required repositories:
and
into this directory.
The script sets your
GOPATH
to this temporary directory.
Three additional environment variables are set:
K8S_ROOT
The script requires two arguments to run successfully:
A YAML configuration file (
reference.yml
)
A release version, for example:
1.17
The configuration file contains a
generate-command
field.
The
field defines a series of build instructions
from
kubernetes-sigs/reference-docs/Makefile
. The
variable
determines the version of the release.
The
updat

e-imported-docs.py
script performs the following steps:
Clones the related repositories specified in a configuration file. For the
purpose of generating reference docs, the repository that is cloned by
default is
.
Runs commands under the cloned repositories to prepare the docs generator and
then generates the HTML and Markdown files.
Copies the generated HTML and Markdown files to a local clone of the
repository under locations specified in the configuration file.
Updates
kubectl
command links from
kubectl
.md to the refer to
the sections in the
kubectl
command reference.
When the generated files are in your local clone of the
repository, you can submit them in a
to
.
Each configuration file may contain multiple repos that will be imported together. When
necessary, you can customize the configuration file by manually editing it. You
may create new config files for importing other groups of documents.
The following is an example of the YAML configuration file:
repos
:
-
name
:
community
remote
:
https://github.com/kubernetes/community.git
branch
:
master
files
:
-
src
:
contributors/devel/README.md
dst
:
docs/imported/community/devel.md
-
src
:
contributors/guide/README.md
dst
:
docs/imported/community/guide.md
Single page Markdown documents, imported by the tool, must adhere to
the
.
Customizing reference.yml
Open
&lt;web-base&gt;/update-imported-docs/reference.yml
for editing.
Do not change the content for the
field unless you understand
how the command is used to build the references.
You should not need to update
. At times, changes in the
upstream source code, may require changes to the configuration file
(for example: golang version dependencies and third-party library changes).
If you encounter build issues, contact the SIG-Docs team on the
#sig-docs Kubernetes Slack channel
.
Note:
The
is an optional entry, which can be used to run a
given command or a short script to generate the docs from within a repository.
In
,
files
contains a list of
src
and
dst
fields.
The
src
field contains the location of a generated Markdown file in the cloned
build directory, and the
dst
field specifies
where to copy this file in the cloned
repos
:
-
name
:
reference-docs
remote
:
https://github.com/kubernetes-sigs/reference-docs.git
files
:
-
src
:
gen-compdocs/build/kube-apiserver.md
dst
:
content/en/docs/reference/command-line-tools-reference/kube-apiserver.md
...
Note that when there are many files to be copied from the same source directory
to the same destination directory, you can use wildcards in the value given to
src
. You must provide the directory name as the value for
dst
.
files
:
-
src
:
gen-compdocs/build/kubeadm*.md
dst
:
content/en/docs/reference/setup-tools/kubeadm/generated/
Running the update-imported-docs tool
tool as follows:
cd
&lt;web-base&gt;/update-imported-docs
./update-imported-docs.py &lt;configuration-file.yml&gt; &lt;release-version&gt;
./update-imported-docs.py reference.yml 1.17
Fixing Links
The
release.yml
configuration file contains instructions to fix relative links.
To fix relative links within your imported files, set the
gen-absolute-links
property to
true
. You can find an example of this in
.
Adding and committing changes in kubernetes/website
List the files that were generated and copied to
:
cd
git status
The output shows the new and modified files. The generated output varies
depending upon changes made to the upstream source code.
Generated component tool files
content/en/docs/reference/command-line-tools-reference/cloud-controller-manager.md
content/en/docs/reference/command-line-tools-reference/kube-apise

rver.md
content/en/docs/reference/command-line-tools-reference/kube-controller-manager.md
content/en/docs/reference/command-line-tools-reference/kube-proxy.md
content/en/docs/reference/command-line-tools-reference/kube-scheduler.md
content/en/docs/reference/setup-tools/kubeadm/generated/kubeadm.md
content/en/docs/reference/kubectl/kubectl.md
Generated kubectl command reference files
static/docs/reference/generated/kubectl/kubectl-commands.html
static/docs/reference/generated/kubectl/navData.js
static/docs/reference/generated/kubectl/scroll.js
static/docs/reference/generated/kubectl/stylesheet.css
static/docs/reference/generated/kubectl/tabvisibility.js
static/docs/reference/generated/kubectl/node_modules/bootstrap/dist/css/bootstrap.min.css
static/docs/reference/generated/kubectl/node_modules/highlight.js/styles/default.css
static/docs/reference/generated/kubectl/node_modules/jquery.scrollto/jquery.scrollTo.min.js
static/docs/reference/generated/kubectl/node_modules/jquery/dist/jquery.min.js
static/docs/reference/generated/kubectl/css/font-awesome.min.css
Generated Kubernetes API reference directories and files
static/docs/reference/generated/kubernetes-api/v1.26/js/query.scrollTo.min.js
Run
git add
and
git commit
to commit the files.
Creating a pull request
Create a pull request to the
repository. Monitor your
pull request, and respond to review comments as needed. Continue to monitor
your pull request until it is merged.
A few minutes after your pull request is merged, your updated reference
topics will be visible in the
published documentation
.
To generate the individual reference documentation by manually setting up the required build repositories and
running the build targets, see the following guides:
Generating Reference Documentation for Kubernetes Components and Tools
Feedback
Yes
No
.
or
.
Last modified October 27, 2022 at 3:04 PM PST:
Tweak reference docs section index (3c33ffc914)
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, pleas

Horizontal Pod Autoscaling | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

Tasks
a
or
), with the
Pods
.
vertical
Horizontal pod autoscaling does not apply to objects that can't be scaled (for example:
a
DaemonSet
.)
The HorizontalPodAutoscaler is implemented as a Kubernetes API resource and a
.
The resource determines the behavior of the controller.
The horizontal pod autoscaling controller, running within the Kubernetes
, periodically adjusts the
desired scale of its target (for example, a Deployment) to match observed metrics such as average
CPU utilization, average memory utilization, or any other custom metric you specify.
There is
walkthrough example
of using
horizontal pod autoscaling.
How does a HorizontalPodAutoscaler work?
graph BT
hpa[Horizontal Pod Autoscaler] --&gt; scale[Scale]
subgraph rc[RC / Deployment]
scale
end
scale -.-&gt; pod1[Pod 1]
scale -.-&gt; pod2[Pod 2]
scale -.-&gt; pod3[Pod N]
classDef hpa fill:#D5A6BD,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D;
classDef rc fill:#F9CB9C,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D;
classDef scale fill:#B6D7A8,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D;
classDef pod fill:#9FC5E8,stroke:#1E1E1D,stroke-width:1px,color:#1E1E1D;
class hpa hpa;
class rc rc;
class scale scale;
class pod1,pod2,pod3 pod
enabled
Figure 1. HorizontalPodAutoscaler controls the scale of a Deployment and its ReplicaSet
Kubernetes implements horizontal pod autoscaling as a control loop that runs intermittently
(it is not a continuous process). The interval is set by the
--horizontal-pod-autoscaler-sync-period
(and the default interval is 15 seconds).
Once during each period, the controller manager queries the resource utilization against the
metrics specified in each HorizontalPodAutoscaler definition. The controller manager
finds the target resou

rce defined by the
,
then selects the pods based on the target resource's
labels, and obtains the metrics from either the resource metrics API (for per-pod resource metrics),
or the custom metrics API (for all other metrics).
For per-pod resource metrics (like CPU), the controller fetches the metrics
from the resource metrics API for each Pod targeted by the HorizontalPodAutoscaler.
Then, if a target utilization value is set, the controller calculates the utilization
value as a percentage of the equivalent
resource request
on the containers in each Pod. If a target raw value is set, the raw metric values are used directly.
The controller then takes the mean of the utilization or the raw value (depending on the type
of target specified) across all targeted Pods, and produces a ratio used to scale
the number of desired replicas.
Please note that if some of the Pod's containers do not have the relevant resource request set,
CPU utilization for the Pod will not be defined and the autoscaler will
not take any action for that metric. See the
algorithm details
section below
for more information about how the autoscaling algorithm works.
For per-pod custom metrics, the controller functions similarly to per-pod resource metrics,
except that it works with raw values, not utilization values.
For object metrics and external metrics, a single metric is fetched, which describes
the object in question. This metric is compared to the target
value, to produce a ratio as above. In the
API
version, this value can optionally be divided by the number of Pods before the
comparison is made.
The common use for HorizontalPodAutoscaler is to configure it to fetch metrics from
aggregated APIs
(
,
, or
). The
API is
usually provided by an add-on named Metrics Server, which needs to be launched separately.
For more information about resource metrics, see
.
Support for metrics APIs
explains the stability guarantees and support status for these
different APIs.
The HorizontalPodAutoscaler controller accesses corresponding workload resources that support scaling (such as Deployments
and StatefulSet). These resources each have a subresource named
scale
, an interface that allows you to dynamically set the
number of replicas and examine each of their current states.
For general information about subresources in the Kubernetes API, see
.
From the most basic perspective, the HorizontalPodAutoscaler controller
operates on the ratio between desired metric value and current metric
value:
desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]
For example, if the current metric value is
200m
, and the desired value
is
100m
, the number of replicas will be doubled, since
200.0 / 100.0 == 2.0
If the current value is instead
50m
, you'll halve the number of
replicas, since
50.0 / 100.0 == 0.5
. The control plane skips any scaling
action if the ratio is sufficiently close to 1.0 (within a globally-configurable
tolerance, 0.1 by default).
When a
targetAverageValue
or
targetAverageUtilization
is specified,
the
currentMetricValue
is computed by taking the average of the given
metric across all Pods in the HorizontalPodAutoscaler's scale target.
Before checking the tolerance and deciding on the final values, the control
plane also considers whether any metrics are missing, and how many Pods
are
Ready
.
All Pods with a deletion timestamp set (objects with a deletion timestamp are
in the process of being shut down / removed) are ignored, and all failed Pods
are discarded.
If a particular Pod is missing metrics, it is set aside for later; Pods
with missing metrics will be used to adjust the final scaling amount.
When scaling on CPU, if any pod has yet to become ready (it's still
initializing, or possibly is unhealthy)
or
the most recent metric point for
the pod was before i

t became ready, that pod is set aside as well.
Due to technical constraints, the HorizontalPodAutoscaler controller
cannot exactly determine the first time a pod becomes ready when
determining whether to set aside certain CPU metrics. Instead, it
considers a Pod "not yet ready" if it's unready and transitioned to
ready within a short, configurable window of time since it started.
This value is configured with the
--horizontal-pod-autoscaler-initial-readiness-delay
flag, and its default is 30
seconds. Once a pod has become ready, it considers any transition to
ready to be the first if it occurred within a longer, configurable time
since it started. This value is configured with the
--horizontal-pod-autoscaler-cpu-initialization-period
flag, and its
default is 5 minutes.
The
currentMetricValue / desiredMetricValue
base scale ratio is then
calculated using the remaining pods not set aside or discarded from above.
If there were any missing metrics, the control plane recomputes the average more
conservatively, assuming those pods were consuming 100% of the desired
value in case of a scale down, and 0% in case of a scale up. This dampens
the magnitude of any potential scale.
Furthermore, if any not-yet-ready pods were present, and the workload would have
scaled up without factoring in missing metrics or not-yet-ready pods,
the controller conservatively assumes that the not-yet-ready pods are consuming 0%
of the desired metric, further dampening the magnitude of a scale up.
After factoring in the not-yet-ready pods and missing metrics, the
controller recalculates the usage ratio. If the new ratio reverses the scale
direction, or is within the tolerance, the controller doesn't take any scaling
action. In other cases, the new ratio is used to decide any change to the
number of Pods.
original
value for the average utilization is reported
back via the HorizontalPodAutoscaler status, without factoring in the
not-yet-ready pods or missing metrics, even when the new usage ratio is
used.
If multiple metrics are specified in a HorizontalPodAutoscaler, this
calculation is done for each metric, and then the largest of the desired
replica counts is chosen. If any of these metrics cannot be converted
into a desired replica count (e.g. due to an error fetching the metrics
from the metrics APIs) and a scale down is suggested by the metrics which
can be fetched, scaling is skipped. This means that the HPA is still capable
of scaling up if one or more metrics give a
greater than
the current value.
Finally, right before HPA scales the target, the scale recommendation is recorded. The
controller considers all recommendations within a configurable window choosing the
highest recommendation from within that window. This value can be configured using the
--horizontal-pod-autoscaler-downscale-stabilization
flag, which defaults to 5 minutes.
This means that scaledowns will occur gradually, smoothing out the impact of rapidly
fluctuating metric values.
API Object
The Horizontal Pod Autoscaler is an API resource in the Kubernetes
API group. The current stable version can be found in
the
API version which includes support for scaling on
memory and custom metrics. The new fields introduced in
are preserved as annotations when working with
.
When you create a HorizontalPodAutoscaler API object, make sure the name specified is a valid
.
More details about the API object can be found at
HorizontalPodAutoscaler Object
.
Stability of workload scale
When managing the scale of a group of replicas using the HorizontalPodAutoscaler,
it is possible that the number of replicas keeps fluctuating frequently due to the
dynamic nature of the metrics evaluated. This is sometimes referred to as
thrashing
,
or
flapping
. It's similar to the concept of
hysteresis
in cybernetics.
Autoscaling during rolling update
Kubernetes lets you perform a rolling update on a Deployment. In

 that
case, the Deployment manages the underlying ReplicaSets for you.
When you configure autoscaling for a Deployment, you bind a
HorizontalPodAutoscaler to a single Deployment. The HorizontalPodAutoscaler
manages the
replicas
field of the Deployment. The deployment controller is responsible
for setting the
replicas
of the underlying ReplicaSets so that they add up to a suitable
number during the rollout and also afterwards.
If you perform a rolling update of a StatefulSet that has an autoscaled number of
replicas, the StatefulSet directly manages its set of Pods (there is no intermediate resource
similar to ReplicaSet).
Support for resource metrics
Any HPA target can be scaled based on the resource usage of the pods in the scaling target.
When defining the pod specification the resource requests like
cpu
and
memory
should
be specified. This is used to determine the resource utilization and used by the HPA controller
to scale the target up or down. To use resource utilization based scaling specify a metric source
like this:
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
60
With this metric the HPA controller will keep the average utilization of the pods in the scaling
target at 60%. Utilization is the ratio between the current usage of resource to the requested
resources of the pod. See
Algorithm
for more details about how the utilization
is calculated and averaged.
Note:
Since the resource usages of all the containers are summed up the total pod utilization may not
accurately represent the individual container resource usage. This could lead to situations where
a single container might be running with high usage and the HPA will not scale out because the overall
pod usage is still within acceptable limits.
Container resource metrics
Kubernetes v1.20 [alpha]
The HorizontalPodAutoscaler API also supports a container metric source where the HPA can track the
resource usage of individual containers across a set of Pods, in order to scale the target resource.
This lets you configure scaling thresholds for the containers that matter most in a particular Pod.
For example, if you have a web application and a logging sidecar, you can scale based on the resource
use of the web application, ignoring the sidecar container and its resource use.
If you revise the target resource to have a new Pod specification with a different set of containers,
you should revise the HPA spec if that newly added container should also be used for
scaling. If the specified container in the metric source is not present or only present in a subset
of the pods then those pods are ignored and the recommendation is recalculated. See
Algorithm
for more details about the calculation. To use container resources for autoscaling define a metric
source as follows:
type
:
ContainerResource
containerResource
:
name
:
cpu
container
:
target
:
type
:
:
60
In the above example the HPA controller scales the target such that the average utilization of the cpu
in the
container of all the pods is 60%.
Note:
If you change the name of a container that a HorizontalPodAutoscaler is tracking, you can
make that change in a specific order to ensure scaling remains available and effective
whilst the change is being applied. Before you update the resource that defines the container
(such as a Deployment), you should update the associated HPA to track both the new and
old container names. This way, the HPA is able to calculate a scaling recommendation
throughout the update process.
Once you have rolled out the container name change to the workload resource, tidy up by removing
the old container name from the HPA specification.
Scaling on custom metrics
(the
API version previously provided this ability as a beta feature)
Provided that you use the
API version, you can configure a HorizontalPodAutoscaler
to

 scale based on a custom metric (that is not built in to Kubernetes or any Kubernetes component).
The HorizontalPodAutoscaler controller then queries for these custom metrics from the Kubernetes
API.
See
for the requirements.
Scaling on multiple metrics
(the
API version, you can specify multiple metrics for a
HorizontalPodAutoscaler to scale on. Then, the HorizontalPodAutoscaler controller evaluates each metric,
and proposes a new scale based on that metric. The HorizontalPodAutoscaler takes the maximum scale
recommended for each metric and sets the workload to that size (provided that this isn't larger than the
overall maximum that you configured).
By default, the HorizontalPodAutoscaler controller retrieves metrics from a series of APIs. In order for it to access these
APIs, cluster administrators must ensure that:
The
The corresponding APIs are registered:
For resource metrics, this is the
API, generally provided by
.
It can be launched as a cluster add-on.
For custom metrics, this is the
API. It's provided by "adapter" API servers provided by metrics solution vendors.
Check with your metrics pipeline to see if there is a Kubernetes metrics adapter available.
For external metrics, this is the
API. It may be provided by the custom metrics adapters provided above.
For more information on these different metrics paths and how they differ please see the relevant design proposals for
the HPA V2
,
and
.
For examples of how to use them see
the walkthrough for using custom metrics
and
the walkthrough for using external metrics
.
Configurable scaling behavior
(the
If you use the
v2
HorizontalPodAutoscaler API, you can use the
behavior
field
(see the
)
to configure separate scale-up and scale-down behaviors.
You specify these behaviours by setting
scaleUp
and / or
scaleDown
under the
behavior
field.
You can specify a
stabilization window
that prevents
flapping
the replica count for a scaling target. Scaling policies also let you controls the
rate of change of replicas while scaling.
Scaling policies
One or more scaling policies can be specified in the
behavior
section of the spec.
When multiple policies are specified the policy which allows the highest amount of
change is the policy which is selected by default. The following example shows this behavior
while scaling down:
behavior
:
scaleDown
:
policies
:
-
type
:
Pods
value
:
4
:
60
-
type
:
Percent
value
:
10
:
60
indicates the length of time in the past for which the policy must hold true.
The first policy
(Pods)
allows at most 4 replicas to be scaled down in one minute. The second policy
(Percent)
allows at most 10% of the current replicas to be scaled down in one minute.
Since by default the policy which allows the highest amount of change is selected, the second policy will
only be used when the number of pod replicas is more than 40. With 40 or less replicas, the first policy will be applied.
For instance if there are 80 replicas and the target has to be scaled down to 10 replicas
then during the first step 8 replicas will be reduced. In the next iteration when the number
of replicas is 72, 10% of the pods is 7.2 but the number is rounded up to 8. On each loop of
the autoscaler controller the number of pods to be change is re-calculated based on the number
of current replicas. When the number of replicas falls below 40 the first policy
(Pods)
is applied
and 4 replicas will be reduced at a time.
The policy selection can be changed by 

specifying the
selectPolicy
field for a scaling
direction. By setting the value to
Min
which would select the policy which allows the
smallest change in the replica count. Setting the value to
Disabled
completely disables
scaling in that direction.
Stabilization window
The stabilization window is used to restrict the
flapping
of
replica count when the metrics used for scaling keep fluctuating. The autoscaling algorithm
uses this window to infer a previous desired state and avoid unwanted changes to workload
scale.
For example, in the following example snippet, a stabilization window is specified for
scaleDown
.
behavior
:
scaleDown
:
stabilizationWindowSeconds
:
300
When the metrics indicate that the target should be scaled down the algorithm looks
into previously computed desired states, and uses the highest value from the specified
interval. In the above example, all desired states from the past 5 minutes will be considered.
This approximates a rolling maximum, and avoids having the scaling algorithm frequently
remove Pods only to trigger recreating an equivalent Pod just moments later.
Default Behavior
To use the custom scaling not all fields have to be specified. Only values which need to be
customized can be specified. These custom values are merged with default values. The default values
match the existing behavior in the HPA algorithm.
behavior
:
scaleDown
:
:
300
policies
:
-
type
:
Percent
value
:
100
:
15
scaleUp
:
:
0
policies
:
-
type
:
Percent
value
:
100
:
15
-
type
:
Pods
value
:
4
:
15
:
Max
For scaling down the stabilization window is
300
seconds (or the value of the
flag if provided). There is only a single policy
for scaling down which allows a 100% of the currently running replicas to be removed which
means the scaling target can be scaled down to the minimum allowed replicas.
For scaling up there is no stabilization window. When the metrics indicate that the target should be
scaled up the target is scaled up immediately. There are 2 policies where 4 pods or a 100% of the currently
running replicas will be added every 15 seconds till the HPA reaches its steady state.
Example: change downscale stabilization window
To provide a custom downscale stabilization window of 1 minute, the following
behavior would be added to the HPA:
behavior
:
scaleDown
:
:
60
Example: limit scale down rate
To limit the rate at which pods are removed by the HPA to 10% per minute, the
following behavior would be added to the HPA:
behavior
:
scaleDown
:
policies
:
-
type
:
Percent
value
:
10
:
60
To ensure that no more than 5 Pods are removed per minute, you can add a second scale-down
policy with a fixed size of 5, and set
to minimum. Setting
to
Min
means
that the autoscaler chooses the policy that affects the smallest number of Pods:
behavior
:
scaleDown
:
policies
:
-
type
:
Percent
value
:
10
:
60
-
type
:
Pods
value
:
5
:
60
:
Min
Example: disable scale down
The
value of
Disabled
turns off scaling the given direction.
So to prevent downscaling the following policy would be used:
behavior
:
scaleDown
:
:
Disabled
Support for HorizontalPodAutoscaler in kubectl
HorizontalPodAutoscaler, like every API resource, is supported in a standard way by
kubectl
.
You can create a new autoscaler using
command.
You can list autoscalers by
or get detailed description by
.
Finally, you can delete an autoscaler using
kubectl delete hpa
.
In addition, there is a special
command for creating a HorizontalPodAutoscaler object.
For instance, executing
kubectl autoscale rs foo --min=2 --max=5 --cpu-percent=80
will create an autoscaler for ReplicaSet
foo
, with target CPU utilization set to
80%

and the number of replicas between 2 and 5.
Implicit maintenance-mode deactivation
You can implicitly deactivate the HPA for a target without the
need to change the HPA configuration itself. If the target's desired replica count
is set to 0, and the HPA's minimum replica count is greater than 0, the HPA
stops adjusting the target (and sets the
Condition on itself
to
false
) until you reactivate it by manually adjusting the target's desired
replica count or HPA's minimum replica count.
Migrating Deployments and StatefulSets to horizontal autoscaling
When an HPA is enabled, it is recommended that the value of
of
the Deployment and / or StatefulSet be removed from their
manifest(s)
. If this isn't done, any time
a change to that object is applied, for example via
kubectl apply -f deployment.yaml
, this will instruct Kubernetes to scale the current number of Pods
to the value of the
key. This may not be
desired and could be troublesome when an HPA is active.
Keep in mind that the removal of
may incur a one-time
degradation of Pod counts as the default value of this key is 1 (reference
Deployment Replicas
).
Upon the update, all Pods except 1 will begin their termination procedures. Any
deployment application afterwards will behave as normal and respect a rolling
update configuration as desired. You can avoid this degradation by choosing one of the following two
methods based on how you are modifying your deployments:
Client Side Apply (this is the default)
kubectl apply edit-last-applied deployment/&lt;deployment_name&gt;
In the editor, remove
. When you save and exit the editor,
kubectl
applies the update. No changes to Pod counts happen at this step.
You can now remove
from the manifest. If you use source code management,
also commit your changes or take whatever other steps for revising the source code
are appropriate for how you track updates.
From here on out you can run
you can follow the
transferring ownership
guidelines, which cover this exact use case.
If you configure autoscaling in your cluster, you may also want to consider running a
cluster-level autoscaler such as
Cluster Autoscaler
.
For more information on HorizontalPodAutoscaler:
Read a
for horizontal pod autoscaling.
Read documentation for
.
If you would like to write your own custom metrics adapter, check out the
boilerplate
to get started.
Read the
for HorizontalPodAutoscaler.
Feedback
Yes
No
.
or
.
Last modified November 26, 2022 at 1:33 PM PST:
Replace an image with mermaid (b319f844db)
API Object
Home
Blog
Training
Partners
Community
CC BY 4.0
ICP

Externalizing config using MicroProfile, ConfigMaps and Secrets | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggregation Laye

r
Tasks
Debug Pods
Determine th

e Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

ation (v1)
Tutorials
here
.
here
.
combined into a single set of properties t

hat the application can access via an API. Together,
Feedback
Yes
No
.
or
.
Last modified June 18, 2022 at 5:15 PM PST:
Batch fix links (4) (2f4a067fea)
Home
Blog
Training
Partners
Community
CC BY 4.0

Managing Secrets using Configuration File | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Oper

ator pattern
Tasks
Debug Pods
Determine the Reas

on for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Cl

uster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

(v1)
Tasks
minikube
You can define the
Secret
object in a manifest first, in JSON or YAML format,
and then create that object. The
Secret
resource contains two maps:
data
and
.
The
data
field is used to store arbitrary data, encoded using base64. The
field is provided for convenience, and it allows you to provide
the same data as unencoded strings.
The keys of
data
and
must consist of alphanumeric characters,
-
,
_
or
.
.
The following example stores two strings in a Secret using the
data
field.
Convert the strings to base64:
echo
-n
'admin'
| base64
echo
-n
'1f2d1e2e67df'
| base64
Note:
The serialized JSON and YAML values of Secret data are encoded as base64 strings. Newlines are not valid within these strings and must be omitted. When using the
base64
utility on Darwin/macOS, users should avoid using the
-b
option to split long lines. Conversely, Linux users
should
add the option
-w 0
to
base64
commands or the pipeline
base64 | tr -d '\n'
if the
-w
option is not available.
YWRtaW4=
MWYyZDFlMmU2N2Rm
Create the manifest:
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
password
:
Note that the name of a Secret object must be a valid
.
Create the Secret using
:
kubectl apply -f ./secret.yaml
secret/mysecret created
To verify that the Secret was created and to decode the Secret data, refer to
.
Specify unencoded data when creating a Secret
For certain scenarios, you may wish to use the
field instead. This
field allows you to put a non-base64 encoded string directly into the Secret,
and the string will be encoded for you when the Secret is created or updated.
A practical example of this might be where you are deploying an application
that uses a Secret to store a configuration file, and y

ou want to populate
parts of that configuration file during your deployment process.
For example, if your application uses the following configuration file:
apiUrl
:
"https://my.api.com/api/v1"
username
:
"&lt;user&gt;"
password
:
You could store this in a Secret using the following definition:
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
:
:
|
apiUrl: "https://my.api.com/api/v1"
username: &lt;user&gt;
password: &lt;password&gt;
When you retrieve the Secret data, the command returns the encoded values,
and not the plaintext values you provided in
.
For example, if you run the following command:
kubectl get secret mysecret -o yaml
:
v1
data
:
:
YXBpVXJsOiAiaHR0cHM6Ly9teS5hcGkuY29tL2FwaS92MSIKdXNlcm5hbWU6IHt7dXNlcm5hbWV9fQpwYXNzd29yZDoge3twYXNzd29yZH19
kind
:
Secret
metadata
:
:
2018-11-15T20:40:59Z
name
:
mysecret
namespace
:
default
:
"7225"
uid
:
c280ad2e-e916-11e8-98f2-025000000001
type
:
Opaque
Specify both
data
and
If you specify a field in both
data
and
, the value from
is used.
For example, if you define the following Secret:
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
:
username
:
administrator
The
Secret
object is created as follows:
:
v1
data
:
username
:
YWRtaW5pc3RyYXRvcg==
kind
:
Secret
metadata
:
:
2018-11-15T20:46:46Z
name
:
mysecret
namespace
:
default
:
"7579"
uid
:
91460ecb-e917-11e8-98f2-025000000001
type
:
Opaque
decodes to
.
Edit a Secret
To edit the data in the Secret you created using a manifest, modify the
data
or
field in your manifest and apply the file to your
cluster. You can edit an existing
Secret
object unless it is
immutable
.
For example, if you want to change the password from the previous example to
birdsarentreal
, do the following:
Encode the new password string:
echo
-n
'birdsarentreal'
| base64
YmlyZHNhcmVudHJlYWw=
Update the
data
field with your new password string:
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
password
:
Apply the manifest to your cluster:
secret/mysecret configured
Kubernetes updates the existing
Secret
object. In detail, the
kubectl
tool
notices that there is an existing
Secret
object with the same name.
kubectl
fetches the existing object, plans changes to it, and submits the changed
Secret
object to your cluster control plane.
If you specified
instead,
kubectl
uses
instead.
Clean up
To delete the Secret you have created:
kubectl delete secret mysecret
Read more about the
Secret concept
manage Secrets using kubectl
manage Secrets using kustomize
Feedback
Yes
No
.
or
.
Last modified November 13, 2022 at 9:10 AM PST:
Tweak tasks/configmap-secret (37864abbb4)
data
and
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Upgrade A Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

v1)
Tasks
kubectl
kubeadm
kubeadm
kubectl
.
Caution:
plugins.
kubectl
.
drain
You can use
The
kubectl

pod.yaml
kind
to
.
Refer to
and
Feedback
Yes
No
.
or
.
Last modified December 12, 2022 at 11:29 PM PST:
[en]modify the invalid link (29194d9435)
kubeadm
Home
Blog
Training
Partners
Community
CC BY 4.0

Manage Cluster Daemons | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Perform common tasks for managing a DaemonSet, such as performing a rolling update.
1:
2:
1 - Perform a Rolling Update on a DaemonSet
minikube
OnDelete
: With
OnDelete
only
before.
With
to
.
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
spec
:
selector
:
:
name
:
:
type
:
:
:
1
template
:
metadata
:
labels
:
name
:
spec
:
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
-
name
:
image
:
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
mountPath
:
readOnly
:
true
:
30
volumes
:
-
name
:
varlog
hostPath
:
path
:
/var/log
-
name
:
hostPath
:
path
:
.
:
=
If you haven't created the DaemonSet in the system, check your Dae

monSet
=
=
DaemonSet
kubectl
commands.
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
spec
:
selector
:
:
name
:
:
type
:
:
:
1
template
:
metadata
:
labels
:
name
:
spec
:
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
-
name
:
image
:
resources
:
limits
:
memory
:
200Mi
requests
:
cpu
:
100m
memory
:
200Mi
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
mountPath
:
readOnly
:
true
:
30
volumes
:
-
name
:
varlog
hostPath
:
path
:
/var/log
-
name
:
hostPath
:
path
:
,
use
:
,
use
:
, use
:
kubectl
set
=
daemonset
causes:
.
by comparing the output of
name
=
Note:
either.
Clock skew
If
progress.
Clean up
Delete 

DaemonSet from a namespace :
See
See
2 - Perform a Rollback on a DaemonSet
This page shows how to perform a rollback on a
DaemonSet
.
minikube
Your Kubernetes server must be at or later than version 1.7.
.
You should already know how to
perform a rolling update on a
DaemonSet
.
Step 1: Find the DaemonSet revision you want to roll back to
You can skip this step if you only want to roll back to the last revision.
List all revisions of a DaemonSet:
history
daemonset &lt;daemonset-name&gt;
This returns a list of DaemonSet revisions:
daemonsets "&lt;daemonset-name&gt;"
REVISION        CHANGE-CAUSE
1               ...
2               ...
...
Change cause is copied from DaemonSet annotation
to its revisions upon creation. You may specify
--record=true
in
kubectl
to record the command executed in the change cause annotation.
To see the details of a specific revision:
history
daemonset &lt;daemonset-name&gt; --revision
=
1
This returns the details of that revision:
daemonsets "&lt;daemonset-name&gt;" with revision #1
Labels:       foo=bar
app:
 Image:        ...
 Port:         ...
 Environment:  ...
 Mounts:       ...
Volumes:      ...
Step 2: Roll back to a specific revision
# Specify the revision number you get from Step 1 in --to-revision
kubectl rollout undo daemonset &lt;daemonset-name&gt; --to-revision
=
&lt;revision&gt;
If it succeeds, the command returns:
daemonset "&lt;daemonset-name&gt;" rolled back
Note:
If
--to-revision
flag is not specified, kubectl picks the most recent revision.
Step 3: Watch the progress of the DaemonSet rollback
kubectl rollout undo daemonset
tells the server to start rolling back the
DaemonSet. The real rollback is done asynchronously inside the cluster
.
To watch the progress of the rollback:
kubectl rollout status ds/&lt;daemonset-name&gt;
When the rollback is complete, the output is similar to:
daemonset "&lt;daemonset-name&gt;" successfully rolled out
Understanding DaemonSet revisions
In the previous
step, you got a list of DaemonSet
revisions. Each revision is stored in a resource named ControllerRevision.
To see what is stored in each revision, find the DaemonSet revision raw
kubectl get controllerrevision -l &lt;daemonset-selector-key&gt;
=
&lt;daemonset-selector-value&gt;
This returns a list of ControllerRevisions:
NAME                               CONTROLLER                     REVISION   AGE
&lt;daemonset-name&gt;-&lt;revision-hash&gt;   DaemonSet/&lt;daemonset-name&gt;     1          1h
&lt;daemonset-name&gt;-&lt;revision-hash&gt;   DaemonSet/&lt;daemonset-name&gt;     2          1h
Each ControllerRevision stores the annotations and template of a DaemonSet
revision.
takes a specific ControllerRevision and replaces
DaemonSet template with the template stored in the ControllerRevision.
is equivalent to updating DaemonSet template to a
previous revision through other commands, such as
or
.
Note:
DaemonSet revisions only roll forward. That is to say, after a
rollback completes, the revision number (
.revision
field) of the
ControllerRevision being rolled back to will advance. For example, if you
have revision 1 and 2 in the system, and roll back from revi

sion 2 to revision
1, the ControllerRevision with
.revision: 1
will become
.revision: 3
.
See
troubleshooting DaemonSet rolling
update
.
Home
Blog
Training
Partners
Community
CC BY 4.0

ListMeta | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Set Up kubectl on Windows
Debug Pods
Troubleshooting Cluste

rs
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Resources with AppAr

mor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-proxy Configuration (v

1alpha1)
Reference
ListMeta
ListMeta
continue
(string)
(int64)
(string)
selfLink
(string)
Feedback
Yes
No
.
Open an issue in the GitHub repo if you 

want to
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Deploy an App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Netw

Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

 Aggregation Layer
Tasks
Debug Pods
Debug a Statefu

lSet
Auditing
Run Jobs
TLS
Tutorials
Security

ity Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-a

piserver Configuration (v1)
Tasks
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Insta

lpha1)
Reference
Kubeadm
Kubeadm
and
.
or
Feedback
Yes
No
.
or
.
Last modified September 08, 2021 at 10:35 AM PST:
Change reference to kubeadm-stacked-color.png (826fb8dc90)
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-apiserver Audit Configuration (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Event
EventList
Policy
Event
EventList
Field
string
kind
string
Event
level
Level
auditID
stage
Stage
string
verb
string
user
sourceIPs
[]string
userAgent
string
objectRef
to the external type, and serialized as JSON. Omitted for non-

resource requests. Only logged
EventList
Field
string
kind
string
EventList
metadata
items
[]Event
Policy
Field
string
kind
string
Policy
metadata
metadata
field.
rules
[]Stage
bool
Field
string
kind
string
metadata
items
[]Policy
Field
group
string
resources
[]string
'
'pods/
[]string
Level
(Alias of
string
)
Event
Event
Field
resource
string
namespace
string
name
string
uid
apiGroup
string
The

 empty string represents the core API group.
string
string
string
Policy
Field
level
Level
users
[]string
[]string
verbs
[]string
resources
[]string
[]string
Examples:
"/healthz
[]Stage
bool
Stage
(Alias of
string
)
Event
Policy
Feedback
Yes
No
.
or
.
Event
EventList
Policy
Level
Stage
Home
Blog
Training
Partners
Community
CC BY 4.0

JSONPath Support | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

alpha1)
Reference
Use the
range
,
end
.
Note:
The
$
{
"kind"
:
"List"
,
"items"
:[
{
"kind"
:
"None"
,
:{
"name"
:
},
"status"
:{
:{
"cpu"
:
"4"
},
:[{
"type"
:
,
"address"
:
}]
}
},
{
"kind"
:
"None"
,
:{
"name"
:
},
"status"
:{
:{
"cpu"
:
"8"
},
:[
{
"type"
:
,
"address"
:
},
{
"type"
:
"another"
,
"address"
:
}
]
}
}
],
"users"
:[
{
"name"
:
"myself"
,
"user"
: {}
},
{
"name"
:
"e2e"
,
"user"
: {
:
"admin"
,
:
"secret"
}
}
]
}
Function
Example
Result
text
@
{@}
.
or
[]
{.kind}
,
or
List
..
{..name}
*
myself
[,]
?()
filter
secret
range
,
end
''
kubectl
=
jsonpath
=
'{@}'
=
jsonpath
=
=
jsonpath
=
=
jsonpath
=
=
jsonpath
=
'{range .items[*]}{.metadata.n

ame}{"\t"}{.status.startTime}{"\n"}{end}'
Note:
double
\t\
\n\
"}{end}"
Note:
jq
.
jsonpath
=
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

PersistentVolume | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
: v1
kind
metadata
(
)
spec
(
)
status
(
)
capacity
Quantity
)
claimRef
(
)

([]
)
([]
)
(string)
(string)
storageClassName is the name of StorageClass to which this persistent volume belongs. Empty value means that this volume does not belong to any StorageClass.
(string)
Local
hostPath
(string)
local
(string)
(string)
(int32)
part

ition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty).
(boolean)
azureDisk
(string)
(string)
(string)
(boolean)
azureFile
(boolean)
(string)
cephfs
(string)
(boolean)
(string)
(string)
(string)
(string)
cinder
cinder represents a cinder volume attached and mounted on kubelets host machine. More 

info:
(string)
(boolean)
(string)
(string)
csi
(string)
(string)
(string)
(string)
(string)
SecretReference represents a Secret Reference. It has enough information to retrieve secret in any names

pace
(string)
(string)
(string)
(string)
nodeStageSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeStageVolume and NodeStageVolume and NodeUnstageVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
(string)
(string)
(boolean)
fc
fc.fsType
(string)
fc.lun
(int32)
(boolean)
fc.wwids
(string)
(boolean)
SecretReference represents a Secret Reference. It has enoug

h information to retrieve secret in any namespace
(string)
(string)
flocker
(string)
(string)
gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin. More info:
(string)
(int32)
(boolean)
iscsi
iscsi.iqn
iscsi.lun
(boolean)
(boolean)
(string)
(string)
initiatorName is the custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterf

ace simultaneously, new iSCSI interface &lt;target portal&gt;:&lt;volume name&gt; will be created for the connection.
(string)
(boolean)
(string)
(string)
nfs
nfs.path
(boolean)
(string)
(string)
(boolean)
quobyte
(string)
(boolean)
(string)
(string)
rbd
rbd represents a Rados Block Device 

mount on the host that shares a pod's lifetime. More info:
rbd.image
(string)
(string)
rbd.pool
(string)
(boolean)
(string)
(string)
rbd.user
(string)
scaleIO
(string)
(string)
(string)
(string)
(boolean)
(boolean)
(string)
(string)
(string)
storageos
https://examples

.k8s.io/volumes/storageos/README.md
(string)
(boolean)
(
)
(string)
(string)
(string)
(string)
(string)
message
(string)
phase
(string)
reason
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch

(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
S

tack Overflow
.
or
.
Local
get
get
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

PriorityLevelConfiguration v1beta3 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extendin

)
Reference
kind
metadata
(
)
metadata
spec
(
)
spec
status
(
)
status
type
type
"Exempt"
means that requests of this priority level are not subject to a limit (and thus are never queued) and do not detract from the capacity made available to other priority levels. A value of
"Limited"
are
limited
limited
type
is
"Limited"
.
(int32)
, if present, configures a limit on how many seats this priority level can borrow from other priority levels. The limit is known as this level's BorrowingConcurrencyLimit (BorrowingCL) and i

s a limit on the total number of seats that this level may borrow at any one time. This field holds the ratio of that limit to the level's nominal concurrency limit. When this field is non-nil, it must hold a non-negative integer and the limit is calculated as follows.
nil
(int32)
type
queuing
type
is
"Queue"
.
(int32)
handSize
handSize
queues
(int32)
(int32)
queues
is the number of queues for this priority level. The queues exist independently at each apiserver. The value must be positive. Setting it to 1 effectively precludes shufflesharding and thus makes the distinguisher method of associated flow schemas irrelevant. This field has a default value of 64.
(int32)
([]PriorityLevelConfigurati

onCondition)
type
(Time)
(string)
message
(string)
reason
(string)
status
(string)
type
kind
metadata
(
ListMeta
)
metadata
items
([]
items
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurat

ions/{name}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
create
update
update
patch
partially update

 the specified PriorityLevelConfiguration
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Operating etcd clusters for Kubernetes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

ha1)
Tasks
back up
plan
.
minikube
.
stable.
.
The minimum recommended version of etcd to run in production is
3.2.10+
.
.
=
http://
:2379
\
=
http://
:2379
.
Make sure
production and back it up periodically. A five-m

ember cluster is recommended
.
.
,
,
,
, and
=
http://
$IP1
$IP2
$IP3
$IP4
$IP5
=
http://
$IP1
$IP2
$IP3
$IP4
$IP5
:2379
.
$LB
.
.
peer.key
and
peer.cert
and
and
and
and
that are
etcd.ca
flag. Specifying flags
and
.
flags
,
and
.
Note:
.
failed member involves two steps: removing the failed member and adding a 

new
member.
,
, and
. When
member1
fails,
.
member1
:
=
flag, then
etcd.
dir.
=
10.0.0.4
:
export
ETCD_NAME
=
"member4"
export
=
export
=
existing
etcd
[
flags
]
flag, then
.
command or by copying the
$ENDPOINT
to the file
:
=
3
$ENDPOINT
=
3
=
volume.
Snapshot usin

g etcdctl options
=
3
etcdctl -h
=
3
=
\
--cacert
=
=
=
\
where
,
cert-file
and
key-file
the
.
=
3
=
3
export
=
3
.
. Replace
and
instead.
Note:
restore instances of etcd. Instead, follow these steps to restore etcd:
stop
all
,
,
kubelet
Note:
Feedback
Yes
No
.
or

.
Last modified August 31, 2022 at 2:12 AM PST:
Update to etcdctl restore command syntax (99bd0a8585)
Home
Blog
Training
Partners
Community
CC BY 4.0

Running Automated Tasks with a CronJob | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Run Jobs
CronJob
to run
Jobs
Cron
CronJobs
.
minikube
:
batch/v1
kind
:
CronJob
metadata
:
name
:
hello
spec
:
schedule
:
"* * * * *"
:
spec
:
template
:
spec
:
:
-
name
:
hello
image
:
:
command
:
- /bin/sh
- -c
:
OnFailure
jobs
--watch
You can stop watching the job and view the cron job again to see that it scheduled the jo

b:
hello
Note:
pods
=
$(
=
job-name
=
=
jsonpath
={
.items
[
*
]
}
)
Show the pod log:
$pods
:
.
,
kind
, and
metadata
fields.
manifests
, see the
,
and
.spec
section.
Note:
not
Schedule
The
.spec
.
It takes a
Cron
0 * * * *
or
@hourly
,
:
with
0-23/2
*/2
.
Note:
?
*
, that is,
The
Job
or
kind
.
.spec
, see
.
The
If the
200
The
Allow
Forbid
pr

evious job run hasn't finished yet, the cron job skips the new job run
Replace
Suspend
The
true
Caution:
When
true
to
false
The
and
0
Feedback
Yes
No
.
or
.
Last modified September 01, 2022 at 12:17 AM PST:
Remove CronJob info about graduating to stable (c6afc7a5e5)
Schedule
Suspend
Home
Blog
Training
Partners
Community
CC BY 4.0

Component tools | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
kubelet
4:
5:
6:
7:
1 - Feature Gates
This page contains an overview of the various feature gates an administrator
See
Overview
Use
-h
=
=
true
in the
.
.
The
Note:
.
Feature
Default
Stage
Since
Until
false
Alpha
1.8
1.8
true
Beta
1.9
false
Alpha
1.18
1.19
true
Beta
1.20
false
Alpha
1.7
1.15
true
Beta
1.16
false
Alpha
1.26
false
Alpha
1.20
1.25
true
Beta
1.26
false
Alpha
1.22
false
Alpha
1.26
false
Alpha
1.18
1.23
true
Beta
1.24
AppArmor
true
Beta
1.4
false
Alpha
1.23
true
Beta
1.23
false
Alpha
1.22
1.22
true
Beta
1.23
false
Alpha
1.23
1.24
false
Beta
1.25
false
Alpha
1.23
false
Alpha
1.25
false
Alpha
1.21
false
Alpha
1.26
false
Alpha
1.25
false
Alpha
1.24
false
Alpha
1.24
1.24
true
Beta
1.25
false
Alpha
1.26
false
Alpha
1.12
false
Alpha
1.23
1.24
true
Beta
1.25
false
Alpha
1.22
false
Alpha
1.23
false
Alpha
1.20
1.20
false
Beta
1.21
1.21
true
Beta
1.22
false
Alpha
1.26
false
Alpha
1.26
-
false
Alpha
1.22
1.25
true
Beta
1.26
false
Beta
1.5
false
Alpha
1.23
1.23
true
Beta
1.24
false
Alpha
1.20
1.20
tru

e
Beta
1.21
false
Alpha
1.23
1.23
true
Beta
1.24
false
Alpha
1.20
false
Alpha
1.16
false
Alpha
1.23
false
Alpha
1.25
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.23
false
Alpha
1.23
false
Alpha
1.21
true
Beta
1.23
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.23
1.23
true
Beta
1.24
KMSv2
false
Alpha
1.25
false
Alpha
1.22
false
Alpha
1.13
1.14
true
Beta
1.15
false
Alpha
1.21
1.22
true
Beta
1.23
false
Alpha
1.25
false
Alpha
1.25
false
Alpha
1.15
-
false
Alpha
1.21
1.21
true
Beta
1.22
false
Alpha
1.24
-
true
Beta
1.24
-
false
Alpha
1.25
false
Alpha
1.24
false
Alpha
1.21
1.21
true
Beta
1.22
MemoryQoS
false
Alpha
1.22
false
Alpha
1.24
1.24
false
Beta
1.25
false
Alpha
1.26
-
false
Alpha
1.25
false
Alpha
1.24
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.24
1.25
true
Beta
1.26
NodeSwap
false
Alpha
1.22
false
Alpha
1.23
1.23
true
Beta
1.24
OpenAPIV3
false
Alpha
1.23
1.23
OpenAPIV3
true
Beta
1.24
false
Alpha
1.26
false
Alpha
1.23
false
Alpha
1.21
1.21
true
Beta
1.22
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.25
false
Alpha
1.26
false
Alpha
1.21
1.21
false
Beta
1.22
1.24
true
Beta
1.25
false
Alpha
1.12
false
Alpha
1.22
1.25
true
Beta
1.26
false
Alpha
1.11
false
Alpha
1.22
false
Alpha
1.23
false
Alpha
1.15
1.15
true
Beta
1.16
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.7
1.11
true
Beta
1.12
false
Alpha
1.25
false
Alpha
1.22
1.24
true
Beta
1.25
false
Alpha
1.23
1.24
true
Beta
1.25
false
Alpha
1.20
1.21
true
Beta
1.22
false
Alpha
1.22
false
Alpha
1.26
false
Alpha
1.20
false
Alpha
1.14
1.14
true
Beta
1.15
false
Alpha
1.21
1.22
false
Beta
1.23
1.23
true
Beta
1.24
false
Alpha
1.16
1.17
true
Beta
1.18
false
Alpha
1.26
false
Beta
1.26
false
Alpha
1.26
false
Alpha
1.25
ValidatingAdmissionPol

icy
false
Alpha
1.26
false
Alpha
1.21
-
WinDSR
false
Alpha
1.14
false
Alpha
1.14
1.19
true
Beta
1.20
false
Alpha
1.26
Feature
Default
Stage
Since
Until
false
Alpha
1.7
1.7
true
Beta
1.8
1.11
true
GA
1.12
-
false
Alpha
1.8
1.9
true
Beta
1.10
1.25
true
GA
1.26
-
false
Alpha
1.15
1.15
true
Beta
1.16
1.24
true
GA
1.25
-
false
Alpha
1.14
1.16
true
Beta
1.17
1.24
true
GA
1.25
-
false
Alpha
1.14
1.16
false
Beta
1.17
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
false
Alpha
1.15
1.18
false
Beta
1.19
1.22
true
Beta
1.23
1.23
true
GA
1.24
false
Alpha
1.15
1.20
false
Beta
1.21
1.23
true
Beta
1.24
1.25
true
GA
1.26
false
Alpha
1.14
1.16
false
Beta
1.17
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
false
Alpha
1.18
1.18
false
Beta
1.19
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
false
Alpha
1.19
1.20
true
Beta
1.21
1.23
true
GA
1.24
-
true
GA
1.25
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.24
true
GA
1.25
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.25
true
GA
1.26
-
false
Alpha
1.8
1.9
true
Beta
1.10
1.25
true
GA
1.26
-
false
Alpha
1.19
1.19
true
Beta
1.20
1.24
true
GA
1.25
-
DryRun
false
Alpha
1.12
1.12
DryRun
true
Beta
1.13
1.18
DryRun
true
GA
1.19
-
false
Alpha
1.20
1.20
true
Beta
1.21
1.23
true
GA
1.24
-
false
Alpha
1.20
1.21
true
Beta
1.22
1.25
true
GA
1.26
false
Alpha
1.16
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
true
GA
1.20
-
false
Alpha
1.14
1.15
true
Beta
1.16
1.23
true
GA
1.24
-
false
Alpha
1.11
1.14
true
Beta
1.15
1.23
true
GA
1.24
-
false
Alpha
1.8
1.10
true
Beta
1.11
1.23
true
GA
1.24
-
false
Alpha
1.23
1.23
true
Beta
1.24
1.24
true
GA
1.25
-
false
Alpha
1.22
1.22
false
Beta
1.23
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
false
Alpha
1.20
1.23
true
Beta
1.24
1.25
true
GA
1.26
-
true
Beta
1.24
1.25
true
GA
1.26
-
false
Alpha
1.7
1.9
true
Beta
1.10
1.24
true
GA
1.25
-
false
Alpha
1.20
1.23
true
Beta
1.24
1.25
true
GA
1.26

-
false
Alpha
1.21
1.21
true
Beta
1.22
1.24
true
GA
1.25
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.24
true
GA
1.25
false
Alpha
1.16
1.19
true
Beta
1.20
1.23
true
GA
1.24
-
false
Alpha
1.14
1.15
true
Beta
1.16
1.21
true
GA
1.22
-
false
Alpha
1.24
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.25
true
GA
1.26
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.25
true
GA
1.26
-
Alpha
,
Beta
or
GA
stage.
An
Alpha
A
Beta
Note:
Beta
A
stable
LIST
or
GET
)
)
LIST
or
GET
requests.
See
See
: Enable
which
of a
PVC
.
AppArmor
See
for mor

e details.
API.
See
and
.
a group
a group
enabled.
enabled.

during a
.
Check the
csi
: Enable support for CSI volume health monitoring on node.
and
.
timeZone
CronJobs
field of a
in
.
the
See
.
fsGroup
passing
fsGroup
:
.
,
and
kubelet
.
DryRun
dry run
requests
and
serving
via
CRI
.
This feature gate exists in case any of your existing workloads depend 

on a
.
.
.
.
MKNODE
,
See
.
Delete
to 0 for
are
windows
and
linux
.
Job
.
Ready
condition
.
Ready
status
of a
Job
status.
Job
KMSv2
: Enable kubelet exe

c credential providers for
.
See
.
: Add support for distributed tracing in the kubelet.
See
.
.
sizeLimit
.
: When
field for
.
MemoryQoS
: Enable
in
.
type
endPort
status
and
in
NodeSwap
OpenAPIV3
field of a
when unhealthy pods should be considered for evictio

n. Please see
.
: Enable
procMount
.
See
.
: Sets the
See
volume.
or
emptyDir
volumes).
: Allows
StatefulSetSt

artOrdinal
.
Hints
for more
details.
.
a group
a group
: Enable
WinDSR
The
, set
.
See
2 - Feature Gates (removed)
the
or the
Feature
Default
Stage
From
To
false
Alpha
1.6
1.10
-
1.11
1.11
false
Alpha
1.6
1.7
-
1.8
1.8
false
Beta
1.4
1.6
true
GA
1.7
1.9
true
Beta
1.17
1.20
true
GA
1.21
1.25
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.21
false
Alpha
1.11
1.21
fa

lse
1.22
1.22
false
Alpha
1.9
1.12
true
Beta
1.13
1.17
true
GA
1.18
1.21
false
Alpha
1.13
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.23
false
Alpha
1.10
1.10
true
Beta
1.11
1.20
true
GA
1.21
1.22
false
Alpha
1.11
1.13
true
Beta
1.14
1.17
true
GA
1.18
1.21
false
Alpha
1.12
1.13
true
Beta
1.14
1.17
true
GA
1.18
1.21
false
Alpha
1.17
1.20
-
1.21
1.21
false
Alpha
1.17
1.20
-
1.21
1.21
false
Alpha
1.17
1.20
-
1.21
1.21
false
Alpha
1.17
1.20
-
1.21
1.21
false
Alpha
1.14
1.17
true
Beta
1.18
1.23
true
GA
1.24
1.25
false
Alpha
1.17
1.20
-
1.21
1.21
false
Beta
1.19
1.21
-
1.22
1.22
false
Alpha
1.12
1.13
true
Beta
1.14
1.16
true
GA
1.17
1.22
false
Alpha
1.9
1.9
true
Beta
1.10
1.12
true
GA
1.13
1.16
false
Alpha
1.20
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.24
false
Alpha
1.19
1.19
true
Beta
1.20
1.22
true
GA
1.23
1.25
false
Alpha
1.18
1.19
true
Beta
1.20
1.22
true
GA
1.23
1.25
false
Alpha
1.20
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.23
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.9
1.9
true
Beta
1.10
1.13
true
GA
1.14
1.16
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
1.18
false
Alpha
1.14
1.14
true
Beta
1.15
1.15
true
GA
1.16
1.18
false
Alpha
1.10
1.10
true
Beta
1.11
1.15
true
GA
1.16
1.18
false
Alpha
1.8
1.8
true
Beta
1.9
1.15
true
GA
1.16
1.18
false
Alpha
1.13
1.14
true
Beta
1.15
1.15
true
GA
1.16
1.18
false
Alpha
1.19
1.19
true
Beta
1.20
1.23
true
GA
1.24
1.25
false
Alpha
1.13
1.18
-
1.19
1.19
false
Alpha
1.4
1.10
true
Beta
1.11
1.21
false
1.22
1.25
false
Alpha
1.11
1.11
-
1.12
-
true
Alpha
1.3
1.7
true
GA
1.8
1.12
true
1.16
1.17
false
Alpha
1.8
1.12
-
1.13
1.23
false
Alpha
1.16
1.16
false
Beta
1.17
1.17
true
Beta
1.18
1.20
true
GA
1.21
1.24
false
Alpha
1.20
1.20
true
GA
1.21
1.24
false
Alpha
1.18
1.18
true
Beta
1.19
1.21
true
GA
1.22
1.24
EvenPodsSp

read
false
Alpha
1.16
1.17
true
Beta
1.18
1.18
true
GA
1.19
1.21
false
Alpha
1.5
1.12
false
1.13
1.16
true
GA
1.18
1.22
true
Beta
1.10
1.12
true
GA
1.13
1.16
false
Alpha
1.19
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
false
Alpha
1.18
1.18
true
Beta
1.19
1.21
true
GA
1.22
1.24
HugePages
false
Alpha
1.8
1.9
HugePages
true
Beta
1.10
1.13
HugePages
true
GA
1.14
1.16
false
Alpha
1.10
1.19
false
1.20
1.20
false
Alpha
1.15
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
false
Alpha
1.18
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.24
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.21
1.21
true
Beta
1.22
1.22
true
GA
1.23
1.24
false
Alpha
1.7
1.13
-
1.14
1.14
false
Alpha
1.8
1.9
-
1.10
1.10
false
Alpha
1.11
1.11
true
Beta
1.12
1.12
true
GA
1.13
1.16
false
Alpha
1.16
1.18
true
Beta
1.19
1.20
false
GA
1.21
1.22
false
Alpha
1.9
1.16
false
1.17
1.17
false
Alpha
1.8
1.9
true
Beta
1.10
1.11
true
GA
1.12
1.14
true
Beta
1.21
1.21
true
GA
1.22
1.23
false
Alpha
1.16
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.22
NodeLease
false
Alpha
1.12
1.13
NodeLease
true
Beta
1.14
1.16
NodeLease
true
GA
1.17
1.23
false
Alpha
1.15
1.18
true
Beta
1.19
1.23
true
GA
1.24
1.25
false
Alpha
1.9
1.9
-
1.10
1.10
false
Alpha
1.7
1.9
true
Beta
1.10
1.13
true
GA
1.14
1.16
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.3
1.4
true
Beta
1.5
1.20
true
GA
1.21
1.25
false
Alpha
1.16
1.17
true
Beta
1.18
1.23
true
GA
1.24
1.25
false
Alpha
1.8
1.10
true
Beta
1.11
1.13
true
GA
1.14
1.18
false
Alpha
1.11
1.11
true
Beta
1.12
1.13
true
GA
1.14
1.16
false
Alpha
1.10
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.19
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.15
1.16
-
1.17
1.17
false
Alpha
1.9
1.18
-
1.19
1.19
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
false
Alpha
1.13
1.19
true
Beta
1.20
1.20
true
GA
1.21
1.22
true
Beta
1.8
1.18
true
GA
1.19
1.21
true
Beta
1.14
1.20
true
GA
1.21
1.22
false
Alpha
1.12
1.13
true
Beta
1.14
1.19
true

GA
1.20
1.24
false
Alpha
1.12
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.22
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
false
Alpha
1.18
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.25
false
Alpha
1.18
1.19
true
Beta
1.20
1.20
true
GA
1.21
1.23
false
Alpha
1.18
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.22
false
Alpha
1.20
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
1.20
false
Alpha
1.8
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.22
false
Alpha
1.17
1.19
false
1.20
1.22
false
Alpha
1.19
1.19
true
Beta
1.20
1.21
true
GA
1.22
1,24
false
Alpha
1.16
1.17
true
Beta
1.18
1.19
true
GA
1.20
1.23
true
Beta
1.10
1.10
true
GA
1.11
1.24
false
Beta
1.5
1.5
true
Beta
1.6
1.17
true
1.18
1.21
false
1.22
1.24
false
Alpha
1.8
1.8
false
Beta
1.9
1.9
true
Beta
1.10
1.10
true
GA
1.11
1.20
false
Alpha
1.14
1.14
true
Beta
1.15
1.19
true
GA
1.20
1.23
false
Alpha
1.10
1.13
true
Beta
1.14
1.19
true
GA
1.20
1.23
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
Sysctls
true
Beta
1.11
1.20
Sysctls
true
GA
1.21
1.22
false
Alpha
1.12
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
false
Alpha
1.6
1.12
true
Beta
1.13
1.17
true
GA
1.18
1.20
false
Alpha
1.8
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
false
Alpha
1.10
1.11
true
Beta
1.12
1.19
true
GA
1.20
1.21
false
Alpha
1.11
1.11
true
Beta
1.12
1.19
true
GA
1.20
1.21
false
Alpha
1.12
1.13
true
Beta
1.14
1.21
true
1.22
1.24
false
Alpha
1.15
1.15
true
Beta
1.16
1.17
true
GA
1.18
1.21
false
Alpha
1.9
1.9
true
Beta
1.10
1.12
true
GA
1.13
1.16
false
Alpha
1.12
1.16
true
Beta
1.17
1.19
true
GA
1.20
1.22
true
GA
1.10
1.24
false
Alpha
1.14
1.14
true
Beta
1.15
1.16
true
GA
1.17
1.24
true
Beta
1.19
1.21
true
GA
1.22
1.24
false
Alpha
1.19
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.24
false
Alpha
1.14
1.15
true
Beta
1.16
1.17
true
GA
1.18
1.20
false
Alpha
1.16
1.16
true
Beta
1.17
1.17
true
GA
1.18
1.20
A

ccelerators
for
.
See
: Include volume count on node to be considered for
See
with
flag
.
Check
See
See
csi
.
prevents the regist

ration of in-tree GCE PD plugin.
kubelet and volume controllers and enables shims and translation logic to route
of the
.
.
field.
CronJob
dnsConfig
property.
Check
: Enable
/status
and
/scale
.
.
.
.
.
tim

eout on aggregated discovery calls.
.
nodeName
field.
.
.
critical
.
See
.
huge pages
.
HugePages
huge pages
.
: Enable
: Enable
dual stack
Job
Scope
and
Namespace
to
.
See
.
and
.
.
label
NodeLease
: Enable
local
local
volume.
and
feature.
.
PodRea

dinessGate
.
ConfigMap
named
See
SCTP
protocol
".
See
Pod's
field
.
startup
exec
,
attach
and
requests.
See
in the
and
: Enable support to suspend and resume Jobs. For mor

e details, see
.
Sysctls
pod. See
sysctls
: Allow a
.
: Enable the injection of service account tokens into a Pod through a
projected
volume
.
local
: Enable
subPath
.
.
3 - kubelet
Synopsis
Options
here
.)
0.0.0.0
or
::
--config
flag. See
*
--config
flag. See
here
.)
Enables anonymous requ

ests to the Kubelet server. Requests that are not rejected by another authentication method are treated as anonymous requests. Anonymous requests have a username of
--config
flag. See
Use the
--config
flag. See
2m0s
--config
flag. See
5m0s
--config
flag. See
30s
--config
flag. See
.
and
cgroupfs
cgroupfs
,
systemd
--config
flag. See
''
true
--config
flag. See
The path to

 the cloud provider configuration file. Empty string for no configuration file. (DEPRECATED: will be removed in 1.24 or later, in favor of removing cloud providers code from kubelet.)
Comma-separated list of DNS server IP address. This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst".
Note:
--config
flag. See
--config
flag. See
10Mi
10Mi
--config
flag. See
remote
docker
,
remote
,
.
--config
flag. See
true
--config
flag. See
100ms
--config
flag. See
none
none
,
static
--config
flag. See
A set of key=value

 CPU Manager policy options to use, to fine tune their behaviour. If not supplied, keep the default behaviour. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
10s
10s
, or
1m
--config
flag. See
--enable-controller-attach-detach     Default:
true
--config
flag. See
true
--config
flag. See
true
--config
flag. See
pods
none
,
pods
,
, and
and
none
here
--config
flag. See
--config
flag. See
--config
flag. See
--config
flag. See
--config
flag. See
5m0s
Duration

 for which the kubelet has to wait before transitioning out of an eviction pressure condition. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--config
flag. See
A set of eviction grace periods (e.g.
--config
flag. See
false
When set to
true
here
--config
flag. See
mount
mount
true
--config
flag. See
A set of
key=value
EphemeralContainers=true|false (

BETA - default=true)
--config
flag. See
20s
Duration between checking config files for new data. (D

EPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
,
and
none
--config
flag. See
127.0.0.1
The IP address for the healthz server to serve on (set to
0.0.0.0
or
::
--config
flag. See
0
--config
flag. See
-h, --help
20s
--config
flag. See
--config
flag. See
--config
flag. See
,
fwmark
--config
flag. See
fwmark
--config
flag. See
If enabled, the kubelet will integrate with the kernel memcg

 notification to determine if memory eviction thresholds are crossed rather than polling. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--config
flag. See
Content type of requests sent to apiserver. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
--config
flag. See
A set of
(e.g.
cpu
,
memory
and local
here
--config
flag. See
''
flag. Ex.
--config
flag. See
--config
flag. See
":0"
:
here
.)
here
.)
here
.)
here
.)
5s
'0'
--config
flag. See
[Experimental] In JSON format, write error messages to stde

rr and info messages to stdout. The default is to write a single stream to stdout. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
text
text
,
json
.
,
,
,
--log-dir
,
,
,
,
,
,
,
.
Non-default choices are currently alpha and subject to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
true
here
.)
true
iptables
--config
flag. See
--config
flag. See
--config
flag. See
default
--config
flag. See
--config
flag. See
or
or
None
'None'
,
'Static'
--config
flag. See
'300ms'
,
'10s'
or
'2h45m'
or
2m0s
'300ms'
,
'10s'
or
'2h45m'
--config
flag. See

'::'
','
,
'beta.kubernetes.io/arch'
,
,
,
,
,
,
,
,
,
,
)
. If
-1
--config
flag. See
10s
--config
flag. See
here
.)
The
--config
flag. See
--config
flag. See
docker
--config
flag. See
-1
--config
flag. See
, so
0
--config
flag. See
The port for the kubelet to serve on. (DEPRECATED: This parameter should be set via the config file specified by the k

ubelet's
--config
flag. See
--config
flag. See
--config
flag. See
(e.g.
memory
--config
flag. See
0
--config
flag. See
true
true
is
false
). No-op if
is
false
--config
flag. See
0
--config
flag. See
and
--config
flag. See
,
and
. See
here
--config
flag. See
--config
flag. See
--root-dir s

tring     Default:
--config
flag. See
when the certificate expiration approaches. Requires the
--config
flag. See
--runonce
If
true
2m0s
pull
,
logs
,
exec
and
attach
--config
flag. See
true
aufs
--config
flag. See
If
true
here
.)
If
true
here
.)
here
.)
4h0m0s
0
5m
--config
flag. See
1m0s
--config
flag. See
'/'
--config
flag. See
A set of
(e.g.
) pairs that describe resources reserved for non-kubernetes components. Currently o

nly
cpu
and
memory
here
--config
flag. See
''
flag. Ex.
--config
flag. See
File containing x509 Certificate used for serving HTTPS (with intermediate certs, if any, concatenated after server cert). If
and
--config
flag. See
,
,
,
--config
flag. See
--config
flag. See
'none'
'none'
,
,
,
--config
flag. See
container
,
'pod'
pattern=N
The full path of the directory in whic

h to search for additional third party volume plugins. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
1m0s
0
--config
flag. See
4 - kube-apiserver
Synopsis
Options
Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the back

end to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict.
File with authorization policy in json line by line format

, used with --authorization-mode=ABAC, on the secure port.
admission plugins that should be disabled although they are in the default enabled plugins list (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityCon

textDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionPolicy, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
The prefix to prepend to all

 resource paths in etcd.
LegacyServic

eAccountTokenTracking=true|false (ALPHA - default=false)
-h, --help
--kubelet-preferred-address-types st

rings     Default: "Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP"
The OpenID claim to use as the user name. Note that claims other than the default ('sub') is not guaranteed to be unique and immutable. This flag is exp

erimental, please see the authentication documentation for further details.
Turns on projected service account expiration extension during token generation, which helps safe transition from legacy token to bound service account token feature. If this flag is enabled, admission injected tokens would be extended up to 1 year to prevent unexpected failure during transition, ig

noring value of service-account-max-token-expiration.
--storage-media-ty

pe string     Default: "application/vnd.kubernetes.protobuf"
-v, --v int
Watch cache size settings for some resources (pods, nodes, etc.), comma separated. The individual setting format: resource[.group]#size, where resource is lowercase plural (no version), group is omitted for resources of apiVersion v1 (the legacy core API) and included for others, and size is a number. This option is only meaningful for resources built into the apiserver, not ones defined by CRDs or aggregated from external servers, and is only consulted if the watch-cache is enabled. The only meaningful size setting to supply here is zer

o, which means to disable watch caching for the associated resource; all non-zero values are equivalent and mean to not disable watch caching for that resource
5 - kube-controller-manager
Synopsis
Options
The path to the cloud provider configuration file. Empty

 string for no configuration file.
The number of namespace obj

ects that are allowed to sync concurrently. Larger number = more responsive namespace termination, but more CPU (and network) load
T

he length of endpoint updates batching period. Processing of pod changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of endpoints updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated
KubeletInUserNamespace=tru

e|false (ALPHA - default=false)
-h, --help
--horizontal-pod-autoscaler-sync-per

iod duration     Default: 15s
The length of EndpointSlice updates batching pe

riod for EndpointSliceMirroring controller. Processing of EndpointSlice changes will be delayed by this duration to join them with potential upcoming updates and reduce the overall number of EndpointSlice updates. Larger number = higher endpoint programming latency, but lower number of endpoints revision generated
Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNIN

G: generally do not depend on authorization being already done for incoming requests.
--tl

s-private-key-file string
-v, --v int
6 - kube-proxy
Synopsis
Options
--cleanup
--conntrack-tcp-timeout-establi

shed duration     Default: 24h0m0s
MinimizeIPTablesRestore=true|false (ALP

HA - default=false)
-h, --help
Enable strict ARP by set

ting arp_ignore to 1 and arp_announce to 2
Burst to use while talking with kubernetes apiserver
Range of host ports (beginPort-endPort, single port or beginPort+offset, inclusive) that may be consumed in order to proxy service traffic. If (unspecified, 0, or 0-0) then ports 

will be randomly chosen.
-v, --v int
7 - kube-scheduler
Synopsis
See
Options
The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or

 an unspecified address (0.0.0.0 or ::), all interfaces will be used.
DEPRECATED: enable lock contention profiling, if profiling is enabled. This parameter is ignored if a config file is specified in --config.
KubeletInUserNames

pace=true|false (ALPHA - default=false)
-h, --help
DEPRECATED: path to kube

config file with authorization and master location information. This parameter is ignored if a config file is specified in --config.
List of request header 

prefixes to inspect. X-Remote-Extra- is suggested.
-v, --v int
Home
Blog
Training
Partners
Community
CC BY 4.0
ICP 

Apply Pod Security Standards at the Namespace Level | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tutorials
Security
Note
baseline
.
KinD
kubectl
Create a
KinD
kind create cluster --name psa-ns-level --image kindest/node:v1.23.0

example
:
\
=
baseline
\
=
latest
enforce
the
baseline
warn
and
audit
for
\
=
baseline
\
pod-security.kubernete

s.io/enforce-version
=
latest
\
=
\
=
latest
\
=
\
=
latest
example
cat
kind: Pod
metadata:
spec:
ports:
EOF
example
default
example
default
namespace
Clean up
Run
Run a
Apply
baseline
enforce
warn
and
audit
mode.
Feedback
Yes
No
.
or
.
Last modified September 10, 2022 at 12:28 PM PST:
Fix cluster cleanup command (61ce6e27ef)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

CertificateSigningRequest | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

n (v1)
Reference
kind
metadata
(
)
spec
(
status
(
)
status contains information about whether the request is approved or denied, and the certificate issued by the signer, or the failure condition indicating signer failure.
request
"kubernetes.io/kube-apiserver-client-kubelet": issues client certificates that kubelets use to authenticate to

 kube-apiserver.
(int32)
extra
groups
uid
(string)
usages
Requests for TLS client certificates typically request: "digital signature", "key encipherment", "client auth".
username
(string)
([]byte)
certificate is populated with an issued certificate by the signer after an Approved condition is present. This field is 

set via the /status subresource. Once populated, this field is immutable.
base64(
...
)
(Time)
(Time)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
Para

meters
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
PUT /apis/certificates.k8s.io/v1/certificatesigningrequests/{name}/status
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch

name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
get
list
create
update
update
update
patch
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Lin

ux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Generating Reference Documentation for kubectl Commands | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator 

This page shows how to generate the
kubectl
Note:
This topic shows how to generate reference documentation for
like
and
.
This topic does not show how to generate the
kubectl
options reference page. For instructions on how to generate the kubectl options
reference page, see
.
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
GOPATH
.
mkdir -p
$HOME
export
GOPATH
=
$HOME
go get -u github.com/spf13/pflag
go get -u github.com/spf13/cobra
go get -u gopkg.in/yaml.v2
$GOPATH
$GOPATH
Remove the spf13 package from
$GOPATH/src/k8s.io/kubernetes/vendor/github.com
.
rm -rf
$GOPATH
/src/k8s.io/kubernetes/vendor/github.com/spf13
The kubernetes/kubernetes repository provides the
kubectl
and
kustomize
source code.
Determine the base directory of your clone of the
For example, if you followed the preceding step to get the repository, your
base directory is
.
$GOPATH/src/github.com/&lt;your-us

ername&gt;/website.
.
.
In your local k8s.io/kubernetes repository, check out the branch of interest,
and make sure it is up to date. For example, if you want to generate docs for
Kubernetes 1.25.0, you could use these commands:
cd
git checkout v1.25.0
git pull https://github.com/kubernetes/kubernetes 1.25.0
If you do not need to edit the
kubectl
source code, follow the instructions for
.
Editing the kubectl source code
The kubectl command reference documentation is automatically generated from
the kubectl source code. If you want to change the reference documentation, the first step
is to change one or more comments in the kubectl source code. Make the change in your
local kubernetes/kubernetes repository, and then submit a pull request to the master branch of
github.com/kubernetes/kubernetes
.
PR 56673
is an example of a pull request that fixes a typo in the kubectl source code.
Monitor your pull request, and respond to reviewer comments. Continue to monitor your
pull request until it is merged into the target branch of the kubernetes/kubernetes repository.
Cherry picking your change into a release branch
Your change is now in the master branch, which is used for development of the next
Kubernetes release. If you want your change to appear in the docs for a Kubernetes
version that has already been released, you need to propose that your change be
cherry picked into the release branch.
For example, suppose the master branch is being used to develop Kubernetes
1.26
and you want to backport your change to the release-1.25 branch. For
instructions on how to do this, see
Propose a Cherry Pick
.
Monitor your cherry-pick pull request until it is merged into the release branch.
Note:
Proposing a cherry pick requires that you have permission to set a label and a
milestone in your pull request. If you don't have those permissions, you will
need to work with someone who can set the label and milestone for you.
Go to
. On you command line, set the following environment variables.
Set
K8S_ROOT
to
.
Set
to
.
Set
For example, if you want to build docs for Kubernetes 1.25, set
to 1.25.
export
=
$GOPATH
export
K8S_ROOT
=
$GOPATH
export
=
1.25
Creating a versioned directory
The
createversiondirs
build target creates a versioned directory
and copies the kubectl reference configuration files to the versioned directory.
.
In the
cd
make createversiondirs
Checking out a release tag in k8s.io/kubernetes
In your local
repository, checkout the branch that has
the version of Kubernetes that you want to document. For example, if you want
to generate docs for Kubernetes 1.25.0, check out the
v1.25
tag. Make sure
you local branch is up to date.
cd
git pull https://github.com/kubernetes/kubernetes v1.25.0
Running the doc generation code
, run the
copycli
build target. The command runs as
root
:
cd
make copycli
The
copycli
command cleans the temporary build directory, generates the kubectl command files,
and copies the collated kubectl command reference HTML page and assets to
.
Locate the generated 

files
[
-e
"&lt;rdocs-base&gt;/gen-kubectldocs/generators/build/index.html"
]
echo
||
echo
[
-e
"&lt;rdocs-base&gt;/gen-kubectldocs/generators/build/navData.js"
]
echo
||
echo
Locate the copied files
Verify that all generated files have been copied to your
:
cd
git status
The output should include the modified files:
The output may also include:
static/docs/reference/generated/kubectl/node_modules/font-awesome/css/font-awesome.min.css
Locally test the documentation
Build the Kubernetes documentation in your local
.
cd
1
View the
.
Run
git add
and
git commit
.
Feedback
Yes
No
.
or
.
Last modified June 17, 2021 at 6:33 PM PST:
Pick example versions based on current release (191c2bf4ee)
Locate the generated files
Home
Blog
Training
Partners
Community
CC BY 4.0

Create static Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
API server
);
Kubelet
mirror Pod
This means that the Pods running on a node are visible on the API server,
Note:
DaemonSet
instead.
Note:
The
spec
(e.g.,
,
ConfigMap
,
Secret
, etc).
Note:
.
minikube
.
CRI-O
or a
.
,
my-node1
.
:
# Run this comman

d on the node where kubelet is running
cat
kind: Pod
metadata:
labels:
spec:
- name: web
ports:
- name: web
EOF
or add the
.
argument
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
role
:
myrole
spec
:
:
-
name
:
web
image
:
nginx
ports
:
-
name
:
web
:
80
protocol
:
TCP
When the kubelet starts, it automatically starts all defined static Pods. As you have
crictl ps
Note:
crictl
NAME
.
Note:
Labels
selectors
, etc.
kubectl
doesn't
sleep
20
crictl ps
89db4553e1eeb   docker.io/library/nginx@sha256:...    19 seconds ag

o    Running    web     1          34533c6729106
#
sleep
20
crictl ps
sleep
20
crictl ps
Feedback
Yes
No
.
or
.
Last modified November 20, 2022 at 7:39 PM PST:
updated docs regarding ephemeral pods not being supported by static pods kubernetes/kubernetes#113935 (65ce32de4e)
Home
Blog
Training
Partners
Community
CC BY 4.0

Schedule GPUs | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

d Set Up kubectl on macOS
Debug Pods
Get a Shell to a Ru

nning Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Cont

ainer's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configur

ation (v1)
Tasks
Configure and schedule GPUs for use as a resource by nodes in a cluster.
Kubernetes includes
stable
support for managing AMD and NVIDIA GPUs
(graphical processing units) across different nodes in your cluster, using
device plugins
.
This page describes how users can consume GPUs, and outlines
some of the limitations in the implementation.
Using device plugins
Kubernetes implements device plugins to let Pods access specialized hardware features such as GPUs.
Note:
As an administrator, you have to install GPU drivers from the corresponding
hardware vendor on the nodes and run the corresponding device plugin from the
GPU vendor. Here are some links to vendors' instructions:
AMD
Intel
NVIDIA
Once you have installed the plugin, your cluster exposes a custom schedulable resource such as
amd.com/gpu
or
.
You can consume these GPUs from your containers by requesting
the custom GPU resource, the same way you request
cpu
or
memory
.
However, there are some limitations in how you specify the resource
requirements for custom devices.
GPUs are only supposed to be specified in the
limits
section, which means:
You can specify GPU
limits
without specifying
requests
, because
Kubernetes will use the limit as the request value by default.
You can specify GPU in both
limits
and
requests
but these two values
must be equal.
You cannot specify GPU
requests
limits
.
Here's an example manifest for a Pod that requests a GPU:
:
v1
kind
:
Pod
metadata
:
name
:
example-vector-add
spec
:
:
OnFailure
:
-
name
:
image
:
"registry.example/example-vector-add:v42"
resources
:
limits
:
gpu-vendor.example/example-gpu
:
1
# requesting 1 GPU
Clusters containing different types of GPUs
If different nodes in your cluster have different types of GPUs, then you
can use
Node Labels and Node Selectors
to schedule pods to appropriate nodes.
# Label your nodes with the accelerator type they have.
kubectl label nodes node1
=
example-gpu-x100
kubectl label nodes node2
=
other-gpu-k915
That label key
is just an example; you can use
a different label key if you prefer.
Automatic node labelling
If you're usin

g AMD GPU devices, you can deploy
Node Labeller
.
Node Labeller is a
that automatically
labels your nodes with GPU device properties.
At the moment, that controller can add labels for:
Device ID (-device-id)
VRAM Size (-vram)
Number of SIMD (-simd-count)
Number of Compute Unit (-cu-count)
Firmware and Feature Versions (-firmware)
GPU Family, in two letters acronym (-family)
SI - Southern Islands
CI - Sea Islands
KV - Kaveri
VI - Volcanic Islands
CZ - Carrizo
AI - Arctic Islands
RV - Raven
kubectl describe node cluster-node-23
Name:               cluster-node-23
Labels:             beta.amd.com/gpu.cu-count.64=1
                    beta.amd.com/gpu.device-id.6860=1
                    beta.amd.com/gpu.family.AI=1
                    beta.amd.com/gpu.simd-count.256=1
                    beta.amd.com/gpu.vram.16G=1
                    kubernetes.io/hostname=cluster-node-23
Annotations:        node.alpha.kubernetes.io/ttl: 0
…
With the Node Labeller in use, you can specify the GPU type in the Pod spec:
:
v1
kind
:
Pod
metadata
:
name
:
cuda-vector-add
spec
:
:
OnFailure
:
-
name
:
# https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
image
:
resources
:
limits
:
:
1
affinity
:
:
:
:
– matchExpressions
:
– key
:
beta.amd.com/gpu.family.AI
# Arctic Islands GPU family
operator
:
Exist
This ensures that the Pod will be scheduled to a node that has the GPU type
you specified.
Feedback
Yes
No
.
or
.
Last modified October 18, 2022 at 11:44 AM PST:
node: devicemgr: docs: Additional updates based on review comments (0a0fb70fc2)
Home
Blog
Training
Partners
Community
CC BY 4.0

Namespace | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

Reference
Namespace
Namespace
Namespace
: v1
kind
metadata
(
)
spec
(
)
status
(
)
type
(Time)
(string)
(string)
phase
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Namespace
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
get
GET /

api/v1/namespaces/{name}/status
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
202 (
Namespace
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
If you plan to report an issue with this page, mention that the page is auto-generated in your issue description.

 The fix may need to happen elsewhere in the Kubernetes project.
Feedback
Yes
No
.
or
.
Namespace
get
get
list
create
update
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubectl Reference Docs
<!-- Latest compiled and minified CSS -->
GETTING STARTED
create
clusterrole
clusterrolebinding
configmap
cronjob
ingress
job
namespace
poddisruptionbudget
priorityclass
quota
role
rolebinding
secret
secret docker-registry
secret generic
secret tls
service
service clusterip
service externalname
service loadbalancer
service nodeport
serviceaccount
token
get
run
expose
delete
APP MANAGEMENT
apply
edit-last-applied
set-last-applied
view-last-applied
annotate
autoscale
debug
diff
edit
kustomize
label
patch
replace
rollout
history
pause
restart
resume
status
undo
scale
set
env
image
resources
selector
subject
wait
WORKING WITH APPS
attach
auth
can-i
reconcile
cp
describe
exec
logs
proxy
top
node
pod
CLUSTER MANAGEMENT
approve
deny
dump
cordon
drain
taint
uncordon
KUBECTL SETTINGS AND USAGE
alpha
events
config
delete-cluster
delete-context
delete-user
get-clusters
get-contexts
get-users
rename-context
set
set-cluster
set-context
set-credentials
unset
use-context
view
explain
options
plugin
list
version
Copyright 2020 The Kubernetes Authors.
example
This section contains the most basic commands for getting a workload
running on your cluster.
run
will start running 1 or more instances of a container image on your cluster.
expose
will load balance traffic across the running instances, and can create a HA proxy for accessing the containers from outside the cluster.
Once your workloads are running, you can use the commands in the
section to
inspect them.
create
Create a pod using the data in pod.json
kubectl
create
-f ./pod.json
Create a pod based on the JSON passed into stdin
cat pod.json | kubectl
create
-f -
Edit the data in registry.yaml in JSON then create the resource using the edited data
kubectl
create
-f registry.yaml --
edit
-o json
Create a resource from a file or from stdin.
JSON and YAML formats are accepted.
Usage
$ kubectl create -f FILENAME
Flags
Name
Shorthand
Default
Usage
allow-missing-template-keys
true
dry-run
none
Must be "none", "server", or "client". If client strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.
edit
false
Edit the API resource before creating
field-manager
kubectl-create
Name of the manager used to track field ownership.
filename
f
[]
Filename, directory, or URL to files to use to create the resource
kustomize
k
Process the kustomization directory. This flag can't be used together with -f or -R.
output
o
Output format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).
raw
Raw URI to POST to the server.  Uses the transport specified by the kubeconfig file.
record
false
Record current kubectl command in the resource annotation. If set to false, do not record the command. If set to true, record the command. If not set, default to updating the existing annotation value only if one already exists.
recursive
R
false
Process the directory used in -f, --filename recursively. Useful when you want to manage related manifests organized within the same directory.
save-config
false
If true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.
selector
l
Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.
show-managed-fields
false
template
Template string or path to tem

plate file to use when -o=go-template, -o=go-template-file. The template format is golang templates
[
http://golang.org/pkg/text/template/#pkg-overview
]
.
validate
strict
Must be one of: strict (or true), warn, ignore (or false).
"true" or "strict" will use a schema to validate the input and fail the request if invalid. It will perform server side validation if ServerSideFieldValidation is enabled on the api-server, but will fall back to less reliable client-side validation if not.
"warn" will warn about unknown or duplicate fields without blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" otherwise.
"false" or "ignore" will not perform any schema validation, silently dropping any unknown or duplicate fields.
windows-line-endings
false
Only relevant if --edit=true. Defaults to the line ending native to your platform.
Create a cluster role named "pod-reader" that allows user to perform "get", "watch" and "list" on pods
kubectl create clusterrole pod-reader
--verb
=get,list,watch
--resource
=pods
Create a cluster role named "pod-reader" with ResourceName specified
--verb
=get
=pods
--resource-name
=readablepod
=anotherpod
Create a cluster role named "foo" with API Group specified
kubectl create clusterrole foo
--verb
=rs.apps
Create a cluster role named "foo" with SubResource specified
--verb
=pods,pods/status
Create a cluster role name "foo" with NonResourceURL specified
"foo"
--verb
=get
--non-resource-url
=/logs/*
Create a cluster role name "monitoring" with AggregationRule specified
kubectl
create
--aggregation-rule="rbac.example.com/aggregate-to-monitoring=true"
Create a cluster role.
Usage
$ kubectl create clusterrole NAME --verb=verb --resource=resource.group [--resource-name=resourcename] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
aggregation-rule
An aggregation label selector for combining ClusterRoles.
true
dry-run
none
non-resource-url
[]
A partial url that user should have access to.
output
o
resource
[]
Resource that the rule applies to
resource-name
[]
Resource in the white list that the rule applies to, repeat this flag for multiple items
false
false
template
Template string or path to template file to use when -o=go-template, -o=go-template-file. The template format is golang templates
[
]
.
validate
strict
"warn" will warn about unknown or duplicate fields without blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" o

therwise.
verb
[]
Verb that applies to the resources contained in the rule
Create a cluster role binding for user1, user2, and group1 using the cluster-admin cluster role
kubectl create clusterrolebinding cluster-admin
=cluster-admin
--user
=user1
--user
=user2
--group
=group1
Create a cluster role binding for a particular cluster role.
Usage
$ kubectl create clusterrolebinding NAME --clusterrole=NAME [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
ClusterRole this ClusterRoleBinding should reference
dry-run
none
group
[]
Groups to bind to the clusterrole. The flag can be repeated to add multiple groups.
output
o
false
[]
Service accounts to bind to the clusterrole, in the format &lt;namespace&gt;:&lt;name&gt;. The flag can be repeated to add multiple service accounts.
false
template
[
]
.
validate
strict
configmap
Create a new config map named my-config based on folder bar
kubectl
create
configmap my-config --
from
-file=
path
/
to
/bar
Create a new config map named my-config with specified keys instead of file basenames on disk
kubectl create configmap my-config --
from
-
file
=key1=
/path/
to
/bar/
file1.txt --
from
-
file
=key2=
/path/
to
/bar/
file2.txt
Create a new config map named my-config with key1=config1 and key2=config2
kubectl create configmap my-config
=key1=config1
=key2=config2
Create a new config map named my-config from the key=value pairs in the file
kubectl
create
from
-file=
path
/
to
/bar
Create a new config map named my-config from an env file
kubectl
create
from
-env-file=
path
/
to
/foo.env --
from
path
/
to
/bar.env
Create a config map based on a file, directory, or specified literal value.
A single config map may package one or more key/value pairs.
When creating a config map based on a file, the key will default to the basename of the file, and the value will default to the file content.  If the basename is an invalid key, you may specify an alternate key.
When creating a c

onfig map based on a directory, each file whose basename is a valid key in the directory will be packaged into the config map.  Any directory entries except regular files are ignored (e.g. subdirectories, symlinks, devices, pipes, etc).
Usage
$ kubectl create configmap NAME [--from-file=[key=]source] [--from-literal=key1=value1] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
append-hash
false
Append a hash of the configmap to its name.
dry-run
none
from-env-file
[]
Specify the path to a file to read lines of key=val pairs to create a configmap.
from-file
[]
Key file can be specified using its file path, in which case file basename will be used as configmap key, or optionally with a key and file path, in which case the given key will be used.  Specifying a directory will iterate each named file in the directory whose basename is a valid configmap key.
from-literal
[]
Specify a key and literal value to insert in configmap (i.e. mykey=somevalue)
output
o
false
false
template
[
]
.
validate
strict
cronjob
Create a cron job
kubectl create cronjob my-job
--image
=busybox
--schedule
=
"*/1 * * * *"
Create a cron job with a command
--image
=busybox
=
"*/1 * * * *"
-- date
Create a cron job with the specified name.
Usage
$ kubectl create cronjob NAME --image=image --schedule='0/5 * * * ?' -- [COMMAND] [args...]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
image
Image name to run.
output
o
restart
job's restart policy. supported values: OnFailure, Never
false
If true, the configuration of current object will be saved in its annotation. Otherwise, the annotation will be unchanged. This flag is useful when 

you want to perform kubectl apply on this object in the future.
schedule
A schedule in the Cron format the job should be run with.
false
template
[
]
.
validate
strict
Create a deployment named my-dep that runs the busybox image
my
-dep
--image=busybox
Create a deployment with a command
my
-dep
--image=busybox -- date
Create a deployment named my-dep that runs the nginx image with 3 replicas
kubectl create deployment my-dep
--image
=nginx
=3
Create a deployment named my-dep that runs the busybox image and expose port 5701
--image
=busybox
--port
=5701
Create a deployment with the specified name.
Usage
$ kubectl create deployment NAME --image=image -- [COMMAND] [args...]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
image
[]
Image names to run.
output
o
port
-1
The port that this container exposes.
replicas
r
1
Number of replicas to create. Default is 1.
false
false
template
[
]
.
validate
strict
ingress
Create a single ingress called 'simple' that directs requests to foo.com/bar to svc # svc1:8080 with a tls secret "my-cert"
kubectl create ingress
simple
--rule
=
"foo.com/bar=svc1:8080,tls=my-cert"
Create a catch all ingress of "/path" pointing to service svc:port and Ingress Class as "otheringress"
catch
-all --
class
=otheringress --rule=
"/path=svc:port"
Creat

e an ingress with two annotations: ingress.annotation1 and ingress.annotations2
kubectl create ingress annotated
--class
=default
--rule
=
"foo.com/bar=svc:port"
\
--annotation ingress.
annotation1
=foo \
annotation2
=bla
Create an ingress with the same host and multiple paths
kubectl create ingress multipath
--class
=default \
--rule
=
"foo.com/=svc:port"
\
--rule
=
"foo.com/admin/=svcadmin:portadmin"
Create an ingress with multiple hosts and the pathType as Prefix
kubectl create ingress ingress1
--class
=default \
--rule
=
"foo.com/path*=svc:8080"
\
--rule
=
"bar.com/admin*=svc2:http"
Create an ingress with TLS enabled using the default ingress certificate and different path types
kubectl create ingress ingtls
--class
=default \
--rule
=
"foo.com/=svc:https,tls"
\
--rule
=
"foo.com/path/subpath*=othersvc:8080"
Create an ingress with TLS enabled using a specific secret and pathType as Prefix
kubectl create ingress ingsecret --
class
=
default
\
--rule=
"foo.com/*=svc:8080,tls=secret1"
Create an ingress with a default backend
kubectl create ingress ingdefault
--class
=default \
--default-backend
=defaultsvc:http \
--rule
=
Create an ingress with the specified name.
Usage
$ kubectl create ingress NAME --rule=host/path=service:port[,tls[=secret]]
Flags
Name
Shorthand
Default
Usage
true
[]
Annotation to insert in the ingress object, in the format annotation=value
class
Ingress Class to be used
default-backend
Default service for backend, in format of svcname:port
dry-run
none
output
o
rule
[]
Rule in format host/path=service:port
[
,tls=secretname
]
. Paths containing the leading character '*' are considered pathType=Prefix. tls argument is optional.
false
false
template
[
]
.
validate
strict
job
Create a job
kubectl create job
my
-job
Create a job with a command
job
my-
job
--image=
busybox
-- date
Create a job from a cron job named "a-cronjob"
kubectl
create
job
test
-job
--from=cronjob/a-cronjob
Create a job with the specified name.
Usage
$ kubectl create job NAME --image=image [--from=cronjob/name] -- [COMMAND] [args...]
Flags
Name
Shorthand
Default
Usage
true
If true, ignore any errors in templates when a field or map key is missing in 

the template. Only applies to golang and jsonpath output formats.
dry-run
none
from
The name of the resource to create a Job from (only cronjob is supported).
image
output
o
false
false
template
[
]
.
validate
strict
namespace
Create a new namespace named my-namespace
kubectl
create
namespace
my-
namespace
Create a namespace with the specified name.
Usage
$ kubectl create namespace NAME [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
output
o
false
false
template
[
]
.
validate
strict
Create a pod disruption budget named my-pdb that will select all pods with the app=rails label # and require at least one of them being available at any point in time
kubectl create poddisruptionbudget my-pdb

=app=rails
--min-available
=1
Create a pod disruption budget named my-pdb that will select all pods with the app=nginx label # and require at least half of the pods selected to be available at any point in time
kubectl create pdb my-pdb
=app=nginx
=50%
Create a pod disruption budget with the specified name, selector, and desired minimum available pods.
Usage
$ kubectl create poddisruptionbudget NAME --selector=SELECTOR --min-available=N [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
The maximum number or percentage of unavailable pods this budget requires.
min-available
The minimum number or percentage of available pods this budget requires.
output
o
false
selector
A label selector to use for this budget. Only equality-based selector requirements are supported.
false
template
[
]
.
validate
strict
Create a priority class named high-priority
kubectl create priorityclass high-priority
--value
=1000
--description
=
"high priority"
Create a priority class named default-priority that is considered as the global default priority
kubectl create priorityclass
default
-priority --
value
=
1000
--
global
-
default
=
true
--description=
"default priority"
Create a priority class named high-priority that cannot preempt pods with lower priority
--value
=1000
=
--preemption-policy
=
"Never"
Create a priority class with the specified name, value, globalDefault and description.
Usage
$ kubectl create priorityclass NAME --value=VALUE --global-default=BOOL [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none

false
global-default specifies whether this PriorityClass should be considered as the default priority.
output
o
preemption-policy
preemption-policy is the policy for preempting pods with lower priority.
false
false
template
[
]
.
validate
strict
value
0
the value of this priority class.
quota
Create a new resource quota named my-quota
kubectl create quota my-quota --hard=cpu=
1
,memory=
1
G,pods=
2
,services=
3
,replicationcontrollers=
2
,resourcequotas=
1
,secrets=
5
,persistentvolumeclaims=
10
Create a new resource quota named best-effort
kubectl create quota best-effort
--hard
=pods=100
--scopes
=BestEffort
Create a resource quota with the specified name, hard limits, and optional scopes.
Usage
$ kubectl create quota NAME [--hard=key1=value1,key2=value2] [--scopes=Scope1,Scope2] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
hard
A comma-delimited set of resource=quantity pairs that define a hard limit.
output
o
false
scopes
A comma-delimited set of quota scopes that must all match each object tracked by the quota.
false
template
[
]
.
validate
strict
"false" or "ignore" 

will not perform any schema validation, silently dropping any unknown or duplicate fields.
role
Create a role named "pod-reader" that allows user to perform "get", "watch" and "list" on pods
kubectl create role pod-reader
--verb
=get
--verb
=list
--verb
=watch
=pods
Create a role named "pod-reader" with ResourceName specified
--verb
=get
=pods
Create a role named "foo" with API Group specified
kubectl create role foo
--verb
=rs.apps
Create a role named "foo" with SubResource specified
--verb
Create a role with single rule.
Usage
$ kubectl create role NAME --verb=verb --resource=resource.group/subresource [--resource-name=resourcename] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
output
o
resource
[]
[]
false
false
template
[
]
.
validate
strict
verb
[]
Create a role binding for user1, user2, and group1 using the admin cluster role
kubectl create rolebinding admin
=admin
--user
=user1
--user
=user2
--group
=group1
Create a role binding for a particular role or cluster role.
Usage
$ kubectl create rolebinding NAME --clusterrole=NAME|--role=NAME [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
ClusterRole this RoleBinding should reference
dry-run
none
group
[]
Groups to bind to the role. The flag can be repeated to add multiple groups.
output
o
O

utput format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file).
role
Role this RoleBinding should reference
false
[]
Service accounts to bind to the role, in the format &lt;namespace&gt;:&lt;name&gt;. The flag can be repeated to add multiple service accounts.
false
template
[
]
.
validate
strict
secret
Create a secret using specified subcommand.
Usage
$ kubectl create secret
If you don't already have a .dockercfg file, you can create a dockercfg secret directly by using:
secret
docker-registry my-secret
--docker-server
=DOCKER_REGISTRY_SERVER
=DOCKER_USER
--docker-password
=DOCKER_PASSWORD
=DOCKER_EMAIL
Create a new secret named my-secret from ~/.docker/config.json
secret
=.dockerconfigjson=path/to/.docker/config.json
Create a new secret for use with Docker registries.
Dockercfg secrets are used to authenticate against Docker registries.
When using the Docker command line to push images, you can authenticate to a given registry by running:
    '$ docker login DOCKER_REGISTRY_SERVER --username=DOCKER_USER --password=DOCKER_PASSWORD --email=DOCKER_EMAIL'.
That produces a ~/.dockercfg file that is used by subsequent 'docker push' and 'docker pull' commands to authenticate to the registry. The email address is optional.
When creating applications, you may have a Docker registry that requires authentication.  In order for the
  nodes to pull images on your behalf, they must have the credentials.  You can provide this information
  by creating a dockercfg secret and attaching it to your service account.
Usage
$ kubectl create docker-registry NAME --docker-username=user --docker-password=password --docker-email=email [--docker-server=string] [--from-file=[key=]source] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
false
Append a hash of the secret to its name.
docker-email
Email for Docker registry
docker-password
Password for Docker registry authentication
docker-server
Server location for Docker registry
docker-username
Username for Docker registry authentication
dry-run
none
from-file
[]
Key files can be specified using their file path, in which case a default name will be given to them, or optionally with a name and file path, i

n which case the given name will be used.  Specifying a directory will iterate each named file in the directory that is a valid secret key.
output
o
false
false
template
[
]
.
validate
strict
Create a new secret named my-secret with keys for each file in folder bar
secret
generic my-secret
=path/to/bar
Create a new secret named my-secret with specified keys instead of names on disk
secret
=ssh-privatekey=path/to/id_rsa
=ssh-publickey=path/to/id_rsa.pub
Create a new secret named my-secret with key1=supersecret and key2=topsecret
secret
=key1=supersecret
=key2=topsecret
Create a new secret named my-secret using a combination of a file and a literal
secret
=passphrase=topsecret
Create a new secret named my-secret from env files
secret
=path/to/foo.env
=path/to/bar.env
Create a secret based on a file, directory, or specified literal value.
A single secret may package one or more key/value pairs.
When creating a secret based on a file, the key will default to the basename of the file, and the value will default to the file content. If the basename is an invalid key or you wish to chose your own, you may specify an alternate key.
When creating a secret based on a directory, each file whose basename is a valid key in the directory will be packaged into the secret. Any directory entries except regular files are ignored (e.g. subdirectories, symlinks, devices, pipes, etc).
Usage
$ kubectl create generic NAME [--type=string] [--from-file=[key=]source] [--from-literal=key1=value1] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
false
dry-run
none
[]
Specify the path to a file to read lines of key=val pairs to create a secret.
from-file
[]
Key files can be specified using their file path, in which case a default name will be given to them, or optionally with a name and file path, in which case the given name will be used.  Specifying a directory will iterate each named fil

e in the directory that is a valid secret key.
[]
Specify a key and literal value to insert in secret (i.e. mykey=somevalue)
output
o
false
false
template
[
]
.
type
The type of secret to create
validate
strict
secret tls
Create a new TLS secret named tls-secret with the given key pair
secret
tls tls-secret
--cert
=path/to/tls.cert
--key
=path/to/tls.key
Create a TLS secret from the given public/private key pair.
The public/private key pair must exist beforehand. The public key certificate must be .PEM encoded and match the given private key.
Usage
$ kubectl create tls NAME --cert=path/to/cert/file --key=path/to/key/file [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
false
cert
Path to PEM encoded public key certificate.
dry-run
none
key
Path to private key associated with given certificate.
output
o
false
false
template
[
]
.
validate
strict
service
Create a service using a specified subcommand.
Usage
$ kubectl create service
service c

lusterip
Create a new ClusterIP service named my-cs
service
clusterip my-cs
--tcp
=5678:8080
Create a new ClusterIP service named my-cs (in headless mode)
service
--clusterip
=
"None"
Create a ClusterIP service with the specified name.
Usage
$ kubectl create clusterip NAME [--tcp=&lt;port&gt;:&lt;targetPort&gt;] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
clusterip
Assign your own ClusterIP or set to 'None' for a 'headless' service (no loadbalancing).
dry-run
none
output
o
false
false
tcp
[]
Port pairs can be specified as '&lt;port&gt;:&lt;targetPort&gt;'.
template
[
]
.
validate
strict
Create a new ExternalName service named my-ns
service
externalname my-ns --external-name bar.com
Create an ExternalName service with the specified name.
ExternalName service references to an external DNS address instead of only pods, which will allow application authors to reference services that exist off platform, on other clusters, or locally.
Usage
$ kubectl create externalname NAME --external-name external.name [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
external-name
External name of service
output
o
false
false
tcp
[]
template
Template string or path to template 

file to use when -o=go-template, -o=go-template-file. The template format is golang templates
[
]
.
validate
strict
Create a new LoadBalancer service named my-lbs
service
loadbalancer my-lbs
--tcp
Create a LoadBalancer service with the specified name.
Usage
$ kubectl create loadbalancer NAME [--tcp=port:targetPort] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
output
o
false
false
tcp
[]
template
[
]
.
validate
strict
Create a new NodePort service named my-ns
service
nodeport my-ns
--tcp
Create a NodePort service with the specified name.
Usage
$ kubectl create nodeport NAME [--tcp=port:targetPort] [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
node-port
0
Port used to expose the service on each node in a cluster.
output
o
false
If true, the configuration of current object will be saved in its annotat

ion. Otherwise, the annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.
false
tcp
[]
template
[
]
.
validate
strict
Create a new service account named my-service-account
kubectl
create
serviceaccount my-service-
account
Create a service account with the specified name.
Usage
$ kubectl create serviceaccount NAME [--dry-run=server|client|none]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
output
o
false
false
template
[
]
.
validate
strict
token
Request a token to authenticate to the kube-apiserver as the service account "myapp" in the current namespace
kubectl
create
token
myapp
Request a token for a service account in a custom namespace
kubectl
create
token
myapp
--namespace myns
Request a token with a custom expiration
kubectl
create
token
myapp
--duration 10m
Request a token with a custom audience
kubectl
create
token
myapp --audience https:
//example.com
Request a token bound to an instance of a Secret object
kubectl create token myapp --bound-object-kind
Secret
--bound-object-name mysecret
Request a token bound to an instance of a Secret object with a specific uid
kubectl create token myapp --bound-object-kind Secret --bound-object-name mysecret --bound-object-uid
0d4691ed
-
659b-4935
-a832-355f77ee47cc
Request a service account token.
Usage

$ kubectl create token SERVICE_ACCOUNT_NAME
Flags
Name
Shorthand
Default
Usage
true
audience
[]
Audience of the requested token. If unset, defaults to requesting a token for use with the Kubernetes API server. May be repeated to request a token valid for multiple audiences.
bound-object-kind
Kind of an object to bind the token to. Supported kinds are Pod, Secret. If set, --bound-object-name must be provided.
bound-object-name
Name of an object to bind the token to. The token will expire when the object is deleted. Requires --bound-object-kind.
bound-object-uid
UID of an object to bind the token to. Requires --bound-object-kind and --bound-object-name. If unset, the UID of the existing object is used.
duration
0s
Requested lifetime of the issued token. The server may return a token with a longer or shorter lifetime.
output
o
false
template
[
]
.
get
List all pods in ps output format
kubectl
get
pods
List all pods in ps output format with more information (such as node name)
kubectl
get
pods -o wide
List a single replication controller with specified NAME in ps output format
kubectl
get
replicationcontroller web
List deployments in JSON output format, in the "v1" version of the "apps" API group
kubectl
get
.v1
.apps
-o
json
List a single pod in JSON output format
kubectl
get
-o
json
pod web-pod
-13
je7
List a pod identified by type and name specified in "pod.yaml" in JSON output format
kubectl
get
-f pod.yaml -o
json
List resources from a directory with kustomization.yaml - e.g. dir/kustomization.yaml
kubectl
get
-k dir/
Return only the phase value of the specified pod
kubectl
get
-o
template
pod/web-pod
-13
je7 --
template
={{.status.phase}}
List resource information in custom columns
kubectl
get
pod test-pod -o
=CONTAINER:.spec.containers[0].name,IMAGE:.spec.containers[0].image
List all replication controllers and services together in ps output format
kubectl
get
rc,services
List one or more resources by their type and names
kubectl
get
rc/web service/frontend pods/web-pod-13je7
List status subresource for a single pod.
kubectl
get
-13
je7
--subresource status
Display one or many resources.
Prints a table of the most important information about the specified resources. You can filter the list using a label selector and the --selector flag. If the desired resource type is namespaced you will only see results in your current namespace unless you pass --all-namespaces.
By specifying the output as 'template' and providing a Go template as the value of the --template flag, you can filter the attributes of the fetched resources.
Use "kubectl api-resources" for a complete list of supported resources.
Usage
$ kubectl get [(-o|--output=)json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file|custom-columns|custom-columns-file|wide] (TYPE[.VERSION][.GROUP] [NAME | -l label] | TYPE[.VERSION][.GROUP]/NAME ...) [flags]
Flags
Name
Shorthand
Default
Usage
all-namespaces
A
false
If present, list the requested object(s) across all namespaces. Namespace in current context is ignored even if specified with --namespace.
true
chunk-size
500
Return large lists in chunks rather than all at once. Pass 0 to disable

. This flag is beta and may change in the future.
field-selector
Selector (field query) to filter on, supports '=', '==', and '!='.(e.g. --field-selector key1=value1,key2=value2). The server only supports a limited number of field queries per type.
filename
f
[]
Filename, directory, or URL to files identifying the resource to get from a server.
ignore-not-found
false
If the requested object does not exist the command will return exit code 0.
kustomize
k
label-columns
L
[]
Accepts a comma separated list of labels that are going to be presented as columns. Names are case-sensitive. You can also use multiple flag options like -L label1 -L label2...
no-headers
false
When using the default or custom-column output format, don't print headers (default print headers).
output
o
Output format. One of: (json, yaml, name, go-template, go-template-file, template, templatefile, jsonpath, jsonpath-as-json, jsonpath-file, custom-columns, custom-columns-file, wide). See custom columns
[
https://kubernetes.io/docs/reference/kubectl/#custom-columns
]
, golang template
[
]
and jsonpath template
[
https://kubernetes.io/docs/reference/kubectl/jsonpath/
]
.
output-watch-events
false
Output watch event objects when --watch or --watch-only is used. Existing objects are output as initial ADDED events.
raw
Raw URI to request from the server.  Uses the transport specified by the kubeconfig file.
recursive
R
false
selector
l
server-print
true
If true, have the server return the appropriate table output. Supports extension APIs and CRDs.
show-kind
false
If present, list the resource type for the requested object(s).
show-labels
false
When printing, show all labels as the last column (default hide labels column)
false
sort-by
If non-empty, sort list types using this field specification.  The field specification is expressed as a JSONPath expression (e.g. '{.metadata.name}'). The field in the API resource specified by this JSONPath expression must be an integer or a string.
If specified, gets the subresource of the requested object. Must be one of
[
status scale
]
. This flag is alpha and may change in the future.
template
[
]
.
use-openapi-print-columns
false
If true, use x-kubernetes-print-column metadata (if present) from the OpenAPI schema for displaying a resource.
watch
w
false
After listing/getting the requested object, watch for changes.
watch-only
false
Watch for changes to the requested object(s), without listing/getting first.
run
Start a nginx pod
kubectl
run
nginx
--image
=nginx
Start a hazelcast pod and let the container expose port 5701
kubectl
run
hazelcast
--image
=hazelcast/hazelcast
--port
=5701
Start a hazelcast pod and set environment variables "DNS_DOMAIN=cluster" and "POD_NAMESPACE=default" in the container
kubectl
run
hazelcast
--image
--env
=
"DNS_DOMAIN=cluster"
--env
=
"POD_NAMESPACE=default"
Start a hazelcast pod and set labels "app=hazelcast" and "env=prod" in the container
kubectl
run
hazelcast
--image
--labels
=
"app=hazelcast,env=prod"
Dry run; print the corresponding API objects without creating them
kubectl
run
nginx
--image
=nginx
--dry-run
=client
Start a nginx pod, but overload the spec with a partial set of values parsed from JSON
kubectl run nginx
--ima

ge=nginx
--overrides=
'{
:
"v1"
,
"spec"
: {
...
} }'
Start a busybox pod and keep it in the foreground, don't restart it if it exits
kubectl
run
-i -t busybox
--image
=busybox
--restart
=Never
Start the nginx pod using the default command, but use custom arguments (arg1 .. argN) for that command
kubectl
run
nginx
--image
=nginx -- &lt;arg1&gt; &lt;arg2&gt;
..
. &lt;argN&gt;
Start the nginx pod using a different command and custom arguments
kubectl
run
nginx
--image
=nginx --command -- &lt;cmd&gt; &lt;arg1&gt;
..
Create and run a particular image in a pod.
Usage
$ kubectl run NAME --image=image [--env="key=value"] [--port=port] [--dry-run=server|client] [--overrides=inline-json] [--command] -- [COMMAND] [args...]
Flags
Name
Shorthand
Default
Usage
true
[]
Annotations to apply to the pod.
attach
false
If true, wait for the Pod to start running, and then attach to the Pod as if 'kubectl attach ...' were called.  Default false, unless '-i/--stdin' is set, in which case the default is true. With '--restart=Never' the exit code of the container process is returned.
command
false
If true and extra arguments are present, use them as the 'command' field in the container, rather than the 'args' field which is the default.
dry-run
none
env
[]
Environment variables to set in the container.
expose
false
If true, create a ClusterIP service associated with the pod.  Requires
--port
.
kubectl-run
image
The image for the container to run.
image-pull-policy
The image pull policy for the container.  If left empty, this value will not be specified by the client and defaulted by the server.
labels
l
Comma separated labels to apply to the pod. Will override previous values.
leave-stdin-open
false
If the pod is started in interactive mode or with stdin, leave stdin open after the first attach completes. By default, stdin will be closed after the first attach completes.
output
o
override-type
merge
The method used to override the generated object: json, merge, or strategic.
overrides
An inline JSON override for the generated object. If this is non-empty, it is used to override the generated object. Requires that the object supply a valid apiVersion field.
pod-running-timeout
1m0s
The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one pod is running
port
false
If true, run the container in privileged mode.
quiet
q
false
If true, suppress prompt messages.
record
false
restart
Always
The restart policy for this Pod.  Legal values
[
Always, OnFailure, Never
]
.
rm
false
If true, delete the pod after it exits.  Only valid when attaching to the container, e.g. with '--attach' or with '-i/--stdin'.
false
false
stdin
i
false
Keep stdin open on the container in the pod, even if nothing is attached.
template
Template string or path to

 template file to use when -o=go-template, -o=go-template-file. The template format is golang templates
[
]
.
tty
t
false
Allocate a TTY for the container in the pod.
expose
Create a service for a replicated nginx, which serves on port 80 and connects to the containers on port 8000
kubectl expose rc nginx
--port
=80
=8000
Create a service for a replication controller identified by type and name specified in "nginx-controller.yaml", which serves on port 80 and connects to the containers on port 8000
kubectl expose -f nginx-controller.yaml
--port
=80
=8000
Create a service for a pod valid-pod, which serves on port 444 with the name "frontend"
kubectl expose pod valid-pod
--port
=444
--name
=frontend
Create a second service based on the above service, exposing the container port 8443 as port 443 with the name "nginx-https"
service
nginx
--port
=443
=8443
--name
=nginx-https
Create a service for a replicated streaming application on port 4100 balancing UDP traffic and named 'video-stream'.
kubectl expose rc streamer
--port
=4100
--protocol
=UDP
--name
=video-stream
Create a service for a replicated nginx using replica set, which serves on port 80 and connects to the containers on port 8000
kubectl expose rs nginx
--port
=80
=8000
Create a service for an nginx deployment, which serves on port 80 and connects to the containers on port 8000
kubectl expose deployment nginx
--port
=80
=8000
Expose a resource as a new Kubernetes service.
Looks up a deployment, service, replica set, replication controller or pod by name and uses the selector for that resource as the selector for a new service on the specified port. A deployment or replica set will be exposed as a service only if its selector is convertible to a selector that service supports, i.e. when the selector contains only the matchLabels component. Note that if no port is specified via --port and the exposed resource has multiple ports, all will be re-used by the new service. Also if no labels are specified, the new service will re-use the labels from the resource it exposes.
Possible resources include (case insensitive):
pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs)
Usage
$ kubectl expose (-f FILENAME | TYPE NAME) [--port=port] [--protocol=TCP|UDP|SCTP] [--target-port=number-or-name] [--name=name] [--external-ip=external-ip-of-service] [--type=type]
Flags
Name
Shorthand
Default
Usage
true
cluster-ip
ClusterIP to be assigned to the service. Leave empty to auto-allocate, or set to 'None' to create a headless service.
dry-run
none
external-ip
Additional external IP address (not managed by Kubernetes) to accept for the service. If this IP is routed to a node, the service can be accessed by this IP in addition to its generated service IP.
kubectl-expose
filename
f
[]
Filename, directory, or URL to files identifying the resource to expose a service
kustomize
k
labels
l
Labels to apply to the service created by this call.
load-balancer-ip
IP to assign to the LoadBalancer. If empty, an ephemeral IP will be created and used (cloud-provider specific).
name
The name for the newly created object.
output
o
merge
The method used to override the generate

d object: json, merge, or strategic.
overrides
port
The port that the service should serve on. Copied from the resource being exposed, if unspecified
protocol
The network protocol for the service to be created. Default is 'TCP'.
record
false
recursive
R
false
false
selector
A label selector to use for this service. Only equality-based selector requirements are supported. If empty (the default) infer the selector from the replication controller or replica set.)
session-affinity
If non-empty, set the session affinity for the service to this; legal values: 'None', 'ClientIP'
false
target-port
Name or number for the port on the container that the service should direct traffic to. Optional.
template
[
]
.
type
Type for this service: ClusterIP, NodePort, LoadBalancer, or ExternalName. Default is 'ClusterIP'.
delete
Delete a pod using the type and name specified in pod.json
kubectl
delete
Delete resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml
kubectl
delete
-k dir
Delete resources from all files that end with '.json' - i.e. expand wildcard characters in file names
kubectl
delete
-f
'*.json'
Delete a pod based on the type and name in the JSON passed into stdin
delete
-f -
Delete pods and services with same names "baz" and "foo"
kubectl
delete
pod,service baz foo
Delete pods and services with label name=myLabel
kubectl
delete
pods,services -l
name
=myLabel
Delete a pod with minimal delay
kubectl
delete
pod foo
--now
Force delete a pod on a dead node
kubectl
delete
pod foo
--force
Delete all pods
kubectl
delete
pods
--all
Delete resources by file names, stdin, resources and names, or by resources and label selector.
JSON and YAML formats are accepted. Only one type of argument may be specified: file names, resources and names, or resources and label selector.
Some resources, such as pods, support graceful deletion. These resources define a default period before they are forcibly terminated (the grace period) but you may override that value with the --grace-period flag, or pass --now to set a grace-period of 1. Because these resources often represent entities in the cluster, deletion may not be acknowledged immediately. If the node hosting a pod is down or cannot reach the API server, termination may take significantly longer than the grace period. To force delete a resource, you must specify the --force flag. Note: only a subset of resources support graceful deletion. In absence of the support, the --grace-period flag is ignored.
IMPORTANT: Force deleting pods does not wait for confirmation that the pod's processes have been terminated, which can leave those processes running until the node detects the deletion and completes graceful deletion. If your processes use shared storage or talk to a remote API and depend on the name of the pod to identify themselves, force deleting those pods may result in multiple processes running on dif

ferent machines using the same identification which may lead to data corruption or inconsistency. Only force delete pods when you are sure the pod is terminated, or if your application can tolerate multiple copies of the same pod running at once. Also, if you force delete pods, the scheduler may place new pods on those nodes before the node has released those resources and causing those pods to be evicted immediately.
Note that the delete command does NOT do resource version checks, so if someone submits an update to a resource right when you submit a delete, their update will be lost along with the rest of the resource.
After a CustomResourceDefinition is deleted, invalidation of discovery cache may take up to 6 hours. If you don't want to wait, you might want to run "kubectl api-resources" to refresh the discovery cache.
Usage
$ kubectl delete ([-f FILENAME] | [-k DIRECTORY] | TYPE [(NAME | -l label | --all)])
Flags
Name
Shorthand
Default
Usage
all
false
Delete all resources, in the namespace of the specified resource types.
A
false
cascade
Must be "background", "orphan", or "foreground". Selects the deletion cascading strategy for the dependents (e.g. Pods created by a ReplicationController). Defaults to background.
dry-run
none
filename
f
[]
containing the resource to delete.
force
false
If true, immediately remove resources from API and bypass graceful deletion. Note that immediate deletion of some resources may result in inconsistency or data loss and requires confirmation.
grace-period
-1
Period of time in seconds given to the resource to terminate gracefully. Ignored if negative. Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion).
false
Treat "resource not found" as a successful delete. Defaults to "true" when --all is specified.
kustomize
k
Process a kustomization directory. This flag can't be used together with -f or -R.
now
false
If true, resources are signaled for immediate shutdown (same as --grace-period=1).
output
o
Output mode. Use "-o name" for shorter output (resource/name).
raw
Raw URI to DELETE to the server.  Uses the transport specified by the kubeconfig file.
recursive
R
false
selector
l
timeout
0s
The length of time to wait before giving up on a delete, zero means determine a timeout from the size of the object
wait
true
If true, wait for resources to be gone before returning. This waits for finalizers.
This section contains commands for creating, updating, deleting, and
viewing your workloads in a Kubernetes cluster.
apply
Apply the configuration in pod.json to a pod
kubectl
apply
Apply resources from a directory containing kustomization.yaml - e.g. dir/kustomization.yaml
kubectl
apply
-k dir/
Apply the JSON passed into stdin to a pod
cat pod.json |
kubectl
apply
-f -
Apply the configuration from all files that end with '.json' - i.e. expand wildcard characters in file names
kubectl
apply -f
'*.json'
Note: --prune is still in Alpha # Apply the configuration in manifest.yaml that matches label app=nginx and delete all other resources that are not in the 

file and match label app=nginx
kubectl apply --prune -f manifest.yaml -l
app
=nginx
Apply the configuration in manifest.yaml and delete all the other config maps that are not in the file
kubectl
apply
--
prune
-
f
manifest
.
yaml
--
all
--
prune
-
whitelist=core/v1/ConfigMap
Apply a configuration to a resource by file name or stdin. The resource name must be specified. This resource will be created if it doesn't exist yet. To use 'apply', always create the resource initially with either 'apply' or 'create --save-config'.
Alpha Disclaimer: the --prune functionality is not yet complete. Do not use unless you are aware of what the current state is. See
https://issues.k8s.io/34274
.
Usage
$ kubectl apply (-f FILENAME | -k DIRECTORY)
Flags
Name
Shorthand
Default
Usage
all
false
Select all resources in the namespace of the specified resource types.
true
cascade
dry-run
none
kubectl-client-side-apply
filename
f
[]
The files that contain the configurations to apply.
force
false
force-conflicts
false
If true, server-side apply will force the changes against conflicts.
-1
kustomize
k
openapi-patch
true
If true, use openapi to calculate diff when the openapi presents and the resource can be found in the openapi spec. Otherwise, fall back to use baked-in types.
output
o
overwrite
true
Automatically resolve conflicts between the modified and live configuration by using values from the modified configuration
prune
false
Automatically delete resource objects, that do not appear in the configs and are created by either apply or create --save-config. Should be used with either -l or --all.
prune-whitelist
[]
Overwrite the default whitelist with &lt;group/version/kind&gt; for --prune
record
false
recursive
R
false
selector
l
server-side
false
If true, apply runs in the server instead of the client.
false
template
[
]
.
timeout
0s
The length of time to wait before giving up on a dele

te, zero means determine a timeout from the size of the object
validate
strict
wait
false
Edit the last-applied-configuration annotations by type/name in YAML
kubectl
apply
edit-
last
-applied deployment/nginx
Edit the last-applied-configuration annotations by file in JSON
kubectl
apply
edit-
last
-applied -f deploy.yaml -o json
Edit the latest last-applied-configuration annotations of resources from the default editor.
The edit-last-applied command allows you to directly edit any API resource you can retrieve via the command-line tools. It will open the editor defined by your KUBE_EDITOR, or EDITOR environment variables, or fall back to 'vi' for Linux or 'notepad' for Windows. You can edit multiple objects, although changes are applied one at a time. The command accepts file names as well as command-line arguments, although the files you point to must be previously saved versions of resources.
The default format is YAML. To edit in JSON, specify "-o json".
The flag --windows-line-endings can be used to force Windows line endings, otherwise the default for your operating system will be used.
In the event an error occurs while updating, a temporary file will be created on disk that contains your unapplied changes. The most common error when updating a resource is another editor changing the resource on the server. When this occurs, you will have to apply your changes to the newer version of the resource, or update your temporary saved copy to include the latest resource version.
Usage
$ kubectl apply edit-last-applied (RESOURCE/NAME | -f FILENAME)
Flags
Name
Shorthand
Default
Usage
true
filename
f
[]
Filename, directory, or URL to files to use to edit the resource
kustomize
k
output
o
record
false
recursive
R
false
false
template
[
]
.
validate
strict
"warn" will warn about unknown or duplicate fields without

 blocking the request if server-side field validation is enabled on the API server, and behave as "ignore" otherwise.
false
Defaults to the line ending native to your platform.
Set the last-applied-configuration of a resource to match the contents of a file
kubectl
apply
-f
deploy.
yaml
Execute set-last-applied against each configuration file in a directory
kubectl
apply
-f
path/
Set the last-applied-configuration of a resource to match the contents of a file; will create the annotation if it does not already exist
kubectl
apply
-f
deploy.
yaml
--create-annotation=true
Set the latest last-applied-configuration annotations by setting it to match the contents of a file. This results in the last-applied-configuration being updated as though 'kubectl apply -f
' was run, without updating any other parts of the object.
Usage
$ kubectl apply set-last-applied -f FILENAME
Flags
Name
Shorthand
Default
Usage
true
create-annotation
false
Will create 'last-applied-configuration' annotations if current objects doesn't have one
dry-run
none
filename
f
[]
Filename, directory, or URL to files that contains the last-applied-configuration annotations
output
o
false
template
[
]
.
View the last-applied-configuration annotations by type/name in YAML
kubectl
apply
view
-
last
View the last-applied-configuration annotations by file in JSON
kubectl
apply
view
-
last
View the latest last-applied-configuration annotations by type/name or file.
The default output will be printed to stdout in YAML format. You can use the -o option to change the output format.
Usage
$ kubectl apply view-last-applied (TYPE [NAME | -l label] | TYPE/NAME | -f FILENAME)
Flags
Name
Shorthand
Default
Usage
all
false
Select all resources in the namespace of the specified resource types
filename
f
[]
kustomize
k
output
o
yaml
Output format. Must be one of (yaml, json)
recursive
R
false
selector
l
annotate
Update pod 'foo' with the annotation 'description' and the value 'my frontend' # If the same annotation is set multiple times, only the last value will be applied
kubectl annotate pods foo description
=
'my frontend'
Update a pod identified by type and name in "pod.json"
kubectl annotate -f pod.json
=
Update pod 'foo' with the annotation 'description' and the value 'my frontend running nginx', overwriting any existing value
kubectl
annotate
--overwrite
pods
foo
description=
Update all pods in t

he namespace
kubectl
annotate
pods
--all
Update pod 'foo' only if the resource is unchanged from version 1
kubectl annotate pods foo
=
--resource-version
=1
Update pod 'foo' by removing an annotation named 'description' if it exists # Does not require the --overwrite flag
-
Update the annotations on one or more resources.
All Kubernetes objects support the ability to store additional data with the object as annotations. Annotations are key/value pairs that can be larger than labels and include arbitrary string values such as structured JSON. Tools and system extensions may use annotations to store their own data.
Attempting to set an annotation that already exists will fail unless --overwrite is set. If --resource-version is specified and does not match the current resource version on the server the command will fail.
Usage
$ kubectl annotate [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--resource-version=version]
Flags
Name
Shorthand
Default
Usage
all
false
Select all resources, in the namespace of the specified resource types.
A
false
If true, check the specified action in all namespaces.
true
dry-run
none
kubectl-annotate
filename
f
[]
Filename, directory, or URL to files identifying the resource to update the annotation
kustomize
k
list
false
If true, display the annotations for a given resource.
local
false
If true, annotation will NOT contact api-server but run locally.
output
o
overwrite
false
If true, allow annotations to be overwritten, otherwise reject annotation updates that overwrite existing annotations.
record
false
recursive
R
false
resource-version
If non-empty, the annotation update will only succeed if this is the current resource-version for the object. Only valid when specifying a single resource.
selector
l
false
template
[
]
.
autoscale
Auto scale a deployment "foo", with the number of pods between 2 and 10, no target CPU utilization specified so a default autoscaling policy will be used
kubectl autoscale deployment foo
--min
=2
--max
=10
Auto scale a replication controller "foo", with the number of pods between 1 and 5, target

 CPU utilization at 80%
kubectl autoscale rc foo
--max
=5
--cpu-percent
=80
Creates an autoscaler that automatically chooses and sets the number of pods that run in a Kubernetes cluster.
Looks up a deployment, replica set, stateful set, or replication controller by name and creates an autoscaler that uses the given resource as a reference. An autoscaler can automatically increase or decrease number of pods deployed within the system as needed.
Usage
$ kubectl autoscale (-f FILENAME | TYPE NAME | TYPE/NAME) [--min=MINPODS] --max=MAXPODS [--cpu-percent=CPU]
Flags
Name
Shorthand
Default
Usage
true
cpu-percent
-1
The target average CPU utilization (represented as a percent of requested CPU) over all the pods. If it's not specified or negative, a default autoscaling policy will be used.
dry-run
none
kubectl-autoscale
filename
f
[]
Filename, directory, or URL to files identifying the resource to autoscale.
kustomize
k
max
-1
The upper limit for the number of pods that can be set by the autoscaler. Required.
min
-1
The lower limit for the number of pods that can be set by the autoscaler. If it's not specified or negative, the server will apply a default value.
name
The name for the newly created object. If not specified, the name of the input resource will be used.
output
o
record
false
recursive
R
false
false
false
template
[
]
.
debug
Create an interactive debugging session in pod mypod and immediately attach to it. # (requires the EphemeralContainers feature to be enabled in the cluster)
kubectl
debug
mypod -it
--image
=busybox
Create a debug container named debugger using a custom automated debugging image. # (requires the EphemeralContainers feature to be enabled in the cluster)
kubectl
debug
--image
=myproj/debug-tools -c debugger mypod
Create a copy of mypod adding a debug container and attach to it
kubectl
debug
mypod -it
--image
=busybox
--copy-to
=my-debugger
Create a copy of mypod changing the command of mycontainer
kubectl
debug
mypod -it
--copy-to
=mycontainer -- sh
Create a copy of mypod changing all container images to busybox
kubectl
debug
mypod
--copy-to
=*=busybox
Create a copy of mypod adding a debug container and changing container images
kubectl
debug
mypod -it
--copy-to
--image
=debian
=app=app:debug,sidecar=sidecar:debug
Create an interactive debugging session on a node and immediately attach to it. # The container will run in t

he host namespaces and the host's filesystem will be mounted at /host
node
/mynode
-it --
image=
busybox
Debug cluster resources using interactive debugging containers.
'debug' provides automation for common debugging tasks for cluster objects identified by resource and name. Pods will be used by default if no resource is specified.
The action taken by 'debug' varies depending on what resource is specified. Supported actions include:
Workload: Create a copy of an existing pod with certain attributes changed, for example changing the image tag to a new version.
Workload: Add an ephemeral container to an already running pod, for example to add debugging utilities without restarting the pod.
*  Node: Create a new pod that runs in the node's host namespaces and can access the node's filesystem.
Usage
$ kubectl debug (POD | TYPE[[.VERSION].GROUP]/NAME) [ -- COMMAND [args...] ]
Flags
Name
Shorthand
Default
Usage
arguments-only
false
If specified, everything after -- will be passed to the new container as Args instead of Command.
attach
false
If true, wait for the container to start running, and then attach as if 'kubectl attach ...' were called.  Default false, unless '-i/--stdin' is set, in which case the default is true.
container
c
Container name to use for debug container.
copy-to
Create a copy of the target Pod with this name.
env
[]
image
Container image to use for debug container.
The image pull policy for the container. If left empty, this value will not be specified by the client and defaulted by the server.
quiet
q
false
If true, suppress informational messages.
replace
false
When used with '--copy-to', delete the original Pod.
same-node
false
When used with '--copy-to', schedule the copy of target Pod on the same node.
set-image
[]
When used with '--copy-to', a list of name=image pairs for changing container images, similar to how 'kubectl set image' works.
share-processes
true
When used with '--copy-to', enable process namespace sharing in the copy.
stdin
i
false
Keep stdin open on the container(s) in the pod, even if nothing is attached.
target
When using an ephemeral container, target processes in this container name.
tty
t
false
Allocate a TTY for the debugging container.
diff
Diff resources included in pod.json
kubectl
diff
-f
pod
.json
Diff file read from stdin
cat service.yaml
| kubectl diff -f -
Diff configurations specified by file name or stdin between the current online configuration, and the configuration as it would be if applied.
The output is always YAML.
KUBECTL_EXTERNAL_DIFF environment variable can be used to select your own diff command. Users can use external commands with params too, example: KUBECTL_EXTERNAL_DIFF="colordiff -N -u"
By default, the "diff" command available in your path will be run with the "-u" (unified diff) and "-N" (treat absent files as empty) options.
Exit status: 0 No differences were found. 1 Differences were found. &gt;1 Kubectl or diff failed with an error.
Note: KUBECTL_EXTERNAL_DIFF, if used, is expected to follow that convention.
Usage
$ kubectl diff -f FILENAME
Flags
Name
Shorthand
Default
Usage
filename
f
[]
Filename, directory, or URL to files contains the configuration to diff
false
kustomize
k
prune
false
Include resources that would be deleted by pruning. Can be used with -l and default shows all resources would be pruned
prune-allowlist
[]
recursive
R
false
selector
l
Selector (label query) to filter on, supports '=',

 '==', and '!='.(e.g. -l key1=value1,key2=value2). Matching objects must satisfy all of the specified label constraints.
false
false
If true, include managed fields in the diff.
edit
Edit the service named 'registry'
kubectl
edit
svc/registry
Use an alternative editor
=
"nano"
kubectl
edit
Edit the job 'myjob' in JSON using the v1 API format
kubectl
edit
job.v1.batch/myjob -o json
Edit the deployment 'mydeployment' in YAML and save the modified config in its annotation
kubectl
edit
deployment/mydeployment -o yaml --
save
-config
Edit the deployment/mydeployment's status subresource
kubectl
edit
deployment mydeployment
=
'status'
Edit a resource from the default editor.
The edit command allows you to directly edit any API resource you can retrieve via the command-line tools. It will open the editor defined by your KUBE_EDITOR, or EDITOR environment variables, or fall back to 'vi' for Linux or 'notepad' for Windows. You can edit multiple objects, although changes are applied one at a time. The command accepts file names as well as command-line arguments, although the files you point to must be previously saved versions of resources.
Editing is done with the API version used to fetch the resource. To edit using a specific API version, fully-qualify the resource, version, and group.
Usage
$ kubectl edit (RESOURCE/NAME | -f FILENAME)
Flags
Name
Shorthand
Default
Usage
true
kubectl-edit
filename
f
[]
kustomize
k
output
o
output-patch
false
Output the patch if the resource is edited.
record
false
recursive
R
false
false
false
If specified, edit will operate on the subresource of the requested object. Must be one of
[
status
]
template
[
]
.
validate
strict
"true" or "strict" will use a schema to validate the input and fail the re

quest if invalid. It will perform server side validation if ServerSideFieldValidation is enabled on the api-server, but will fall back to less reliable client-side validation if not.
false
kustomize
Build the current working directory
Build some shared configuration directory
kubectl kustomize /
home
/
config
/production
Build from github
kubectl kustomize https:
//gi
thub.com
/kubernetes-sigs/
kustomize.git
/examples/
helloWorld?ref=v1.
0.6
Build a set of KRM resources using a 'kustomization.yaml' file. The DIR argument must be a path to a directory containing 'kustomization.yaml', or a git repository URL with a path suffix specifying same with respect to the repository root. If DIR is omitted, '.' is assumed.
Usage
$ kubectl kustomize DIR
Flags
Name
Shorthand
Default
Usage
as-current-user
false
use the uid and gid of the command executor to run the function in the container
enable-alpha-plugins
false
enable kustomize plugins
enable-helm
false
Enable use of the Helm chart inflator generator.
enable-managedby-label
false
enable adding app.kubernetes.io/managed-by
env
e
[]
a list of environment variables to be used by functions
helm-command
helm
helm command (path to executable)
load-restrictor
LoadRestrictionsRootOnly
if set to 'LoadRestrictionsNone', local kustomizations may load files from outside their root. This does, however, break the relocatability of the kustomization.
mount
[]
a list of storage options read from the filesystem
network
false
enable network access for functions that declare it
network-name
bridge
the docker network to run the container in
output
o
If specified, write output to this path.
reorder
legacy
Reorder the resources just before output. Use 'legacy' to apply a legacy reordering (Namespaces first, Webhooks last, etc). Use 'none' to suppress a final reordering.
label
Update pod 'foo' with the label 'unhealthy' and the value 'true'
kubectl
label
pods foo unhealthy=
true
Update pod 'foo' with the label 'status' and the value 'unhealthy', overwriting any existing value
kubectl
label
--overwrite pods foo status=unhealthy
Update all pods in the namespace
kubectl
label
pods --all status=unhealthy
Update a pod identified by the type and name in "pod.json"
kubectl
label
-f pod.json status=unhealthy
kubectl label pods foo
status
=unhealthy
=1
Update pod 'foo' by removing a label named 'bar' if it exists # Does not require the --overwrite flag
kubectl
label
pods foo bar-
Update the labels on a resource.
A label key and value must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to  63 characters each.
Optionally, the key can begin with a DNS subdomain prefix and a single '/', like example.com/my-app.
If --overwrite is true, then existing labels can be overwritten, otherwise attempting to overwrite a label will result in an error.
If --resource-version is specified, then updates will use this resource version, otherwise the existing resource-version will be used.
Usage
$ kubectl label [--overwrite] (-f FILENAME | TYPE NAME) KEY_1=VAL_1 ... KEY_N=VAL_N [--resource-version=version]
Flags
Name
Shorthand
Default
Usage
all
false
Select all resources, in the namespace of the specified resource types
A
false
true
dry-run
none
Must be "none", "server", or "client". If client

 strategy, only print the object that would be sent, without sending it. If server strategy, submit server-side request without persisting the resource.
kubectl-label
filename
f
[]
Filename, directory, or URL to files identifying the resource to update the labels
kustomize
k
list
false
If true, display the labels for a given resource.
local
false
If true, label will NOT contact api-server but run locally.
output
o
overwrite
false
If true, allow labels to be overwritten, otherwise reject label updates that overwrite existing labels.
record
false
recursive
R
false
If non-empty, the labels update will only succeed if this is the current resource-version for the object. Only valid when specifying a single resource.
selector
l
false
template
[
]
.
patch
Partially update a node using a strategic merge patch, specifying the patch as JSON
node
k8s-node-1
-p '{
"spec"
:{
"unschedulable"
:
true
}}'
Partially update a node using a strategic merge patch, specifying the patch as YAML
node
-p $'
spec
:\n unschedulable:
true
'
Partially update a node identified by the type and name specified in "node.json" using strategic merge patch
kubectl patch -f
node
.
json
-p '{
"spec"
:{
:
true
}}'
Update a container's image; spec.containers[*].name is required because it's a merge key
kubectl
patch
pod valid-pod -p '{
"spec"
:{
:[{
"name"
:
"kubernetes-serve-hostname"
,
"image"
:
"new image"
}]}}'
Update a container's image using a JSON patch with positional arrays
kubectl patch pod valid-pod --
type
=
'jso
n' -p='[{
"op"
:
"replace"
,
"path"
:
"/spec/containers/0/image"
,
"value"
:
}]'
Update a deployment's replicas through the scale subresource using a merge patch.
kubectl patch deployment nginx-deployment --subresource=
'scal
e' --
type
=
'merg
e' -p '{
"spec"
:{
"replicas"
:
2
}}'
Update fields of a resource using strategic merge patch, a JSON merge patch, or a JSON patch.
Usage
$ kubectl patch (-f FILENAME | TYPE NAME) [-p PATCH|--patch-file FILE]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
kubectl-patch
filename
f
[]
Filename, directory, or URL to files identifying the resource to update
k

ustomize
k
local
false
If true, patch will operate on the content of the file, not the server-side resource.
output
o
patch
p
The patch to be applied to the resource JSON file.
patch-file
A file containing a patch to be applied to the resource.
record
false
recursive
R
false
false
If specified, patch will operate on the subresource of the requested object. Must be one of
[
]
template
[
]
.
type
strategic
The type of patch being provided; one of
[
json merge strategic
]
replace
Replace a pod using the data in pod.json
kubectl
replace
Replace a pod based on the JSON passed into stdin
kubectl
replace
-f -
Update a single-container pod's image version (tag) to v4
kubectl get pod mypod -o yaml |
sed
's/\(image: myimage\):.*$/\
1
:v4/' |
kubectl
replace
-f -
Force replace, delete and then re-create the resource
--force
-f
./pod.json
Replace a resource by file name or stdin.
JSON and YAML formats are accepted. If replacing an existing resource, the complete resource spec must be provided. This can be obtained by
$ kubectl get TYPE NAME -o yaml
Usage
$ kubectl replace -f FILENAME
Flags
Name
Shorthand
Default
Usage
true
cascade
dry-run
none
kubectl-replace
filename
f
[]
The files that contain the configurations to replace.
force
false
-1
kustomize
k
output
o
raw
Raw URI to PUT to the server.  Uses the transport specified by the kubeconfig file.
recursive
R
false
false
show-ma

naged-fields
false
If specified, replace will operate on the subresource of the requested object. Must be one of
[
]
template
[
]
.
timeout
0s
validate
strict
wait
false
rollout
Rollback to the previous deployment
undo
deployment/
abc
Check the rollout status of a daemonset
status
daemonset/foo
Restart a deployment
restart
deployment/abc
Restart deployments with the app=nginx label
restart
--selector=app=nginx
Manage the rollout of one or many resources.
Valid resource types include:
*  statefulsets
Usage
$ kubectl rollout SUBCOMMAND
history
View the rollout history of a deployment
history
abc
View the details of daemonset revision 3
history
daemonset/abc
--revision=3
View previous rollout revisions and configurations.
Usage
$ kubectl rollout history (TYPE NAME | TYPE/NAME) [flags]
Flags
Name
Shorthand
Default
Usage
true
filename
f
[]
kustomize
k
output
o
recursive
R
false
revision
0
See the details, including podTemplate of the revision specified
selector
l
false
template
[
]
.
pause
Mark the nginx deployment as paused # Any current state of the deployment will continue its function; new updates # to the deployment will not have an effect as long as the deployment is paused
pause
deployment/nginx
Mark the provided resource as paused.
Paused resources will not be reconciled by a controller. Use "kubectl rollout resume" to resume a paused resource. Currently only deployments support being paused.
Usage
$ kubectl rollout pause RESOURCE
Flags
Name
Shorthand
Default
Usage
true
If true, ignore any errors in templates when a field or map key is missing in the template. Only applies

 to golang and jsonpath output formats.
kubectl-rollout
filename
f
[]
kustomize
k
output
o
recursive
R
false
selector
l
false
template
[
]
.
restart
restart
Restart a daemon set
restart
restart
Restart a resource.
Resource rollout will be restarted.
Usage
$ kubectl rollout restart RESOURCE
Flags
Name
Shorthand
Default
Usage
true
filename
f
[]
kustomize
k
output
o
recursive
R
false
selector
l
false
template
[
]
.
resume
Resume an already paused deployment
resume
Resume a paused resource.
Paused resources will not be reconciled by a controller. By resuming a resource, we allow it to be reconciled again. Currently only deployments support being resumed.
Usage
$ kubectl rollout resume RESOURCE
Flags
Name
Shorthand
Default
Usage
true
filename
f
[]
kustomize
k
output
o
recursive
R
false
selector
l
Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2). Matching objects must satisfy all of the specified labe

l constraints.
false
template
[
]
.
status
Watch the rollout status of a deployment
status
Show the status of the rollout.
By default 'rollout status' will watch the status of the latest rollout until it's done. If you don't want to wait for the rollout to finish then you can use --watch=false. Note that if a new rollout starts in-between, then 'rollout status' will continue watching the latest revision. If you want to pin to a specific revision and abort if it is rolled over by another revision, use --revision=N where N is the revision you need to watch for.
Usage
$ kubectl rollout status (TYPE NAME | TYPE/NAME) [flags]
Flags
Name
Shorthand
Default
Usage
filename
f
[]
kustomize
k
recursive
R
false
revision
0
Pin to a specific revision for showing its status. Defaults to 0 (last revision).
selector
l
timeout
0s
The length of time to wait before ending watch, zero means never. Any other values should contain a corresponding time unit (e.g. 1s, 2m, 3h).
watch
w
true
Watch the status of the rollout until it's done.
undo
Roll back to the previous deployment
undo
abc
Roll back to daemonset revision 3
undo
daemonset/
abc
--
to
-revision=
3
Roll back to the previous deployment with dry-run
kubectl rollout undo --dry-
run
=server deployment/abc
Roll back to a previous rollout.
Usage
$ kubectl rollout undo (TYPE NAME | TYPE/NAME) [flags]
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
filename
f
[]
kustomize
k
output
o
recursive
R
false
selector
l
false
template
[
]
.
to-revision
0
The revision to rollback to. Default to 0 (last revision).
scale
Scale a replica set named 'foo' to 3
=3 rs/foo
Scale a resource identified by type and name specified in "foo.yaml" to 3
=3 -f foo.yaml
If the deployment named mysql's current size is 2, scale mysql to 3
--current-replicas
=2
=3 deployment/my

sql
Scale multiple replication controllers
kubectl
scale
--replicas=
5
rc/foo rc/
bar
rc/baz
Scale stateful set named 'web' to 3
=3 statefulset/web
Set a new size for a deployment, replica set, replication controller, or stateful set.
Scale also allows users to specify one or more preconditions for the scale action.
If --current-replicas or --resource-version is specified, it is validated before the scale is attempted, and it is guaranteed that the precondition holds true when the scale is sent to the server.
Usage
$ kubectl scale [--resource-version=version] [--current-replicas=count] --replicas=COUNT (-f FILENAME | TYPE NAME)
Flags
Name
Shorthand
Default
Usage
all
false
true
current-replicas
-1
Precondition for current size. Requires that the current size of the resource match this value in order to scale. -1 (default) for no condition.
dry-run
none
filename
f
[]
Filename, directory, or URL to files identifying the resource to set a new size
kustomize
k
output
o
record
false
recursive
R
false
replicas
0
The new desired number of replicas. Required.
Precondition for resource version. Requires that the current resource version match this value in order to scale.
selector
l
false
template
[
]
.
timeout
0s
The length of time to wait before giving up on a scale operation, zero means don't wait. Any other values should contain a corresponding time unit (e.g. 1s, 2m, 3h).
set
These commands help you make changes to existing application resources.
Usage
$ kubectl set SUBCOMMAND
env
Update deployment 'registry' with a new environment variable
kubectl
set
env deployment/registry
STORAGE_DIR
=/local
List the environment variables defined on a deployments 'sample-build'
kubectl
set
env deployment/
sample
-
build
--list
List the environment variables defined on all pods
kubectl
set
env pods
--all
--list
Output modified deployment in YAML, and does not alter the object on the server
kubectl
set
env deployment/sample-build
=/data -o yaml
Update all containers in all replication controllers in the project to have ENV=prod
kubectl
set
env rc --all
ENV
=prod
Import environment from a secret
kubectl
set
env
--from
=secret/mysecret deployment/myapp
Import environment from a config map with a prefix
kubectl
set
env
--from
=configmap/myconfigmap
--prefix
=MYSQL_ deployment/myapp
Import specific keys from a config map
kubectl
set
env
--keys
=my-example-key
--from
=configmap/myconfigmap deployment/mya

pp
Remove the environment variable ENV from container 'c1' in all deployment configs
kubectl
set
env deployments
--all
--containers=
"c1"
ENV-
Remove the environment variable ENV from a deployment definition on disk and # update the deployment config on the server
env
-f deploy.json
ENV
-
Set some of the local shell environment into a deployment config on the server
env |
grep
RAILS_ |
kubectl
set
env -e - deployment/registry
Update environment variables on a pod template.
List environment variable definitions in one or more pods, pod templates. Add, update, or remove container environment variable definitions in one or more pod templates (within replication controllers or deployment configurations). View or modify the environment variable definitions on all containers in the specified pods or pod templates, or just those that match a wildcard.
If "--env -" is passed, environment variables can be read from STDIN using the standard env syntax.
pod (po), replicationcontroller (rc), deployment (deploy), daemonset (ds), statefulset (sts), cronjob (cj), replicaset (rs)
Usage
$ kubectl set env RESOURCE/NAME KEY_1=VAL_1 ... KEY_N=VAL_N
Flags
Name
Shorthand
Default
Usage
all
false
If true, select all resources in the namespace of the specified resource types
true
c
*
The names of containers in the selected pod templates to change - may use wildcards
dry-run
none
env
e
[]
Specify a key-value pair for an environment variable to set into each container.
kubectl-set
filename
f
[]
Filename, directory, or URL to files the resource to update the env
from
The name of a resource from which to inject environment variables
keys
[]
Comma-separated list of keys to import from specified resource
kustomize
k
list
false
If true, display the environment and any changes in the standard format. this flag will removed when we have kubectl view env.
local
false
If true, set env will NOT contact api-server but run locally.
output
o
overwrite
true
If true, allow environment to be overwritten, otherwise reject updates that overwrite existing environment.
prefix
Prefix to append to variable names
recursive
R
false
resolve
false
If true, show secret or configmap references when listing variables
selector
l
false
template
[
]
.
image
Set a deployment's nginx container image to 'nginx:1.9.1', and its busybox container image to 'busybox'
kubectl
set
image deployment/nginx
busybox
=busybox
nginx
=nginx:1.9.1
Update all deployments' and rc's nginx container's image to 'nginx:1.9.1'
kubectl
set
image deployments,rc
nginx
=nginx:1.9.1 --all
Update image of all containers of daemonset abc to 'nginx:1.9.1'
kubectl
set
image daemonset abc *=nginx:
1.9
.1
Print result (

in yaml format) of updating nginx container image from local file, without hitting the server
kubectl
set
image -f path/
to
/file.yaml
nginx
=nginx:1.9.1 --local -o yaml
Update existing container image(s) of resources.
Usage
$ kubectl set image (-f FILENAME | TYPE NAME) CONTAINER_NAME_1=CONTAINER_IMAGE_1 ... CONTAINER_NAME_N=CONTAINER_IMAGE_N
Flags
Name
Shorthand
Default
Usage
all
false
true
dry-run
none
filename
f
[]
kustomize
k
local
false
If true, set image will NOT contact api-server but run locally.
output
o
record
false
recursive
R
false
selector
l
false
template
[
]
.
resources
Set a deployments nginx container cpu limits to "200m" and memory to "512Mi"
kubectl
set
resources deployment nginx
-c
=nginx
--limits
=cpu=200m,memory=512Mi
Set the resource request and limits for all containers in nginx
kubectl
set
--limits
--requests
=cpu=100m,memory=256Mi
Remove the resource requests for resources on containers in nginx
kubectl
set
--limits
=cpu=0,memory=0
Print the result (in yaml format) of updating nginx container limits from a local, without hitting the server
kubectl
set
resources -f path/
to
--limits
=cpu=200m,memory=512Mi --local -o yaml
Specify compute resource requirements (CPU, memory) for any resource that defines a pod template.  If a pod is successfully scheduled, it is guaranteed the amount of resource requested, but may burst up to its specified limits.
For each compute resource, if a limit is specified and a request is omitted, the request will default to the limit.
Possible resources include (case insensitive): Use "kubectl api-resources" for a complete list of supported resources..
Usage
$ kubectl set resources (-f FILENAME | TYPE NAME)  ([--limits=LIMITS &amp; --requests=REQUESTS]
Flags
Name
Shorthand
Default
Usage
all
false
true
c
*
The names of containers

 in the selected pod templates to change, all containers are selected by default - may use wildcards
dry-run
none
filename
f
[]
kustomize
k
limits
The resource requirement requests for this container.  For example, 'cpu=100m,memory=256Mi'.  Note that server side components may assign requests depending on the server configuration, such as limit ranges.
local
false
If true, set resources will NOT contact api-server but run locally.
output
o
record
false
recursive
R
false
requests
selector
l
false
template
[
]
.
selector
Set the labels and selector before creating a deployment/service pair
service
clusterip my-svc
=
"None"
-o yaml
--dry-run
=client | kubectl
set
kubectl create deployment my-dep -o yaml
--dry-run
=client | kubectl label --local -f -
=qa -o yaml | kubectl create -f -
Set the selector on a resource. Note that the new selector will overwrite the old selector if the resource had one prior to the invocation of 'set selector'.
A selector must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to  63 characters. If --resource-version is specified, then updates will use this resource version, otherwise the existing resource-version will be used. Note: currently selectors can only be set on Service objects.
Usage
$ kubectl set selector (-f FILENAME | TYPE NAME) EXPRESSIONS [--resource-version=version]
Flags
Name
Shorthand
Default
Usage
all
false
true
dry-run
none
filename
f
[]
identifying the resource.
local
false
output
o
record
false
Record current kubectl command in 

the resource annotation. If set to false, do not record the command. If set to true, record the command. If not set, default to updating the existing annotation value only if one already exists.
recursive
R
true
If non-empty, the selectors update will only succeed if this is the current resource-version for the object. Only valid when specifying a single resource.
false
template
[
]
.
Set deployment nginx-deployment's service account to serviceaccount1
kubectl
set
deployment nginx-deployment serviceaccount1
Print the result (in YAML format) of updated nginx deployment with the service account from local file, without hitting the API server
kubectl
set
sa -f nginx-deployment.yaml serviceaccount1 --local
--dry-run
=client -o yaml
Update the service account of pod template resources.
Possible resources (case insensitive) can be:
replicationcontroller (rc), deployment (deploy), daemonset (ds), job, replicaset (rs), statefulset
Usage
$ kubectl set serviceaccount (-f FILENAME | TYPE NAME) SERVICE_ACCOUNT
Flags
Name
Shorthand
Default
Usage
all
false
true
dry-run
none
filename
f
[]
kustomize
k
local
false
If true, set serviceaccount will NOT contact api-server but run locally.
output
o
record
false
recursive
R
false
false
template
[
]
.
subject
Update a cluster role binding for serviceaccount1
kubectl
set
subject clusterrolebinding admin
=namespace:serviceaccount1
Update a role binding for user1, user2, and group1
kubectl
set
subject rolebinding admin
--user
=user1
--user
=user2
--group
=group1
Print the result (in YAML format) of updating rolebinding subjects from a local, without hitting the server
--role
=admin
--user
=admin -o yaml
--dry-run
set
subject --local -f -
--user
=foo -o yaml
Update the user, group, or service account in a role binding or cluster role binding.
Usage
$ kubectl set subject (-f FILENAME | TYPE NAME) [--user=username] [--group=groupname] [--serviceaccount=namespace:serviceaccountname] [--dry-run=server|client|none]
Flags
Name
Sh

orthand
Default
Usage
all
false
true
dry-run
none
filename
f
[]
Filename, directory, or URL to files the resource to update the subjects
group
[]
Groups to bind to the role
kustomize
k
local
false
If true, set subject will NOT contact api-server but run locally.
output
o
recursive
R
false
selector
l
[]
Service accounts to bind to the role
false
template
[
]
.
wait
Wait for the pod "busybox1" to contain the status condition of type "Ready"
kubectl
wait
--for=condition=Ready pod/busybox1
The default value of status condition is true; you can wait for other targets after an equal delimiter (compared after Unicode simple case folding, which is a more general form of case-insensitivity):
kubectl
wait
--
for
=condition=Ready=
false
pod/busybox1
Wait for the pod "busybox1" to contain the status phase to be "Running".
kubectl
wait
--
for
=jsonpath=
'{.status.phase}'
=Running pod/busybox1
Wait for the pod "busybox1" to be deleted, with a timeout of 60s, after having issued the "delete" command
kubectl delete pod/busybox1
--for
=delete pod/busybox1
--timeout
=60s
The command takes multiple resources and waits until the specified condition is seen in the Status field of every given resource.
Alternatively, the command can wait for the given set of resources to be deleted by providing the "delete" keyword as the value to the --for flag.
A successful message will be printed to stdout indicating when the specified condition has been met. You can use -o option to change to output destination.
Usage
$ kubectl wait ([-f FILENAME] | resource.group/resource.name | resource.group [(-l label | --all)]) [--for=delete|--for condition=available|--for=jsonpath='{}'=value]
Flags
Name
Shorthand
Default
Usage
all
false
A
false
true
filename
f
[]
for
The condition to wait on:
[
delete|condition=condition-name
[
=condition-value
]
|jsonpath='{JSONPath expression}'=JSONPath Condition
]
. The default condition-value is true.  Condition va

lues are compared after Unicode simple case folding, which is a more general form of case-insensitivity.
local
false
output
o
recursive
R
true
selector
l
Selector (label query) to filter on, supports '=', '==', and '!='.(e.g. -l key1=value1,key2=value2)
false
template
[
]
.
timeout
30s
The length of time to wait before giving up.  Zero means check once and don't wait, negative means wait for a week.
This section contains commands for inspecting and debugging your
applications.
logs
will print the logs from the specified pod + container.
exec
can be used to get an interactive shell on a pod + container.
describe
will print debug information about the given resource.
attach
Get output from running pod mypod; use the 'kubectl.kubernetes.io/default-container' annotation # for selecting the container to be attached or the first container in the pod will be chosen
kubectl
attach
mypod
Get output from ruby-container from pod mypod
kubectl
attach
mypod -c ruby-container
Switch to raw terminal mode; sends stdin to 'bash' in ruby-container from pod mypod # and sends stdout/stderr from 'bash' back to the client
kubectl
attach
mypod -c ruby-container -i -t
Get output from the first pod of a replica set named nginx
kubectl
attach
rs/nginx
Attach to a process that is already running inside an existing container.
Usage
$ kubectl attach (POD | TYPE/NAME) -c CONTAINER
Flags
Name
Shorthand
Default
Usage
container
c
Container name. If omitted, use the kubectl.kubernetes.io/default-container annotation for selecting the container to be attached or the first container in the pod will be chosen
1m0s
quiet
q
false
Only print output from the remote session
stdin
i
false
Pass stdin to the container
tty
t
false
Stdin is a TTY
auth
Inspect authorization
Usage
$ kubectl auth
can-i
Check to see if I can create pods in any namespace
kubectl auth can-i
create
pods
Check to see if I can list deployments in my current namespace
list
deployments.apps
Check to see if I can do everything in my current namespace ("*" means all)
kubectl auth can-
i
'*'
'*'
Check to see if I can get the job named "bar" in namespace "foo"
list
jobs.batch/bar -n foo
Check to see if I can read pod logs
get
pods
=log
Check to see if I can access the URL /logs/
get
/logs/
List all allowed actions in namespace "foo"
kubectl auth can-i --list --
namespace
=
foo
Check whether an action is allowed.
VERB is a logical Kubernetes API verb like 'get', 'list', 'watch', 'delete', etc. TYPE is a Kubernetes resource. Shortcuts and groups will be resolved. NONRESOURCEURL is a partial URL that starts with "/". NAME is the name of a particular Kubernetes resource. This command pairs nicely with impersonation. See --as global flag.
Usage
$ kubectl auth can-i VERB [TYPE | TYPE/NAME | NONRESOURCEURL]
Flags
Name
Shorthand
Default
Usage
A
false
list
false
If true, prints all allowed actions.
false
If true, prints allowed actions without headers
quiet
q
false
If true, suppress output and just return the exit code.
SubResource such as pod/log or deployment/scal

e
reconcile
Reconcile RBAC resources from a file
kubectl auth reconcile -f
my
-rbac-rules.yaml
Reconciles rules for RBAC role, role binding, cluster role, and cluster role binding objects.
Existing roles are updated to include the permissions in the input objects, and remove extra permissions if --remove-extra-permissions is specified.
Existing bindings are updated to include the subjects in the input objects, and remove extra subjects if --remove-extra-subjects is specified.
This is preferred to 'apply' for RBAC resources so that semantically-aware merging of rules and subjects is done.
Usage
$ kubectl auth reconcile -f FILENAME
Flags
Name
Shorthand
Default
Usage
true
dry-run
none
filename
f
[]
Filename, directory, or URL to files identifying the resource to reconcile.
kustomize
k
output
o
recursive
R
false
remove-extra-permissions
false
If true, removes extra permissions added to roles
remove-extra-subjects
false
If true, removes extra subjects added to rolebindings
false
template
[
]
.
cp
!!!Important Note!!! # Requires that the 'tar' binary is present in your container # image.  If 'tar' is not present, 'kubectl cp' will fail. # # For advanced use cases, such as symlinks, wildcard expansion or # file mode preservation, consider using 'kubectl exec'. # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace
tar cf -
/tmp/
foo | kubectl exec -i -n
&lt;some-namespace&gt;
&lt;some-pod&gt;
-- tar xf - -C
/tmp/
bar
Copy /tmp/foo from a remote pod to /tmp/bar locally
kubectl exec -n
-- tar cf -
/tmp/
foo | tar xf - -C
/tmp/
bar
Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the default namespace
kubectl cp /tmp/foo_dir &lt;some-pod&gt;
:/tmp/bar_dir
Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container
kubectl cp
/tmp/
foo
:
/tmp/
bar -c
&lt;specific-container&gt;
Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace
kubectl cp
/tmp/
foo
/
:
/tmp/
bar
kubectl cp
/
:
/tmp/
foo
/tmp/
bar
Usage
$ kubectl cp &lt;file-spec-src&gt; &lt;file-spec-dest&gt;
Flags
Name
Shorthand
Default
Usage
container
c
no-preserve
false
The copied file/directory's ownership and permissions will not be preserved in the container
retries
0
Set number of retries to complete a copy operation from a container. Specify 0 to disable or any negative value for infinite retrying. The default is 0 (no retry).
describe
Describe a node
kubectl describe nodes kubernetes-
node
-emt8
.c.m

yproject.internal
Describe a pod
kubectl
describe
pods/nginx
Describe a pod identified by type and name in "pod.json"
kubectl
describe
-f
pod
.json
Describe all pods
Describe pods by label name=myLabel
kubectl describe po -l
name
=myLabel
Describe all pods managed by the 'frontend' replication controller # (rc-created pods get the name of the rc as a prefix in the pod name)
kubectl describe pods frontend
Show details of a specific resource or group of resources.
Print a detailed description of the selected resources, including related resources such as events or controllers. You may select a single object by name, all objects of that type, provide a name prefix, or label selector. For example:
$ kubectl describe TYPE NAME_PREFIX
will first check for an exact match on TYPE and NAME_PREFIX. If no such resource exists, it will output details for every resource that has a name prefixed with NAME_PREFIX.
Usage
$ kubectl describe (-f FILENAME | TYPE [NAME_PREFIX | -l label] | TYPE/NAME)
Flags
Name
Shorthand
Default
Usage
A
false
500
Return large lists in chunks rather than all at once. Pass 0 to disable. This flag is beta and may change in the future.
filename
f
[]
Filename, directory, or URL to files containing the resource to describe
kustomize
k
recursive
R
false
selector
l
show-events
true
If true, display events related to the described object.
exec
Get output from running the 'date' command from pod mypod, using the first container by default
kubectl
exec
mypod --
date
Get output from running the 'date' command in ruby-container from pod mypod
kubectl
exec
mypod -c ruby-
container
--
date
kubectl
exec
container
-i -t -- bash -il
List contents of /usr from the first container of pod mypod and sort by modification time # If the command you want to execute in the pod has any flags in common (e.g. -i), # you must use two dashes (--) to separate your command's flags/arguments # Also note, do not surround your command and its flags/arguments with quotes # unless that is how you would execute it normally (i.e., do ls -t /usr, not "ls -t /usr")
kubectl exec mypod -i -
t
-- ls -
t
/usr
Get output from running 'date' command from the first pod of the deployment mydeployment, using the first container by default
kubectl
exec
deploy/mydeployment --
date
Get output from running 'date' command from the first pod of the service myservice, using the first container by default
kubectl
exec
svc/myservice --
date
Execute a command in a container.
Usage
$ kubectl exec (POD | TYPE/NAME) [-c CONTAINER] [flags] -- COMMAND [args...]
Flags
Name
Shorthand
Default
Usage
container
c
filename
f
[]
to use to exec into the resource
1m0s
quiet
q
false
stdin
i
false
tty
t
false
logs
Return snapshot logs from pod nginx with only one container
kubectl logs nginx
Return snapshot logs f

rom pod nginx with multi containers
kubectl logs nginx --
all
-containers=
true
Return snapshot logs from all containers in pods defined by label app=nginx
app
=nginx
--all-containers
=
true
Return snapshot of previous terminated ruby container logs from pod web-1
kubectl logs -p -c ruby web
-1
Begin streaming the logs of the ruby container in pod web-1
kubectl logs -f -c ruby web
-1
Begin streaming the logs from all containers in pods defined by label app=nginx
app
=nginx
=
true
Display only the most recent 20 lines of output in pod nginx
--tail
=20 nginx
Show all logs from pod nginx written in the last hour
--since
=1h nginx
Show logs from a kubelet with an expired serving certificate
--insecure-skip-tls-verify-backend nginx
Return snapshot logs from first container of a job named hello
job
/hello
Return snapshot logs from container nginx-1 of a deployment named nginx
kubectl logs deployment/nginx -c nginx
-1
Print the logs for a container in a pod or specified resource. If the pod has only one container, the container name is optional.
Usage
$ kubectl logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER]
Flags
Name
Shorthand
Default
Usage
all-containers
false
Get all containers' logs in the pod(s).
container
c
Print the logs of this container
follow
f
false
Specify if the logs should be streamed.
ignore-errors
false
If watching / following pod logs, allow for any errors that occur to be non-fatal
insecure-skip-tls-verify-backend
false
Skip verifying the identity of the kubelet that logs are requested from.  In theory, an attacker could provide invalid log content back. You might want to use this if your kubelet serving certificates have expired.
limit-bytes
0
Maximum bytes of logs to return. Defaults to no limit.
max-log-requests
5
Specify maximum number of concurrent logs to follow when using by a selector. Defaults to 5.
20s
prefix
false
Prefix each log line with the log source (pod name and container name)
previous
p
false
If true, print the logs for the previous instance of the container in a pod if it exists.
selector
l
since
0s
Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of since-time / since may be used.
since-time
Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time / since may be used.
tail
-1
Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise 10, if a selector is provided.
false
Include timestamps on each line in the log output
Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod
kubectl port-forward pod/mypod
5000
6000
Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in a pod selected by the deployment
kubectl port-forward deployment/mydeployment
5000
6000
Listen on port 8443 locally, forwarding to the targetPort of the service's port named "https" in a pod selected by the service
kubectl port-forward service/myservice
8443
:https
Listen on port 8888 locally, forwarding to 5000 in the pod
8888
:
5000
Listen on port 8888 on all addresses, forwarding to 5000 in the pod
kubectl port-forward --address
0.0
.0
.0
pod/mypod
8888
:
5000
Listen on port 8888 on localhost and selected IP, forwarding to 5000 in the pod
kubectl port-forward --address localhost,
10.19
.21
.23
pod/mypod
8888
:
5000
Listen on a random port locally, forwarding to 5000 in the pod
kubectl port-forward pod/mypod :
5000
Use resource type/name such as depl

oyment/mydeployment to select a pod. Resource type defaults to 'pod' if omitted.
If there are multiple pods matching the criteria, a pod will be selected automatically. The forwarding session ends when the selected pod terminates, and a rerun of the command is needed to resume forwarding.
Usage
$ kubectl port-forward TYPE/NAME [options] [LOCAL_PORT:]REMOTE_PORT [...[LOCAL_PORT_N:]REMOTE_PORT_N]
Flags
Name
Shorthand
Default
Usage
address
[localhost]
Addresses to listen on (comma separated). Only accepts IP addresses or localhost as a value. When localhost is supplied, kubectl will try to bind on both 127.0.0.1 and ::1 and will fail if neither of these addresses are available to bind.
1m0s
proxy
To proxy all of the Kubernetes API and nothing else
kubectl
proxy
--api-prefix
=/
To proxy only part of the Kubernetes API and also some static files # You can get pods info with 'curl localhost:8001/api/v1/pods'
kubectl
proxy
--www
=/my/files
--www-prefix
=/static/
=/api/
To proxy the entire Kubernetes API at a different root # You can get pods info with 'curl localhost:8001/custom/api/v1/pods'
kubectl
proxy
=/custom/
Run a proxy to the Kubernetes API server on port 8011, serving static content from ./local/www/
kubectl
proxy
--port
=8011
--www
=./local/www/
Run a proxy to the Kubernetes API server on an arbitrary local port # The chosen port for the server will be output to stdout
kubectl
proxy
--port
=0
Run a proxy to the Kubernetes API server, changing the API prefix to k8s-api # This makes e.g. the pods API available at localhost:8001/k8s-api/v1/pods/
kubectl
proxy
=/k8s-api
Creates a proxy server or application-level gateway between localhost and the Kubernetes API server. It also allows serving static content over specified HTTP path. All incoming data enters through one port and gets forwarded to the remote Kubernetes API server port, except for the path matching the static content path.
Usage
$ kubectl proxy [--port=PORT] [--www=static-dir] [--www-prefix=prefix] [--api-prefix=prefix]
Flags
Name
Shorthand
Default
Usage
accept-hosts
^localhost$,^127.0.0.1$,^[::1]$
Regular expression for hosts that the proxy should accept.
accept-paths
^.*
Regular expression for paths that the proxy should accept.
address
127.0.0.1
The IP address on which to serve on.
api-prefix
/
Prefix to serve the proxied API under.
append-server-path
false
If true, enables automatic path appending of the kube context server path to each request.
disable-filter
false
If true, disable request filtering in the proxy. This is dangerous, and can leave you vulnerable to XSRF attacks, when used with an accessible port.
keepalive
0s
keepalive specifies the keep-alive period for an active network connection. Set to 0 to disable keepalive.
port
p
8001
The port on which to run the proxy. Set to 0 to pick a random port.
reject-methods
^$
Regular expression for HTTP methods that the proxy should reject (example --reject-methods='POST,PUT,PATCH').
reject-paths
^/api/.
/pods/.
/exec,^/api/.
/pods/.
/attach
Regular expression for paths that the proxy should reject. Paths specified here will be rejected even accepted by --accept-paths.
unix-socket
u
Unix socket on which to run the proxy.
www
w
Also serve static files from the given directory under the specified prefix.
www-prefix
P
/static/
Prefix to serve static files under, if static file directory is specified.
top
Display Resource (CPU/Memory) usage.
The top command allows you to see the resource consumption for nodes or pods.
This command requires Metrics Server to be correctly configured and working on the server.
Usage
$ kubectl top
node
Show metrics for all nodes
node
Show metrics for a given node
node
NODE_NAME
Display resource (CPU/memory) usage of nodes.
The top-node command allows you to see the resource consumption of nodes.
Usage
$ ku

bectl top node [NAME | -l label]
Flags
Name
Shorthand
Default
Usage
false
If present, print output without headers
selector
l
show-capacity
false
Print node resources based on Capacity instead of Allocatable(default) of the nodes.
sort-by
If non-empty, sort nodes list using specified field. The field can be either 'cpu' or 'memory'.
use-protocol-buffers
true
Enables using protocol-buffers to access Metrics API.
pod
Show metrics for all pods in the default namespace
kubectl
top
pod
Show metrics for all pods in the given namespace
kubectl top pod --
namespace
=
NAMESPACE
Show metrics for a given pod and its containers
kubectl top pod POD_NAME
--containers
Show metrics for the pods defined by label name=myLabel
kubectl top pod -l
name
=myLabel
Display resource (CPU/memory) usage of pods.
The 'top pod' command allows you to see the resource consumption of pods.
Due to the metrics pipeline delay, they may be unavailable for a few minutes since pod creation.
Usage
$ kubectl top pod [NAME | -l label]
Flags
Name
Shorthand
Default
Usage
A
false
false
If present, print usage of containers within a pod.
false
If present, print output without headers.
selector
l
sort-by
If non-empty, sort pods list using specified field. The field can be either 'cpu' or 'memory'.
sum
false
Print the sum of the resource usage
true
Print the supported API versions
Print the supported API versions on the server, in the form of "group/version".
Usage
$ kubectl api-versions
Usage
$ kubectl certificate SUBCOMMAND
approve
Approve CSR 'csr-sqgzp'
kubectl certificate approve csr-sqgzp
Approve a certificate signing request.
kubectl certificate approve allows a cluster admin to approve a certificate signing request (CSR). This action tells a certificate signing controller to issue a certificate to the requestor with the attributes requested in the CSR.
SECURITY NOTICE: Depending on the requested attributes, the issued certificate can potentially grant a requester access to cluster resources or to authenticate as a requested identity. Before approving a CSR, ensure you understand what the signed certificate can do.
Usage
$ kubectl certificate approve (-f FILENAME | NAME)
Flags
Name
Shorthand
Default
Usage
true
filename
f
[]
force
false
Update the CSR even if it is already approved.
kustomize
k
output
o
recursive
R
false
false
template
Template string or path to template file to use when 

-o=go-template, -o=go-template-file. The template format is golang templates
[
]
.
deny
Deny CSR 'csr-sqgzp'
deny
csr-sqgzp
Deny a certificate signing request.
kubectl certificate deny allows a cluster admin to deny a certificate signing request (CSR). This action tells a certificate signing controller to not to issue a certificate to the requestor.
Usage
$ kubectl certificate deny (-f FILENAME | NAME)
Flags
Name
Shorthand
Default
Usage
true
filename
f
[]
force
false
Update the CSR even if it is already denied.
kustomize
k
output
o
recursive
R
false
false
template
[
]
.
Print the address of the control plane and cluster services
kubectl
cluster-
info
Display addresses of the control plane and services with label kubernetes.io/cluster-service=true. To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Usage
$ kubectl cluster-info
dump
Dump current cluster state to stdout
kubectl
cluster-
info
dump
Dump current cluster state to /path/to/cluster-state
kubectl cluster-
info
dump
--output-directory
=/path/to/cluster-state
Dump all namespaces to stdout
kubectl
cluster
-
info
dump
Dump a set of namespaces to /path/to/cluster-state
kubectl
cluster
-info dump --namespaces
default
,kube-
system
--output-directory=/path/to/
cluster
-state
Dump cluster information out suitable for debugging and diagnosing cluster problems.  By default, dumps everything to stdout. You can optionally specify a directory with --output-directory.  If you specify a directory, Kubernetes will build a set of files in that directory.  By default, only dumps things in the current namespace and 'kube-system' namespace, but you can switch to a different namespace with the --namespaces flag, or specify --all-namespaces to dump all namespaces.
The command also dumps the logs of all of the pods in the cluster; these logs are dumped into different directories based on namespace and pod name.
Usage
$ kubectl cluster-info dump
Flags
Name
Shorthand
Default
Usage
A
false
If true, dump all namespaces.  If true, --namespaces is ignored.
true
[]
A comma separated list of namespaces to dump.
output
o
json
output-directory
Where to output the files.  If empty or '-' uses stdout, otherwise creates a directory hierarchy in that directory
20s
false
template
[

]
.
cordon
Mark node "foo" as unschedulable
kubectl cordon foo
Usage
$ kubectl cordon NODE
Flags
Name
Shorthand
Default
Usage
dry-run
none
selector
l
drain
Drain node "foo", even if there are pods not managed by a replication controller, replica set, job, daemon set or stateful set on it
kubectl drain foo
--force
As above, but abort if there are pods not managed by a replication controller, replica set, job, daemon set or stateful set, and use a grace period of 15 minutes
--grace-period
=900
The given node will be marked unschedulable to prevent new pods from arriving. 'drain' evicts the pods if the API server supports
https://kubernetes.io/docs/concepts/workloads/pods/disruptions/
. Otherwise, it will use normal DELETE to delete the pods. The 'drain' evicts or deletes all pods except mirror pods (which cannot be deleted through the API server).  If there are daemon set-managed pods, drain will not proceed without --ignore-daemonsets, and regardless it will not delete any daemon set-managed pods, because those pods would be immediately replaced by the daemon set controller, which ignores unschedulable markings.  If there are any pods that are neither mirror pods nor managed by a replication controller, replica set, daemon set, stateful set, or job, then drain will not delete any pods unless you use --force.  --force will also allow deletion to proceed if the managing resource of one or more pods is missing.
'drain' waits for graceful termination. You should not operate on the machine until the command completes.
When you are ready to put the node back into service, use kubectl uncordon, which will make the node schedulable again.
https://kubernetes.io/images/docs/kubectl_drain.svg
Usage
$ kubectl drain NODE
Flags
Name
Shorthand
Default
Usage
500
delete-emptydir-data
false
Continue even if there are pods using emptyDir (local data that will be deleted when the node is drained).
delete-local-data
false
disable-eviction
false
Force drain to use delete, even if eviction is supported. This will bypass checking PodDisruptionBudgets, use with caution.
dry-run
none
force
false
Continue even if there are pods that do not declare a controller.
-1
Period of time in seconds given to each pod to terminate gracefully. If negative, the default value specified in the pod will be used.
ignore-daemonsets
false
Ignore DaemonSet-managed pods.
pod-selector
Label selector to filter pods on the node
selector
l
skip-wait-for-delete-timeout
0
If pod DeletionTimestamp older than N seconds, skip waiting for the pod.  Seconds must be greater than 0 to skip.
timeout
0s
The length of time to wait before giving up, zero means infinite
taint
Update node 'foo' with a taint with key 'dedicated' and value 'special-user' and effect 'NoSchedule' # If a taint with that key and effect already exists, its value is replaced as specified
dedicated
=special-user:NoSchedule
Remove f

rom node 'foo' the taint with key 'dedicated' and effect 'NoSchedule' if one exists
kubectl
taint
nodes
foo
dedicated
:NoSchedule-
Remove from node 'foo' all the taints with key 'dedicated'
kubectl
taint nodes foo dedicated-
Add a taint with key 'dedicated' on nodes having label mylabel=X
node
-l
myLabel=
X
dedicated=
foo:PreferNoSchedule
Add to node 'foo' a taint with key 'bar' and no value
kubectl
taint
nodes
foo
bar
:NoSchedule
A taint consists of a key, value, and effect. As an argument here, it is expressed as key=value:effect.
The key must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to  253 characters.
The value is optional. If given, it must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to  63 characters.
The effect must be NoSchedule, PreferNoSchedule or NoExecute.
Currently taint can only apply to node.
Usage
$ kubectl taint NODE NAME KEY_1=VAL_1:TAINT_EFFECT_1 ... KEY_N=VAL_N:TAINT_EFFECT_N
Flags
Name
Shorthand
Default
Usage
all
false
Select all nodes in the cluster
true
dry-run
none
kubectl-taint
output
o
overwrite
false
If true, allow taints to be overwritten, otherwise reject taint updates that overwrite existing taints.
selector
l
false
template
[
]
.
validate
strict
uncordon
Mark node "foo" as schedulable
kubectl uncordon foo
Usage
$ kubectl uncordon NODE
Flags
Name
Shorthand
Default
Usage
dry-run
none
selector
l
alpha
These commands correspond to alpha features that are not enabled in Kubernetes clusters by default.
Usage
$ kubectl alpha
events
List recent events in the default namespace.
kubectl alpha events
List recent events in all namespaces.
List recent events for the specified pod, then wait 

for more events and list them as they arrive.
kubectl
alpha
events
--
for
pod/web
-
pod
-
13je7
--
watch
List recent events in given format. Supported ones, apart from default, are json and yaml.
kubectl alpha events -oyaml
List recent only events in given event types
--types
=Warning,Normal
Experimental: Display events
Prints a table of the most important information about events. You can request events for a namespace, for all namespace, or filtered to only those pertaining to a specified resource.
Usage
$ kubectl alpha events [(-o|--output=)json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-as-json|jsonpath-file] [--for TYPE/NAME] [--watch] [--event=Normal,Warning]
Flags
Name
Shorthand
Default
Usage
A
false
true
500
for
Filter events to only those pertaining to the specified resource.
false
When using the default output format, don't print headers.
output
o
false
template
[
]
.
types
[]
Output only events of given types.
watch
w
false
After listing the requested events, watch for more events.
Print the supported API resources
Print the supported API resources with more information
Print the supported API resources sorted by a column
kubectl api-
resources
--
sort
-by=
name
Print the supported namespaced resources
kubectl
api-resources --namespaced=
true
Print the supported non-namespaced resources
kubectl
false
Print the supported API resources with a specific APIGroup
--api-group
=rbac.authorization.k8s.io
Print the supported API resources on the server.
Usage
$ kubectl api-resources
Flags
Name
Shorthand
Default
Usage
api-group
Limit to resources in the specified API group.
cached
false
Use the cached list of resources if available.
true
If false, non-namespaced resources will be returned, otherwise returning namespaced resources by default.
false
output
o
Output format. One of: (wide, name).
sort-by
If non-empty, sort list of resources using specified field. The field can be either 'name' or 'kind'.
verbs
[]
Limit to resources that support the specified verbs.
Installing bash completion on macOS using homebrew ## If running Bash 3.2 included with macOS
brew
install
bash-completion
or, if running Bash 4.1+
brew install bash-
completion@
2
If kubectl is installed via homebrew, this should start working immediately ## If you've installed via other means, you may need add the completion to your completion directory
kubectl completion bash &gt;
$(
brew
--
prefix
)
/etc/bash_completion.d/kubectl
Installing bash completion on Linux ## If bash-completion is not installed on Linux, install the 'bash-completion' package ## via your distribution's package manager. ## Load the kubectl completion code for bash into the current shell
source &lt;(
kubectl
completion bash)
Write bash completion code to a file and source it from .bash_profile
kubectl
bash
&gt

; ~/.kube/completion.
bash.inc
printf
"
Kubectl shell completion
source
'
$HOME
/.kube/completion.bash.inc'
" &gt;&gt;
$HOME
/.bash_profile
source
$HOME
Load the kubectl completion code for zsh[1] into the current shell
kubectl
completion zsh)
Set the kubectl completion code for zsh[1] to autoload on startup
kubectl
completion zsh &gt;
"
${fpath[1]}
/_kubectl"
Load the kubectl completion code for fish[2] into the current shell
kubectl completion fish
| source
To load completions for each session, execute once:
kubectl completion fish &gt; ~
/.config/
fish
/completions/
kubectl.fish
Load the kubectl completion code for powershell into the current shell
kubectl completion powershell |
Out
-
String
| Invoke-
Set kubectl completion code for powershell to run on startup ## Save completion code to a script and execute in the profile
kubectl completion powershell &gt; $HOME\.kube\completion.ps1
Add
-Content
$PROFILE
"
$HOME
\.kube\completion.ps1"
Execute completion code in the profile
Add-Content $PROFILE
"if (Get-Command kubectl -ErrorAction SilentlyContinue) {
kubectl completion powershell
| Out-String | Invoke-Expression
}
"
Add completion code directly to the $PROFILE script
kubectl
completion powershell &gt;&gt;
$PROFILE
Output shell completion code for the specified shell (bash, zsh, fish, or powershell). The shell code must be evaluated to provide interactive completion of kubectl commands.  This can be done by sourcing it from the .bash_profile.
Detailed instructions on how to do this are available here:
for macOS:
https://kubernetes.io/docs/tasks/tools/install-kubectl-macos/#enable-shell-autocompletion
for linux:
https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#enable-shell-autocompletion
for windows:
https://kubernetes.io/docs/tasks/tools/install-kubectl-windows/#enable-shell-autocompletion
Note for zsh users:
[
1
]
zsh completions are only supported in versions of zsh &gt;= 5.2.
Usage
$ kubectl completion SHELL
config
Modify kubeconfig files using subcommands like "kubectl config set current-context my-context"
The loading order follows these rules:
1.  If the --kubeconfig flag is set, then only that file is loaded. The flag may only be set once and no merging takes place.
2.  If $KUBECONFIG environment variable is set, then it is used as a list of paths (normal path delimiting rules for your system). These paths are merged. When a value is modified, it is modified in the file that defines the stanza. When a value is created, it is created in the first file that exists. If no files in the chain exist, then it creates the last file in the list.
3.  Otherwise, ${HOME}/.kube/config is used and no merging takes place.
Usage
$ kubectl config SUBCOMMAND
Display the current-context
kubectl
config
Display the current-context.
Usage
$ kubectl config current-context
Delete the minikube cluster
kubectl
config
minikube
Delete the specified cluster from the kubeconfig.
Usage
$ kubectl config delete-cluster NAME
Delete the context for the minikube cluster
kubectl
config
minikube
Delete the specified context from the kubeconfig.
Usage
$ kubectl config delete-context NAME
Delete the minikube user
kubectl
config
minikube
Delete the specified user from the kubeconfig.
Usage
$ kubectl config delete-user NAME
List the clusters that kubectl knows about
kubectl
config
Display clusters defined in the kubeconfig.
Usage
$ kubectl config get-clusters
List all the contexts in your kubeconfig file
kubectl
config
Describe one context in your kubeconfig file
kubectl
config
my-context
Display one or many contexts from the kubeconfig file.
Usage
$ kubectl config get-contexts [(-o|--output=)name)]
Flags
Name
Shorthand
Default
Usage
false
When using the default or custom-column output format, don't print headers (def

ault print headers).
output
o
Output format. One of: (name).
get-users
List the users that kubectl knows about
kubectl
config
get-users
Display users defined in the kubeconfig.
Usage
$ kubectl config get-users
Rename the context 'old-name' to 'new-name' in your kubeconfig file
kubectl
config
rename-context old-name new-name
Renames a context from the kubeconfig file.
CONTEXT_NAME is the context name that you want to change.
NEW_NAME is the new name you want to set.
Note: If the context being renamed is the 'current-context', this field will also be updated.
Usage
$ kubectl config rename-context CONTEXT_NAME NEW_NAME
set
Set the server field on the my-cluster cluster to
https://1.2.3.4
kubectl
config
set
clusters.my-cluster.server https:
//1.2.3.4
Set the certificate-authority-data field on the my-cluster cluster
kubectl
config
set
clusters.my-cluster.certificate-authority-data $(echo
"cert_data_here"
| base64 -i -)
Set the cluster field in the my-context context to my-cluster
kubectl
config
set
contexts.my-context.cluster my-cluster
Set the client-key-data field in the cluster-admin user using --set-raw-bytes option
kubectl
config
set
users.cluster-admin.client-key-data cert_data_here
--set-raw-bytes
=
true
Set an individual value in a kubeconfig file.
PROPERTY_NAME is a dot delimited name where each token represents either an attribute name or a map key.  Map keys may not contain dots.
PROPERTY_VALUE is the new value you want to set. Binary fields such as 'certificate-authority-data' expect a base64 encoded string unless the --set-raw-bytes flag is used.
Specifying an attribute name that already exists will merge new fields on top of existing values.
Usage
$ kubectl config set PROPERTY_NAME PROPERTY_VALUE
Flags
Name
Shorthand
Default
Usage
set-raw-bytes
false
When writing a
[
]
byte PROPERTY_VALUE, write the given string directly without base64 decoding.
Set only the server field on the e2e cluster entry without touching other values
kubectl
config
set-cluster e2e
--server
=https://1.2.3.4
Embed certificate authority data for the e2e cluster entry
kubectl
config
set-cluster e2e --embed-certs
--certificate-authority
=~/.kube/e2e/kubernetes.ca.crt
Disable cert checking for the e2e cluster entry
kubectl
config
=
true
Set custom TLS server name to use for validation for the e2e cluster entry
kubectl
config
--tls-server-name
=my-cluster-name
Set proxy url for the e2e cluster entry
kubectl
config
--proxy-url
Set a cluster entry in kubeconfig.
Specifying a name that already exists will merge new fields on top of existing values for those fields.
Usage
$ kubectl config set-cluster NAME [--server=server] [--certificate-authority=path/to/certificate/authority] [--insecure-skip-tls-verify=true] [--tls-server-name=example.com]
Flags
Name
Shorthand
Default
Usage
false
embed-certs for the cluster entry in kubeconfig
proxy-url
proxy-url for the cluster entry in kubeconfig
Set the user field on the gce context entry without touching other values
kubectl
config
set-context gce
--user
Set a context entry in kubeconfig.
Usage
$ kubectl config set-context [NAME | --current] [--cluster=cluster_nickname] [--user=user_nickname] [--namespace=namespace]
Flags
Name
Shorthand
Default
Usage
current
false
Modify the current context
Set only the "client-key" field on the "cluster-admin" # entry, without touching other values
kubectl
config
set-credentials cluster-admin
--client-key
=~/.kube/admin.key
Set basic auth for the "cluster-admin" entry
kubectl
config
--username
=admin
--password
=uXFGweU9l35qcif
Embed client certificate data in the "cluster-admin" entry
kubectl
config
--client-certificate
=~/.kube/admin.crt
--embed

-certs
=
true
Enable the Google Compute Platform auth provider for the "cluster-admin" entry
kubectl
config
=gcp
Enable the OpenID Connect auth provider for the "cluster-admin" entry with additional args
kubectl
config
=oidc
=client-id=foo
=client-secret=bar
Remove the "client-secret" config value for the OpenID Connect auth provider for the "cluster-admin" entry
kubectl
config
=oidc
=client-secret-
Enable new exec auth plugin for the "cluster-admin" entry
kubectl
config
--exec-command
=/path/to/the/executable
--exec-api-version
=client.authentication.k8s.io/v1beta1
Define new exec auth plugin args for the "cluster-admin" entry
kubectl
config
--exec-arg
=arg1
=arg2
Create or update exec auth plugin environment variables for the "cluster-admin" entry
kubectl
config
--exec-env
=key1=val1
=key2=val2
Remove exec auth plugin environment variables for the "cluster-admin" entry
kubectl
config
=var-to-remove-
Set a user entry in kubeconfig.
Specifying a name that already exists will merge new fields on top of existing values.
Client-certificate flags:
  --client-certificate=certfile --client-key=keyfile
Bearer token flags:
  --token=bearer_token
Basic auth flags:
  --username=basic_user --password=basic_password
Bearer token and basic auth are mutually exclusive.
Usage
$ kubectl config set-credentials NAME [--client-certificate=path/to/certfile] [--client-key=path/to/keyfile] [--token=bearer_token] [--username=basic_user] [--password=basic_password] [--auth-provider=provider_name] [--auth-provider-arg=key=value] [--exec-command=exec_command] [--exec-api-version=exec_api_version] [--exec-arg=arg] [--exec-env=key=value]
Flags
Name
Shorthand
Default
Usage
Auth provider for the user entry in kubeconfig
auth-provider-arg
[]
'key=value' arguments for the auth provider
false
Embed client cert/key for the user entry in kubeconfig
exec-api-version
API version of the exec credential plugin for the user entry in kubeconfig
exec-arg
[]
New arguments for the exec credential plugin command for the user entry in kubeconfig
exec-command
Command for the exec credential plugin for the user entry in kubeconfig
exec-env
[]
'key=value' environment values for the exec credential plugin
unset
Unset the current-context
kubectl
config
unset current-context
Unset namespace in foo context
kubectl
config
unset contexts.foo.namespace
Unset an individual value in a kubeconfig file.
Usage
$ kubectl config unset PROPERTY_NAME
Use the context for the minikube cluster
kubectl
config
use-context minikube
Set the current-context in a kubeconfig file.
Usage
$ kubectl config use-context CONTEXT_NAME
view
Show merged kubeconfig settings
kubectl
config
view
Show merged kubeconfig settings and raw certificate data
kubectl
config
view --raw
Get the password for the e2e user
kubectl
config
view -o
jsonpath
=
Display merged kubeconfig settings or a specified kubeconfig file.
You can use --output jsonpath={...} to extract specific values using a jsonpath expression.
Usage
$ kubectl config view
Flags
Name
Shorthand
Default
Usage
true
flatten
false
Flatten the resulting kubeconfig file into self-contained output (useful for creating portable kubeconfig files)
merge
true
Merge the full hierarchy of kubeconfig files
minify
false
Remove all information no

t used by current-context from the output
output
o
yaml
raw
false
Display raw byte data
false
template
[
]
.
explain
Get the documentation of the resource and its fields
Get the documentation of a specific field of a resource
kubectl
explain
pods
.spec
.containers
List the fields for supported resources.
This command describes the fields associated with each supported API resource. Fields are identified via a simple JSONPath identifier:
&lt;type&gt;.&lt;fieldName&gt;
[
.&lt;fieldName&gt;
]
Add the --recursive flag to display all of the fields at once without descriptions. Information about each field is retrieved from the server in OpenAPI format.
Usage
$ kubectl explain RESOURCE
Flags
Name
Shorthand
Default
Usage
api-version
Get different explanations for particular API version (API group/version)
recursive
false
Print the fields of fields (Currently only 1 level deep)
options
Print flags inherited by all commands
Print the list of flags inherited by all commands
Usage
$ kubectl options
plugin
Plugins provide extended functionality that is not part of the major command-line distribution. Please refer to the documentation and examples for more information about how write your own plugins.
The easiest way to discover and install plugins is via the kubernetes sub-project krew. To install krew, visit
https://krew.sigs.k8s.io/docs/user-guide/setup/install/
Usage
$ kubectl plugin [flags]
list
List all available plugins
kubectl
plugin
list
List all available plugin files on a user's PATH.
Available plugin files are those that are: - executable - anywhere on the user's PATH - begin with "kubectl-"
Usage
$ kubectl plugin list
Flags
Name
Shorthand
Default
Usage
name-only
false
If true, display only the binary name of each plugin, rather than its full path
version
Print the client and server versions for the current context
kubectl
version
Print the client and server version information for the current context.
Usage
$ kubectl version
Flags
Name
Shorthand
Default
Usage
client
false
If true, shows client version only (no server required).
output
o
One of 'yaml' or 'json'.
short
false
If true, print just the version number.
<!--<script src="actions.js"></script>-->

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
kubectl
4:
5:
6:
,
kubectl
.
kubectl
config
in the
flag.
kubectl
kubectl
;
.
docker
kubectl
Syntax
kubectl
kubectl
[
command
]
[
TYPE
]
[
NAME
]
[
flags
]
where
command
,
TYPE
,
NAME
, and
flags
are:
command
create
,
get
,
describe
,
delete
.
TYPE
NAME
.
.
Example:
.
Example:
Example:
flags
-s
or
--server
Caution:
By default
kubectl
and
If the
environment variable is set, cli operati

ons on namespaced resources will default to the variable value. For example, if the variable is set to
seattle
,
seattle
If:
, and
the
the
=
kubectl
Operation
Syntax
alpha
annotate
apply
attach
auth
autoscale
config
convert
cordon
cp
create
delete
describe
diff
drain
edit
kubectl edit (-f 

FILENAME | TYPE NAME | TYPE/NAME) [flags]
events
exec
explain
expose
get
kustomize
label
logs
options
patch
plugin
proxy
replace
rollout
run
scale
set
taint
top
uncordon
version
wait
kubectl
NAME
KIND
bindings
v1
true
Binding
cs
v1
false
cm
v1
true
ConfigMap
endpoints
ep
v1
true
Endpoints
events
ev
v1
true
Event
limits
v1
true
ns
v1
false
Namespace
nodes
no
v1
false
Node
persistentvol

umeclaims
pvc
v1
true
pv
v1
false
pods
po
v1
true
Pod
v1
true
rc
v1
true
quota
v1
true
secrets
v1
true
Secret
sa
v1
true
services
svc
v1
true
Service
false
false
crd,crds
false
false
apps/v1
true
ds
apps/v1
true
DaemonSet
deploy
apps/v1
true
rs
apps/v1
true
sts
apps/v1
true
false
true
false
false
false
hpa
true
cronjobs
cj
batch/v1
true
CronJob
jobs
batch/v1
true
Job
csr
false
leases
true
Lease
true
events
ev
true
Event
false
false
false
ingresses
ing
true
Ingress
netpol
true
false
pdb
policy/v1
true
psp
false
false
false
true
roles
true
Role
pc
false
false
CSIDriver
csinodes
false
CSINode
true
sc
false
false
kubectl
kubectl
-o
or
--output
kubectl
command.
Syntax
kubectl
[
command
]
[
TYPE
]
[
NAME
]
kubectl
.
file.
-o json
jsonpath
jsonpath
file.
-o name
-o wide
-o yaml
Example
In t

his example, the following command outputs the details for a single pod as a YAML formatted object:
kubectl
option.
or
.
Examples
Inline:
=
=
where the
NAME          RSRC
NAME           RSRC
kubectl
flag to the
command.
Examples
=
false
NAME       AGE
--sort-by
kubectl
--sort-by
jsonpath
Syntax
kubectl
[
command
]
[
TYPE
]
[
NAME
]
--sort-by
=
Example
=
kubectl
=
=
server01

s &lt;rc-name&gt;
Note:
The
-o
or
--output
-w
or
--watch
=
kubectl
exec
kubectl
exec
kubectl
exec
kubectl
plugins:
#!/bin/sh
echo
kubectl
, use
the

cat ./k

ubectl-whoami
have
=
$(
)
Read the
kubectl
the
reference
kubectl
Read about
in kubectl
.
1 - kubectl Cheat Sheet
kubectl
BASH
source
&lt;
(
)
echo
kubectl
alias
k
=
kubectl
complete
ZSH
source
&lt;
(
)
echo
A note on
Appending
:
kubectl -A
kubectl
=
jsonpath
=
jsonpath
=
jsonpath
=
=
=
=
=
ggckad-s2
=
=
foo
\
unset
users.foo
alias
kx
=
alias
kn
=
'f() { [ "$1" ] &amp;&amp; kubectl config set-context --current --na

mespace $1 || kubectl config view --minify | grep namespace | cut -d" " -f6 ; } ; f'
apply
.
.yaml
,
.yml
, and
.json
=
nginx
=
echo
=
=
"*/1 * * * *"
--
echo
cat
kind: Pod
metadata:
spec:
args:
- sleep
---
kind: Pod
metadata:
spec:
args:
- sleep
- "1000"
EOF
cat
metadata:
data:
EOF
=
=
=
=
app
=
\
jsonpath
=
\
-o
jsonpath
=
=
=
=
=
Running
jsonpath
=
sel
=
${
$(
=
json | jq -j
)
%?
}
echo
$(
=
$sel
--output
=
jsonpath
={
}
)
JSONPATH
=
'{range .items[*]}{@.met

adata.name}:{range @.status.conditions[*]}{@.type}={@.status};{end}{end}'
\
jsonpath
=
"
$JSONPATH
"
| grep
=
jsonpath
=
=
=
Warning
for
pod in
$(
=
jsonpath
={
}
)
;
do
echo
$pod
kubectl
exec
-it
$pod
-- env;
done
=
status
kubectl
set
www
=
image:v2
history
=
2
=
80
=
8000
=
awesome
=
=
2
--max
=
10
=
'json'
-p
=
=
'[{"op": "remove", "path": "/spec/template/s

pec/containers/0/livenessProbe"}]'
=
'json'
-p
=
=
'scale'
--type
=
'merge'
-p
=
"nano"
=
3
rs/foo
=
3
=
2
=
3
=
5
name
=
myLabel
=
true
| awk
name
=
myLabel
name
=
name
=
=
=
=
=
kubectl
exec
kubectl
exec
kubectl
exec
=
cpu
kubectl cp my-namespace/my-p

od:/tmp/foo /tmp/bar
Note:
kubectl cp
kubectl cp
will fail.
.
exec
kubectl
exec
5000
kubectl
exec
=
=
dedicated
=
API group
, and
Kind
:
=
true
=
false
=
list,get
=
-o
(or
--output
kubectl
command.
file
-o=json
jsonpath
jsonpath
file
-o=name
-o=wide
-o=yaml
:
=
=
# All images r

unning in namespace: default, grouped by Pod
=
=
=
=
=
=
.
-v
or
--v
here
.
Verbosity
--v=0
always
--v=1
--v=2
--v=3
--v=4
--v=5
--v=6
--v=7
--v=8
--v=9
Read the
JsonPath
.
See
kubectl
options.
Also read
.
2 - kubectl Commands
3 - kubectl
Synopsis
Options
-h, --help
Defines the maximum size a log file can grow to. Unit is megabytes. If t

he value is 0, the maximum file size is unlimited.
See Also
kubectl cp
- Build a kustomization targ

et from a directory or a remote url.
4 - JSONPath Support
Use the
range
,
end
.
Note:
The
$
{
"kind"
:
"List"
,
"items"
:[
{
"kind"
:
"None"
,
:{
"name"
:
},
"status"
:{
:{
"cpu"
:
"4"
},
:[{
"type"
:
,
"address"
:
}]
}
},
{
"kind"
:
"None"
,
:{
"name"
:
},
"status"
:{
:{
"cpu"
:
"8"
},
:[
{
"type"
:
,
"address"
:
},
{
"type"
:
"another"
,
"address"
:
}
]
}
}
],
"users"
:[
{
"name"
:
"myself"
,
"user"
: {}
},
{
"name"
:
"e2e"
,
"user"
: {
:
"admin"
,
:
"secret"
}
}
]
}
Function
Example
Result
text
@
{@}
.
or
[]
{.kind}
,
or
List
..
{..name}
*
myself
[,]
?()
filter
secret
range
,
end
''
kubectl
=
jsonpath
=
'{@}'
=
jsonpath
=
=
jsonpath
=
=
jsonpath
=
=
jsonpath
=
'{range .items[*]}{.metadata.name}{"\t"}{.status.startTime}{"\n"}{end}'
Note:
double
"{range .items[*]

}{.metadata.name}{\"
\t\
\n\
"}{end}"
Note:
jq
.
jsonpath
=
5 - kubectl for Docker Users
kubectl
kubectl
command.
docker run
.
docker:
=
always -e
DOMAIN
=
docker ps
kubectl:
=
kubectl
set
DOMAIN
=
cluster
Note:
kubectl
=
80
--name
=
service
.
[
-i
]
[
--tty
]
=
Unlike
--attach
stdin
,
stdout
and
stderr
).
docker ps
.
docker:
kubectl:
ubuntu                      0/1       Completed   0          20s
.
docker:
docker ps
...
kubectl:
NAME       

       READY     STATUS    RESTARTS   AGE
...
.
docker:
docker ps
docker
exec
kubectl:
kubectl
exec
docker:
docker
exec
# exit
kubectl:
kubectl
exec
# exit
.
.
docker:
kubectl:
.
.
docker:
docker ps
kubectl:
app
=
nginx-app
app
=
nginx-app
Note:
.
To get the version of client and server, see
.
docker:
kubectl:
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.9+a3d1dfa6f4335", GitCommit:"9b77fed11

a9843ce3780f70dd251e92901c43072", GitTreeState:"dirty", BuildDate:"2017-08-29T20:32:58Z", OpenPaasKubernetesVersion:"v1.03.02", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
.
docker:
 Dirs: 248
CPUs: 12
kubectl:
6 - kubectl Usage Conventions
kubectl
.
Using
kubectl
-o name
,
-o json
,
-o yaml
,
, or
.
get
,
patch
,
edit
and
replace
status
and
scale
status
For
:v1234
,
v1.2.3
,
:latest
).
flags.
You can use
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Network Pl

ugins
Tasks
Monitoring, Logging, and

 Debugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configuring

 a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Image Policy API (v1alp

ha1)
Concepts
Overview
Overview
running
Each VM is a full machine running all the co

mponents, including its own operating
Continuous development, integration, and deployment: provides for reliable
Kubernetes lets you store and manage sensitive information,

 such as passwords, OAuth tokens,
are optional and pluggable. Kubernetes provides the building blocks for building developer
.
Ready to
?
Feedback
Yes
No
.
or
.
Last modified January 03, 2023 at 10:07 AM PST:
Nit: case consistency for list items (#37829) (a745da1835)
Home
Blog
Training
Partners
Community
CC BY 4.0

TLS bootstrapping | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
found
here
.
file
file
a cluster.
not
have a
kubeconfi

g
file
file
kubectl
kubelet
(key).
group
.
boot

strap token
here
type
,
namespace
and
name
here
.
head -c
16
' '
.
Add the
here
for
to create a
.
.
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
cfssl
Since these signed certificates will, in turn, be used by the kubelet to authenticate as a regular kubelet to kube-apis

erver, it is important that the CA
=
=
=
=
Approval
.
.
to the
:
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
to the
that
:
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
Group
name
:
apiGroup
:
roleRef
:
kind
:
name
:
apiGroup
:
The
API
to
A path to a
.
Its format is identi

cal to a normal
:
v1
kind
:
Config
clusters
:
-
cluster
:
:
server
:
name
:
bootstrap
contexts
:
-
context
:
cluster
:
bootstrap
user
:
name
:
bootstrap
:
bootstrap
:
{}
users
:
-
name
:
user
:
token
:
server
token
any
is
a standard
kubectl
does not
is used
.
client
serving
and
flags
.
can
beta
field of
Enabling
both
and
of
the
--rotate-server-cert

ificates
Note:
serving
security
reasons
. To use
and
contains
)
contains
,
and
in
to verify
)
manager.
and
.
Feedback
Yes
No
.
or
.
Last modified December 09, 2022 at 8:31 AM PST:
fix documentation for RotateKubeletClientCertificate (ab4812140f)
Approval
Home
Blog
Training
Partners
Community
CC BY 4.0

Service Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
Service
2:
Endpoints
3:
4:
Ingress
5:
1 - Service
Service
: v1
kind
: Service
metadata
(
)
spec
(
)
status
(
)
selector
ports
port
(string)
(string)
(int32)
The port on each node on which this service is exposed when type is NodePort or LoadBalancer. Usually assigned by the system. If a value is specified, in-range, and not in use it will be used, otherwise the operation will fail. If not specified, a port will be allocated if this Service requires one. If this field is specified when creating a Service which d

oes not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type from NodePort to ClusterIP). More info:
(string)
type
(string)
(string)
clusterIP
(string)
clusterIP is the IP address of the service and is usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be blank) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above). Valid values are "None", empty string (""), or a valid IP address. Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections

 are preferred and proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. More info:
(string)
(string)
(string)
loadBalancerClass is the class of the load balancer implementation this Service belongs to. If specified, the value of this field must be a label-style identifier, with an optional prefix, e.g. "internal-vip" or "example.com/internal-vip". Unprefixed names are reserved for end-users. This field can only be set when the Service type is 'LoadBalancer'. If not set, the default load balancer implementation is used, today this is typically done throug

h the cloud provider integration, but should apply for any default implementation. If set, it is assumed that a load balancer implementation is watching for Services with a matching class. Any default load balancer implementation (e.g. cloud providers) should ignore Services that set this field. This field can only be set when creating or updating a Service to type 'LoadBalancer'. Once set, it can not be changed. This field will be wiped when a service is updated to a non 'LoadBalancer' type.
(string)
type
(string)
(string)
(int32)
(boolean)
sessionAffinityConf

ig.clientIP.timeoutSeconds
(int32)
(boolean)
type
(int64)
(string)
(string)
**
(string)
Error is to record the problem with the service port The format of the error shall comply with the following rules: - built-in error values shall be specified in

 this file and those shall use
: v1
kind
metadata
(
ListMeta
)
items
([]
Service
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
202 (
Service
update
name
(
in path
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
patch
name
(
in path
namespace
(
in path
namespace
body

:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Service
): OK
202 (
Service
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - Endpoints
Endpoints
 Subsets: [
   {
   },
   {
   },
]
: v1
kind
metadata
(
)
subsets
{
}
(string), require

d
(string)
(string)
(
)
(string)
(string)
(
)
(string)
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Endpoints
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
reso

(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
202 (
Endpoints
update
name
(
in path
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3 - EndpointSlice
kind
metadata
(
)
endpoints
endpoints is a list of unique endpoints in this slice. Each slice may inclu

de a maximum of 1000 endpoints.
(boolean)
(boolean)
(boolean)
(string)
(string)
(
)
(string)
ports
(int32)
(string)
ports.n

ame
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
)

: string, required
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
4 - Ingress
Ingress
kind
: Ingress
metadata
(
)
spec
(
)
status
(
)
(
)
(string)
annotation (simple constant name) was never formally defined, it was widely supported by Ingress controllers to create a direct binding between Ingress controller and Ingress resources. Newly created Ingress resources should prefer using the field. However, even though the annotation is officially deprecated, for backwards compatibility reasons, ingress controllers should still honor that annotation if present.
rules
IngressRule repr

esents the rules mapping the paths under a specified host to the related backend services. Incoming requests are first evaluated for a host match, then routed to the backend associated with the matching IngressRuleValue.
(string)
2. The
:
/
?
(
(string)
tls
TLS configuration. Currently the Ingress only supports a single TLS port, 443. If multiple members of this list specify different hosts, they will be multiplexed on the same port according to the hostname specified through the SNI TLS extension, if the ingress controller fulfilling the ingress supports SNI.
tls.hosts
Atomic: will be replaced du

ring a merge
(string)
resource
(
)
service
(string)
(int32)
(string)
(string)
(string)
items
([]
Ingress
(string)
kind
(string)
metadata
(
ListMeta
)
Standard object's meta

data. More info:
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
202 (
Ingress
update
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in que

ry
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5 - IngressClass
kind
metadata
(
)
spec
(
)
(string)
(string)
APIGroup is the group for the resource being re

ferenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
resourceVersio

n
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

Determine the Reason for Pod Failure | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Debug Pods
Debug Init Con

tainers
Auditing
Run Jobs
TLS
Tutorials
Security

andards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
.
minikube
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
debian
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
]
command
and
args
the
=
yaml
:
v1
kind
:
Pod
...
lastState
:
:
:
...
exitCode
:
0
:
...
message
:
|
...
=
kub

ectl get pod multi-container-pod -o go-template
=
field of a Container, which has a default
value of
or
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
debian
:
File
to "
See the
field in
Container
.
.
.
Feedback
Yes
No
.
or
.
Last modified September 23, 2022 at 6:29 PM PST:
Update Determine the Reason for Pod Failure (1ca77d26b9)
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Pod Initialization | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
minikube
.
:
v1
kind
:
Pod
metadata
:
name
:
init-demo
spec
:
:
-
name
:
nginx
image
:
nginx
ports
:
-
:
80
:
-
name
:
workdir
mountPath
:
:
-
name
:
install
image
:
command
:
- wget
-
"-O"
-
:
-
name
:
workdir
mountPath
:
dnsPolicy
:
Default
volumes
:
-
name
:
workdir
emptyDir
:
{}
/work-dir
Volume at

  ...
  ...
.
.
Volumes
.
Feedback
Yes
No
.
or
.
Last modified April 27, 2022 at 6:02 PM PST:
Fixed all hyperlinks of the task "Monitoring, Logging, and Debugging" in all pages. (d1076700c7)
Home
Blog
Training
Partners
Community
CC BY 4.0

Learn Kubernetes Basics | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
1.1:
1.2:
2:
2.1:
2.2:
3:
3.1:
3.2:
4:
4.1:
4.2:
5:
5.1:
5.2:
6:
6.1:
6.2:
This tutorial provides a walkthrough of the basics of the Kubernetes cluster orchestration system. Each module contains some background information on major Kubernetes features and concepts, and includes an interactive online tutorial. These interactive tutorials let you manage a simple cluster and its containerized applications for yourself.
Using the interactive tutorials, you can learn to:
Deploy a containerized application on a cluster.
Scale the deployment.
Update the containerized application with a new software version.
Debug the containerized application.
The tutorials use Katacoda to run a virtual terminal in your web browser that runs Minikube, a small-scale local deployment of Kubernetes that can run anywhere. There's no need to install any software or configure anything; each interactive tutorial runs directly out of your web browser itself.
What can Kubernetes do for you?
With modern web services, users expect applications to be available 24/7, and developers expect to deploy new versions of those applications several times a day. Containerization helps package software to serve these goals, enabling applications to be released and updated without downtime. Kubernetes helps you make sure those containerized applications run where and when you want, and helps them find the resources and tools they need to work. Kubernetes is a production-ready, open source platform designed with Google's accumulated experience in container orchestration, combined with best-of-breed ideas from the community.
Kubernetes Basics Modules
1. Create a Kubernetes cluster
2. Deploy an app
3. Explore your app
4. Expose your app publicly
5. Scale up your app
6. Update your app
1 - Create a Cluster
cluster
1.1 - Using Minikube to Create a Cluster
Kubernetes is an open-source platform and is produc

tion-ready.
The
Nodes
Summary:
Minikube
etcd
›
1.2 - Interactive Tutorial - Creating a Cluster
Home
2 - Deploy an App
2.1 - Using kubectl to Create a Deployment
Learn about application Deployments.
Deploy your first app on Kubernetes with kubectl.
Kubernetes Deployments
Once you have a running Kubernetes cluster, you can deploy your containerized applications on top of it.
To do so, you create a Kubernetes
configuration. The Deployment instructs Kubernetes
how to create and update instances of your application. Once you've created a Deployment, the Kubernetes
control plane schedules the application instances included in that Deployment to run on individual Nodes in the
cluster.
Once the application instances are created, a Kubernetes Deployment Controller continuously monitors those instances. If the Node hosting an instance goes down or is deleted, the Deployment controller replaces the instance with an instance on another Node in the cluster.
This provides a self-healing mechanism to address machine failure or maintenance.
In a pre-orchestration world, installation scripts would often be used to start applications, but they did not allow recovery from machine failure. By both creating your application instances and keeping them running across Nodes, Kubernetes Deployments provide a fundamentally different approach to application management.
Summary:
Kubectl
A Deployment is responsible for creating and updating instances of your application
Deploying your first

 app on Kubernetes
You can create and manage a Deployment by using the Kubernetes command line interface,
Kubectl
. Kubectl uses the Kubernetes API to interact with the cluster. In this module, you'll learn the most common Kubectl commands needed to create Deployments that run your applications on a Kubernetes cluster.
When you create a Deployment, you'll need to specify the container image for your application and the number of replicas that you want to run. You can change that information later by updating your Deployment; Modules
5
and
6
of the bootcamp discuss how you can scale and update your Deployments.
Applications need to be packaged into one of the supported container formats in order to be deployed on Kubernetes
For your first Deployment, you'll use a hello-node application packaged in a Docker container that uses NGINX to echo back all the requests. (If you didn't already try creating a hello-node application and deploying it using a container, you can do that first by following the instructions from the
Hello Minikube tutorial
).
Now that you know what Deployments are, let's go to the online tutorial and deploy our first app!
›
2.2 - Interactive Tutorial - Deploying an App
.
Home
3 - Explore Your App
3.1 - Viewing Pods and Nodes
2
Pod
Summary:
Pods
Nodes
Nodes
Node
A container runtime (like Docker) responsible for pulling the container image from

 a registry, unpacking the container, and running the application.
In Module
2
›
3.2 - Interactive Tutorial - Exploring Your App
Home
4 - Expose Your App Publicly
4.1 - Using a Service to Expose Your App
Pods
lifecycle
selector
type
ClusterIP
NodePort
CNAME
kube-dns
.
selector
in the spec. A

 Service created without
selector
.
Summary
›
4.2 - Interactive Tutorial - Exposing Your App
Home
5 - Scale Your App
5.1 - Running Multiple Instances of Your App
Service
Scaling
Summary:
Previous
Next
›
5.2 - Interactive Tutorial - Scaling Your App
Home
6 - Update Your App
6.1 - Performing a Rolling Update
In the previous modul

e we scaled our application to run multiple instances. This is a requirement for performing updates without affecting application availability. By default, the maximum number of Pods that can be unavailable during the update and the maximum number of new Pods that can be created, is one. Both options can be configured to either numbers or percentages (of Pods).
Summary:
Previous
Next
›
6.2 - Interactive Tutorial - Updating Your App
Home
Blog
Training
Partners
Community
CC BY 4.0

Managing Secrets using Kustomize | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
kubectl
supports using the
Kustomize object management tool
to manage Secrets
and ConfigMaps. You create a
resource generator
using Kustomize, which
generates a Secret that you can apply to the API server using
kubectl
.
minikube
You can generate a Secret by defining a
in a
file that references other existing files,
.env
files, or
literal values. For example, the following instructions create a Kustomization
file for the username
admin
and the password
1f2d1e2e67df
.
Create the Kustomization file
Literals
Files
.env files
:
-
name
:
database-creds
literals
:
- password=1f2d1e2e67df
Store the credentials in files with the values encoded in base64:
echo
-n
'admin'
echo
-n
The
-n
flag ensures that there's no newline character at the end of your
files.
Create the
file:
:
-
name
:
files
:
- username.txt
You can also define the secretGenerator in the
file by
providing
.env
files. For example, the following
file
pulls in data from an
.env.secret
file:
:
-
name
:
db-user-pass
envs
:
- .env.secret
In all cases, you don't need to base64 encode the values. The name of the YAML
file
must
be
or
kustomization.yml
.
Apply the kustomization file
To create the Secret, apply the directory that contains the kustomization file:
kubectl apply -k &lt;directory-path&gt;
secret/database-creds-5hdh7hhgfk created
When a Secret is generated, the Secret name is created by hashing
the Secret data and appending the hash value to the name. This ensures that
a new Secret is generated each time the data is modified.
.
In your
file, modify the data, 

such as the
password
.
Apply the directory that contains the kustomization file:
secret/db-user-pass-6f24b56cc8 created
The edited Secret is created as a new
Secret
object, instead of updating the
existing
Secret
object. You might need to update references to the Secret in
your Pods.
Clean up
To delete a Secret, use
kubectl
:
kubectl delete secret db-user-pass
manage Secrets using config file
Feedback
Yes
No
.
or
.
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

TokenReview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

Reference
kind
metadata
(
)
spec
(
status
(
)
audiences
token
(string)
audiences
(boolean)
error
(string)
user
user.uid
(string)
A unique value that identifies this user across time. If this user is deleted and another user by the same nam

e is added, they will have different UIDs.
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
Feedback
Yes
No
.
or
.
create
Home
Blog
Training
Partners
Community
CC BY 4.0

Use an HTTP Proxy to Access the Kubernetes API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

Tasks
minikube
.
=
=
8080
curl
,
wget
,
{
    "v1"
  ],
    {
    }
  ]
}
{
  },
  "items": [
    {
        },
        ...
}
.
Feedback
Yes
No
.
or
.
Last modified July 09, 2020 at 10:21 AM PST:
Avoid using deprecated commands in task (621e5e7236)
Before you 

begin
Home
Blog
Training
Partners
Community
CC BY 4.0

SubjectAccessReview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kind
metadata
(
)
spec
(
status
(
)
extra
groups
(string)
(string)
(string)
(string)
(string)
(string)
(string)
Subresource is one of the existing resou

rce types. "" means none.
(string)
(string)
uid
(string)
user
(string)
allowed
denied
(boolean)
(string)
reason
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
Feedback
Yes
No
.
or
.
create
Home
Blog
Training
Partners
Community
CC BY 4.0

Using NodeLocal DNSCache in Kubernetes Clusters | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Insta

Tasks
minikube
.
serviceIP
(
default
This image shows how NodeLocal DNSCache handles DNS queri

es.
Note:
"
kubedns
=
`
jsonpath
={
}
`
domain
=
localdns
=
is "
is the
sed -i
$localdns
$domain
$kubedns
/g"
and
the
pods.
as well as
sed -i
$localdns
$domain
$kubedns
/g"
.
The
Run
to use
flag,
.
namespace
CoreDNS
.
kube-dns
namespace
shown in
.
The
kube-dns
The
From
,
Memory usage can be reduced b

y specifying smaller cache sizes.
using the
If a
The
You can determine a suitable memory limit by running node-local-dns pods without a limit and
in
Feedback
Yes
No
.
or
.
Last modified August 25, 2022 at 10:05 PM PST:
Favor EndpointSlice over Endpoints (1eef742465)
Home
Blog
Training
Partners
Community
CC BY 4.0

Developing and debugging services locally using telepresence | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggrega

tion Layer
Tasks
Debug Pods
Det

ermine the Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

dards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
Note:
kubectl
, run
 
...
Use the
Where:
And
Running this command tell

s Telepresence to send remote traffic to your local service instead of the service in the remote Kubernetes cluster. Make edits to your service source code locally, save, and see the corresponding changes when accessing your remote application take effect immediately. You can also run your local service using a debugger or any other local development tool.
.
Feedback
Yes
No
.
or
.
Last modified August 21, 2022 at 9:45 PM PST:
Fix typo in tasks/debug-application-cluster/local-debugging.md (3f08d3163e)
Home
Blog
Training
Partners
Community
CC BY 4.0

Contributing new content | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

This section contains information you should know before contributing new
content.
subgraph second[Before you begin]
S[ ] -.-
A[Sign the CNCF CLA] --&gt; B[Choose Git branch]
B --&gt; C[One language per PR]
C --&gt; F[Check out
contributor tools]
end
subgraph first[Contributing Basics]
T[ ] -.-
D[Write docs in markdown
and build site with Hugo] --- E[source in GitHub]
E --- G['/content/../docs' folder contains docs
for multiple languages]
G --- H[Review Hugo page content
types and shortcodes]
end
first ----&gt; second
class A,B,C,D,E,F,G,H grey
enabled
Figure - Contributing new content preparation
The figure above depicts the information you should know
prior to submitting new content. The information details follow.
Contributing basics
Write Kubernetes documentation in Markdown and build the Kubernetes site
using
Hugo
.
Kubernetes documentation uses
CommonMark
as its flavor of Markdown.
The source is in
GitHub
. You can find
Kubernetes documentation at
. Some of the reference
documentation is automatically generated from scripts in
the
update-imported-docs/
describe the
presentation of documentation content in Hugo.
You can use
Docsy shortcodes
or
custom Hugo shortcodes
to contribute to Kubernetes documentation.
In addition to the standard Hugo shortcodes, we use a number of
in our
documentation to control the presentation of content.
Documentation source is available in multiple languages in
/content/
. Each
language has its own folder with a two-letter code determined by the
ISO 639-1 standard
. For example, English documentation source is stored in
.
For more information about contributing to documentation in multiple languages
or starting a new translation,
see
localization
.
Sign the CNCF CLA
All Kubernetes contributors
must
read
the
Contributor guide
and
sign the Contributor License Agreement (CLA)
.
Pull requests from contributors who haven't signed the CLA fail the automated
tests. The name and email you provide must match those found in
your
git config
, and your git name and email must match those u

sed for the
CNCF CLA.
Choose which Git branch to use
When opening a pull request, you need to know in advance which branch to base
your work on.
Scenario
Branch
Existing or new English language content for the current release
main
Content for a feature change release
The branch which corresponds to the major and minor version the feature change is in, using the pattern
dev-&lt;version&gt;
. For example, if a feature changes in the
v1.27
release, then add documentation changes to the
dev-1.27
branch.
Content in other languages (localizations)
Use the localization's convention. See the
Localization branching strategy
If you're still not sure which branch to choose, ask in
#sig-docs
on Slack.
Note:
If you already submitted your pull request and you know that the
base branch was wrong, you (and only you, the submitter) can change it.
Languages per PR
Limit pull requests to one language per PR. If you need to make an identical
change to the same code sample in multiple languages, open a separate PR for
each language.
Tools for contributors
The
doc contributors tools
directory in the
repository contains tools to help your
contribution journey go more smoothly.
Feedback
Yes
No
.
or
.
Last modified December 07, 2021 at 7:00 PM PST:
removed the italics (c214d37d86)
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm reset phase | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Kubeadm
,
phase
Synopsis
Options
-h, --help
preflight
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
Path to the CRI socket to connect. If empty kubeadm will try to auto-detect this val

ue; use this option only if you have more than one CRI installed or if you have non-standard CRI socket.
--dry-run
-h, --help
or
Feedback
Yes
No
.
or
.
Last modified December 09, 2021 at 6:55 PM PST:
Update kubeadm reference for 1.23 (36a57f5b4d)
Home
Blog
Training
Partners
Community
CC BY 4.0

ValidatingWebhookConfiguration | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Reference
kind
metadata
(
)
.
webhooks
name
([]byte)
caBundle
service
service
or
url
service
.
name
namespace
(string)
path
(int32)
If specified, the port on the 

service that hosting webhook. Default to 443 for backward compatibility.
port
(string)
url
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"st

aging"
]
}
]
}
See
(
)
any
(string)
(int32)
kind
metadata
(
ListMeta
)
items
([]

get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string

(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

SelfSubjectRulesReview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kind
metadata
(
)
spec
(
status
s are allowed, but only as th

e full, final step in the path. "
"
(string)
namespace
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
Feedback
Yes
No
.
or
.
create
Home
Blog
Training
Partners
Community
CC BY 4.0

Accessing Clusters | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
This topic discusses multiple ways to interact with clusters.
When accessing the Kubernetes API for the first time, we suggest using the
Kubernetes CLI,
kubectl
.
a
,
Many of the
examples
kubectl
, and complete documentation is found in the
kubectl reference
.
Kubectl handles locating and authenticating to the apiserver.
If you want to directly access the REST API with an http client like
curl or wget, or a browser, there are several ways to locate and authenticate:
Run kubectl in proxy mode.
Recommended approach.
Uses stored apiserver location.
Verifies identity of apiserver using self-signed cert. No MITM possible.
Authenticates to apiserver.
In future, may do intelligent client-side load-balancing and failover.
Provide the location and credentials directly to the http client.
Alternate approach.
Works with some types of client code that are confused by using a proxy.
Need to import a root cert into your browser to protect against MITM.
locating the apiserver and authenticating.
=
8080
See
Then you can explore the API with curl, wget, or a browser, replacing localhost
with [::1] for IPv6, like so:
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Use
and
kubectl describe secret...
to create a token for the default service account with grep/cut:
First, create the Secret, requesting a token for the default ServiceAccount:
metadata:
EOF
Next, wait for the token controller to populate the Secret with a token:
while
! kubectl describe secret default-token 

| grep -E
'^token'
do
echo
&gt;&amp;
2
sleep
1
done
Capture and use the generated token:
APISERVER
=
$(
kubectl config view --minify | grep server | cut -f 2- -d
":"
| tr -d
" "
)
TOKEN
=
$(
kubectl describe secret default-token | grep -E
'^token'
| cut -f2 -d
':'
| tr -d
" "
)
curl
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Using
jsonpath
:
APISERVER
=
$(
kubectl config view --minify -o
jsonpath
=
'{.clusters[0].cluster.server}'
)
TOKEN
=
$(
jsonpath
=
)
curl
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
The above examples use the
~/.kube
On some clusters, the apiserver does not require authentication; it may serve
for this.
Controlling Access to the API
describes how a cluster admin can configure this.
Kubernetes officially supports
Go
and
Python
client libraries.
Go client
, see
INSTALL.md
for detailed installation instructions. See
https://github.com/kubernetes/client-go
Write an application atop of the client-go clients. Note that client-go defines its own API objects, so if needed, please import API definitions from client-go rather than from the main repository, e.g.,
as the kubectl CLI does to locate and authenticate to the apiserver. See this
example
.
If the application is deployed as a Pod in the cluster, please refer to the
next section
.
To use
. See
example
.
Other languages
There are
for accessing the API from other languages.
See documentation for other libraries for how they authenticate.
Accessing the API from a Pod
When accessing the API from a pod, locating and authenticating
to the API server are somewhat different.
Please check
The previous section describes how to connect to the Kubernetes API server.
For information about connecting to other services running on a Kubernetes cluster, see
Access Cluster Services
.
The redirect capabilities have been deprecated and removed. Please use a proxy (see below) instead.
So Many Proxies
The
:
The
:
client to p

roxy uses HTTPS (or http if apiserver so configured)
The
kube proxy
:
proxies UDP and TCP
use UDP/TCP only
Feedback
Yes
No
.
or
.
Last modified July 21, 2022 at 1:41 PM PST:
Fix the grammar by using the verb form 'set up' where appropriate instead of the noun 'setup' (d6a1ba2a6d)
Go client
Home
Blog
Training
Partners
Community
CC BY 4.0

Contribute to K8s docs | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Contribute to K8s docs
1:
2:
2.1:
2.2:
2.3:
3:
3.1:
3.2:
4:
5:
5.1:
5.2:
6:
6.1:
6.2:
6.3:
6.4:
6.5:
6.6:
6.7:
7:
7.1:
7.2:
7.3:
7.4:
7.5:
7.6:
8:
9:
Kubernetes welcomes improvements from all contributors, new and experienced!
Note:
To learn more about contributing to Kubernetes in general, see the
contributor documentation
.
You can also read the
CNCF
page
about contributing to Kubernetes.
This website is maintained by
Kubernetes SIG Docs
.
Kubernetes documentation contributors:
Improve existing content
Create new content
Translate the documentation
Manage and publish the documentation parts of the Kubernetes release cycle
Anyone can open an issue about documentation, or contribute a change with a
pull request (PR) to the
GitHub repository
.
You need to be comfortable with
git
and
GitHub
to work effectively in the Kubernetes community.
To get involved with documentation:
Sign the CNCF
Contributor License Agreement
.
documentation repository
and the website's
static site generator
.
Make sure you understand the basic processes for
opening a pull request
and
reviewing changes
.
flowchart TB
subgraph third[Open PR]
U[ ] -.-
Q[Improve content] --- N[Create content]
N --- O[Translate docs]
O --- P[Manage/publish docs parts
of K8s release cycle]
end
subgraph second[Review]
T[ ] -.-
D[Look over the
K8s/website
repository] --- E[Check out the
Hugo static site
generator]
E --- F[Understand basic
GitHub commands]
F --- G[Review open PR
and change review
processes]
end
subgraph first[Sign up]
S[ ] -.-
B[Sign the CNCF
License Agreement] --- C[Join sig-docs
Slack channel]
C --- V[Join kubernetes-sig-docs
mailing list]
V --- M[Attend weekly
sig-docs calls
or slack meetings]
end
Contributor]) --&gt; first
A --&gt; second
A --&gt; third
A --&gt; H[Ask Questions!!!]
class A,B,C,D,E,F,G,H,M,Q,N,O,P,V grey
class S,T,U spacewhite
class first,second,third white
enabled
Figure 1. Getting started for a new contributor.
Figure 1 outlines a roadmap for new contributors. You can follow some or all of the steps for
Sign up
and
Review
. Now you are ready to open PRs that achieve your contribution objectives with some listed under
Open PR
. Again, questions are always welcome!
Some tasks require more trust and more access in

 the Kubernetes organization.
See
for more details about
roles and permissions.
Your first contribution
You can prepare for your first contribution by reviewing several steps beforehand. Figure 2 outlines the steps and the details follow.
subgraph second[First Contribution]
S[ ] -.-
G[Review PRs from other
K8s members] --&gt;
A[Check K8s/website
issues list for
good first PRs] --&gt; B[Open a PR!!]
end
subgraph first[Suggested Prep]
T[ ] -.-
D[Read contribution overview] --&gt;E[Read K8s content
and style guides]
E --&gt; F[Learn about Hugo page
content types
and shortcodes]
end
class A,B,D,E,F,G grey
enabled
Figure 2. Preparation for your first contribution.
Read the
Contribution overview
to
learn about the different ways you can contribute.
Check
issues list
for issues that make good entry points.
Open a pull request using GitHub
to existing documentation and learn more about filing issues in GitHub.
Review pull requests
from other
Kubernetes community members for accuracy and language.
Read the Kubernetes
content
and
style guides
page content types
and
.
Next steps
Learn to
work from a local clone
of the repository.
Document
features in a release
.
Participate in
SIG Docs
, and become a
member or reviewer
.
Start or help with a
.
Get involved with SIG Docs
SIG Docs
is the group of contributors who
publish and maintain Kubernetes documentation and the website. Getting
involved with SIG Docs is a great way for Kubernetes contributors (feature
development or otherwise) to have a large impact on the Kubernetes project.
SIG Docs communicates with different methods:
Join
#sig-docs
on the Kubernetes Slack instance
. Make sure to
introduce yourself!
Join the
kubernetes-sig-docs
mailing list
,
where broader discussions take place and official decisions are recorded.
Join the
SIG Docs video meeting
held every two weeks. Meetings are always announced on
#sig-docs
and added to the
Kubernetes community meetings calendar
. You'll need to download the
Zoom client
or dial in using a phone.
Join the SIG Docs async Slack standup meeting on those weeks when the in-person Zoom video meeting does not take place. Meetings are always announced on
#sig-docs
. You can contribute to any one of the threads up to 24 hours after meeting announcement.
Other ways to contribute
Visit the
Kubernetes community site
. Participate on Twitter or Stack Overflow, learn about local Kubernetes meetups and events, and more.
Read the
contributor cheatsheet
to get involved with Kubernetes feature development.
Visit the contributor site to learn more about
Kubernetes Contributors
and
additional contributor resources
.
Submit a
blog post or case study
.
1 - Suggesting content improvements
Click the
Click
.
After submi

tting, check in on your issue occasionally or turn on GitHub notifications.
.
Go to
GitHub
new issue.
with a
#
.
Follow the
2 - Contributing new content
content.
S[ ] -.-
end
T[ ] -.-
end
enabled
using
Hugo
.
GitHub
the
You can use
or
in our
/content/
. Each
.
see
.
must
read
the
and
.
your
git config
, and your git name and email must match those used for the
CNCF CLA.
Choose which Git branch

 to use
Scenario
Branch
main
v1.27
dev-1.27
branch.
#sig-docs
on Slack.
Note:
The
2.1 - Opening a pull request
Note:
.
section.
GitHub)]
0[ ] -.-
end
end
enabled
form.
Note:
Select
.
Select
.
The
The
Subject
The
Body
Leave the
checkbox selected.
Note:
.
Select
.

Go to the
tab.
#sig-docs
git
installed
S[ ] -.-
end
T[ ] -.-
end
enabled
Select
Fork
.
:
cd
website
1
website
upstream
remote:
cd
website
origin
and
upstream
and
's
:
Note:
.
main
with
.
.
.
.
#sig-docs
branch is
:
A

t any time, use the
git status
git status

Note:
, which can
Note:
Ctrl+C
,
hugo
Hugo
.
1
cd
Ctrl+C
,
end
to your PR]
end
enabled
Select
.
Select
.
From the
From the
compare
drop-dow

n menu, select your branch.
Select
.
Title
or
in the
Select the
button.
.
Netlify
.
Details
Details
.
-a
--amend
Use
Note:
If you use
Note:
,
#sig-docs
's
git status
,
, and
===
.
Note:
.
git status
Note:
,
#sig-docs
Commits
git log
Note:
vim
git rebase -i HEAD~&lt;number_of_commits_in_branch&gt;
-i
pick d875112ca Original com

mit

...

pick
squash
and
pick
.
Note:
.
To:
and
into
The
comments.
README.md
,
, and
Read
Reviewing
2.2 - Documenting a feature for a release
#sig-docs
.
the
community
in the
, which is
the official way to find out about all new features schedu

led to go into the
release.
content is not finalized until late in the release cycle, so use caution.
Open a
draft
dev-1.27
,
then click
.
PR(s) and
issue(s).
.
#sig-docs
table
to
dev-1.27
branch by the relea

se deadline, work with the
2.3 - Submitting blog posts and case studies
.
Check the
for
Sometimes this is a delicate balance. The
blog team
Posts about other CNCF projects may or may not be on topic. We recommend asking the blo

g team
The
license
Hugo
generator
.
.
Do
not
causes
---
layout
:
blog
title
:
date
:
slug
:
---
Add blog post
.
draft post
evergreen
:
true
Tutorials
CNCF
.
3 - Reviewing changes
This section describes how to review content.
3.1 - Reviewing pull requests
section in the Ku

bernetes website repository to see open pull requests.
Read the
and
Read the
S[ ] -.-
end
T[ ] -.-
end
by label]
enabled
Go to
.
work in progress
tab.
Details
link of the
line in the
Go to the
+
Leaving a "/approve" comment is recommende

d most of the time.
move
?
Content
Website
Docs Home
listing?
Other
;
(it's still OK to accept the change if it is genuinely an improvement).
nit:
.
;
3.2 - Reviewing for approvers and reviewers
SIG Docs
Reviewers
and
Approvers
do a few extra things
when reviewing a change.
Every week a specific docs approver volunteers to triage
and review pull requests. This
person is the "PR Wrangler" for the week. See the
PR Wrangler scheduler
for more information. To become a PR Wrangler, attend the weekly SIG Docs meeting and volunteer. Even if you are not on the schedule for the current week, you can still review pull
requests (PRs) that are not already under active review.
In addition to the rotation, a bot assigns reviewers and approvers
for the PR based on the owners for the affected files.
Revie

wing a PR
Kubernetes documentation follows the
Kubernetes code review process
.
Everything described in
Reviewing a pull request
applies, but Reviewers and Approvers should also do the following:
Using the
/assign
Prow command to assign a specific reviewer to a PR as needed. This is extra important
when it comes to requesting technical review from code contributors.
Note:
Look at the
reviewers
field in the front-matter at the top of a Markdown file to see who can
provide technical review.
Making sure the PR follows the
Content
and
Style
guides; link the author to the relevant part of the guide(s) if it doesn't.
Using the GitHub
Request Changes
option when applicable to suggest changes to the PR author.
Changing your review status in GitHub using the
/approve
or
/lgtm
Prow commands, if your suggestions are implemented.
Leaving PR comments is helpful, but there might be times when you need to commit
into another person's PR instead.
Do not "take over" for another person unless they explicitly ask
you to, or you want to resurrect a long-abandoned PR. While it may be faster
in the short term, it deprives the person of the chance to contribute.
The process you use depends on whether you need to edit a file that is already
in the scope of the PR, or a file that the PR has not yet touched.
You can't commit into someone else's PR if either of the following things is
true:
If the PR author pushed their branch directly to the
https://github.com/kubernetes/website/
repository. Only a reviewer with push access can commit to another user's PR.
Note:
Encourage the author to push their branch to their fork before
opening the PR next time.
The PR author explicitly disallows edits from approvers.
Prow commands for reviewing
Prow
is
the Kubernetes-based CI/CD system that runs jobs against pull requests (PRs). Prow
enables chatbot-style commands to handle GitHub actions across the Kubernetes
organization, like
adding and removing labels
, closing issues, and assigning an approver. Enter Prow commands as GitHub comments using the
/&lt;command-name&gt;
format.
The most common prow commands reviewers and approvers use are:
Prow Command
Role Restrictions
/lgtm
Organization members
Signals that you've finished reviewing a PR and are satisfied with the changes.
/approve
Approvers
Approves a PR for merging.
/assign
Anyone
Assigns a person to review or approve a PR
/close
Closes an issue or PR.
/hold
Anyone
Adds the
do-not-merge/hold
label, indicating the PR cannot be automatically merged.
/hold cancel
Anyone
Removes the
label.
To view the commands that you can use in a PR, see the
Prow Command Reference
.
In general, SIG Docs follows the
Kubernetes issue triage
process and uses the same labels.
This GitHub Issue
filter
finds issues that might need triage.
Triaging an issue
Validate the issue
Make sure the issue is about website documentation. Some issues can be closed quickly by
answering a question or pointing the reporter to a resource. See the
Support requests or code bug reports
Assess whether the issue has merit.
Add the
triage/needs-information
label if the issue doesn't have enough
detail to be actionable or the template is not filled out adequately.
Close the issue if it has both the
lifecycle/stale
and
labels.
Add a priority label (the
Issue Triage Guidelines
define priority labels in detail)
Issue labels
Label
priority/critical-urgent
Do this right now.
priority/important-soon
Do this within 3 months.
priority/important-longterm
Do this within 6 months.
priority/backlog
Deferrable indefinitely. Do when resources are available.
priority/awaiting-more-evidence
Placeholder for a potentially good issue so it doesn't get lost.
help
or
good first issue
Suitable for someone with very little Kubernetes or SIG Docs experience. See
Help Wanted and Good First

 Issue Labels
At your discretion, take ownership of an issue and submit a PR for it
(especially if it's quick or relates to work you're already doing).
If you have questions about triaging an issue, ask in
#sig-docs
on Slack or
the
kubernetes-sig-docs mailing list
.
To add a label, leave a comment in one of the following formats:
/&lt;label-to-add&gt;
/good-first-issue
)
/&lt;label-category&gt; &lt;label-to-add&gt;
/triage needs-information
or
/language ja
)
To remove a label, leave a comment in one of the following formats:
/remove-&lt;label-to-remove&gt;
/remove-help
)
/remove-&lt;label-category&gt; &lt;label-to-remove&gt;
/remove-triage needs-information
)
In both cases, the label must already exist. If you try to add a label that does not exist, the command is
silently ignored.
For a list of all labels, see the
website repository's Labels section
. Not all labels are used by SIG Docs.
Issue lifecycle labels
Issues are generally opened and closed quickly.
However, sometimes an issue is inactive after its opened.
Other times, an issue may need to remain open for longer than 90 days.
Label
After 90 days with no activity, an issue is automatically labeled as stale. The issue will be automatically closed if the lifecycle is not manually reverted using the
/remove-lifecycle stale
command.
lifecycle/frozen
An issue with this label will not become stale after 90 days of inactivity. A user manually adds this label to issues that need to remain open for much longer than 90 days, such as those with a
label.
Handling special issue types
SIG Docs encounters the following types of issues often enough to document how
to handle them.
Duplicate issues
If a single problem has one or more issues open for it, combine them into a single issue.
You should decide which issue to keep open (or
open a new issue), then move over all relevant information and link related issues.
Finally, label all other issues that describe the same problem with
triage/duplicate
and close them. Only having a single issue to work on reduces confusion
and avoids duplicate work on the same problem.
Dead link issues
If the dead link issue is in the API or
kubectl
documentation, assign them
/priority critical-urgent
until the problem is fully understood. Assign all other dead link issues
/priority important-longterm
, as they must be manually fixed.
Blog issues
We expect
entries to become
outdated over time. Therefore, we only maintain blog entries less than a year old.
If an issue is related to a blog entry that is more than one year old,
close the issue without fixing.
Some docs issues are actually issues with the underlying code, or requests for
assistance when something, for example a tutorial, doesn't work.
For issues unrelated to docs, close the issue with the
kind/support
label and a comment
directing the requester to support venues (Slack, Stack Overflow) and, if
relevant, the repository to file an issue for bugs with features (
is a great place to start).
Sample response to a request for support:
This issue sounds more like a request for support and less
like an issue specifically for docs. I encourage you to bring
your question to the `#kubernetes-users` channel in
[Kubernetes slack](https://slack.k8s.io/). You can also search
resources like
[Stack Overflow](https://stackoverflow.com/questions/tagged/kubernetes)
for answers to similar questions.

You can also open issues for Kubernetes functionality in
https://github.com/kubernetes/kubernetes.

If this is a documentation issue, please re-open this issue.
Sample code bug report response:
This sounds more like an issue with the code than an issue with
the documentation. Please open an issue at
https://github.com/kubernetes/kubernetes/issues.

If this is 

a documentation issue, please re-open this issue.
4 - Localizing Kubernetes documentation
This page shows you how to
localize
the docs for a different language.
Contribute to an existing localization
You can help add or improve the content of an existing localization. In
, you can find a channel for each
localization. There is also a general
SIG Docs Localizations Slack
channel
where you
can say hello.
Note:
For extra details on how to contribute to a specific localization,
look for a localized version of this page.
Find your two-letter language code
First, consult the
ISO 639-1
standard
to find your
localization's two-letter language code. For example, the two-letter code for
Korean is
ko
.
Fork and clone the repo
First,
create your own
fork
of the
Then, clone your fork and
cd
into it:
git clone https://github.com/&lt;username&gt;/website
cd
website
The website content directory includes subdirectories for each language. The
localization you want to help out with is inside
content/&lt;two-letter-code&gt;
.
Suggest changes
Create or update your chosen localized page based on the English original. See
translating content
If you notice a technical inaccuracy or other problem with the upstream
(English) documentation, you should fix the upstream documentation first and
then repeat the equivalent fix by updating the localization you're working on.
Limit changes in a pull requests to a single localization. Reviewing pull
requests that change content in multiple localizations is problematic.
Follow
Suggesting Content Improvements
to propose changes to that localization. The process is similar to proposing
changes to the upstream (English) content.
Start a new localization
If you want the Kubernetes documentation localized into a new language, here's
what you need to do.
Because contributors can't approve their own pull requests, you need
at least
two contributors
to begin a localization.
All localization teams must be self-sufficient. The Kubernetes website is happy
to host your work, but it's up to you to translate it and keep existing
localized content current.
You'll need to know the two-letter language code for your language. Consult the
to find your localization's two-letter language code. For example, the
two-letter code for Korean is
ko
.
When you start a new localization, you must localize all the
minimum required content
before
the Kubernetes project can publish your changes to the live
website.
SIG Docs can help you work on a separate branch so that you
can incrementally work towards that goal.
Find community
Let Kubernetes SIG Docs know you're interested in creating a localization! Join
the
SIG Docs Slack channel
and
the
SIG Docs Localizations Slack channel
.
Other localization teams are happy to help you get started and answer your
questions.
Please also consider participating in the
SIG Docs Localization Subgroup meeting
.
The mission of the SIG Docs localization subgroup is to work across the SIG Docs
localization teams to collaborate on defining and documenting the processes for
creating localized contribution guides. In addition, the SIG Docs localization
subgroup looks for opportunities to create and share common tools across
localization teams and identify new requirements for the SIG Docs Leadership
team. If you have questions about this meeting, please inquire on the
SIG Docs
Localizations Slack channel
.
You can also create a Slack channel for your localization in the
repository. For an example of adding a Slack channel, see
the PR for
adding a channel for Persian
.
Join the Kubernetes GitHub organization
When you've opened a localization PR, you can become members of the Kubernetes
GitHub organization. Each person on the team needs to create their own
in the
Add your localization team in GitHub
Next, add your Kubernetes localization team to
sig-docs

/teams.yaml
.
For an example of adding a localization team, see the PR to add the
Spanish localization team
.
Members of
@kubernetes/sig-docs-**-owners
can approve PRs that change content
within (and only within) your localization directory:
/content/**/
. For each
localization, The
@kubernetes/sig-docs-**-reviews
team automates review
assignments for new PRs. Members of
@kubernetes/website-maintainers
can create
new localization branches to coordinate translation efforts. Members of
@kubernetes/website-milestone-maintainers
can use the
/milestone
Prow command
to assign a milestone to issues or PRs.
Configure the workflow
Next, add a GitHub label for your localization in the
kubernetes/test-infra
repository. A label lets you filter issues and pull requests for your specific
language.
For an example of adding a label, see the PR for adding the
Italian language label
.
Modify the site configuration
The Kubernetes website uses Hugo as its web framework. The website's Hugo
configuration resides in the
file. You'll need to modify
to support a new localization.
Add a configuration block for the new language to
under the
existing
[languages]
block. The German block, for example, looks like:
[languages.de]
title =
description =
"Produktionsreife Container-Verwaltung"
languageName =
"Deutsch (German)"
languageNameLatinScript =
"Deutsch"
contentDir =
"content/de"
weight =
8
The language selection bar lists the value for
languageName
. Assign "language
name in native script and language (English language name in Latin script)" to
languageName = "한국어 (Korean)"
or
languageName = "Deutsch (German)"
.
languageNameLatinScript
can be used to access the language name in Latin
script and use it in the theme. Assign "language name in latin script" to
languageNameLatinScript ="Korean"
or
languageNameLatinScript = "Deutsch"
.
When assigning a
weight
parameter for your block, find the language block with
the highest weight and add 1 to that value.
For more information about Hugo's multilingual support, see
"
Multilingual Mode
".
Add a new localization directory
Add a language-specific subdirectory to the
content
folder in
the repository. For example, the two-letter code for German is
de
:
mkdir content/de
You also need to create a directory inside
data/i18n
for
localized strings
; look at existing localizations
for an example. To use these new strings, you must also create a symbolic link
from
i18n/&lt;localization&gt;.toml
to the actual string configuration in
data/i18n/&lt;localization&gt;/&lt;localization&gt;.toml
(remember to commit the symbolic
link).
For example, for German the strings live in
data/i18n/de/de.toml
, and
i18n/de.toml
is a symbolic link to
.
Localize the community code of conduct
Open a PR against the
cncf/foundation
repository to add the code of conduct in your language.
Setting up the OWNERS files
To set the roles of each user contributing to the localization, create an
OWNERS
file inside the language-specific subdirectory with:
reviewers
: A list of kubernetes teams with reviewer roles, in this case,
the
sig-docs-**-reviews
team created in
.
approvers
: A list of kubernetes teams with approvers roles, in this case,
the
sig-docs-**-owners
.
labels
: A list of GitHub labels to automatically apply to a PR, in this
case, the language label created in
.
More information about the
OWNERS
file can be found at
go.k8s.io/owners
.
The
Spanish OWNERS file
, with
language code
es
, looks like this:
# See the OWNERS docs at https://go.k8s.io/owners
# This is the localization project for Spanish.
# Teams and members are visible at https://github.com/orgs/kubernetes/teams.
reviewers
:
- sig-docs-es-reviews
approvers
:
- sig-docs-es-owners
labels
:
- language/es
After adding the language-specific

OWNERS
file, update the
root
file with the new
Kubernetes teams for the localization,
and
.
For each team, add the list of GitHub users requested in
,
in alphabetical order.
--- a/OWNERS_ALIASES
+++ b/OWNERS_ALIASES
@@ -48,6 +48,14 @@ aliases:
- stewart-yu
- xiangpengzhao
- zhangxiaoyu-zidif
+  sig-docs-es-owners: # Admins for Spanish content
+    - alexbrand
+    - raelga
+  sig-docs-es-reviews: # PR reviews for Spanish content
+    - electrocucaracha
+    - glo-pena
+    - raelga
sig-docs-fr-owners: # Admins for French content
- perriea
- remyleone
Next,
open a pull request
(PR) to add a localization to the
repository. The PR must
include all the
before it
can be approved.
For an example of adding a new localization, see the PR to enable
docs in French
.
Add a localized README file
To guide other localization contributors, add a new
README-**.md
to the top
level of
, where
**
is the two-letter language code. For example, a German README file would be
README-de.md
.
Guide localization contributors in the localized
file.
Include the same information contained in
README.md
as well as:
A point of contact for the localization project
Any information specific to the localization
After you create the localized README, add a link to the file from the main
English
README.md
, and include contact information in English. You can provide
a GitHub ID, email address,
, or another
method of contact. You must also provide a link to your localized Community Code
of Conduct.
Launching your new localization
When a localization meets the requirements for workflow and minimum output, SIG
Docs does the following:
Enables language selection on the website.
Publicizes the localization's availability through
Cloud Native Computing Foundation
(CNCF)
channels, including the
Kubernetes blog
.
Translating content
Localizing
all
the Kubernetes documentation is an enormous task. It's okay to
start small and expand over time.
Minimum required content
At a minimum, all localizations must include:
URLs
Home
All heading and subheading URLs
Setup
Tutorials
,
Site strings
All site strings
in a new localized TOML file
Releases
Translated documents must reside in their own
content/**/
subdirectory, but otherwise, follow the
same URL path as the English source. For example, to prepare the
tutorial for translation into German,
create a subfolder under the
content/de/
folder and copy the English source:
mkdir -p content/de/docs/tutorials
cp content/en/docs/tutorials/kubernetes-basics.md content/de/docs/tutorials/kubernetes-basics.md
Translation tools can speed up the translation process. For example, some
editors offer plugins to quickly translate text.
Caution:
Machine-generated translation is insufficient on its own. Localization requires
extensive human review to meet minimum standards of quality.
To ensure accuracy in grammar and meaning, members of your localization team
should carefully review all machine-generated translations before publishing.
Source files
Localizations must be based on the English files from a specific release
targeted by the localization team. Each localization team can decide which
release to target, referred to as the
target version
below.
To find source files for your target version:
Navigate to the Kubernetes website repository at
https://github.com/kubernetes/website
.
Select a branch for your target version from the following table:
Target version
Branch
Latest version
main
Previous version
release-1.25
Next version
dev-1.27
The
main
branch holds content for the current release
v1.26
.
The release team creates a
release-1.26
branch before the next
release: v1.27.
Site strings in i18n
Localizat

ions must include the contents of
data/i18n/en/en.toml
in a new language-specific file. Using German as an example:
.
Add a new localization directory and file to
data/i18n/
. For example, with
German (
de
):
mkdir -p data/i18n/de
cp data/i18n/en/en.toml data/i18n/de/de.toml
Revise the comments at the top of the file to suit your localization, then
translate the value of each string. For example, this is the German-language
placeholder text for the search form:
[ui_search_placeholder]
other =
"Suchen"
Localizing site strings lets you customize site-wide text and features: for
example, the legal copyright text in the footer on each page.
Language-specific localization guide
As a localization team, you can formalize the best practices your team follows
by creating a language-specific localization guide.
Korean Localization Guide
, which
includes content on the following subjects:
Sprint cadence and releases
Branch strategy
Pull request workflow
Glossary of localized and non-localized terms
Markdown conventions
Kubernetes API object terminology
Language-specific Zoom meetings
If the localization project needs a separate meeting time, contact a SIG Docs
Co-Chair or Tech Lead to create a new reoccurring Zoom meeting and calendar
invite. This is only needed when the team is large enough to sustain and require
a separate meeting.
Per CNCF policy, the localization teams must upload their meetings to the SIG
Docs YouTube playlist. A SIG Docs Co-Chair or Tech Lead can help with the
process until SIG Docs automates it.
Branching strategy
Because localization projects are highly collaborative efforts, we
encourage teams to work in shared localization branches - especially
when starting out and the localization is not yet live.
To collaborate on a localization branch:
A team member of
opens a localization branch from a source branch on
.
Your team approvers joined the
team when
you
added your localization team
to
the
We recommend the following branch naming scheme:
dev-&lt;source version&gt;-&lt;language code&gt;.&lt;team milestone&gt;
For example, an approver on a German localization team opens the localization
branch
dev-1.12-de.1
directly against the
repository,
based on the source branch for Kubernetes v1.12.
Individual contributors open feature branches based on the localization
branch.
For example, a German contributor opens a pull request with changes to
kubernetes:dev-1.12-de.1
from
username:local-branch-name
.
Approvers review and merge feature branches into the localization branch.
Periodically, an approver merges the localization branch with its source
branch by opening and approving a new pull request. Be sure to squash the
commits before approving the pull request.
Repeat steps 1-4 as needed until the localization is complete. For example,
subsequent German localization branches would be:
dev-1.12-de.2
,
dev-1.12-de.3
, etc.
Teams must merge localized content into the same branch from which the content
was sourced. For example:
A localization branch sourced from
main
must be merged into
main
.
.
Note:
If your localization branch was created from
main
branch, but it is not merged
into
main
before the new release branch
created,
merge it into both
main
and new release branch
. To
merge your localization branch into the new release branch
, you need to switch the upstream branch of your
localization branch to
.
At the beginning of every team milestone, it's helpful to open an issue
comparing upstream changes between the previous localization branch and the
current localization branch. There are two scripts for comparing upstream
changes.
upstream_changes.py
is useful for checking the changes made to a

 specific file. And
diff_l10n_branches.py
is useful for creating a list of outdated files for a specific localization
branch.
While only approvers can open a new localization branch and merge pull requests,
anyone can open a pull request for a new localization branch. No special
permissions are required.
For more information about working from forks or directly from the repository,
see
"fork and clone the repo"
.
Upstream contributions
SIG Docs welcomes upstream contributions and corrections to the English source.
5 - Participating in SIG Docs
member
,
reviewer
, or
approver
.
teams
on GitHub:
@name
OWNERS
files.
uses
two
:
approve
OWNERS
and
OWNERS
.
lgtm
and
approve
hold
prevent automatic merging of a giv

en pull request (by adding a
/hold
comment
/lgtm
comment).
Any Kubernetes member can add the
lgtm
/lgtm
comment.
/approve
or
.
5.1 - Roles and responsibilities
Anyone
Slack
or the
.
After
.
Members
.
Anyone
Use the
/lgtm
Note:
Using
/lgtm
Use the
/hold
Use the
/assign
:
Find two
reviewers
or
approvers
to
sponsor
your
or on the
.
Note:
)
+1
Note:
Reviewers
Anyone
and
Members
Note:
You can be a SIG Docs reviewer, or a reviewer for docs i

n a specific subject area.
.
Using
/lgtm
/lgtm
/approve
A
/lgtm
lgtm
label.
,
To apply:
file
in the
Note:
.
).
Approvers
Anyone
,
Members
and
Reviewers
/approve
comment
/lgtm
/lgtm
/lgtm
/approve
Warning:
.
/approve
a PR.
.
,
To apply:
file in the
Note:
.
Read about
5.2 - PR wranglers
SIG Docs
approvers
.
Duties
Review
Style
and
Content
guides.
size/

XS
size/XXL
Make sure PR contributors sign the
CLA
.
Use
this
sig/
label(s).
SIG
on the PR.
Use the
/approve
/lgtm
issues as
.
:
Needs LGTM
:
:
/approve
Quick Wins
dev-
using:

/close
Note:
The
Others can 

reach out on the
for requesting to shadow an assigned PR Wrangler for a specific week. Feel free to reach out to Brad Topol (
.
.
6 - Documentation style overview
6.1 - Documentation Content Guide
This page contains guidelines for Kubernetes documentation.
If you have questions about what's allowed, join the #sig-docs channel in
and ask!
You can register for Kubernetes Slack at
https://slack.k8s.io/
.
For information on creating new content for the Kubernetes
docs, follow the
.
Overview
Source for the Kubernetes website, including the docs, resides in the
Located in the
kubernetes/website/content/&lt;language_code&gt;/docs
folder, the
majority of Kubernetes documentation is specific to the
project
.
What's allowed
Kubernetes docs allow content for third-party projects only when:
Content documents software in the Kubernetes project
Content documents software that's out of project but necessary for Kubernetes to function
Content is canonical on kubernetes.io, or links to canonical content elsewhere
Third party content
Kubernetes documentation includes applied examples of projects in the Kubernetes project—projects that live in the
and
kubernetes-sigs
GitHub organizations.
Links to active content in the Kubernetes project are always allowed.
Kubernetes requires some third party content to function. Examples include container runtimes (containerd, CRI-O, Docker),
networking policy
(CNI plugins),
Ingress controllers
, and
logging
.
Docs can link to third-party open source software (OSS) outside the Kubernetes project only if it's necessary for Kubernetes to function.
Dual sourced content
Wherever possible, Kubernetes docs link to canonical sources instead of hosting
dual-sourced content.
Dual-sourced content requires double the effort (or more!) to maintain
and grows stale more quickly.
Note:
If you're a maintainer for a Kubernetes project and need help hosting your own docs,
ask for help in
#sig-docs on Kubernetes Slack
.
If you have questions about allowed content, join the
#sig-docs channel and ask!
Read the
.
6.2 - Documentation Style Guide
This page gives writing style guidelines for the Kubernetes documentation.
These are guidelines, not rules. Use your best judgment, and feel free to
propose changes to this document in a pull request.
For additional information on creating new content for the Kubernetes
documentation, read the
.
Changes to the style guide are made by SIG Docs as a group. To propose a change
or addition,
add it to the agenda
for an upcoming SIG Docs meeting, and attend the meeting to participate in the
discussion.
Note:
Goldmark Markdown Renderer
with some adjustments along with a few
Hugo Shortcodes
to support
glossary entries, tabs, and representing feature state.
Language
Kubernetes documentation has been translated into multiple languages
(see
Localization READMEs
).
The way of localizing the docs for a different language is described in
Localizing Kubernetes Documentation
.
The English-language documentation uses U.S. English spelling and grammar.
Documentation formatting standards
Use upper camel case for API objects
When you refer specifically to interacting with an API object, use
UpperCamelCase
, also known as Pascal case. You may see different capitalization, such as "configMap", in the
. When writing general documentation, it's better to use upper camel case, calling it "ConfigMap" instead.
When you are generally discussing an API object, use
se

ntence-style capitalization
.
The following examples focus on capitalization. For more information about formatting API object names, review the related guidance on
Code Style
.
Do and Don't - Use Pascal case for API objects
Do
Don't
The HorizontalPodAutoscaler resource is responsible for ...
The Horizontal pod autoscaler is responsible for ...
A PodList object is a list of pods.
A Pod List object is a list of pods.
The Volume object contains a
hostPath
field.
The volume object contains a hostPath field.
Every ConfigMap object is part of a namespace.
Every configMap object is part of a namespace.
For managing confidential data, consider using the Secret API.
For managing confidential data, consider using the secret API.
Use angle brackets for placeholders
Use angle brackets for placeholders. Tell the reader what a placeholder
represents, for example:
Display information about a pod:
kubectl describe pod &lt;pod-name&gt; -n &lt;namespace&gt;
If the namespace of the pod is
default
, you can omit the '-n' parameter.
Use bold for user interface elements
Do and Don't - Bold interface elements
Do
Don't
Click
Fork
.
Click "Fork".
Select
Other
.
Select "Other".
Use italics to define or introduce new terms
Do and Don't - Use italics for new terms
Do
Don't
A
cluster
is a set of nodes ...
A "cluster" is a set of nodes ...
These components form the
.
.
Use code style for filenames, directories, and paths
Do and Don't - Use code style for filenames, directories, and paths
Do
Don't
Open the
envars.yaml
file.
Open the envars.yaml file.
Go to the
/docs/tutorials
Go to the /docs/tutorials directory.
Open the
/_data/concepts.yaml
file.
Open the /_data/concepts.yaml file.
Use the international standard for punctuation inside quotes
Do and Don't - Use the international standard for punctuation inside quotes
Do
Don't
events are recorded with an associated "stage".
events are recorded with an associated "stage."
The copy is called a "fork".
The copy is called a "fork."
Inline code formatting
Use code style for inline code, commands, and API objects
For inline code in an HTML document, use the
&lt;code&gt;
tag. In a Markdown
document, use the backtick (
`
).
Do and Don't - Use code style for inline code, commands, and API objects
Do
Don't
The
command creates a
Pod
.
The "kubectl run" command creates a pod.
The kubelet on each node acquires a
Lease
…
The kubelet on each node acquires a lease…
A
represents durable storage…
A Persistent Volume represents durable storage…
For declarative management, use
.
For declarative management, use "kubectl apply".
Enclose code samples with triple backticks. (```)
Enclose code samples with any other syntax.
Use single backticks to enclose inline code. For example,
var example = true
.
Use two asterisks (
**
_
) to enclose inline code. For example,
.
Use triple backticks before and after a multi-line block of code for fenced code blocks.
Use multi-line blocks of code to create diagrams, flowcharts, or other illustrations.
Use meaningful variable names that have a context.
Use variable names such as 'foo','bar', and 'baz' that are not meaningful and lack context.
Remove trailing spaces in the code.
Add trailing spaces in the code, where these are important, because the screen reader will read out the spaces as well.
Note:
The website supports syntax highlighting for code samples, but specifying a language is optional. Syntax highlighting in the code block should conform to the
contrast guidelines.
Use code style for object field names and namespaces
Do and Don't - Use code style for object field names
Do
Don't
Set the value of the
replicas
Set the value of the "replicas" field in the configuration file.
exec
field is an ExecAction object.
The value of the "exec" field is an ExecAction object.
Run the process as a DaemonSet

 in the
Run the process as a DaemonSet in the kube-system namespace.
Use code style for Kubernetes command tool and component names
Do and Don't - Use code style for Kubernetes command tool and component names
Do
Don't
The kubelet preserves node stability.
The
kubelet
preserves node stability.
The
kubectl
handles locating and authenticating to the API server.
The kubectl handles locating and authenticating to the apiserver.
Run the process with the certificate,
kube-apiserver --client-ca-file=FILENAME
.
Run the process with the certificate, kube-apiserver --client-ca-file=FILENAME.
Starting a sentence with a component tool or component name
Do and Don't - Starting a sentence with a component tool or component name
Do
Don't
The
kubeadm
tool bootstraps and provisions machines in a cluster.
kubeadm
The kube-scheduler is the default scheduler for Kubernetes.
kube-scheduler is the default scheduler for Kubernetes.
Use a general descriptor over a component name
Do and Don't - Use a general descriptor over a component name
Do
Don't
The Kubernetes API server offers an OpenAPI spec.
The apiserver offers an OpenAPI spec.
Aggregated APIs are subordinate API servers.
Aggregated APIs are subordinate APIServers.
Use normal style for string and integer field values
For field values of type string or integer, use normal style without quotation marks.
Do and Don't - Use normal style for string and integer field values
Do
Don't
to Always.
to "Always".
image
to nginx:1.16.
image
to
.
replicas
field to 2.
replicas
field to
2
.
Referring to Kubernetes API resources
This section talks about how we reference API resources in the documentation.
Clarification about "resource"
Kubernetes uses the word "resource" to refer to API resources, such as
pod
,
, and so on. We also use "resource" to talk about CPU and memory requests and limits. Always refer to API resources as "API resources" to avoid confusion with CPU and memory resources.
When to use Kubernetes API terminologies
The different Kubernetes API terminologies are:
Resource type: the name used in the API URL (such as
pods
,
)
Resource: a single instance of a resource type (such as
pod
,
secret
)
Object: a resource that serves as a "record of intent". An object is a desired state for a specific part of your cluster, which the Kubernetes control plane tries to maintain.
Always use "resource" or "object" when referring to an API resource in docs. For example, use "a
Secret
object" over just "a
Secret
".
API resource names
Always format API resource names using
, also known as PascalCase, and code formatting.
tag. In a Markdown document, use the backtick (
`
).
Don't split an API object name into separate words. For example, use
, not Pod Template List.
For more information about PascalCase and code formatting, please review the related guidance on
and
.
For more information about Kubernetes API terminologies, please review the related guidance on
.
Code snippet formatting
Don't include the command prompt
Do and Don't - Don't include the command prompt
Do
Don't
$ kubectl get pods
Separate commands from output
=
wide
Versioning Kubernetes examples
Code examples and configuration examples that include version information should be consistent with the accompanying text.
If the information is ver

sion specific, the Kubernetes version needs to be defined in the
Task template
or the
Tutorial template
. Once the page is saved, the
section is shown as
.
To specify the Kubernetes version for a task or tutorial page, include
min-kubernetes-server-version
in the front matter of the page.
If the example YAML is in a standalone file, find and review the topics that include it as a reference.
Verify that any topics using the standalone YAML have the appropriate version information defined.
If a stand-alone YAML file is not referenced from any topics, consider deleting it instead of updating it.
For example, if you are writing a tutorial that is relevant to Kubernetes version 1.8, the front-matter of your markdown file should look something like:
---
title
:
&lt;your tutorial title here&gt;
:
v1.8
---
In code and configuration examples, do not include comments about alternative versions.
Be careful to not include incorrect statements in your examples as comments, such as:
:
v1
# earlier versions use...
kind
:
Pod
...
Kubernetes.io word list
A list of Kubernetes-specific terms and words to be used consistently across the site.
Term
Usage
Kubernetes should always be capitalized.
Docker
Docker should always be capitalized.
SIG Docs
SIG Docs rather than SIG-DOCS or other variations.
On-premises
On-premises or On-prem rather than On-premise or other variations.
Shortcodes
Hugo
help create different rhetorical appeal levels. Our documentation supports three different shortcodes in this category:
Note
{{&lt; note &gt;}}
,
Caution
{{&lt; caution &gt;}}
, and
Warning
{{&lt; warning &gt;}}
.
Surround the text with an opening and closing shortcode.
Use the following syntax to apply a style:
No need to include a prefix; the shortcode automatically provides one. (Note:, Caution:, etc.)
{{&lt; /note &gt;}}
Note:
The prefix you choose is the same text for the tag.
Note
Use
to highlight a tip or a piece of information that may be helpful to know.
You can _still_ use Markdown inside these callouts.
Note:
You can
still
use Markdown inside these callouts.
in a list:
1. Use the note shortcode in a list

1. A second item with an embedded note

   {{&lt; note &gt;}}
   Warning, Caution, and Note shortcodes, embedded in lists, need to be indented four spaces. See [Common Shortcode Issues](#common-shortcode-issues).
   {{&lt; /note &gt;}}

1. A third item in a list

1. A fourth item in a list
Use the note shortcode in a list
A second item with an embedded note
Note:
Warning, Caution, and Note shortcodes, embedded in lists, need to be indented four spaces. See
Common Shortcode Issues
.
A third item in a list
A fourth item in a list
Caution
Use
to call attention to an important piece of information to avoid pitfalls.
The callout style only applies to the line directly above the tag.
{{&lt; /caution &gt;}}
Caution:
Warning
Use
to indicate danger or a piece of information that is crucial to follow.
Beware.
{{&lt; /warning &gt;}}
Warning:
Beware.
This button lets users run Minikube in their browser using the Katacoda Terminal.
It lowers the barrier of entry by allowing users to use Minikube with one click instead of going through the complete
Minikube and Kubectl installation process locally.
The Embedded Live Environment is configured to run
and lets users complete tutorials in the same window
as the documentation.
Caution:
The session is limited to 15 minutes.
{{&lt; kat-button 

&gt;}}
Ordered Lists
Shortcodes will interrupt numbered lists unless you indent four spaces before the notice and the tag.
1. Preheat oven to 350˚F

1. Prepare the batter, and pour into springform pan.
   `{{&lt; note &gt;}}Grease the pan for best results.{{&lt; /note &gt;}}`

1. Bake for 20-25 minutes or until set.
Preheat oven to 350˚F
Prepare the batter, and pour into springform pan.
Note:
Grease the pan for best results.
Bake for 20-25 minutes or until set.
Include Statements
Shortcodes inside include statements will break the build. You must insert them in the parent document, before and after you call the include. For example:
{{&lt; include "task-tutorial-prereqs.md" &gt;}}
Markdown elements
Line breaks
Use a single newline to separate block-level content like headings, lists, images, code blocks, and others. The exception is second-level headings, where it should be two newlines. Second-level headings follow the first-level (or the title) without any preceding paragraphs or texts. A two line spacing helps visualize the overall structure of content in a code editor better.
Headings and titles
People accessing this documentation may use a screen reader or other assistive technology (AT).
Screen readers
are linear output devices, they output items on a page one at a time. If there is a lot of content on a page, you can use headings to give the page an internal structure. A good page structure helps all readers to easily navigate the page or filter topics of interest.
Do and Don't - Headings
Do
Don't
Update the title in the front matter of the page or blog post.
Use first level heading, as Hugo automatically converts the title in the front matter of the page into a first-level heading.
Use ordered headings to provide a meaningful high-level outline of your content.
Use headings level 4 through 6, unless it is absolutely necessary. If your content is that detailed, it may need to be broken into separate articles.
Use pound or hash signs (
#
) for non-blog post content.
Use underlines (
---
or
===
) to designate first-level headings.
Use sentence case for headings in the page body. For example,
Use title case for headings in the page body. For example,
Extend Kubectl With Plugins
Use title case for the page title in the front matter. For example,
title: Kubernetes API Server Bypass Risks
Use sentence case for page titles
in the front matter. For example, don't use
title: Kubernetes API server bypass risks
Paragraphs
Do and Don't - Paragraphs
Do
Don't
Try to keep paragraphs under 6 sentences.
Indent the first paragraph with space characters. For example, ⋅⋅⋅Three spaces before a paragraph will indent it.
Use three hyphens (
---
) to create a horizontal rule. Use horizontal rules for breaks in paragraph content. For example, a change of scene in a story, or a shift of topic within a section.
Use horizontal rules for decoration.
Links
Do and Don't - Links
Do
Don't
Write hyperlinks that give you context for the content they link to. For example: Certain ports are open on your machines. See
Check required ports
Use ambiguous terms such as "click here". For example: Certain ports are open on your machines. See
here
Write Markdown-style links:
[link text](URL)
[Hugo shortcodes](/docs/contribute/style/hugo-shortcodes/#table-captions)
and the output is
.
Write HTML-style links:
&lt;a href="/media/examples/link-element-example.css" target="_blank"&gt;Visit our tutorial!&lt;/a&gt;
, or create links that open in new tabs or windows. For example:
[example website](https://example.com){target="_blank"}
Lists
Group items in a list that are related to each other and need to appear in a specific order or to indicate a correlation between multiple items. When a screen reader comes across a list—whether it is an ordered or unorder

ed list—it will be announced to the user that there is a group of list items. The user can then use the arrow keys to move up and down between the various items in the list.
Website navigation links can also be marked up as list items; after all they are nothing but a group of related links.
End each item in a list with a period if one or more items in the list are complete sentences. For the sake of consistency, normally either all items or none should be complete sentences.
Note:
Ordered lists that are part of an incomplete introductory sentence can be in lowercase and punctuated as if each item was a part of the introductory sentence.
Use the number one (
1.
) for ordered lists.
Use (
+
), (
*
), or (
-
) for unordered lists.
Leave a blank line after each list.
Indent nested lists with four spaces (for example, ⋅⋅⋅⋅).
List items may consist of multiple paragraphs. Each subsequent paragraph in a list item must be indented by either four spaces or one tab.
Tables
The semantic purpose of a data table is to present tabular data. Sighted users can quickly scan the table but a screen reader goes through line by line. A table caption is used to create a descriptive title for a data table. Assistive technologies (AT) use the HTML table caption element to identify the table contents to the user within the page structure.
Add table captions using
for tables.
Content best practices
This section contains suggested best practices for clear, concise, and consistent content.
Use present tense
Do and Don't - Use present tense
Do
Don't
This command starts a proxy.
This command will start a proxy.
Exception: Use future or past tense if it is required to convey the correct
meaning.
Use active voice
Do and Don't - Use active voice
Do
Don't
You can explore the API using a browser.
The API can be explored using a browser.
The YAML file specifies the replica count.
The replica count is specified in the YAML file.
Exception: Use passive voice if active voice leads to an awkward construction.
Use simple and direct language
Use simple and direct language. Avoid using unnecessary phrases, such as saying "please."
Do and Don't - Use simple and direct language
Do
Don't
To create a ReplicaSet, ...
In order to create a ReplicaSet, ...
See the configuration file.
Please see the configuration file.
View the pods.
With this next command, we'll view the pods.
Address the reader as "you"
Do and Don't - Addressing the reader
Do
Don't
You can create a Deployment by ...
We'll create a Deployment by ...
In the preceding output, you can see...
In the preceding output, we can see ...
Avoid Latin phrases
Prefer English terms over Latin abbreviations.
Do and Don't - Avoid Latin phrases
Do
Don't
For example, ...
e.g., ...
That is, ...
i.e., ...
Exception: Use "etc." for et cetera.
Patterns to avoid
Avoid using "we"
Using "we" in a sentence can be confusing, because the reader might not know
whether they're part of the "we" you're describing.
Do and Don't - Patterns to avoid
Do
Don't
Version 1.4 includes ...
In version 1.4, we have added ...
Kubernetes provides a new feature for ...
We provide a new feature ...
This page teaches you how to use pods.
In this page, we are going to learn about pods.
Avoid jargon and idioms
Some readers speak English as a second language. Avoid jargon and idioms to help them understand better.
Do and Don't - Avoid jargon and idioms
Do
Don't
Internally, ...
Under the hood, ...
Create a new cluster.
Turn up a new cluster.
Avoid statements about the future
Avoid making promises or giving hints about the future. If you need to talk about
an alpha feature, put the text under a heading that identifies it as alpha
An exception to this rule is documentation about announced deprecations
targeting removal in future versions. One example of documentation like this
is the
Deprecated API migration guide
.
Avoid statements that will soon be out of date
Avoid words like "currently" and "new." A feature that is new today might

 not be
considered new in a few months.
Do and Don't - Avoid statements that will soon be out of date
Do
Don't
In version 1.4, ...
In the current version, ...
The Federation feature provides ...
The new Federation feature provides ...
Avoid words that assume a specific level of understanding
Avoid words such as "just", "simply", "easy", "easily", or "simple". These words do not add value.
Do and Don't - Avoid insensitive words
Do
Don't
Include one command in ...
Include just one command in ...
Run the container ...
Simply run the container ...
You can remove ...
You can easily remove ...
These steps ...
These simple steps ...
writing a new topic
.
using page templates
.
.
6.3 - Diagram Guide
generate
.svg
or
.png
S[ ]-.-
C[build
diagrams
D[on-line
end
diagrams]
and
captions]
click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank
click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICA

geCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank
enabled
Using
.
Using the
.
.
process.
Note:
Mermaid
Mermaid
to add, edit and collaborate on diagrams for new or existing documentation.
Note:
below.
The
is
.svg
or
.png
files.
Note:
The live editor is the easiest and fastest way to create and ed

it Mermaid diagrams.
graph TB
B[Inline
A --&gt; B
A --&gt; C
A --&gt; D
click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBBW0NvbnRyaWJ1dG9yXVxuICAgIEJbSW5saW5lPGJyPjxicj5NZXJtYWlkIGNvZGU8YnI-YWRkZWQgdG8gLm1kIGZpbGVdXG4gICAgQ1tNZXJtYWlkK1NWRzxicj48YnI-QWRkIG1lcm1haWQtZ2VuZXJhdGVkPGJyPnN2ZyBmaWxlIHRvIC5tZCBmaWxlXVxuICAgIERbRXh0ZXJuYWwgdG9vbDxicj48YnI-QWRkIGV4dGVybmFsLXRvb2wtPGJyPmdlbmVyYXRlZCBzdmcgZmlsZTxicj50byAubWQgZmlsZV1cblxuICAgIEEgLS0-IEJcbiAgICBBIC0tPiBDXG4gICAgQSAtLT4gRFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3giLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
enabled
Inline
method.
graph LR
click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggTFJc

biAgICBBWzEuIFVzZSBsaXZlIGVkaXRvcjxicj4gdG8gY3JlYXRlL2VkaXQ8YnI-ZGlhZ3JhbV0gLS0-XG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdIC0tPlxuICAgIENbMy4gQ29weSBNZXJtYWlkIGNvZGU8YnI-dG8gcGFnZSBtYXJrZG93biBmaWxlXSAtLT5cbiAgICBEWzQuIEFkZCBjYXB0aW9uXVxuIFxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMsRCBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ" _blank
enabled
.md
Note:
in the
.md
.md
file:
---
---
...
    graph TB
    A --&gt; B

Figure 17. A to B
more text
Note:
.
.md
file.
.svg
.md
file.
local
Caution:
The Mermaid live editor feature set may not support the K8s/

website Mermaid feature set.
diagram]
end
A --&gt; B
B --&gt; C
C --&gt; w
click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0

" _blank
enabled
.svg
images/
folder.
Use the
.md
file.
caption
boxnet
.
folder.
Use the
.md
the
.svg
{
{&lt;
figure
src=
alt=
class=
caption=
&gt;
}
}
.
Note:
The
.svg
.
.svg
.svg
.svg
local
.svg
or
.png
.svg
diagram]
access]
to reference svg or
png file in
page .md file] --&gt;
end
A --&gt; B
B --&gt; C
C --&gt; w
click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0byBjcmVhdGUvZWRpdDxicj5kaWFncmFtXVxuICAgIEJbMi4gSWYgcG9zc2libGUsIHNhdmU8YnI-ZGlhZ3JhbSBjb29yZGluYXRlczxicj5mb3IgY29udHJpYnV0b3I8YnI-YWNjZXNzXVxuICAgIENbMy4gR2VuZXJhdGUgLnN2ZyA8YnI-b3IucG5nIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgI

G4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ"
enabled
.xml
.svg
or
.png
folder.
Use the
.md
file.
caption
Here is the
diagram:
.svg
or
.png
.xml
.
local
Examples
Note:
page.
graph TB
n3(Node3)
n4(Node4)
end
n1(Node1)
n2(Node2)
end
click n3 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
click n4 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiAgICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2x

vcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
enabled
graph TB
       n3(Node3)
       n4(Node4)
   end
       n1(Node1)
       n2(Node2)
   end
 
page.
graph LR;
ingress;
end
click client "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U6I2JiYixzdHJva2Utd2lkdGg6MnB4LGNvbG9yOiMzMjZjZTU7XG4gIGNsYXNzIGluZ3Jlc3Msc2VydmljZSxwb2QxLHBvZDIgazhzO1xuICBjbGFzcyBjbGllbnQgcGxhaW47XG4gIGNsYXNzIGNsdXN0ZXIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI

ice--&gt;pod1[Pod];
 end
    actor me
classDef
class
figure 7
,
classDef
and
class
#326ce5
.
user.
flowchart
A[Diagram
in text]
click A "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE51bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmF

tIjpmYWxzZX0" _blank
enabled
Note:
Diagram
The
and
.svg
Here is the
.svg
:
src
,
alt
,
class
and
caption
,
and
classes.
Note:
Inline
See
.svg
caption
Use the
and
diagrams.
Inline
method.
Figure
You must puncuate the caption with a period. Keep the caption text short.
BELOW
match.
or
suitable. If you are not sure, add a diag

ram referral to your text to see if
shortcode
link
Here is the
Tips
.svg
or
.png
With the
and
.svg
.svg
must
save it
.
6.4 - Writing a new topic
Open a PR
.
Type
Concept
Nodes
.
Task
Tutorial
A tutorial page shows how to accomplish a goal that ties together several Kubernetes features. A tutorial might provide several sequences of steps that readers can actually do as they read the page. Or it might provide explanations of related pieces of code. For example, a tutorial could provide a walkthrough of a code sample. A tutorial can include brief explanations of the Kubernetes features that are being tied together, but should link to related concept topics for d

eep explanations of individual features.
Use a
to create
hugo new
"kubernetes" in the filename, because "kubernetes" is already in the
title
.
---
---
create
contents.
_index.md
_index.md
weight:
10
20
.
.
,
one of the
codenew
where
examples
/content/en/example

s/pods/storage/gce-volume.yaml
.
Note:
&lt;
and before
the
&gt;
under
.
Note:
When adding new YAML files to the
file. The
.
/images
.
.
6.5 - Page content types
Concept
Task
Tutorial
Reference
heading
heading
whatsnext
cleanup
synopsis
seealso
options
whatsnext
The
heading
files.
:
other =
:
other =
"다음 내용"
Concept
overview
body
whatsnext
The
overview
and
body
whatsnext
heading
For
overview
For
body
For
whatsnext
Task
to conceptual topics that provide related background and knowledge.
overview
prereq

uisites
steps
whatsnext
The
overview
,
steps
, and
and
whatsnext
with the
heading
#
For
overview
For
include
For
steps
in
steps
.
For
whatsnext
.
Tutorial
overview
cleanup
whatsnext
The
overview
,
, and
,
cleanup
, and
whatsnext
with the
heading
#
For
overview
For
For
For
For
cleanup
For
whatsnext
.
Reference
synopsis
options
examples
seealso
kubectl
6.6 - Content organization
This site uses Hugo. In Hugo,
is a core concept.
Note:
Hugo Tip:
Start Hugo with
hugo server --navigateToChanged
for content edit-sessions.
Page Lists
Page Order
The documentation side menu, the documentation page browser etc. are listed using Hugo's default sort order, which sorts by weight (from 1), date (newest first), and finally by the link title.
Given that, if you want to move a page or a section up, set a weight in the page's front matter:
title
:
My Page
weight
:
10
Note:
For page weights, it can be smart not to use 1, 2, 3 ..., but some other interval, say 10, 20, 30... This allows you to insert pages where you want later.
Additionally, each weight within the same directory (section) should not be overlapped with the other weights. This makes sure that content is always organized correctly, especi

ally in localized content.
Documentation Main Menu
The
main menu is built from the sections below
docs/
with the
main_menu
flag set in front matter of the
_index.md
section content file:
main_menu
:
true
Note that the link title is fetched from the page's
linkTitle
, so if you want it to be something different than the title, change it in the content file:
main_menu
:
true
title
:
Page Title
linkTitle
:
Title used in links
Note:
The above needs to be done per language. If you don't see your section in the menu, it is probably because it is not identified as a section by Hugo. Create a
_index.md
content file in the section folder.
Documentation Side Menu
The documentation side-bar menu is built from the
current section tree
starting below
docs/
.
It will show all sections and their pages.
If you don't want to list a section or page, set the
toc_hide
flag to
true
in front matter:
toc_hide
:
true
When you navigate to a section that has content, the specific section or page (e.g.
_index.md
) is shown. Else, the first page inside that section is shown.
Documentation Browser
The page browser on the documentation home page is built using all the sections and pages that are directly below the
docs section
.
toc_hide
flag to
true
toc_hide
:
true
The Main Menu
The site links in the top-right menu -- and also in the footer -- are built by page-lookups. This is to make sure that the page actually exists. So, if the
case-studies
section does not exist in a site (language), it will not be linked to.
Page Bundles
In addition to standalone content pages (Markdown files), Hugo supports
.
One example is
. It is considered a
leaf bundle
. Everything below the directory, including the
index.md
, will be part of the bundle. This also includes page-relative links, images that can be processed etc.:
en/docs/home/contribute/includes
├── example1.md
├── example2.md
├── index.md
└── podtemplate.json
Another widely used example is the
includes
bundle. It sets
headless: true
in front matter, which means that it does not get its own URL. It is only used in other pages.
en/includes
├── default-storage-class-prereqs.md
├── partner-script.js
├── partner-style.css
├── task-tutorial-prereqs.md
├── user-guide-content-moved.md
└── user-guide-migration-notice.md
Some important notes to the files in the bundles:
For translated bundles, any missing non-content files will be inherited from languages above. This avoids duplication.
All the files in a bundle are what Hugo calls
Resources
and you can provide metadata per language, such as parameters and title, even if it does not supports front matter (YAML files etc.). See
Page Resources Metadata
.
The value you get from
.RelPermalink
of a
Resource
is page-relative. See
Permalinks
.
Styles
The
SASS
source of the stylesheets for this site is stored in
assets/sass
and is automatically built by Hugo.
6.7 - Custom Hugo Shortcodes
This page explains the custom Hugo shortcodes that can be used in Kubernetes Markdown documentation.
Read more about shortcodes in the
Hugo documentation
.
Feature state
In a Markdown page (
.md
file) on this site, you can add a shortcode to
display version and state of the documented feature.
Feature state demo
Below is a demo of the feature state snippet, which displays the feature as
stable in the latest Kubernetes version.
{{&lt; feature-state state="stable" &gt;}}
Renders to:
The valid values for
state
are:
alpha
beta
stable
Feature state code
The displayed Kubernetes version defaults to that of the page or the site. You can change the
feature state version by passing the
for_k8s_version
shortcode parameter. For example:
{{&lt; feature-state for_k8s_version="v1.10" state="beta" &gt;}}
FEATURE ST

ATE:
Kubernetes v1.10 [beta]
Glossary
There are two glossary shortcodes:
glossary_tooltip
and
glossary_definition
.
You can reference glossary terms with an inclusion that automatically updates
and replaces content with the relevant links from
our glossary
.
When the glossary term is moused-over, the glossary entry displays a tooltip.
The glossary term also displays as a link.
As well as inclusions with tooltips, you can reuse the definitions from the glossary in
page content.
The raw data for glossary terms is stored at
the glossary directory
,
with a content file for each glossary term.
Glossary demo
For example, the following include within the Markdown renders to
cluster
with a tooltip:
{{&lt; glossary_tooltip text="cluster" term_id="cluster" &gt;}}
Here's a short glossary definition:
{{&lt; glossary_definition prepend="A cluster is" term_id="cluster" length="short" &gt;}}
which renders as:
A cluster is a set of worker machines, called
nodes
,
You can also include a full definition:
{{&lt; glossary_definition term_id="cluster" length="all" &gt;}}
nodes
,
Pods
that are
Links to API Reference
You can link to a page of the Kubernetes API reference using the
api-reference
shortcode, for example to the
Pod
reference:
{{&lt; api-reference page="workload-resources/pod-v1" &gt;}}
The content of the
page
parameter is the suffix of the URL of the API reference page.
You can link to a specific place into a page by specifying an
anchor
parameter, for example to the
PodSpec
reference or the
environment-variables
section of the page:
{{&lt; api-reference page="workload-resources/pod-v1" anchor="PodSpec" &gt;}}
{{&lt; api-reference page="workload-resources/pod-v1" anchor="environment-variables" &gt;}}
You can change the text of the link by specifying a
text
parameter, for
example by linking to the
{{&lt; api-reference page="workload-resources/pod-v1" anchor="environment-variables" text="Environment Variable" &gt;}}
Table captions
You can make tables more accessible to screen readers by adding a table caption. To add a
caption
to a table,
enclose the table with a
table
shortcode and specify the caption with the
caption
Note:
Table captions are visible to screen readers but invisible when viewed in standard HTML.
Here's an example:
{{
&lt;
table
caption
=
"Configuration parameters"
&gt;
}}
Parameter | Description | Default
:---------|:------------|:-------
`timeout` | The timeout for requests | `30s`
`logLevel` | The log level for log output | `INFO`
{{
&lt;
/
table
&gt;
}}
The rendered table looks like this:
Configuration parameters
Parameter
Default
timeout
The timeout for requests
30s
logLevel
The log level for log output
INFO
If you inspect the HTML for the table, you should see this element immediately
after the opening
&lt;table&gt;
element:
&lt;
caption
style
=
"display: none;"
&gt;Configuration parameters&lt;/
caption
&gt;
Tabs
In a markdown page (
.md
file) on this site, you can add a tab set to display
multiple flavors of a given solution.
The
tabs
shortcode takes these parameters:
name
: The name as shown on the tab.
codelang
: If you provide inner content to the
tab
shortcode, you can tell Hugo
what code language to use for highlighting.
include
: The file to include in the tab. If the tab lives in a Hugo
,
the file -- which can be any MIME type supported by Hugo -- is looked up in the bundle itself.
If not, the content page th

at needs to be included is looked up relative to the current page.
Note that with the
include
, you do not have any shortcode inner content and must use the
self-closing syntax. For example,
{{&lt; tab name="Content File #1" include="example1" /&gt;}}
. The language needs to be specified
under
codelang
or the language is taken based on the file name.
Non-content files are code-highlighted by default.
If your inner content is markdown, you must use the
%
-delimiter to surround the tab.
{{% tab name="Tab 1" %}}This is **markdown**{{% /tab %}}
You can combine the variations mentioned above inside a tab set.
Below is a demo of the tabs shortcode.
Note:
The tab
name
in a
tabs
definition must be unique within a content page.
Tabs demo: Code highlighting
{{
&lt;
tabs
name
=
"tab_with_code"
&gt;
}}
{{
&lt;
tab
name
=
"Tab 1"
codelang
=
"bash"
&gt;
}}
echo "This is tab 1."
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
tab
name
=
"Tab 2"
codelang
=
"go"
&gt;
}}
println "This is tab 2."
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
Tab 1
Tab 2
echo
"This is tab 1."
println
"This is tab 2."
Tabs demo: Inline Markdown and HTML
{{
&lt;
tabs
name
=
"tab_with_md"
&gt;
}}
{{
%
tab
name
=
"Markdown"
%
}}
This is **some markdown.**
{{
&lt;
note
&gt;
}}
It can even contain shortcodes.
{{
&lt;
/
note
&gt;
}}
{{
%
/
tab
%
}}
{{
&lt;
tab
name
=
"HTML"
&gt;
}}
&lt;
div
&gt;
&lt;
h3
&gt;Plain HTML&lt;/
h3
&gt;
&lt;
p
&gt;This is some &lt;
i
&gt;plain&lt;/
i
&gt; HTML.&lt;/
p
&gt;
&lt;/
div
&gt;
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
Markdown
HTML
This is
some markdown.
Note:
Plain HTML
This is some
plain
HTML.
Tabs demo: File include
{{
&lt;
tabs
name
=
"tab_with_file_include"
&gt;
}}
{{
&lt;
tab
name
=
"Content File #1"
include
=
"example1"
/&gt;
}}
{{
&lt;
tab
name
=
"Content File #2"
include
=
"example2"
/&gt;
}}
{{
&lt;
tab
name
=
"JSON File"
include
=
"podtemplate"
/&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
Content File #1
Content File #2
JSON File
This is an
example
content file inside the
includes
leaf bundle.
Note:
Included content files can also contain shortcodes.
This is another
example
includes
{
:
"v1"
,
"kind"
:
"PodTemplate"
,
: {
"name"
:
"nginx"
},
: {
: {
"labels"
: {
"name"
:
"nginx"
},
"generateName"
:
"nginx-"
},
"spec"
: {
: [{
"name"
:
"nginx"
,
"image"
:
"dockerfile/nginx"
,
"ports"
: [{
"containerPort"
:
80
}]
}]
}
}
}
Third party content marker
Running Kubernetes requires third-party software. For example: you
usually need to add a
DNS server
to your cluster so that name resolution works.
When we link to third-party software, or otherwise mention it,
we follow the
and we also mark those third party items.
Using these shortcodes adds a disclaimer to any documentation page
that uses them.
Lists
For a list of several third-party items, add:
{{% thirdparty-content %}}
just below the heading for the section that includes all items.
Items
If you have a list where most of the items refer to in-project
software (for example: Kubernetes itself, and the separate
Descheduler
component), then there is a different form to use.
Add the shortcode:
{{% thirdparty-content single="true" %}}
before the item, or just below the heading for the specific item.
Version strings
To generate a version string for inclusion in the documentation, you can choose from
several version shortcodes. Each version shortcode displays a version string derived from
the value of a version parameter found in the site configuration file,
.
The two most commonly used version parameters are
latest
and
version
.
{{&lt; param "version" &gt;}}
The
shortcode generates the value of the current
version of the Kubernetes documentation from the
version
site parameter. The
param
shortcode accepts the name of one site parameter, in this case:
version
.
Note:
In previously r

eleased documentation,
latest
and
version
parameter values
are not equivalent. After a new version is released,
latest
is incremented
and the value of
version
for the documentation set remains unchanged. For
example, a previously released version of the documentation displays
version
as
v1.19
and
latest
as
v1.20
.
v1.26
{{&lt; latest-version &gt;}}
The
shortcode returns the value of the
latest
site parameter.
The
latest
site parameter is updated when a new version of the documentation is released.
This parameter does not always match the value of
version
in a documentation set.
v1.26
{{&lt; latest-semver &gt;}}
The
shortcode generates the value of
latest
without the "v" prefix.
1.26
{{&lt; version-check &gt;}}
The
shortcode checks if the
page parameter is present and then uses this value to compare to
version
.
.
{{&lt; latest-release-notes &gt;}}
The
shortcode generates a version string
from
latest
and removes the "v" prefix. The shortcode prints a new URL for
the release note CHANGELOG page with the modified version string.
https://git.k8s.io/kubernetes/CHANGELOG/CHANGELOG-1.26.md
Hugo
.
.
.
.
advanced contributing
.
7 - Updating Reference Documentation
7.1 - Reference Documentation Quickstart
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
website
remote on
GitHub (
main
website
fork.
cd
.
Note:
see the
.
The
The
kubectl
The
under
/tmp
and
GOPATH
K8S_ROOT
)
A release vers

ion, for example:
1.17
field.
The
from
. The
variable
The
default is
.
Updates
kubectl
kubectl
kubectl
to
.
repos
:
-
name
:
community
remote
:
branch
:
master
files
:
-
src
:
dst
:
-
src
:
dst
:
the
.
Open
.
Note:
The
In
,
files
src
and
dst
fields.
The
src
dst
repos
:
-
name
:
remote
:
files
:
-
src
:
dst
:
...
src
dst
.
files
:
-
src
:
dst
:
cd
The
true
.
:
cd
git status
The output shows the new and modified file

s. The generated output varies
Run
git add
and
git commit
.
7.2 - Contributing to the Upstream Kubernetes Code
This page shows how to contribute to the upstream
project.
You can fix bugs found in the Kubernetes API documentation or the content of
the Kubernetes components such as
kubeadm
,
, and
.
If you instead want to regenerate the reference documentation for the Kubernetes
API or the
kube-*
components from the upstream code, see the following instructions:
Generating Reference Documentation for the Kubernetes Components and Tools
Git
Golang
Docker
etcd
make
Your
GOPATH
environment variable must be set, and the location of
etcd
must be in your
PATH
You need to know how to create a

 pull request to a GitHub repository.
Typically, this involves creating a fork of the repository.
Creating a Pull Request
and
GitHub Standard Fork &amp; Pull Request Workflow
.
The big picture
The reference documentation for the Kubernetes API and the
kube-*
such as
,
are automatically generated
from the source code in the
upstream Kubernetes
.
When you see bugs in the generated documentation, you may want to consider
creating a patch to fix it in the upstream project.
Cloning the Kubernetes repository
If you don't already have the kubernetes/kubernetes repository, get it now:
mkdir
$GOPATH
/src
cd
$GOPATH
/src
go get github.com/kubernetes/kubernetes
$GOPATH/src/github.com/kubernetes/kubernetes.
.
.
Editing the Kubernetes source code
The Kubernetes API reference documentation is automatically generated from
an OpenAPI spec, which is generated from the Kubernetes source code. If you
want to change the API reference documentation, the first step is to change one
or more comments in the Kubernetes source code.
The documentation for the
kube-*
components is also generated from the upstream
source code. You must change the code related to the component
you want to fix in order to fix the generated documentation.
Making changes to the upstream source code
Note:
The following steps are an example, not a general procedure. Details
will be different in your situation.
Here's an example of editing a comment in the Kubernetes source code.
In your local kubernetes/kubernetes repository, check out the default branch,
and make sure it is up to date:
cd
git checkout master
git pull https://github.com/kubernetes/kubernetes master
Suppose this source file in that default branch has the typo "atmost":
kubernetes/kubernetes/staging/src/k8s.io/api/apps/v1/types.go
In your local environment, open
types.go
, and change "atmost" to "at most".
Verify that you have changed the file:
git status
The output shows that you are on the master branch, and that the
types.go
source file has been modified:
On branch master
...
modified:   staging/src/k8s.io/api/apps/v1/types.go
Committing your edited file
Run
git add
and
git commit
to commit the changes you have made so far. In the next step,
you will do a second commit. It is important to keep your changes separated into two commits.
Generating the OpenAPI spec and related files
Go to
and run these scripts:
hack/update-generated-swagger-docs.sh
hack/update-openapi-spec.sh
hack/update-generated-protobuf.sh
Run
git status
to see what was generated.
...
modified:   api/openapi-spec/swagger.json
modified:   api/openapi-spec/v3/apis__apps__v1_openapi.json
modified:   pkg/generated/openapi/zz_generated.openapi.go
modified:   staging/src/k8s.io/api/apps/v1/generated.proto
modified:   staging/src/k8s.io/api/apps/v1/types_swagger_doc_generated.go
View the contents of
api/openapi-spec/swagger.json
to make sure the typo is fixed.
For example, you could run
git diff -a api/openapi-spec/swagger.json
.
This is important, because
swagger.json
is the input to the second stage of
the doc generation process.
Run
git add
and
git commit
to commit your changes. Now you have two commits:
one that contains the edited
types.go
file, and one that contains the generated OpenAPI spec
and related files. Keep these two commits separate. That is, do not squash your 

commits.
to the
master branch of the
to monitor your pull request until it is merged.
PR 57758
is an example of a pull request that fixes a typo in the Kubernetes source code.
Note:
It can be tricky to determine the correct source file to be changed. In the
preceding example, the authoritative source file is in the
staging
directory
in the
repository. But in your situation,the
staging
directory
might not be the place to find the authoritative source. For guidance, check the
README
files in
repository and in related repositories, such as
kubernetes/apiserver
.
Cherry picking your commit into a release branch
In the preceding section, you edited a file in the master branch and then ran scripts
to generate an OpenAPI spec and related files. Then you submitted your changes in a pull request
to the master branch of the kubernetes/kubernetes repository. Now suppose you want to backport
your change into a release branch. For example, suppose the master branch is being used to develop
Kubernetes version 1.26, and you want to backport your change into the
release-1.25 branch.
Recall that your pull request has two commits: one for editing
types.go
and one for the files generated by scripts. The next step is to propose a cherry pick of your first
commit into the release-1.25 branch. The idea is to cherry pick the commit
that edited
types.go
, but not the commit that has the results of running the scripts. For instructions, see
.
Note:
Proposing a cherry pick requires that you have permission to set a label and a milestone in your
pull request. If you don't have those permissions, you will need to work with someone who can set the label
and milestone for you.
When you have a pull request in place for cherry picking your one commit into the
release-1.25 branch, the next step is to run these scripts in the
release-1.25 branch of your local environment.
hack/update-api-reference-docs.sh
Now add a commit to your cherry-pick pull request that has the recently generated OpenAPI spec
and related files. Monitor your pull request until it gets merged into the
At this point, both the master branch and the release-1.25 branch have your updated
types.go
file and a set of generated files that reflect the change you made to
types.go
. Note that the
generated OpenAPI spec and other generated files in the release-1.25 branch are not necessarily
the same as the generated files in the master branch. The generated files in the release-1.25 branch
contain API elements only from Kubernetes 1.25. The generated files in the master branch might contain
API elements that are not in 1.25, but are under development for 1.26.
Generating the published reference docs
The preceding section showed how to edit a source file and then generate
several files, including
in the
The
file is the OpenAPI definition file to use for generating
the API reference documentation.
You are now ready to follow the
guide to generate the
published Kubernetes API reference documentation
.
7.3 - Generating Reference Documentation for the Kubernetes API
using the

.
OpenAPI
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
GOPATH
.
mkdir -p
$HOME
export
GOPATH
=
$HOME
$GOPATH
$GOPATH
.
.
.
.
Set
K8S_ROOT
to
.
Set
to
.
Set
to 1.17.0.
export
=
${
GOPATH
}
export
K8S_ROOT
=
${
GOPATH
}
export
=
1.17.0
The
.
In the
cd
The
copyapi
.
:
cd
[
-e
]
echo
||
echo
[
-e
]
echo
||
echo
, and
cd
git status
static/docs/reference/g

enerated/kubernetes-api/v1.26/fonts/fontawesome-webfont.eot
Open
---
---

Open
Verify the
.
cd
1
In
run
git add
and
git commit
to the
7.4 - Generating Reference Documentation for kubectl Commands
kubectl
Note:
like
and
.
kubectl
.
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
GOPATH
.
mkdir -p
$HOME
export
GOPATH
=
$HOME
$GOPATH
$GOPATH
.
rm -rf
$GOPATH
kubectl
and
kustomize
Determine 

the base directory of your clone of the
.
$GOPATH/src/github.com/&lt;your-username&gt;/website.
.
.
cd
kubectl
.
.
PR 56673
1.26
.
Note:
Go to
Set
K8S_ROOT
to
.
Set
to
.
Set
to 1.25.
export
=
$GOPATH
export
K8S_ROOT
=
$GOPATH
export
=
1.25
The
.
In the
cd
v1.25
you local branch is 

up to date.
cd
, run the
copycli
root
:
cd
The
copycli
.
[
-e
]
echo
||
echo
[
-e
]
echo
||
echo
:
cd
git status
.
cd
1
View the
.
Run
git add
and
git commit
.
7.5 - Generating Reference Pages for Kubernetes Components and Tools
This page shows how to build the Kubernetes component and tool reference pages.
Start with the
Prerequisites section
in the Reference Documentation Quickstart guide.
Follow the
to generate the Kubernetes component and tool reference pages.
Contributing to the Upstream Kubernetes Project for Documentation
7.6 -
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
8 - Advanced contributing
This page assumes that you understand how to
contribute to new content
and
review others' work
, and are ready
to learn about more ways to contribute. You need to use the Git command line
client and other tools for some of these tasks.
Propose im

provements
SIG Docs
members
can propose improvements.
After you've been contributing to the Kubernetes documentation for a while, you
may have ideas for improving the
Style Guide
, the
Content Guide
, the toolchain used to build
the documentation, the website style, the processes for reviewing and merging
pull requests, or other aspects of the documentation. For maximum transparency,
these types of proposals need to be discussed in a SIG Docs meeting or on the
.
In addition, it can help to have some context about the way things
currently work and why past decisions have been made before proposing sweeping
changes. The quickest way to get answers to questions about how the documentation
currently works is to ask in the
#sig-docs
Slack channel on
kubernetes.slack.com
After the discussion has taken place and the SIG is in agreement about the desired
outcome, you can work on the proposed changes in the way that is the most
appropriate. For instance, an update to the style guide or the website's
functionality might involve opening a pull request, while a change related to
documentation testing might involve working with sig-testing.
Coordinate docs for a Kubernetes release
SIG Docs
approvers
can coordinate docs for a Kubernetes release.
Each Kubernetes release is coordinated by a team of people participating in the
sig-release Special Interest Group (SIG). Others on the release team for a given
release include an overall release lead, as well as representatives from
sig-testing and others. To find out more about Kubernetes release processes,
refer to
https://github.com/kubernetes/sig-release
.
The SIG Docs representative for a given release coordinates the following tasks:
Monitor the feature-tracking spreadsheet for new or changed features with an
impact on documentation. If the documentation for a given feature won't be ready
for the release, the feature may not be allowed to go into the release.
Attend sig-release meetings regularly and give updates on the status of the
docs for the release.
Review and copyedit feature documentation drafted by the SIG responsible for
implementing the feature.
Merge release-related pull requests and maintain the Git feature branch for
the release.
Mentor other SIG Docs contributors who want to learn how to do this role in
the future. This is known as "shadowing".
Publish the documentation changes related to the release when the release
artifacts are published.
Coordinating a release is typically a 3-4 month commitment, and the duty is
rotated among SIG Docs approvers.
Serve as a New Contributor Ambassador
SIG Docs
approvers
can serve as New Contributor Ambassadors.
New Contributor Ambassadors welcome new contributors to SIG-Docs,
suggest PRs to new contributors, and mentor new contributors through their first
few PR submissions.
Responsibilities for New Contributor Ambassadors include:
Monitoring the
for questions from new contributors.
Working with PR wranglers to identify
for new contributors.
Mentoring new contributors through their first few PRs to the docs repo.
Helping new contributors create the more complex PRs they need to become Kubernetes members.
Sponsoring contributors
on their path to becoming Kubernetes members.
Hosting a monthly meeting to help and mentor new contributors.
Current New Contributor Ambassadors are announced at each SIG-Docs meeting and in the
Kubernetes #sig-docs channel
.
Sponsor a new contributor
SIG Docs
reviewers
can sponsor new contributors.
After a new contributor has successfully submitted 5 substantive pull requests
to one or more Kubernetes repositories, they are eligible to apply for
membership
in the Kubernetes organization. The contributor's membership needs to be
backed by two sponsors who are already reviewers.
New docs contributors can request sponsors by asking in the #sig-docs channel
on the
Kubernetes Slack instance
or on the
.
If you feel confident abo

ut the applicant's work, you volunteer to sponsor them.
When they submit their membership application, reply to the application with a
"+1" and include details about why you think the applicant is a good fit for
membership in the Kubernetes organization.
Serve as a SIG Co-chair
SIG Docs
members
can serve a term as a co-chair of SIG Docs.
A Kubernetes member must meet the following requirements to be a co-chair:
Understand SIG Docs workflows and tooling: git, Hugo, localization, blog subproject
Understand how other Kubernetes SIGs and repositories affect the SIG Docs
workflow, including:
teams in k/org
, the
process in k/community
,
plugins in
k/test-infra
, and the role of
SIG Architecture
.
In addition, understand how the
Kubernetes docs release process
works.
Approved by the SIG Docs community either directly or via lazy consensus.
Commit at least 5 hours per week (and often more) to the role for a minimum of 6 months
Responsibilities
The role of co-chair is one of service: co-chairs build contributor capacity, handle process and policy, schedule and run meetings, schedule PR wranglers, advocate for docs in the Kubernetes community, make sure that docs succeed in Kubernetes release cycles, and keep SIG Docs focused on effective priorities.
Responsibilities include:
Keep SIG Docs focused on maximizing developer happiness through excellent documentation
Exemplify the
community code of conduct
and hold SIG members accountable to it
Learn and set best practices for the SIG by updating contribution guidelines
Schedule and run SIG meetings: weekly status updates, quarterly retro/planning sessions, and others as needed
Schedule and run doc sprints at KubeCon events and other conferences
Recruit for and advocate on behalf of SIG Docs with the
CNCF
and its platinum partners, including Google, Oracle, Azure, IBM, and Huawei
Keep the SIG running smoothly
Running effective meetings
To schedule and run effective meetings, these guidelines show what to do, how to do it, and why.
Uphold the
:
Hold respectful, inclusive discussions with respectful, inclusive language.
Set a clear agenda
:
Set a clear agenda of topics
Publish the agenda in advance
For weekly meetings, copypaste the previous week's notes into the "Past meetings" section of the notes
Collaborate on accurate notes
:
Record the meeting's discussion
Consider delegating the role of note-taker
Assign action items clearly and accurately
:
Record the action item, who is assigned to it, and the expected completion date
Moderate as needed
:
If discussion strays from the agenda, refocus participants on the current topic
Make room for different discussion styles while keeping the discussion focused and honoring folks' time
Honor folks' time
:
Begin and end meetings on time.
Use Zoom effectively
:
Zoom guidelines for Kubernetes
Claim the host role when you log in by entering the host key
Recording meetings on Zoom
When you're ready to start the recording, click Record to Cloud.
When you're ready to stop recording, click Stop.
The video uploads automatically to YouTube.
9 - Viewing Site Analytics
.
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Founda

tion has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Configure the Aggregation Layer | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Tasks
minikube
.
Note:
Caution:
.
.
A request to an API path that is served by an extension apiserver begins the same way as all API requests: communication to the Kubernetes apiserver

. This path already has been registered with the Kubernetes apiserver by the extension apiserver.
.
Everything to this point has been standard Kubernetes API requests, authentication and authorization.
.
.
Note:
any
all
confi

gmap, so they can be retrieved and used by extension apiservers.
Was signed by the CA whose certificate matches the retrieved CA certificate.
in the
named
If the
CN=
O=
. See the
.
CN=
If
both
and
-

-requestheader-client-ca-file
same
CN=
will
not
Warning:
Do
not
flag:
:
kind
:
metadata
:
name
:
spec
:
group
:
version
:
:
:
service
:
namespace
:
name
:
caBundle
:
.
The
service
:
kind
:
...
spec
:
...
service
:
namespace
:
name
:
port
:
1234
caBundle
:
...
.
.
Feedback
Yes
No
.
or
.
Enable Kubernet

es Apiserver flags
Home
Blog
Training
Partners
Community
CC BY 4.0

Feature Gates (removed) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
the
or the
Feature
Default
Stage
From
To
false
Alpha
1.6
1.10
-
1.11
1.11
false
Alpha
1.6
1.7
-
1.8
1.8
false
Beta
1.4
1.6
true
GA
1.7
1.9
true
Beta
1.17
1.20
true
GA
1.21
1.25
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.21
false
Alpha
1.11
1.21
false
1.22
1.22
false
Alpha
1.9
1.12
true
Beta
1.13
1.17
true
GA
1.18
1.21
false
Alpha
1.13
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.23
false
Alpha
1.10
1.10
true
Beta
1.11
1.20
true
GA
1.21
1.22
false
Alpha
1.11
1.13
true
Beta
1.14
1.17
true
GA
1.18
1.21
false
Alpha
1.12
1.13
true
Beta
1.14
1.17
true
GA
1.18
1.21
false
Alpha
1.17
1.20
-
1.21
1.21
false
Alpha
1.17
1.20
-
1.21
1.21
false
Alpha
1.17
1.20
-
1.21
1.21

false
Alpha
1.17
1.20
-
1.21
1.21
false
Alpha
1.14
1.17
true
Beta
1.18
1.23
true
GA
1.24
1.25
false
Alpha
1.17
1.20
-
1.21
1.21
false
Beta
1.19
1.21
-
1.22
1.22
false
Alpha
1.12
1.13
true
Beta
1.14
1.16
true
GA
1.17
1.22
false
Alpha
1.9
1.9
true
Beta
1.10
1.12
true
GA
1.13
1.16
false
Alpha
1.20
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.24
false
Alpha
1.19
1.19
true
Beta
1.20
1.22
true
GA
1.23
1.25
false
Alpha
1.18
1.19
true
Beta
1.20
1.22
true
GA
1.23
1.25
false
Alpha
1.20
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.23
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.9
1.9
true
Beta
1.10
1.13
true
GA
1.14
1.16
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
1.18
false
Alpha
1.14
1.14
true
Beta
1.15
1.15
true
GA
1.16
1.18
false
Alpha
1.10
1.10
true
Beta
1.11
1.15
true
GA
1.16
1.18
false
Alpha
1.8
1.8
true
Beta
1.9
1.15
true
GA
1.16
1.18
false
Alpha
1.13
1.14
true
Beta
1.15
1.15
true
GA
1.16
1.18
false
Alpha
1.19
1.19
true
Beta
1.20
1.23
true
GA
1.24
1.25
false
Alpha
1.13
1.18
-
1.19
1.19
false
Alpha
1.4
1.10
true
Beta
1.11
1.21
false
1.22
1.25
false
Alpha
1.11
1.11
-
1.12
-
true
Alpha
1.3
1.7
true
GA
1.8
1.12
true
1.16
1.17
false
Alpha
1.8
1.12
-
1.13
1.23
false
Alpha
1.16
1.16
false
Beta
1.17
1.17
true
Beta
1.18
1.20
true
GA
1.21
1.24
false
Alpha
1.20
1.20
true
GA
1.21
1.24
false
Alpha
1.18
1.18
true
Beta
1.19
1.21
true
GA
1.22
1.24
false
Alpha
1.16
1.17
true
Beta
1.18
1.18
true
GA
1.19
1.21
false
Alpha
1.5
1.12
false
1.13
1.16
true
GA
1.18
1.22
true
Beta
1.10
1.12
true
GA
1.13
1.16
false
Alpha
1.19
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
false
Alpha
1.18
1.18
true
Beta
1.19
1.21
true
GA
1.22
1.24
HugePages
false
Alpha
1.8
1.9
HugePages
true
Beta
1.10
1.13
HugePages
true
GA
1.14
1.16
false
Alpha
1.10
1.19
false
1.20
1.20
false
Alpha
1.15
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
false
Alpha
1.18
1.18
ImmutableEphemera

lVolumes
true
Beta
1.19
1.20
true
GA
1.21
1.24
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.21
1.21
true
Beta
1.22
1.22
true
GA
1.23
1.24
false
Alpha
1.7
1.13
-
1.14
1.14
false
Alpha
1.8
1.9
-
1.10
1.10
false
Alpha
1.11
1.11
true
Beta
1.12
1.12
true
GA
1.13
1.16
false
Alpha
1.16
1.18
true
Beta
1.19
1.20
false
GA
1.21
1.22
false
Alpha
1.9
1.16
false
1.17
1.17
false
Alpha
1.8
1.9
true
Beta
1.10
1.11
true
GA
1.12
1.14
true
Beta
1.21
1.21
true
GA
1.22
1.23
false
Alpha
1.16
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.22
NodeLease
false
Alpha
1.12
1.13
NodeLease
true
Beta
1.14
1.16
NodeLease
true
GA
1.17
1.23
false
Alpha
1.15
1.18
true
Beta
1.19
1.23
true
GA
1.24
1.25
false
Alpha
1.9
1.9
-
1.10
1.10
false
Alpha
1.7
1.9
true
Beta
1.10
1.13
true
GA
1.14
1.16
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.3
1.4
true
Beta
1.5
1.20
true
GA
1.21
1.25
false
Alpha
1.16
1.17
true
Beta
1.18
1.23
true
GA
1.24
1.25
false
Alpha
1.8
1.10
true
Beta
1.11
1.13
true
GA
1.14
1.18
false
Alpha
1.11
1.11
true
Beta
1.12
1.13
true
GA
1.14
1.16
false
Alpha
1.10
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.19
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.15
1.16
-
1.17
1.17
false
Alpha
1.9
1.18
-
1.19
1.19
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
false
Alpha
1.13
1.19
true
Beta
1.20
1.20
true
GA
1.21
1.22
true
Beta
1.8
1.18
true
GA
1.19
1.21
true
Beta
1.14
1.20
true
GA
1.21
1.22
false
Alpha
1.12
1.13
true
Beta
1.14
1.19
true
GA
1.20
1.24
false
Alpha
1.12
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.22
false
Alpha
1.11
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
false
Alpha
1.18
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.25
false
Alpha
1.18
1.19
true
Beta
1.20
1.20
true
GA
1.21
1.23
false
Alpha
1.18
1.18
true
Beta
1.19
1.19
true
GA
1.20
1.22
false
Alpha
1.20
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
false
Alpha
1.15
1.15
ServiceLoadBalancerFinal

izer
true
Beta
1.16
1.16
true
GA
1.17
1.20
false
Alpha
1.8
1.18
true
Beta
1.19
1.20
true
GA
1.21
1.22
false
Alpha
1.17
1.19
false
1.20
1.22
false
Alpha
1.19
1.19
true
Beta
1.20
1.21
true
GA
1.22
1,24
false
Alpha
1.16
1.17
true
Beta
1.18
1.19
true
GA
1.20
1.23
true
Beta
1.10
1.10
true
GA
1.11
1.24
false
Beta
1.5
1.5
true
Beta
1.6
1.17
true
1.18
1.21
false
1.22
1.24
false
Alpha
1.8
1.8
false
Beta
1.9
1.9
true
Beta
1.10
1.10
true
GA
1.11
1.20
false
Alpha
1.14
1.14
true
Beta
1.15
1.19
true
GA
1.20
1.23
false
Alpha
1.10
1.13
true
Beta
1.14
1.19
true
GA
1.20
1.23
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
1.25
Sysctls
true
Beta
1.11
1.20
Sysctls
true
GA
1.21
1.22
false
Alpha
1.12
1.20
true
Beta
1.21
1.22
true
GA
1.23
1.24
false
Alpha
1.6
1.12
true
Beta
1.13
1.17
true
GA
1.18
1.20
false
Alpha
1.8
1.11
true
Beta
1.12
1.16
true
GA
1.17
1.18
false
Alpha
1.10
1.11
true
Beta
1.12
1.19
true
GA
1.20
1.21
false
Alpha
1.11
1.11
true
Beta
1.12
1.19
true
GA
1.20
1.21
false
Alpha
1.12
1.13
true
Beta
1.14
1.21
true
1.22
1.24
false
Alpha
1.15
1.15
true
Beta
1.16
1.17
true
GA
1.18
1.21
false
Alpha
1.9
1.9
true
Beta
1.10
1.12
true
GA
1.13
1.16
false
Alpha
1.12
1.16
true
Beta
1.17
1.19
true
GA
1.20
1.22
true
GA
1.10
1.24
false
Alpha
1.14
1.14
true
Beta
1.15
1.16
true
GA
1.17
1.24
true
Beta
1.19
1.21
true
GA
1.22
1.24
false
Alpha
1.19
1.20
true
Beta
1.21
1.21
true
GA
1.22
1.24
false
Alpha
1.14
1.15
true
Beta
1.16
1.17
true
GA
1.18
1.20
false
Alpha
1.16
1.16
true
Beta
1.17
1.17
true
GA
1.18
1.20
for
.
See
See
fo

r more details.
with
flag
.
Check
See
See
csi
.
InTreePluginOpenStackUnr

egister
of the
.
.
field.
CronJob
dnsConfig
property.
Check
: Enable
/status
and
/scale
.
.
.
.
.
.
nodeName
field.
.
.
critical
.
applied to Service Extern

alIPs.
See
.
huge pages
.
HugePages
huge pages
.
: Enable
: Enable
dual stack
Job
Scope
and
Namespace
to
.
See
.
and
.
.
label
NodeLease
: Enable
local
local
volume.
and
feature.
.
.
: Enable reso

urce quota scope selectors.
ConfigMap
named
See
SCTP
protocol
".
See
Pod's
field
.
startup
exec
,
attach
and
requests.
See
in the
and
.
Sysctls
pod. See
sysctls
: Allow a
.
projected
volume
.

local
: Enable
subPath
.
.
Feedback
Yes
No
.
or
.
Last modified December 30, 2022 at 9:30 PM PST:
Fix feature gates (ae757599af)
Home
Blog
Training
Partners
Community
CC BY 4.0

Node metrics data | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
The
kubelet
.
minikube
:
curl
:
Note:
0.6.x,
.
cAdvisor
in your
discuss
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Generating Reference Pages for Kubernetes Components and Tools | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

 API Aggregation Layer
Tasks
Debug Pods
Debug a Sta

tefulSet
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod S

ecurity Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
ku

be-apiserver Configuration (v1)
Follow the
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Configuration | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
1.1:
1.2:
2:
1 - Example: Configuring a Java Microservice
1.1 - Externalizing config using MicroProfile, ConfigMaps and Secrets
here
.
here
.
1.2 - Interactive Tutorial - Configuring a Java Microservice
2 - Configuring Redis using 

a ConfigMap
task.
minikube
.
kubectl
.
cat
metadata:
data:
EOF
config
The
key
and
path
under
on the
config
volume.
The
config
by
.
from the
:
v1
kind
:
Pod
metadata
:
name
:
redis
spec
:
:
-
name
:
redis
image
:
command
:
-
env
:
-
name
:
MASTER
value
:
"true"
ports
:
-
:
6379
resources
:
limits
:
cpu
:
"0.1"
:
-
mountPath
:
name
:
data
-
mountPath
:
name
:
config
volumes
:
-
name
:
data
emptyDir
:
{}
-
name
:
config
configMap
:
name
:
items
:
-
key
:
path
:

key in the
key:
Data
====
Use
redis-cli
kubectl
exec
Check
maxmemory
:
1
)
2
)
"0"
:
:
1
)
2
)
:
v1
kind
:
ConfigMap
metadata
:
name
:
data
:
:
|
kubectl describe configmap/example

-redis-config
Data
====
----
redis-cli
via
kubectl
exec
Check
maxmemory
:
1
)
2
)
"0"
Returns:
1
)
2
)
kubectl
exec
Check
maxmemory
:
1
)
2
)
"2097152"
:
1
)
2
)
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Define Environment Variables for a Container | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Oper

(v1)
Tasks
minikube
env
or
envFrom
and
value
Pod:
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
purpose
:
spec
:
:
-
name
:
image
:
env
:
-
name
:
value
:
-
name
:
value
:
purpose
=
kubectl
exec
...
Note:
env
or
envFrom
field
Note:
elsewhere in the configuratio

n, for example in commands and arguments that
GREETING
,
HONORIFIC
, and
NAME
,
, and
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
bash
env
:
-
name
:
GREETING
value
:
-
name
:
HONORIFIC
value
:
-
name
:
NAME
value
:
command
:
[
"echo"
]
args
:
[
]
.
.
See
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Well-Known Labels, Annotations and Taints | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
Example:
).
One of the
.
Example:
).
Note:
Example:
).
.
One of the
.
Example:
).
One of the
.
Example:
).
One of the
.
Example:
).
One of the
.
Example:
).
the Git
One of the
.
Example:
"true"
"false"
Example:
Example:
Example:
)
selector
.
Example:
your container can consume 

unlimited CPU and memory.
Pods deployed after you define a LimitRange will have these limits applied to them.
.
instead.
instead.
Example:
The
Note:
onstart
true
Example:
flag to the
kubelet
.
Example:
--record
to a
kubectl
Example:
Example:
true
secrets
field.
Example:
Example:
int32
type.
Example:
"true"
"false"
If this annotation is not

 set, then the Cluster Autoscaler follows its overall behaviour (i.e evict the DaemonSets based on its configuration).
Note:
Note:
bandwidth
) and
).
Example:
Quantity
.
10M
Note:
bandwidth
) and
).
Example:
Quantity
.
10M
Note:
.
Example:
.
).
See
.
Note:
.
See
.
Note:
.
Example:
See
in the
Example:
The
Example:
See
.
Example:
kubelet
. However, you should consider setting this on nodes if it makes 

sense in your topology.
.
A zone represents a logical failure domain. It is common for Kubernetes clusters to span multiple zones for increased availability. While the exact definition of a zone is left to infrastructure implementations, common properties of a zone include very low network latency within a zone, no-cost network traffic within a zone, and failure independence from other zones. For example, nodes within a zone might share a network switch, but nodes in different zones should not.
.
If
). With
Example:
Example :
on itself, then its storage attach and detach operation

s are being managed
by the
.
Example:
Example:
Example:
with
Services
name
of the
Example:
name
of the
Example:
unique ID
of the
Example:
.
Example:
Example:
"true"
Example:
Example:
.
Note:
Example:
"true"
Note:
Starting in v1.18, this annotation is depreca

ted in favor of
.
Example:
When a single StorageClass resource has this annotation set to
"true"
Example:
--node-ip
Example:
.
Example:
or
without
-c
or
Example:
The
Endpoints
Service
Example:
.
You should
not
Note:
Example:
Use
instead.
Example:
Example:
Ready
being
Unknown
.
Example:
Example:
and
observed on a Node. The observed values are then compared to the corresponding thresholds that can

 be set on the kubelet to determine if the Node condition and taint should be added/removed.
Example:
The kubelet detects disk pressure based on
,
,
and
Example:
Example:
Example:
Caution:
Refer to
Example:
Example:
.
Example:
Value
must
be one of
,
baseline
, or
the
enforce
label
prohibits
See
Example:
Value
must
be
latest
.
See
Example:
Value
must
be one of
,
baseline
, or
the
audit
See
Example:
pod-security.kubernetes.io/audit-v

ersion: "1.26"
Value
must
be
latest
.
See
Example:
Value
must
be one of
,
baseline
, or
the
warn
See
Example:
Value
must
be
latest
.
See
Example:
.
The
.spec
When you
,
takes
.spec
.
Example:
true
or
false
.
is being
Refer to
and the
Example:
Note:
.
missing-s

an.invalid-cert.kubernetes.io/$hostname
page.
kubeadm
Example:
init
/
join
Example:
Example:
Example:
Example:
Example:
; kubeadm
1 - Audit Annotations
Event
.
Note:
Event
.
Event API
(API group
).
Example:
Value
must
be one of
user
,
namespace
, or
Example:
Value
must
be
,
,
Standard
must
be
latest
See
Example:
Value details 

an audit policy violation, it contains the
level
See
Example:
See
Auditing
Example:
decision
See
Auditing
Example:
.
.
Example:
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

d Set Up kubectl on Linux
Debug Pods
De

bug Running Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at

 the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Configuration (v1alpha1

)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Use a SOCKS5 Proxy to Access the Kubernetes API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

Tasks
minikube
.
ssh
Note:
SOCKS5
proxies.
You will use SSH client and server software to create a secure SOCKS5 tunnel between the local and
graph LR;
end
SOCKS5
sshd[SSH
end
enabled
This command starts a SOCKS5 proxy b

etween your client machine and the remote server.
ssh -D
1080
-D 1080
-q
-N
export
=
curl
Note:
localhost
localhost
.
The
curl
kubectl
proxy-url
element
cluster
:
v1
clusters
:
-
cluster
:
:
LRMEMMW2
server
:
proxy-url
:
name
:
default
contexts
:
-
context
:
cluster
:
default
user
:
default
name
:
default
:
default
kind
:
Config
:
{}
users
:
-
name
:
default
user
:
:
:
kubectl
Clean up
CTRL+C
Type
Feedback
Yes
No
.
or
.
Last modified June 14, 2022 at 10:56 PM PST:
Update content/en/docs/tasks/extend-kubernetes/socks5-proxy-access-api.md (1f22d45a3b)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Run a Replicated Stateful Application | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
This page shows how to run a replicated stateful application using a
.
This application is a replicated MySQL database. The example topology has a
single primary server and multiple replicas, using asynchronous row-based
replication.
Note:
This is not a production configuration
. MySQL settings remain on insecure defaults to keep the focus
on general patterns for running stateful applications in Kubernetes.
minikube
You need to either have a
dynamic PersistentVolume provisioner
with a default
,
or
statically provision PersistentVolumes
yourself to satisfy the
used here.
This tutorial assumes you are familiar with
and
,
as well as other core concepts like
Pods
,
Services
, and
.
Some familiarity with MySQL helps, but this tutorial aims to present
general patterns that should be useful for other systems.
You are using the default namespace or another namespace that does not contain any conflicting objects.
Deploy a replicated MySQL topology with a StatefulSet.
Send MySQL client traffic.
Observe resistance to downtime.
Scale the StatefulSet up and down.
Deploy MySQL
The example MySQL deployment consists of a ConfigMap, two Services,
and a StatefulSet.
Create the ConfigMap from the following YAML configuration file:
application/mysql/mysql-configmap.yaml
:
v1
kind
:
ConfigMap
metadata
:
name
:
mysql
labels
:
app
:
mysql
:
mysql
data
:
primary.cnf
:
|
# Apply this config only on the primary.
[mysqld]
log-bin
replica.cnf
:
|
# Apply this config only on replicas.
[mysqld]
super-read-only
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-configmap.yaml
This ConfigMap provides
my.cnf
overrides that let you independently control
configuration on the primary MySQL server and its replicas.
In this case, you want the primary server to be able to serve replication logs to replicas
and you want replicas to reject any writes that don't come via replication.
There's nothing

 special about the ConfigMap itself that causes different
portions to apply to different Pods.
Each Pod decides which portion to look at as it's initializing,
based on information provided by the StatefulSet controller.
Create Services
Create the Services from the following YAML configuration file:
application/mysql/mysql-services.yaml
# Headless service for stable DNS entries of StatefulSet members.
:
v1
kind
:
Service
metadata
:
name
:
mysql
labels
:
app
:
mysql
:
mysql
spec
:
ports
:
-
name
:
mysql
port
:
3306
clusterIP
:
None
selector
:
app
:
mysql
---
# Client service for connecting to any MySQL instance for reads.
# For writes, you must instead connect to the primary: mysql-0.mysql.
:
v1
kind
:
Service
metadata
:
name
:
mysql-read
labels
:
app
:
mysql
:
mysql
readonly
:
"true"
spec
:
ports
:
-
name
:
mysql
port
:
3306
selector
:
app
:
mysql
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-services.yaml
The headless Service provides a home for the DNS entries that the StatefulSet
creates for each
Pod that's part of the set.
Because the headless Service is named
mysql
, the Pods are accessible by
resolving
&lt;pod-name&gt;.mysql
from within any other Pod in the same Kubernetes
cluster and namespace.
The client Service, called
, is a normal Service with its own
cluster IP that distributes connections across all MySQL Pods that report
being Ready. The set of potential endpoints includes the primary MySQL server and all
replicas.
Note that only read queries can use the load-balanced client Service.
Because there is only one primary MySQL server, clients should connect directly to the
primary MySQL Pod (through its DNS entry within the headless Service) to execute
writes.
Create the StatefulSet
Finally, create the StatefulSet from the following YAML configuration file:
application/mysql/mysql-statefulset.yaml
:
apps/v1
kind
:
metadata
:
name
:
mysql
spec
:
selector
:
:
app
:
mysql
:
mysql
:
mysql
replicas
:
3
template
:
metadata
:
labels
:
app
:
mysql
:
mysql
spec
:
:
-
name
:
init-mysql
image
:
mysql:5.7
command
:
- bash
-
"-c"
- |
set -ex
# Generate mysql server-id from pod ordinal index.
[[ $HOSTNAME =~ -([0-9]+)$ ]] || exit 1
ordinal=${BASH_REMATCH[1]}
echo [mysqld] &gt; /mnt/conf.d/server-id.cnf
# Add an offset to avoid reserved server-id=0 value.
echo server-id=$((100 + $ordinal)) &gt;&gt; /mnt/conf.d/server-id.cnf
# Copy appropriate conf.d files from config-map to emptyDir.
if [[ $ordinal -eq 0 ]]; then
cp /mnt/config-map/primary.cnf /mnt/conf.d/
else
cp /mnt/config-map/replica.cnf /mnt/conf.d/
fi
:
-
name
:
conf
mountPath
:
/mnt/conf.d
-
name
:
config-map
mountPath
:
/mnt/config-map
-
name
:
clone-mysql
image
:
gcr.io/google-samples/xtrabackup:1.0
command
:
- bash
-
"-c"
- |
set -ex
# Skip the clone if data already exists.
[[ -d /var/lib/mysql/mysql ]] &amp;&amp; exit 0
# Skip the clone on primary (ordinal index 0).
[[ `hostname` =~ -([0-9]+)$ ]] || exit 1
[[ $ordinal -eq 0 ]] &amp;&amp; exit 0
# Clone data from previous peer.
ncat --recv-only mysql-$(($ordinal-1)).mysql 3307 | xbstream -x -C /var/lib/mysql
# Prepare the backup.
xtrabackup --prepare --target-dir=/var/lib/mysql
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
/etc/mysql/conf.d
:
-
name
:
mysql
image
:
mysql:5.7
env
:
-
name
:
MYSQL_ALLOW_EMPTY_PASSWORD
value
:
"1"
ports
:
-
name
:
mysql
:
3306
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
resources
:
requests
:
cpu
:
500m
memory
:
1Gi
:
exec
:
command
:
[
"mysqladmin"
,
"ping"
]
:
30
:
10
:
5
:
exec
:
# Check we can execute queries ove

r TCP (skip-networking is off).
command
:
[
"mysql"
,
"-h"
,
,
"-e"
,
"SELECT 1"
]
:
5
:
2
:
1
-
name
:
xtrabackup
image
:
ports
:
-
name
:
:
3307
command
:
- bash
-
"-c"
- |
set -ex
cd /var/lib/mysql
# Determine binlog position of cloned data, if any.
if [[ -f xtrabackup_slave_info &amp;&amp; "x$(&lt;xtrabackup_slave_info)" != "x" ]]; then
# XtraBackup already generated a partial "CHANGE MASTER TO" query
# because we're cloning from an existing replica. (Need to remove the tailing semicolon!)
cat xtrabackup_slave_info | sed -E 's/;$//g' &gt; change_master_to.sql.in
# Ignore xtrabackup_binlog_info in this case (it's useless).
rm -f xtrabackup_slave_info xtrabackup_binlog_info
elif [[ -f xtrabackup_binlog_info ]]; then
# We're cloning directly from primary. Parse binlog position.
[[ `cat xtrabackup_binlog_info` =~ ^(.*?)[[:space:]]+(.*?)$ ]] || exit 1
rm -f xtrabackup_binlog_info xtrabackup_slave_info
echo "CHANGE MASTER TO MASTER_LOG_FILE='${BASH_REMATCH[1]}',\
MASTER_LOG_POS=${BASH_REMATCH[2]}" &gt; change_master_to.sql.in
fi
# Check if we need to complete a clone by starting replication.
if [[ -f change_master_to.sql.in ]]; then
echo "Waiting for mysqld to be ready (accepting connections)"
until mysql -h 127.0.0.1 -e "SELECT 1"; do sleep 1; done
echo "Initializing replication from clone position"
mysql -h 127.0.0.1 \
-e "$(&lt;change_master_to.sql.in), \
MASTER_HOST='mysql-0.mysql', \
MASTER_USER='root', \
MASTER_PASSWORD='', \
MASTER_CONNECT_RETRY=10; \
START SLAVE;" || exit 1
# In case of container restart, attempt this at-most-once.
mv change_master_to.sql.in change_master_to.sql.orig
fi
# Start a server to send backups when requested by peers.
exec ncat --listen --keep-open --send-only --max-conns=1 3307 -c \
"xtrabackup --backup --slave-info --stream=xbstream --host=127.0.0.1 --user=root"
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
volumes
:
-
name
:
conf
emptyDir
:
{}
-
name
:
configMap
:
name
:
mysql
:
-
metadata
:
name
:
data
spec
:
:
[
]
resources
:
requests
:
storage
:
10Gi
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-statefulset.yaml
You can watch the startup progress by running:
app
=
mysql --watch
After a while, you should see all 3 Pods become
Running
:
mysql-0   2/2       Running   0          2m
mysql-1   2/2       Running   0          1m
mysql-2   2/2       Running   0          1m
Press
Ctrl+C
to cancel the watch.
Note:
If you don't see any progress, make sure you have a dynamic PersistentVolume
provisioner enabled, as mentioned in the
.
This manifest uses a variety of techniques for managing stateful Pods as part of
a StatefulSet. The next section highlights some of these techniques to explain
what happens as the StatefulSet creates Pods.
Understanding stateful Pod initialization
The StatefulSet controller starts Pods one at a time, in order by their
ordinal index.
It waits until each Pod reports being Ready before starting the next one.
In addition, the controller assigns each Pod a unique, stable name of the form
&lt;statefulset-name&gt;-&lt;ordinal-index&gt;
, which results in Pods named
mysql-0
,
mysql-1
, and
mysql-2
.
The Pod template in the above StatefulSet manifest takes advantage of these
properties to perform orderly startup of MySQL replication.
Generating configuration
Before starting any of the containers in the Pod spec, the Pod first runs any
in the order defined.
The first init container, named
, generates special MySQL config
files based on the ordinal index.
The script determines its own ordinal index by extracting it from the end of
the Pod name, which is returned by

 the
hostname
command.
Then it saves the ordinal (with a numeric offset to avoid reserved values)
into a file called
server-id.cnf
in the MySQL
conf.d
This translates the unique, stable identity provided by the StatefulSet
into the domain of MySQL server IDs, which require the same properties.
The script in the
container also applies either
or
from the ConfigMap by copying the contents into
conf.d
.
Because the example topology consists of a single primary MySQL server and any number of
replicas, the script assigns ordinal
0
to be the primary server, and everyone
else to be replicas.
Combined with the StatefulSet controller's
deployment order guarantee
,
this ensures the primary MySQL server is Ready before creating replicas, so they can begin
replicating.
Cloning existing data
In general, when a new Pod joins the set as a replica, it must assume the primary MySQL
server might already have data on it. It also must assume that the replication
logs might not go all the way back to the beginning of time.
These conservative assumptions are the key to allow a running StatefulSet
to scale up and down over time, rather than being fixed at its initial size.
The second init container, named
, performs a clone operation on
a replica Pod the first time it starts up on an empty PersistentVolume.
That means it copies all existing data from another running Pod,
so its local state is consistent enough to begin replicating from the primary server.
MySQL itself does not provide a mechanism to do this, so the example uses a
popular open-source tool called Percona XtraBackup.
During the clone, the source MySQL server might suffer reduced performance.
To minimize impact on the primary MySQL server, the script instructs each Pod to clone
from the Pod whose ordinal index is one lower.
This works because the StatefulSet controller always ensures Pod
N
is
Ready before starting Pod
N+1
.
Starting replication
After the init containers complete successfully, the regular containers run.
The MySQL Pods consist of a
mysql
container that runs the actual
mysqld
server, and an
container that acts as a
sidecar
.
The
sidecar looks at the cloned data files and determines if
it's necessary to initialize MySQL replication on the replica.
If so, it waits for
mysqld
to be ready and then executes the
CHANGE MASTER TO
and
START SLAVE
commands with replication parameters
extracted from the XtraBackup clone files.
Once a replica begins replication, it remembers its primary MySQL server and
reconnects automatically if the server restarts or the connection dies.
Also, because replicas look for the primary server at its stable DNS name
(
mysql-0.mysql
), they automatically find the primary server even if it gets a new
Pod IP due to being rescheduled.
Lastly, after starting replication, the
container listens for
connections from other Pods requesting a data clone.
This server remains up indefinitely in case the StatefulSet scales up, or in
case the next Pod loses its PersistentVolumeClaim and needs to redo the clone.
Sending client traffic
You can send test queries to the primary MySQL server (hostname
)
by running a temporary container with the
mysql:5.7
image and running the
mysql
client binary.
kubectl run mysql-client --image
=
mysql:5.7 -i --rm --restart
=
Never --
\
mysql -h mysql-0.mysql
CREATE DATABASE test;
CREATE TABLE test.messages (message VARCHAR(250));
INSERT INTO test.messages VALUES ('hello');
EOF
Use the hostname
to send test queries to any server that reports
being Ready:
=
mysql:5.7 -i -t --rm --restart
=
Never --
\
mysql -h mysql-read -e
"SELECT * FROM test.messages"
You should get output like this:
Waiting for pod default/mysql-client to be running, status is Pending, pod ready: false
+---------+
| message |
| hello   |
pod "mysql-client" deleted
To demonstrate that the
m

ysql-read
Service distributes connections across
servers, you can run
SELECT @@server_id
in a loop:
kubectl run mysql-client-loop --image
=
=
Never --
\
bash -ic
"while sleep 1; do mysql -h mysql-read -e 'SELECT @@server_id,NOW()'; done"
You should see the reported
@@server_id
change randomly, because a different
endpoint might be selected upon each connection attempt:
+-------------+---------------------+
| @@server_id | NOW()               |
|         100 | 2006-01-02 15:04:05 |
|         102 | 2006-01-02 15:04:06 |
|         101 | 2006-01-02 15:04:07 |
You can press
Ctrl+C
when you want to stop the loop, but it's useful to keep
it running in another window so you can see the effects of the following steps.
Simulate Pod and Node failure
To demonstrate the increased availability of reading from the pool of replicas
instead of a single server, keep the
loop from above
running while you force a Pod out of the Ready state.
Break the Readiness probe
The
readiness probe
for the
mysql
container runs the command
mysql -h 127.0.0.1 -e 'SELECT 1'
to make sure the server is up and able to execute queries.
One way to force this readiness probe to fail is to break that command:
kubectl
exec
mysql-2 -c mysql -- mv /usr/bin/mysql /usr/bin/mysql.off
This reaches into the actual container's filesystem for Pod
mysql-2
and
renames the
mysql
command so the readiness probe can't find it.
After a few seconds, the Pod should report one of its containers as not Ready,
which you can check by running:
kubectl get pod mysql-2
Look for
1/2
in the
READY
column:
mysql-2   1/2       Running   0          3m
At this point, you should see your
loop continue to run,
although it never reports
102
anymore.
Recall that the
script defined
server-id
as
100 + $ordinal
,
so server ID
102
corresponds to Pod
mysql-2
.
Now repair the Pod and it should reappear in the loop output
after a few seconds:
kubectl
exec
mysql-2 -c mysql -- mv /usr/bin/mysql.off /usr/bin/mysql
Delete Pods
The StatefulSet also recreates Pods if they're deleted, similar to what a
ReplicaSet does for stateless Pods.
kubectl delete pod mysql-2
The StatefulSet controller notices that no
mysql-2
Pod exists anymore,
and creates a new one with the same name and linked to the same
You should see server ID
102
disappear from the loop output for a while
and then return on its own.
Drain a Node
If your Kubernetes cluster has multiple Nodes, you can simulate Node downtime
(such as when Nodes are upgraded) by issuing a
drain
.
First determine which Node one of the MySQL Pods is on:
kubectl get pod mysql-2 -o wide
The Node name should show up in the last column:
NAME      READY     STATUS    RESTARTS   AGE       IP            NODE
mysql-2   2/2       Running   0          15m       10.244.5.27   kubernetes-node-9l2t
Then, drain the Node by running the following command, which cordons it so
no new Pods may schedule there, and then evicts any existing Pods.
Replace
&lt;node-name&gt;
with the name of the Node you found in the last step.
Caution:
Draining a Node can impact other workloads and applications
running on the same node. Only perform the following step in a test
cluster.
# See above advice about impact on other workloads
kubectl drain &lt;node-name&gt; --force --delete-emptydir-data --ignore-daemonsets
Now you can watch as the Pod reschedules on a different Node:
kubectl get pod mysql-2 -o wide --watch
It should look something like this:
NAME      READY   STATUS          RESTARTS   AGE 

      IP            NODE
mysql-2   2/2     Terminating     0          15m       10.244.1.56   kubernetes-node-9l2t
[...]
mysql-2   0/2     Pending         0          0s        &lt;none&gt;        kubernetes-node-fjlm
mysql-2   0/2     Init:0/2        0          0s        &lt;none&gt;        kubernetes-node-fjlm
mysql-2   0/2     Init:1/2        0          20s       10.244.5.32   kubernetes-node-fjlm
mysql-2   0/2     PodInitializing 0          21s       10.244.5.32   kubernetes-node-fjlm
mysql-2   1/2     Running         0          22s       10.244.5.32   kubernetes-node-fjlm
mysql-2   2/2     Running         0          30s       10.244.5.32   kubernetes-node-fjlm
And again, you should see server ID
102
disappear from the
loop output for a while and then return.
Now uncordon the Node to return it to a normal state:
Scaling the number of replicas
When you use MySQL replication, you can scale your read query capacity by
adding replicas.
For a StatefulSet, you can achieve this with a single command:
kubectl scale statefulset mysql  --replicas
=
5
Watch the new Pods come up by running:
app
=
Once they're up, you should see server IDs
103
and
104
start appearing in
the
loop output.
You can also verify that these new servers have the data you added before they
existed:
=
=
Never --
\
mysql -h mysql-3.mysql -e
| message |
| hello   |
Scaling back down is also seamless:
kubectl scale statefulset mysql --replicas
=
3
Note:
Although scaling up creates new PersistentVolumeClaims
automatically, scaling down does not automatically delete these PVCs.
This gives you the choice to keep those initialized PVCs around to make
scaling back up quicker, or to extract data before deleting them.
You can see this by running:
app
=
mysql
Which shows that all 5 PVCs still exist, despite having scaled the
StatefulSet down to 3:
data-mysql-0   Bound     pvc-8acbf5dc-b103-11e6-93fa-42010a800002   10Gi       RWO           20m
data-mysql-1   Bound     pvc-8ad39820-b103-11e6-93fa-42010a800002   10Gi       RWO           20m
data-mysql-2   Bound     pvc-8ad69a6d-b103-11e6-93fa-42010a800002   10Gi       RWO           20m
data-mysql-3   Bound     pvc-50043c45-b1c5-11e6-93fa-42010a800002   10Gi       RWO           2m
data-mysql-4   Bound     pvc-500a9957-b1c5-11e6-93fa-42010a800002   10Gi       RWO           2m
If you don't intend to reuse the extra PVCs, you can delete them:
kubectl delete pvc data-mysql-3
kubectl delete pvc data-mysql-4
Cancel the
loop by pressing
Ctrl+C
in its terminal,
or running the following from another terminal:
kubectl delete pod mysql-client-loop --now
Delete the StatefulSet. This also begins terminating the Pods.
kubectl delete statefulset mysql
Verify that the Pods disappear.
They might take some time to finish terminating.
app
=
mysql
You'll know the Pods have terminated when the above returns:
Delete the ConfigMap, Services, and PersistentVolumeClaims.
kubectl delete configmap,service,pvc -l
app
=
mysql
If you manually provisioned PersistentVolumes, you also need to manually
delete them, as well as release the underlying resources.
If you used a dynamic provisioner, it automatically deletes the
PersistentVolumes when it sees that you deleted the PersistentVolumeClaims.
Some dynamic provisioners (such as those for EBS and PD) also release the
underlying resources upon deleting the PersistentVolumes.
scaling a StatefulSet
.
debugging a StatefulSet
.
Learn more a

bout
.
force deleting StatefulSet Pods
.
Look in the
Helm Charts repository
for other stateful application examples.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
Node
2:
Namespace
3:
Event
4:
5:
Lease
6:
7:
8:
9:
Binding
10:
11:
1 - Node
Node
: v1
kind
: Node
metadata
(
)
spec
(
NodeSpec
)
status
(
)
NodeSpec
(string)
(string)
(string)
podCIDR
(string)
podCIDRs
(string)
taints
([]Taint)
(Time)
Time is a wrapper around time.Time which supports correct marsha

ling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package offers.
(string)
(boolean)
addresses
type
Quantity
)
capacity
Quantity
)
type
(Time)
(Time)
(string)
(string)
config
config.active.configMap.

(string)
(string)
(string)
(string)
(string)
LastKnownGood reports the checkpointed config the node will fall back to when it encounters an error attempting to use the Assigned config. The Assigned config becomes the LastKnownGood config when the node determines that the Assigned config is stable and correct. This is currently implemented as a 10-minute soak period starting when the local record of Assigned config is updated. If the Assigned config is Active at the end of this period, it becomes the LastKnownGood. Note that if Spec.ConfigSource is reset to nil (use local defaults), the LastKnownGood is also immediately reset to nil, because the local default config is al

ways assumed good. You should not make assumptions about the node's method of determining config stability and correctness, as this may change or become configurable in the future.
(string)
(string)
images
(int64)
nodeInfo
phase
(string)
The field is never populated, and now is deprecated.
L

ist of volumes that are attached to the node.
NodeList
: v1
kind
: NodeList
metadata
(
ListMeta
)
items
([]
Node
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
NodeList
): OK
create
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
202 (
Node
update
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
update
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
grac

ePeriodSeconds
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - Namespace
Namespace
: v1
kind
metadata
(
)
spec
(
)
status
(
)
type
(Time)
(string)
(string)
phase
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Namespace
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
resou

rceVersion
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
202 (
Namespace
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
3 - Event
Event
kind
: Event
metadata
(
)

eventTime
action
(string)
(int32)
(Time)
(Time)
(string)
(string)
note
(string)
reason
(string)
regarding
(
)
related
(
)
(string)
(string)
series
type
(string)
EventList
kind
metadata
(
ListMeta
)
items
([]
Event
get
read the specified Event
GET /apis/events.k8s.io/v

1/namespaces/{namespace}/events/{name}
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
create
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
202 (
Event
update
name
(
in path
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
p

ropagationPolicy
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
4 - APIService
kind
metadata
(
)
spec
(
)
status
(
)
caBundle
([]byte)
group
(string)
(boolean)
service
(string)
(string)
(int32)
port
version
(string)
Version is the API version this server host

s. For example, "v1"
type
(Time)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
): string, requi

red
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5 - Lease
Lease
kind
: Lease
metadata
(
)
More info:
spec
(
LeaseSpec
)
LeaseSpec
(string)
(int32)
(int32)
renewTime
LeaseList
kind
metadata
(
ListMeta
)
items
([]
Lease
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
list
namespac

e
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
create
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
202 (
Lease
update
name
(
in path
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
6 - RuntimeClass
RuntimeClass defines a class of container 

runtime supported in the cluster.
kind
metadata
(
)
More info:
handler
overhead
Quantity
)
.
(string)
(string)
(string)
(string)
(int64)
TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative val

ues will be treated as 0 (evict immediately) by the system.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7 - FlowSchema v1beta3
FlowSchema d

efines the schema of a group of flows. Note that a flow is made up of a set of inbound API requests with similar attributes and is identified by a pair of strings: the name of the FlowSchema and a "flow distinguisher".
kind
metadata
(
)
metadata
spec
(
)
spec
status
(
)
status
name
nil
type
(int32)
rules
rules
kind
group
for some well-known gro

up names. Required.
name
namespace
user
name
"
verbs
and
apiGroups
resources
verbs
(boolean)
([]string

)
is true.
type
(Time)
(string)
message
(string)
reason
(string)
status
(string)
type
kind
metadata
(
ListMeta
)
metadata
items
([]
items
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
HTTP Re

quest
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8 - PriorityLevelConfiguration v1beta3
kind
metadata
(
)
metadata
spec
(
)
spec
status
(
)
status
type
type
"Exempt"
means that requests of this priority level are not subject to a limit (and thus are never queued) and do not detract fro

m the capacity made available to other priority levels. A value of
"Limited"
are
limited
limited
type
is
"Limited"
.
(int32)
nil
(int32)
type
queuing
type
is
"Queue"
.
(int32)
handSize
handSize
queues
(int32)
(int32)
queues
l

imited.nominalConcurrencyShares
(int32)
type
(Time)
(string)
message
(string)
reason
(string)
status
(string)
type
kind
metadata
(
ListMeta
)
metadata
items
([]
items
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string

pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
9 - Binding
Binding
Binding ties one object to another; for example, a p

od is bound to a node by a scheduler. Deprecated in 1.7, please use the bindings subresource of pods instead.
: v1
kind
: Binding
metadata
(
)
target
(
create
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
create
name
(
in path
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
10 - ComponentStatus
: v1
kind
metadata
(
)
type
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
11 - ClusterCIDR v1alpha1
import "k8s.io/api/ne

tworking/v1alpha1"
kind
metadata
(
)
spec
(
)
ipv4
(string)
ipv6
(string)
([]
)
([]
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in q

uery
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure a Security Context for a Pod or Container | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Ope

 (v1)
Tasks
.
:
:
AppArmor
:
Seccomp
has
minikube
.
field
field is a
object.
and an
emptyDir
volume:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
runAsUser
:
1000
:
3000
fsGroup
:
2000
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
,
"sleep 1h"
]
:
-
name
:
mountPath
:
:
:
false
In the c

onfiguration file, the
runAsUser
will be root(0). Any files created will also be owned by user 1000 and group 3000 when
Since
fsGroup
kubectl
exec
ps
runAsUser
:
...
/data
cd
/data
ls -l
fsGroup
.
cd
demo
echo
ls -l
testfile
fsGroup
.
id
gid
field.
If the
gid
exit
fsGroup
mounted.
-
fsGroup
Always
:
runAsUser
:
1000
:
3000
fsGroup
:
2000
:
Note:
secret
,
configMap
,
and
emptydir
.
, the
fsGroup
as specified by CSI, the driver is expected to

 mount the volume with the
provided
fsGroup
fsGroup
.
field
field is a
object.
Security settings that you specify for a Container apply only to
field:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
runAsUser
:
1000
:
-
name
:
image
:
:
runAsUser
:
2000
:
false
kubectl
exec
ps aux
of
runAsUser
...
exit
With
,
field.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
sec-ctx-3
image
:
kubectl
exec
ps aux
cd
/proc/1
cat status
...
...
exit
adds the
and
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
sec-ctx-4
image
:
:
:
add
:
[
,
]
kubectl
exec
cd
/proc/1
cat status

capabilities bitmap for the process:
...
...
, and bit 25 is
.
See
Note:
CAP_XXX
.
omit the
CAP_
, include
SYS_TIME
field
in the
field is a
type
and
.
type
include
,
, and
Localhost
.
. It
flag).
...
:
:
type
:
:
...
:
:
type
:
Localhost
:
field in
the
field is an
...
:
:
level
:
Note:
.
and
must
.
have
set.
CSI
legacy
iscsi
-o context
by setting
in
Note:
nodes are
fsGroup
and
are
fsGroup
fsGroup
. See the
se

LinuxOptions
level
Warning:
After you specify an MCS label for a Pod, all Pods with the same label can access the Volume.
Clean up
document
Feedback
Yes
No
.
or
.
Last modified December 05, 2022 at 6:38 PM PST:
Fix nits in markdown links (a437285212)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Set up Konnectivity service | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
minikube
.
:
kind
:
:
-
name
:
cluster
:
:
GRPC
transport
:
uds
:
udsName
:
.
Set the
spec
:
:
:
-
name
:
mountPath
:
readOnly
:
false
volumes
:
-
name
:
hostPath
:
path
:
type
:
For example, you can us

e the OpenSSL command line tool to issue a X.509 certificate,
375
-sha256
SERVER
=
$(
jsonpath
=
)
=
true
"
$SERVER
"
=
true
static Pod
in your cluster. If not, you can deploy the Konnectivity
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
:
true
:
-
name
:
image
:
command
:
[
]
args
:
[
,
,
,
,
,
,
,
,
,
,
,
,
]
:
httpGet
:
scheme
:
HTTP
host
:
127.0.0.1
port
:
8134
path
:
/healthz
:
30
:
60
ports
:
-
name
:
agentport
:
8132
hostPort
:
8132
-
name
:
adminport
:
8133
hostPort
:
8133
-
name
:
:
8134
hostPort
:
8134
:
-
name
:
k8s-certs
mountPath
:
readOnly
:
true
-
name
:
mountPath
:
readOnly
:
true
-
name
:
mountPath
:
readOnly
:
false
volumes
:
-
name
:
k8s-certs
hostPath
:
path
:
-
name
:
hostPath
:
path
:
type
:
-
name
:
hostPath
:
path
:
type
:
:
apps/v1
kind
:
DaemonSet
metadata
:
labels
:
:
Reconcile
k8s-app
:
konnec

tivity-agent
namespace
:
name
:
spec
:
selector
:
:
k8s-app
:
template
:
metadata
:
labels
:
k8s-app
:
spec
:
:
:
-
key
:
operator
:
"Exists"
:
-
image
:
name
:
command
:
[
]
args
:
[
,
,
,
,
,
,
]
:
-
mountPath
:
name
:
:
httpGet
:
port
:
8134
path
:
/healthz
:
15
:
15
:
volumes
:
-
name
:
projected
:
sources
:
-
:
path
:
audience
:
:
kind
:
metadata
:
name
:
labels
:
:
"true"
:
Reconcile
roleRef
:
apiGroup
:
kind
:
name
:
subjects
:
-
apiGroup
:
kind
:
User
name
:
---
:
v1
kind
:
metadata
:
name
:
namespace
:
labels
:
:
"true"
:
Reconcile
Feedback
Yes
No
.
or
.
Last modified October 11, 2022 at 4:37 PM PST:
docs(setup-konnectivity.md): remove duplicate openssl option (1e2fd3cad8)
Home
Blog
Training
Partners
Community
CC BY 4.0

Define a Command and Arguments for a Container | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator patter

n
Tasks
Debug Pods
Determine the Reason for Pod 

Failure
Auditing
Run Jobs
TLS
Tutorials
Security

Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-a

piserver Configuration (v1alpha1)
Tasks
in a
Pod
.
minikube
.
command
the
args
Note:
The
command
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
purpose
:
spec
:
:
-
name
:
image
:
debian
command
:
[
]
args
:
[
,
]
:
OnFailure
env
:
-
nam

e
:
MESSAGE
value
:
command
:
[
]
args
:
[
]
and
Secrets
.
Note:
"$(VAR)"
. This is
command
or
args
field.
command:
[
"/bin/sh"
]
args:
[
"-c"
,
"while true; do echo hello; sleep 10;done"
]
.
.
See
Container
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tutorials
cluster
Feedback
Yes
No
.
or
.
Last modified July 12, 2021 at 9:01 AM PST:
Improvement to create cluster page (3766686671)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
1 - Protocols for Services
Service
,
SCTP
TCP
UDP
HTTP
and
HTTPS
TLS
protocol
SCTP
TCP
UDP
Services,
HTTP
configure a load balancer outside of your Kubernetes cluster, in a special mode
TCP
and add an
Note:
Ingress
to expose
http
or
https
. Use
http
https
if the
.
TCP
.
TLS
as
The connection from the load balancer to your workl

oad can also be TLS,
TCP
2 - Ports and Protocols
Protocol
Direction
Port Range
Purpose
Used By
TCP
Inbound
6443
All
TCP
Inbound
2379-2380
TCP
Inbound
10250
TCP
Inbound
10259
Self
TCP
Inbound
10257
Self
Protocol
Direction
Port Range
Purpose
Used By
TCP
Inbound
10250
TCP
Inbound
All
.
3 - Virtual IPs and Service Proxies
Every
node
).
The
virtual IP
Services
of
type
other than
.
cleanup
The kube-proxy's configuration is done via a Con

figMap, and the ConfigMap for
iptables
proxy mode
clusterIP
and
port
,
userspace
Example
earlier
All of the kube-proxy instances in the cluster observe the creation of the new
Service.
iptables
section
of the
):
...
iptables
:
:
1s
:
30s
...
The
0s
, multiple
Pod deletio

n events would get aggregated together, so kube-proxy might
1s
may
The
to avoid
1h
enable the
feature
gate
for
.
If you enable that feature gate and you were previously overriding
1s
or
metrics
In
ipvs
calls
netlink
state.
these are:
rr
lc
dh
sh
sed
nq
Note:
proxied 

to an appropriate backend without the clients knowing anything
to
ClientIP
None
).
Note:
virtual
IP
etcd
prior to creating each Service. The map object must exist in the registry for
ClusterIP
.
lower
Services.
and
fields
Cluster
and
Local
Cluster
Local
Local

Cluster
and
Local
to
Cluster
Local
to only
Local
If the
Local
all
NodePort
and
.
read
.
Read about
Services
Read about
Ingresses
Read the
Home
Blog
Training
Partners
Community
CC BY 4.0

Coarse Parallel Processing Using a Work Queue | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Run Jobs
Job
.
minikube
.
... [ previous 

line repeats several times .. hit return when it stops ] ...
....
[
]
....
....
[
]
....

Next we will verify we can create a queue, and publish and consume messages.
export
=
=
-q foo -d
foo
=
=
-q foo -c
1
cat
echo
Hello
-c 1
)
cat
printed.
=
-q job1  -d
job1
for
do
=
$f
done
import
sys
import
time
print
(
+
sys
.
stdin
.
0
])
time
.
sleep(
10
)
.
,
and
worker.py
For the
Docker Hub
, tag your ap

p image with
Registry
, tag
.
:
batch/v1
kind
:
Job
metadata
:
name
:
job-wq-1
spec
:
:
8
:
2
template
:
metadata
:
name
:
job-wq-1
spec
:
:
-
name
:
c
image
:
env
:
-
name
:
value
:
-
name
:
QUEUE
value
:
job1
:
OnFailure
In this example, each pod works on one item from the queue and then exits.
kubectl
wait
--for
=
condition
=
complete
--timeout
=
   c:
    Port:
Events:
.
example
I

n this example, we use the
A
Caveats
back to the api-server, then the Job will not appear to be complete, even though all items
Feedback
Yes
No
.
or
.
Caveats
Home
Blog
Training
Partners
Community
CC BY 4.0

Workload Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
Pod
2:
3:
4:
5:
6:
7:
8:
DaemonSet
9:
Job
10:
CronJob
11:
12:
13:
14:
15:
16:
17:
1 - Pod
Pod
: v1
kind
: Pod
metadata
(
)
spec
(
PodSpec
)
status
(
PodStatus
)
PodSpec
([]
Container
name
([]
Container
)
name
List of initialization containers belonging to the pod. Init containers are executed in order prior to containers being started. If any init container fails, the pod is considered to have failed and is handled according to its restartPolicy. The name for an init container or normal container must be unique among all containers. Init containers may not have Lifecycle actions, Readiness probes, Liveness probes, or Startup probes. The resourceRequirements of an init container are taken into account during scheduling by finding the highest request/limit for each resource type, and then using the max of of that value or the sum of the normal containers. Limits are applied to init containers in a similar fashion. Init containers cannot currently be added or removed. Cannot be updated. More info:
([]
)
name
([]
)
name
(boolean)
os
(PodOS)
If the OS field is set to linux, the following fields must be unset: -securityContext.win

dowsOptions
os.name
Volumes
volumes
([]
Volume
)
name
nodeName
(string)
affinity
Affinity is a group of affinity scheduling rules.
(
)
(
)
(
)
.
(string)
(string)
(string)
(string)
(int64)
(string)
If specified, the pod wil

l be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.
(string)
(string)
priority
(int32)
(string)
, it is the maximum permitted difference between the number of matching pods in the target topology and the global minimum. The global minimum is the minimum number of matching pods in an eligible domain or zero if the number of eligible domains is less than MinDomains. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 2/2/1: In this case, the global minimum is 1. | zone1 | zone2 | zone3 | | P P | P P | P | - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming pod can be scheduled onto any zone. When
but giving higher precedence to topologies that wo

uld help reduce the
skew.
more
(
)
(int32)
MinDomains indicates a minimum number of eligible domains. When the number of eligible domains with matching topology keys is less than minDomains, Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. And when the number of eligible domains with matching topology keys equals or greater than minDomains, this value has no effect on scheduling. As a result, when the number of eligible domains is less than minDomains, scheduler won't schedule more than maxSkew Pods to those domains. If value is nil, the constraint behaves as if MinDomains is equal to 1. Valid values are integers greater than 0. When value is not nil, WhenUnsatisfiable must be DoNotSchedule.
(string)
(string)
overhead
Quantity
)
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission co

ntroller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info:
Lifecycle
(string)
(int64)
(int64)
hostname
(string)
Specifies the hostname of the Pod If not specified, the pod's hostname will be set to a system-defined value.
(boolean)
subdomain
(string)
ip
(string)
dnsConfig
A list of DNS resolver options. This will be merged with the base options generated from DNSPolicy. Du

plicated entries will be removed. Resolution options given in Options will override those that appear in the base DNSPolicy.
(string)
Required.
(string)
dnsPolicy
(string)
(boolean)
hostPID
(boolean)
hostIPC
(boolean)
(boolean)
(string)
(boolean)
PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext. Field values of container.securityContext take precedence over field values of PodSecurityContext.
(int64)
(boolean)
(int64)
([]int64)
A list of groups applied to the first process run in each container, in addition to the container's primary GID, the fsGroup (if specified), and group memberships defined in the container image for the uid of the container process. If unspecified, no additional groups are added to any container. Note that group memberships defined in the container image for the uid of the container process are still effect

ive, even if they are not included in this list. Note that this field cannot be set when spec.os.name is windows.
(int64)
(string)
(string)
The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
(string)
(string)
(string)
(string)
(string)
https://github.com/kubernetes-sig

s/windows-gmsa
(string)
(boolean)
(string)
hostUsers
(boolean)
name
ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume them by name.
(string)
(string)

Gates
name
.
(string)
Container
name
Image
image
(string)
(string)
command
args
(string)
Ports
ports
(string)
(int32)
Number of port to expose on the host. If specified, this must be a valid port number, 0 &lt; x &lt; 65536. I

f HostNetwork is specified, this must match ContainerPort. Most containers do not need this.
(string)
(string)
env
name
env.name
env.value
(string)
(string)
(boolean)
(
)
,
(
)
(string)
(boolean)
envFrom
(string)
(boolean)
(string)

(string)
(boolean)
Volumes
mountPath
(string)
(boolean)
(string)
(string)
Resources
resources
Compute Resources required by this container. Cannot be updated. More info:
Quantity
)
Quantity
)
Lifecycle
lifecycle
(
)
PostStart is called immediately after a container is cre

ated. If the handler fails, the container is terminated and restarted according to its restart policy. Other management of the container blocks until the hook completes. More info:
(
)
(string)
(string)
(
Probe
)
(
Probe
)
(
Probe
)
(int64)
(boolean)
Indicates that the container must run as a non-root user. If true, the Kubelet will validate

 the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
(int64)
(boolean)
(string)
(boolean)
(boolean)
(string)
(string)
(string)
securityContext.seLin

uxOptions.type
(string)
(string)
(string)
(string)
(boolean)
(string)
Debugging
stdin
(boolean)
stdinOnce
(boolean)
tty
(boolean)
name
(string)
The container runtime must implement support for this feature. If the runtime does not support namespa

ce targeting then the result of setting this field is undefined.
Image
image
(string)
(string)
command
args
(string)
env
name
env.name
env.value
(string)
(string)
(boolean)
(
)
,
(
)
(SecretKeySelector

)
(string)
(boolean)
envFrom
(string)
(boolean)
(string)
(string)
(boolean)
Volumes
mountPath
Pod volumes to mount into the container's filesystem. Subpath mounts are not allowed for ephemeral containers. Cannot be updated.
(string)
(boolean)
(string)
(string)
Lifecycle
(string)
Optional: Path at which the file to which the container's termination message will be written is mounted into the container's filesystem. Message written is intended to be brief final status, such as an asse

rtion failure message. Will be truncated by the node if greater than 4096 bytes. The total message length across all containers will be limited to 12kb. Defaults to /dev/termination-log. Cannot be updated.
(string)
Debugging
stdin
(boolean)
stdinOnce
(boolean)
tty
(boolean)
(int64)
(boolean)
(int64)
(boolean)
(string)
(boolean)
securityC

ontext.allowPrivilegeEscalation
(boolean)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(boolean)
HostProcess determines if a container should be run as a 'Host Process' container. This field is alpha-level and will only be honored by components that enable the WindowsHostProcessContainers feature flag. Setting this field without the feature flag will result in errors when validating the Pod. All of a Pod's containers must have the same effective H

ostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers). In addition, if HostProcess is true then HostNetwork must also be set to true.
(string)
ports
(string)
(int32)
(string)
(string)
resources
Quantity
)
Quantity
)
lifecycle
(
)
(
)
PreStop is called immediately before a container is terminated due to an API request or management event such as liveness/startup probe failure, preemption, resource contention, etc. The handler is not called if the container crashes or exits. The Pod's termination grace period countdown begins before the PreStop hook is executed. Regardless of the outcome of the handler, the containe

r will eventually terminate within the Pod's termination grace period (unless delayed by finalizers). Other management of the container blocks until the hook completes or until the termination grace period is reached. More info:
(
Probe
)
(
Probe
)
(
Probe
)
exec
httpGet
(string)
(string)
(string)
tcpSocket
(string)
An empty preferred scheduling term matches all objects with implicit weight 0 (i.e. it's a no-op). A null preferred sche

duling term matches no objects (i.e. is also a no-op).
([]
)
([]
)
([]
)
([]
)
Pod affinity is a group of inter pod affinity scheduling rules.
(

)
(
)
(
)
(
)
The scheduler will prefer to schedule pods to nodes that satisfy the anti-affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling anti-affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node has pods which matches the corresponding podAffinityTerm; the node(s) with the highest sum are the most prefe

rred.
(
)
(
)
(
)
(
)
A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. null selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches al

l namespaces.
Probe
exec
httpGet
(string)
(string)
(string)
tcpSocket
(string)
(int32)
(int64)
(int32)
(int32)
Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is

 1. More info:
(int32)
(int32)
grpc
**
grpc.port
(string)
.
PodStatus
(string)
hostIP
(string)
startTime
(Time)
phase
(string)
The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. The conditions array, the reason and message fields, and the individual container status arrays contain more detail about the pod's status. There are five possible phase values:
More info:
message
(string)
reason
(string)
podIP
(string)
podIPs
([]PodIP)
ip
*IP address information for entries in the (plu

ral) PodIPs field. Each entry includes:
podIPs.ip
(string)
type
(Time)
(Time)
(string)
(string)
qosClass
(string)
.
(string)
(Time)
(string)
(Time)
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods th

at the time package offers.
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
(string)
initContainerStatuses.lastState.terminated.startedAt
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(boolean)
The list has one entry p

er container in the manifest. More info:
.
(string)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
(string)
(

int32), required
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(boolean)
.
(string)
(Time)
(string)
(Time)
ephemeralContainerStatuses.state.terminated.fini

shedAt
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
(string)
ephemeralContainerStatuses.lastState.terminated.startedAt
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(boolean)
PodList
PodList is

 a list of Pods.
items
([]
Pod
(string)
kind
(string)
metadata
(
ListMeta
)
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
name
(
in path
namespace
(
in path
namespace
container
(
in query
): string
follow
(
in query
): boolean
(
in query
): boolean
insecureSkipTLSVerifyBackend indicates that the apiserver should not confirm the validity of the serving certificate of the backend it is connecting to. This will make the HTTPS connection between the apiserver and the backend insecure. This means the apiserver cannot verify the log data it is receiving came from the real kubelet. If the kubelet is configured to verify the apiserver's TLS credentials, it does not mean the connection to the real kubelet is vulnerable to a man in the middle attack (e.g. an attacker could not intercept the actual log data coming from the real kubelet).
(
in query
): integer
pretty
(
in query
): string
pretty
previous
(
in query
): boolean
(
in query
): integer
tailLines
(
in query
): integer
(
in query
): boolean
Response
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
list
list or watch objects o

f kind Pod
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
create
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
202 (
Pod
update
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
replace ephemeralcontainers of the specified Pod
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): stri

ng
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Pod
): OK
202 (
Pod
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - PodTemplate
: v1
kind
metadata
(
)
template
(
)
metadata
(
)
spec
(
PodSpec
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
prett

y
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3 - ReplicationController
: v1
kind
metadata
(
)
If the 

Labels of a ReplicationController are empty, they are defaulted to be the same as the Pod(s) that the replication controller manages. Standard object's metadata. More info:
spec
(
)
status
(
)
selector
template
(
)
replicas
(int32)
(int32)
replicas
(int32)
(int32)
(int32)
type
(Time)
(string)
(string)
(int64)
: v1
kind
metadata
(
ListMeta
)
Standard list meta

items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK

201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
4 - ReplicaSet
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
https://kubernetes.io/docs/concep

ts/overview/working-with-objects/labels/#label-selectors
template
(
)
replicas
(int32)
(int32)
replicas
(int32)
(int32)
(int32)
type
(Time)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
resourceVersionMat

ch
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
Param

eters
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5 - Deployment
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
template
(
replicas
(int32)
(int32)
strategy
DeploymentStrategy describes how to replace existing pods with new ones.
(string)
The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. Defaults to 25%. Example: when this is set to 30%, the old ReplicaSet can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old ReplicaSet can be scaled down further, followed by scaling up the new ReplicaSet, ensuring that the

 total number of pods available at all times during the update is at least 70% of desired pods.
(int32)
(int32)
paused
(boolean)
replicas
(int32)
(int32)
(int32)
(int32)
(int32)
(int32)
type
Represents the latest available observations of a deployment's current state.
(Time)
(Time)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pre

tty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
delete a 

name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
6 - StatefulSet
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
template
(
replicas
(int32)
(string)
Type indicates the type of the StatefulSetUpdateStrategy. Default

 is RollingUpdate.
(int32)
(string)
Parallel
(int32)
([]
)
(int32)
(string)
Retain
causes PVCs to not be affected by StatefulSet dele

tion. The
Delete
(string)
Retain
Delete
ordinals
(int32)
replicas
(int32)
(int32)
(int32)
(int32)
(int32)
type
(Time)
(string)
(string)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
ge

t
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
St

atefulSet
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7 - ControllerRevision
ControllerRevision implements an immutable snapshot of state data. Clients are responsible for serializing and deserializing the objects that contain their internal state. Once a ControllerRevision has been successfully created, it can not be updated. The API Server will fail validation of all requests that attempt to mutate the Data field. ControllerRevisions may, however, be deleted. Note that, due to its use by both the DaemonSet and StatefulSet controllers for update and rollback, this object is beta. However, it may be subject to name and representation changes in future releases, and clients should not depend on its stability. It is primarily for internal use by controllers.
: apps/v1
kind
metadata
(
)
revision
data
}
}
}
}
{
},
}
So what happens? Decode first uses json or yaml to unmarshal the serialized data into your external MyAPIObject. That causes the raw JSON to

 be stored, but not unpacked. The next step is to copy (using pkg/conversion) into the internal struct. The runtime package's DefaultScheme has conversion functions installed which will unpack the JSON stored in RawExtension, turning it into the correct object type, and storing it in the Object. (TODO: In the case where the object is of an unknown type, a runtime.Unknown object will be created and stored.)*
: apps/v1
kind
metadata
(
ListMeta
)
More info:
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(

in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8 - DaemonSet
DaemonSet
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
template
(
(int32)
(string)
The maximum number of nodes with an existing available DaemonSet pod that can have an updated DaemonSet pod during during an update. Value can be an absolute number (ex: 5) or a percentage

 of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up to a minimum of 1. Default value is 0. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their a new pod created before the old pod is marked as deleted. The update starts by launching new pods on 30% of nodes. Once an updated pod is available (Ready for at least minReadySeconds) the old DaemonSet pod on that node is marked deleted. If the old pod becomes unavailable for any reason (Ready transitions to false, is evicted, or is drained) an updated pod is immediatedly created on that node without considering surge limits. Allowing surge implies the possibility that the resources consumed by the daemonset on any given node can double if the readiness check fails, and so resource intensive daemonsets should take into account that they may cause evictions during disruption.
(int32)
(int32)
(int32)
(int32)
(int32)
Count of hash c

ollisions for the DaemonSet. The DaemonSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.
type
(Time)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
DaemonSet
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
202 (
DaemonSet
update
name
(
in path
namespace
(
in path

namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
9 - Job
Job
: batch/v1
kind
: Job
metadata
(
)
spec
(
JobSpec
)
status
(
JobStatus
)
JobSpec
Replicas
template
(
Describes

 the pod that will be created when executing a job. More info:
(int32)
Lifecycle
(int32)
(string)
Indexed
.
Indexed
Indexed
.
(int32)
(int64)
(int32)
suspend
(boolean)
Selector
selector
(
)
(boolean)
unset unless you are certain what you are doing. When false or unset, the system pick labels unique to this job and appends thos

e labels to the pod template. When true, the user is responsible for picking unique labels and specifying the selector. Failure to pick a unique label may cause this and other jobs to not function correctly. However, You may see
Represents the relationship between the container exit code(s) and the specified values. Containers completed with success (exit code 0) are excluded from the requir

ement check. Possible values are:
(string)
JobStatus
startTime
(Time)
(Time)
active
(int32)
failed
(int32)
succeeded
(int32)
(string)
type
(Time)
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time pac

kage offers.
(Time)
(string)
(string)
counter.
Beta level
ready
(int32)
JobList
: batch/v1
kind
: JobList
metadata
(
ListMeta
)
items
([]
Job
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
create
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
fieldManage

r
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
202 (
Job
update
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
10 - CronJob
CronJob
: batch/v1
kind
: CronJob
metadata
(
)
spec
(
)
https://git.k8s.io/community/contributors/devel/sig-architect

ure/api-conventions.md#spec-and-status
status
(
)
(
)
(
JobSpec
)
schedule
.
timeZone
(string)
(string)
Specifies how to treat concurrent executions of a Job. Valid values are: - "Allow" (default): allows CronJobs to run concurrently; - "Forbid": forbids concurrent runs, skipping next run if previous run hasn't finished yet; - "Replace": cancels currently running job and replaces it with a new one
(int64)
suspend
(boolean)
(int32)
(int32)
active
([]
)
(Time)
(Time)
: batch/v1
kind
metadata
(
ListMeta
)
items
([]
CronJob
get
name
(
in p

ath
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
202 (
CronJob
update
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
name
(
in path
namespace
(
in path
):

 string, required
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
11 - HorizontalPodAutoscaler
kind
metadata
(
)
spec
(
)
.
status
(
)
(string)
(int32)
(int32)
currentCPUUtilizationP

ercentage
(int32)
(Time)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
fieldVa

lidation
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
12 - HorizontalPodAutoscaler
kind
metadat

a
(
)
spec
(
)
.
status
(
)
(string)
(int32)
behavior
(string)
(int32)
StabilizationWindowSeconds is the number of seconds for which past recommendations should be considered while sc

aling up or scaling down. StabilizationWindowSeconds must be greater than or equal to zero and less than or equal to 3600 (one hour). If not set, use the default values: - For scale up: 0 (i.e. no stabilization is done). - For scale down: 300 (i.e. the stabilization window is 300 seconds long).
(string)
(int32)
metrics
type
ContainerResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as spec

ified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory). The values will be averaged together before being compared to the target. Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source. Only one "target" type should be set.
(int32)
(
Quantity
)
(
Quantity
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
CrossVersionObjectReference contains enough information to

 let you identify the referred resource.
(string)
(
)
(int32)
(
Quantity
)
(
Quantity
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing each pod in the current scale target (e.g. CPU or memory). Such metric

s are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
(int32)
(
Quantity
)
(
Quantity
)
type
(Time)
(string)
(string)
ContainerResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing a single container in each po

d in the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
(int32)
(
Quantity
)
(
Quantity
)
(int32)
(
Quantity
)
(
Quantity
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
DescribedOb

ject specifies the descriptions of a object,such as kind,name apiVersion
(string)
(
)
(int32)
(
Quantity
)
(
Quantity
)
(
)
(int32)
currentAverageUtilization is the current value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource

 for the pods.
(
Quantity
)
(
Quantity
)
(int32)
(Time)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
HorizontalPodAutosca

ler
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
13 - PriorityClass

io/api/scheduling/v1"
kind
metadata
(
)
value
(string)
(boolean)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
DELETE /apis/scheduling.k8s.io/v1/pr

iorityclasses/{name}
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
14 - PodScheduling v1alpha1
kind
metadata
(
)
spec
(
status
(
)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): 

OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
HTTP Re

quest
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
15 - ResourceClaim v1alpha1
kind
metadata
(
)
spec
(
status
(
)
(string)
(string)
APIGroup is the group for the resource being referenced. It is empty for the core API. This matches the group in the APIVersion that is used when creating the reso

urces.
([]
)
([]
)
(string)
(boolean)
(boolean)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
HTTP Requ

est
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/resource.k8s.io/v1alpha1/namespaces/{namespace}/resourcecla

ims/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
16 - ResourceClaimTemplate v1alpha1
kind
metadata
(
)
spec
(
spec
(
metadata
(
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
labelSel

ector
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
resourceVersi

on
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
17 - ResourceClass v1alpha1
kind
metadata
(
)
(string)
(string)
Only nodes matching the selector will be considered by the scheduler when trying to find a Node that fits a Pod when that Pod uses a ResourceClaim that has not been allocated yet.
([]
)
([]
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): i

nteger
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
DELETE /apis/resource.k8s.io/v1alpha1/resourceclasses
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

Interactive Tutorial - Exposing Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Network Pl

ha1)
Tutorials
Home
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Pod
DaemonSet
Job
CronJob
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundatio

n, please see our

Kubernetes Security and Disclosure Information | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

Reference
Join the
.
.
A public disclosure date is negotiated by the Kubernetes Security Response Committee and the bug submitter. We prefer to fully disclose the bug as soon as possible once a user mitigation is available. It is reasonable to delay disclosure when the bug or the fix is not yet fully understood, the solution is not well-tested, or for vendor coordination. The timeframe for disclosure is from immediate (especially if it's already publicly known) to a few weeks. For a 

vulnerability with a straightforward mitigation, we expect report date to disclosure date to be on the order of 7 days. The Kubernetes Security Response Committee holds the final say when setting a disclosure date.
Feedback
Yes
No
.
or
.
Last modified February 07, 2022 at 11:56 AM PST:
Removed: link to broken RSS feed (9486fa2c65)
Home
Blog
Training
Partners
Community
CC BY 4.0

Custom Hugo Shortcodes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

.
.md
state
are:
alpha
beta
stable
Glossary
and
.
.
,
cluster
nodes
,
nodes
,
Pods
that are
fault-tolerance and high availabil

ity.
Pod
page
anchor
PodSpec
text
caption
to a table,
table
caption
Note:
{{
&lt;
table
caption
=
&gt;
}}
{{
&lt;
/
table
&gt;
}}
Parameter
Default
timeout
30s
logLevel
INFO
element:
&lt;
caption
style
=
caption
&gt;
Tabs
.md
The
tabs
name
codelang
tab
include
,
If not, the content page that needs to be included is looked up relative to the current page.
include
under
codelang
%
Note:
The tab
name
in a
tabs
{{
&lt;
tabs
name
=
&gt;
}}
{{
&lt;
tab
name
=
"Tab 1"
codelang
=
"bash"
&gt;
}}
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
tab
name
=
"Tab 2"
codelang
=
"go"
&gt;
}}
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
Tab 1
Tab 2
echo
println
{{
&lt;
tabs
name
=
&gt;
}}
{{
%
tab
name
=
%
}}
{{
&lt;
note
&gt;
}}
{{
&lt;
/
note
&gt;
}}
{{
%
/
tab
%
}}
{{
&lt;
tab
name
=
"HTML"
&gt;
}}
&lt;
div
&gt;
&lt;
h3
h3
&gt;
&lt;
p
i
i
p
&gt;
&lt;/
div
&gt;
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
Markdown
HTML
This is
Note:
It can even con

tain shortcodes.
Plain HTML
plain
HTML.
{{
&lt;
tabs
name
=
&gt;
}}
{{
&lt;
tab
name
=
include
=
/&gt;
}}
{{
&lt;
tab
name
=
include
=
/&gt;
}}
{{
&lt;
tab
name
=
include
=
/&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
JSON File
This is an
example
includes
Note:
example
includes
{
:
"v1"
,
"kind"
:
,
: {
"name"
:
"nginx"
},
: {
: {
"labels"
: {
"name"
:
"nginx"
},
:
"nginx-"
},
"spec"
: {
: [{
"name"
:
"nginx"
,
"image"
:
,
"ports"
: [{
:
80
}]
}]
}
}
}
DNS server
Lists
Items
.
latest
and
version
.
The
version
param
version
.
Note:
In previously released documentation,
latest
and
version
latest
version
version
as
v1.19
and
latest
as
v1.20
.
v1.26
The
latest
The
latest
version
v1.26
The
latest
1.26
The
version
.
.
The
from
latest
Hugo
.
.
.
.
.
Feedback
Yes
No
Thanks for the feedback. If you have a specific, answe

rable question about how to use Kubernetes, ask it on
.
or
.
Glossary
Tabs
Lists
Items
Home
Blog
Training
Partners
Community
CC BY 4.0

Tools for Monitoring Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
pods
,
services
, and
or
utility.
and
API.
kubelet
for CPU and
cAdvisor
).
or
API.
, a CNCF pro

ject, can natively monitor Kubernetes, nodes, and Prometheus itself.
Learn about additional debugging tools, including:
Logging
exec
Feedback
Yes
No
.
or
.
Last modified March 05, 2022 at 11:29 AM PST:
Remove 'Legacy Docker integration' from cAdvisor sentence (db056570a9)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
7:
1 - Documentation Content Guide
and ask!
.
.
Overview
project
.
and
, and
logging
.
Note:
.
Read the
.
2 - Documentation Style Guide
.
Note:
to support
Language
(see
).
.
UpperCamelCa

se
sentence-style capitalization
.
Code Style
.
Do
Don't
hostPath
field.
default
Do
Don't
Click
Fork
.
Select
Other
.
Do
Don't
A
cluster
.
.
Do
Don't
Open the
file.
Go to the
Open the
file.
Do
Don't
`
).
Do
Don't
The
Pod
.
Lease
…
A
.
.
**
_
.
Note:
D

o
Don't
replicas
exec
Run the process as a DaemonSet in the
Do
Don't
The
kubelet
The
kubectl
.
Do
Don't
The
kubeadm
kubeadm
Do
Don't
Do
Don't
to Always.
image
image
to
.
replicas
field to 2.
replicas
field to
2
.
pod
,
pods
,
)
pod
,
secret
)
Secret
Secret
".
`
).
and
.
.
Do
Don't
=
wide
NAME     READY     STATUS    RESTARTS 

  AGE    IP           NODE
If the information is version specific, the Kubernetes version needs to be defined in the
or the
.
---
title
:
:
v1.8
---
:
v1
kind
:
Pod
...
Term
Usage
Docker
SIG Docs
Hugo
Note
,
Caution
, and
Warning
.
Note:
Note
Use
Note:
You can
still
in a list:

Note:
.
Caution
Use
Caution:
Warning
Use
Beware.
Warning:
Beware.
It lowers the barrier of entry by allowing users to use Minikube with one click instead of going through the compl

ete
Caution:
{{&lt; kat-button &gt;}}

Note:
Do
Don't
#
---
or
===
Do
Don't
---
Links
Do
Don't
here
.
[example 

website](https://example.com){target="_blank"}
Lists
Group items in a list that are related to each other and need to appear in a specific order or to indicate a correlation between multiple items. When a screen reader comes across a list—whether it is an ordered or unordered list—it will be announced to the user that there is a group of list items. The user can then use the arrow keys to move up and down between the various items in the list.
Note:
1.
Use (
+
), (
*
), or (
-
Tables
Do
Don't
meaning.
Do
Don't
Do
Don't
Do
Don't
Do
Don't
e.g., ...
i.e., ...
Do
Don't
Do
Don't
An exception to this rule is docume

ntation about announced deprecations
is the
.
Avoid words like "currently" and "new." A feature that is new today might not be
Do
Don't
Do
Don't
.
.
.
3 - Diagram Guide
generate
.svg
or
.png
S[ ]-.-
C[build
diagrams
D[on-line
end
diagrams]
and
captions]
click C "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgc3ViZ3JhcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0i

hcGggbVtNZXJtYWlkLmpzXVxuICAgIGRpcmVjdGlvbiBUQlxuICAgICAgICBTWyBdLS4tXG4gICAgICAgIENbYnVpbGQ8YnI-ZGlhZ3JhbXM8YnI-d2l0aCBtYXJrZG93bl0gLS0-XG4gICAgICAgIERbb24tbGluZTxicj5saXZlIGVkaXRvcl1cbiAgICBlbmRcbiAgICBBW1doeSBhcmUgZGlhZ3JhbXM8YnI-dXNlZnVsP10gLS0-IG1cbiAgICBtIC0tPiBOWzMgeCBtZXRob2RzPGJyPmZvciBjcmVhdGluZzxicj5kaWFncmFtc11cbiAgICBOIC0tPiBUW0V4YW1wbGVzXVxuICAgIFQgLS0-IFhbU3R5bGluZzxicj5hbmQ8YnI-Y2FwdGlvbnNdXG4gICAgWCAtLT4gVltUaXBzXVxuICAgIFxuIFxuICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIHNwYWNld2hpdGUgZmlsbDojZmZmZmZmLHN0cm9rZTojZmZmLHN0cm9rZS13aWR0aDowcHgsY29sb3I6IzAwMFxuICAgIGNsYXNzIEEsQyxELE4sWCxtLFQsViBib3hcbiAgICBjbGFzcyBTIHNwYWNld2hpdGUiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOnRydWV9" _blank
enabled
Using
.
Using the
.
.
process.
Note:
Mermaid
Mermaid
Note:
below.
The
is
Generate

s a URL for each saved diagram. The URL is displayed in the URL
.svg
or
.png
files.
Note:
graph TB
B[Inline
A --&gt; B
A --&gt; C
A --&gt; D
enabled
Inline
method.
graph LR
to page markdown file

] --&gt;
enabled
.md
Note:
in the
.md
.md
file:
---
---
...
    graph TB
    A --&gt; B

more text
Note:
.
.md
file.
No need for s

eparate
.svg
.md
file.
local
Caution:
diagram]
end
A --&gt; B
B --&gt; C
C --&gt; w
click D "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgbGl2ZSBlZGl0b3I8YnI-IHRvIGNyZWF0ZS9lZGl0PGJyPmRpYWdyYW1dXG4gICAgQlsyLiBTdG9yZSBkaWFncmFtPGJyPlVSTCBzb21ld2hlcmVdXG4gICAgQ1szLiBHZW5lcmF0ZSAuc3ZnIGZpbGU8YnI-YW5kIGRvd25sb2FkIHRvPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSAuc3ZnPGJyPmZpbGUgaW4gcGFnZTxicj4ubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbkEgLS0-IEJcbkIgLS0-IENcbkMgLS0-IHdcblx

uICAgIGNsYXNzRGVmIGJveCBmaWxsOiNmZmYsc3Ryb2tlOiMwMDAsc3Ryb2tlLXdpZHRoOjFweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzIEEsQixDLEQsRSx3IGJveFxuICAgICIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
enabled
.svg
images/
folder.
Use the
.md
file.
caption
boxnet
.
folder.
Use the
.md
the
.svg
{
{&lt;
figure
src=
alt=
class=
caption=
&gt;
}
}
.
Note:
The
.svg
.
.svg
.svg
.svg
local
.svg
or
.png
.svg
diagram]
access]
png file in
end
A --&gt; B
B --&gt; C
C --&gt; w
click A "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZmxvd2NoYXJ0IExSXG4gICAgQVsxLiBVc2UgZXh0ZXJuYWw8YnI-dG9vbCB0b

mFwcHJvcHJpYXRlPGJyPmltYWdlcy8gZm9sZGVyXVxuICAgIHN1YmdyYXBoIHdbIF1cbiAgICBkaXJlY3Rpb24gVEJcbiAgICBEWzQuIFVzZSBmaWd1cmUgc2hvcnRjb2RlPGJyPnRvIHJlZmVyZW5jZSBzdmcgb3I8YnI-cG5nIGZpbGUgaW48YnI-cGFnZSAubWQgZmlsZV0gLS0-XG4gICAgRVs1LiBBZGQgY2FwdGlvbl1cbiAgICBlbmRcbiAgICBBIC0tPiBCXG4gICAgQiAtLT4gQ1xuICAgIEMgLS0-IHdcbiAgICBjbGFzc0RlZiBib3ggZmlsbDojZmZmLHN0cm9rZTojMDAwLHN0cm9rZS13aWR0aDoxcHgsY29sb3I6IzAwMDtcbiAgICBjbGFzcyBBLEIsQyxELEUsdyBib3hcbiAgICAiLCJtZXJtYWlkIjoie1xuICBcInRoZW1lXCI6IFwiZGVmYXVsdFwiXG59IiwidXBkYXRlRWRpdG9yIjpmYWxzZSwiYXV0b1N5bmMiOnRydWUsInVwZGF0ZURpYWdyYW0iOmZhbHNlfQ"
enabled
.xml
.svg
or
.png
folder.
Use the
.md
file.
caption
Here is the
diagram:
.svg
or
.png
.xml
.
local
Examples
Note:
page.
graph TB
n3(Node3)
n4(Node4)
end
n1(Node1)
n2(Node2)
end
click n4 "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggVEJcbiA

gICBzdWJncmFwaCBcInpvbmVCXCJcbiAgICAgICAgbjMoTm9kZTMpXG4gICAgICAgIG40KE5vZGU0KVxuICAgIGVuZFxuICAgIHN1YmdyYXBoIFwiem9uZUFcIlxuICAgICAgICBuMShOb2RlMSlcbiAgICAgICAgbjIoTm9kZTIpXG4gICAgZW5kXG5cbiAgICBjbGFzc0RlZiBwbGFpbiBmaWxsOiNkZGQsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojMDAwO1xuICAgIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICAgIGNsYXNzRGVmIGNsdXN0ZXIgZmlsbDojZmZmLHN0cm9rZTojYmJiLHN0cm9rZS13aWR0aDoycHgsY29sb3I6IzMyNmNlNTtcbiAgICBjbGFzcyBuMSxuMixuMyxuNCBrOHM7XG4gICAgY2xhc3Mgem9uZUEsem9uZUIgY2x1c3RlcjtcbiIsIm1lcm1haWQiOiJ7XG4gIFwidGhlbWVcIjogXCJkZWZhdWx0XCJcbn0iLCJ1cGRhdGVFZGl0b3IiOmZhbHNlLCJhdXRvU3luYyI6dHJ1ZSwidXBkYXRlRGlhZ3JhbSI6dHJ1ZX0" _blank
enabled
graph TB
       n3(Node3)
       n4(Node4)
   end
       n1(Node1)
       n2(Node2)
   end
 
page.
graph LR;
ingress;
end
click client "https://mermaid-js.github.io/mermaid-live-editor/edit/#eyJjb2RlIjoiZ3JhcGggIExSXG4gIGNsaWVudChbY2xpZW50XSktLiBJbmdyZXNzLW1hbmFnZWQgPGJyPiBsb2FkIGJhbGFuY2VyIC4tPmluZ3Jlc3NbSW5ncmVzc107XG4gIGluZ3Jlc3MtLT58cm91dGluZyBydWxlfHNlcnZpY2VbU2VydmljZV07XG4gIHN1YmdyYXBoIGNsdXN0ZXJcbiAgaW5ncmVzcztcbiAgc2VydmljZS0tPnBvZDFbUG9kXTtcbiAgc2VydmljZS0tPnBvZDJbUG9kXTtcbiAgZW5kXG4gIGNsYXNzRGVmIHBsYWluIGZpbGw6I2RkZCxzdHJva2U6I2ZmZixzdHJva2Utd2lkdGg6NHB4LGNvbG9yOiMwMDA7XG4gIGNsYXNzRGVmIGs4cyBmaWxsOiMzMjZjZTUsc3Ryb2tlOiNmZmYsc3Ryb2tlLXdpZHRoOjRweCxjb2xvcjojZmZmO1xuICBjbGFzc0RlZiBjbHVzdGVyIGZpbGw6I2ZmZixzdHJva2U

SI6ZmFsc2V9" _blank
enabled
graph LR;
 ingress;
 end
    actor me
classDef
class
figure 7
,
classDef
and
class
#326ce5
.
user.
flowchart
A[Diagram
in text]
click A "https://mermaid-js.github.io/mermaid-live-editor/edit#eyJjb2RlIjoiZmxvd2NoYXJ0XG4gICAgQVtEaWFncmFtPGJyPjxicj5JbmxpbmUgTWVybWFpZCBvcjxicj5TVkcgaW1hZ2UgZmlsZXNdXG4gICAgQltEaWFncmFtIENhcHRpb248YnI-PGJyPkFkZCBGaWd1cmUgTnVtYmVyLiBhbmQ8YnI-Q2FwdGlvbiBUZXh0XVxuICAgIENbRGlhZ3JhbSBSZWZlcnJhbDxicj48YnI-UmVmZXJlbmVuY2UgRmlndXJlIE5

1bWJlcjxicj5pbiB0ZXh0XVxuXG4gICAgY2xhc3NEZWYgYm94IGZpbGw6I2ZmZixzdHJva2U6IzAwMCxzdHJva2Utd2lkdGg6MXB4LGNvbG9yOiMwMDA7XG4gICAgY2xhc3MgQSxCLEMgYm94IiwibWVybWFpZCI6IntcbiAgXCJ0aGVtZVwiOiBcImRlZmF1bHRcIlxufSIsInVwZGF0ZUVkaXRvciI6ZmFsc2UsImF1dG9TeW5jIjp0cnVlLCJ1cGRhdGVEaWFncmFtIjpmYWxzZX0" _blank
enabled
Note:
Diagram
The
and
.svg
Here is the
.svg
:
src
,
alt
,
class
and
caption
,
and
classes.
Note:
Inline
See
.svg
caption
Use the
and
diagrams.
Inline
method.
Figure
BELOW
match.
or

shortcode
link
Here is the
Tips
.svg
or
.png
With the
and
.svg
.svg
must
save it
.
4 - Writing a new topic
Open a PR
.
Type
Concept
Nodes
.
Task
Tutorial
A tutorial page shows how to accomplish a goal that ties together several Kubernetes features. A tutorial might provide several sequences of steps that readers can actually do as they read the page. Or it 

might provide explanations of related pieces of code. For example, a tutorial could provide a walkthrough of a code sample. A tutorial can include brief explanations of the Kubernetes features that are being tied together, but should link to related concept topics for deep explanations of individual features.
Use a
to create
hugo new
title
.
---
---
create
contents.
_index.md
_index.md
weight:
10
20
.
.
,
one of the
the topic. In your topic file

, use the
codenew
where
examples
.
Note:
&lt;
and before
the
&gt;
under
.
Note:
file. The
.
/images
.
.
5 - Page content types
Concept
Task
Tutorial
Reference
heading
heading
whatsnext
cleanup
synopsis
seealso
options
whatsnext
The
heading
files.
:
other =
:
other =
"다음 내용"
Concept
overview
body
whatsnext
The
overview
and
body
whatsnext
heading
For
overview
For
body
For
whatsnext
Task
sequence of steps. Task pages have minimal explanati

on, but often provide links
overview
steps
whatsnext
The
overview
,
steps
, and
and
whatsnext
with the
heading
#
For
overview
For
include
For
steps
in
steps
.
For
whatsnext
.
Tutorial
overview
cleanup
whatsnext
The
overview
,
, and
,
cleanup
, and
whatsnext
with the
heading
#
For
overview
For
For
For
For
cleanup
For
whatsnext
.
Reference
synopsis
options
examples
seealso
kubectl
6 - Content organization
Note:
Hugo Tip:
Page Lists
Page Order
title
:
My Page
weight
:
10
Note:
For page weights, it can be smart not to use 1, 2, 3 ..., but some

 other interval, say 10, 20, 30... This allows you to insert pages where you want later.
Additionally, each weight within the same directory (section) should not be overlapped with the other weights. This makes sure that content is always organized correctly, especially in localized content.
The
docs/
with the
main_menu
_index.md
main_menu
:
true
linkTitle
main_menu
:
true
title
:
Page Title
linkTitle
:
Note:
_index.md
docs/
.
toc_hide
flag to
true
toc_hide
:
true
_index.md
.
toc_hide
flag to
true
toc_hide
:
true
.
index.md
includes
Resources
.
of a
Resource
.
Styles
The
SASS
7 - Custom Hugo Shortcodes
.
.md
state
are:
alpha
beta
stable
The 

displayed Kubernetes version defaults to that of the page or the site. You can change the
Glossary
and
.
.
,
cluster
nodes
,
nodes
,
Pods
that are
Pod
page
anchor
PodSpec
text
caption
to a table,
table
caption
Note:
{{
&lt;
table
caption
=
&gt;
}}
{{
&lt;
/
table
&gt;
}}
Parameter
Default
timeout
30s
logLevel
INFO
element:
&lt;
caption
style
=
caption
&gt;
Tabs
.md
The
tabs
name
codelang
tab
shortcode, you

 can tell Hugo
include
,
include
under
codelang
%
Note:
The tab
name
in a
tabs
{{
&lt;
tabs
name
=
&gt;
}}
{{
&lt;
tab
name
=
"Tab 1"
codelang
=
"bash"
&gt;
}}
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
tab
name
=
"Tab 2"
codelang
=
"go"
&gt;
}}
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
Tab 1
Tab 2
echo
println
{{
&lt;
tabs
name
=
&gt;
}}
{{
%
tab
name
=
%
}}
{{
&lt;
note
&gt;
}}
{{
&lt;
/
note
&gt;
}}
{{
%
/
tab
%
}}
{{
&lt;
tab
name
=
"HTML"
&gt;
}}
&lt;
div
&gt;
&lt;
h3
h3
&gt;
&lt;
p
i
i
p
&gt;
&lt;/
div
&gt;
{{
&lt;
/
tab
&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
Markdown
HTML
This is
Note:
Plain HTML
plain
HTML.
{{
&lt;
tabs
name
=
&gt;
}}
{{
&lt;
tab
name
=
include
=
/&gt;
}}
{{
&lt;
tab
name
=
include
=
/&gt;
}}
{{
&lt;
tab
name
=
include
=
/&gt;
}}
{{
&lt;
/
tabs
&gt;
}}
JSON File
This is an
example
includes
Note:
example
includes
{
:
"v1"
,
"kind"
:
,
: {
"name"
:
"nginx"
},
: {
: {
"labels"
: {
"name"
:
"nginx"
},
:
"nginx-"
},
"spec"
: {
: [{
"name"
:
"nginx"
,
"image"
:
,
"ports"
: [{
:
80
}]
}]
}
}
}
DNS server
Lists
Items
.
latest
and
version
.
{{&lt; param "version" 

&gt;}}
The
version
param
version
.
Note:
latest
and
version
latest
version
version
as
v1.19
and
latest
as
v1.20
.
v1.26
The
latest
The
latest
version
v1.26
The
latest
1.26
The
version
.
.
The
from
latest
Hugo
.
.
.
.
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
a list of
.
1 - Kubelet Checkpoint API
root
user. It
post
CRI
.
tar
tar
namespace
(
in path
Namespace
pod
(
in path
Pod
container
(
in path
Container
timeout
(
in query
): integer
CRI
Response
200: OK
namespace
,
pod
or
container
2 - Kubelet Device Manager API Versions
This page provides details of version compatibility 

between the Kubernetes
,
v1alpha1
v1beta1
-
✓
-
✓
-
✓
-
✓
-
✓
-
✓
Key:
✓
+
-
3 - Articles on dockershim Removal and on Using CRI-compatible Runtimes
,
(
)
. (
)
"
Mirantis:
Tripwire:
Home
Blog
Training
Partners
Community
CC BY 4.0

Assign Pods to Nodes using Node Affinity | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
minikube
.
=
worker0
=
worker1
=
worker2
disktype
=
ssd
where
label:
worker0
node has a
label.
.
label.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
affinity
:
:
:
:
-
:
-
key
:
disktype
operator
:
In
values
:
- ssd
:
-
name
:
nginx
image
:
nginx
imagePullPo

licy
:
=
wide
.
label.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
affinity
:
:
:
-
weight
:
1
:
:
-
key
:
disktype
operator
:
In
values
:
- ssd
:
-
name
:
nginx
image
:
nginx
:
kubectl apply -f https://k8s.io/examples/pods/pod-nginx-preferred-affinity.yaml
=
wide
.
Feedback
Yes
No
.
or
.
Last modified August 28, 2022 at 1:28 AM PST:
[en] fix typo assign-pods-nodes-using-node-affinity.md (100f074e0c)
Home
Blog
Training
Partners
Community
CC BY 4.0

Securing a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

on (v1)
Tasks
minikube
.
Choose an authentication mechanism for the API servers to use that matches the common access patterns
roles are provided that offer re

asonable default separation of responsibility depending on what
Node
and
RBAC
policies
in a
namespace
Generally, most application workloads need limited access to host resources so they can
Baseline
or
the node, or add rules to block them. On most Linux 

distributions, you can do that by

The
and the
This section describes some common patterns for protecting clusters from compromise.
Caution:
The
Alpha and beta Kubernetes features are in active 

development and may have limitations or bugs
do not use.
like the
.
If you use
and allow
kube-
Secret
or
ConfigMap
encryption-at-rest for extension APIs defined in CustomResourceDefinitions was added to
Join the
Feedback
Yes
No
.
or
.
Last modified November 29, 2022 at 7:18 PM PST:
Document custom resource encryption support (#38109) (765de36b1d)
Preve

nting containers from loading unwanted kernel modules
Home
Blog
Training
Partners
Community
CC BY 4.0

ValidatingAdmissionPolicyBindingList v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Inst

alling Addons
Tasks
Migrate from Po

dSecurityPolicy to the Built-In PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externali

zing config using MicroProfile, ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Client A

uthentication (v1beta1)
Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Access Clusters Using the Kubernetes API | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

ha1)
Tasks
minikube
.
kubectl
.
a
,
Many of the
examples
.
curl
or
wget
=
8080
&amp;
See
{
: [
"v1"
],
"serverAddr

essByClientCIDRs"
: [
{
:
,
:
}
]
}
Using
grep/cut
approach:
jsonpath
=
export
=
APISERVER
=
$(
jsonpath
=
)
metadata:
EOF
while
'^token'
do
echo
&gt;&amp;
2
sleep
1
done
TOKEN
=
$(
jsonpath
=
)
curl -X GET
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
~/.kube
for this.
Go
,
Python
,
Java
,
dotnet
,
, and
Haskell
Go client
See
Note:
example
:
package
main
import
(
"context"
"fmt"
)
func
main
() {
config, _
:=
(
""
,
)
:=
(config)
pods, _
:=
CoreV1
().
Pods
(
""
).
List
(context.
TODO
fmt.
Printf
(
,
len
}
.
To use
. See
P

ython Client Library page
example
:
from
import
config
.
v1
=
client
.
print
(
)
ret
=
v1
.
=
False
)
for
i
in
ret
.
items:
print
(
"
%s
\t
%s
\t
%s
"
%
(i
.
status
.
pod_ip, i
.
metadata
.
.
metadata
.
name))
, run:
cd
java
See
example
:
package
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
/**
*
* &lt;p&gt;Easiest way to run this: mvn exec:java
*
*/
public
class
{
public
static
void
main
(
String
[]
args
)
throws
,
{
=
;
=
.
(
.
(
new
(
))).
build
();
.
(
client
);
=
new
CoreV1Api
();
=
api
.
(
null
,
null
,
null
,
null
,
null
,
null
,
null
,
null
,
null
);
System
.
out
.
println
(
);
for
(
V1Pod item
:
list
.
getItems
())
{
System
.
out
.
println
(
item
.
().
getName
());
}
}
}
To use
See
example
:
using
System
;
using
k8s
;
namespace
simple
{
internal
class
PodList
{
private
static
void
Main(
string
[] args)
{
var
new
);
var
"default"
);
foreach
(
var
item
in
{
}
if
0
)
{
"Empty!"
);
}
}
}
}
To install
. See
example
:
const
k8s
=
require(
);
const
kc
=
new
const
k8sApi
=
'default'
});
Ha

skell client
See
The
example
:
::
IO
()
=
do
oidcCache
&lt;-
$
newTVar
$
Map
.
fromList
[]
&lt;-
$
mgr
kcfg
(
CoreV1
.
Accept
MimeJSON
))
&gt;&gt;=
print
Feedback
Yes
No
.
or
.
Last modified October 05, 2022 at 5:11 PM PST:
Add missing period (64ca4f1e2e)
Home
Blog
Training
Partners
Community
CC BY 4.0

Perform a Rollback on a DaemonSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

alpha1)
Tasks
DaemonSet
.
minikube
.
DaemonSet
.
history
1               ...
2               ...
...
in
kubectl
history
=
1
app:
 Image:        ...
 Port:         ...
=
Note:
If
.

=
revision.
or
.
Note:
.revision
have revision 1 and 2 in the system, and roll back from revision 2 to revision
.
See
update
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Connect a Frontend to a Backend Using Services | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
This task shows how to create a
frontend
and a
backend
microservice. The backend
microservice is a hello greeter. The frontend exposes the backend using nginx and a
Service
object.
Create and run a sample
hello
backend microservice using a
object.
Use a Service object to send traffic to the backend microservice's multiple replicas.
Create and run a
nginx
frontend microservice, also using a Deployment object.
Configure the frontend microservice to send traffic to the backend microservice.
Use a Service object of
to expose the frontend microservice
outside the cluster.
minikube
.
This task uses
Services with external load balancers
, which
require a supported environment. If your environment does not support this, you can use a Service of type
NodePort
instead.
Creating the backend using a Deployment
The backend is a simple hello greeter microservice. Here is the configuration
file for the backend Deployment:
service/access/backend-deployment.yaml
---
:
apps/v1
kind
:
metadata
:
name
:
backend
spec
:
selector
:
:
app
:
hello
tier
:
backend
track
:
stable
replicas
:
3
template
:
metadata
:
labels
:
app
:
hello
tier
:
backend
track
:
stable
spec
:
:
-
name
:
hello
image
:
"gcr.io/google-samples/hello-go-gke:1.0"
ports
:
-
name
:
http
:
80
...
Create the backend Deployment:
kubectl apply -f https://k8s.io/examples/service/access/backend-deployment.yaml
View information about the backend Deployment:
kubectl describe deployment backend
Name:                           backend
Namespace:                      default
CreationTimestamp:              Mon, 24 Oct 2016 14:21:02 -0700
Labels:                         app=hello
                                tier=backend
                                track=stable
Annotations:                    deployment.kubernetes.io/revision=1
Selector:  

                     app=hello,tier=backend,track=stable
Replicas:                       3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:                   RollingUpdate
MinReadySeconds:                0
RollingUpdateStrategy:          1 max unavailable, 1 max surge
  Labels:       app=hello
                tier=backend
                track=stable
   hello:
    Image:              "gcr.io/google-samples/hello-go-gke:1.0"
    Port:               80/TCP
  Type          Status  Reason
  ----          ------  ------
  Available     True    MinimumReplicasAvailable
  Progressing   True    NewReplicaSetAvailable
OldReplicaSets:                 &lt;none&gt;
NewReplicaSet:                  hello-3621623197 (3/3 replicas created)
Events:
...
Creating the
hello
Service object
The key to sending requests from a frontend to a backend is the backend
Service. A Service creates a persistent IP address and DNS name entry
so that the backend microservice can always be reached. A Service uses
selectors
to find
the Pods that it routes traffic to.
First, explore the Service configuration file:
service/access/backend-service.yaml
---
:
v1
kind
:
Service
metadata
:
name
:
hello
spec
:
selector
:
app
:
hello
tier
:
backend
ports
:
-
protocol
:
TCP
port
:
80
:
http
...
In the configuration file, you can see that the Service, named
hello
routes
traffic to Pods that have the labels
app: hello
and
tier: backend
.
Create the backend Service:
kubectl apply -f https://k8s.io/examples/service/access/backend-service.yaml
At this point, you have a
backend
Deployment running three replicas of your
hello
application, and you have a Service that can route traffic to them. However, this
service is neither available nor resolvable outside the cluster.
Creating the frontend
Now that you have your backend running, you can create a frontend that is accessible
outside the cluster, and connects to the backend by proxying requests to it.
The frontend sends requests to the backend worker Pods by using the DNS name
given to the backend Service. The DNS name is
hello
, which is the value
of the
name
examples/service/access/backend-service.yaml
The Pods in the frontend Deployment run a nginx image that is configured
to proxy requests to the
hello
backend Service. Here is the nginx configuration file:
service/access/frontend-nginx.conf
# The identifier Backend is internal to nginx, and used to name this specific upstream
upstream Backend {
    # hello is the internal DNS name used by the backend Service inside Kubernetes
    server hello;
}

server {
    listen 80;

    location / {
        # The following statement will proxy traffic to the upstream named Backend
        proxy_pass http://Backend;
    }
}
Similar to the backend, the frontend has a Deployment and a Service. An important
difference to notice between the backend and frontend services, is that the
configuration for the frontend Service has
, which means that
the Service uses a load balancer provisioned by your cloud provider and will be
accessible from outside the cluster.
service/access/frontend-service.yaml
---
:
v1
kind
:
Service
metadata
:
name
:
frontend
spec
:
selector
:
app
:
hello
tier
:
frontend
ports
:
-
protocol
:
"TCP"
port
:
80
:
80
type
:
...
service/access/frontend-deployment.yaml
---
:
apps/v1
kind
:
metadata
:
name
:
frontend
spec
:
selector
:
:
app
:
hello
tier
:
frontend
track
:
stable
replicas
:
1
template
:
metadata
:
labels
:
app
:
hello
tier
:
frontend
track
:
stable
spec
:
:
-
name
:
nginx
image
:
"gcr.io/google-samples/hello-frontend:1.0"
lifecycle
:
preStop
:
exec
:
command
:
[
"/usr/sbin/nginx"
,
"-s"
,
"quit"
]
...
Create the frontend Deployment and Service

:
kubectl apply -f https://k8s.io/examples/service/access/frontend-deployment.yaml
kubectl apply -f https://k8s.io/examples/service/access/frontend-service.yaml
The output verifies that both resources were created:
Note:
The nginx configuration is baked into the
. A better way to do this would
be to use a
ConfigMap
,
so that you can change the configuration more easily.
Interact with the frontend Service
Once you've created a Service of type LoadBalancer, you can use this
command to find the external IP:
kubectl get service frontend --watch
This displays the configuration for the
frontend
Service and watches for
changes. Initially, the external IP is listed as
&lt;pending&gt;
:
NAME       TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)  AGE
frontend   LoadBalancer   10.51.252.116   &lt;pending&gt;     80/TCP   10s
As soon as an external IP is provisioned, however, the configuration updates
to include the new IP under the
heading:
NAME       TYPE           CLUSTER-IP      EXTERNAL-IP        PORT(S)  AGE
frontend   LoadBalancer   10.51.252.116   XXX.XXX.XXX.XXX    80/TCP   1m
That IP can now be used to interact with the
frontend
service from outside the
cluster.
Send traffic through the frontend
The frontend and backend are now connected. You can hit the endpoint
by using the curl command on the external IP of your frontend Service.
curl http://
${
EXTERNAL_IP
}
# replace this with the EXTERNAL-IP you saw earlier
The output shows the message generated by the backend:
{
"message"
:
"Hello"
}
To delete the Services, enter this command:
kubectl delete services frontend backend
To delete the Deployments, the ReplicaSets and the Pods that are running the backend and frontend applications, enter this command:
kubectl delete deployment frontend backend
Services
DNS for Service and Pods
Feedback
Yes
No
.
or
.
Last modified November 12, 2021 at 9:32 PM PST:
Fix broken anchor link of #type-nodeport in service.md (#30388) (7a513e6dee)
hello
Home
Blog
Training
Partners
Community
CC BY 4.0

Restrict a Container's Access to Resources with AppArmor | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tutorials
Security
application to reduce its potential attack surface and provide greater in-depth defense. It is
enforcing
complain
Make sure:
will
=
jsonpath
=
file:
Y
If the Kubelet contains AppArmor s

upport (&gt;= v1.4), it will refuse to run a Pod with AppArmor
Note:
Docker
,
CRI-O
or
.
.
=
jsonpath
=
gke-test-default-pool-239f5d02-x1kf: kubelet is posting ready status. AppArmor enabled
Note:
).
:
Where
See the
kubectl
exec
Example
Fi

rst, we need to load the profile we want to use onto our nodes. This profile denies all file writes:
flags
=(
)
{
file,
deny /** w,
}
.
NODES
=(
)
for
NODE in
${
NODES
[*]
}
;
do
ssh
$NODE
file,
deny /** w,
}
EOF'
done
:
v1
kind
:
Pod
metadata
:
name
:
:
:
spec
:
:
-
name
:
hello
image
:
command
:
[
"sh"
,
"-c"
,
]
14s        14s         1         hello-apparmor   Pod                                Normal    Scheduled   {default-scheduler }                           Successfully assigned hello-apparmor to gke-test-default-pool-239f5d02-gyn2
kubectl
exec
kubectl
exec
:
v1
kind
:
Pod
metadata
:
name
:
:
:
spec
:
:
-
name
:
hello
image
:
command
:
[
"sh"
,
"-c"
,
]
EOF
Star

t Time:    Tue, 30 Aug 2016 17:58:56 -0700
IP:
  hello:
    Image ID:
    Port:
    Command:
      sh
      -c
    Mounts:
Volumes:
Events:
. An event was also recorded with the same message.
Through a
DaemonSet
here
.
image.
Example
.
error.
Note:
and
.
bane
dmesg
.
Pod Annotati

on
key
:
Where
value
.
Feedback
Yes
No
.
or
.
Last modified October 08, 2022 at 11:31 AM PST:
Update deprecated kubectl command in apparmor doc (0e31a40737)
Example
Home
Blog
Training
Partners
Community
CC BY 4.0

Node | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Node
Node
Node
: v1
kind
: Node
metadata
(
)
spec
(
NodeSpec
)
status
(
)
NodeSpec
(string)
(string)
(string)
podCIDR
(string)
podCIDRs
podCIDRs represents the IP ranges assigned to the node for usag

e by Pods on that node. If this field is specified, the 0th entry must match the podCIDR field. It may contain at most 1 value for each of IPv4 and IPv6.
(string)
taints
([]Taint)
(Time)
(string)
(boolean)
addresses
type
Quantity
)
capacity
Quantity
)
type
(Time)
(Time)
(string)
(string)
config
NodeConfigSource specifies a source of node configuration. Exactly one subfield (excluding metadata) must be non-nil. This API is de

precated since 1.22
(string)
(string)
(string)
(string)
(string)
Error describes any problems reconciling the Spec.ConfigSource to the Active config. Errors may occur, for example, attempting to checkpoint Spec.ConfigSource to the local Assigned record, attempting to checkpoint the payload associated with Spec.ConfigSource, attempting to load or validate the Assigned config, etc. Errors may occur at different points while syncing config. Earlier errors (e.g. download or checkpointing errors) will not result in a rollback to LastKnownGood, and may resolve across Kubelet retries. Later errors (e.g. loading or validating a checkpointed config) will result in a rollback to LastKnownGood. In the latter case, it is usually possible to resolve the error by fixing the config assigned in Spec.ConfigSource. You can find additional information for debugging by searching the error message in the Ku

belet log. Error is a human-readable description of the error state; machines can check whether or not Error is empty, but should not rely on the stability of the Error text across Kubelet versions.
LastKnownGood reports the checkpointed config the node will fall back to when it encounters an error attempting to use the Assigned config. The Assigned config becomes the LastKnownGood config when the node determines that the Assigned config is stable and correct. This is currently implemented as a 10-minute soak period starting when the local record of Assigned config is updated. If the Assigned config is Active at the end of this period, it becomes the LastKnownGood. Note that if Spec.ConfigSource is reset to nil (use local defaults), the LastKnownGood is also immediately reset to nil, because the local default config is always assumed good. You should not make assumptions about the node's method of determining config stability and correctness, as this may change or become configurable in the future.
(string)
(string)
images
(int64)
nodeInfo
MachineID reported by the node

. For unique machine identification in the cluster this field is preferred. Learn more from man(5) machine-id:
phase
(string)
NodeList
: v1
kind
: NodeList
metadata
(
ListMeta
)
items
([]
Node
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
NodeList
): OK
create
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
202 (
Node
update
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
update
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string

er
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Node
NodeSpec
NodeList
get
get
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Networ

k Plugins
Tasks
Monitoring, Logging,

 and Debugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configu

ring a Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Image Policy API (v

1alpha1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure RunAsUserName for Windows pods and containers | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
runAsUser
field (
(
field.
field set:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
:
:
:
-
name
:
image
:
command
:
[
"ping"
,
"-t"
,
]
:
:
windows
kubectl
exec
echo
field (
(
field.
field is set at the Po

d level and the Container level:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
:
:
:
-
name
:
image
:
command
:
[
"ping"
,
"-t"
,
]
:
:
:
:
:
windows
kubectl
exec
echo
, where
DOMAIN\
DOMAIN
and
USER
:
The
0x00-0x1F
,
0x7F
)
The
DOMAIN
.
.
(dot) or
-
(dash).
The
USER
only
.
field:
,
,
,
.
here
and
here
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Image Policy API (v1alpha1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

ation (v1)
Reference
Field
string
kind
string
metadata
metadata
field.
spec
status
Field
image
string
Field
.
namespace
string
Field
allowed
bool
reason
string
Feedback
Yes
No
.
Open an i

ssue in the GitHub repo if you want to
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Using kubectl to Create a Deployment | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Compute, Storage, an

d Networking Extensions
Tasks
Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission C

ontroller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials

Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Rate Limit Configuration 

Tutorials
cluster.
Summary:
Kubectl
Deploying your first app on Kubernetes
Kubectl
5
and
6
For your first Deployment, you'll use a hello-node application packaged in a Docker container that uses NGINX to echo back all the requests. (If you didn't al

ready try creating a hello-node application and deploying it using a container, you can do that first by following the instructions from the
).
›
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
T

Reference
read
.
including
and
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Using a KMS provider for data encryption | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
minikube
.
KMS v1
KMS v2
kms
in the
providers
KMS v1
name
endpoint
cachesize

timeout
KMS v2
name
endpoint
cachesize
timeout
See
.
KMS v1
api.pb.go
api.proto
KMS v2
api.pb.go
api.proto
Notes
KMS v1
v1beta1
v1beta1
as
.
v1beta1
unix
)
/@
, i.e.
KMS v2
v2alpha1
v2alpha1
as
, "ok" as
unix
)
/@
, i.e.
. Care must be taken when using this type of socket as they do not have concept of ACL (unlike traditional file based sockets).

 However, they are subject to Linux networking namespace, so will only be accessible to containers within the same pod unless host networking is used.
The KMS plugin can encode the ciphertext with additional metadata that may be required before sending it to the KMS for decryption.
kms
provider
Set the
KMS v1
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
kms
:
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
-
kms
:
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
KMS v2
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
kms
:
:
v2
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
-
kms
:
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
Setting
to
true
KMS v1 only
KMS v2 only
No KMS
None
.
,
etc.
providers
identity
.
,
etcdctl
command line program to retrieve the contents 

of your secret data.
secret1
in the
default
=
mykey
=
mydata
Using the
etcdctl
=
3
[
...
]
where
[...]
kms
kms
Add the
kms
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
kms
:
name
:
endpoint
:
cachesize
:
100
-
aescbc
:
keys
:
-
name
:
key1
secret
:
kms
provider.
Place the
identity
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
kms
:
name
:
endpoint
:
cachesize
:
100
Feedback
Yes
No
.
or
.
Last modified November 28, 2022 at 8:03 PM PST:
kms: updates website to document --encryption-provider-config-automatic-reload api server flag and healthcheck endpoint behavior (d132365aae)
KMS v1
KMS v2
KMS v1
KMS v2
KMS v1
KMS v2
Home
Blog
Training
Partners
Community
CC BY 4.0

Contributing to the Upstream Kubernetes Code | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

project.
kubeadm
,
, and
.
API or the
kube-*
Git
Golang
Docker
etcd
make
Your
GOPATH
etcd
PATH
and
.
kube-*
such as
,
.
mkdir
$GOPATH
/src
cd
$GOPATH
/src
.
.
want to change the API reference documentation, the first step is to change

 one
kube-*
Note:
cd
types.go
git status
types.go
...
Run
git add
and
git commit
Go to
Run
git status
...
.
Run
git add
and
git commit
types.go
and related files. Keep these two commits separate. That is, do not squash your commits.
to the
PR 57758
Note:
staging
directory
in the
staging
directory
README
files in
.
types.go
types.go
, but not the commit that has the results of running the s

cripts. For instructions, see
.
Note:
types.go
types.go
in the
The
.
Feedback
Yes
No
.
or
.
Last modified April 05, 2022 at 10:15 AM PST:
add necessary tools, correct git output example (c231ba5be3)
Home
Blog
Training
Partners
Community
CC BY 4.0

Run a Single-Instance Stateful Application | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

stall Tools
Debug Pods
De

bug Init Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod

 Security Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

onfiguration (v1alpha1)
Tasks
This page shows you how to run a single-instance stateful application
in Kubernetes using a PersistentVolume and a Deployment. The
application is MySQL.
Create a PersistentVolume referencing a disk in your environment.
Create a MySQL Deployment.
Expose MySQL to other pods in the cluster at a known DNS name.
minikube
.
,
or
used here.
You can run a stateful application by creating a Kubernetes Deployment
and connecting it to an existing PersistentVolume using a
PersistentVolumeClaim. For example, this YAML file describes a
Deployment that runs MySQL and references the PersistentVolumeClaim. The file
defines a volume mount for /var/lib/mysql, and then creates a
PersistentVolumeClaim that looks for a 20G volume. This claim is
satisfied by any existing volume that meets the requirements,
or by a dynamic provisioner.
Note: The password is defined in the config yaml, and this is insecure. See
Kubernetes Secrets
for a secure solution.
application/mysql/mysql-deployment.yaml
:
v1
kind
:
Service
metadata
:
name
:
mysql
spec
:
ports
:
-
port
:
3306
selector
:
app
:
mysql
clusterIP
:
None
---
:
apps/v1
kind
:
metadata
:
name
:
mysql
spec
:
selector
:
:
app
:
mysql
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
mysql
spec
:
:
-
image
:
mysql:5.6
name
:
mysql
env
:
# Use secret in real usage
-
name
:
value
:
password
ports
:
-
:
3306
name
:
mysql
:
-
name
:
mountPath
:
volumes
:
-
name
:
:
claimName
:
application/mysql/mysql-pv.yaml
:
v1
kind
:
metadata
:
name
:
mysql-pv-volume
labels
:
type
:
local
spec
:
storageClas

sName
:
manual
capacity
:
storage
:
20Gi
:
hostPath
:
path
:
---
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
20Gi
Deploy the PV and PVC of the YAML file:
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-pv.yaml
Deploy the contents of the YAML file:
kubectl apply -f https://k8s.io/examples/application/mysql/mysql-deployment.yaml
kubectl describe deployment mysql
Name:                 mysql
 Namespace:            default
 CreationTimestamp:    Tue, 01 Nov 2016 11:18:45 -0700
 Labels:               app=mysql
 Annotations:          deployment.kubernetes.io/revision=1
 Selector:             app=mysql
 Replicas:             1 desired | 1 updated | 1 total | 0 available | 1 unavailable
 StrategyType:         Recreate
 MinReadySeconds:      0
 Pod Template:
   Labels:       app=mysql
   Containers:
    mysql:
     Image:      mysql:5.6
     Port:       3306/TCP
     Environment:
       MYSQL_ROOT_PASSWORD:      password
     Mounts:
       /var/lib/mysql from mysql-persistent-storage (rw)
   Volumes:
    mysql-persistent-storage:
     Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
     ClaimName:  mysql-pv-claim
     ReadOnly:   false
 Conditions:
   Type          Status  Reason
   ----          ------  ------
   Available     False   MinimumReplicasUnavailable
   Progressing   True    ReplicaSetUpdated
 OldReplicaSets:       &lt;none&gt;
 NewReplicaSet:        mysql-63082529 (1/1 replicas created)
 Events:
   FirstSeen    LastSeen    Count    From                SubobjectPath    Type        Reason            Message
   ---------    --------    -----    ----                -------------    --------    ------            -------
   33s          33s         1        {deployment-controller }             Normal      ScalingReplicaSet Scaled up replica set mysql-63082529 to 1
List the pods created by the Deployment:
kubectl get pods -l app=mysql
NAME                   READY     STATUS    RESTARTS   AGE
 mysql-63082529-2z3ki   1/1       Running   0          3m
Inspect the PersistentVolumeClaim:
kubectl describe pvc mysql-pv-claim
Name:         mysql-pv-claim
 Namespace:    default
 StorageClass:
 Status:       Bound
 Volume:       mysql-pv-volume
 Labels:       &lt;none&gt;
 Annotations:    pv.kubernetes.io/bind-completed=yes
                 pv.kubernetes.io/bound-by-controller=yes
 Capacity:     20Gi
 Access Modes: RWO
 Events:       &lt;none&gt;
Accessing the MySQL instance
The preceding YAML file creates a service that
allows other Pods in the cluster to access the database. The Service option
clusterIP: None
lets the Service DNS name resolve directly to the
Pod's IP address. This is optimal when you have only one Pod
behind a Service and you don't intend to increase the number of Pods.
Run a MySQL client to connect to the server:
kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
This command creates a new Pod in the cluster running a MySQL client
and connects it to the server through the Service. If it connects, you
know your stateful MySQL database is up and running.
Waiting for pod default/mysql-client-274442439-zyp6i to be running, status is Pending, pod ready: false

mysql&gt;
Updating
The image or any other part of the Deployment can be updated as usual
with the
command. Here are some precautions that are
specific to stateful apps:
Don't scale the app. This setup is for single-instance apps
only. The underlying PersistentVolume can only be mounted to one
Pod. For clustered stateful apps, see the
StatefulSet documentation
.
Use
strategy:
type: Recreate
in

 the Deployment configuration
YAML file. This instructs Kubernetes to
not
use rolling
updates. Rolling updates will not work, as you cannot have more than
one Pod running at a time. The
Recreate
strategy will stop the
first pod before creating a new one with the updated configuration.
Deleting a deployment
Delete the deployed objects by name:
kubectl delete deployment,svc mysql
kubectl delete pvc mysql-pv-claim
kubectl delete pv mysql-pv-volume
If you manually provisioned a PersistentVolume, you also need to manually
delete it, as well as release the underlying resource.
PersistentVolume when it sees that you deleted the PersistentVolumeClaim.
underlying resource upon deleting the PersistentVolume.
.
kubectl run documentation
Volumes
and
Feedback
Yes
No
.
or
.
Last modified February 11, 2021 at 6:29 PM PST:
style: separate commands from output (e0e68de173)
Updating
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes API Reference Docs
Overview
API Groups
WORKLOADS APIS
Container v1 core
CronJob v1 batch
Write Operations
Create
Patch
Replace
Delete
Delete Collection
Read Operations
Read
List
List All Namespaces
Watch
Watch List
Watch List All Namespaces
Status Operations
Patch Status
Read Status
Replace Status
DaemonSet v1 apps
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Deployment v1 apps
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Misc Operations
Read Scale
Replace Scale
Patch Scale
Job v1 batch
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Pod v1 core
Create
Create Eviction
Patch
Replace
Delete
Read
List
Watch
Watch List
EphemeralContainers Operations
Patch EphemeralContainers
Read EphemeralContainers
Replace EphemeralContainers
Proxy Operations
Create Connect Portforward
Create Connect Proxy
Create Connect Proxy Path
Delete Connect Proxy
Delete Connect Proxy Path
Get Connect Portforward
Get Connect Proxy
Get Connect Proxy Path
Head Connect Proxy
Head Connect Proxy Path
Replace Connect Proxy
Replace Connect Proxy Path
Read Log
ReplicaSet v1 apps
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Scale
ReplicationController v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Scale
StatefulSet v1 apps
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Scale
SERVICE APIS
Endpoints v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
EndpointSlice v1 discovery.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Ingress v1 networking.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
IngressClass v1 networking.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Service v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CONFIG AND STORAGE APIS
ConfigMap v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CSIDriver v1 storage.

k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CSINode v1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Secret v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PersistentVolumeClaim v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
StorageClass v1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CSIStorageCapacity v1beta1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Volume v1 core
VolumeAttachment v1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
METADATA APIS
ControllerRevision v1 apps
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CustomResourceDefinition v1 apiextensions.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Event v1 events.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
LimitRange v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
HorizontalPodAutoscaler v1 autoscaling
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
MutatingWebhookConfiguration v1 admissionregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ValidatingWebhookConfiguration v1 admissionregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PodTemplate v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PodDisruptionBudget v1 policy
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PriorityClass v1 scheduling.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PodSecurityPolicy v1beta1 policy
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CLUSTER APIS
APIService v1 apiregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Binding v1 core
Create
CertificateSigningRequest v1 certificates.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ClusterRole v1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ClusterRoleBinding v1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Compon

entStatus v1 core
Read
List
FlowSchema v1beta1 flowcontrol.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Lease v1 coordination.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
LocalSubjectAccessReview v1 authorization.k8s.io
Create
Namespace v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Node v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PersistentVolume v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PriorityLevelConfiguration v1beta1 flowcontrol.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ResourceQuota v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Role v1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
RoleBinding v1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
RuntimeClass v1 node.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
SelfSubjectAccessReview v1 authorization.k8s.io
Create
SelfSubjectRulesReview v1 authorization.k8s.io
Create
ServiceAccount v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
StorageVersion v1alpha1 internal.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
SubjectAccessReview v1 authorization.k8s.io
Create
TokenRequest v1 authentication.k8s.io
TokenReview v1 authentication.k8s.io
Create
NetworkPolicy v1 networking.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
DEFINITIONS
APIGroup v1 meta
APIResource v1 meta
APIServiceCondition v1 apiregistration.k8s.io
APIVersions v1 meta
AWSElasticBlockStoreVolumeSource v1 core
Affinity v1 core
AggregationRule v1 rbac.authorization.k8s.io
AllowedCSIDriver v1beta1 policy
AllowedFlexVolume v1beta1 policy
AllowedHostPath v1beta1 policy
AttachedVolume v1 core
AzureDiskVolumeSource v1 core
AzureFilePersistentVolumeSource v1 core
AzureFileVolumeSource v1 core
BoundObjectReference v1 authentication.k8s.io
CSINodeDriver v1 storage.k8s.io
CSIPersistentVolumeSource v1 core
CSIVolumeSource v1 core
Capabilities v1 core
CephFSPersistentVolumeSource v1 core
CephFSVolumeSource v1 core
CertificateSigningRequestCondition v1 certificates.k8s.io
CinderPersistentVolumeSource v1 core
CinderVolumeSource v1 core
ClientIPConfig v1 core
ComponentCondition v1 core
Condition v1 meta
ConfigMap

EnvSource v1 core
ConfigMapKeySelector v1 core
ConfigMapNodeConfigSource v1 core
ConfigMapProjection v1 core
ConfigMapVolumeSource v1 core
ContainerImage v1 core
ContainerPort v1 core
ContainerResourceMetricSource v2beta2 autoscaling
ContainerResourceMetricStatus v2beta2 autoscaling
ContainerState v1 core
ContainerStateRunning v1 core
ContainerStateTerminated v1 core
ContainerStateWaiting v1 core
CrossVersionObjectReference v1 autoscaling
CustomResourceColumnDefinition v1 apiextensions.k8s.io
CustomResourceConversion v1 apiextensions.k8s.io
CustomResourceDefinitionCondition v1 apiextensions.k8s.io
CustomResourceDefinitionNames v1 apiextensions.k8s.io
CustomResourceDefinitionVersion v1 apiextensions.k8s.io
CustomResourceSubresourceScale v1 apiextensions.k8s.io
CustomResourceSubresourceStatus v1 apiextensions.k8s.io
CustomResourceSubresources v1 apiextensions.k8s.io
CustomResourceValidation v1 apiextensions.k8s.io
DaemonEndpoint v1 core
DaemonSetCondition v1 apps
DaemonSetUpdateStrategy v1 apps
DeleteOptions v1 meta
DeploymentCondition v1 apps
DownwardAPIProjection v1 core
DownwardAPIVolumeFile v1 core
DownwardAPIVolumeSource v1 core
EmptyDirVolumeSource v1 core
Endpoint v1 discovery.k8s.io
EndpointAddress v1 core
EndpointConditions v1 discovery.k8s.io
EndpointHints v1 discovery.k8s.io
EndpointPort v1 core
EndpointSubset v1 core
EnvFromSource v1 core
EnvVar v1 core
EnvVarSource v1 core
EphemeralContainer v1 core
EphemeralContainers v1 core
EphemeralVolumeSource v1 core
EventSeries v1 events.k8s.io
EventSource v1 core
Eviction v1beta1 policy
ExecAction v1 core
ExternalDocumentation v1 apiextensions.k8s.io
ExternalMetricSource v2beta2 autoscaling
ExternalMetricStatus v2beta2 autoscaling
FCVolumeSource v1 core
FSGroupStrategyOptions v1beta1 policy
FieldsV1 v1 meta
FlexPersistentVolumeSource v1 core
FlexVolumeSource v1 core
FlockerVolumeSource v1 core
FlowDistinguisherMethod v1beta1 flowcontrol.apiserver.k8s.io
FlowSchemaCondition v1beta1 flowcontrol.apiserver.k8s.io
ForZone v1 discovery.k8s.io
GCEPersistentDiskVolumeSource v1 core
GitRepoVolumeSource v1 core
GlusterfsPersistentVolumeSource v1 core
GlusterfsVolumeSource v1 core
GroupSubject v1beta1 flowcontrol.apiserver.k8s.io
GroupVersionForDiscovery v1 meta
HPAScalingPolicy v2beta2 autoscaling
HPAScalingRules v2beta2 autoscaling
HTTPGetAction v1 core
HTTPHeader v1 core
HTTPIngressPath v1 networking.k8s.io
HTTPIngressRuleValue v1 networking.k8s.io
Handler v1 core
HorizontalPodAutoscalerBehavior v2beta2 autoscaling
HorizontalPodAutoscalerCondition v2beta2 autoscaling
HostAlias v1 core
HostPathVolumeSource v1 core
HostPortRange v1beta1 policy
IDRange v1beta1 policy
IPBlock v1 networking.k8s.io
ISCSIPersistentVolumeSource v1 core
ISCSIVolumeSource v1 core
IngressBackend v1 networking.k8s.io
IngressClassParametersReference v1 networking.k8s.io
IngressRule v1 networking.k8s.io
IngressServiceBackend v1 networking.k8s.io
IngressTLS v1 networking.k8s.io
JSON v1 apiextensions.k8s.io
JSONSchemaProps v1 apiextensions.k8s.io
JSONSchemaPropsOrArray v1 apiextensions.k8s.io
JSONSchemaPropsOrBool v1 apiextensions.k8s.io
JobCondition v1 batch
JobTemplateSpec v1 batch
KeyToPath v1 core
LabelSelector v1 meta
LabelSelectorRequirement v1 meta
Lifecycle v1 core
LimitRangeItem v1 core
LimitResponse v1beta1 flowcontrol.apiserver.k8s.io
LimitedPriorityLevelConfiguration v1beta1 flowcontrol.apiserver.k8s.io
ListMeta v1 meta
LoadBalancerIngress v1 core
LoadBalancerStatus v1 core
LocalObjectReference v1 core
LocalVolumeSource v1 core
ManagedFieldsEntry v1 meta
MetricIdentifier v2beta2 autoscaling
MetricSpec v2beta2 autoscaling
MetricStatus v2beta2 autoscaling
MetricTarget v2beta2 autoscaling
MetricValueStatus v2beta2 autoscaling
MicroTime v1 meta
MutatingWebhook v1 admissionregistration.k8s.io
NFSVolumeSource v1 core
NamespaceCondition v1 core
NetworkPolicyEgressRule v1 networking.k8s.io
NetworkPolicyIngressRule v1 networking.k8s.io
NetworkPolicyPeer v1 networking.k8s.io
NetworkPolicyPort v1 networking.k8s.io

NodeAddress v1 core
NodeAffinity v1 core
NodeCondition v1 core
NodeConfigSource v1 core
NodeConfigStatus v1 core
NodeDaemonEndpoints v1 core
NodeSelector v1 core
NodeSelectorRequirement v1 core
NodeSelectorTerm v1 core
NodeSystemInfo v1 core
NonResourceAttributes v1 authorization.k8s.io
NonResourcePolicyRule v1beta1 flowcontrol.apiserver.k8s.io
NonResourceRule v1 authorization.k8s.io
ObjectFieldSelector v1 core
ObjectMeta v1 meta
ObjectMetricSource v2beta2 autoscaling
ObjectMetricStatus v2beta2 autoscaling
ObjectReference v1 core
Overhead v1 node.k8s.io
OwnerReference v1 meta
Patch v1 meta
PersistentVolumeClaimCondition v1 core
PersistentVolumeClaimTemplate v1 core
PersistentVolumeClaimVolumeSource v1 core
PhotonPersistentDiskVolumeSource v1 core
PodAffinity v1 core
PodAffinityTerm v1 core
PodAntiAffinity v1 core
PodCondition v1 core
PodDNSConfig v1 core
PodDNSConfigOption v1 core
PodIP v1 core
PodReadinessGate v1 core
PodSecurityContext v1 core
PodsMetricSource v2beta2 autoscaling
PodsMetricStatus v2beta2 autoscaling
PolicyRule v1 rbac.authorization.k8s.io
PolicyRulesWithSubjects v1beta1 flowcontrol.apiserver.k8s.io
PortStatus v1 core
PortworxVolumeSource v1 core
Preconditions v1 meta
PreferredSchedulingTerm v1 core
PriorityLevelConfigurationCondition v1beta1 flowcontrol.apiserver.k8s.io
PriorityLevelConfigurationReference v1beta1 flowcontrol.apiserver.k8s.io
Probe v1 core
ProjectedVolumeSource v1 core
Quantity resource core
QueuingConfiguration v1beta1 flowcontrol.apiserver.k8s.io
QuobyteVolumeSource v1 core
RBDPersistentVolumeSource v1 core
RBDVolumeSource v1 core
ReplicaSetCondition v1 apps
ReplicationControllerCondition v1 core
ResourceAttributes v1 authorization.k8s.io
ResourceFieldSelector v1 core
ResourceMetricSource v2beta2 autoscaling
ResourceMetricStatus v2beta2 autoscaling
ResourcePolicyRule v1beta1 flowcontrol.apiserver.k8s.io
ResourceRequirements v1 core
ResourceRule v1 authorization.k8s.io
RoleRef v1 rbac.authorization.k8s.io
RollingUpdateStatefulSetStrategy v1 apps
RuleWithOperations v1 admissionregistration.k8s.io
RunAsGroupStrategyOptions v1beta1 policy
RunAsUserStrategyOptions v1beta1 policy
RuntimeClassStrategyOptions v1beta1 policy
SELinuxOptions v1 core
SELinuxStrategyOptions v1beta1 policy
Scale v1 autoscaling
ScaleIOPersistentVolumeSource v1 core
ScaleIOVolumeSource v1 core
Scheduling v1 node.k8s.io
ScopeSelector v1 core
ScopedResourceSelectorRequirement v1 core
SeccompProfile v1 core
SecretEnvSource v1 core
SecretKeySelector v1 core
SecretProjection v1 core
SecretReference v1 core
SecretVolumeSource v1 core
SecurityContext v1 core
ServerAddressByClientCIDR v1 meta
ServerStorageVersion v1alpha1 internal.apiserver.k8s.io
ServiceAccountSubject v1beta1 flowcontrol.apiserver.k8s.io
ServiceAccountTokenProjection v1 core
ServiceBackendPort v1 networking.k8s.io
ServicePort v1 core
ServiceReference v1 admissionregistration.k8s.io
SessionAffinityConfig v1 core
StatefulSetCondition v1 apps
StatefulSetUpdateStrategy v1 apps
Status v1 meta
StatusCause v1 meta
StatusDetails v1 meta
StorageOSPersistentVolumeSource v1 core
StorageOSVolumeSource v1 core
StorageVersionCondition v1alpha1 internal.apiserver.k8s.io
Subject v1beta1 flowcontrol.apiserver.k8s.io
SubjectRulesReviewStatus v1 authorization.k8s.io
SupplementalGroupsStrategyOptions v1beta1 policy
Sysctl v1 core
TCPSocketAction v1 core
Taint v1 core
Time v1 meta
Toleration v1 core
TopologySelectorLabelRequirement v1 core
TopologySelectorTerm v1 core
TopologySpreadConstraint v1 core
TypedLocalObjectReference v1 core
UserInfo v1 authentication.k8s.io
UserSubject v1beta1 flowcontrol.apiserver.k8s.io
ValidatingWebhook v1 admissionregistration.k8s.io
VolumeAttachmentSource v1 storage.k8s.io
VolumeDevice v1 core
VolumeError v1 storage.k8s.io
VolumeMount v1 core
VolumeNodeAffinity v1 core
VolumeNodeResources v1 storage.k8s.io
VolumeProjection v1 core
VsphereVirtualDiskVolumeSource v1 core
WatchEvent v1 meta
WebhookClientConfig v1 admissionregistration.k8s.io
WebhookConve

rsion v1 apiextensions.k8s.io
WeightedPodAffinityTerm v1 core
WindowsSecurityContextOptions v1 core
OLD API VERSIONS
APIService v1beta1 apiregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
APIServiceCondition v1beta1 apiregistration.k8s.io
AggregationRule v1beta1 rbac.authorization.k8s.io
AggregationRule v1alpha1 rbac.authorization.k8s.io
CSIDriver v1beta1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CSINode v1beta1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CSINodeDriver v1beta1 storage.k8s.io
CSIStorageCapacity v1alpha1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CertificateSigningRequest v1beta1 certificates.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CertificateSigningRequestCondition v1beta1 certificates.k8s.io
ClusterRole v1beta1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ClusterRole v1alpha1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ClusterRoleBinding v1beta1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ClusterRoleBinding v1alpha1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ContainerResourceMetricSource v2beta1 autoscaling
ContainerResourceMetricStatus v2beta1 autoscaling
CronJob v1beta1 batch
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CrossVersionObjectReference v2beta2 autoscaling
CrossVersionObjectReference v2beta1 autoscaling
CustomResourceColumnDefinition v1beta1 apiextensions.k8s.io
CustomResourceConversion v1beta1 apiextensions.k8s.io
CustomResourceDefinition v1beta1 apiextensions.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CustomResourceDefinitionCondition v1beta1 apiextensions.k8s.io
CustomResourceDefinitionNames v1beta1 apiextensions.k8s.io
CustomResourceDefinitionVersion v1beta1 apiextensions.k8s.io
CustomResourceSubresourceScale v1beta1 apiextensions.k8s.io
CustomResourceSubresourceStatus v1beta1 apiextensions.k8s.io
CustomResourceSubresources v1beta1 apiextensions.k8s.io
CustomResourceValidation v1beta1 apiextensions.k8s.io
Endpoint v1beta1 discovery.k8s.io
EndpointConditions v1beta1 discovery.k8s.io
EndpointHints v1beta1 discovery.k8s.io
EndpointPort v1 discovery.k8s.io
EndpointPort v1beta1 discovery.k8s.io
EndpointSlice v1beta1 discovery.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Event v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Event v1beta1 events.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
EventSeries v1 core
EventSeries v1beta1 events.k8s.io
ExternalDocumentation v1beta1 apiextensions.k8s.io
ExternalMetricSource v2beta1 autoscaling
ExternalMetricStatus v2beta1 autoscaling
ForZone v1beta1 discovery.

k8s.io
HTTPIngressPath v1beta1 extensions
HTTPIngressPath v1beta1 networking.k8s.io
HTTPIngressRuleValue v1beta1 extensions
HTTPIngressRuleValue v1beta1 networking.k8s.io
HorizontalPodAutoscaler v2beta2 autoscaling
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
HorizontalPodAutoscaler v2beta1 autoscaling
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
HorizontalPodAutoscalerCondition v2beta1 autoscaling
Ingress v1beta1 extensions
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Ingress v1beta1 networking.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
IngressBackend v1beta1 extensions
IngressBackend v1beta1 networking.k8s.io
IngressClass v1beta1 networking.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
IngressClassParametersReference v1beta1 networking.k8s.io
IngressRule v1beta1 extensions
IngressRule v1beta1 networking.k8s.io
IngressTLS v1beta1 extensions
IngressTLS v1beta1 networking.k8s.io
JSON v1beta1 apiextensions.k8s.io
JSONSchemaProps v1beta1 apiextensions.k8s.io
JSONSchemaPropsOrArray v1beta1 apiextensions.k8s.io
JSONSchemaPropsOrBool v1beta1 apiextensions.k8s.io
JobTemplateSpec v1beta1 batch
Lease v1beta1 coordination.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
LocalSubjectAccessReview v1beta1 authorization.k8s.io
Create
MetricSpec v2beta1 autoscaling
MetricStatus v2beta1 autoscaling
MutatingWebhook v1beta1 admissionregistration.k8s.io
MutatingWebhookConfiguration v1beta1 admissionregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
NonResourceAttributes v1beta1 authorization.k8s.io
NonResourceRule v1beta1 authorization.k8s.io
ObjectMetricSource v2beta1 autoscaling
ObjectMetricStatus v2beta1 autoscaling
Overhead v1beta1 node.k8s.io
Overhead v1alpha1 node.k8s.io
PodDisruptionBudget v1beta1 policy
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PodsMetricSource v2beta1 autoscaling
PodsMetricStatus v2beta1 autoscaling
PolicyRule v1beta1 rbac.authorization.k8s.io
PolicyRule v1alpha1 rbac.authorization.k8s.io
PriorityClass v1beta1 scheduling.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PriorityClass v1alpha1 scheduling.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ResourceAttributes v1beta1 authorization.k8s.io
ResourceMetricSource v2beta1 autoscaling
ResourceMetricStatus v2beta1 autoscaling
ResourceRule v1beta1 authorization.k8s.io
Role v1beta1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Role v1alpha1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
RoleBinding v1beta1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read Ope

rations
Read
List
Watch
Watch List
RoleBinding v1alpha1 rbac.authorization.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
RoleRef v1beta1 rbac.authorization.k8s.io
RoleRef v1alpha1 rbac.authorization.k8s.io
RuleWithOperations v1beta1 admissionregistration.k8s.io
RuntimeClass v1beta1 node.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
RuntimeClass v1alpha1 node.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Scheduling v1beta1 node.k8s.io
Scheduling v1alpha1 node.k8s.io
SelfSubjectAccessReview v1beta1 authorization.k8s.io
Create
SelfSubjectRulesReview v1beta1 authorization.k8s.io
Create
ServiceReference v1 apiextensions.k8s.io
ServiceReference v1 apiregistration.k8s.io
ServiceReference v1beta1 admissionregistration.k8s.io
ServiceReference v1beta1 apiextensions.k8s.io
ServiceReference v1beta1 apiregistration.k8s.io
StorageClass v1beta1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Subject v1 rbac.authorization.k8s.io
Subject v1beta1 rbac.authorization.k8s.io
Subject v1alpha1 rbac.authorization.k8s.io
SubjectAccessReview v1beta1 authorization.k8s.io
Create
SubjectRulesReviewStatus v1beta1 authorization.k8s.io
TokenRequest v1 storage.k8s.io
TokenRequest v1beta1 storage.k8s.io
TokenReview v1beta1 authentication.k8s.io
Create
UserInfo v1beta1 authentication.k8s.io
ValidatingWebhook v1beta1 admissionregistration.k8s.io
ValidatingWebhookConfiguration v1beta1 admissionregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
VolumeAttachment v1beta1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
VolumeAttachment v1alpha1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
VolumeAttachmentSource v1beta1 storage.k8s.io
VolumeAttachmentSource v1alpha1 storage.k8s.io
VolumeError v1beta1 storage.k8s.io
VolumeError v1alpha1 storage.k8s.io
VolumeNodeResources v1beta1 storage.k8s.io
WebhookClientConfig v1 apiextensions.k8s.io
WebhookClientConfig v1beta1 admissionregistration.k8s.io
WebhookClientConfig v1beta1 apiextensions.k8s.io
Copyright 2016-2021 The Kubernetes Authors.
Generated at: 2021-08-11 10:59:31 (CEST)
API Version:
v1.21.0
API OVERVIEW
Welcome to the Kubernetes API.  You can use the Kubernetes API to read
and write Kubernetes resource objects via a Kubernetes API endpoint.
Resource Categories
This is a high-level overview of the basic types of resources provide by the Kubernetes API and their primary functions.
Workloads
are objects you use to manage and run your containers on the cluster.
Discovery &amp; LB
resources are objects you use to "stitch" your workloads together into an externally accessible, load-balanced Service.
Config &amp; Storage
resources are objects you use to inject initialization data into your applications, and to persist data that is external to your container.
Cluster
resources objects define how the cluster itself is configured; these are typically used only by cluster operators.
Metadata
resources are objects you use to configure the behavior of other resources within the cluster, such as
for scaling workloads.
Resource Objects
Resource objects typically have 3 components:
Resource ObjectMeta
: This is metadata about the resource, such as its name, type, api version, annotations, and labels.  This contains
fields that maybe updated both by the end user and the system (e.g. annotations).
: This is defined by the user and describes the desire

d state of system.  Fill this in when creating or updating an object.
ResourceStatus
: This is filled in by the server and reports the current state of the system.  In most cases, users don't need to change this.
Resource Operations
Most resources provide the following Operations:
Create
Create operations will create the resource in the storage backend.  After a resource is create the system will apply
the desired state.
Update
Updates come in 2 forms:
Replace
and
Patch
:
Replace
:
Replacing a resource object will update the resource by replacing the existing spec with the provided one.  For
read-then-write operations this is safe because an optimistic lock failure will occur if the resource was modified
between the read and write.
Note
: The
will be ignored by the system and will not be updated.
To update the status, one must invoke the specific status update operation.
Note: Replacing a resource object may not result immediately in changes being propagated to downstream objects.  For instance
replacing a
ConfigMap
or
Secret
resource will not result in all
Pod
s seeing the changes unless the
Pod
s are
restarted out of band.
Patch
:
Patch will apply a change to a specific field.  How the change is merged is defined per field.  Lists may either be
replaced or merged.  Merging lists will not preserve ordering.
Patches will never cause optimistic locking failures, and the last write will win.
Patches are recommended
when the full state is not read before an update, or when failing on optimistic locking is undesirable.
When patching
complex types, arrays and maps, how the patch is applied is defined on a per-field basis and may either replace
the field's current value, or merge the contents into the current value.
Read
Reads come in 3 forms:
Get
,
List
and
Watch
:
Get
: Get will retrieve a specific resource object by name.
List
: List will retrieve all resource objects of a specific type within a namespace, and the results can be restricted to resources matching a selector query.
: Like
List
but retrieves resources across all namespaces.
Watch
: Watch will stream results for an object(s) as it is updated.  Similar to a callback, watch is used to respond to resource changes.
Delete
Delete will delete a resource.  Depending on the specific resource, child objects may or may not be garbage collected by the server.  See
notes on specific resource objects for details.
Additional Operations
Resources may define additional operations specific to that resource type.
Rollback
: Rollback a PodTemplate to a previous version.  Only available for some resource types.
Read / Write Scale
: Read or Update the number of replicas for the given resource.  Only available for some resource types.
Read / Write Status
: Read or Update the Status for a resource object.  The Status can only changed through these update operations.
API Groups
The API Groups and their versions are summarized in the following table.
Group
Version
admissionregistration.k8s.io
v1, v1beta1
apiregistration.k8s.io
apps
v1
v1, v2beta2, v2beta1
batch
coordination.k8s.io
core
v1
discovery.k8s.io
policy
v1beta1
flowcontrol.apiserver.k8s.io
v1beta1
internal.apiserver.k8s.io
v1alpha1
node.k8s.io
v1, v1beta1, v1alpha1
scheduling.k8s.io
storage.k8s.io
WORKLOADS
Workloads resources are responsible for managing and running your containers on the cluster.
are created
by Controllers through
Pods
. Pods run Containers and provide environmental dependencies such as shared or
persistent storage
Volumes
and
or
Secret
data injected into the container.
The most common Controllers ar

e:
for stateless persistent apps (e.g. HTTP servers).
for stateful persistent apps (e.g. databases).
Jobs
for run-to-completion apps (e.g. batch Jobs).
example
Container Config to run nginx (must be embedded in a PodSpec to run).
# Run the nginx:1.14 image
image: nginx:1.14
Group
Version
Kind
core
v1
Container
Warning:
Containers are only ever created within the context of a
Pod
.  This is usually done using a Controller.  See Controllers:
,
Job
, or
Appears In:
PodSpec [core/v1]
Field
args
string array
Arguments to the entrypoint. The docker image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell
command
Entrypoint array. Not executed within a shell. The docker image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell
env
EnvVar
array
patch strategy
:
merge
patch merge key
:
name
envFrom
EnvFromSource
array
image
string
Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets.
string
Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. Cannot be updated. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images
lifecycle
Lifecycle
Probe
Periodic probe of container liveness. Container will be restarted if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
name
string
ports
ContainerPort
array
:
merge
:
List of ports to expose from the container. Exposing a port here gives the system additional information about the network connections a container uses, but is primarily informational. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Cannot be updated.
Probe
Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. More info: https://kuberne

tes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
resources
ResourceRequirements
Compute Resources required by this container. Cannot be updated. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
Security options the pod should run with. More info: https://kubernetes.io/docs/concepts/policy/security-context/ More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
Probe
StartupProbe indicates that the Pod has successfully initialized. If specified, no other probes are executed until this completes successfully. If this probe fails, the Pod will be restarted, just as if the livenessProbe failed. This can be used to provide different probe parameters at the beginning of a Pod's lifecycle, when it might take a long time to load data or warm a cache, than during steady-state operation. This cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
stdin
boolean
stdinOnce
boolean
string
string
tty
boolean
VolumeDevice
array
:
merge
:
VolumeMount
array
:
merge
:
mountPath
string
ContainerStatus v1 core
PodStatus [core/v1]
Field
string
Container's ID in the format 'docker://&lt;container_id&gt;'.
image
string
The image the container is running. More info: https://kubernetes.io/docs/concepts/containers/images
imageID
string
lastState
ContainerState
name
string
ready
boolean
integer
The number of times the container

 has been restarted, currently based on the number of dead containers that have not yet been removed. Note that this is calculated from dead containers. But those containers are subject to garbage collection. This value will get capped at 5 by GC.
started
boolean
state
Group
Version
Kind
batch
v1
CronJob
Other API versions of this object exist:
v1beta1
CronJobList [batch/v1]
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
Specification of the desired behavior of a cron job, including the schedule. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Current status of a cron job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
CronJobSpec v1 batch
CronJob [batch/v1]
Field
string
integer
JobTemplateSpec
schedule
string
The schedule in Cron format, see https://en.wikipedia.org/wiki/Cron.
integer
Optional deadline in seconds for starting the job if it misses scheduled time for any reason.  Missed jobs executions will be counted as failed ones.
integer
suspend
boolean
This flag tells the controller to suspend subsequent executions, it does not apply to already started executions.  Defaults to false.
CronJobStatus v1 batch
Field
active
array
Time
Time
CronJobList v1 batch
Field
string
items
CronJob
array
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: h

ttps://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
Create
Path Parameters
Parameter
namespace
Query Parameters
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Body Parameters
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
202
CronJob
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
CronJob
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
The dur

ation in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue fie

ld to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
CronJob
OK
List
Parameter
namespace
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified c

ontinue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when re

trieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind CronJob. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/batch/v1/watch/namespaces/{namespace}/cronjobs/{name}
Parameter
name
na

mespace
Parameter
continue
limit
pretty
tim

eoutSeconds
watch
Response
Code
200
WatchEvent
OK
Watch List
watch individual changes to a list of CronJob. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/batch/v1/watch/namespaces/{namespace}/cronjobs
Parameter
namespace
Parameter
continue
limit
pretty
If 'true', then the output is pret

ty printed.
watch
Response
Code
200
OK
GET /apis/batch/v1/watch/cronjobs
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued wi

ll be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
CronJob
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
CronJob
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
example

DaemonSet Config to print the `hostname` on each Node in the cluster every 10 seconds.
kind: DaemonSet
metadata:
  # Unique key of the DaemonSet instance
  name: daemonset-example
spec:
  selector:
    matchLabels:
      app: daemonset-example
  template:
    metadata:
      labels:
        app: daemonset-example
    spec:
      # This container is run once on each Node in the cluster
      - name: daemonset-example
        image: ubuntu:trusty
        command:
        - /bin/sh
        args:
        - -c
        # This script is run through `sh -c &lt;script&gt;`
        - &gt;-
          while [ true ]; do
          echo "DaemonSet running on $(hostname)" ;
          sleep 10 ;
          done
Group
Version
Kind
apps
v1
DaemonSet
DaemonSetList [apps/v1]
Field
string
kind
string
metadata
spec
The desired behavior of this daemon set. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
The current status of this daemon set. This data may be out of date by some window of time. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
DaemonSetSpec v1 apps
DaemonSet [apps/v1]
Field
integer
integer
selector
A label query over pods that are managed by the daemon set. Must match in order to be controlled. It must match the pod template's labels. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
template
An object that describes the pod that will be created. The DaemonSet will create exactly one copy of this pod on every node that matches the template's node selector (or on every node if no node selector is specified). More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#pod-template
DaemonSetUpdateStrategy
DaemonSetStatus v1 apps
Field
integer
DaemonSetCondition
array
:
merge
:
type
integer
The number of nodes that are running at least 1 daemon pod and are supposed to run the daemon pod. More info: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
integer
The total number of nodes that should be running the daemon pod

 (including nodes correctly running the daemon pod). More info: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
integer
integer
The number of nodes that are running the daemon pod, but are not supposed to run the daemon pod. More info: https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
integer
The number of nodes that should be running the daemon pod and have one or more of the daemon pod running and ready.
integer
integer
integer
DaemonSetList v1 apps
Field
string
items
DaemonSet
array
kind
string
metadata
ListMeta
RollingUpdateDaemonSet v1 apps
DaemonSetUpdateStrategy [apps/v1]
Field
maxSurge
The maximum number of nodes with an existing available DaemonSet pod that can have an updated DaemonSet pod during during an update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). This can not be 0 if MaxUnavailable is 0. Absolute number is calculated from percentage by rounding up to a minimum of 1. Default value is 0. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their a new pod created before the old pod is marked as deleted. The update starts by launching new pods on 30% of nodes. Once an updated pod is available (Ready for at least minReadySeconds) the old DaemonSet pod on that node is marked deleted. If the old pod becomes unavailable for any reason (Ready transitions to false, is evicted, or is drained) an updated pod is immediatedly created on that node without considering surge limits. Allowing surge implies the possibility that the resources consumed by the daemonset on any given node can double if the readiness check fails, and so resource intensive daemonsets should take into account that they may cause evictions during disruption. This is an alpha field and requires enabling DaemonSetUpdateSurge feature gate.
The maximum number of DaemonSet pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of total number of DaemonSet pods at the start of the update (ex: 10%). Absolute number is calculated from percentage by rounding down to a minimum of one. This cannot be 0 if MaxSurge is 0 Default value is 1. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their pods stopped for an update at any given time. The update starts by stopping at most 30% of those DaemonSet pods and then brings up new DaemonSet pods in their place. Once the new pods are available, it then proceeds

 onto other DaemonSet pods, thus ensuring that at least 70% of original number of DaemonSet pods are available at all times during the update.
Create
kubectl request example
curl request example
kubectl
command
$ echo 'apiVersion: apps/v1
metadata:
spec:
  selector:
  template:
    metadata:
      labels:
    spec:
        command:
        - /bin/sh
        args:
        - -c
        - &gt;-
          sleep 10 ;
          done
' | kubectl create -f -
curl
command (
requires
to be running
)
$ kubectl proxy
$ curl -X POST -H 'Content-Type: application/yaml' --data '
metadata:
spec:
  selector:
  template:
    metadata:
      labels:
    spec:
        command:
        - /bin/sh
        args:
        - -c
        - &gt;-
          sleep 10 ;
          done
' http://127.0.0.1:8001/apis/apps/v1/namespaces/default/daemonsets
kubectl response example
curl response example
Output
daemonset "daemonset-example" created
Response Body
{
  "kind": "DaemonSet",
  "apiVersion": "apps/v1",
    "name": "daemonset-example",
    "selfLink": "/apis/apps/v1/namespaces/default/daemonsets/daemonset-example",
    "uid": "65552ced-b0e2-11e6-aef0-42010af00229",
    "resourceVersion": "3558",
    "generation": 1,
    "creationTimestamp": "2016-11-22T18:35:09Z",
    "labels": {
      "app": "daemonset-example"
    }
  },
  "spec": {
    "selector": {
      "matchLabels": {
        "app": "daemonset-example"
      }
    },
    "template": {
        "creationTimestamp": null,
          "app": "daemonset-example"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "daemonset-example",
            "image": "ubuntu:trusty",
            "command": [
              "/bin/sh"
            ],
            "args": [
              "-c",
              "while [ true ]; do echo \"DaemonSet running on $(hostname)\" ; sleep 10 ; done"
            ],
            "resources": {},
            "terminationMessagePath": "/dev/termination-log",
            "imagePullPolicy": "IfNotPresent"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {}
      }
    }
  },
  "status": {
    "currentNumberScheduled": 0,
    "numberMisscheduled": 0,
    "desiredNumberScheduled": 0
  }
}
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
DaemonSet
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
202
DaemonSet
Accepted
Patch
PATCH

 /apis/apps/v1/namespaces/{namespace}/daemonsets/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
DaemonSet
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
DaemonSet
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
Delete
kubectl
command
$ kubectl delete daemonset daemonset-example
curl
command (
requires
)
$ curl -X DELETE -H 'Content-Type: application/yaml' --data '
gracePeriodSeconds: 0
orphanDependents: false
' 'http://127.0.0.1:8001/apis/apps/v1/namespaces/default/daemonsets/daemonset-example'
Output
daemonset "daemonset-example" deleted
{
  "kind": "Status",
  "metadata": {},
  "status": "Success",
  "code": 200
}
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage 

collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ens

ures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
kubectl
command
$ kubectl get daemonset daemonset-example -o json
curl
command (
requires
)
$ curl -X GET http://127.0.0.1:8001/apis/apps/v1/namespaces/default/daemonsets/daemonset-example
Parameter
name
namespace
Parameter
pretty
Response
Code
200
DaemonSet
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send anothe

r list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expirati

on (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind DaemonSet. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/apps/v1/watch/namespaces/{namespace}/daemonsets/{name}
Parameter
name
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are re

turned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual c

hanges to a list of DaemonSet. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apps/v1/watch/namespaces/{namespace}/daemonsets
Parameter
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is hig

hly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
watch individual changes to a list of DaemonSet. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apps/v1/watch/daemonsets
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at t

he time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
DaemonSet
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
DaemonSet
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
DaemonSet
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
example
Deployment Config to run 3 nginx instances (max rollback set to 10 revisions).
metadata:
  # Unique key of the Deployment instance
  name: deployment-example
spec:
  # 3 Pods should exist at all times.
  replicas: 3
  selector:
      app: nginx
  template:

    metadata:
      labels:
        # Apply this label to pods and default
        # the Deployment label selector to this value
        app: nginx
    spec:
      - name: nginx
        # Run this image
        image: nginx:1.14
Group
Version
Kind
apps
v1
DeploymentList [apps/v1]
Field
string
kind
string
metadata
Standard object metadata.
spec
status
DeploymentSpec v1 apps
Deployment [apps/v1]
Field
integer
paused
boolean
integer
replicas
integer
integer
selector
strategy
DeploymentStrategy
:
template
DeploymentStatus v1 apps
Field
integer
integer
DeploymentCondition
array
:
merge
:
type
integer
integer
Total number of ready pods targeted by this deployment.
replicas
integer
integer
integer
DeploymentList v1 apps
Field
string
APIVersion defines the versioned schema of this represen

tation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
items
array
kind
string
metadata
ListMeta
DeploymentStrategy v1 apps
DeploymentSpec [apps/v1]
Field
RollingUpdateDeployment
type
string
RollingUpdateDeployment v1 apps
DeploymentStrategy [apps/v1]
Field
maxSurge
Create
kubectl
command
metadata:
spec:
  revisionHistoryLimit: 10
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        ports:
curl
command (
requires
)
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        ports:
' http://127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments
Output
deployment "deployment-example" created
{
  "kind": "Deployment",
    "name": "deployment-example",
    "selfLink": "/apis/apps/v1/namespaces/default/deployments/deployment-example",
    "uid": "4ccca349-9cb1-11e6-9c54-42010a800148",
    "resourceVersion": "2118306",
    "creationTimestamp": "2016-10-28T01:53:19Z",
      "app": "nginx"
    }
  },
  "spec": {
    "replicas": 3,
        "app": "nginx"
      }
    },
    "template":

 {
          "app": "nginx"
        }
      },
      "spec": {
          {
            "name": "nginx",
            "image": "nginx:1.14",
            "ports": [
              {
                "containerPort": 80,
                "protocol": "TCP"
              }
            ],
          }
        ],
      }
    },
    "strategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "maxUnavailable": 1,
        "maxSurge": 1
      }
    },
    "revisionHistoryLimit": 10
  },
  "status": {}
}
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
kubectl
command
$ kubectl patch deployment deployment-example -p \
	'{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"nginx:1.16"}]}}}}'
curl
command (
requires
)
$ curl -X PATCH -H 'Content-Type: application/strategic-merge-patch+json' --data '
{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"nginx:1.16"}]}}}}' \
	'http://127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments/deployment-example'
Output
"deployment-example" patched
{
    "uid": "5dc3a8e6-b0ee-11e6-aef0-42010af00229",
    "resourceVersion": "164489",
    "generation": 11,
    "creationTimestamp": "2016-11-22T20:00:50Z",
    },
    "annotations": {
      "deployment.kubernetes.io/revision": "5"
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "image": "nginx:1.16",
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    },
  },
    "observedGeneration": 10,

    "updatedReplicas": 3,
    "availableReplicas": 3
  }
}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
kubectl
command
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        image: nginx:1.16
        ports:
' | kubectl replace -f -
curl
command (
requires
)
$ curl -X PUT -H 'Content-Type: application/yaml' --data '
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        ports:
' http://127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments/deployment-example
Output
deployment "deployment-example" replaced
{
    "resourceVersion": "2119082",
    "generation": 5,
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    },
  },
    "observedGeneration": 4,
  }
}

Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
kubectl
command
$ kubectl delete deployment deployment-example
curl
command (
requires
)
' 'http://127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments/deployment-example'
Output
deployment "deployment-example" deleted
{
}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not

 recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https://ku

bernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
kubectl
command
$ kubectl get deployment deployment-example -o json
curl
command (
requires
)
$ curl -X GET http://127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments/deployment-example
Output
{
    "uid": "1b33145a-9c63-11e6-9c54-42010a800148",
    "resourceVersion": "2064726",
    "generation": 4,
    "creationTimestamp": "2016-10-27T16:33:35Z",
    },
      "deployment.kubernetes.io/revision": "1"
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    }
  },
  }
}
{
    },
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
        "max

Unavailable": 1,
      }
    }
  },
  }
}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
kubectl
command
$ kubectl get deployment -o json
curl
command (
requires
)
$ curl -X GET 'http://127.0.0.1:8001/apis/apps/v1/namespaces/default/deployments'
Output
{
  "kind": "List",
  "items": [
    {
      "kind": "Deployment",
      "apiVersion": "app/v1beta1",
        "name": "docs",
        "selfLink": "/apis/app/v1beta1/namespaces/default/deployments/docs",
        "uid": "ef49e1d2-915e-11e6-be81-42010a80003f",
        "resourceVersion": "1924126",
        "generation": 21,
        "creationTimestamp": "2016-10-13T16:06:00Z",
          "run": "docs"
        },
        "annotations": {
          "deployment.kubernetes.io/revision": "10",
          "replicatingperfection.net/push-image": "true"
        }
      },
      "spec": {
        "replicas": 1,
        "selector": {
          "matchLabels": {
            "run": "docs"
          }
        },
        "template": {
          "metadata": {
            "creationTimestamp": null,
            "labels": {
              "auto-pushed-image-pwittrock/api-docs": "1477496453",
              "run": "docs"
            }
          },
          "spec": {
            "containers": [
              {
                "name": "docs",
                "image": "pwittrock/api-docs:v9",
                "resources": {},
                "terminationMessagePath": "/dev/termination-log",
                "imagePullPolicy": "Always"
              }
            ],
            "restartPolicy": "Always",
            "terminationGracePeriodSeconds": 30,
            "dnsPolicy": "ClusterFirst",
            "securityContext": {}
          }
        },
        "strategy": {
          "type": "RollingUpdate",
          "rollingUpdate": {
            "maxUnavailable": 1,
            "maxSurge": 1
          }
        }
      },
      "status": {
        "observedGeneration": 21,
        "updatedReplicas": 1,
        "availableReplicas": 1
      }
    },
    {
        "name": "deployment-example",
        "selfLink": "/apis/app/v1beta1/namespaces/default/deployments/deployment-example",
        "uid": "1b33145a-9c63-11e6-9c54-42010a800148",
        "resourceVersion": "2064726",
        "generation": 4,
        "creationTimestamp": "2016-10-27T16:33:35Z",
        },
          "deployment.kubernetes.io/revision": "1"
        }
      },
      "spec": {
        "replicas": 3,
            "app": "nginx"
          }
        },
              "app": "nginx"
            }
          },
          "spec": {
              {
                "name": "nginx",
                "image": "nginx:1.14",
                "ports": [
                  {
                    "containerPort": 80,
                    "protocol": "TCP"
                  }
                ],
                "terminationMessagePath": "/dev/termina

tion-log",
                "imagePullPolicy": "IfNotPresent"
              }
            ],
          }
        },
          }
        }
      },
        "observedGeneration": 4,
        "updatedReplicas": 3,
        "availableReplicas": 3
      }
    }
  ]
}
{
  "items": [
    {
        },
        }
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
              {
              }
            ],
          }
        },
          }
        }
      },
      }
    },
    {
        },
        }
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
              {
                "ports": [
                  {
                  }
                ],
              }
            ],
            "s

ecurityContext": {}
          }
        },
          }
        }
      },
      }
    }
  ]
}
Parameter
namespace
Parameter
pretty
continue
limit
res

ourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and

 ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
kubectl
command
$ kubectl get deployment deployment-example --watch -o json
curl
command (
requires
)
$ curl -X GET 'http://127.0.0.1:8001/apis/apps/v1/watch/namespaces/default/deployments/deployment-example'
Output
{
	"type": "ADDED",
	"object": {
		"kind": "Deployment",
		"apiVersion": "apps/v1",
		"metadata": {
			"name": "deployment-example",
			"namespace": "default",
			"selfLink": "/apis/apps/v1/namespaces/default/deployments/deployment-example",
			"uid": "64c12290-9cbf-11e6-9c54-42010a800148",
			"resourceVersion": "2128095",
			"generation": 2,
			"creationTimestamp": "2016-10-28T03:34:12Z",
			"labels": {
				"app": "nginx"
			},
			"annotations": {
				"deployment.kubernetes.io/revision": "3"
			}
		},
		"spec": {
			"replicas": 3,
			"selector": {
				"matchLabels": {
					"app": "nginx"
				}
			},
			"template": {
				"metadata": {
					"creationTimestamp": null,
					"labels": {
						"app": "nginx"
					}
				},
				"spec": {
					"containers": [
						{
							"name": "nginx",
							"image": "nginx:1.14",
							"ports": [
								{
									"containerPort": 80,
									"protocol": "TCP"
								}
							],
							"resources": {
							},
							"terminationMessagePath": "/dev/termination-log",
							"imagePullPolicy": "IfNotPresent"
						}
					],
					"restartPolicy": "Always",
					"terminationGracePeriodSeconds": 30,
					"dnsPolicy": "ClusterFirst",
					"securityContext": {
					}
				}
			},
			"strategy": {
				"type": "RollingUpdate",
				"rollingUpdate": {
					"maxUnavailable": 1,
					"maxSurge": 1
				}
			}
		},
		"status": {
			"observedGeneration": 2,
			"updatedReplicas": 3,
			"availableReplicas": 3
		}
	}
}
{
			},
			}
		},
		"spec": {
				}
			},
					}
				},
				"spec": {
						{
							"ports": [
								{
								}
							],
			
				},
						}
					],
					}
				}
			},
				}
			}
		},
		}
	}
}
watch changes to an object of kind Deployment. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that 

is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Deployment. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apps/v1/watch/namespaces/{namespace}/deployments
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and wi

ll return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/apps/v1/watch/deployments
Parameter
continue
limit

 `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
PUT /apis/apps/v1/namespaces/{namespace}/deplo

yments/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Read Scale
read scale of the specified Deployment
GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale
Parameter
name
name of the Scale
namespace
Parameter
pretty
Response
Code
200
Scale
OK
replace scale of the specified Deployment
PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
partially update scale of the specified Deployment
PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Scale
OK
example
Job Config to print pi up to 2000 digits (then exit).
kind: Job
metadata:
  # Unique key of the Job instance
  name: example-job
spec:
  template:
    metadata:
      name: example-job
    spec:
      - name: pi
        image: perl
        command: ["perl"]
        args: ["-Mbignum=bpi", "-wle", "print bpi(2000)"]
      # Do not restart containers after they exit
Group
Version
K

ind
batch
v1
Job
JobList [batch/v1]
Field
string
kind
string
metadata
spec
JobSpec
Specification of the desired behavior of a job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
JobStatus
Current status of a job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
JobSpec v1 batch
Job [batch/v1]
JobTemplateSpec [batch/v1]
JobTemplateSpec [batch/v1beta1]
Field
integer
integer
string
CompletionMode specifies how Pod completions are tracked. It can be `NonIndexed` (default) or `Indexed`.  `NonIndexed` means that the Job is considered complete when there have been .spec.completions successfully completed Pods. Each Pod completion is homologous to each other.  `Indexed` means that the Pods of a Job get an associated completion index from 0 to (.spec.completions - 1), available in the annotation batch.kubernetes.io/job-completion-index. The Job is considered complete when there is one successfully completed Pod for each index. When value is `Indexed`, .spec.completions must be specified and `.spec.parallelism` must be less than or equal to 10^5.  This field is alpha-level and is only honored by servers that enable the IndexedJob feature gate. More completion modes can be added in the future. If the Job controller observes a mode that it doesn't recognize, the controller skips updates for the Job.
integer
Specifies the desired number of successfully finished pods the job should be run with.  Setting to nil means that the success of any pod signals the success of all pods, and allows parallelism to have any positive value.  Setting to 1 means that parallelism is limited to 1 and the success of that pod signals the success of the job. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
boolean
manualSelector controls generation of pod labels and pod selectors. Leave `manualSelector` unset unless you are certain what you are doing. When false or unset, the system pick labels unique to this job and appends those labels to the pod template.  When true, the user is responsible for picking unique labels and specifying the selector.  Failure to pick a unique label may cause this and other jobs to not function correctly.  However, You may see `manualSelector=true` in jobs that were created with the old `extensions/v1beta1` API. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/#specifying-your-own-pod-selector
integer
Specifies the maximum desired number of pods the job should run at any given time. The actual number of pods running in steady state will be less than this number when ((.spec.comple

tions - .status.successful) &lt; .spec.parallelism), i.e. when the work left to do is less than max parallelism. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
selector
A label query over pods that should match the pod count. Normally, the system sets this field for you. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
suspend
boolean
Suspend specifies whether the Job controller should create Pods or not. If a Job is created with suspend set to true, no Pods are created by the Job controller. If a Job is suspended after creation (i.e. the flag goes from false to true), the Job controller will delete all active Pods associated with this Job. Users must design their workload to gracefully handle this. Suspending a Job will reset the StartTime field of the Job, effectively resetting the ActiveDeadlineSeconds timer too. This is an alpha field and requires the SuspendJob feature gate to be enabled; otherwise this field may not be set to true. Defaults to false.
template
Describes the pod that will be created when executing a job. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
integer
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won't be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. This field is alpha-level and is only honored by servers that enable the TTLAfterFinished feature.
JobStatus v1 batch
Field
active
integer
The number of actively running pods.
string
Time
JobCondition
array
:
merge
:
type
The latest available observations of an object's current state. When a Job fails, one of the conditions will have type "Failed" and status true. When a Job is suspended, one of the conditions will have type "Suspended" and status true; when the Job is resumed, the status of this condition will become false. When a Job is completed, one of the conditions will have type "Complete" and status true. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
failed
integer
startTime
Time
succeeded
integer
JobList v1 batch
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contrib

utors/devel/sig-architecture/api-conventions.md#resources
items
Job
array
kind
string
metadata
ListMeta
Create
kubectl
command
$ echo 'apiVersion: batch/v1
kind: Job
metadata:
spec:
  template:
    metadata:
    spec:
      - name: pi
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
curl
command (
requires
)
kind: Job
metadata:
spec:
  template:
    metadata:
    spec:
      - name: pi
' http://127.0.0.1:8001/apis/batch/v1/namespaces/default/jobs
Output
job "example-job" created
{
  "kind": "Job",
  "apiVersion": "batch/v1",
    "name": "example-job",
    "selfLink": "/apis/batch/v1/namespaces/default/jobs/example-job",
    "uid": "d93a3569-a2be-11e6-a008-fa043d458cc7",
    "resourceVersion": "7479",
    "creationTimestamp": "2016-11-04T18:45:25Z"
  },
  "spec": {
    "parallelism": 1,
    "completions": 1,
        "controller-uid": "d93a3569-a2be-11e6-a008-fa043d458cc7"
      }
    },
        "name": "example-job",
          "controller-uid": "d93a3569-a2be-11e6-a008-fa043d458cc7",
          "job-name": "example-job"
        }
      },
      "spec": {
          {
            "name": "pi",
            "image": "perl",
              "perl",
              "-Mbignum=bpi",
              "-wle",
              "print bpi(2000)"
            ],
            "imagePullPolicy": "Always"
          }
        ],
        "restartPolicy": "Never",
      }
    }
  },
}
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Job
Response
Code
200
Job
OK
201
Job
Created
202
Job
Accepted
Patch
Parameter
name
namespace
Para

meter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Job
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Job
Response
Code
200
Job
OK
201
Job
Created
Delete
kubectl
command
$ kubectl delete job example-job
curl
command (
requires
)
' 'http://127.0.0.1:8001/apis/batch/v1/namespaces/default/jobs/example-job'
Output
job "example-job" deleted
{
}
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the

 garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Deprecated: please

 use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
kubectl
command
$ kubectl get job example-job -o json
curl
command (
requires
)
$ curl -X GET http://127.0.0.1:8001/apis/batch/v1/namespaces/default/jobs/example-job
Output
{
    "resourceVersion": "7482",
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
              "perl",
              "-wle",
            ],
          }
        ],
      }
    }
  },
    "startTime": "2016-11-04T18:45:25Z",
    "active": 1
  }
}
{
  },
  "spec": {
      }
    },
        }

      },
      "spec": {
          {
              "perl",
              "-wle",
            ],
          }
        ],
      }
    }
  },
  }
}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Job
OK
List
kubectl
command
$ kubectl get job -o json
curl
command (
requires
)
$ curl -X GET 'http://127.0.0.1:8001/apis/batch/v1/namespaces/default/jobs'
Output
{
  "kind": "JobList",
    "selfLink": "/apis/batch/v1/namespaces/default/jobs",
    "resourceVersion": "7589"
  },
  "items": [
    {
        "name": "",
        "selfLink": "/apis/batch/v1/namespaces/default/jobs/",
        "uid": "d93a3569-a2be-11e6-a008-fa043d458cc7",
        "resourceVersion": "7482",
        "creationTimestamp": "2016-11-04T18:45:25Z"
      },
      "spec": {
        "parallelism": 1,
        "completions": 1,
            "controller-uid": "d93a3569-a2be-11e6-a008-fa043d458cc7"
          }
        },
            "name": "",
              "controller-uid": "d93a3569-a2be-11e6-a008-fa043d458cc7",
              "job-name": ""
            }
          },
          "spec": {
              {
                "name": "pi",
                "image": "perl",
                "command": [
                  "perl",
                  "-Mbignum=bpi",
                  "-wle",
                  "print bpi(2000)"
                ],
              }
            ],
            "restartPolicy": "Never",
          }
        }
      },
        "startTime": "2016-11-04T18:45:25Z",
        "active": 1
      }
    }
  ]
}
{
  },
  "items": [
    {
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
   
           {
                  "perl",
                  "-wle",
                ],
              }
            ],
          }
        }
      },
      }
    }
  ]
}
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as 

a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
JobList
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees t

hat the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
JobList
OK
Watch
kubectl
command
$ kubectl get job example-job --watch -o json
curl
command (
requires
)
$ curl -X GET 'http://127.0.0.1:8001/apis/batch/v1/watch/namespaces/default/jobs/example-job'
Output
{
		"kind": "Job",
		"apiVersion": "batch/v1",
			"name": "example-job",
			"selfLink": "/apis/batch/v1/namespaces/default/jobs/example-job",
			"uid": "d93a3569-a2be-11e6-a008-fa043d458cc7",
			"resourceVersion": "7482",
			"creationTimestamp": "2016-11-04T18:45:25Z"
		},
		"spec": {
			"parallelism": 1,
			"completions": 1,
					"controller-uid": "d93a3569-a2be-11e6-a008-fa043d458cc7"
				}
			},
					"name": "example-job",
						"controller-uid": "d93a3569-a2be-11e6-a008-fa043d458cc7",
						"job-name": "example-job"
					}
				},
				"spec": {
						{
							"name": "pi",
							"image": "perl",
							"command": [
								"perl",
								"-Mbignum=bpi",
								"-wle",
								"print bpi(2000)"
							],
							},
							"imagePullPolicy": "Always"
						}
					],
					"restartPolicy": "Never",
					}
				}
			}
		},
			"startTime": "2016-11-04T18:45:25Z",
			"active": 1
		}
	}
}
{
		},
		"spec": {
				}
			},
					}
				},
				"spec": {
						{
			
					"perl",
								"-wle",
							],
							},
						}
					],
					}
				}
			}
		},
		}
	}
}
watch changes to an object of kind Job. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/batch/v1/watch/namespaces/{namespace}/jobs/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they s

ee all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Job. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/batch/v1/watch/namespaces/{namespace}/jobs
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty,

 clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/batch/v1/watch/jobs
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve 

the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Job
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Job
OK
Parameter
name
namespace
Paramete

r
pretty
dryRun
Parameter
body
Job
Response
Code
200
Job
OK
201
Job
Created
Pod v1 core
example
Pod Config to print "Hello World".
kind: Pod
metadata:
  name: pod-example
spec:
  - name: ubuntu
    image: ubuntu:trusty
    command: ["echo"]
    args: ["Hello World"]
Group
Version
Kind
core
v1
Pod
Warning:
It is recommended that users create Pods only through a Controller, and not directly.  See Controllers:
,
Job
, or
.
PodList [core/v1]
Field
string
kind
string
metadata
spec
PodSpec
Specification of the desired behavior of the pod. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
PodStatus
Most recently observed status of the pod. This data may not be up to date. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
PodSpec v1 core
Pod [core/v1]
PodTemplateSpec [core/v1]
Field
integer
affinity
Affinity
boolean
Container
array
:
merge
:
name
dnsConfig
PodDNSConfig
dnsPolicy
string
boolean
array
:
merge
:
name
List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated acti

ons such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod's ephemeralcontainers subresource. This field is alpha-level and is only honored by servers that enable the EphemeralContainers feature.
HostAlias
array
:
merge
:
ip
hostIPC
boolean
boolean
hostPID
boolean
hostname
string
array
:
merge
:
name
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod
Container
array
:
merge
:
name
List of initialization containers belonging to the pod. Init containers are executed in order prior to containers being started. If any init container fails, the pod is considered to have failed and is handled according to its restartPolicy. The name for an init container or normal container must be unique among all containers. Init containers may not have Lifecycle actions, Readiness probes, Liveness probes, or Startup probes. The resourceRequirements of an init container are taken into account during scheduling by finding the highest request/limit for each resource type, and then using the max of of that value or the sum of the normal containers. Limits are applied to init containers in a similar fashion. Init containers cannot currently be added or removed. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
nodeName
string
object
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
overhead
object
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead This field is alpha-level as of Kubernetes v1.16, and is only honored by servers that enable the PodOverhead feature.
string
PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset. This field is beta-level, gated by the NonPreemptingPriority feature-gate.
priority
integer
T

he priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority.
string
array
If specified, all readiness gates will be evaluated for pod readiness. A pod is ready when all its containers are ready AND all conditions specified in the readiness gates have status equal to "True" More info: https://git.k8s.io/enhancements/keps/sig-network/0007-pod-ready%2B%2B.md
string
Restart policy for all containers within the pod. One of Always, OnFailure, Never. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy
string
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod.  If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the "legacy" RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/runtime-class.md This is a beta feature as of Kubernetes v1.14.
string
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty.  See type description for default values of each field.
string
string
ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
boolean
boolean
subdomain
string
integer
Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a

 kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds.
array
TopologySpreadConstraint
array
:
merge
:
volumes
Volume
array
:
merge,retainKeys
:
name
List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes
PodStatus v1 core
Field
PodCondition
array
:
merge
:
type
Current service state of pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-conditions
ContainerStatus
array
The list has one entry per container in the manifest. Each entry is currently the output of `docker inspect`. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-and-container-status
array
Status for any ephemeral containers that have run in this pod. This field is alpha-level and is only populated by servers that enable the EphemeralContainers feature.
hostIP
string
array
The list has one entry per init container in the manifest. The most recent successful init container will have ready = true, the most recently started container will have startTime set. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-and-container-status
message
string
string
phase
string
The phase of a Pod is a simple, high-level summary of where the Pod is in its lifecycle. The conditions array, the reason and message fields, and the individual container status arrays contain more detail about the pod's status. There are five possible phase values:  Pending: The pod has been accepted by the Kubernetes system, but one or more of the container images has not been created. This includes time before being scheduled as well as time spent downloading images over the network, which could take a while. Running: The pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting. Succeeded: All containers in the pod have terminated in success, and will not be restarted. Failed: All containers in the pod have terminated, and at least one container has terminated in failure. The container either exited with non-zero status or was terminated by the system. Unknown: For some reason the state of the pod could not be obtained, typically due to an error in communicating with the host of the pod.  More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase
podIP
string
podIPs
PodIP
array
:
merge
:
ip
podIPs holds the IP addresses allocated to the pod. If this field is specifi

ed, the 0th entry must match the podIP field. Pods may be allocated at most 1 value for each of IPv4 and IPv6. This list is empty if no IPs have been allocated yet.
qosClass
string
The Quality of Service (QOS) classification assigned to the pod based on resource requirements See PodQOSClass type for available QOS classes More info: https://git.k8s.io/community/contributors/design-proposals/node/resource-qos.md
reason
string
startTime
Time
PodList v1 core
Field
string
items
Pod
array
List of pods. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md
kind
string
metadata
ListMeta
Standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Pod
Response
Code
200
Pod
OK
201
Pod
Created
202
Pod
Accepted
create eviction of a Pod
POST /api/v1/namespaces/{namespace}/pods/{name}/eviction
Parameter
name
name of the Eviction
namespace
Parameter
dryRun
pretty
Parameter
body
Eviction
Response
Code
200
Eviction
OK
201
Eviction
Created
202
Eviction
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the 

request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
Pod
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Pod
Response
Code
200
Pod
OK
201
Pod
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Pod
OK
202
Pod
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue)

 and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersion sets a constraint on what resourc

e versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Pod
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is

 not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
PodList
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue f

ield to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
PodList
OK
Watch
watch changes to an object of kind Pod. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/pods/{name}
Parameter
name
namespace
Parameter
continue
A selector to restrict the li

st of returned objects by their fields. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Pod. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/pods
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received

 with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /api/v1/watch/pods
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not r

ecognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry

 run stages will be processed
force
Parameter
body
Patch
Response
Code
200
Pod
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Pod
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Pod
Response
Code
200
Pod
OK
201
Pod
Created
Parameter
name
name of the EphemeralContainers
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
Descrip

tion
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
connect POST requests to portforward of Pod
POST /api/v1/namespaces/{namespace}/pods/{name}/portforward
Parameter
name
name of the PodPortForwardOptions
namespace
Parameter
ports
List of ports to forward Required when using WebSockets
Response
Code
200
string
OK
connect POST requests to proxy of Pod
POST /api/v1/namespaces/{namespace}/pods/{name}/proxy
Parameter
name
name of the PodProxyOptions
namespace
Parameter
path
Path is the URL path to use for the current proxy request to pod.
Response
Code
200
string
OK
POST /api/v1/namespaces/{namespace}/pods/{name}/proxy/{path}
Parameter
name
namespace
path
path to the resource
Parameter
path
Response
Code
200
string
OK
connect DELETE requests to proxy of Pod
DELETE /api/v1/namespaces/{namespace}/pods/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
DELETE /api/v1/namespaces/{namespace}/pods/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
connect GET requests to portforward of Pod
GET /api/v1/namespaces/{namespace}/pods/{name}/portforward
Parameter
name
namespace
Parameter
ports
Response
Code
200
string
OK
connect GET requests to proxy of Pod
GET /api/v1/namespaces/{namespace}/pods/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK

/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
connect HEAD requests to proxy of Pod
HEAD /api/v1/namespaces/{namespace}/pods/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
HEAD /api/v1/namespaces/{namespace}/pods/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
connect PUT requests to proxy of Pod
PUT /api/v1/namespaces/{namespace}/pods/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
PUT /api/v1/namespaces/{namespace}/pods/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Read Log
Parameter
name
namespace
Parameter
container
follow
insecureSkipTLSVerifyBackend indicates that the apiserver should not confirm the validity of the serving certificate of the backend it is connecting to.  This will make the HTTPS connection between the apiserver and the backend insecure. This means the apiserver cannot verify the log data it is receiving came from the real kubelet.  If the kubelet is configured to verify the apiserver's TLS credentials, it does not mean the connection to the real kubelet is vulnerable to a man in the middle attack (e.g. an attacker could not intercept the actual log data coming from the real kubelet).
pretty
previous
tailLines
If true, add an 

RFC3339 or RFC3339Nano timestamp at the beginning of every line of log output. Defaults to false.
Response
Code
200
string
OK
example
ReplicaSet Config to run 3 nginx instances.
kind: ReplicaSet
metadata:
  # Unique key of the ReplicaSet instance
  name: replicaset-example
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      # Run the nginx image
Group
Version
Kind
apps
v1
Warning:
In many cases it is recommended to create a
instead of ReplicaSet.
ReplicaSetList [apps/v1]
Field
string
kind
string
metadata
If the Labels of a ReplicaSet are empty, they are defaulted to be the same as the Pod(s) that the ReplicaSet manages. Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
Spec defines the specification of the desired behavior of the ReplicaSet. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Status is the most recently observed status of the ReplicaSet. This data may be out of date by some window of time. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
ReplicaSetSpec v1 apps
ReplicaSet [apps/v1]
Field
integer
replicas
integer
Replicas is the number of desired replicas. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#what-is-a-replicationcontroller
selector
Selector is a label query over pods that should match the replica count. Label keys and values that must match in order to be controlled by this replica set. It must match the pod template's labels. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
template
Template is the object that describes the pod that will be created if insufficient replicas are detected. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#pod-template
ReplicaSetStatus v1 apps
Field
integer
ReplicaSetCondition
array
:
merge
:
type
integer
integer
integer
The number of ready replicas for this replica set.
replicas
integer
Replicas is the most recently

 oberved number of replicas. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#what-is-a-replicationcontroller
ReplicaSetList v1 apps
Field
string
items
array
List of ReplicaSets. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is 

making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
A selector to restrict the list of returned objects by their field

s. Defaults to everything.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
object name and auth scope, such as for teams and project

s
Parameter
pretty
continue
limit
Timeout for the list/watch call. This limits the dura

tion of the call, regardless of any activity or inactivity.
watch
Response
Code
200
OK
Parameter
continue
limit
pretty
resourc

eVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ReplicaSet. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/apps/v1/watch/namespaces/{namespace}/replicasets/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued reque

sts. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ReplicaSet. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apps/v1/watch/namespaces/{namespace}/replicasets
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whe

ther more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/apps/v1/watch/replicasets
Parameter
continue
limit
limit is a maximum

 number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
replace st

atus of the specified ReplicaSet
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Read Scale
read scale of the specified ReplicaSet
GET /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/scale
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Scale
OK
replace scale of the specified ReplicaSet
PUT /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
partially update scale of the specified ReplicaSet
PATCH /apis/apps/v1/namespaces/{namespace}/replicasets/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Scale
OK
example
ReplicationController Config to run 3 nginx instances.
kind: ReplicationController
metadata:
  # Unique key of the ReplicationController instance
  name: replicationcontroller-example
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
      contai

ners:
Group
Version
Kind
core
v1
Warning:
instead of a ReplicationController.
ReplicationControllerList [core/v1]
Field
string
kind
string
metadata
If the Labels of a ReplicationController are empty, they are defaulted to be the same as the Pod(s) that the replication controller manages. Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
Spec defines the specification of the desired behavior of the replication controller. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Status is the most recently observed status of the replication controller. This data may be out of date by some window of time. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
ReplicationControllerSpec v1 core
ReplicationController [core/v1]
Field
integer
replicas
integer
Replicas is the number of desired replicas. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller
selector
object
Selector is a label query over pods that should match the Replicas count. If Selector is empty, it is defaulted to the labels present on the Pod template. Label keys and values that must match in order to be controlled by this replication controller, if empty defaulted to labels on Pod template. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
template
Template is the object that describes the pod that will be created if insufficient replicas are detected. This takes precedence over a TemplateRef. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#pod-template
ReplicationControllerStatus v1 core
Field
integer
ReplicationControllerCondition
array
:
merge
:
type
integer
integer
integer
replicas
integer
Replicas is the most recently oberved number of replicas. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller
Replicatio

nControllerList v1 core
Field
string
items
array
List of replication controllers. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is maki

ng these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
fieldSel

ector
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /api/v1/namespaces/{namespace}/re

plicationcontrollers
Parameter
namespace
Parameter
pretty
continue
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-co

ncepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be ser

ved from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ReplicationController. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/replicationcontrollers/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be i

dentical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ReplicationController. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/replicationcontrollers
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a l

imit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /api/v1/watch/replicationcontrollers
Parameter
continue
A

 selector to restrict the list of returned objects by their fields. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Par

ameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Read Scale
read scale of the specified ReplicationController
GET /api/v1/namespaces/{namespace}/replicationcontrollers/{name}/scale
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Scale
OK
replace scale of the specified ReplicationController
PUT /api/v1/namespaces/{namespace}/replicationcontrollers/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
partially update scale of the specified ReplicationController
PATCH /api/v1/namespaces/{namespace}/replicationcontrollers/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response

Code
200
Scale
OK
Group
Version
Kind
apps
v1
StatefulSetList [apps/v1]
Field
string
kind
string
metadata
spec
status
StatefulSetSpec v1 apps
StatefulSet [apps/v1]
Field
string
podManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. The default policy is `OrderedReady`, where pods are created in increasing order (pod-0, then pod-1, etc) and the controller will wait until each pod is ready before continuing. When scaling down, the pods are removed in the opposite order. The alternative policy is `Parallel` which will create pods in parallel to match the desired scale without waiting, and on scale down will delete all pods at once.
replicas
integer
integer
selector
selector is a label query over pods that should match the replica count. It must match the pod template's labels. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors
string
template
template is the object that describes the pod that will be created if insufficient replicas are detected. Each pod stamped out by the StatefulSet will fulfill this Template, but have a unique identity from the rest of the StatefulSet.
StatefulSetUpdateStrategy
array
StatefulSetStatus v1 apps
Field
integer
cond

itions
StatefulSetCondition
array
:
merge
:
type
integer
string
integer
integer
readyReplicas is the number of Pods created by the StatefulSet controller that have a Ready Condition.
replicas
integer
string
integer
StatefulSetList v1 apps
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned by other people

. Force flag must be unset for non-apply patch requests.
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token

 received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unse

t
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit t

o receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue wi

ll be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind StatefulSet. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/apps/v1/watch/namespaces/{namespace}/statefulsets/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that 

can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of StatefulSet. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apps/v1/watch/namespaces/{namespace}/statefulsets
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "nex

t key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/apps/v1/watch/statefulsets
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it mus

t restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/

#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Read Scale
read scale of the specified StatefulSet
GET /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Scale
OK
replace scale of the specified StatefulSet
PUT /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
partially update scale of the specified StatefulSet
PATCH /apis/apps/v1/namespaces/{namespace}/statefulsets/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only c

ontain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
Scale
OK
SERVICE APIs
Service API resources are responsible for stitching your workloads together into an accessible Loadbalanced Service.  By default,
Workloads
are only accessible within the cluster, and they must be exposed externally using a either
a *LoadBalancer* or *NodePort*
Service
.  For development, internally accessible
Workloads can be accessed via proxy through the api master using the
command.
Common resource types:
Services
for providing a single ip endpoint loadbalanced across multiple Workload replicas.
Ingress
for providing a https(s) endpoint http(s) routed to one or more *Services*.
Group
Version
Kind
core
v1
Endpoints
EndpointsList [core/v1]
Field
string
kind
string
metadata
subsets
EndpointSubset
array
EndpointsList v1 core
Field
string
items
Endpoints
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The v

alue must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Endpoints
Response
Code
200
Endpoints
OK
201
Endpoints
Created
202
Endpoints
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Endpoints
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Endpoints
Response
Code
200
Endpoints
OK
201
Endpoints
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan'

 - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calcu

lated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Endpoints
OK
List
Parameter
namespace
Parameter
pretty
continue
A selec

tor to restrict the list of returned objects by their fields. Defaults to everything.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after

 the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind Endpoints. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/endpoints/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recogn

ize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Endpoints. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/endpoints
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any speci

fic interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
watch individual chang

es to a list of Endpoints. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/endpoints
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-a

pi/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
EndpointSliceList [discovery/v1]
Field
string
string
endpoints
Endpoint
array
kind
string
metadata
ports
EndpointPort
array
EndpointSliceList v1 discovery
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
name of the E

ndpointSlice
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
HTT

P Request
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
propagationPoli

cy
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items

 exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not

 miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind EndpointSlice. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/discovery.k8s.io/v1/watch/namespaces/{namespace}/endpointslices/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired er

ror together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of EndpointSlice. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/discovery.k8s.io/v1/watch/namespaces/{namespace}/endpointslices
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enab

led in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
GET /apis/discovery.k8s.io/v1/watch/endpointslices
Para

meter
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the call, regar

dless of any activity or inactivity.
watch
Response
Code
200
OK
Group
Version
Kind
v1
Ingress
v1beta1
v1beta1
IngressList [networking/v1]
Field
string
kind
string
metadata
spec
Spec is the desired state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Status is the current state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
IngressSpec v1 networking
Ingress [networking/v1]
Field
string
IngressClassName is the name of the IngressClass cluster resource. The associated IngressClass defines which controller will implement the resource. This replaces the deprecated `kubernetes.io/ingress.class` annotation. For backwards compatibility, when that annotation is set, it must be given precedence over this field. The controller may emit a warning if the field and annotation have different values. Implementations of this API should ignore Ingresses without a class specified. An IngressClass resource may be marked as default, which can be used to set a default value for this field. For more information, refer to the IngressClass documentation.
rules
IngressRule
array
tls
IngressTLS
array
IngressStatus v1 networking
Field
LoadBalancerStatus
IngressList v1 networking
Field
string
items
Ingress
array
kind
string
metadata
ListMeta
Standard object's m

etadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
202
Ingress
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value must be non-ne

gative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers m

ay choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid wh

ether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results 

from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind Ingress. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/networking.k8s.io/v1/watch/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
o

bject name and auth scope, such as for teams and projects
Parameter
continue
limit
pretty
timeoutSecon

ds
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Ingress. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/networking.k8s.io/v1/watch/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
continue
limit
pretty
If 'true', then the output is 

pretty printed.
watch
Response
Code
200
OK
GET /apis/networking.k8s.io/v1/watch/ingresses
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first req

uest is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200

Ingress
OK
201
Ingress
Created
Group
Version
Kind
v1
v1beta1
IngressClassList [networking/v1]
Field
string
kind
string
metadata
spec
Spec is the desired state of the IngressClass. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
IngressClassSpec v1 networking
IngressClass [networking/v1]
Field
string
IngressClassParametersReference
IngressClassList v1 networking
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - Al

l: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen

 minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit

nMatch
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued r

equests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
watch changes to an object of kind IngressClass. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/networking.k8s.io/v1/watch/ingressclasses/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are avai

lable. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of IngressClass. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/networking.k8s.io/v1/watch/ingressclasses
Parameter
continue
limit
limit is a maximum number of responses to

 return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
example
Service Config to load balance traffic across all Pods with the app=nginx label.  Receives on and sends to port 80.  Exposes an externally accessible endpoint.
metadata:
  # Unique key of the Service instance
  name: service-example
spec:
  ports:
    # Accept traffic sent to port 80
    - name: http
      port: 80
      targetPort: 80
  selector:
    # Loadbalance traffic across Pods matching
    # this label selector
    app: nginx
  # Create an HA proxy in the cloud provider
  # with an External IP address - *Only supported
  # by some cloud providers*
  type: LoadBalancer
Group
Version
Kind
core
v1
Service
ServiceList [core/v1]
Field
string
kind
string
metadata
spec
Spec defines the behavior of a service. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Most recently observed status of the service. Populated by the system. Read-only. More info: https://git

.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
ServiceSpec v1 core
Service [core/v1]
Field
boolean
allocateLoadBalancerNodePorts defines if NodePorts will be automatically allocated for services with type LoadBalancer.  Default is "true". It may be set to "false" if the cluster load-balancer does not rely on NodePorts. allocateLoadBalancerNodePorts may only be set for services with type LoadBalancer and will be cleared if the type is changed to any other type. This field is alpha-level and is only honored by servers that enable the ServiceLBNodePortControl feature.
clusterIP
string
clusterIP is the IP address of the service and is usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be blank) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above).  Valid values are "None", empty string (""), or a valid IP address. Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required.  Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
ClusterIPs is a list of IP addresses assigned to this service, and are usually assigned randomly.  If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be empty) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above).  Valid values are "None", empty string (""), or a valid IP address.  Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required.  Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName.  If this field is not specified, it will be initialized from the clusterIP field.  If this field is specified, clients must ensure that clusterIPs[0] and clusterIP have the same value.  Unless the "IPv6DualStack" feature gate is enabled, this field is limited to one value, which must be the same as the clusterIP field.  If the feature gate is enabled, this field may hold a maximum of two entries (dual-stack IPs, in either order).  These IPs must correspond to the values of the ipFamilies field. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
externalIPs is a list of IP addresses for which nodes in the cluster will also accept traffic for this service.  These IPs are not managed by Kubernetes.  The user is responsible for ensuring that traffic arrives at a node with this IP.  A common example is external load-balancers that are not part of the Kubernetes system.
string
externalName is the external reference that discovery mechanisms will return as an alias for this service (e.g. a DNS CNAME record). No 

proxying will be involved.  Must be a lowercase RFC-1123 hostname (https://tools.ietf.org/html/rfc1123) and requires `type` to be "ExternalName".
string
externalTrafficPolicy denotes if this Service desires to route external traffic to node-local or cluster-wide endpoints. "Local" preserves the client source IP and avoids a second hop for LoadBalancer and Nodeport type services, but risks potentially imbalanced traffic spreading. "Cluster" obscures the client source IP and may cause a second hop to another node, but should have good overall load-spreading.
integer
healthCheckNodePort specifies the healthcheck nodePort for the service. This only applies when type is set to LoadBalancer and externalTrafficPolicy is set to Local. If a value is specified, is in-range, and is not in use, it will be used.  If not specified, a value will be automatically allocated.  External systems (e.g. load-balancers) can use this port to determine if a given node holds endpoints for this service or not.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type).
string
InternalTrafficPolicy specifies if the cluster internal traffic should be routed to all endpoints or node-local endpoints only. "Cluster" routes internal traffic to a Service to all endpoints. "Local" routes traffic to node-local endpoints only, traffic is dropped if no node-local endpoints are ready. The default value is "Cluster".
IPFamilies is a list of IP families (e.g. IPv4, IPv6) assigned to this service, and is gated by the "IPv6DualStack" feature gate.  This field is usually assigned automatically based on cluster configuration and the ipFamilyPolicy field. If this field is specified manually, the requested family is available in the cluster, and ipFamilyPolicy allows it, it will be used; otherwise creation of the service will fail.  This field is conditionally mutable: it allows for adding or removing a secondary IP family, but it does not allow changing the primary IP family of the Service.  Valid values are "IPv4" and "IPv6".  This field only applies to Services of types ClusterIP, NodePort, and LoadBalancer, and does apply to "headless" services.  This field will be wiped when updating a Service to type ExternalName.  This field may hold a maximum of two entries (dual-stack families, in either order).  These families must correspond to the values of the clusterIPs field, if specified. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field.
string
IPFamilyPolicy represents the dual-stack-ness requested or required by this Service, and is gated by the "IPv6DualStack" feature gate.  If there is no value provided, then this field will be set to SingleStack. Services can be "SingleStack" (a single IP family), "PreferDualStack" (two IP families on dual-stack configured clusters or a single IP family on single-stack clusters), or "RequireDualStack" (two IP families on dual-stack configured clusters, otherwise fail). The ipFamilies and clusterIPs fields depend on the value of this field.  This field will be wiped when updating a service to type ExternalName.
string
loadBalancerClass is the class of the load balancer implementation this Service belongs to. If specified, the value of this field must be a label-style identifier, with an optional prefix, e.g. "internal-vip" or "example.com/internal-vip". Unprefixed names are reserved for end-users. This field can only be set when the Service type is 'LoadBalancer'. If not set, the default load balancer implementation is used, today this is typically done through the cloud provider integration, but should apply for any default implementation. If set, it is assumed that a load balancer implementation is watching for Services with a matching class. Any default load balancer

 implementation (e.g. cloud providers) should ignore Services that set this field. This field can only be set when creating or updating a Service to type 'LoadBalancer'. Once set, it can not be changed. This field will be wiped when a service is updated to a non 'LoadBalancer' type.
string
Only applies to Service Type: LoadBalancer LoadBalancer will get created with the IP specified in this field. This feature depends on whether the underlying cloud-provider supports specifying the loadBalancerIP when a load balancer is created. This field will be ignored if the cloud-provider does not support the feature.
If specified and supported by the platform, this will restrict traffic through the cloud-provider load-balancer will be restricted to the specified client IPs. This field will be ignored if the cloud-provider does not support the feature." More info: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/
ports
ServicePort
array
:
merge
:
port
The list of ports that are exposed by this service. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
boolean
selector
object
Route service traffic to pods with label keys and values matching this selector. If empty or not present, the service is assumed to have an external process managing its endpoints, which Kubernetes will not modify. Only applies to types ClusterIP, NodePort, and LoadBalancer. Ignored if type is ExternalName. More info: https://kubernetes.io/docs/concepts/services-networking/service/
string
Supports "ClientIP" and "None". Used to maintain session affinity. Enable client IP based session affinity. Must be ClientIP or None. Defaults to None. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
SessionAffinityConfig
topologyKeys
topologyKeys is a preference-order list of topology keys which implementations of services should use to preferentially sort endpoints when accessing this Service, it can not be used at the same time as externalTrafficPolicy=Local. Topology keys must be valid label keys and at most 16 keys may be specified. Endpoints are chosen based on the first topology key with available backends. If this field is specified and all entries have no backends that match the topology of the client, the service has no backends for that client and connections should fail. The special value "*" may be used to mean "any topology". This catch-all value, if used, only makes sense as the last value in the list. If this is not specified or empty, no topology constraints will be applied. This field is alpha-level and is only honored by servers that enable the ServiceTopology feature. This field is deprecated and will be removed in a future version.
type
string
type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. "ClusterIP" allocates a cluster-internal IP address for load-balancing to endpoints. Endpoints are determined by the selector or if that is not specified, by manual constru

ction of an Endpoints object or EndpointSlice objects. If clusterIP is "None", no virtual IP is allocated and the endpoints are published as a set of endpoints rather than a virtual IP. "NodePort" builds on ClusterIP and allocates a port on every node which routes to the same endpoints as the clusterIP. "LoadBalancer" builds on NodePort and creates an external load-balancer (if supported in the current cloud) which routes to the same endpoints as the clusterIP. "ExternalName" aliases this service to the specified externalName. Several other fields do not apply to ExternalName services. More info: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
ServiceStatus v1 core
Field
Condition
array
:
merge
:
type
ServiceList v1 core
Field
string
items
Service
array
kind
string
metadata
ListMeta
Create
kubectl
command
$ echo 'kind: Service
metadata:
spec:
  ports:
      port: 80
  selector:
      app: nginx
curl
command (
requires
)
metadata:
spec:
  ports:
      port: 80
  selector:
      app: nginx
' http://127.0.0.1:8001/api/v1/namespaces/default/services
Output
service "service-example" created
{
  "kind": "Service",
    "name": "service-example",
    "selfLink": "/api/v1/namespaces/default/services/service-example",
    "uid": "93e5c731-9d30-11e6-9c54-42010a800148",
    "resourceVersion": "2205767",
    "creationTimestamp": "2016-10-28T17:04:24Z"
  },
  "spec": {
    "ports": [
      {
        "name": "http",
        "protocol": "TCP",
        "port": 80,
        "targetPort": 80,
        "nodePort": 32417
      }
    ],
    },
    "clusterIP": "10.183.250.161",
    "type": "LoadBalancer",
    "sessionAffinity": "None"
  },
    "loadBalancer": {}
  }
}
Parameter
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, a

s defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Service
Response
Code
200
Service
OK
201
Service
Created
202
Service
Accepted
Patch
kubectl
command
$ kubectl patch service  -p \
	'{"spec":{"ports":[{"name":"http","port":80,"targetPort":8080}]}}'
curl
command (
requires
)
{"spec":{"ports":[{"name":"http","port":80,"targetPort":8080}]}}' \
	'http://127.0.0.1:8001/api/v1/namespaces/default/services/'
Output
"" patched
{
    "selfLink": "/api/v1/namespaces/default/services/deployment-example",
    "resourceVersion": "2205995",
  },
  "spec": {
    "ports": [
      {
        "targetPort": 8080,
      }
    ],
    },
  },
    "loadBalancer": {
      "ingress": [
        {
          "ip": "104.198.186.106"
        }
      ]
    }
  }
}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Service
OK
Replace
kubectl
command
$ echo 'apiVersion: v1
metadata:
  resourceVersion: "2205995"
spec:
  clusterIP: 10.183.250.161
  ports:
  - name: http
    nodePort: 32417
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: nginx
  sessionAffinity: None
curl
command (
requires
)
metadata:
spec:
  ports:
    port: 80
  selector:
    app: nginx
' http://127.0.0.1:8001/api/v1/namespaces/default/services/deployment-example
Output
service "deployment-example" replaced
{
    "uid": "93e5c731-9d30-11e6-9c54-42010a80014

8",
    "resourceVersion": "2208672",
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
        {
        }
      ]
    }
  }
}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Service
Response
Code
200
Service
OK
201
Service
Created
Delete
kubectl
command
$ kubectl delete service deployment-example
curl
command (
requires
)
' 'http://127.0.0.1:8001/api/v1/namespaces/default/services/deployment-example'
Output
service "deployment-example" deleted
{
}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Read
kubectl
command
$ kubectl get service deployment-example -o json
curl
command (
req

uires
)
$ curl -X GET http://127.0.0.1:8001/api/v1/namespaces/default/services/deployment-example
Output
{
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
        {
        }
      ]
    }
  }
}
{
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
        {
        }
      ]
    }
  }
}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Service
OK
List
kubectl
command
$ kubectl get service -o json
curl
command (
requires
)
$ curl -X GET 'http://127.0.0.1:8001/api/v1/namespaces/default/services'
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot

, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue t

oken. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
kubectl
command
$ kubectl get service deployment-example --watch -o json
curl
command (
requires
)
$ curl -X GET 'http://127.0.0.1:8001/api/v1/watch/namespaces/default/services/deployment-example'
Output
{
		"kind": "Service",
		"apiVersion": "v1",
			"selfLink": "/api/v1/namespaces/default/services/deployment-example",
			"uid": "93e5c731-9d30-11e6-9c54-42010a800148",
			"resourceVersion": "2205995",
			"creationTimestamp": "2016-10-28T17:04:24Z"
		},
		"spec": {
			"ports": [
				{
					"name": "http",
					"protocol": "TCP",
					"port": 80,
					"targetPort": 8080,
					"nodePo

rt": 32417
				}
			],
			},
			"clusterIP": "10.183.250.161",
			"type": "LoadBalancer",
			"sessionAffinity": "None"
		},
			"loadBalancer": {
				"ingress": [
					{
						"ip": "104.198.186.106"
					}
				]
			}
		}
	}
}
{
		},
		"spec": {
			"ports": [
				{
					"nodePort": 32417
				}
			],
			},
		},
					{
					}
				]
			}
		}
	}
}
watch changes to an object of kind Service. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/services/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results.

 If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Service. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/services
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to retu

rn for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /api/v1/watch/services
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  T

his field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Service
OK
read status of the specified Ser

vice
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Service
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Service
Response
Code
200
Service
OK
201
Service
Created
connect POST requests to proxy of Service
POST /api/v1/namespaces/{namespace}/services/{name}/proxy
Parameter
name
name of the ServiceProxyOptions
namespace
Parameter
path
Path is the part of URLs that include service endpoints, suffixes, and parameters to use for the current proxy request to service. For example, the whole request URL is http://localhost/api/v1/namespaces/kube-system/services/elasticsearch-logging/_search?q=user:kimchy. Path is _search?q=user:kimchy.
Response
Code
200
string
OK
POST /api/v1/namespaces/{namespace}/services/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
connect DELETE requests to proxy of Service
DELETE /api/v1/namespaces/{namespace}/services/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
DELETE /api/v1/namespaces/{namespace}/services/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Path is the part of URLs that include service endpoints, suffixes, and parameters to use for the current proxy request to service. For example, the whole request URL is http://localhost/api/v1/namespaces/kube-system/services/elasticsearch-logging/_search?q=user:kimchy. Path is _se

arch?q=user:kimchy.
Response
Code
200
string
OK
connect GET requests to proxy of Service
GET /api/v1/namespaces/{namespace}/services/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
GET /api/v1/namespaces/{namespace}/services/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
connect HEAD requests to proxy of Service
HEAD /api/v1/namespaces/{namespace}/services/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
HEAD /api/v1/namespaces/{namespace}/services/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
connect PUT requests to proxy of Service
PUT /api/v1/namespaces/{namespace}/services/{name}/proxy
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
PUT /api/v1/namespaces/{namespace}/services/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Path is the part of URLs that include service endpoints, suffixes, and parameters to use for the current proxy request to service. For example, the whole request URL is http://localhost/api/v1/namespaces/k

ube-system/services/elasticsearch-logging/_search?q=user:kimchy. Path is _search?q=user:kimchy.
Response
Code
200
string
OK
CONFIG &amp; STORAGE
Config and Storage resources are responsible for injecting data into your applications and persisting data externally to your container.
for providing text key value pairs injected into the application through environment variables, command line arguments, or files
Secrets
for providing binary data injected into the application through files
Volumes
for providing a filesystem external to the Container.  Maybe shared across Containers within the same Pod and have a lifetime persisting beyond a Container or Pod.
Group
Version
Kind
core
v1
ConfigMap
ConfigMapList [core/v1]
Field
string
object
data
object
immutable
boolean
kind
string
metadata
ConfigMapList v1 core
Field
string
items
ConfigMap
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less tha

n or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
ConfigMap
Response
Code
200
ConfigMap
OK
201
ConfigMap
Created
202
ConfigMap
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
ConfigMap
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
ConfigMap
Response
Code
200
ConfigMap
OK
201
ConfigMap
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the de

pendents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is return

ed.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
ConfigMap
OK
List
Parameter
namespace
Parameter
pretty
continue
A selector to restri

ct the list of returned objects by their fields. Defaults to everything.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first l

ist request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ConfigMap. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/configmaps/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the

 specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ConfigMap. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/configmaps
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interv

al, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
watch individual changes to a li

st of ConfigMap. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/configmaps
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
CSIDriver v1 storage.k8s.io
Group
Version
Kind
v1
CSIDriver
v1beta1
CSIDriverList [storage/v1]
Field
string
kind
string
metadata
Standard object metadata. metadata.Name indicates the name of the CSI driver that this object refers to; it MUST be the same name returned by the CSI GetPluginName() call for that driver. The driver name must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), dots (.), and alphanumerics between. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
CSIDriverSpec v1 storage
CSIDriver [storage/v1]
Field
boolean
attachRequired indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume() method), and that the Kubernetes attach detach controller should call the attach volume interface which checks the volumeattachment status and waits until the volume is attached before proceeding to mounting. The CSI external-attacher coordinates with CSI volume driver and updates the volumeattachment status when the attach operation is complete. If the CSIDriverRegistry feature gate is enabled and the value is specified to false, the attach operation will be skipped. Otherwise the attach operation will be called.  This field is immutable.
string
Defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupPolicy values for additional details. This field is alpha-level, and is only honored by servers that enable the CSIVolumeFSGroupPolicy feature gate.  This field is immutable.
boolean
If set to true, podInfoOnMount indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during mount operations. If set to false, pod information will not be passed on mount. Default is false. The CSI driver specifies podInfoOnMount as part of driver deployment. If true, Kubelet will pass pod information as VolumeContext in the CSI NodePublishVolume() calls. The CSI driver is responsible for parsing and validating the information passed in as VolumeContext. The following VolumeConext will be passed if podInfoOnMount is set to true. This list might grow, but the prefix will be used. "csi.storage.k8s.io/pod.name": pod.Name "csi.storage.k8s.io/pod.namespace": pod.Namespace "csi.storage.k8s.io/pod.uid": string(pod.UID) "csi.storage.k8s.io/ephemeral": "true" if the volume is an ephemeral inline volume                                 defined by a CSIVolumeSource, otherwise "false"  "csi.storage.k8s.io/ephemeral" is a new feature in Kubernetes 1.16. It is only required for drivers which support both the "Persistent" and "Ephemeral" VolumeLifecycleMode. Other drivers can leave pod info disabled and/or ignore t

his field. As Kubernetes 1.15 doesn't support this field, drivers can only support one mode when deployed on such a cluster and the deployment determines which mode that is, for example via a command line parameter of the driver.  This field is immutable.
boolean
RequiresRepublish indicates the CSI driver wants `NodePublishVolume` being periodically called to reflect any possible change in the mounted volume. This field defaults to false.  Note: After a successful initial NodePublishVolume call, subsequent calls to NodePublishVolume should only update the contents of the volume. New mount points will not be seen by a running container.  This is a beta feature and only available when the CSIServiceAccountToken feature is enabled.
boolean
If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.  The check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.  Alternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.  This field is immutable.  This is a beta field and only available when the CSIStorageCapacity feature is enabled. The default is false.
array
TokenRequests indicates the CSI driver needs pods' service account tokens it is mounting volume for to do necessary authentication. Kubelet will pass the tokens in VolumeContext in the CSI NodePublishVolume calls. The CSI driver should parse and validate the following VolumeContext: "csi.storage.k8s.io/serviceAccount.tokens": {   "&lt;audience&gt;": {     "token": &lt;token&gt;,     "expirationTimestamp": &lt;expiration timestamp in RFC3339&gt;,   },   ... }  Note: Audience in each TokenRequest should be different and at most one token is empty string. To receive a new token after expiry, RequiresRepublish can be used to trigger NodePublishVolume periodically.  This is a beta feature and only available when the CSIServiceAccountToken feature is enabled.
volumeLifecycleModes defines what kind of volumes this CSI volume driver supports. The default if the list is empty is "Persistent", which is the usage defined by the CSI specification and implemented in Kubernetes via the usual PV/PVC mechanism. The other mode is "Ephemeral". In this mode, volumes are defined inline inside the pod spec with CSIVolumeSource and their lifecycle is tied to the lifecycle of that pod. A driver has to be aware of this because it is only going to get a NodePublishVolume call for such a volume. For more information about implementing this mode, see https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html A driver can support one or more of these modes and more modes may be added in the future. This field is beta.  This field is immutable.
CSIDriverList v1 storage
Field
string
items
CSIDriver
array
kind
string
metadata
ListMeta
Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

Create
Parameter
pretty
dryRun
Parameter
body
CSIDriver
Response
Code
200
CSIDriver
OK
201
CSIDriver
Created
202
CSIDriver
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
CSIDriver
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
CSIDriver
Response
Code
200
CSIDriver
OK
201
CSIDriver
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set

, but not both.
Parameter
body
Response
Code
200
CSIDriver
OK
202
CSIDriver
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smalle

r chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
CSIDriver
OK
List
Parameter
pretty
continue
fiel

dSelector
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind CSIDriver. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/storage.k8s.io/v1/watch/csidrivers/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the t

oken received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of CSIDriver. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/storage.k8s.io/v1/watch/csidrivers
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a co

ntinue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
CSINode
v1beta1
CSINodeList [storage/v1]
Field
string
kind
string
Kind is a string value representing the REST resource this object repr

esents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
spec
CSINodeSpec v1 storage
CSINode [storage/v1]
Field
drivers
CSINodeDriver
array
:
merge
:
name
CSINodeList v1 storage
Field
string
items
CSINode
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
CSINode
Response
Code
200
CSINode
OK
201
CSINode
Created
202
CSINode
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
CSINode
OK
Replace
Parameter
name
Parameter
pretty
dryRun
fie

ldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
CSINode
Response
Code
200
CSINode
OK
201
CSINode
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
CSINode
OK
202
CSINode
Accepted
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero in

dicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
CSINode
OK
List
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they as

sume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind CSINode. deprecated: use the 'watch' parameter with a list operation instead, filtered t

o a single item with the 'fieldSelector' parameter.
GET /apis/storage.k8s.io/v1/watch/csinodes/{name}
Parameter
name
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See htt

ps://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of CSINode. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/storage.k8s.io/v1/watch/csinodes
Parameter
continue
limit
pretty
If 'true', then the output is pretty printed

.
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Secret
SecretList [core/v1]
Field
string
data
object
Data contains the secret data. Each key must consist of alphanumeric characters, '-', '_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here. Described in https://tools.ietf.org/html/rfc4648#section-4
immutable
boolean
kind
string
metadata
object
type
string
Used to facilitate programmatic handling of secret data.
SecretList v1 core
Field
string
items
Secret
array
Items is a list of secret objects. More info: https://kubernetes.io/docs/concepts/configuration/secret
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further proc

essing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Secret
Response
Code
200
Secret
OK
201
Secret
Created
202
Secret
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Secret
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Secret
Response
Code
200
Secret
OK
201
Secret
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default po

licy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible

 objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Secret
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may s

tart a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest 

snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind Secret. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/secrets/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query p

arameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Secret. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/secrets
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmar

ks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resource

Version.
Response
Code
200
OK
GET /api/v1/watch/secrets
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be 

set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
A
must be allocated in the cluster to use this.
PersistentVolumeClaimList [core/v1]
StatefulSetSpec [apps/v1]
Field
string
kind
string
metadata
spec
Spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
status
Status represents the current information/status of a persistent volume claim. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
PersistentVolumeClaimSpec v1 core
PersistentVolumeClaim [core/v1]
PersistentVolumeClaimTemplate [core/v1]
Field
AccessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
This field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) * An existing custom resource that implements data population (Alpha) In order to use custom resource types that implement data population, the AnyVolumeDataSource feature gate must be enabled. If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source.
resources
Resources represents the minimum resources the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources
selector
A label query over volumes to consider for binding.
string
Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1
string
string
VolumeName is the binding reference to the PersistentVolume backing this claim.
PersistentVolumeClaimStatus v1 core
Field
AccessModes contains the actual access modes the volume backing the PVC has. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
capacity
object
Represents the actual resources of the underlying volume.
PersistentVolumeClaimCondition
array
:
merge
:
type
Current Condition of persistent volume claim. If underlying persistent volume is being resized then t

he Condition will be set to 'ResizeStarted'.
phase
string
Phase represents the current phase of PersistentVolumeClaim.
PersistentVolumeClaimList v1 core
Field
string
items
array
A list of persistent volume claims. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid va

lues are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecog

nized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
If 'true', then the output is pretty pri

nted.
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
resourceVersionMatch determines how resourceVersion is applied to lis

t calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PersistentVolumeClaim. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/persistentvolumeclaims/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may

 assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PersistentVolumeClaim. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/persistentvolumeclaims
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist

, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /api/v1/watch/persistentvolumeclaims
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field i

s not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK

Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
v1beta1
StorageClassList [storage/v1]
Field
boolean
TopologySelectorTerm
array
string
kind
string
metadata
object
string
string
string
VolumeBindingMode indicates how PersistentVolumeClaims should be provisioned and bound.  When unset, VolumeBindingImmediate is used. This field is only honored by servers that enable the VolumeScheduling feature.
StorageClassList v1 storage
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architect

ure/api-conventions.md#resources
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be proces

sed
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choo

se not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client n

eeds a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind StorageClass. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/storage.k8s.io/v1/watch/storageclasses/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving mo

re results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of StorageClass. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/storage.k8s.io/v1/watch/storageclasses
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do no

t implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a stream of a

dd, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1alpha1
CSIStorageCapacityList [storage/v1beta1]
Field
string
capacity
Quantity
Capacity is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.  The semantic is currently (CSI spec 1.2) defined as: The available capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable and treated like zero capacity.
kind
string
Quantity
MaximumVolumeSize is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.  This is defined since CSI spec 1.4.0 as the largest size that may be used in a CreateVolumeRequest.capacity_range.required_bytes field to create a volume with the same parameters as those in GetCapacityRequest. The corresponding value in the Kubernetes API is ResourceRequirements.Requests in a volume claim.
metadata
Standard object's metadata. The name has no particular meaning. It must be be a DNS subdomain (dots allowed, 253 characters). To ensure that there are no conflicts with other CSI drivers on the cluster, the recommendation is to use csisc-&lt;uuid&gt;, a generated name, or a reverse-domain name which ends with the unique CSI driver name.  Objects are namespaced.  More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
string
CSIStorageCapacityList v1beta1 storage
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/storage.k8s.io/v1beta1/namespaces/{namespace}/csistoragecapacities
Pa

rameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/storage.k8s.io/v1beta1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/storage.k8s.io/v1beta1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/storage.k8s.io/v1beta1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately

. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/storage.k8s.io/v1beta1/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit 

argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/storage.k8s.io/v1beta1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/storage.k8s.io/v1beta1/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
Parameter
pretty
continue

limit
watch
Response
Code
200
OK
GET /apis/storage.k8s.io/v1beta1/csistoragecapacities
Parameter
co

ntinue
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/storage.k8s.io/v1beta1/watch/namespaces/{namespac

e}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https:/

/kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of CSIStorageCapacity. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/storage.k8s.io/v1beta1/watch/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list th

e version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/storage.k8s.io/v1beta1/watch/csistoragecapacities
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects 

returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Volume
Field
AWSElasticBlockStoreVolumeSource
AWSElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
azureDisk
AzureDiskVolumeSource
azureFile
AzureFileVolumeSource
cephfs
CephFSVolumeSource
CephFS represents a Ceph FS mount on the host that shares a pod's lifetime
cinder
CinderVolumeSource
Cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
configMap
ConfigMapVolumeSource
ConfigMap represents a configMap that should populate this volume
csi
CSIVolumeSource
CSI (Container Storage Interface) represents ephemeral storage that is handled by certain external CSI drivers (Beta feature).
DownwardAPI represents downward API about the pod that should populate this volume
emptyDir
EmptyDirVolumeSource
EmptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
ephemeral
EphemeralVolumeSource
Ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed.  Use this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity    tracking are needed, c) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through    a PersistentVolumeClaim (see EphemeralVolumeSource for more    information on the connection between this volume type    and PersistentVolumeClaim).  Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod.  Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information.  A pod can use both types of ephemeral volumes and persistent volumes at the same time.  This is a beta feature and only avail

able when the GenericEphemeralVolume feature gate is enabled.
fc
FCVolumeSource
FC represents a Fibre Channel resource that is attached to a kubelet's host machine and then exposed to the pod.
FlexVolumeSource
flocker
FlockerVolumeSource
Flocker represents a Flocker volume attached to a kubelet's host machine. This depends on the Flocker control service being running
GCEPersistentDiskVolumeSource
GCEPersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
gitRepo
GitRepoVolumeSource
GitRepo represents a git repository at a particular revision. DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container.
glusterfs
GlusterfsVolumeSource
Glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md
hostPath
HostPathVolumeSource
HostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
iscsi
ISCSIVolumeSource
ISCSI represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md
name
string
Volume's name. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
nfs
NFSVolumeSource
NFS represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
PersistentVolumeClaimVolumeSource
PersistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
PhotonPersistentDiskVolumeSource
PhotonPersistentDisk represents a PhotonController persistent disk attached and mounted on kubelets host machine
PortworxVolumeSource
PortworxVolume represents a portworx volume attached and mounted on kubelets host machine
projected
ProjectedVolumeSource
Items for all in one resources secrets, configmaps, and downward API
quobyte
QuobyteVolumeSource
Quobyte represents a Quobyte mount on the host that shares a pod's lifetime
rbd
RBDVolumeSource
RBD represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md
scaleIO
ScaleIOVolumeSource
ScaleIO represents a ScaleIO persistent volume attached and mounted on Kubernetes nodes.
secret
SecretVolumeSource
Secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret
storageos
StorageOSVolumeSource
StorageOS represents a StorageOS volume attached and mounted on Kubernetes nodes.
VsphereVirtualDiskVolumeSource
VsphereVolume represents a vSphere volume attached and mounted on kubelets host machine
Group
Version
Kind
v1
v1beta1
v1alpha1
VolumeAttachmentList [storage/v1]
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig

-architecture/api-conventions.md#resources
kind
string
metadata
Standard object metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
status
VolumeAttachmentSpec v1 storage
VolumeAttachment [storage/v1]
Field
attacher
string
nodeName
string
source
VolumeAttachmentSource
VolumeAttachmentStatus v1 storage
Field
VolumeError
attached
boolean
object
VolumeAttachmentList v1 storage
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
Descri

ption
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
pretty
continue
The continue optio

n should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 

'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If

 limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
watch changes to an object of kind VolumeAttachment. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/storage.k8s.io/v1/watch/volumeattachments/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a 

list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of VolumeAttachment. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/storage.k8s.io/v1/watch/volumeattachments
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET

 /apis/storage.k8s.io/v1/volumeattachments/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
METADATA
Metadata resources are responsible for configuring behavior of your other Resources within the Cluster.
(HPA) for automatically scaling the replicacount of your workloads in response to load.
for configuring how many replicas in a given workload maybe made concurrently unavailable when performing maintenance.
Event
for notification of resource lifecycle events in the cluster.
Group
Version
Kind
apps
v1
ControllerRevisionList [apps/v1]
Field
string
data
kind
string
metadata
revision
integer
ControllerRevisionList v1 apps
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun di

rective will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" fin

alizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, n

o objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token re

ceived with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (g

enerally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ControllerRevision. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/apps/v1/watch/namespaces/{namespace}/controllerrevisions/{name}
Parameter
name
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients sho

uld not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
O

K
Watch List
watch individual changes to a list of ControllerRevision. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apps/v1/watch/namespaces/{namespace}/controllerrevisions
Parameter
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how re

sourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
GET /apis/apps/v1/watch/controllerrevisions
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated du

ring a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
CustomResourceDefinitionList [apiextensions/v1]
Field
string
kind
string
metadata
spec
status
CustomResourceDefinitionSpec v1 apiextensions
CustomResourceDefinition [apiextensions/v1]
Field
CustomResourceConversion
group
string
group is the API group of the defined custom resource. The custom resources are served under `/apis/&lt;group&gt;/...`. Must match the name of the CustomResourceDefinition (in the form `&lt;names.plural&gt;.&lt;group&gt;`).
names
CustomResourceDefinitionNames
boolean
preserveUnknownFields indicates that object fields which are not specified in the OpenAPI schema should be preserved when persisting to storage. apiVersion, kind, metadata and known fields inside metadata are always preserved. This field is deprecated in favor of setting `x-preserve-unknown-fields` to true in `spec.versions[*].schema.openAPIV3Schema`. See https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields for details.
scope
string
scope indicates whether the defined custom resource is cluster- or namespace-scoped. Allowed values are `Cluster` and `Namespaced`.
versions
CustomResourceDefinitionVersion
array
versions is the list of all API versions of the defined custom resource. Version names are used to compute the order in which served versions are listed in API discovery. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major 

version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10.
CustomResourceDefinitionStatus v1 apiextensions
Field
CustomResourceDefinitionCondition
array
storedVersions lists all versions of CustomResources that were ever persisted. Tracking these versions allows a migration path for stored versions in etcd. The field is mutable so a migration controller can finish a migration to another version (ensuring no old objects are left in storage), and then remove the rest of the versions from this list. Versions may not be removed from `spec.versions` while they exist in this list.
CustomResourceDefinitionList v1 apiextensions
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
repla

ce the specified CustomResourceDefinition
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response

, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read
read the specified CustomResourceDefini

tion
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/ap

i-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind CustomResourceDefinition. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/apiextensions.k8s.io/v1/watch/customresourcedefinitions/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the firs

t request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of CustomResourceDefinition. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apiextensions.k8s.io/v1/watch/customresourcedefinitions
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more result

s are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request

. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Event
v1beta1
EventList [events/v1]
Field
action
string
string
integer
Time
Time
EventSource
eventTime
MicroTime
kind
string
metadata
note
string
reason
string
regarding
related
string
reportingController is the name of the controller that emitted this Event, e.g. `kubernetes.io/kubelet`. This field cannot be empty for new Events.
string
reportingInstance is the ID of the controller instance, e.g. `kubelet-xyzf`. This field cannot be empty for new Events and it can have at most 128 characters.
series
EventSeries
type
string
EventList v1 events
Field
string
items
Event
array
items is a li

st of schema objects.
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
202
Event
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Event
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should

 not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to ret

rieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Event
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is se

rver defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
EventList
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the se

rver will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
EventList
OK
Watch
watch changes to an object of kind Event. deprecated: use the 'watch' parameter wit

h a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/events.k8s.io/v1/watch/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied

 to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Event. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/events.k8s.io/v1/watch/namespaces/{namespace}/events
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large 

result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/events.k8s.io/v1/watch/events
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not suppo

rted if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
LimitRangeList [core/v1]
Field
string
kind
string
metadata
spec
Spec defines the limits enforced. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
LimitRangeSpec v1 core
LimitRange [core/v1]
Field
limits
LimitRangeItem
array
LimitRangeList v1 core
Field
string
items
array
Items is a list of LimitRange objects. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
If 'true', then the output is prett

y printed.
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed 

from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest sna

pshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together wit

h a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind LimitRange. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/limitranges/{name}
Parameter
name
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ig

nored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of LimitRange. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespa

ce}/limitranges
Parameter
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concept

s/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
GET /api/v1/watch/limitranges
Parameter
continue
limit
pretty
resourceVersion sets a constrain

t on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1
v2beta2
v2beta1
HorizontalPodAutoscalerList [autoscaling/v1]
Field
string
kind
string
metadata
spec
behaviour of autoscaler. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status.
status
HorizontalPodAutoscalerSpec v1 autoscaling
HorizontalPodAutoscaler [autoscaling/v1]
Field
integer
integer
minReplicas is the lower limit for the number of replicas to which the autoscaler can scale down.  It defaults to 1 pod.  minReplicas is allowed to be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one Object or External metric is configured.  Scaling is active as long as at least one metric value is available.
CrossVersionObjectReference
integer
HorizontalPodAutoscalerStatus v1 autoscaling
Field
integer
integer
integer
Time
integer
HorizontalPodAutoscalerList v1 autoscaling
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest in

ternal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201

Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value 

is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
allowWatchBookma

rks
continue
limit
watch
Watch for changes to the described resources

 and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list 

calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind HorizontalPodAutoscaler. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/autoscaling/v1/watch/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. 

This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of HorizontalPodAutoscaler. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/autoscaling/v1/watch/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the con

tinue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/autoscaling/v1/watch/horizontalpodautoscalers
Parameter
continue
A selector to restrict the list of returned objects by t

heir labels. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
namespace
object name and auth scope, such as for teams and proj

ects
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
v1beta1
MutatingWebhookConfigurationList [admissionregistration/v1]
Field
string
kind
string
metadata
Standard object metadata; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata.
webhooks
MutatingWebhook
array
:
merge
:
name
MutatingWebhookConfigurationList v1 admissionregistration
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only cont

ain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.fi

nalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first 

list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
A selector to restrict the list 

of returned objects by their fields. Defaults to everything.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind MutatingWebhookConfiguration. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/admissionregistration.k8s.io/v1/watch/mutatingwebhookconfigurations/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the

 client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of MutatingWebhookConfiguration. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/admissionregistration.k8s.io/v1/watch/mutatingwebhookconfigurations
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query resu

lt with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
ValidatingWebhookConfigurationList [admissionregistration/v1]
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reje

ct unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
metadata
webhooks
ValidatingWebhook
array
:
merge
:
name
ValidatingWebhookConfigurationList v1 admissionregistration
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis

/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be incl

uded in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read
read the specified 

Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
resourceVersion sets a constraint on what resource versions a request m

ay be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ValidatingWebhookConfiguration. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/admissionregistration.k8s.io/v1/watch/validatingwebhookconfigurations/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issui

ng a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ValidatingWebhookConfiguration. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/admissionregistration.k8s.io/v1/watch/validatingwebhookconfigurations
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all reques

ted objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
PodTemplateList [core/v1]
Field
string
kind
string
metadata
template
Template defines the pods that will be created from this pod template. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
PodTemplateSpec v1 core
DaemonSetSpec [apps/v1]
JobSpec [batch/v1]
PodTemplate [core/v1]
ReplicaSetSpec [apps/v1]
ReplicationControllerSpec [core/v1]
Field
metadata
spec
PodSpec
PodTemplateList v1 core
Field
string
items
array
kind
string
Kind is a string value representing the 

REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. 

An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the ne

xt set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server d

efined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the serve

r will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PodTemplate. deprecated: use the 'watch' para

meter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/podtemplates/{name}
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is a

pplied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PodTemplate. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/podtemplates
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very lar

ge result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /api/v1/watch/podtemplates
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Group
Version
Kind
policy
v1
v1beta1
PodDisruptionBudgetList [policy/v1]
Field
string
kind
string
metadata
spec
status
Most recently observed status of the PodDisruptionBudget.
PodDisruptionBudgetSpec v1 policy
PodDisruptionBudget [policy/v1]
Field
An eviction is allowed if at least "minAvailable" pods selected by "selector" will still be available after the eviction, i.e. even in the absence of the evicted pod.  So for example you can prevent all voluntary evictions by specifying "100%".
selector
:
replace
PodDisruptionBudgetStatus v1 policy
Field
Condition
array
:
merge
:
type
Conditions contain conditions for PDB. The disruption controller sets the DisruptionAllowed condition. The following are known values for the reason field (additional reasons could be added in the future): - SyncFailed: The controller encountered an err

or and wasn't able to compute               the number of allowed disruptions. Therefore no disruptions are               allowed and the status of the condition will be False. - InsufficientPods: The number of pods are either at or below the number                     required by the PodDisruptionBudget. No disruptions are                     allowed and the status of the condition will be False. - SufficientPods: There are more pods than required by the PodDisruptionBudget.                   The condition will be True, and the number of allowed                   disruptions are provided by the disruptionsAllowed property.
integer
integer
object
integer
integer
integer
PodDisruptionBudgetList v1 policy
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Path P

arameters
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
Descrip

tion
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's 

finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
A selector to restrict the list of returned objects by their labe

ls. Defaults to everything.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not s

upported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PodDisruptionBudget. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/policy/v1/watch/namespaces/{namespace}/poddisruptionbudgets/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expira

tion (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PodDisruptionBudget. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/policy/v1/watch/namespaces/{namespace}/poddisruptionbudgets
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server wil

l send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
watch individual changes to a list of PodDisruptionBudget. deprecated:

 use the 'watch' parameter with a list operation instead.
GET /apis/policy/v1/watch/poddisruptionbudgets
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#re

source-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
v1beta1
v1alpha1
PriorityClassList [scheduling/v1]
Field
string
string
boolean
globalDefault specifies whether this PriorityClass should be considered as the default priority for pods that do not hav

e any priority class. Only one PriorityClass can be marked as `globalDefault`. However, if more than one PriorityClasses exists with their `globalDefault` field set to true, the smallest value of such global default PriorityClasses will be used as the default priority.
kind
string
metadata
string
value
integer
PriorityClassList v1 scheduling
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
Descr

iption
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue

  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/scheduling.k8s.io/v1/prioritycla

sses/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
resourceVersionMatch determines how 

resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PriorityClass. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/scheduling.k8s.io/v1/watch/priorityclasses/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is 

using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PriorityClass. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/scheduling.k8s.io/v1/watch/priorityclasses
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume

 that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
policy
v1beta1
PodSecurityPolicyList [policy/v1beta1]
Field
string
kind
string
metadata
spec
spec defines the policy enforced.
PodSecurityPolicySpec v1beta1 policy
PodSecurityPolicy [policy/v1beta1]
Field
boolean
allowPrivilegeEscalation determines if a pod can request to allow privilege escalation. If unspecified, defaults to true.
AllowedCSIDriver
array
AllowedCSIDrivers is an allowlist of inline CSI drivers that must be explicitly set to be embedded within a pod spec. An empty value indicates that any CSI driver can be used for inline ephemeral volumes. This is a beta field, and is only honored if the API server enables the CSIInlineVolume feature gate.
allowedCapabilities is a list of capabilities that can be requested to add to the container. Capabilities in this field may be added at the pod author's discretion. You must not list a capability in both allowedCapabilities and requiredDropCapabilities.
AllowedFlexVolume
array
allowedFlexVolumes is an allowlist of Flexvolumes.  Empty or nil indicates that all Flexvolumes may be used.  This parameter is effective only when the usage of the Flexvolumes is allowed in the "volumes" field.
AllowedHostPath
array
allowedHostPaths is an allowlist of host paths. Empty indicates that all host paths may be used.
AllowedProcMountTypes is an allowlist of allowed ProcMountTypes. Empty or nil indicates that onl

y the DefaultProcMountType may be used. This requires the ProcMountType feature flag to be enabled.
allowedUnsafeSysctls is a list of explicitly allowed unsafe sysctls, defaults to none. Each entry is either a plain sysctl name or ends in "*" in which case it is considered as a prefix of allowed sysctls. Single * means all unsafe sysctls are allowed. Kubelet has to allowlist all allowed unsafe sysctls explicitly to avoid rejection.  Examples: e.g. "foo/*" allows "foo/bar", "foo/baz", etc. e.g. "foo.*" allows "foo.bar", "foo.baz", etc.
defaultAddCapabilities is the default set of capabilities that will be added to the container unless the pod spec specifically drops the capability.  You may not list a capability in both defaultAddCapabilities and requiredDropCapabilities. Capabilities added here are implicitly allowed, and need not be included in the allowedCapabilities list.
boolean
defaultAllowPrivilegeEscalation controls the default setting for whether a process can gain more privileges than its parent process.
forbiddenSysctls is a list of explicitly forbidden sysctls, defaults to none. Each entry is either a plain sysctl name or ends in "*" in which case it is considered as a prefix of forbidden sysctls. Single * means all sysctls are forbidden.  Examples: e.g. "foo/*" forbids "foo/bar", "foo/baz", etc. e.g. "foo.*" forbids "foo.bar", "foo.baz", etc.
fsGroup
FSGroupStrategyOptions
fsGroup is the strategy that will dictate what fs group is used by the SecurityContext.
hostIPC
boolean
hostIPC determines if the policy allows the use of HostIPC in the pod spec.
boolean
hostNetwork determines if the policy allows the use of HostNetwork in the pod spec.
hostPID
boolean
hostPID determines if the policy allows the use of HostPID in the pod spec.
hostPorts
HostPortRange
array
hostPorts determines which host port ranges are allowed to be exposed.
boolean
privileged determines if a pod can request to be run as privileged.
boolean
readOnlyRootFilesystem when set to true will force containers to run with a read only root file system.  If the container specifically requests to run with a non-read only root file system the PSP should deny the pod. If set to false the container may run with a read only root file system if it wishes but it will not be forced to.
requiredDropCapabilities are the capabilities that will be dropped from the container.  These are required to be dropped and cannot be added.
RunAsGroupStrategyOptions
RunAsGroup is the strategy that will dictate the allowable RunAsGroup values that may be set. If this field is omitted, the pod's RunAsGroup can take any value. This field requires the RunAsGroup feature gate to be enabled.
runAsUser
RunAsUserStrategyOptions
runAsUser is the strategy that will dictate the allowable RunAsUser values that may be set.
RuntimeClassStrategyOptions
runtimeClass is the strategy that will dictate the allowable RuntimeClasses for a pod. If this field is omitted, the pod's runtimeClassName field is unrestricted. Enforcement of this field depends on the RuntimeClass feature gate being enabled.
seLinux
SELinuxStrategyOptions
seLinux is the strategy that will dictate the allowable labels that may be set.
SupplementalGroupsStrategyOptions
supplementalGroups is the strategy that will dictate what supplemental groups are used by the SecurityContext.
volumes
volumes is an allowlist of volume plugins. Empty indicates that no volumes may be used. To allow all volumes you may use '\*'.
PodSecurityPolicyList v1beta1 policy
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized value

s. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
items
array
kind
string
metadata
ListMeta
Create
create a PodSecurityPolicy
POST /apis/policy/v1beta1/podsecuritypolicies
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
partially update the specified PodSecurityPolicy
PATCH /apis/policy/v1beta1/podsecuritypolicies/{name}
Parameter
name
name of the PodSecurityPolicy
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
replace the specified PodSecurityPolicy
PUT /apis/policy/v1beta1/podsecuritypolicies/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
delete a PodSecurityPolicy
DELETE /apis/policy/v1beta1/podsecuritypolicies/{name}
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or 

unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
202
Accepted
delete collection of PodSecurityPolicy
DELETE /apis/policy/v1beta1/podsecuritypolicies
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) 

in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
read the specified PodSecurityPolicy
GET /apis/policy/v1beta1/podsecuritypolicies/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
list or watch objects of kind PodSecurityPolicy
GET /apis/policy/v1beta1/podsecuritypolicies
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whe

ther due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
PodSecurityPolicyList
OK
Watch
watch changes to an object of kind PodSecurityPolicy. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/policy/v1beta1/watch/podsecuritypolicies/{name}
Parameter
name
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMAR

K event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PodSecurityPolicy. deprecated: use the 'watch' parameter with a l

ist operation instead.
GET /apis/policy/v1beta1/watch/podsecuritypolicies
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  D

efaults to unset
watch
Response
Code
200
OK
CLUSTER
Cluster resources are responsible for defining configuration of the cluster itself, and are generally only used by cluster operators.
Group
Version
Kind
v1
v1beta1
APIServiceList [apiregistration/v1]
Field
string
kind
string
metadata
spec
status
APIServiceSpec v1 apiregistration
APIService [apiregistration/v1]
Field
caBundle
string
group
string
integer
GroupPriorityMininum is the priority this group should have at least. Higher priority means that the group is preferred by clients over lower priority ones. Note that other versions of this group might specify even higher GroupPriorityMininum values such that the whole group gets a higher priority. The primary sort is based on GroupPriorityMinimum, ordered highest number to lowest (20 before 10). The secondary sort is based on the alphabetical comparison of the name of the object.  (v1.bar before v1.foo) We'd recommend something like: *.k8s.io (except extensions) at 18000 and PaaSes (OpenShift, Deis) are recommended to be in the 2000s
boolean
InsecureSkipTLSVerify disables TLS certificate verification when communicating with this server. This is strongly discouraged.  You should use the CABundle instead.
service
ServiceReference
Service is a reference to the service for this API server.  It must communicate on port 443. If the Service is nil, that means the handling for the API groupversion is handled locally on this server. The call will simply delegate to the normal handler chain to be fulfilled.
version
string
Version is the API version this server hosts.  For example, "v1"
integer
VersionPriority controls the ordering of this API version inside of its group.  Must be greater than zero. The primary sort is based on VersionPriority, ordered highest to lowest (20 before 10). Since it's inside of a group, the number can be small, probably in the 10s. In case of equal version priorities, the version string will be used to compute the order inside a group. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2

, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10.
APIServiceStatus v1 apiregistration
Field
APIServiceCondition
array
:
merge
:
type
APIServiceList v1 apiregistration
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/apiregistration.k8s.io/v1/a

piservices/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the

 server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value

 from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind APIService. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/apiregistration.k8s.io/v1/watch/apiservices/{name}
Parameter
name
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks 

may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a stream of add, update, and remove

 notifications. Specify resourceVersion.
Response
Code
200
OK
Watch List
watch individual changes to a list of APIService. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/apiregistration.k8s.io/v1/watch/apiservices
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is

 highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
core
v1
Binding
Field
string
kind
string
metadata
target
Create
POST /api

/v1/namespaces/{namespace}/bindings
Parameter
namespace
Parameter
dryRun
pretty
Parameter
body
Binding
Response
Code
200
Binding
OK
201
Binding
Created
202
Binding
Accepted
Group
Version
Kind
v1
v1beta1
CertificateSigningRequestList [certificates/v1]
Field
string
kind
string
metadata
spec
spec contains the certificate request, and is immutable after creation. Only the request, signerName, and usages fields can be set on creation. Other fields are derived by Kubernetes and cannot be modified by users.
status
CertificateSigningRequestSpec v1 certificates
CertificateSigningRequest [certificates/v1]
Field
extra
object
groups
request
string
string
signerName indicates the requested signer, and is a qualified name.  List/watch requests for CertificateSigningRequests can filter on this field using a "spec.signerName=NAME" fieldSelector.  Well-known Kubernetes signers are:  1. "kubernetes.io/kube-apiserver-client": issues client certificates that can be used to authenticate to kube-apiserver.   Requests for this signer are never auto-approved by kube-controller-manager, can be issued by the "csrsigning" controller in kube-controller-manager.  2. "kubernetes.io/kube-apiserver-client-kubelet": issues client certificates that kubelets use to authenticate to kube-apiserver.   Requests for this signer can be auto-approved by the "csrapproving" controller in kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager.  3. "kubernetes.io/kubelet-serving" issues serving certificates that kubelets use to serve TLS endpoints, which kube-apiserver can connect to securely.   Requests for this signer are never auto-approved by kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager.  More details are av

ailable at https://k8s.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers  Custom signerNames can also be specified. The signer defines:  1. Trust distribution: how trust (CA bundles) are distributed.  2. Permitted subjects: and behavior when a disallowed subject is requested.  3. Required, permitted, or forbidden x509 extensions in the request (including whether subjectAltNames are allowed, which types, restrictions on allowed values) and behavior when a disallowed extension is requested.  4. Required, permitted, or forbidden key usages / extended key usages.  5. Expiration/certificate lifetime: whether it is fixed by the signer, configurable by the admin.  6. Whether or not requests for CA certificates are allowed.
uid
string
usages
usages specifies a set of key usages requested in the issued certificate.  Requests for TLS client certificates typically request: "digital signature", "key encipherment", "client auth".  Requests for TLS serving certificates typically request: "key encipherment", "digital signature", "server auth".  Valid values are:  "signing", "digital signature", "content commitment",  "key encipherment", "key agreement", "data encipherment",  "cert sign", "crl sign", "encipher only", "decipher only", "any",  "server auth", "client auth",  "code signing", "email protection", "s/mime",  "ipsec end system", "ipsec tunnel", "ipsec user",  "timestamping", "ocsp signing", "microsoft sgc", "netscape sgc"
username
string
CertificateSigningRequestStatus v1 certificates
Field
string
certificate is populated with an issued certificate by the signer after an Approved condition is present. This field is set via the /status subresource. Once populated, this field is immutable.  If the certificate signing request is denied, a condition of type "Denied" is added and this field remains empty. If the signer cannot issue the certificate, a condition of type "Failed" is added and this field remains empty.  Validation requirements:  1. certificate must contain one or more PEM blocks.  2. All PEM blocks must have the "CERTIFICATE" label, contain no headers, and the encoded data   must be a BER-encoded ASN.1 Certificate structure as described in section 4 of RFC5280.  3. Non-PEM content may appear before or after the "CERTIFICATE" PEM blocks and is unvalidated,   to allow for explanatory text as described in section 5.2 of RFC7468.  If more than one PEM block is present, and the definition of the requested spec.signerName does not indicate otherwise, the first block is the issued certificate, and subsequent blocks should be treated as intermediate certificates and presented in TLS handshakes.  The certificate is encoded in PEM format.  When serialized as JSON or YAML, the data is additionally base64-encoded, so it consists of:      base64(     -----BEGIN CERTIFICATE-----     ...     -----END CERTIFICATE-----     )
CertificateSigningRequestCondition
array
CertificateSigningRequestList v1 certificates
Field
string
items
array
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpo

int the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. V

alue must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the ava

ilable results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue tok

en. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind CertificateSigningRequest. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/certificates.k8s.io/v1/watch/certificatesigningrequests/{name}
Parameter
name
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this fiel

d is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of CertificateSigningRequest. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/certificates.k8s.io/v1/watch/certificatesigningrequests
Parameter
Descripti

on
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the call, regardless of any ac

tivity or inactivity.
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
v1beta1
v1alpha1
ClusterRoleList [rbac/v1]
Field
AggregationRule
string
kind
string
metadata
ObjectM

eta
rules
array
ClusterRoleList v1 rbac
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Qu

ery Parameters
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be

 used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except

 for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ClusterRole. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/rbac.authorization.k8s.io/v1/watch/clusterroles/{name}
Parameter
name
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's di

scretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
Descrip

tion
200
OK
Watch List
watch individual changes to a list of ClusterRole. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/rbac.authorization.k8s.io/v1/watch/clusterroles
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for 

list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
v1alpha1
ClusterRoleBindingList [rbac/v1]
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
ClusterRoleBindingList v1 rbac
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and on

ly contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error 

together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommende

d that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent sna

pshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ClusterRoleBinding. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose n

ot to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ClusterRoleBinding. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/rbac.authorization.k8s.io/v1/watch/clusterrolebindings
Parameter
continue
limit
limit is a maximum number of responses to return 

for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
ComponentStatus v1 core
Group
Version
Kind
core
v1
ComponentStatusList [core/v1]
Field
string
ComponentCondition
array
:
merge
:
type
kind
string
metadata
ComponentStatusList v1 core
Field
string
items
array
kind
string
metadata
ListMeta
Standard list metadata. More i

nfo: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served fro

m. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
FlowSchemaList [flowcontrol/v1beta1]
Field
string
kind
string
metadata
`metadata` is the standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
`spec` is the specification of the desired behavior of a FlowSchema. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
`status` is the current status of a FlowSchema. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
FlowSchemaSpec v1beta1 flowcontrol
FlowSchema [flowcontrol/v1beta1]
Field
FlowDistinguisherMethod
`distinguisherMethod` defines how to compute the flow distinguisher for requests that match this schema. `nil` specifies that the distinguisher is disabled and thus will always be the empty string.
integer
`matchingPrecedence` is used to choose among the FlowSchemas that match a given request. The chosen FlowSchema is among those with the numerically lowest (which we take to be logically highest) MatchingPrecedence.  Each MatchingPrecedence value must be ranged in [1,10000]. Note that if the precedence is not specified, it will be set to 1000 as default.
PriorityLevelConfigurationReference
`priorityLevelConfiguration` should reference a PriorityLevelConfiguration in the cluster. If the reference cannot be resolved, the FlowSchema will be ignored and marked as invalid in its status. Required.
rules
PolicyRulesWithSubjects
array
`rules` describes which requests will match this flow schema. This FlowSchema matches a request if and only if at least one member of rules matches the request. if it is an empty slice, there will be no requests matching the FlowSchema.
FlowSchemaStatus v1beta1 flowcontrol
Field
FlowSchemaCondition
array
`conditions` is a list of the current states of FlowSchema.
FlowSchemaList v1beta1 flowcontrol
Field
string
items
array
`items` is a list of FlowSchemas.
kind
string
Ki

nd is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
`metadata` is the standard list metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
Create
POST /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas/{name}
Parameter
name
Parameter
pretty
dryRun
The duration in seconds before t

he object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return 

all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it mu

st restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind FlowSchema. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/watch/flowschemas/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the

 server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of FlowSchema. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/watch/flowschemas
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not imp

lement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a stream of add, u

pdate, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta1/flowschemas/{name}/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Lease
v1beta1
LeaseList [coordination/v1]
Field
string
kind
string
metadata
spec
LeaseSpec
Specification of the Lease. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
LeaseSpec v1 coordination
Lease [coordination/v1]
Field
MicroTime
string
integer
leaseDurationSeconds 

is a duration that candidates for a lease need to wait to force acquire it. This is measure against time of last observed RenewTime.
integer
renewTime
MicroTime
LeaseList v1 coordination
Field
string
items
Lease
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Lease
Response
Code
200
Lease
OK
201
Lease
Created
202
Lease
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Lease
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun

Parameter
body
Lease
Response
Code
200
Lease
OK
201
Lease
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
fieldSelec

tor
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Lease
OK
List
Parameter
Descripti

on
namespace
Parameter
pretty
continue
limit

t
watch
Response
Code
200
LeaseList
OK
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource

-versions for details.  Defaults to unset
watch
Response
Code
200
LeaseList
OK
Watch
watch changes to an object of kind Lease. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/coordination.k8s.io/v1/watch/namespaces/{namespace}/leases/{name}
Parameter
name
namespace
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Lease. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/coordination.k8s.io/v1/watch/namespaces/{namespace}/leases
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clien

ts should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/coordination.k8s.io/v1/watch/leases
Parameter
continue
A selector to restrict the list of returne

d objects by their labels. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
Field
string
kind
string
metadata
spec
Spec holds information about the request being evaluated.  spec.namespace must be equal to the namespace you made the request against.  If empty, it is defaulted.
status
Create
Parameter
namespace
Parameter
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will 

be processed
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
core
v1
Namespace
NamespaceList [core/v1]
Field
string
kind
string
metadata
spec
Spec defines the behavior of the Namespace. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Status describes the current status of a Namespace. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
NamespaceSpec v1 core
Namespace [core/v1]
Field
Finalizers is an opaque list of values that must be empty to permanently remove object from storage. More info: https://kubernetes.io/docs/tasks/administer-cluster/namespaces/
NamespaceStatus v1 core
Field
NamespaceCondition
array
:
merge
:
type
phase
string
Phase is the current lifecycle phase of the namespace. More info: https://kubernetes.io/docs/tasks/administer-cluster/namespaces/
NamespaceList v1 core
Field
string
items
Namespace
array
Items is the list of Namespace objects in the list. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https:/

/golang.org/pkg/unicode/#IsPrint.
Parameter
body
Namespace
Response
Code
200
Namespace
OK
201
Namespace
Created
202
Namespace
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Namespace
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Namespace
Response
Code
200
Namespace
OK
201
Namespace
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Read
GET 

/api/v1/namespaces/{name}
Parameter
name
Parameter
pretty
Response
Code
200
Namespace
OK
List
Parameter
pretty
continue
limit
resourceVersionMatch determines how resourceVersion is app

lied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind Namespace. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large resu

lt can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Namespace. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  Th

e server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Namespace
OK
Parameter
name
Parameter
pretty
Response
Code
200
Namespace
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Namespace
Response
Code
200
Namespace
OK
201
Namespace
Created
Group
Version
Kind
core
v1
Node
NodeList [core/v1]

Field
string
kind
string
metadata
spec
NodeSpec
Spec defines the behavior of a node. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Most recently observed status of the node. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
NodeSpec v1 core
Node [core/v1]
Field
NodeConfigSource
If specified, the source to get node configuration from The DynamicKubeletConfig feature gate must be enabled for the Kubelet to use this field
string
Deprecated. Not all kubelets will set this field. Remove field after 1.13. see: https://issues.k8s.io/61966
podCIDR
string
podCIDRs
:
merge
string
taints
Taint
array
boolean
Unschedulable controls node schedulability of new pods. By default, node is schedulable. More info: https://kubernetes.io/docs/concepts/nodes/node/#manual-node-administration
NodeStatus v1 core
Field
addresses
NodeAddress
array
:
merge
:
type
List of addresses reachable to the node. Queried from cloud provider, if available. More info: https://kubernetes.io/docs/concepts/nodes/node/#addresses Note: This field is declared as mergeable, but the merge key is not sufficiently unique, which can cause data corruption when it is merged. Callers should instead use a full-replacement patch. See http://pr.k8s.io/79391 for an example.
object
capacity
object
Capacity represents the total resources of a node. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity
array
:
merge
:
type
Conditions is an array of current observed node conditions. More info: https://kubernetes.io/docs/concepts/nodes/node/#condition
config
NodeConfigStatus
NodeDaemonEndpoints
images
ContainerImage
array
nodeInfo
NodeSystemInfo
Set of ids/uuids to uniquely identify the node. More info: https://kubernetes.io/docs/concepts/nodes/node/#info
phase
string
NodePhase is the recently observed lifecycle phase of the node. More info: https://kubernetes.io/docs/concepts/nodes/node/#phase The field is never populated, and now is deprecated.
AttachedVolume
array
NodeList v1 core
Field
Desc

ription
string
items
Node
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Node
Response
Code
200
Node
OK
201
Node
Created
202
Node
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Node
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Node
Response
Code
200
Node
OK
201
Node
Created
Delete
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - A

ll: all dry run stages will be processed
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not t

o support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
Node
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, 

the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
NodeList
OK
Watch
watch changes to an object of kind Node. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/nodes/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with i

dentical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Node. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/nodes
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor

 may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
partially update status of the spe

cified Node
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Node
OK
Parameter
name
Parameter
pretty
Response
Code
200
Node
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Node
Response
Code
200
Node
OK
201
Node
Created
connect POST requests to proxy of Node
POST /api/v1/nodes/{name}/proxy
Parameter
name
name of the NodeProxyOptions
Parameter
path
Path is the URL path to use for the current proxy request to node.
Response
Code
200
string
OK
POST /api/v1/nodes/{name}/proxy/{path}
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
connect DELETE requests to proxy of Node
DELETE /api/v1/nodes/{name}/proxy
Parameter
name
Parameter
path
Response
Code
200
string
OK
DELETE /api/v1/nodes/{name}/proxy/{path}
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
connect GET requests to proxy of Node
GET /api/v1/nodes/{name}/proxy
Parameter
name
Parameter
path
Response
Code
200
string

OK
GET /api/v1/nodes/{name}/proxy/{path}
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
connect HEAD requests to proxy of Node
HEAD /api/v1/nodes/{name}/proxy
Parameter
name
Parameter
path
Response
Code
200
string
OK
HEAD /api/v1/nodes/{name}/proxy/{path}
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
connect PUT requests to proxy of Node
PUT /api/v1/nodes/{name}/proxy
Parameter
name
Parameter
path
Response
Code
200
string
OK
PUT /api/v1/nodes/{name}/proxy/{path}
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
Group
Version
Kind
core
v1
These are assigned to
Pods
using
.
PersistentVolumeList [core/v1]
Field
string
kind
string
metadata
spec
Spec defines a specification of a persistent volume owned by the cluster. Provisioned by an administrator. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
status
Status represents the current information/status for the persistent volume. Populated by the system. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
PersistentVolumeSpec v1 core
PersistentVolume [core/v1]
VolumeAttachmentSource [storage/v1]
VolumeAttachmentSource [storage/v1beta1]
VolumeAttachmentSource [storage/v1alpha1]
Field
AccessModes contains all ways the volume can be mounted. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes
azureDisk
azureFile
AzureFilePersistentVolumeSource
AzureFile represents an Azure File Service m

ount on the host and bind mount to the pod.
capacity
object
A description of the persistent volume's resources and capacity. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity
cephfs
CephFSPersistentVolumeSource
cinder
CinderPersistentVolumeSource
claimRef
ClaimRef is part of a bi-directional binding between PersistentVolume and PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName is the authoritative bind between PV and PVC. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding
csi
CSIPersistentVolumeSource
CSI represents storage that is handled by an external CSI driver (Beta feature).
fc
FlexPersistentVolumeSource
flocker
Flocker represents a Flocker volume attached to a kubelet's host machine and exposed to the pod for its usage. This depends on the Flocker control service being running
GCEPersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
glusterfs
GlusterfsPersistentVolumeSource
Glusterfs represents a Glusterfs volume that is attached to a host and exposed to the pod. Provisioned by an admin. More info: https://examples.k8s.io/volumes/glusterfs/README.md
hostPath
HostPath represents a directory on the host. Provisioned by a developer or tester. This is useful for single-node development and testing only! On-host storage is not supported in any way and WILL NOT WORK in a multi-node cluster. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
iscsi
ISCSIPersistentVolumeSource
ISCSI represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin.
local
LocalVolumeSource
Local represents directly-attached storage with node affinity
A list of mount options, e.g. ["ro", "soft"]. Not validated - mount will simply fail if one is invalid. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options
nfs
NFS represents an NFS mount on the host. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
VolumeNodeAffinity
NodeAffinity defines constraints that limit what nodes this volume can be accessed from. This field influences the scheduling of pods that use this volume.
string
What happens to a persistent volume when released from its claim. Valid options are Retain (default for manually created PersistentVolumes), Delete (default for dynamically provisioned PersistentVolumes), and Recycle (deprecated). Recycle must be supported by the volume plugin underlying this PersistentVolume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#reclaiming
quobyte
rbd
RBDPersistentVolumeSource
RBD represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examp

les.k8s.io/volumes/rbd/README.md
scaleIO
ScaleIOPersistentVolumeSource
string
Name of StorageClass to which this persistent volume belongs. Empty value means that this volume does not belong to any StorageClass.
storageos
StorageOSPersistentVolumeSource
StorageOS represents a StorageOS volume that is attached to the kubelet's host machine and mounted into the pod More info: https://examples.k8s.io/volumes/storageos/README.md
string
PersistentVolumeStatus v1 core
Field
message
string
A human-readable message indicating details about why the volume is in this state.
phase
string
Phase indicates if a volume is available, bound to a claim, or released by a claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#phase
reason
string
Reason is a brief CamelCase string that describes any failure and is meant for machine parsing and tidy display in the CLI.
PersistentVolumeList v1 core
Field
string
items
array
List of persistent volumes. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) b

ut optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will resp

ond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Timeout for the list/watch call. 

This limits the duration of the call, regardless of any activity or inactivity.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit

n
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PersistentVolume. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/persistentvolumes/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single l

ist call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PersistentVolume. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/persistentvolumes
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence o

f the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name a

ssociated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta1
PriorityLevelConfigurationList [flowcontrol/v1beta1]
Field
string
kind
string
metadata
spec
`spec` is the specification of the desired behavior of a "request-priority". More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
`status` is the current status of a "request-priority". More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
PriorityLevelConfigurationSpec v1beta1 flowcontrol
PriorityLevelConfiguration [flowcontrol/v1beta1]
Field
limited
LimitedPriorityLevelConfiguration
`limited` specifies how requests are handled for a Limited priority level. This field must be non-empty if and only if `type` is `"Limited"`.
type
string
`type` indicates whether this priority level is subject to limitation on request execution.  A value of `"Exempt"` means that requests of this priority level are not subject to a limit (and thus are never queued) and do not detract from the capacity made available to other priority levels.  A value of `"Limited"` means that (a) requests of this priority level _are_ subject to limits and (b) some of the server's limited capacity is made available exclusively to this priority level. Required.
PriorityLevelConfigurationStatus v1beta1 flowcontrol
Field
PriorityLevelConfigurationCondition
array
`conditions` is the current state of "request-priority".
PriorityLevelConfigurationList v1beta1 flowcontrol
Field
string
items
array
`items` is a list of request-priorities.
kind
string
metadata
ListMeta
Create
POST /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations
Parameter
pretty
If 'true', then the o

utput is pretty printed.
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be deprecate

d in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no 

objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapsho

t, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PriorityLevelConfiguration. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/watch/prioritylevelconfigurations/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of co

ntinue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PriorityLevelConfiguration. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/watch/prioritylevelconfigurations
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at 

any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
partiall

y update status of the specified PriorityLevelConfiguration
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta1/prioritylevelconfigurations/{name}/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
core
v1
ResourceQuotaList [core/v1]
Field
string
kind
string
metadata
spec
Spec defines the desired quota. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Status defines the actual enforced quota and its current usage. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
ResourceQuotaSpec v1 core
ResourceQuota [core/v1]
Fi

eld
hard
object
hard is the set of desired hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/
ScopeSelector
scopes
ResourceQuotaStatus v1 core
Field
hard
object
Hard is the set of enforced hard limits for each named resource. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/
used
object
ResourceQuotaList v1 core
Field
string
items
array
Items is a list of ResourceQuota objects. More info: https://kubernetes.io/docs/concepts/policy/resource-quotas/
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset fo

r non-apply patch requests.
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue

dryRun
limit
timeout

Seconds
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive

 smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be ide

ntical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ResourceQuota. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/resourcequotas/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be 

used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ResourceQuota. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/resourcequotas
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  

limit
pretty
watch
Response
Code
200
OK
GET /api/v1/watch/resourcequotas
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart t

heir list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. T

his field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Role
v1beta1
v1alpha1
RoleList [rbac/v1]
Field
string
kind
string
metadata
rules
array
RoleList v1 rbac
Field
string
items
Role
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the reques

t. Valid values are: - All: all dry run stages will be processed
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
202
Role
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Role
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, b

ut not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a 

very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Role
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their k

eys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
RoleList
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token 

received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
RoleList
OK
Watch
watch changes to an object of kind Role. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/rbac.authorization.k8s.io/v1/watch/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the ser

ver. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Role. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/rbac.authorization.k8s.io/v1/watch/namespaces/{namespace}/roles
Parameter
namespace
Parameter
a

llowWatchBookmarks
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the call, regardless of any activi

ty or inactivity.
watch
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1/watch/roles
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for det

ails.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
v1alpha1
RoleBindingList [rbac/v1]
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
RoleBindingList v1 rbac
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An inval

id or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty

continue
dryRun
limit
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Accep

table values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items)

 in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
l

imit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind RoleBinding. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/rbac.authorization.k8s.io/v1/watch/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will res

pond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of RoleBinding. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/rbac.authorization.k8s.io/v1/watch/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with 

identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1/watch/rolebindings
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not as

sume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
RuntimeC

lass v1 node.k8s.io
Group
Version
Kind
v1
v1beta1
v1alpha1
RuntimeClassList [node/v1]
Field
string
handler
string
Handler specifies the underlying runtime and configuration that the CRI implementation will use to handle pods of this class. The possible values are specific to the node &amp; CRI configuration.  It is assumed that all handlers are available on every node, and handlers of the same name are equivalent on every node. For example, a handler called "runc" might specify that the runc OCI runtime (using native Linux containers) will be used to run the containers in a pod. The Handler must be lowercase, conform to the DNS Label (RFC 1123) requirements, and is immutable.
kind
string
metadata
overhead
Overhead
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see  https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/ This field is in beta starting v1.18 and is only honored by servers that enable the PodOverhead feature.
RuntimeClassList v1 node
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
If 'true',

 then the output is pretty printed.
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result wit

h identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
re

sourceVersion
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects retu

rned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
watch changes to an object of kind RuntimeClass. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/node.k8s.io/v1/watch/runtimeclasses/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fe

wer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of RuntimeClass. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/node.k8s.io/v1/watch/runtimeclasses
Parameter
continue
A selector to restrict the list of ret

urned objects by their fields. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
Field
string
kind
string
metadata
spec
Spec holds information about the request being evaluated.  user and groups must be empty
status
SelfSubjectAccessReviewSpec v1 authorization
SelfSubjectAccessReview [authorization/v1]
Field
NonResourceAttributes
ResourceAttributes
Create
Query Paramet

ers
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1
v1beta1
Field
string
kind
string
metadata
spec
status
SubjectRulesReviewStatus
SelfSubjectRulesReviewSpec v1 authorization
SelfSubjectRulesReview [authorization/v1]
Field
namespace
string
Create
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
core
v1
ServiceAccountList [core/v1]
Field
string
boolean
array
ImagePullSecrets is a list of references to secrets in the same namespace to use for pulling any images in pods that reference this ServiceAccount. ImagePullSecrets are distinct from Secrets because Secrets can be mounted in the pod, but ImagePullSecrets are only accessed by the kubelet. More info: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
kind
string
Kind is a string value

 representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
secrets
array
:
merge
:
name
Secrets is the list of secrets allowed to be used by pods running using this ServiceAccount. More info: https://kubernetes.io/docs/concepts/configuration/secret
ServiceAccountList v1 core
Field
string
items
array
List of ServiceAccounts. More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Path Paramet

ers
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list r

equest will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read

Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
re

watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a c

onsistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ServiceAccount. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/serviceaccounts/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients shoul

d only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ServiceAccount. deprecated: use the 'watch' parameter with a list operation instead.
GET /api/v1/watch/namespaces/{namespace}/serviceaccounts
Parameter
namespace
Parameter
continue
A selector to restrict the list of returned o

bjects by their fields. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
GET /api/v1/watch/serviceaccounts
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snap

shot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
StorageVersion
StorageVersionList [apiserverinternal/v1alpha1]
Field
string
kind
string
metadata
The name is &lt;group&gt;.&lt;resource&gt;.
spec
StorageVersionSpec
Spec is an empty spec. It is here to comply with Kubernetes API style.
status
StorageVersionStatus
API server instances report the version they can decode and the version they encode objects to when persisting objects in the b

ackend.
StorageVersionSpec v1alpha1 apiserverinternal
StorageVersion [apiserverinternal/v1alpha1]
Field
StorageVersionStatus v1alpha1 apiserverinternal
Field
commonEncodingVersion
string
If all API server instances agree on the same encoding storage version, then this field is set to that version. Otherwise this field is left empty. API servers should finish updating its storageVersionStatus entry before serving write operations, so that this field will be in sync with the reality.
StorageVersionCondition
array
The latest available observations of the storageVersion's state.
storageVersions
ServerStorageVersion
array
The reported versions per API server instance.
StorageVersionList v1alpha1 apiserverinternal
Field
string
items
array
kind
string
metadata
ListMeta
Create
create a StorageVersion
POST /apis/internal.apiserver.k8s.io/v1alpha1/storageversions
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
partially update the specified StorageVersion
PATCH /apis/internal.apiserver.k8s.io/v1alpha1/storageversions/{name}
Parameter
name
name of the StorageVersion
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
replace the specified StorageVersion
PUT /apis/internal.apiserver.k8s.io/v1alpha1/storageversions/{name}
Parameter
name
Parameter
pretty
dryRun

Parameter
body
Response
Code
200
OK
201
Created
Delete
delete a StorageVersion
DELETE /apis/internal.apiserver.k8s.io/v1alpha1/storageversions/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
delete collection of StorageVersion
DELETE /apis/internal.apiserver.k8s.io/v1alpha1/storageversions
Parameter
pretty
continue
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be p

rocessed
limit
Parameter
body
Response
Code
200
Status
OK
Read
read the specified StorageVersion
GET /apis/internal.apiserver.k8s.io/v1alpha1/storageversions/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
list or watch objects of kind StorageVersion
GET /apis/internal.apiserver.k8s.io/v1alpha1/storageversions
Parameter
pretty
If 'tru

e', then the output is pretty printed.
continue
limit
Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactiv

ity.
watch
Response
Code
200
StorageVersionList
OK
Watch
watch changes to an object of kind StorageVersion. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/internal.apiserver.k8s.io/v1alpha1/watch/storageversions/{name}
Parameter
name
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what r

esource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of StorageVersion. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/internal.apiserver.k8s.io/v1alpha1/watch/storageversions
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued req

uests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
partially update status of the specified StorageVersion
PATCH /apis/internal.apiserver.k8s.io/v1alpha1/storageversions/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
read status of the specified StorageVersion
GET /apis/internal.apiserver.k8s.io/v1alpha1/storageversions/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
replace status of the specified StorageVersion
PUT /apis/internal.apiserver.k8s.io/v1alpha1/storageversions/{name}/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
v1beta1
Field
Descripti

on
string
kind
string
metadata
spec
status
SubjectAccessReviewSpec v1 authorization
LocalSubjectAccessReview [authorization/v1]
SubjectAccessReview [authorization/v1]
Field
extra
object
Extra corresponds to the user.Info.GetExtra() method from the authenticator.  Since that is input to the authorizer it needs a reflection here.
groups
uid
string
user
string
SubjectAccessReviewStatus v1 authorization
Field
allowed
boolean
denied
boolean
string
reason
string
Reason is optional.  It indicates why a request was allowed or denied.
Create
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1
v1beta1
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resource

s
kind
string
metadata
spec
status
TokenRequestSpec v1 authentication
TokenRequest [authentication/v1]
Field
audiences
Audiences are the intendend audiences of the token. A recipient of a token must identitfy themself with an identifier in the list of audiences of the token, and otherwise should reject the token. A token issued for multiple audiences may be used to authenticate against any of the audiences listed but implies a high degree of trust between the target audiences.
BoundObjectReference
integer
TokenRequestStatus v1 authentication
Field
Time
token
string
Group
Version
Kind
v1
v1beta1
Field
string
kind
string
metadata
spec
status
TokenReviewSpec v1 authentication
TokenReview [authentication/v1]
Field
audiences
token
string
TokenReviewStatus v1 authentication
Field
audiences
boolean
error
str

ing
user
UserInfo
Create
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1
NetworkPolicyList [networking/v1]
Field
string
kind
string
metadata
spec
NetworkPolicySpec v1 networking
NetworkPolicy [networking/v1]
Field
egress
NetworkPolicyEgressRule
array
ingress
NetworkPolicyIngressRule
array
List of rule types that the NetworkPolicy relates to. Valid options are ["Ingress"], ["Egress"], or ["Ingress", "Egress"]. If this field is not specified, it will default based on the existence of Ingress or Egress rules; policies that contain an Egress section are assumed to affect Egress, and all policies (whether or not they contain an Ingress section) are assumed to affect Ingress. If you

 want to write an egress-only policy, you must explicitly specify policyTypes [ "Egress" ]. Likewise, if you want to write a policy that specifies that no egress is allowed, you must specify a policyTypes value that include "Egress" (since such a policy would not include an Egress section and would otherwise default to just [ "Ingress" ]). This field is beta-level in 1.8
NetworkPolicyList v1 networking
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized 

dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
When present, indicates that modifications 

should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
prett

y
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
resourceVersionMatch determines

 how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the ti

me the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind NetworkPolicy. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/networking.k8s.io/v1/watch/namespaces/{namespace}/networkpolicies/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field

 is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of NetworkPolicy. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/networking.k8s.io/v1/watch/namespaces/{namespace}/networkpolicies
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list

 call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/networking.k8s.io/v1/watch/networkpolicies
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after

 the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
This section contains definitions for objects used in the Kubernetes APIs.
Group
Version
Kind
meta
v1
APIGroup
APIGroup contains the name, the supported versions, and the preferred version of a group.
APIGroupList [meta/v1]
Field
string
kind
string
name
string
name is the name of the group.
preferredVersion
GroupVersionForDiscovery
preferredVersion is the version preferred by the API server, which probably is the storage version.
serverAddressByClientCIDRs
ServerAddressByClientCIDR
array
a map of client CIDR to server address that is serving this group. This is to help clients reach servers in the most network-efficient way possible. Clients can use the appropriate serve

r address as per the CIDR that they match. In case of multiple matches, clients should use the longest matching CIDR. The server returns only those CIDRs that it thinks that the client can match. For example: the master will return an internal IP CIDR only, if the client reaches the server using an internal IP. Server looks at X-Forwarded-For header or X-Real-Ip header or request.RemoteAddr (in that order) to get the client IP.
versions
array
versions are the versions supported in this group.
Group
Version
Kind
meta
v1
APIResource
APIResource specifies the name of a resource and whether it is namespaced.
APIResourceList [meta/v1]
Field
categories is a list of the grouped resources this resource belongs to (e.g. 'all')
group
string
group is the preferred group of the resource.  Empty implies the group of the containing resource list. For subresources, this may have a different value, for example: Scale".
kind
string
kind is the kind for the resource (e.g. 'Foo' is the kind for a resource 'foo')
name
string
name is the plural name of the resource.
boolean
namespaced indicates if a resource is namespaced or not.
shortNames is a list of suggested short names of the resource.
singularName
string
singularName is the singular name of the resource.  This allows clients to handle plural and singular opaquely. The singularName is more correct for reporting status on a single item and both singular and plural are allowed from the kubectl CLI interface.
storageVersionHash
string
The hash value of the storage version, the version this resource is converted to when written to the data store. Value must be treated as opaque by clients. Only equality comparison on the value is valid. This is an alpha feature and may change or be removed in the future. The field is populated by the apiserver only if the StorageVersionHash feature gate is enabled. This field will remain optional even if it graduates.
verbs
verbs is a list of supported kube verbs (this includes get, list, watch, create, update, patch, delete, deletecollection, and proxy)
version
string
version is the preferred version of the resource.  Empty implies the version of the containing resource list For subresources, this may have a different value, for example: v1 (while inside a v1beta1 version of the core resource's group)".
Group
Version
Kind
v1
v1beta1
APIServiceStatus [apiregistration/v1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
meta
v1
APIVersions
APIVersions lists the versions that are available, to allow clients to discover the API at /api, which is the root path of the legacy v1 API.
Field
string
kind
string
S

erverAddressByClientCIDR
array
a map of client CIDR to server address that is serving this group. This is to help clients reach servers in the most network-efficient way possible. Clients can use the appropriate server address as per the CIDR that they match. In case of multiple matches, clients should use the longest matching CIDR. The server returns only those CIDRs that it thinks that the client can match. For example: the master will return an internal IP CIDR only, if the client reaches the server using an internal IP. Server looks at X-Forwarded-For header or X-Real-Ip header or request.RemoteAddr (in that order) to get the client IP.
versions
versions are the api versions that are available.
Group
Version
Kind
core
v1
Represents a Persistent Disk resource in AWS.

An AWS EBS disk must exist before mounting to a container. The disk must also be in the same AWS zone as the kubelet. An AWS EBS disk can only be mounted as read/write once. AWS EBS volumes support ownership management and SELinux relabeling.
PersistentVolumeSpec [core/v1]
Volume [core/v1]
Field
fsType
string
Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
partition
integer
The partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty).
readOnly
boolean
Specify "true" to force and set the ReadOnly property in VolumeMounts to "true". If omitted, the default is "false". More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
volumeID
string
Unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
Group
Version
Kind
core
v1
Affinity
Field
podAffinity
Group
Version
Kind
v1
v1beta1
v1alpha1
ClusterRole [rbac/v1]
Field
array
Group
Version
Kind
policy
v1beta1
AllowedCSIDriver represents a single inline CSI Driver that is allowed to be used.
PodSecurityPolicySpec [policy/v1beta1]
Field
name
string
Name is the registered name of the CSI driver
Group
Version
Kind
policy
v1beta1
AllowedFlexVolume represents a single Flexvolume that is allowed to be used.
Field
driver
string
driver is the name of the Flexvolume driver.
Group
Version
Kind
policy
v1beta1
AllowedHostPath defines the host volume conditions that will be enabled by a policy for pods to use

. It requires the path prefix to be defined.
Field
pathPrefix
string
pathPrefix is the path prefix that the host volume must match. It does not support `*`. Trailing slashes are trimmed when validating the path prefix with a host path.  Examples: `/foo` would allow `/foo`, `/foo/` and `/foo/bar` `/foo` would not allow `/food` or `/etc/foo`
readOnly
boolean
when set to true, will allow host volumes matching the pathPrefix only if all volume mounts are readOnly.
Group
Version
Kind
core
v1
NodeStatus [core/v1]
Field
string
name
string
Group
Version
Kind
core
v1
Field
cachingMode
string
Host Caching mode: None, Read Only, Read Write.
diskName
string
The Name of the data disk in the blob storage
diskURI
string
The URI the data disk in the blob storage
fsType
string
Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified.
kind
string
Expected values Shared: multiple blob disks per storage account  Dedicated: single blob disk per storage account  Managed: azure managed data disk (only in managed availability set). defaults to shared
readOnly
boolean
Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.
Group
Version
Kind
core
v1
Field
readOnly
boolean
string
the name of secret that contains Azure Storage Account Name and Key
string
the namespace of the secret that contains Azure Storage Account Name and Key default is the same as the Pod
shareName
string
Share Name
Group
Version
Kind
core
v1
Field
readOnly
boolean
string
shareName
string
Share Name
Group
Version
Kind
v1
TokenRequestSpec [authentication/v1]
Field
string
kind
string
name
string
uid
string
Group
Version
Kind
v1
v1beta1
CSINodeSpec [storage/v1]
Field
VolumeNodeResources
name
string
nodeID
string
nodeID of the node from the driver point of view. This field enables Kubernetes to communicate with storage systems that do not share the same nomenclature f

or nodes. For example, Kubernetes may refer to a given node as "node1", but the storage system may refer to the same node as "nodeA". When Kubernetes issues a command to the storage system to attach a volume to a specific node, it can use this field to refer to the node name using the ID that the storage system will understand, e.g. "nodeA" instead of "node1". This field is required.
Group
Version
Kind
core
v1
Field
controllerExpandSecretRef
SecretReference
ControllerExpandSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerExpandVolume call. This is an alpha field and requires enabling ExpandCSIVolumes feature gate. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
controllerPublishSecretRef
ControllerPublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI ControllerPublishVolume and ControllerUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
driver
string
Driver is the name of the driver to use for this volume. Required.
fsType
string
Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs".
nodePublishSecretRef
NodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
nodeStageSecretRef
NodeStageSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeStageVolume and NodeStageVolume and NodeUnstageVolume calls. This field is optional, and may be empty if no secret is required. If the secret object contains more than one secret, all secrets are passed.
readOnly
boolean
Optional: The value to pass to ControllerPublishVolumeRequest. Defaults to false (read/write).
volumeAttributes
object
Attributes of the volume to publish.
volumeHandle
string
VolumeHandle is the unique volume name returned by the CSI volume plugin’s CreateVolume to refer to the volume on all subsequent calls. Required.
Group
Version
Kind
core
v1
Field
driver
string
Driver is the name of the CSI driver that handles this volume. Consult with your admin for the correct name as registered in the cluster.
fsType
string
Filesystem type to mount. Ex. "ext4", "xfs", "ntfs". If not provided, the empty value is passed to the associated CSI driver which will determine the default filesystem to apply.
LocalObjectRefe

rence
NodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and  may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed.
readOnly
boolean
Specifies a read-only configuration for the volume. Defaults to false (read/write).
object
VolumeAttributes stores driver-specific properties that are passed to the CSI driver. Consult your driver's documentation for supported values.
Group
Version
Kind
core
v1
Capabilities
SecurityContext [core/v1]
Field
add
drop
Group
Version
Kind
core
v1
Field
monitors
Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
path
string
Optional: Used as the mounted root, rather than the full Ceph tree, default is /
readOnly
boolean
Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
secretFile
string
Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
secretRef
Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
user
string
Optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
Group
Version
Kind
core
v1
Field
monitors
path
string
readOnly
boolean
string
secretRef
user
string
Group
Version
Kind
v1
v1beta1
CertificateSigningRequestStatus [certificates/v1]
Field
Time
Time
lastUpdateTime is the time of the last up

date to this condition
message
string
reason
string
status
string
type
string
type of the condition. Known conditions are "Approved", "Denied", and "Failed".  An "Approved" condition is added via the /approval subresource, indicating the request was approved and should be issued by the signer.  A "Denied" condition is added via the /approval subresource, indicating the request was denied and should not be issued by the signer.  A "Failed" condition is added via the /status subresource, indicating the signer failed to issue the certificate.  Approved and Denied conditions are mutually exclusive. Approved, Denied, and Failed conditions cannot be removed once added.  Only one condition of a given type is allowed.
Group
Version
Kind
core
v1
Field
fsType
string
Filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
readOnly
boolean
Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
secretRef
Optional: points to a secret object containing parameters used to connect to OpenStack.
volumeID
string
volume id used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
Group
Version
Kind
core
v1
Field
fsType
string
readOnly
boolean
secretRef
volumeID
string
Group
Version
Kind
core
v1
ClientIPConfig
SessionAffinityConfig [core/v1]
Field
integer
Group
Version
Kind
core
v1
ComponentStatus [core/v1]
Field
error
string
message
string
status
string
t

ype
string
Group
Version
Kind
meta
v1
Condition
PodDisruptionBudgetStatus [policy/v1]
PodDisruptionBudgetStatus [policy/v1beta1]
ServiceStatus [core/v1]
Field
Time
lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed.  If that is not known, then using the time when the API field changed is acceptable.
message
string
integer
reason
string
status
string
type
string
ConfigMapEnvSource v1 core
Group
Version
Kind
core
v1
ConfigMapEnvSource
ConfigMapEnvSource selects a ConfigMap to populate the environment variables with.

The contents of the target ConfigMap's Data field will represent the key-value pairs as environment variables.
EnvFromSource [core/v1]
Field
name
string
Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
optional
boolean
Group
Version
Kind
core
v1
ConfigMapKeySelector
EnvVarSource [core/v1]
Field
key
string
name
string
optional
boolean
Group
Version
Kind
core
v1
ConfigMapNodeConfigSource
ConfigMapNodeConfigSource contains the information to reference a ConfigMap as a config source for the Node.
NodeConfigSource [core/v1]
Field
kubeletConfigKey
string
name
string
namespace
string
string
uid
string
Group
Version
Kind
core
v1
ConfigMapProjection
Adapts a ConfigMap into a projected volume.

The contents of the target ConfigMap's Data field will be presented in a projected volume as files using the keys in the Data field as the file names, unless the items element is populated with specific mappings of keys to paths. Note that this is identical to a configmap volume source without the default mode.
VolumeProjection [core/v1]
Field
items
KeyToPath
array
If unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose 

name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
name
string
optional
boolean
Specify whether the ConfigMap or its keys must be defined
Group
Version
Kind
core
v1
Adapts a ConfigMap into a volume.

The contents of the target ConfigMap's Data field will be presented in a volume as files using the keys in the Data field as the file names, unless the items element is populated with specific mappings of keys to paths. ConfigMap volumes support ownership management and SELinux relabeling.
Field
integer
Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
items
KeyToPath
array
If unspecified, each key-value pair in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
name
string
optional
boolean
Group
Version
Kind
core
v1
Field
names
Names by which this image is known. e.g. ["k8s.gcr.io/hyperkube:v1.0.7", "dockerhub.io/google_containers/hyperkube:v1.0.7"]
sizeBytes
integer
Group
Version
Kind
core
v1
Container [core/v1]
EphemeralContainer [core/v1]
Field
integer
hostIP
string
hostPort
integer
name
string
protocol
string
Group
Version
Kind
v2beta2
ContainerResourceMetricSource
ContainerResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory).  The values will be averaged together before being compared to the target.  Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.  Only one "target" type should be set.
Other API versions of 

this object exist:
v2beta1
MetricSpec [autoscaling/v2beta2]
Field
container
string
name
string
target
MetricTarget
Group
Version
Kind
v2beta2
ContainerResourceMetricStatus
ContainerResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing a single container in each pod in the current scale target (e.g. CPU or memory).  Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
v2beta1
MetricStatus [autoscaling/v2beta2]
Field
container
string
current
MetricValueStatus
name
string
Group
Version
Kind
core
v1
ContainerStatus [core/v1]
Field
running
ContainerStateRunning
ContainerStateTerminated
waiting
ContainerStateWaiting
Group
Version
Kind
core
v1
ContainerState [core/v1]
Field
startedAt
Time
Group
Version
Kind
core
v1
Field
string
Container's ID in the format 'docker://&lt;container_id&gt;'
exitCode
integer
Time
message
string
reason
string
signal
integer
startedAt
Time
Group
Version
Kind
core
v1
Field
message
string
reason
string
Group
Version
Kind
v1
v2beta2
v2beta1
HorizontalPodAutoscalerSpec [autoscaling/v1]
Field
string
kind
string
Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"
name
string
Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names
Group
Version
Kind
v1
CustomResourceColumnDefinition
v1beta1
CustomResourceDefinitionVersion [apiextensions/v1]
Field
string
format
string
format is an optional Open

API type definition for this column. The 'name' format is applied to the primary identifier column to assist in clients identifying column is the resource name. See https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types for details.
jsonPath
string
name
string
priority
integer
type
string
type is an OpenAPI type definition for this column. See https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types for details.
Group
Version
Kind
v1
v1beta1
CustomResourceDefinitionSpec [apiextensions/v1]
Field
strategy
string
strategy specifies how custom resources are converted between versions. Allowed values are: - `None`: The converter only change the apiVersion and would not touch any other field in the custom resource. - `Webhook`: API Server will call to an external webhook to do the conversion. Additional information   is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhook to be set.
webhook
WebhookConversion
webhook describes how to call the conversion webhook. Required when `strategy` is set to `Webhook`.
Group
Version
Kind
v1
v1beta1
CustomResourceDefinitionStatus [apiextensions/v1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1
v1beta1
Field
categories is a list of grouped resources this custom resource belongs to (e.g. 'all'). This is published in API discovery documents, and used by clients to support invocations like `kubectl get all`.
kind
string
kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as the `kind` attribute in API calls.
listKind
string
listKind is the serialized kind of the list for this resource. Defaults to "`kind`List".
plural
string
plural is the plural name of the resource to serve. The custom resources are served under `/apis/&lt;group&gt;/&lt;version&gt;/.../&lt;plural&gt;`. Must match the name of the CustomResourceDefinition (in the form `&lt;names.plural&gt;.&lt;group&gt;`). Must be all lowercase.
shortNames are short names for the resource, exposed in API discovery documents, and used by clients to support invocations like `kubectl get &lt;shortname&

gt;`. It must be all lowercase.
singular
string
singular is the singular name of the resource. It must be all lowercase. Defaults to lowercased `kind`.
Group
Version
Kind
v1
v1beta1
Field
array
additionalPrinterColumns specifies additional columns returned in Table output. See https://kubernetes.io/docs/reference/using-api/api-concepts/#receiving-resources-as-tables for details. If no columns are specified, a single column displaying the age of the custom resource is used.
boolean
string
deprecationWarning overrides the default warning returned to API clients. May only be set when `deprecated` is true. The default warning indicates this version is deprecated and recommends use of the newest served version of equal or greater stability, if one exists.
name
string
name is the version name, e.g. “v1”, “v2beta1”, etc. The custom resources are served under this version at `/apis/&lt;group&gt;/&lt;version&gt;/...` if `served` is true.
schema
served
boolean
storage
boolean
Group
Version
Kind
v1
CustomResourceSubresourceScale
v1beta1
CustomResourceSubresources [apiextensions/v1]
Field
string
labelSelectorPath defines the JSON path inside of a custom resource that corresponds to Scale `status.selector`. Only JSON paths without the array notation are allowed. Must be a JSON Path under `.status` or `.spec`. Must be set to work with HorizontalPodAutoscaler. The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form. More info: https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions#scale-subresource If there is no value under the given path in the custom resource, the `status.selector` value in the `/scale` subresource will default to the empty string.
string
specReplicasPath defines the JSON path inside of a custom resource that corresponds to Scale `spec.replicas`. Only JSON paths without the array notation are allowed. Must be a JSON Path under `.spec`. If there is no value under the given path in the custom resource, the `/scale` subresource will return an error on GET.
string
statusReplicasPath defines the JSON path inside of a custom resource that corresponds to Scale `status.replicas`. Only JSON paths without the array notation are allowed. Must be a JSON Path under `.status`. If there is no value under the given path in the custom resource, the `status.replicas` value in the `/scale` subresource will default to 0.
Group
Version
Kind
v1
CustomResourceSubresourceStatus
CustomResourceSubresourceStatus defines h

ow to serve the status subresource for CustomResources. Status is represented by the `.status` JSON path inside of a CustomResource. When set, * exposes a /status subresource for the custom resource * PUT requests to the /status subresource take a custom resource object, and ignore changes to anything except the status stanza * PUT/POST/PATCH requests to the custom resource ignore changes to the status stanza
v1beta1
Field
Group
Version
Kind
v1
v1beta1
Field
scale
scale indicates the custom resource should serve a `/scale` subresource that returns an `autoscaling/v1` Scale object.
status
status indicates the custom resource should serve a `/status` subresource. When enabled: 1. requests to the custom resource primary endpoint ignore changes to the `status` stanza of the object. 2. requests to the custom resource `/status` subresource ignore changes to anything other than the `status` stanza of the object.
Group
Version
Kind
v1
v1beta1
Field
Group
Version
Kind
core
v1
DaemonEndpoint
NodeDaemonEndpoints [core/v1]
Field
Port
integer
Group
Version
Kind
apps
v1
DaemonSetStatus [apps/v1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
apps
v1
Field
RollingUpdateDaemonSet
type
string
Group
Version
Kind
meta
v1
Eviction [policy/v1beta1]
Field
string
dryRun
integer
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, th

e default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
kind
string
boolean
Preconditions
string
Group
Version
Kind
apps
v1
DeploymentStatus [apps/v1]
Field
Time
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
DownwardAPIProjection
Field
items
array
Group
Version
Kind
core
v1
DownwardAPIProjection [core/v1]
DownwardAPIVolumeSource [core/v1]
Field
fieldRef
mode
integer
path
string
Required: Path is  the relative path name of the file to be created. Must not be absolute or contain the '..' path. Must be utf-8 encoded. The first item of the relative path must not start with '..'
Group
Version
Kind
core
v1
Field
integer
Optional: mode bits to use on created files by default. Must be a Optional: mode bits used to set permissi

ons on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
items
array
Group
Version
Kind
core
v1
Field
medium
string
What type of storage medium should back this directory. The default is "" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
sizeLimit
Quantity
Total amount of local storage required for this EmptyDir volume. The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir
Group
Version
Kind
v1
Endpoint
v1beta1
EndpointSlice [discovery/v1]
Field
addresses
addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100.
EndpointConditions
object
deprecatedTopology contains topology information part of the v1beta1 API. This field is deprecated, and will be removed when the v1beta1 API is removed (no sooner than kubernetes v1.24).  While this field can hold values, it is not writable through the v1 API, and any attempts to write to it will be silently ignored. Topology information can be found in the zone and nodeName fields instead.
hints
EndpointHints
hostname
string
nodeName
string
nodeName represents the name of the Node hosting this endpoint. This can be used to determine endpoints local to a Node. This field can be enabled with the EndpointSliceNodeName feature gate.
targetRef
zone
string
Group
Version
Kind
core
v1
EndpointAddress
EndpointSubset [core/v1]
Field
hostname
string
ip
string
nodeName
string
targetRef
Endpoin

tConditions v1 discovery.k8s.io
Group
Version
Kind
v1
v1beta1
Endpoint [discovery/v1]
Field
ready
boolean
serving
boolean
serving is identical to ready except that it is set regardless of the terminating state of endpoints. This condition should be set to true for a ready endpoint that is terminating. If nil, consumers should defer to the ready condition. This field can be enabled with the EndpointSliceTerminatingCondition feature gate.
boolean
terminating indicates that this endpoint is terminating. A nil value indicates an unknown state. Consumers should interpret this unknown state to mean that the endpoint is not terminating. This field can be enabled with the EndpointSliceTerminatingCondition feature gate.
Group
Version
Kind
v1
v1beta1
Field
forZones
ForZone
array
Group
Version
Kind
core
v1
v1beta1
Field
string
The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol. This is a beta field that is guarded by the ServiceAppProtocol feature gate and enabled by default.
name
string
The name of this port.  This must match the 'name' field in the corresponding ServicePort. Must be a DNS_LABEL. Optional only if one port is defined.
port
integer
protocol
string
Group
Version
Kind
core
v1
EndpointSubset is a group of addresses with a common set of ports. The expanded set of endpoints is the Cartesian product of Addresses x Ports. For example, given:
  {
    Addresses: [{"ip": "10.10.1.1"}, {"ip": "10.10.2.2"}],
    Ports:     [{"name": "a", "port": 8675}, {"name": "b", "port": 309}]
  }
    a: [ 10.10.1.1:8675, 10.10.2.2:8675 ],
    b: [ 10.10.1.1:309, 10.10.2.2:309 ]
Endpoints [core/v1]
Field
addresses
array
notReadyAddresses
array
ports
array
Group
Version
Kind
core
v1
Field
prefix
string
secretRef
Secre

tEnvSource
Group
Version
Kind
core
v1
EnvVar
Field
name
string
value
string
Variable references $(VAR_NAME) are expanded using the previous defined environment variables in the container and any service environment variables. If a variable cannot be resolved, the reference in the input string will be unchanged. The $(VAR_NAME) syntax can be escaped with a double $$, ie: $$(VAR_NAME). Escaped references will never be expanded, regardless of whether the variable exists or not. Defaults to "".
valueFrom
Group
Version
Kind
core
v1
EnvVar [core/v1]
Field
fieldRef
Selects a field of the pod: supports metadata.name, metadata.namespace, `metadata.labels['&lt;KEY&gt;']`, `metadata.annotations['&lt;KEY&gt;']`, spec.nodeName, spec.serviceAccountName, status.hostIP, status.podIP, status.podIPs.
SecretKeySelector
Group
Version
Kind
core
v1
An EphemeralContainer is a container that may be added temporarily to an existing pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a pod is removed or restarted. If an ephemeral container causes a pod to exceed its resource allocation, the pod may be evicted. Ephemeral containers may not be added by directly updating the pod spec. They must be added via the pod's ephemeralcontainers subresource, and they will appear in the pod spec once added. This is an alpha feature enabled by the EphemeralContainers feature flag.
EphemeralContainers [core/v1]
Field
args
command
env
EnvVar
array
:
merge
:
name
envFrom
array
List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event

 when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be updated.
image
string
Docker image name. More info: https://kubernetes.io/docs/concepts/containers/images
string
lifecycle
Lifecycle
Probe
name
string
ports
array
Probe
resources
SecurityContext is not allowed for ephemeral containers.
Probe
stdin
boolean
stdinOnce
boolean
string
If set, the name of the container from PodSpec that this ephemeral container targets. The ephemeral container will be run in the namespaces (IPC, PID, etc) of this container. If not set then the ephemeral container is run in whatever namespaces are shared for the pod. Note that the container runtime must support this feature.
string
string
tty
boolean
array
:
merge
:
array
:
merge
:
mountPath
string
Container's working directory. If not specified, the container runtime's default will be used, which might be configured in the container image. Cannot be updated

.
Group
Version
Kind
core
v1
A list of ephemeral containers used with the Pod ephemeralcontainers subresource.
Field
string
array
:
merge
:
name
A list of ephemeral containers associated with this pod. New ephemeral containers may be appended to this list, but existing ephemeral containers may not be removed or modified.
kind
string
metadata
Group
Version
Kind
core
v1
Field
volumeClaimTemplate
PersistentVolumeClaimTemplate
Will be used to create a stand-alone PVC to provision the volume. The pod in which this EphemeralVolumeSource is embedded will be the owner of the PVC, i.e. the PVC will be deleted together with the pod.  The name of the PVC will be `&lt;pod name&gt;-&lt;volume name&gt;` where `&lt;volume name&gt;` is the name from the `PodSpec.Volumes` array entry. Pod validation will reject the pod if the concatenated name is not valid for a PVC (for example, too long).  An existing PVC with that name that is not owned by the pod will *not* be used for the pod to avoid using an unrelated volume by mistake. Starting the pod is then blocked until the unrelated PVC is removed. If such a pre-created PVC is meant to be used by the pod, the PVC has to updated with an owner reference to the pod once the pod exists. Normally this should not be necessary, but it may be useful when manually reconstructing a broken cluster.  This field is read-only and no changes will be made by Kubernetes to the PVC after it has been created.  Required, must not be nil.
Group
Version
Kind
v1
v1beta1
Event [events/v1]
Field
count
integer
lastObservedTime
MicroTime
Group
Version
Kind
core
v1
Event [core/v1]
Event [events/v1beta1]
Field
component
string
host
string
Group
Version
Kind
policy
v1beta1
Eviction
Eviction evicts a pod from its node subject to certain policies and safety constraints. This is a subresource of Pod.  A request to cause such an eviction is created by POSTing to .../pods/&lt;pod name&gt;/evictions.
Field
string

deleteOptions
DeleteOptions may be provided
kind
string
metadata
ObjectMeta describes the pod that is being evicted.
Group
Version
Kind
core
v1
ExecAction
Handler [core/v1]
Probe [core/v1]
Field
command
Command is the command line to execute inside the container, the working directory for the command  is root ('/') in the container's filesystem. The command is simply exec'd, it is not run inside a shell, so traditional shell instructions ('|', etc) won't work. To use a shell, you need to explicitly call out to that shell. Exit status of 0 is treated as live/healthy and non-zero is unhealthy.
Group
Version
Kind
v1
ExternalDocumentation
v1beta1
JSONSchemaProps [apiextensions/v1]
Field
string
url
string
Group
Version
Kind
v2beta2
ExternalMetricSource
v2beta1
Field
metric
MetricIdentifier
target
Group
Version
Kind
v2beta2
ExternalMetricStatus
v2beta1
Field
current
metric
Group
Version
Kind
core
v1
Field
fsType
string
lun
integer
Optional: FC target lun number
readOnly
boolean
Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts.
targetWWNs
Optional: FC target worldwide names (WWNs)
wwids
Optional: FC volume world wide identifiers (wwids) Either wwids or combination of targetWWNs and lun must be set, but not both simultaneously.
Group
Version
Kind
policy
v1beta1
FSGroupStrategyOptions defines the strategy type and options used to create the strategy.
Field
ranges
IDRange
array
ranges are the allowed ranges of fs groups.  If you would like to force a single fs group then supply a single range with the same start and end. Required for MustRunAs.
rule
string
rule is the strategy that will dictate what FSGroup is used in the SecurityContext.
Group
Version
Kind
meta
v1
FieldsV1
FieldsV1 stores a set of fields in a da

ta structure like a Trie, in JSON format.

Each key is either a '.' representing the field itself, and will always map to an empty set, or a string representing a sub-field or item. The string will follow one of these four formats: 'f:&lt;name&gt;', where &lt;name&gt; is the name of a field in a struct, or key in a map 'v:&lt;value&gt;', where &lt;value&gt; is the exact json formatted value of a list item 'i:&lt;index&gt;', where &lt;index&gt; is position of a item in a list 'k:&lt;keys&gt;', where &lt;keys&gt; is a map of  a list item's key fields to their unique values If a key maps to an empty Fields value, the field that key represents is part of the set.

The exact format is defined in sigs.k8s.io/structured-merge-diff
ManagedFieldsEntry [meta/v1]
Field
Group
Version
Kind
core
v1
Field
driver
string
Driver is the name of the driver to use for this volume.
fsType
string
Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default filesystem depends on FlexVolume script.
options
object
Optional: Extra command options if any.
readOnly
boolean
secretRef
Optional: SecretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets are passed to the plugin scripts.
Group
Version
Kind
core
v1
Field
driver
string
fsType
string
options
object
readOnly
boolean
secretRef
Group
Version
Kind
core
v1
Field
datasetName
string
Name of the dataset stored as metadata -&gt; name on the dataset for Flocker should be considered as deprecated
datasetUUID
string
UUID of the dataset. This is unique identifier of a Flocker dataset
Group
Version
Kind
v1beta1
FlowSchemaSpec [flowcontrol/v1beta1]
Field
type
string
`type` is the type of flow distinguisher method The supported types are "ByUser" and "ByNamespace". Required.
Group
Version
Kind
v1beta1
FlowSchemaStatus [flowcontrol/v1beta1]
Field
lastTransitionT

ime
Time
`lastTransitionTime` is the last time the condition transitioned from one status to another.
message
string
`message` is a human-readable message indicating details about last transition.
reason
string
`reason` is a unique, one-word, CamelCase reason for the condition's last transition.
status
string
`status` is the status of the condition. Can be True, False, Unknown. Required.
type
string
`type` is the type of the condition. Required.
Group
Version
Kind
v1
ForZone
v1beta1
EndpointHints [discovery/v1]
Field
name
string
Group
Version
Kind
core
v1
Represents a Persistent Disk resource in Google Compute Engine.

A GCE PD must exist before mounting to a container. The disk must also be in the same GCE project and zone as the kubelet. A GCE PD can only be mounted as read/write once or read-only many times. GCE PDs support ownership management and SELinux relabeling.
Field
fsType
string
Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
partition
integer
The partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
pdName
string
Unique name of the PD resource in GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
readOnly
boolean
ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
Group
Version
Kind
core
v1
Represents a volume that is populated with the contents of a git repository. Git repo volumes do not support ownership management. Git repo volumes support SELinux relabeling.

DEPRECATED: GitRepo is deprecated. To provision a container with a git repo, mount an EmptyDir into an InitContainer that clones the repo using git, then mount the EmptyDir into the Pod's container.
Field
directory
string
Target directory name. Must not contain or start with '..'.  If '.' is supplied, the volume directory will be the git repository.  Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name.
repository
string
Repository URL
revision
string
Commit hash for the specified revision.
Group
Version
Kind
core
v1
Represents a Glusterfs mount that lasts the lifetime of a pod. Glusterfs volumes do not support ownership management or SELinux relabeling.
Field
endpoints
string
EndpointsName is the endpoint name that details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
endpointsNamespace
string
EndpointsNamespace is the namespace that contains Glusterfs endpoint. If this field is empty, the EndpointNamespace defaults to the same namespace as the bound PVC. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
path
string
Path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
readOnly
boolean
ReadOnly here 

will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
Group
Version
Kind
core
v1
Field
endpoints
string
path
string
readOnly
boolean
ReadOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
Group
Version
Kind
v1beta1
GroupSubject
Subject [flowcontrol/v1beta1]
Field
name
string
name is the user group that matches, or "*" to match all user groups. See https://github.com/kubernetes/apiserver/blob/master/pkg/authentication/user/user.go for some well-known group names. Required.
Group
Version
Kind
meta
v1
GroupVersion contains the "group/version" and "version" string of a version. It is made a struct to keep extensibility.
APIGroup [meta/v1]
Field
groupVersion
string
groupVersion specifies the API group and version in the form "group/version"
version
string
version specifies the version in the form of "version". This is to save the clients the trouble of splitting the GroupVersion.
Group
Version
Kind
v2beta2
HPAScalingPolicy
HPAScalingRules [autoscaling/v2beta2]
Field
integer
type
string
value
integer
Group
Version
Kind
v2beta2
HPAScalingRules
HorizontalPodAutoscalerBehavior [autoscaling/v2beta2]
Field
policies
array
string
selectPolicy is used to specify which policy should be used. If not set, the default value MaxPolicySelect is used.
integer
Group
Version
Kind
core
v1
HTTPGetAction
Pro

be [core/v1]
Field
host
string
HTTPHeader
array
path
string
port
scheme
string
Group
Version
Kind
core
v1
HTTPGetAction [core/v1]
Field
name
string
value
string
Group
Version
Kind
v1
HTTPIngressPath
v1beta1
v1beta1
HTTPIngressRuleValue [networking/v1]
Field
backend
path
string
Path is matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/'. When unspecified, all paths from incoming requests are matched.
pathType
string
PathType determines the interpretation of the Path matching. PathType can be one of the following values: * Exact: Matches the URL path exactly. * Prefix: Matches based on a URL path prefix split by '/'. Matching is   done on a path element by element basis. A path element refers is the   list of labels in the path split by the '/' separator. A request is a   match for path p if every p is an element-wise prefix of p of the   request path. Note that if the last element of the path is a substring   of the last element in request path, it is not a match (e.g. /foo/bar   matches /foo/bar/baz, but does not match /foo/barbaz). * ImplementationSpecific: Interpretation of the Path matching is up to   the IngressClass. Implementations can treat this as a separate PathType   or treat it identically to Prefix or Exact path types. Implementations are required to support all path types.
Group
Version
Kind
v1
HTTPIngressRuleValue
HTTPIngressRuleValue is a list of http selectors pointing to backends. In the example: http://&lt;host&gt;/&lt;path&gt;?&lt;searchpart&gt; -&gt; backend where where parts of the url correspond to RFC 3986, this resource will be used to match against everything after the last '/' and before the first '?' or '#'.
v1beta1
v1beta1
IngressRule [networking/v1]
Field
paths
array
Group
Version
Kind
core
v1
Handler
Handler defines a specific action that should be taken
Lifecycle [core/v1]
Field
exec
One and only one of the following should be specified. Exec specifies the action to take.
httpGet
tcpSocket
TCPSocketAction
TCPSocket specifies an action involving a TCP port. TCP hooks not yet supported
Group
Version
Kind
v2beta2
HorizontalPodAutoscalerBehavior
HorizontalPodAutoscalerSpec [autoscaling/v2beta2]
Field
scaleDown
scaleDown is scaling policy for scaling Down. If not set, the default value is to allow to scale down to minReplicas pods, with a 300 second stabilization window (i.e., the highest recommendation for the 

last 300sec is used).
scaleUp
scaleUp is scaling policy for scaling Up. If not set, the default value is the higher of:   * increase no more than 4 pods per 60 seconds   * double the number of pods per 60 seconds No stabilization is used.
Group
Version
Kind
v2beta2
HorizontalPodAutoscalerCondition
v2beta1
HorizontalPodAutoscalerStatus [autoscaling/v2beta2]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
HostAlias
Field
hostnames
ip
string
Group
Version
Kind
core
v1
Field
path
string
Path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
type
string
Type for HostPath Volume Defaults to "" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
Group
Version
Kind
policy
v1beta1
HostPortRange defines a range of host ports that will be enabled by a policy for pods to use.  It requires both the start and end to be defined.
Field
max
integer
max is the end of the range, inclusive.
min
integer
min is the start of the range, inclusive.
Group
Version
Kind
policy
v1beta1
IDRange
IDRange provides a min/max of an allowed range of IDs.
FSGroupStrategyOptions [policy/v1beta1]
RunAsGroupStrategyOptions [policy/v1beta1]
RunAsUserStrategyOptions [policy/v1beta1]
SupplementalGroupsStrategyOptions [policy/v1beta1]
Field
max
integer
min
integer
Group
Version
Kind
v1
IPBlock
IPBlock describes a particular CIDR (Ex. "192.168.1.1/24","2001:db9::/64") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should not be included within this rule.
NetworkPolicyPeer [networking/v1]
Field
cidr
string
CIDR is a string representing the IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64"
except
Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.1/24" or "2001:db9::/64" Except values will be rejected if they are outside the CIDR range
Group
Version
Kind
core
v1
Field
chapAuthDiscovery
boolean
whether support iSCSI Discovery CHAP authentication
chapAuthSession
boolean
whether support iSCSI Session CHAP authentication
fsType
string
Filesystem type of the volume that you want to mount. Tip: Ens

ure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi
initiatorName
string
Custom iSCSI Initiator Name. If initiatorName is specified with iscsiInterface simultaneously, new iSCSI interface &lt;target portal&gt;:&lt;volume name&gt; will be created for the connection.
iqn
string
Target iSCSI Qualified Name.
iscsiInterface
string
iSCSI Interface Name that uses an iSCSI transport. Defaults to 'default' (tcp).
lun
integer
iSCSI Target Lun number.
portals
iSCSI Target Portal List. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).
readOnly
boolean
ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false.
secretRef
CHAP Secret for iSCSI target and initiator authentication
targetPortal
string
iSCSI Target Portal. The Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).
Group
Version
Kind
core
v1
Field
boolean
boolean
fsType
string
Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi
string
iqn
string
string
lun
integer
portals
iSCSI Target Portal List. The portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).
readOnly
boolean
secretRef
string
Group
Version
Kind
v1
v1beta1
v1beta1
HTTPIngressPath [networking/v1]
IngressSpec [networking/v1]
Field
resource
service
IngressServiceBackend
Group
Version
Kind
v1
v1beta1
IngressClassSpec [networking/v1]
Field
apiGroup
string
kind
s

tring
name
string
namespace
string
scope
string
Scope represents if this refers to a cluster or namespace scoped resource. This may be set to "Cluster" (default) or "Namespace". Field can be enabled with IngressClassNamespacedParams feature gate.
Group
Version
Kind
v1
v1beta1
v1beta1
Field
host
string
Host is the fully qualified domain name of a network host, as defined by RFC 3986. Note the following deviations from the "host" part of the URI as defined in RFC 3986: 1. IPs are not allowed. Currently an IngressRuleValue can only apply to    the IP in the Spec of the parent Ingress. 2. The `:` delimiter is not respected because ports are not allowed. 	  Currently the port of an Ingress is implicitly :80 for http and 	  :443 for https. Both these may change in the future. Incoming requests are matched against the host before the IngressRuleValue. If the host is unspecified, the Ingress routes all traffic based on the specified IngressRuleValue.  Host can be "precise" which is a domain name without the terminating dot of a network host (e.g. "foo.bar.com") or "wildcard", which is a domain name prefixed with a single wildcard label (e.g. "*.foo.com"). The wildcard character '\*' must appear by itself as the first DNS label and matches only a single label. You cannot have a wildcard label by itself (e.g. Host == "*"). Requests will be matched against the Host field in the following way: 1. If Host is precise, the request matches this rule if the http host header is equal to Host. 2. If Host is a wildcard, then the request matches this rule if the http host header is to equal to the suffix (removing the first label) of the wildcard rule.
http
Group
Version
Kind
v1
IngressBackend [networking/v1]
Field
name
string
port
ServiceBackendPort
Group
Version
Kind
v1
v1beta1
v1beta1
Field
hosts
string
Group
Version
Kind
v1
JSON
v1beta1
JSONSchemaProps [api

extensions/v1]
Field
Group
Version
Kind
v1
JSONSchemaProps is a JSON-Schema following Specification Draft 4 (http://json-schema.org/).
v1beta1
CustomResourceValidation [apiextensions/v1]
Field
$ref
string
$schema
string
JSONSchemaPropsOrBool
allOf
array
anyOf
array
default
JSON
object
object
string
enum
JSON
array
example
JSON
boolean
boolean
format
string
format is an OpenAPI v3 format string. Unknown formats are ignored. The following formats are validated:  - bsonobjectid: a bson object ID, i.e. a 24 characters hex string - uri: an URI as parsed by Golang net/url.ParseRequestURI - email: an email address as parsed by Golang net/mail.ParseAddress - hostname: a valid representation for an Internet host name, as defined by RFC 1034, section 3.1 [RFC1034]. - ipv4: an IPv4 IP as parsed by Golang net.ParseIP - ipv6: an IPv6 IP as parsed by Golang net.ParseIP - cidr: a CIDR as parsed by Golang net.ParseCIDR - mac: a MAC address as parsed by Golang net.ParseMAC - uuid: an UUID that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?[0-9a-f]{4}-?[0-9a-f]{4}-?[0-9a-f]{12}$ - uuid3: an UUID3 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?3[0-9a-f]{3}-?[0-9a-f]{4}-?[0-9a-f]{12}$ - uuid4: an UUID4 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?4[0-9a-f]{3}-?[89ab][0-9a-f]{3}-?[0-9a-f]{12}$ - uuid5: an UUID5 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?5[0-9a-f]{3}-?[89ab][0-9a-f]{3}-?[0-9a-f]{12}$ - isbn: an ISBN10 or ISBN13 number string like "0321751043" or "978-0321751041" - isbn10: an ISBN10 number string like "0321751043" - isbn13: an ISBN13 number string like "978-0321751041" - creditcard: a credit card number defined by the regex ^(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35\d{3})\d{11})$ with any non digit characters mixed in - ssn: a U.S. social security number following the regex ^\d{3}[- ]?\d{2}[- ]?\d{4}$ - hexcolor: an hexadecimal color code like "#FFFFFF: following the regex ^#?([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$ - rgbcolor: an RGB color code like rgb like "rgb(255,255,2559" - byte: base64 encoded binary data - password: any kind of string - date: a date string like "2006-01-02" as defined by full-date in RFC3339 - duration: a duration string like "22 ns" as parsed by Golang time.ParseDuration or compatible with Scala duration format - datetime: a date time string like "2014-12-15T19:30:20.000Z" as defined by date-time in RFC3339.
id
string
items
JSONSchemaPropsOrArray
maxItems
integer
maxLength
integer
integer
maximum
number
minItems
integer
minLength
integer
integer
minimum
number
number
not
nullable
boolean
oneOf
array
pattern
string
object
object
required
title
string
type
string
boolean
boolean
x-kuber

netes-int-or-string
boolean
x-kubernetes-int-or-string specifies that this value is either an integer or a string. If this is true, an empty type is allowed and type as child of anyOf is permitted if following one of the following patterns:  1) anyOf:    - type: integer    - type: string 2) allOf:    - anyOf:      - type: integer      - type: string    - ... zero or more
x-kubernetes-list-map-keys annotates an array with the x-kubernetes-list-type `map` by specifying the keys used as the index of the map.  This tag MUST only be used on lists that have the "x-kubernetes-list-type" extension set to "map". Also, the values specified for this attribute must be a scalar typed field of the child structure (no nesting is supported).  The properties specified must either be required or have a default value, to ensure those properties are present for all list items.
string
x-kubernetes-list-type annotates an array to further describe its topology. This extension must only be used on lists and may have 3 possible values:  1) `atomic`: the list is treated as a single entity, like a scalar.      Atomic lists will be entirely replaced when updated. This extension      may be used on any type of list (struct, scalar, ...). 2) `set`:      Sets are lists that must not have multiple items with the same value. Each      value must be a scalar, an object with x-kubernetes-map-type `atomic` or an      array with x-kubernetes-list-type `atomic`. 3) `map`:      These lists are like maps in that their elements have a non-index key      used to identify them. Order is preserved upon merge. The map tag      must only be used on a list with elements of type object. Defaults to atomic for arrays.
string
x-kubernetes-map-type annotates an object to further describe its topology. This extension must only be used when type is object and may have 2 possible values:  1) `granular`:      These maps are actual maps (key-value pairs) and each fields are independent      from each other (they can each be manipulated by separate actors). This is      the default behaviour for all maps. 2) `atomic`: the list is treated as a single entity, like a scalar.      Atomic maps will be entirely replaced when updated.
boolean
Group
Version
Kind
v1
v1beta1
Field
Group
Version
Kind
v1
v1beta1
Field
Group
Version
Kind
batch
v1
JobStatus [batch/v1]
Field
Time
Time
message
string
reason
string
status
string
type
string
JobTemplateSpec v1 ba

tch
Group
Version
Kind
batch
v1
v1beta1
CronJobSpec [batch/v1]
Field
metadata
Standard object's metadata of the jobs created from this template. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
JobSpec
Specification of the desired behavior of the job. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
Group
Version
Kind
core
v1
KeyToPath
ConfigMapProjection [core/v1]
ConfigMapVolumeSource [core/v1]
SecretProjection [core/v1]
SecretVolumeSource [core/v1]
Field
key
string
The key to project.
mode
integer
Optional: mode bits used to set permissions on this file. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. If not specified, the volume defaultMode will be used. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
path
string
The relative path of the file to map the key to. May not be an absolute path. May not contain the path element '..'. May not start with the string '..'.
Group
Version
Kind
meta
v1
AggregationRule [rbac/v1]
AggregationRule [rbac/v1beta1]
AggregationRule [rbac/v1alpha1]
CSIStorageCapacity [storage/v1beta1]
CSIStorageCapacity [storage/v1alpha1]
ExternalMetricSource [autoscaling/v2beta1]
ExternalMetricStatus [autoscaling/v2beta1]
MetricIdentifier [autoscaling/v2beta2]
MutatingWebhook [admissionregistration/v1]
MutatingWebhook [admissionregistration/v1beta1]
NetworkPolicySpec [networking/v1]
ObjectMetricSource [autoscaling/v2beta1]
ObjectMetricStatus [autoscaling/v2beta1]
PersistentVolumeClaimSpec [core/v1]
PodAffinityTerm [core/v1]
PodDisruptionBudgetSpec [policy/v1]
PodDisruptionBudgetSpec [policy/v1beta1]
PodsMetricSource [autoscaling/v2beta1]
PodsMetricStatus [autoscaling/v2beta1]
TopologySpreadConstraint [core/v1]
ValidatingWebhook [admissionregistration/v1]
ValidatingWebhook [admissionregistration/v1beta1]
Field
LabelSelectorRequirement
array
object
Group
Version
Kind
meta
v1
LabelSelector [meta/v1]
Field
key
string
:
merge
:
key
operator
string
values
Group
Version
Kind
core
v1
Lifecycle
Lifecycle describes 

actions that the management system should take in response to container lifecycle events. For the PostStart and PreStop lifecycle handlers, management of the container blocks until the action is complete, unless the container process fails, in which case the handler is aborted.
Field
postStart
Handler
PostStart is called immediately after a container is created. If the handler fails, the container is terminated and restarted according to its restart policy. Other management of the container blocks until the hook completes. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
preStop
Handler
PreStop is called immediately before a container is terminated due to an API request or management event such as liveness/startup probe failure, preemption, resource contention, etc. The handler is not called if the container crashes or exits. The reason for termination is passed to the handler. The Pod's termination grace period countdown begins before the PreStop hooked is executed. Regardless of the outcome of the handler, the container will eventually terminate within the Pod's termination grace period. Other management of the container blocks until the hook completes or until the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
Group
Version
Kind
core
v1
LimitRangeSpec [core/v1]
Field
default
object
object
max
object
maxLimitRequestRatio
object
min
object
type
string
Group
Version
Kind
v1beta1
LimitResponse
LimitedPriorityLevelConfiguration [flowcontrol/v1beta1]
Field
queuing
QueuingConfiguration
`queuing` holds the configuration parameters for queuing. This field may be non-empty only if `type` is `"Queue"`.
type
string
`type` is "Queue" or "Reject". "Queue" means that requests that can not be executed upon arrival are held in a queue until they can be executed or a queuing limit is reached. "Reject" means that requests that can not be executed upon arrival are rejected. Required.
Group
Version
Kind
v1beta1
LimitedPriorityLevelConfiguration specifies how to handle requests that are subject to limits. It addresses two issues:
 * How are requests for this priority level limited?
 * What should be done with requests that exceed the limit?
PriorityLevelConfigurationSpec [flowcontrol/v1beta1]
Field
assuredConcurrencyShares
integer
`assuredConcurrencyShares` (ACS) configures the execution limit, which is a limit on the number of requests of this priority level that may be exeucting at a given time.  ACS must be a positive number. The server's concurrency limit (SCL) is divided among the concurrency-controlled priority levels in proportion to their assured concurrency shares. This produces the assured concurrency value (ACV) --- the number of requests that may 

be executing at a time --- for each such priority level:              ACV(l) = ceil( SCL * ACS(l) / ( sum[priority levels k] ACS(k) ) )  bigger numbers of ACS mean more reserved concurrent requests (at the expense of every other PL). This field has a default value of 30.
`limitResponse` indicates what to do with requests that can not be executed right now
Group
Version
Kind
meta
v1
ListMeta
APIServiceList [apiregistration/v1beta1]
CSIDriverList [storage/v1beta1]
CSINodeList [storage/v1beta1]
CSIStorageCapacityList [storage/v1alpha1]
CertificateSigningRequestList [certificates/v1beta1]
ClusterRoleBindingList [rbac/v1beta1]
ClusterRoleBindingList [rbac/v1alpha1]
ClusterRoleList [rbac/v1beta1]
ClusterRoleList [rbac/v1alpha1]
CronJobList [batch/v1beta1]
CustomResourceDefinitionList [apiextensions/v1beta1]
EndpointSliceList [discovery/v1beta1]
EventList [core/v1]
EventList [events/v1beta1]
HorizontalPodAutoscalerList [autoscaling/v2beta2]
HorizontalPodAutoscalerList [autoscaling/v2beta1]
IngressClassList [networking/v1beta1]
IngressList [extensions/v1beta1]
IngressList [networking/v1beta1]
LeaseList [coordination/v1beta1]
MutatingWebhookConfigurationList [admissionregistration/v1beta1]
PodDisruptionBudgetList [policy/v1beta1]
PriorityClassList [scheduling/v1beta1]
PriorityClassList [scheduling/v1alpha1]
RoleBindingList [rbac/v1beta1]
RoleBindingList [rbac/v1alpha1]
RoleList [rbac/v1beta1]
RoleList [rbac/v1alpha1]
RuntimeClassList [node/v1beta1]
RuntimeClassList [node/v1alpha1]
Status [meta/v1]
StorageClassList [storage/v1beta1]
ValidatingWebhookConfigurationList [admissionregistration/v1beta1]
VolumeAttachmentList [storage/v1beta1]
VolumeAttachmentList [storage/v1alpha1]
Field
continue
string
continue may be set if the user set a limit on the number of items returned, and indicates that the server has more data available. The value is opaque and may be used to issue another request to the endpoint that served this list to retrieve the next set of available objects. Continuing a consistent list may not be possible if the server configuration has changed or more than a few minutes have passed. The resourceVersion field returned when using this continue val

ue will be identical to the value in the first response, unless you have received this token from an error message.
integer
remainingItemCount is the number of subsequent items in the list which are not included in this list response. If the list request contained label or field selectors, then the number of remaining items is unknown and the field will be left unset and omitted during serialization. If the list is complete (either because it is not chunking or because this is the last chunk), then there are no more remaining items and this field will be left unset and omitted during serialization. Servers older than v1.15 do not set this field. The intended use of the remainingItemCount is *estimating* the size of a collection. Clients should not rely on the remainingItemCount to be set or to be exact.
string
String that identifies the server's internal version of this object that can be used by clients to determine when objects have changed. Value must be treated as opaque by clients and passed unmodified back to the server. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
selfLink
string
selfLink is a URL representing this object. Populated by the system. Read-only.  DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release.
Group
Version
Kind
core
v1
LoadBalancerIngress
LoadBalancerStatus [core/v1]
Field
hostname
string
ip
string
ports
PortStatus
array
Group
Version
Kind
core
v1
IngressStatus [networking/v1]
IngressStatus [extensions/v1beta1]
IngressStatus [networking/v1beta1]
Field
ingress
array
Group
Version
Kind
core
v1
CSIVolumeSource [core/v1]
CephFSVolumeSource [core/v1]
CinderVolumeSource [core/v1]
FlexVolumeSource [core/v1]
ISCSIVolumeSource [core/v1]
RBDVolumeSource [core/v1]
ScaleIOVolumeSource [core/v1]
ServiceAccount [core/v1]
StorageOSVolumeSource [core/v1]
Field
name
string
Group
Version
Kind
core
v1
Field
fsType
string
Filesystem type to mount. It applies only when the Path is a block device. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". The default value is to auto-select a fileystem if unspecified.
path
string
The full path to the volume on the node. It can be either a directory or block device (disk, partition, ...).
Group
Version
Kind
meta
v1
ManagedFieldsEntry
ObjectMeta [meta/v1]
Field
D

escription
string
string
fieldsV1
FieldsV1
manager
string
operation
string
time
Time
Time is timestamp of when these fields were set. It should always be empty if Operation is 'Apply'
Group
Version
Kind
v2beta2
ExternalMetricSource [autoscaling/v2beta2]
ExternalMetricStatus [autoscaling/v2beta2]
ObjectMetricSource [autoscaling/v2beta2]
ObjectMetricStatus [autoscaling/v2beta2]
PodsMetricSource [autoscaling/v2beta2]
PodsMetricStatus [autoscaling/v2beta2]
Field
name
string
selector
Group
Version
Kind
v2beta2
MetricSpec
MetricSpec specifies how to scale based on a single metric (only `type` and one other matching field should be set at once).
v2beta1
Field
container resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing a single container in each pod of the current scale target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source. This is an alpha feature and can be enabled by the HPAContainerMetrics feature flag.
external
object
ObjectMetricSource
pods
PodsMetricSource
pods refers to a metric describing each pod in the current scale target (for example, transactions-processed-per-second).  The values will be averaged together before being compared to the target value.
resource
ResourceMetricSource
type
string
type is the type of metric source.  It should be one of "ContainerResource", "External", "Object", "Pods" or "Resource", each mapping to a matching field in the object. Note: "ContainerResource" type is available on when the feature-gate HPAContainerMetrics is enabled
Group
Version
Kind
v2beta2
MetricStatus
Other API versions of this object 

exist:
v2beta1
Field
external
object
ObjectMetricStatus
pods
PodsMetricStatus
resource
ResourceMetricStatus
type
string
type is the type of metric source.  It will be one of "ContainerResource", "External", "Object", "Pods" or "Resource", each corresponds to a matching field in the object. Note: "ContainerResource" type is available on when the feature-gate HPAContainerMetrics is enabled
Group
Version
Kind
v2beta2
ContainerResourceMetricSource [autoscaling/v2beta2]
ResourceMetricSource [autoscaling/v2beta2]
Field
integer
Quantity
type
string
value
Quantity
Group
Version
Kind
v2beta2
ContainerResourceMetricStatus [autoscaling/v2beta2]
ResourceMetricStatus [autoscaling/v2beta2]
Field
integer
Quantity
value
Quantity
Group
Version
Kind
meta
v1
MicroTime
EventSeries [core/v1]
EventSeries [events/v1]
EventSeries [events/v1beta1]
LeaseSpec [coordination/v1]
LeaseSpec [coordination/v1beta1]
Field
MutatingWebhook v1 admission

registration.k8s.io
Group
Version
Kind
v1
v1beta1
MutatingWebhookConfiguration [admissionregistration/v1]
Field
AdmissionReviewVersions is an ordered list of preferred `AdmissionReview` versions the Webhook expects. API server will try to use first version in the list which it supports. If none of the versions specified in this list supported by API server, validation will fail for this object. If a persisted webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail and be subject to the failure policy.
WebhookClientConfig
string
string
matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent".  - Exact: match a request only if it exactly matches a specified rule. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, but "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/v1beta1 would not be sent to the webhook.  - Equivalent: match a request if modifies a resource listed in rules, even via another API group or version. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, and "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/v1beta1 would be converted to apps/v1 and sent to the webhook.  Defaults to "Equivalent"
name
string
NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.  For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "runlevel",       "operator": "NotIn",       "values": [         "0",         "1"       ]     }   ] }  If instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "environment",       "operator": "In",       "values": [         "prod",         "staging"       ]     }   ] }  See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.  Default to the empty LabelSelector, which matches everything.
ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in the case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the adm

ission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
string
reinvocationPolicy indicates whether this webhook should be called multiple times as part of a single admission evaluation. Allowed values are "Never" and "IfNeeded".  Never: the webhook will not be called more than once in a single admission evaluation.  IfNeeded: the webhook will be called at least one additional time as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call. Webhooks that specify this option *must* be idempotent, able to process objects they previously admitted. Note: * the number of additional invocations is not guaranteed to be exactly one. * if additional invocations result in further modifications to the object, webhooks are not guaranteed to be invoked again. * webhooks that use this option may be reordered to minimize the number of additional invocations. * to validate an object after all mutations are guaranteed complete, use a validating admission webhook instead.  Defaults to "Never".
rules
RuleWithOperations
array
Rules describes what operations on what resources/subresources the webhook cares about. The webhook cares about an operation if it matches _any_ Rule. However, in order to prevent ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks from putting the cluster in a state which cannot be recovered from without completely disabling the plugin, ValidatingAdmissionWebhooks and MutatingAdmissionWebhooks are never called on admission requests for ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects.
string
integer
Group
Version
Kind
core
v1
Field
path
string
Path that is exported by the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
readOnly
boolean
ReadOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
server
string
Server is the hostname or IP address of the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
Group
Version
Kind
core
v1
NamespaceStatus [core/v1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1
Field
ports
NetworkPolicyPort
array
List of destination ports for outgoing traffic. Each item in this lis

t is combined using a logical OR. If this field is empty or missing, this rule matches all ports (traffic not restricted by port). If this field is present and contains at least one item, then this rule allows traffic only if the traffic matches at least one port in the list.
to
NetworkPolicyPeer
array
Group
Version
Kind
v1
Field
from
array
ports
array
Group
Version
Kind
v1
NetworkPolicyEgressRule [networking/v1]
NetworkPolicyIngressRule [networking/v1]
Field
ipBlock
IPBlock
Selects Namespaces using cluster-scoped labels. This field follows standard label selector semantics; if present but empty, it selects all namespaces.  If PodSelector is also set, then the NetworkPolicyPeer as a whole selects the Pods matching PodSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects all Pods in the Namespaces selected by NamespaceSelector.
This is a label selector which selects Pods. This field follows standard label selector semantics; if present but empty, it selects all pods.  If NamespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the Pods matching PodSelector in the Namespaces selected by NamespaceSelector. Otherwise it selects the Pods matching PodSelector in the policy's own Namespace.
Group
Version
Kind
v1
Field
endPort
integer
If set, indicates that the range of ports from port to endPort, inclusive, should be allowed by the policy. This field cannot be defined if the port field is not defined or if the port field is defined as a named (string) port. The endPort must be equal or greater than port. This feature is in Alpha state and should be enabled using the Feature Gate "NetworkPolicyEndPort".
port
The port on the given protocol. This can either be a numerical or named port on a pod. If this field is not provided, this matches all port names and numbers. If present, only traffic on the s

pecified protocol AND port will be matched.
protocol
string
Group
Version
Kind
core
v1
Field
address
string
type
string
Group
Version
Kind
core
v1
Affinity [core/v1]
Field
PreferredSchedulingTerm
array
NodeSelector
Group
Version
Kind
core
v1
Field
Time
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
NodeConfigSource specifies a source of node configuration. Exactly one subfield (excluding metadata) must be non-nil.
NodeConfigStatus [core/v1]
NodeSpec [core/v1]
Field
configMap
Group
Version
Kind
core
v1
Field
active
assigned
error
string
Error describes any problems reconciling the Spec.ConfigSource to the Active config. Errors may occur, for example, attempting to checkpoint Spec.ConfigSource to the local Assigned record, attempting to checkpoint the payload associated with Spec.ConfigSource, attempting to load or validate the 

Assigned config, etc. Errors may occur at different points while syncing config. Earlier errors (e.g. download or checkpointing errors) will not result in a rollback to LastKnownGood, and may resolve across Kubelet retries. Later errors (e.g. loading or validating a checkpointed config) will result in a rollback to LastKnownGood. In the latter case, it is usually possible to resolve the error by fixing the config assigned in Spec.ConfigSource. You can find additional information for debugging by searching the error message in the Kubelet log. Error is a human-readable description of the error state; machines can check whether or not Error is empty, but should not rely on the stability of the Error text across Kubelet versions.
lastKnownGood
Group
Version
Kind
core
v1
Field
Group
Version
Kind
core
v1
NodeAffinity [core/v1]
VolumeNodeAffinity [core/v1]
Field
NodeSelectorTerm
array
Group
Version
Kind
core
v1
NodeSelectorTerm [core/v1]
Field
key
string
operator
string
values
Group
Version
Kind
core
v1
NodeSelector [core/v1]
PreferredSchedulingTerm [core/v1]
Field
array
matchFields
array
Group
Version
Kind
core
v1
Field
string
bootID
string
string
ContainerRuntime Version reported by the node through runtime remote 

API (e.g. docker://1.5.0).
string
string
string
machineID
string
MachineID reported by the node. For unique machine identification in the cluster this field is preferred. Learn more from man(5) machine-id: http://man7.org/linux/man-pages/man5/machine-id.5.html
string
osImage
string
string
SystemUUID reported by the node. For unique machine identification MachineID is preferred. This field is specific to Red Hat hosts https://access.redhat.com/documentation/en-us/red_hat_subscription_management/1/html/rhsm/uuid
Group
Version
Kind
v1
v1beta1
SelfSubjectAccessReviewSpec [authorization/v1]
SubjectAccessReviewSpec [authorization/v1]
Field
path
string
verb
string
Group
Version
Kind
v1beta1
NonResourcePolicyRule
PolicyRulesWithSubjects [flowcontrol/v1beta1]
Field
`nonResourceURLs` is a set of url prefixes that a user should have access to and may not be empty. For example:   - "/healthz" is legal   - "/hea*" is illegal   - "/hea" is legal but matches nothing   - "/hea/*" also matches nothing   - "/healthz/*" matches all per-component health checks. "*" matches all non-resource urls. if it is present, it must be the only entry. Required.
verbs
`verbs` is a list of matching verbs and may not be empty. "*" matches all verbs. If it is present, it must be the only entry. Required.
Group
Version
Kind
v1
NonResourceRule
v1beta1
SubjectRulesReviewStatus [authorization/v1]
Field
NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path.  "*" means all.
verbs
Verb is a list of kubernetes non-resource API verbs, like: get, post, put, delete, patch, head, options.  "*" means all.
Group
Version
Kind
core
v1
DownwardAPIVolumeFile [core/v1]
Field
string
fieldPath
string
Group
Version
Kind
meta
v1
APIService [apiregistration/v1beta1]
Binding [core/v1]
CSIDriver [storage/v1beta1]
CSINode [storage/v1beta1]
CertificateSigningRequest [certificate

s/v1beta1]
ClusterRole [rbac/v1beta1]
ClusterRole [rbac/v1alpha1]
ClusterRoleBinding [rbac/v1]
ClusterRoleBinding [rbac/v1beta1]
ClusterRoleBinding [rbac/v1alpha1]
ConfigMap [core/v1]
ControllerRevision [apps/v1]
CronJob [batch/v1beta1]
CustomResourceDefinition [apiextensions/v1beta1]
EndpointSlice [discovery/v1beta1]
HorizontalPodAutoscaler [autoscaling/v2beta2]
HorizontalPodAutoscaler [autoscaling/v2beta1]
Ingress [extensions/v1beta1]
Ingress [networking/v1beta1]
IngressClass [networking/v1beta1]
Lease [coordination/v1beta1]
LocalSubjectAccessReview [authorization/v1beta1]
MutatingWebhookConfiguration [admissionregistration/v1beta1]
PodDisruptionBudget [policy/v1beta1]
PriorityClass [scheduling/v1]
PriorityClass [scheduling/v1beta1]
PriorityClass [scheduling/v1alpha1]
Role [rbac/v1]
Role [rbac/v1beta1]
Role [rbac/v1alpha1]
RoleBinding [rbac/v1]
RoleBinding [rbac/v1beta1]
RoleBinding [rbac/v1alpha1]
RuntimeClass [node/v1]
RuntimeClass [node/v1beta1]
RuntimeClass [node/v1alpha1]
Scale [autoscaling/v1]
Secret [core/v1]
SelfSubjectAccessReview [authorization/v1beta1]
SelfSubjectRulesReview [authorization/v1beta1]
StorageClass [storage/v1]
StorageClass [storage/v1beta1]
SubjectAccessReview [authorization/v1beta1]
TokenReview [authentication/v1beta1]
ValidatingWebhookConfiguration [admissionregistration/v1]
ValidatingWebhookConfiguration [admissionregistration/v1beta1]
VolumeAttachment [storage/v1beta1]
VolumeAttachment [storage/v1alpha1]
Field
object
Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: http://kubernetes.io/docs/user-guide/annotations
string
The name of the cluster which the object belongs to. This is used to distinguish resources with same name and namespace in different clusters. This field is not set anywhere right now and apiserver is going to ignore it if set in create or update request.
Time
CreationTimestamp is a timestamp representing the server time when this object was created. It is not guaranteed to be set in happens-before order across separate operations. Clients may not set this value. It is represented in RFC3339 form and is in UTC.  Populated by the system. Read-only. Null for lists. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
integer
Number of seconds allowed for this object to gracefully terminate before it 

will be removed from the system. Only set when deletionTimestamp is also set. May only be shortened. Read-only.
Time
DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.  Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
:
merge
Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed. Finalizers may be processed and removed in any order.  Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list.
string
GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.  If this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header).  Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency
integer
labels
object
Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: http://kubernetes.io/docs/user-guide/labels
array
ManagedFields maps workflow-id and version to the set of fields that are managed by that workflow. This is mostly for internal housekeeping, and users typically shouldn't need to set or understand this field. 

A workflow can be the user's name, a controller's name, or the name of a specific apply path like "ci-cd". The set of fields is always in the version that the workflow used when modifying the object.
name
string
Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names
namespace
string
Namespace defines the space within which each name must be unique. An empty namespace is equivalent to the "default" namespace, but "default" is the canonical representation. Not all objects are required to be scoped to a namespace - the value of this field for those objects will be empty.  Must be a DNS_LABEL. Cannot be updated. More info: http://kubernetes.io/docs/user-guide/namespaces
OwnerReference
array
:
merge
:
uid
string
An opaque value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.  Populated by the system. Read-only. Value must be treated as opaque by clients and . More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
selfLink
string
SelfLink is a URL representing this object. Populated by the system. Read-only.  DEPRECATED Kubernetes will stop propagating this field in 1.20 release and the field is planned to be removed in 1.21 release.
uid
string
UID is the unique in time and space value for this object. It is typically generated by the server on successful creation of a resource and is not allowed to change on PUT operations.  Populated by the system. Read-only. More info: http://kubernetes.io/docs/user-guide/identifiers#uids
Group
Version
Kind
v2beta2
v2beta1
Field
metric
target
Group
Version
Kind
v2beta2
v2beta1
Field
current
metric
Group
Version
Kind
core
v1
CronJobStatus [batch/v1]
CronJobStatus [batch/v1beta1]
Endpoint [discovery/v1beta1]
EndpointAddress [core/v1]
Event [events

/v1]
StorageOSPersistentVolumeSource [core/v1]
Field
string
fieldPath
string
kind
string
Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
name
string
namespace
string
Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
string
Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
uid
string
UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
Group
Version
Kind
v1
Overhead
v1beta1
v1alpha1
Field
podFixed
object
Group
Version
Kind
meta
v1
Field
string
boolean
If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned.
boolean
kind
string
name
string
Name of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#names
uid
string
UID of the referent. More info: http://kubernetes.io/docs/user-guide/identifiers#uids
Group
Version
Kind
meta
v1
Patch
Field
Group
Version
Kind
core
v1
PersistentVolumeClaimStatus [core/v1]
Field
Time
Time
message
string
reason
string
Unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
status
string
type
string
Group
Version
Kind
core
v1
PersistentVolumeClaimTemplate is used to produce PersistentVolumeClaim o

bjects as part of an EphemeralVolumeSource.
EphemeralVolumeSource [core/v1]
Field
metadata
spec
Group
Version
Kind
core
v1
Field
claimName
string
ClaimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
readOnly
boolean
Will force the ReadOnly setting in VolumeMounts. Default false.
Group
Version
Kind
core
v1
Field
fsType
string
pdID
string
ID that identifies Photon Controller persistent disk
Group
Version
Kind
core
v1
Field
WeightedPodAffinityTerm
array
PodAffinityTerm
array
Group
Version
Kind
core
v1
Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key &lt;topologyKey&gt; matches that of any node on which a pod of the set of pods is running
PodAffinity [core/v1]
PodAntiAffinity [core/v1]
WeightedPodAffinityTerm [core/v1]
Field
A label query over the set of namespaces that the term applies to. The term is applied to the union of the namespaces selected by this field and the ones listed in the namespaces field. nul

l selector and null or empty namespaces list means "this pod's namespace". An empty selector ({}) matches all namespaces. This field is alpha-level and is only honored when PodAffinityNamespaceSelector feature is enabled.
namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespaces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace"
string
Group
Version
Kind
core
v1
Field
array
array
Group
Version
Kind
core
v1
Field
Time
Time
message
string
reason
string
status
string
Status is the status of the condition. Can be True, False, Unknown. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-conditions
type
string
Type is the type of the condition. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-conditions
Group
Version
Kind
core
v1
Field
nameservers
options
PodDNSConfigOption
array
searches

 v1 core
Group
Version
Kind
core
v1
PodDNSConfig [core/v1]
Field
name
string
Required.
value
string
Group
Version
Kind
core
v1
PodIP
IP address information for entries in the (plural) PodIPs field. Each entry includes:
   IP: An IP address allocated to the pod. Routable at least within the cluster.
Field
ip
string
Group
Version
Kind
core
v1
Field
conditionType
string
Group
Version
Kind
core
v1
PodSecurityContext holds pod-level security attributes and common container settings. Some fields are also present in container.securityContext.  Field values of container.securityContext take precedence over field values of PodSecurityContext.
Field
fsGroup
integer
A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:  1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw----  If unset, the Kubelet will not modify the ownership and permissions of any volume.
string
fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types which support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used.
integer
The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container.
boolean
Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in SecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
runAsUser
integer
The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container.
The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container.  May also be set in SecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container.
The seccomp options to use by the containers in this pod.
integer array
A list of groups applied to the first process run in each container, in addition to the container's primary GID.  If unspecified, no groups will be added to any container.
sysctls
Sysctl
array
Sysctls hold a list of namespaced sysctls used for the pod. Pods with unsupported sysctls (by the container runtime) might fail to launch.
The Windows specific settings applied to all containers. If unspecified, the options within a container's 

SecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
Group
Version
Kind
v2beta2
v2beta1
Field
metric
target
Group
Version
Kind
v2beta2
v2beta1
Field
current
metric
Group
Version
Kind
v1
v1beta1
v1alpha1
Field
apiGroups
APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.
NonResourceURLs is a set of partial urls that a user should have access to.  *s are allowed, but only as the full, final step in the path Since non-resource URLs are not namespaced, this field is only applicable for ClusterRoles referenced from a ClusterRoleBinding. Rules can either apply to API resources (such as "pods" or "secrets") or non-resource URL paths (such as "/api"),  but not both.
ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.
resources
Resources is a list of resources this rule applies to.  ResourceAll represents all resources.
verbs
Verbs is a list of Verbs that apply to ALL the ResourceKinds and AttributeRestrictions contained in this rule.  VerbAll represents all kinds.
Group
Version
Kind
v1beta1
Field
array
`nonResourceRules` is a list of NonResourcePolicyRules that identify matching requests according to their verb and the target non-resource URL.
ResourcePolicyRule
array
`resourceRules` is a slice of ResourcePolicyRules that identify matching requests according to their verb and the target resource. At least one of `resourceRules` and `nonResourceRules` has to be non-empty.
subjects
Subject
array
subjects is the list of normal user, serviceaccount, or group that this rule cares about. There must be at least one member in this slice. A slice that includes both the system:authenticated and system:unaut

henticated user groups matches every request. Required.
Group
Version
Kind
core
v1
LoadBalancerIngress [core/v1]
Field
error
string
Error is to record the problem with the service port The format of the error shall comply with the following rules: - built-in error values shall be specified in this file and those shall use   CamelCase names - cloud provider specific error values must have names that comply with the   format foo.example.com/CamelCase.
port
integer
protocol
string
Group
Version
Kind
core
v1
Field
fsType
string
FSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs". Implicitly inferred to be "ext4" if unspecified.
readOnly
boolean
volumeID
string
VolumeID uniquely identifies a Portworx volume
Group
Version
Kind
meta
v1
DeleteOptions [meta/v1]
Field
string
uid
string
Group
Version
Kind
core
v1
Field
weight
integer
Group
Version
Kind
v1beta1
PriorityLevelConfigurationStatus [flowcontrol/v1beta1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1beta1
Field
name
string
`name` is the name of the priority level configuration being referenced Required.
Group
Version
Kind
core
v1
Probe
Field
exec
integer
httpGet
integer
Number of seconds after the con

tainer has started before liveness probes are initiated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
integer
integer
tcpSocket
integer
Optional duration in seconds the pod needs to terminate gracefully upon probe failure. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this value overrides the value provided by the pod spec. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). This is an alpha field and requires enabling ProbeTerminationGracePeriod feature gate.
integer
Number of seconds after which the probe times out. Defaults to 1 second. Minimum value is 1. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
Group
Version
Kind
core
v1
Field
integer
Mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bits. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
sources
VolumeProjection
array
list of volume projections
Group
Version
Kind
core
resource
Quantity

&lt;quantity&gt;        ::= &lt;signedNumber&gt;&lt;suffix&gt;
  (Note that &lt;suffix&gt; may be empty, from the "" case in &lt;decimalSI&gt;.)
&lt;digit&gt;           ::= 0 | 1 | ... | 9 &lt;digits&gt;          ::= &lt;digit&gt; | &lt;digit&gt;&lt;digits&gt; &lt;number&gt;          ::= &lt;digits&gt; | &lt;digits&gt;.&lt;digits&gt; | &lt;digits&gt;. | .&lt;digits&gt; &lt;sign&gt;            ::= "+" | "-" &lt;signedNumber&gt;    ::= &lt;number&gt; | &lt;sign&gt;&lt;number&gt; &lt;suffix&gt;          ::= &lt;binarySI&gt; | &lt;decimalExponent&gt; | &lt;decimalSI&gt; &lt;binarySI&gt;        ::= Ki | Mi | Gi | Ti | Pi | Ei
  (International System of units; See: http://physics.nist.gov/cuu/Units/binary.html)
&lt;decimalSI&gt;       ::= m | "" | k | M | G | T | P | E
  (Note that 1024 = 1Ki but 1000 = 1k; I didn't choose the capitalization.)
&lt;decimalExponent&gt; ::= "e" &lt;signedNumber&gt; | "E" &lt;signedNumber&gt;

Before serializing, Quantity will be put in "canonical form". This means that Exponent/suffix will be adjusted up or down (with a corresponding incre

ase or decrease in Mantissa) such that:
  a. No precision is lost
  b. No fractional digits will be emitted
  c. The exponent (or suffix) is as large as possible.

Examples:
  1.5 will be serialized as "1500m"
  1.5Gi will be serialized as "1536Mi"

ContainerResourceMetricSource [autoscaling/v2beta1]
ContainerResourceMetricStatus [autoscaling/v2beta1]
EmptyDirVolumeSource [core/v1]
MetricTarget [autoscaling/v2beta2]
MetricValueStatus [autoscaling/v2beta2]
ResourceFieldSelector [core/v1]
ResourceMetricSource [autoscaling/v2beta1]
ResourceMetricStatus [autoscaling/v2beta1]
Field
Group
Version
Kind
v1beta1
LimitResponse [flowcontrol/v1beta1]
Field
handSize
integer
`handSize` is a small positive number that configures the shuffle sharding of requests into queues.  When enqueuing a request at this priority level the request's flow identifier (a string pair) is hashed and the hash value is used to shuffle the list of queues and deal a hand of the size specified here.  The request is put into one of the shortest queues in that hand. `handSize` must be no larger than `queues`, and should be significantly smaller (so that a few heavy flows do not saturate most of the queues).  See the user-facing documentation for more extensive guidance on setting this field.  This field has a default value of 8.
integer
`queueLengthLimit` is the maximum number of requests allowed to be waiting in a given queue of this priority level at a time; excess requests are rejected.  This value must be positive.  If not specified, it will be defaulted to 50.
queues
integer
`queues` is the number of queues for this priority level. The queues exist independently at each apiserver. The value must be positive.  Setting it to 1 effectively precludes shufflesharding and thus makes the distinguisher method of associated flow schemas irrelevant.  This field has a default value of 64.
Group
Version
Kind
core
v1
Field
group
string
Group to map volume access to Default is no group
readOnly
boolean
ReadOnly here will force the Quobyte volume to be mounted with read-only permissions. Defaults to false.
registry
string
Registry represents a single or multiple Quobyte Registry services specified as a string as host:port pair (multiple entries are separated with commas) which acts as the central registry for volumes
tenant
string
Tenant owning the given Quobyte volume in the Backend Used with dynamically provisioned Quobyte volumes, value is set by the plugin
user
string
User to map volume access to Defaults to serivceaccount user
volume
string
Volume is a string that references an already created Quobyte volume by name.
RBD

PersistentVolumeSource v1 core
Group
Version
Kind
core
v1
Field
fsType
string
Filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd
image
string
The rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
keyring
string
Keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
monitors
A collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
pool
string
The rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
readOnly
boolean
ReadOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
secretRef
SecretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
user
string
The rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
Group
Version
Kind
core
v1
Field
fsType
string
image
string
keyring
string
monitors
pool
string
readOnly
boolean
secretRef
user
string
Group
Version
Kind
apps
v1
ReplicaSetStatus [apps/v1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
ReplicationControllerStatus [core/v1]
Field
Time
The last t

ime the condition transitioned from one status to another.
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1
v1beta1
Field
group
string
Group is the API Group of the Resource.  "*" means all.
name
string
namespace
string
Namespace is the namespace of the action being requested.  Currently, there is no distinction between no namespace and all namespaces "" (empty) is defaulted for LocalSubjectAccessReviews "" (empty) is empty for cluster-scoped resources "" (empty) means "all" for namespace scoped resources from a SubjectAccessReview or SelfSubjectAccessReview
resource
string
Resource is one of the existing resource types.  "*" means all.
string
Subresource is one of the existing resource types.  "" means none.
verb
string
Verb is a kubernetes resource API verb, like: get, list, watch, create, update, delete, proxy.  "*" means all.
version
string
Version is the API Version of the Resource.  "*" means all.
Group
Version
Kind
core
v1
Field
string
divisor
Quantity
resource
string
Group
Version
Kind
v2beta2
ResourceMetricSource indicates how to scale on a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory).  The values will be averaged together before being compared to the target.  Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.  Only one "target" type should be set.
v2beta1
Field
name
string
target
Group
Version
Kind
v2beta2
ResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing each pod in the current scale target (e.g. CPU or memory).  Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
v2beta1
Field
current
name
string
Group
Version
Kind
v1beta1
ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one m

ember of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) least one member of namespaces matches the request.
Field
apiGroups
`apiGroups` is a list of matching API groups and may not be empty. "*" matches all API groups and, if present, must be the only entry. Required.
boolean
`clusterScope` indicates whether to match requests that do not specify a namespace (which happens either because the resource is not namespaced or the request targets all namespaces). If this field is omitted or false then the `namespaces` field must contain a non-empty list.
`namespaces` is a list of target namespaces that restricts matches.  A request that specifies a target namespace matches only if either (a) this list contains that target namespace or (b) this list contains "*".  Note that "*" matches any specified namespace but does not match a request that _does not specify_ a namespace (see the `clusterScope` field for that). This list may be empty, but only if `clusterScope` is true.
resources
`resources` is a list of matching resources (i.e., lowercase and plural) with, if desired, subresource.  For example, [ "services", "nodes/status" ].  This list may not be empty. "*" matches all resources and, if present, must be the only entry. Required.
verbs
`verbs` is a list of matching verbs and may not be empty. "*" matches all verbs and, if present, must be the only entry. Required.
Group
Version
Kind
core
v1
Field
limits
object
Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
requests
object
Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
Group
Version
Kind
v1
ResourceRule
v1beta1
Field
apiGroups
APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed.  "*" means all.
ResourceNames is an optional white list of names that the rule applies to.  An empty set means that everything is allowed.  "*" means all.
resources
Resources is a list of resources this rule applies to.  "*" means all in the specified apiGroups.  "*/foo" represents the subresource 'foo' for all resources in the specified apiGroups.
verbs
Verb is a list of kubernetes resource API verbs, like: get, list, watch, create, update, delete, proxy.  "*" means all.
Group
Version
Kind
v1
RoleRef
v1beta1
v1alpha1
Field
apiGroup
string
kind
string
name
string
Group
Vers

ion
Kind
apps
v1
RollingUpdateStatefulSetStrategy
StatefulSetUpdateStrategy [apps/v1]
Field
partition
integer
Partition indicates the ordinal at which the StatefulSet should be partitioned. Default value is 0.
Group
Version
Kind
v1
v1beta1
Field
apiGroups
APIGroups is the API groups the resources belong to. '\*' is all groups. If '\*' is present, the length of the slice must be one. Required.
APIVersions is the API versions the resources belong to. '\*' is all versions. If '\*' is present, the length of the slice must be one. Required.
Operations is the operations the admission hook cares about - CREATE, UPDATE, DELETE, CONNECT or * for all of those operations and any future admission operations that are added. If '\*' is present, the length of the slice must be one. Required.
resources
Resources is a list of resources this rule applies to.  For example: 'pods' means pods. 'pods/log' means the log subresource of pods. '\*' means all resources, but not subresources. 'pods/\*' means all subresources of pods. '\*/scale' means all scale subresources. '\*/\*' means all resources and their subresources.  If wildcard is present, the validation rule will ensure resources do not overlap with each other.  Depending on the enclosing object, subresources might not be allowed. Required.
scope
string
scope specifies the scope of this rule. Valid values are "Cluster", "Namespaced", and "*" "Cluster" means that only cluster-scoped resources will match this rule. Namespace API objects are cluster-scoped. "Namespaced" means that only namespaced resources will match this rule. "*" means that there are no scope restrictions. Subresources match the scope of their parent resource. Default is "*".
Group
Version
Kind
policy
v1beta1
RunAsGroupStrategyOptions defines the strategy type and any options used to create the strategy.
Field
ranges
IDRange
array
ranges are the allowed ranges of gids that may be used. If you would like to force a single gid then supply a single range with the same start and end. Required for MustRunAs.
rule
string
rule is the strategy that will dictate the allowable RunAsGroup values that may be set.
Group
Version
Kind
policy
v1beta1
RunAsUserStrategyOptions defines the strategy type and any options used to create the strategy.
Field
ranges
IDRange
array
ranges are the allowed ranges of uids that may be used. If you would like to force a single uid then supply a single range with the same start and end. Required for MustRunAs.
rule
string
rule is the strategy that will dictate the allowable RunAsUser values that may be set.
Group
Version
Kind
policy
v1beta1
RuntimeClassStrategyOptions define the strategy that will dictate the allowable RuntimeClasses for a pod.
Field
allowedRuntimeClassNames
allowedRuntimeClassNames is an allowlist of RuntimeClass names that may be specified on a pod. A value of "*" means that any RuntimeClass name is allowed, and must be the only item in the list. An empty list requires the RuntimeClassName field to be unset.
default

RuntimeClassName
string
defaultRuntimeClassName is the default RuntimeClassName to set on the pod. The default MUST be allowed by the allowedRuntimeClassNames list. A value of nil does not mutate the Pod.
Group
Version
Kind
core
v1
PodSecurityContext [core/v1]
SELinuxStrategyOptions [policy/v1beta1]
Field
level
string
role
string
type
string
user
string
Group
Version
Kind
policy
v1beta1
SELinuxStrategyOptions defines the strategy type and any options used to create the strategy.
Field
rule
string
rule is the strategy that will dictate the allowable labels that may be set.
seLinuxOptions required to run as; required for MustRunAs More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
Group
Version
Kind
v1
Scale
Scale represents a scaling request for a resource.
Field
string
kind
string
metadata
spec
ScaleSpec
defines the behavior of the scale. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status.
status
ScaleStatus
current status of the scale. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status. Read-only.
Group
Version
Kind
core
v1
Field
fsType
string
Filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Default is "xfs"
gateway
string
The host address of the ScaleIO API Gateway.
protectionDomain
string
The name of the ScaleIO Protection Domain for the configured storage.
readOnly
boolean
secretRef
SecretRef references to the secret for ScaleIO user and other sensitive information. If this is not provided, Login operation will fail.
sslEnabled
boolean
Flag to enable/disable SSL communication with Gateway, default false
storageMode
string
Indicates whether the storage for a volume should be ThickProvisioned or ThinProvisioned. Default is ThinProvisioned.
storagePool
string
The ScaleIO Storage Pool associated with the protection domain.
system
string
The name of the storage system as configured in ScaleIO.
string
The name of a volume already created in the ScaleIO system that is associated with this volume source.
Group
Version
Kind
core
v1
Field
fsType
string
Filesystem

 type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Default is "xfs".
gateway
string
string
readOnly
boolean
secretRef
boolean
string
string
system
string
string
Group
Version
Kind
v1
v1beta1
v1alpha1
Field
object
array
Group
Version
Kind
core
v1
ResourceQuotaSpec [core/v1]
Field
ScopedResourceSelectorRequirement
array
Group
Version
Kind
core
v1
ScopeSelector [core/v1]
Field
operator
string
scopeName
string
values
Group
Version
Kind
core
v1
Field
string
type
string
type indicates which kind of seccomp profile will be applied. Valid options are:  Localhost - a profile defined in a file on the node should be used. RuntimeDefault - the container runtime default profile should be used. Unconfined - no profile should be applied.
Group
Version
Kind
core
v1
SecretEnvSource
SecretEnvSource selects a Secret to populate the environment variables with.

The contents of the target Secret's Data field will represent th

e key-value pairs as environment variables.
Field
name
string
optional
boolean
Group
Version
Kind
core
v1
Field
key
string
The key of the secret to select from.  Must be a valid secret key.
name
string
optional
boolean
Group
Version
Kind
core
v1
SecretProjection
Adapts a secret into a projected volume.

The contents of the target Secret's Data field will be presented in a projected volume as files using the keys in the Data field as the file names. Note that this is identical to a secret volume source without the default mode.
Field
items
KeyToPath
array
If unspecified, each key-value pair in the Data field of the referenced Secret will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the Secret, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
name
string
optional
boolean
Group
Version
Kind
core
v1
CSIPersistentVolumeSource [core/v1]
CephFSPersistentVolumeSource [core/v1]
CinderPersistentVolumeSource [core/v1]
FlexPersistentVolumeSource [core/v1]
ISCSIPersistentVolumeSource [core/v1]
RBDPersistentVolumeSource [core/v1]
ScaleIOPersistentVolumeSource [core/v1]
Field
name
string
Name is unique within a namespace to reference a secret resource.
namespace
string
Namespace defines the space within which the secret name must be unique.
Group
Version
Kind
core
v1
Adapts a Secret into a volume.

The contents of the target Secret's Data field will be presented in a volume as files using the keys in the Data field as the file names. Secret volumes support ownership management and SELinux relabeling.
Field
integer
items
KeyToPath
array
optional
boolean
Specify whether the Secret or its keys must be defined
string
Name of the secret in the pod's namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret
Securit

yContext v1 core
Group
Version
Kind
core
v1
SecurityContext holds security configuration that will be applied to a container. Some fields are present in both SecurityContext and PodSecurityContext.  When both are set, the values in SecurityContext take precedence.
Field
boolean
AllowPrivilegeEscalation controls whether a process can gain more privileges than its parent process. This bool directly controls if the no_new_privs flag will be set on the container process. AllowPrivilegeEscalation is true always when the container is: 1) run as Privileged 2) has CAP_SYS_ADMIN
The capabilities to add/drop when running containers. Defaults to the default set of capabilities granted by the container runtime.
boolean
Run container in privileged mode. Processes in privileged containers are essentially equivalent to root on the host. Defaults to false.
procMount
string
procMount denotes the type of proc mount to use for the containers. The default is DefaultProcMount which uses the container runtime defaults for readonly paths and masked paths. This requires the ProcMountType feature flag to be enabled.
boolean
Whether this container has a read-only root filesystem. Default is false.
integer
The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
boolean
Indicates that the container must run as a non-root user. If true, the Kubelet will validate the image at runtime to ensure that it does not run as UID 0 (root) and fail to start the container if it does. If unset or false, no such validation will be performed. May also be set in PodSecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
runAsUser
integer
The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container.  May also be set in PodSecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
The seccomp options to use by this container. If seccomp options are provided at both the pod &amp; container level, the container options override the pod options.
The Windows specific settings applied to all containers. If unspecified, the options from the PodSecurityContext will be used. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
Group
Version
Kind
meta
v1
ServerAddressByClientCIDR helps the client to determine the server address that they should use, depending on the clientCIDR that they match.
APIVersions [meta/v1]
Field
clientCIDR
string
The CIDR with which clients can match their IP to figure out the server address that they should use.
serverAddress
string
Address of this server, suitable for a client that matches the above CIDR. This can be a hostname, hostname:port, IP or IP:port.
Group
Version
Kind
v1alpha1
An API server instance reports the version it can decode and the version it encodes objects to when persisting objects in the backend.
Appears 

In:
StorageVersionStatus [apiserverinternal/v1alpha1]
Field
apiServerID
string
The ID of the reporting API server.
decodableVersions
The API server can decode objects encoded in these versions. The encodingVersion must be included in the decodableVersions.
encodingVersion
string
The API server encodes the object to this version when persisting it in the backend (e.g., etcd).
Group
Version
Kind
v1beta1
ServiceAccountSubject
Field
name
string
`name` is the name of matching ServiceAccount objects, or "*" to match regardless of name. Required.
namespace
string
`namespace` is the namespace of matching ServiceAccount objects. Required.
Group
Version
Kind
core
v1
ServiceAccountTokenProjection
Field
audience
string
Audience is the intended audience of the token. A recipient of a token must identify itself with an identifier specified in the audience of the token, and otherwise should reject the token. The audience defaults to the identifier of the apiserver.
integer
ExpirationSeconds is the requested duration of validity of the service account token. As the token approaches expiration, the kubelet volume plugin will proactively rotate the service account token. The kubelet will start trying to rotate the token if the token is older than 80 percent of its time to live or if the token is older than 24 hours.Defaults to 1 hour and must be at least 10 minutes.
path
string
Path is the path relative to the mount point of the file to project the token into.
Group
Version
Kind
v1
IngressServiceBackend [networking/v1]
Field
name
string
number
integer
Group
Version
Kind
core
v1
ServiceSpec [core/v1]
Field
string
name
string
nodePort
integer
The port on each node on which this service is exposed when type is NodePort or LoadBalancer.  Usually assigned by the system. If a value is specified, in-range, and not in use it will be used, otherwise the operation will fail.  If not specified, a port will be allocated if this Service requires one.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type from NodePort to ClusterIP). More info: https://kubernetes.io/docs/concepts/ser

vices-networking/service/#type-nodeport
port
integer
protocol
string
Number or name of the port to access on the pods targeted by the service. Number must be in the range 1 to 65535. Name must be an IANA_SVC_NAME. If this is a string, it will be looked up as a named port in the target Pod's container ports. If this is not specified, the value of the 'port' field is used (an identity map). This field is ignored for services with clusterIP=None, and should be omitted or set equal to the 'port' field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#defining-a-service
Group
Version
Kind
v1
v1beta1
v1beta1
v1beta1
WebhookClientConfig [admissionregistration/v1]
Field
name
string
`name` is the name of the service. Required
namespace
string
`namespace` is the namespace of the service. Required
path
string
`path` is an optional URL path which will be sent in any request to this service.
port
integer
If specified, the port on the service that hosting webhook. Default to 443 for backward compatibility. `port` should be a valid port number (1-65535, inclusive).
Group
Version
Kind
core
v1
Field
clientIP
Group
Version
Kind
apps
v1
StatefulSetStatus [apps/v1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
apps
v1
Field
type
string
Group
Version
Kind
meta
v1
Status
Field
string
code
integer
details
StatusDetails
Extended data associated with the reason.  Each reason may define its own extended details. This field is optional and the data returned is not guaranteed to conform to any schema except that defined by the reason type.
kind
string
message
string
A human-readable desc

ription of the status of this operation.
metadata
ListMeta
reason
string
status
string
Status of the operation. One of: "Success" or "Failure". More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
Group
Version
Kind
meta
v1
StatusCause
StatusDetails [meta/v1]
Field
field
string
The field of the resource that has caused this error, as named by its JSON serialization. May include dot and postfix notation for nested attributes. Arrays are zero-indexed.  Fields may appear more than once in an array of causes due to fields having multiple errors. Optional.  Examples:   "name" - the field "name" on the current resource   "items[0].name" - the field "name" on the first array entry in "items"
message
string
A human-readable description of the cause of the error.  This field may be presented as-is to a reader.
reason
string
Group
Version
Kind
meta
v1
Field
causes
array
group
string
kind
string
The kind attribute of the resource associated with the status StatusReason. On some operations may differ from the requested resource Kind. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
name
string
retryAfterSeconds
integer
uid
string
UID of the resource. (when there is a single resource which can be described). More info: http://kubernetes.io/docs/user-guide/identifiers#uids
Group
Version
Kind
core
v1
Field
fsType
string
readOnly
boolean
secretRef
SecretRef specifies the secret to use for obtaining the StorageOS API credentials.  If not specified, default values will be attempted.
string
VolumeName is the human-readable name of the StorageOS volume.  Volume names are only unique within a namespace.
volumeNamespace
string
VolumeNamespace specifies the scope of the volume within StorageOS.  If no namespace is specified then the Pod's namespace will be used.  This

 allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.
Group
Version
Kind
core
v1
Field
fsType
string
readOnly
boolean
secretRef
string
string
VolumeNamespace specifies the scope of the volume within StorageOS.  If no namespace is specified then the Pod's namespace will be used.  This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.
Group
Version
Kind
v1alpha1
Describes the state of the storageVersion at a certain point.
Field
Time
message
string
integer
If set, this represents the .metadata.generation that the condition was set based upon.
reason
string
status
string
type
string
Type of the condition.
Group
Version
Kind
v1beta1
Subject
v1
v1alpha1
Field
group
kind
string
Required
user
UserSubject
Group
Version
Kind
v1
v1beta1
Field
string
incomplete
boolean
array
NonResourceRules is the list of actions the subject is allowed to perform on non-resources. The list ordering isn't significant, may contain

 duplicates, and possibly be incomplete.
array
Group
Version
Kind
policy
v1beta1
SupplementalGroupsStrategyOptions defines the strategy type and options used to create the strategy.
Field
ranges
IDRange
array
ranges are the allowed ranges of supplemental groups.  If you would like to force a single supplemental group then supply a single range with the same start and end. Required for MustRunAs.
rule
string
rule is the strategy that will dictate what supplemental groups is used in the SecurityContext.
Group
Version
Kind
core
v1
Sysctl
Field
name
string
value
string
Group
Version
Kind
core
v1
Field
host
string
port
Group
Version
Kind
core
v1
Taint
Field
effect
string
key
string
timeAdded
Time
value
string
Group
Version
Kind
meta
v1
Time
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON.  Wrappers are provided for many of the factory methods that the time package offers.
APIServiceCondition [apiregistration/v1]
APIServiceCondition [apiregistration/v1beta1]
CertificateSigningRequestCondition [certificates/v1]
CertificateSigningRequestCondition [certificates/v1beta1]
Condition [meta/v1]
ContainerStateRunning [core/v1]
ContainerStateTerminated [core/v1]
CustomResourceDefinitionCondition [apiextensions/v1]
CustomResourceDefinitionCondition [apiextensions/v1beta1]
DaemonSetCondition [apps/v1]
DeploymentCondition [apps/v1]
FlowSchemaCondition [flowcontrol/v1beta1]
HorizontalPodAutoscalerCondition [autoscaling/v2beta2]
HorizontalPodAutoscalerCondition [autoscaling/v2beta1]
HorizontalPodAutoscalerStatus [autoscaling/v1]
HorizontalPodAutoscalerStatus [autoscaling/v2beta1]
JobCondition [batch/v1]
NamespaceCondition [core/v1]
NodeCondition [core/v1]
PersistentVolumeClaimCondition [core/v1]
PodCondition [core/v1]
PriorityLevelConfigurationCondition [flowcontrol/v1beta1]
ReplicaSetCondition [apps/v1]
ReplicationControllerCondition [core/v1]
StatefulSetCondition [apps/v1]
StorageVersionCondition [apiserverinternal/v1alpha1]
Taint [core/v1]
TokenRequestStatus [authentication/v1]
VolumeError [storage/v1]
VolumeError [storage/v1beta1]
VolumeError [storage/v1alpha1]
Field
Group
Version
Kind
core
v1
The pod this Toleration is attached to tolerates any taint that matches the triple &lt;key,value,effect&gt; using the matching operator &lt;operator&gt;.
PodSpec [core/v

1]
Scheduling [node/v1]
Scheduling [node/v1beta1]
Scheduling [node/v1alpha1]
Field
effect
string
key
string
operator
string
integer
value
string
Group
Version
Kind
core
v1
TopologySelectorLabelRequirement
TopologySelectorTerm [core/v1]
Field
key
string
values
Group
Version
Kind
core
v1
Field
matchLabelExpressions
array
Group
Version
Kind
core
v1
Field
maxSkew
integer
MaxSkew describes the degree to which pods may be unevenly distributed. When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference between the number of matching pods in the target topology and the global minimum. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 1/1/0: | zone1 | zone2 | zone3 | |   P   |   P   |       | - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 1/1/1; scheduling it onto zone1(zone2) would make the ActualSkew(2-0) on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming pod can be scheduled onto any zone. When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence to topologies that satisfy it. It's a required field. Default value is 1 and 0 is not allowed.
string
TopologyKey is the key of node labels. Nodes that have a label with this key and identical values are considered to be in the same topology. We consider each &lt;key, value&gt; as a "bucket", and try to put balanced number of pods into each bucket. It's a required field.
whenUnsatisfiable
string
WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy the spread constraint. - DoNotSchedule (default) tells the scheduler not to schedule it. - ScheduleAnyway tells the scheduler to schedule the pod in a

ny location,   but giving higher precedence to topologies that would help reduce the   skew. A constraint is considered "Unsatisfiable" for an incoming pod if and only if every possible node assigment for that pod would violate "MaxSkew" on some topology. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 3/1/1: | zone1 | zone2 | zone3 | | P P P |   P   |   P   | If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler won't make it *more* imbalanced. It's a required field.
Group
Version
Kind
core
v1
IngressBackend [extensions/v1beta1]
IngressBackend [networking/v1beta1]
Field
apiGroup
string
kind
string
name
string
Group
Version
Kind
v1
UserInfo
v1beta1
TokenReviewStatus [authentication/v1]
Field
extra
object
groups
uid
string
username
string
Group
Version
Kind
v1beta1
Field
name
string
`name` is the username that matches, or "*" to match all usernames. Required.
Group
Version
Kind
v1
v1beta1
Field
string
string
matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent".  - Exact: match a request only if it exactly matches a specified rule. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, but "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/

v1beta1 would not be sent to the webhook.  - Equivalent: match a request if modifies a resource listed in rules, even via another API group or version. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, and "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/v1beta1 would be converted to apps/v1 and sent to the webhook.  Defaults to "Equivalent"
name
string
NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.  For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "runlevel",       "operator": "NotIn",       "values": [         "0",         "1"       ]     }   ] }  If instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "environment",       "operator": "In",       "values": [         "prod",         "staging"       ]     }   ] }  See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels for more examples of label selectors.  Default to the empty LabelSelector, which matches everything.
rules
array
string
integer
Group
Version
Kind
v1
VolumeAttachmentSource r

epresents a volume that should be attached. Right now only PersistenVolumes can be attached via external attacher, in future we may allow also inline volumes in pods. Exactly one member can be set.
v1beta1
v1alpha1
VolumeAttachmentSpec [storage/v1]
Field
inlineVolumeSpec
persistentVolumeName
string
Group
Version
Kind
core
v1
Field
string
name
string
Group
Version
Kind
v1
v1beta1
v1alpha1
VolumeAttachmentStatus [storage/v1]
Field
message
string
time
Time
Group
Version
Kind
core
v1
Field
mountPath
string
Path within the container at which the volume should be mounted.  Must not contain ':'.
mountPropagation
string
name
string
readOnly
boolean
subPath
string
string
Group
Version
Kind
core
v1
Field
required
Required specifies hard node constraints that must be met.
Group
Version
Kind
v1
v1beta1
CSINodeDriver [storage/v1]
Field
count
integer
Group
Version
Kind
core
v1
ProjectedVolumeSource [core/v1]
Field
configMap
information about the confi

gMap data to project
information about the downwardAPI data to project
secret
information about the secret data to project
information about the serviceAccountToken data to project
Group
Version
Kind
core
v1
Field
fsType
string
storagePolicyID
string
Storage Policy Based Management (SPBM) profile ID associated with the StoragePolicyName.
storagePolicyName
string
Storage Policy Based Management (SPBM) profile name.
volumePath
string
Path that identifies vSphere volume vmdk
Group
Version
Kind
meta
v1
Event represents a single event to a watched resource.
Field
object
Object is:  * If Type is Added or Modified: the new state of the object.  * If Type is Deleted: the state of the object immediately before deletion.  * If Type is Error: *Status is recommended; other types may make sense    depending on context.
type
string
Group
Version
Kind
v1
v1beta1
v1beta1
Field
caBundle
string
`caBundle` is a PEM encoded CA bundle which will be used to validate the webhook's server certificate. If unspecified, system trust roots on the apiserver are used.
service
`service` is a reference to the service for this webhook. Either `service` or `url` must be specified.  If the webhook is running within the cluster, then you should use `service`.
url
string
`url` gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified.  The `host` should not refer to a service running in the cluster; use the `service` field instead. The host might be resolved via external DNS in some apiservers (e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would be a layering violation). `host` may also be an IP address.  Please note that using `localhost` or `127.0.0.1` as a `host` is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster.  The scheme must be "https"; the URL must begin with "https://".  A path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier.  Attempting to use a user or basic auth e.g. "user:password@" is not allowed. Fragments ("#...") and query parameters ("?...") are not allowed, either.
WebhookConversion v1 apiextensions.k8s.io
Group
Version
Kind
v1
CustomResourceConversion [apiextensions/v1]
Field
clientConfig is the instructions for how to call the webhook if strategy is `Webhook`.
conversionReviewVersions is an ordered list of preferred `ConversionReview` versions the Webhook expects. The API server will use the first version in the list which it supports. If none of the versions specified in this list are supported by API server, conversion will fail for the custom resource. If a persisted Webhook configuration specifies allowed versions and does not inc

lude any versions known to the API Server, calls to the webhook will fail.
Group
Version
Kind
core
v1
Field
podAffinityTerm
weight
integer
Group
Version
Kind
core
v1
Field
gmsaCredentialSpec
string
GMSACredentialSpec is where the GMSA admission webhook (https://github.com/kubernetes-sigs/windows-gmsa) inlines the contents of the GMSA credential spec named by the GMSACredentialSpecName field.
string
string
This section contains older versions of resources shown above.
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
spec
status
APIServiceSpec v1beta1 apiregistration
Field
caBundle
string
group
string
integer
boolean
service
Service is a reference to the service for this API server.  It must communicate on port 443. If the Service is nil, that means the handling for the API groupversion is handled locally on this server. The ca

ll will simply delegate to the normal handler chain to be fulfilled.
version
string
integer
VersionPriority controls the ordering of this API version inside of its group.  Must be greater than zero. The primary sort is based on VersionPriority, ordered highest to lowest (20 before 10). Since it's inside of a group, the number can be small, probably in the 10s. In case of equal version priorities, the version string will be used to compute the order inside a group. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10.
APIServiceStatus v1beta1 apiregistration
Field
array
:
merge
:
type
APIServiceList v1beta1 apiregistration
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/apiregistration.k8s.io/v1beta1/apiservices
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/apiregistration.k8s.io/v1beta1/apiservices/{name}
Parameter
name
Parameter
pretty
dryRun
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be

 unset for non-apply patch requests.
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/apiregistration.k8s.io/v1beta1/apiservices/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/apiregistration.k8s.io/v1beta1/apiservices
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in t

he response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read
read the specified APIServic

e
GET /apis/apiregistration.k8s.io/v1beta1/apiservices/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/apiregistration.k8s.io/v1beta1/apiservices
Parameter
pretty
continue
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to uns

et
watch
Response
Code
200
OK
Watch
GET /apis/apiregistration.k8s.io/v1beta1/watch/apiservices/{name}
Parameter
name
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch List
GET /apis/apiregistration.k8s.io/v1beta1/watch/apiservices
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is s

pecified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
PATCH /apis/apiregistration.k8s.io/v1beta1/apiservices/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/apiregistration.k8s.io/v1beta1/apiservices/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
PUT /apis/apiregistration.k8s.io/v1beta1/apiservices/{name}/status
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as de

fined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta1
v1
APIServiceStatus [apiregistration/v1beta1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
array
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
array
Group
Version
Kind
v1beta1
CSIDriver
v1
Field
string
kind
string
metadata
spec
CSIDriverSpec v1beta1 storage
Field
boolean
string
Defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupP

olicy values for additional details. This field is alpha-level, and is only honored by servers that enable the CSIVolumeFSGroupPolicy feature gate.  This field is immutable.
boolean
If set to true, podInfoOnMount indicates this CSI volume driver requires additional pod information (like podName, podUID, etc.) during mount operations. If set to false, pod information will not be passed on mount. Default is false. The CSI driver specifies podInfoOnMount as part of driver deployment. If true, Kubelet will pass pod information as VolumeContext in the CSI NodePublishVolume() calls. The CSI driver is responsible for parsing and validating the information passed in as VolumeContext. The following VolumeConext will be passed if podInfoOnMount is set to true. This list might grow, but the prefix will be used. "csi.storage.k8s.io/pod.name": pod.Name "csi.storage.k8s.io/pod.namespace": pod.Namespace "csi.storage.k8s.io/pod.uid": string(pod.UID) "csi.storage.k8s.io/ephemeral": "true" if the volume is an ephemeral inline volume                                 defined by a CSIVolumeSource, otherwise "false"  "csi.storage.k8s.io/ephemeral" is a new feature in Kubernetes 1.16. It is only required for drivers which support both the "Persistent" and "Ephemeral" VolumeLifecycleMode. Other drivers can leave pod info disabled and/or ignore this field. As Kubernetes 1.15 doesn't support this field, drivers can only support one mode when deployed on such a cluster and the deployment determines which mode that is, for example via a command line parameter of the driver.  This field is immutable.
boolean
boolean
array
VolumeLifecycleModes defines what kind of volumes this CSI volume driver supports. The default if the list is empty is "Persistent", which is the usage defined by the CSI specification and implemented in Kubernetes via the usual PV/PVC mechanism. The other mode is "Ephemeral". In this mode, volum

es are defined inline inside the pod spec with CSIVolumeSource and their lifecycle is tied to the lifecycle of that pod. A driver has to be aware of this because it is only going to get a NodePublishVolume call for such a volume. For more information about implementing this mode, see https://kubernetes-csi.github.io/docs/ephemeral-local-volumes.html A driver can support one or more of these modes and more modes may be added in the future.  This field is immutable.
CSIDriverList v1beta1 storage
Field
string
items
CSIDriver
array
kind
string
metadata
ListMeta
Create
POST /apis/storage.k8s.io/v1beta1/csidrivers
Parameter
pretty
dryRun
Parameter
body
CSIDriver
Response
Code
200
CSIDriver
OK
201
CSIDriver
Created
202
CSIDriver
Accepted
Patch
PATCH /apis/storage.k8s.io/v1beta1/csidrivers/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
CSIDriver
OK
Replace
PUT /apis/storage.k8s.io/v1beta1/csidrivers/{name}
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less 

than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
CSIDriver
Response
Code
200
CSIDriver
OK
201
CSIDriver
Created
Delete
DELETE /apis/storage.k8s.io/v1beta1/csidrivers/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
CSIDriver
OK
202
CSIDriver
Accepted
DELETE /apis/storage.k8s.io/v1beta1/csidrivers
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for 

the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/storage.k8s.io/v1beta1/csidrivers/{name}
Parameter
name
Parameter
pretty
Response
Code
200
CSIDriver
OK
List
GET /apis/storage.k8s.io/v1beta1/csidrivers
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a ses

sion. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
watch
Response
Code
200
OK
Watch
H

TTP Request
GET /apis/storage.k8s.io/v1beta1/watch/csidrivers/{name}
Parameter
name
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/

api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
GET /apis/storage.k8s.io/v1beta1/watch/csidrivers
Parameter
continue
limit
pretty
resourceVersion s

ets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
CSINode
v1
Field
string
kind
string
metadata
spec
CSINodeSpec v1beta1 storage
Field
drivers
array
:
merge
:
name
CSINodeList v1beta1 storage
Field
string
items
CSINode
array
kind
string
metadata
ListMeta
Create
POST /apis/storage.k8s.io/v1beta1/csinodes
Parameter
pretty
dryRun
Parameter
body
CSINode
Response
Code
200
CSINode
OK
201
CSINode
Created
202
CSINode
Accepted
Patch
PATCH /apis/storage.k8s.io/v1beta1/csinodes/{name}
Parameter
name
Parameter
pretty
dryRun
When present, i

ndicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
CSINode
OK
Replace
PUT /apis/storage.k8s.io/v1beta1/csinodes/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
CSINode
Response
Code
200
CSINode
OK
201
CSINode
Created
Delete
DELETE /apis/storage.k8s.io/v1beta1/csinodes/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
CSINode
OK
202
CSINode
Accepted
DELETE /apis/storage.k8s.io/v1beta1/csinodes
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a con

tinue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersion sets a constraint on what resource versions a request may be serv

ed from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/storage.k8s.io/v1beta1/csinodes/{name}
Parameter
name
Parameter
pretty
Response
Code
200
CSINode
OK
List
GET /apis/storage.k8s.io/v1beta1/csinodes
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects 

created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
watch changes to an object of kind CSINode. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/storage.k8s.io/v1beta1/watch/csinodes/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients sh

ould only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/storage.k8s.io/v1beta1/watch/csinodes
Parameter
continue
A selector to restrict the list of returned objects by the

ir labels. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
CSINodeSpec [storage/v1beta1]
Field
allocatable represents the volume resources of a node that are available for scheduling.
name
string
nodeID
string
topologyKeys is the list of keys supported by the driver. When a driver is initialized on a cluster, it provides a set of topology keys that it understands (e.g. "company.com/zone", "company.com/region"). When a driver is initialized on a node, it provides the same topology keys along with values. Kubelet will expose these topology keys as labels on its own node object. When Kubernetes does topology aware provisioning, it can use this list to determine which labels it should retrieve from the node object and pass back to the driver. It is possible for different nodes to use different topology keys. This can be empty 

if driver does not support topology.
Group
Version
Kind
v1alpha1
v1beta1
Field
string
capacity
Quantity
kind
string
Quantity
metadata
string
CSIStorageCapacityList v1alpha1 storage
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
object name and auth scope, such as 

for teams and projects
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specif

ied type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limi

t is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/storage.k8s.io/v1alpha1/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (

generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
GET /apis/storage.k8s.io/v1alpha1/csistoragecapacities
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/storage.k8s.io/v1alpha1/watch/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
D

name
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#reso

urce-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
GET /apis/storage.k8s.io/v1alpha1/watch/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the 

first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/storage.k8s.io/v1alpha1/watch/csistoragecapacities
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuin

g a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
spec
The certificate request itself and any additional information.
status
Derived information about the request.
CertificateSigningRequestSpec v1beta1 certificates
CertificateSigningRequest [certificates/v1beta1]
Field
extra
object
Extra information about the requesting user. See user.Info interface for details.
groups
Group information about the requesting user. See user.Info interface for details.
request
string
Base64-encoded PKCS#10 CSR data
string
Requested signer for the request. It is a qualified name in the form: `scope-hostname.io/name`. If empty, it will be defaulted:  1. If it's a kubelet client certificate, it is assigned     "kubernetes.io/kube-apiserver-client-kubelet".  2. If it's a kubelet serving certificate, it is assigned     "kubernetes.io/kubelet-serving".  3. Otherwise, it is assigned "kubernetes.io/legacy-unknown". Distribution of trust for signers happens out of band. You can select on this field using `spec.signerName`.
uid
string
UID information about the requesting user. See user.Info interface for details.
usages
allowedUsages specifies a set of usage contexts the key will be valid for. See: https://tools.ietf.org/html/rfc5280#section-4.2.1.3      https://tools.ietf.org/html/rfc5280#section-4.2.1.12 Valid values are:  "signing",  "digital signature",  "content commitment",  "key encipherment",  "key agreement",  "data encipherment",  "cert sign",  "crl sign",  "encipher only",  "decipher only",  "any",  "server auth",  "client auth",  "code signing",  "email protection",  "s/mime",  "ipsec end system",  "ipse

c tunnel",  "ipsec user",  "timestamping",  "ocsp signing",  "microsoft sgc",  "netscape sgc"
username
string
Information about the requesting user. See user.Info interface for details.
CertificateSigningRequestStatus v1beta1 certificates
Field
string
If request was approved, the controller will place the issued certificate here.
array
Conditions applied to the request, such as approval or denial.
CertificateSigningRequestList v1beta1 certificates
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/certificates.k8s.io/v1beta1/certificatesigningrequests
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/{name}
Parameter
name
Parameter
pretty
dryRun

Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/certificates.k8s.io/v1beta1/certificatesigningrequests
Parameter
pretty
continue
dryRun
A select

or to restrict the list of returned objects by their fields. Defaults to everything.
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/certificates.k8s.io/v1beta1/certificatesigningrequests
Parameter
Descr

iption
pretty
continue
limit
Timeout for the list/watch call. This limits the duration of the call, regardless of an

y activity or inactivity.
watch
Response
Code
200
OK
Watch
GET /apis/certificates.k8s.io/v1beta1/watch/certificatesigningrequests/{name}
Parameter
name
Parameter
continue
limit
pretty
r

esourceVersion
watch
Response
Code
200
OK
Watch List
GET /apis/certificates.k8s.io/v1beta1/watch/certificatesigningrequests
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the

 first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
PATCH /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
PUT /apis/certificates.k8s.io/v1beta1/certificatesigningrequests/{name}/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201

Created
Group
Version
Kind
v1beta1
v1
CertificateSigningRequestStatus [certificates/v1beta1]
Field
Time
Time
timestamp for the last update to this condition
message
string
human readable message with details about the request state
reason
string
brief reason for the request state
status
string
Status of the condition, one of True, False, Unknown. Approved, Denied, and Failed conditions may not be "False" or "Unknown". Defaults to "True". If unset, should be treated as "True".
type
string
type of the condition. Known conditions include "Approved", "Denied", and "Failed".
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
string
kind
string
metadata
rules
array
ClusterRoleList v1beta1 rbac
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1beta1/clusterroles
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1beta1/clust

erroles/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1beta1/clusterroles/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1beta1/clusterroles/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1beta1/clusterroles
Parameter
pretty
continue
The cont

inue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/rbac.authorization.k8s.io/v1beta1/clusterroles/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/rbac.authorization.k8s.io/v1beta1/clusterroles
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available

 results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/clusterroles/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to retur

n for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/clusterroles
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are 

after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
string
kind
string
metadata
rules
array
ClusterRoleList v1alpha1 rbac
Field
string
APIVersion defines the versioned s

chema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
items
array
kind
string
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1alpha1/clusterroles
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1alpha1/clusterroles/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1alpha1/clusterroles/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/clusterroles/{name}
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or

 unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/clusterroles
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event a

ll requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/rbac.authorization.k8s.io/v1alpha1/clusterroles/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/rbac.authorization.k8s.io/v1alpha1/clusterroles
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to 

expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterroles/{name}
Parameter
name
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a 

session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List

GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterroles
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to

 unset
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
ClusterRoleBindingList v1beta1 rbac
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply re

quests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without th

e continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/doc

s/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/rbac.authorization.k8s.io/v1beta1/clusterrolebindings
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large res

ult can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/clusterrolebindings/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is sp

ecified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/clusterrolebindings
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the li

st metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
ClusterRoleBindingList v1alpha1 rbac
Field
string
items
array
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventio

ns.md#types-kinds
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. ze

ro means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects retur

ned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/rbac.authorization.k8s.io/v1alpha1/clusterrolebindings
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list start

ing from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterrolebindings/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for

 the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/clusterrolebindings
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at an

y specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
ContainerResourceMetricSource v2beta1 aut

oscaling
Group
Version
Kind
v2beta1
v2beta2
MetricSpec [autoscaling/v2beta1]
Field
container
string
name
string
integer
targetAverageUtilization is the target value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods.
Quantity
targetAverageValue is the target value of the average of the resource metric across all relevant pods, as a raw value (instead of as a percentage of the request), similar to the "pods" metric source type.
Group
Version
Kind
v2beta1
v2beta2
MetricStatus [autoscaling/v2beta1]
Field
container
string
currentAverageUtilization
integer
currentAverageUtilization is the current value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods.  It will only be present if `targetAverageValue` was set in the corresponding metric specification.
currentAverageValue
Quantity
currentAverageValue is the current value of the average of the resource metric across all relevant pods, as a raw value (instead of as a percentage of the request), similar to the "pods" metric source type. It will always be set, regardless of the corresponding metric specification.
name
string
Group
Version
Kind
batch
v1beta1
CronJob
v1
Field
string
kind
string
metadata
spec
status
CronJobSpec v1beta1 batch
Field
string
integer
The number of failed finished jobs to retain. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1.
schedule
string
integer
Optional deadline in seconds for starting the job if it misses scheduled time for any reason.  Missed jobs execut

ions will be counted as failed ones.
integer
The number of successful finished jobs to retain. This is a pointer to distinguish between explicit zero and not specified. Defaults to 3.
suspend
boolean
CronJobStatus v1beta1 batch
Field
active
array
Time
Time
CronJobList v1beta1 batch
Field
string
items
CronJob
array
kind
string
metadata
ListMeta
Create
POST /apis/batch/v1beta1/namespaces/{namespace}/cronjobs
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
202
CronJob
Accepted
Patch
PATCH /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
CronJob
OK
Replace
PUT /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
Parameter
name
namespace
object name and auth scope, such as for tea

ms and projects
Parameter
pretty
dryRun
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
Delete
DELETE /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/batch/v1beta1/namespaces/{namespace}/cronjobs
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a w

atch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}
Parameter
name
name of the C

ronJob
namespace
Parameter
pretty
Response
Code
200
CronJob
OK
List
GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs
Parameter
namespace
Parameter
pretty
continue
limit

watch
Response
Code
200
OK
GET /apis/batch/v1beta1/cronjobs
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during

 a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
GET /apis/batch/v1beta1/watch/namespaces/{namespace}/cronjobs/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit

 is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/batch/v1beta1/watch/namespaces/{namespace}/cronjobs
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to r

eturn for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/batch/v1beta1/watch/cronjobs
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the

 "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
PATCH /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
CronJob
OK
read 

status of the specified CronJob
GET /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status
Parameter
name
namespace
Parameter
pretty
Response
Code
200
CronJob
OK
PUT /apis/batch/v1beta1/namespaces/{namespace}/cronjobs/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
Group
Version
Kind
v2beta2
v1
v2beta1
Field
string
kind
string
name
string
Group
Version
Kind
v2beta1
v1
v2beta2
HorizontalPodAutoscalerSpec [autoscaling/v2beta1]
Field
string
kind
string
name
string
Group
Version
Kind
v1beta1
v1
CustomResourceDefinitionSpec [apiextensions/v1beta1]
CustomResourceDefinitionVersion [apiextensions/v1beta1]
Field
JSONPath
string
JSONPath is a simple JSON path (i.e. with array notation) which is evaluated against each custom resource to produce the value for this column.
string
format
string
format is an optional OpenAPI type definition for this column. The 'name' format is applied to the primary identifier column to assist in clients identifying column is the resource name. See https://github.com/OAI/OpenAPI-Specification/blob/master/versions/2.0.md#data-types for details.
name
string
priority
integer
type
string
Group
Version
Ki

nd
v1beta1
v1
Field
conversionReviewVersions is an ordered list of preferred `ConversionReview` versions the Webhook expects. The API server will use the first version in the list which it supports. If none of the versions specified in this list are supported by API server, conversion will fail for the custom resource. If a persisted Webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail. Defaults to `["v1beta1"]`.
strategy
string
strategy specifies how custom resources are converted between versions. Allowed values are: - `None`: The converter only change the apiVersion and would not touch any other field in the custom resource. - `Webhook`: API Server will call to an external webhook to do the conversion. Additional information   is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.webhookClientConfig to be set.
webhookClientConfig is the instructions for how to call the webhook if strategy is `Webhook`. Required when `strategy` is set to `Webhook`.
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
spec
status
CustomResourceDefinitionSpec v1beta1 apiextensions
Field
array
additionalPrinterColumns specifies additional columns returned in Table output. See https://kubernetes.io/docs/reference/using-api/api-concepts/#receiving-resources-as-tables for details. If present, this field configures columns for all versions. Top-level and per-version columns are mutually exclusive. If no top-level or per-version columns are specified, a single column displaying the age of the custom resource is used.
group
string
names
boolean
preserveUnknownFields indicates that object fields which are not specified in the OpenAPI schema should be preserved when persisting to storage. apiVersion, kind, metadata and known fields inside metadata are always preserved. If false, schemas must be defined for all versions. Defaults to true in v1beta for backwards compatibility. Deprecated: will be required to be false in v1. Preservation of unknown fields can be specified in the validation schema using the `x-kubernetes-preserve-unknown-field

s: true` extension. See https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/#pruning-versus-preserving-unknown-fields for details.
scope
string
scope indicates whether the defined custom resource is cluster- or namespace-scoped. Allowed values are `Cluster` and `Namespaced`. Default is `Namespaced`.
subresources specify what subresources the defined custom resource has. If present, this field configures subresources for all versions. Top-level and per-version subresources are mutually exclusive.
validation describes the schema used for validation and pruning of the custom resource. If present, this validation schema is used to validate all versions. Top-level and per-version schemas are mutually exclusive.
version
string
version is the API version of the defined custom resource. The custom resources are served under `/apis/&lt;group&gt;/&lt;version&gt;/...`. Must match the name of the first item in the `versions` list if `version` and `versions` are both specified. Optional if `versions` is specified. Deprecated: use `versions` instead.
versions
array
versions is the list of all API versions of the defined custom resource. Optional if `version` is specified. The name of the first item in the `versions` list must match the `version` field if `version` and `versions` are both specified. Version names are used to compute the order in which served versions are listed in API discovery. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10.
CustomResourceDefinitionStatus v1beta1 apiextensions
Field
array
CustomResourceDefinitionList v1beta1 apiextensions
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions
Parameter
pret

ty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}
Parameter
name
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent obj

ects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after th

e first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are

 created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/apiextensions.k8s.io/v1beta1/watch/customresourcedefinitions/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified

 continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/apiextensions.k8s.io/v1beta1/watch/customresourcedefinitions
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. I

f this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
PATCH /apis/apiextensions.k8s.i

o/v1beta1/customresourcedefinitions/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
PUT /apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions/{name}/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta1
v1
CustomResourceDefinitionStatus [apiextensions/v1beta1]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1beta1
v1
Field
kind
string
kind is the serialized kind of the resource. It is normally CamelCase and singular. Custom resource instances will use this value as

 the `kind` attribute in API calls.
listKind
string
plural
string
shortNames are short names for the resource, exposed in API discovery documents, and used by clients to support invocations like `kubectl get &lt;shortname&gt;`. It must be all lowercase.
singular
string
Group
Version
Kind
v1beta1
v1
Field
array
additionalPrinterColumns specifies additional columns returned in Table output. See https://kubernetes.io/docs/reference/using-api/api-concepts/#receiving-resources-as-tables for details. Top-level and per-version columns are mutually exclusive. Per-version columns must not all be set to identical values (top-level columns should be used instead). If no top-level or per-version columns are specified, a single column displaying the age of the custom resource is used.
boolean
string
name
string
schema
schema describes the schema used for validation and pruning of this version of the custom resource. Top-level and per-version schemas are mutually exclusive. Per-version schemas must not all be set to identical values (top-level validation schema should be used instead).
served
boolean
storage
boolean
subresources specify what subresources this version of the defined custom resource have. Top-level and per-version subresources are mutually exclusive. Per-version subresources must not all be set to identical values (top-level subresources should be used instead).
Group
Version
Kind
v1beta1
v1
CustomResourceSubresources [apiextensions/v1beta1]
Field
string
labelSelectorPath defines the JSON path inside of a custom resource that corresponds to Scale `status.selector`. Only JSON paths without the array notation are allowed. Must be a JSON Path under `.status` or `.spec`. Must be set to work with HorizontalPodAutoscaler. The field pointed by this JSON path must be a string field (not a complex selector struct) which contains a serialized label selector in string form. More info: https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions#scale-subresource If there is no value under the given path in the c

ustom resource, the `status.selector` value in the `/scale` subresource will default to the empty string.
string
string
Group
Version
Kind
v1beta1
v1
Field
Group
Version
Kind
v1beta1
v1
Field
scale
status
Group
Version
Kind
v1beta1
v1
Field
Group
Version
Kind
v1beta1
Endpoint
v1
Field
addresses
hints
hostname
string
nodeName
string
targetRef
topology
object
topology contains arbitrary topology information associated with the endpoint. These key/value pairs must conform with the label format. https://kubernetes.io/docs/concepts/overview/working-with-objects/labels Topology may include a maximum of 16 key/value pairs. This includes, but is not limit

ed to the following well known keys: * kubernetes.io/hostname: the value indicates the hostname of the node   where the endpoint is located. This should match the corresponding   node label. * topology.kubernetes.io/zone: the value indicates the zone where the   endpoint is located. This should match the corresponding node label. * topology.kubernetes.io/region: the value indicates the region where the   endpoint is located. This should match the corresponding node label. This field is deprecated and will be removed in future api versions.
Group
Version
Kind
v1beta1
v1
Field
ready
boolean
serving
boolean
boolean
Group
Version
Kind
v1beta1
v1
Field
forZones
ForZone
array
forZones indicates the zone(s) this endpoint should be consumed by to enable topology aware routing. May contain a maximum of 8 entries.
Group
Version
Kind
v1
v1beta1
Field
string
The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and http://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.
name
string
port
integer
protocol
string
Group
Version
Kind
v1beta1
v1
v1
Field
string
name
string
The name of this port. All ports in an EndpointSlice must have a unique name. If the EndpointSlice is dervied from a Kubernetes service, this correspond

s to the Service.ports[].name. Name must either be an empty string or pass DNS_LABEL validation: * must be no more than 63 characters long. * must consist of lower case alphanumeric characters or '-'. * must start and end with an alphanumeric character. Default is empty string.
port
integer
protocol
string
Group
Version
Kind
v1beta1
v1
Field
string
string
endpoints
Endpoint
array
kind
string
metadata
ports
array
EndpointSliceList v1beta1 discovery
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/discovery.k8s.io/v1beta1/namespaces/{namespace}/endpointslices
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
partially update the spec

ified EndpointSlice
PATCH /apis/discovery.k8s.io/v1beta1/namespaces/{namespace}/endpointslices/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/discovery.k8s.io/v1beta1/namespaces/{namespace}/endpointslices/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/discovery.k8s.io/v1beta1/namespaces/{namespace}/endpointslices/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the fore

ground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/discovery.k8s.io/v1beta1/namespaces/{namespace}/endpointslices
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be or

phaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/discovery.k8s.io/v1beta1/namespaces/{namespace}/endpointslices/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/discovery.k8s.io/v1beta1/namespaces/{namespace}/endpointslices
Parameter
namespace
Parameter
pretty
continue
A selector to restrict the list of returned objects by their fields. Defaults to everythi

ng.
limit
watch
Response
Code
200
OK
GET /apis/discovery.k8s.io/v1beta1/endpointslices
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in th

e response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/discovery.k8s.io/v1beta1/watch/namespaces/{namespace}/endpointslices/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. I

f the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/discovery.k8s.io/v1beta1/watch/namespaces/{namespace}/endpointslices
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks

 are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List All Namespa

ces
GET /apis/discovery.k8s.io/v1beta1/watch/endpointslices
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersi

on is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Event
v1beta1
Field
action
string
What action was taken/failed regarding to the Regarding object.
string
count
integer
The number of times this event has occurred.
eventTime
MicroTime
Time when this Event was first observed.
Time
The time at which the event was first recorded. (Time of server receipt is in TypeMeta.)
The object that this event is about.
kind
string
Time
The time at which the most recent occurrence of this event was recorded.
message
string
metadata
reason
string
This should be a short, machine understandable string that gives the reason for the transition into the object's current status.
related
Optional secondary object for more complex actions.
reportingComponent
string
Name of the controller that emitted this Event, e.g. `kubernetes.io/kubelet`.
string
ID of the controller instance, e.g. `kubelet-xyzf`.
series
Data about the Event series this event represents or nil if it's a singleton Event.
source
The component reporting this event. Should be a short machine understandable string.
type
string
Type of this event (Normal, Warning), new types could be added in the future
EventList v1 core
Field
string
items
Event
array
List of events
kind
string
metadata
ListMeta
Create
POST /api/v1/namespaces/{namespace}/events
Parameter
namespace
Parameter
pretty
dryRun
fieldManager is

 a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
202
Event
Accepted
Patch
PATCH /api/v1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Event
OK
Replace
PUT /api/v1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
Delete
DELETE /api/v1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable val

ues are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /api/v1/namespaces/{namespace}/events
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result

 was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /api/v1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Event
OK
List
GET /api/v1/namespaces/{namespace}/events
Parameter
namespace
Parameter
pretty
continue
A selector to restr

ict the list of returned objects by their fields. Defaults to everything.
limit
watch
Response
Code
200
EventList
OK
GET /api/v1/events
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request

 will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
EventList
OK
Watch
watch changes to an object of kind Event. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /api/v1/watch/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value i

s no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /api/v1/watch/namespaces/{namespace}/events
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server 

will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
watch individual changes to a list of Event. deprecated: use the 'w

atch' parameter with a list operation instead.
GET /api/v1/watch/events
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Def

aults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
Event
v1
v1
Field
action
string
action is what action was taken/failed regarding to the regarding object. It is machine-readable. This field can have at most 128 characters.
string
integer
Time
Time
eventTime
MicroTime
kind
string
metadata
note
string
reason
string
reason is why the action was taken. It is human-readable. This field can have at most 128 characters.
regarding
related
string
string
series
type
string
type is the type of this event (Normal, Warning), new types could be added in the future. It is machine-readable.
EventList v1beta1 events
Field
string
items
Event
array
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8

s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Create
POST /apis/events.k8s.io/v1beta1/namespaces/{namespace}/events
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
202
Event
Accepted
Patch
PATCH /apis/events.k8s.io/v1beta1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Event
OK
Replace
PUT /apis/events.k8s.io/v1beta1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
Delete
DELETE /apis/events.k8s.io/v1beta1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
The 

duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/events.k8s.io/v1beta1/namespaces/{namespace}/events
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the 

continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/events.k8s.io/v1beta1/namespaces/{namespace}/events/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Event
OK
List
GET /apis/events.k8s.io/v1beta1/namespaces/{namespace}/events
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not rec

ognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
EventList
OK
GET /apis/events.k8s.io/v1beta1/events
Parameter
continue
The cont

limit
pretty
watch
Response
Code
200
EventList
OK
Watch
GET /apis/events.k8s.io/v1beta1/watch/namespaces/{namespace}/events/{name}
Parameter

name
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versio

ns for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
GET /apis/events.k8s.io/v1beta1/watch/namespaces/{namespace}/events
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is retu

rned.
pretty
watch
Response
Code
200
OK
GET /apis/events.k8s.io/v1beta1/watch/events
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, m

odified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
v1beta1
Field
count
integer
Number of occurrences in this series up to the last heartbeat time
MicroTime
Time of the last occurrence observed
Group
Version
Kind
v1beta1
v1
v1
Field
count
integer
MicroTime
Group
Version
Kind
v1beta1
v1
JSONSchemaProps [apiextensions/v1beta1]
Field
string
url
string
Group
Version
Kind
v2beta1
v2beta2
Field
metricName
string
metricName is the name of the metric in question.
metricSelector is used to identify a specific time series within a given metric.
Quantity
targetAverageValue is the target per-pod value of global metric (as a quantity). Mutually exclusive with TargetValue.
targetValue
Quantity
targetValue is the target value of the metric (as a quantity). Mutually exclusive with TargetAverageValue.
Group
Version
Kind
v2beta1
v2beta2
Field
Quantity
currentAverageValue is the current value of metric averaged over autoscaled pods.
currentValue
Quantity
currentValue is the current value of the metric (as a quantity)
string
metricName is the name of a metric used for autoscaling in metric system.
ForZone v1beta1 discovery.k8s.io
Group
Version
Kind
v1beta1
ForZone
v1
EndpointHints [discovery/v1beta1]
Field
name
string
Group
Version
Kind
v1beta1
v1
HTTPIng

ressRuleValue [extensions/v1beta1]
Field
backend
path
string
pathType
string
PathType determines the interpretation of the Path matching. PathType can be one of the following values: * Exact: Matches the URL path exactly. * Prefix: Matches based on a URL path prefix split by '/'. Matching is   done on a path element by element basis. A path element refers is the   list of labels in the path split by the '/' separator. A request is a   match for path p if every p is an element-wise prefix of p of the   request path. Note that if the last element of the path is a substring   of the last element in request path, it is not a match (e.g. /foo/bar   matches /foo/bar/baz, but does not match /foo/barbaz). * ImplementationSpecific: Interpretation of the Path matching is up to   the IngressClass. Implementations can treat this as a separate PathType   or treat it identically to Prefix or Exact path types. Implementations are required to support all path types. Defaults to ImplementationSpecific.
Group
Version
Kind
v1beta1
v1
HTTPIngressRuleValue [networking/v1beta1]
Field
backend
path
string
pathType
string
Group
Version
Kind
v1beta1
v1
IngressRule [extensions/v1beta1]
Field
paths
array
Group
Version
Kind
v1beta1
v1
IngressRule [networking/v1beta1]
Field
paths
array
Group
Version
Kind
v2beta2
v1
v2beta1
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/communit

y/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
metadata
metadata is the standard object metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
spec is the specification for the behaviour of the autoscaler. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status.
status
HorizontalPodAutoscalerSpec v2beta2 autoscaling
Field
behavior
integer
metrics
array
metrics contains the specifications for which to use to calculate the desired replica count (the maximum replica count across all metrics will be used).  The desired replica count is calculated multiplying the ratio between the target value and the current value by the current number of pods.  Ergo, metrics used must decrease as the pod count is increased, and vice-versa.  See the individual metric source types for more information about how each type of metric must respond. If not set, the default metric will be set to 80% average CPU utilization.
integer
HorizontalPodAutoscalerStatus v2beta2 autoscaling
Field
array
array
integer
integer
Time
integer
HorizontalPodAutoscalerList v2beta2 autoscaling
Field
string
items
array
kind
string
Kind is a string

 value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Create
POST /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
nam

espace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
pretty
continue
dryRun
A selector to restrict the list of returned objects by th

eir labels. Defaults to everything.
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookm

arks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
watch
Response
Code
200
OK
list or watch objects of kind Hori

zontalPodAutoscaler
GET /apis/autoscaling/v2beta2/horizontalpodautoscalers
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  

watch
Response
Code
200
OK
Watch
GET /apis/autoscaling/v2beta2/watch/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objec

ts are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/autoscaling/v2beta2/watch/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the contin

ue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/autoscaling/v2beta2/watch/horizontalpodautoscalers
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata

 to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
PATCH /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
replace stat

us of the specified HorizontalPodAutoscaler
PUT /apis/autoscaling/v2beta2/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v2beta1
v1
v2beta2
Field
string
kind
string
metadata
spec
status
HorizontalPodAutoscalerSpec v2beta1 autoscaling
Field
integer
metrics
array
metrics contains the specifications for which to use to calculate the desired replica count (the maximum replica count across all metrics will be used).  The desired replica count is calculated multiplying the ratio between the target value and the current value by the current number of pods.  Ergo, metrics used must decrease as the pod count is increased, and vice-versa.  See the individual metric source types for more information about how each type of metric must respond.
integer
HorizontalPodAutoscalerStatus v2beta1 autoscaling
Field
array
conditions is the set of conditions required for this autoscaler to scale its target, and indicates whether or

 not those conditions are met.
array
integer
integer
Time
integer
HorizontalPodAutoscalerList v2beta1 autoscaling
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT 

/apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond 

with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Timeout for the list/watch call. This

 limits the duration of the call, regardless of any activity or inactivity.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and e

nsures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
GET /apis/autoscaling/v2beta1/horizontalpodautoscalers
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported

 if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
GET /apis/autoscaling/v2beta1/watch/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maxi

mum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/autoscaling/v2beta1/watch/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsiste

nt from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/autoscaling/v2beta1/watch/horizontalpodautoscalers
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
PATCH /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all d

ry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
PUT /apis/autoscaling/v2beta1/namespaces/{namespace}/horizontalpodautoscalers/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v2beta1
v2beta2
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1beta1
Ingress
v1
Field
string
kind
string
metadata
spec
Spec is the desired state of the Ingress. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/

api-conventions.md#spec-and-status
status
IngressSpec v1beta1 extensions
Field
backend
A default backend capable of servicing requests that don't match any rule. At least one of 'backend' or 'rules' must be specified. This field is optional to allow the loadbalancer controller or defaulting logic to specify a global default.
string
rules
array
tls
array
IngressStatus v1beta1 extensions
Field
IngressList v1beta1 extensions
Field
string
items
Ingress
array
kind
string
metadata
ListMeta
Create
POST /apis/extensions/v1beta1/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
202
Ingress
Accepted
Patch
PATCH /apis/extensions/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Query 

Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
Replace
PUT /apis/extensions/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
Delete
DELETE /apis/extensions/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/extensions/v1beta1/namespaces/{namespace}/ingresses
Parameter
namespace
object name and auth scope, suc

h as for teams and projects
Parameter
pretty
continue
dryRun
limit
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the ex

isting finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/extensions/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
List
GET /apis/extensions/v1beta1/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer 

than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
GET /apis/extensions/v1beta1/ingresses
Parameter
continue
A selector to restrict the list of returned objects by their labels.

 Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/extensions/v1beta1/watch/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server w

ill respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/extensions/v1beta1/watch/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical 

query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/extensions/v1beta1/watch/ingresses
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned

 at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
part

ially update status of the specified Ingress
PATCH /apis/extensions/v1beta1/namespaces/{namespace}/ingresses/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
GET /apis/extensions/v1beta1/namespaces/{namespace}/ingresses/{name}/status
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
PUT /apis/extensions/v1beta1/namespaces/{namespace}/ingresses/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
Group
Version
Kind
v1beta1
Ingress
v1
Field
string
kind
string
metadata
spec
status
IngressSpec v1beta1 networki

ng
Field
backend
string
rules
array
tls
array
IngressStatus v1beta1 networking
Field
IngressList v1beta1 networking
Field
string
items
Ingress
array
kind
string
metadata
ListMeta
Create
POST /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
202
Ingress
Accepted
Patch
PATCH /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an er

ror response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
Replace
PUT /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
Delete
DELETE /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
pretty
continue

dryRun
limit
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the 

dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
List
GET /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether 

more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
GET /apis/networking.k8s.io/v1beta1/ingresses
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to 

a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
GET /apis/networking.k8s.io/v1beta1/watch/namespaces/{namespace}/ingresses/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or 

deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/networking.k8s.io/v1beta1/watch/namespaces/{namespace}/ingresses
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid 

whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/networking.k8s.io/v1beta1/watch/ingresses
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the fea

ture gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
PATCH /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses/{name}/status
Par

ameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
GET /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses/{name}/status
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
PUT /apis/networking.k8s.io/v1beta1/namespaces/{namespace}/ingresses/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
Group
Version
Kind
v1beta1
v1
HTTPIngressPath [extensions/v1beta1]
IngressSpec [extensions/v1beta1]
Field
resource
Resource is an ObjectRef to another Kubernetes resource in the namespace of the Ingress object. If resource is specified, serviceName and servicePort must not be specified.
string
Specifies the name of the referenced service.
Specifies the port of the referenced service.
Group
Version
Kind
v1beta1
v1
HTTPIngressPath [networking/v1beta1]
IngressSpec [networking/v1beta1]
Field
resource
string
Group
Version
Kind
v1beta1
v1
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recogni

zed schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
metadata
spec
IngressClassSpec v1beta1 networking
Field
string
IngressClassList v1beta1 networking
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/networking.k8s.io/v1beta1/ingressclasses
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/networking.k8s.io/v1beta1/ingressclasses/{name}
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests 

(application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/networking.k8s.io/v1beta1/ingressclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/networking.k8s.io/v1beta1/ingressclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/networking.k8s.io/v1beta1/ingressclasses
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with 

the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
timeoutSecond

s
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/networking.k8s.io/v1beta1/ingressclasses/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/networking.k8s.io/v1beta1/ingressclasses
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the

 first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
GET /apis/networking.k8s.io/v1beta1/watch/ingressclasses/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the object

s returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/networking.k8s.io/v1beta1/watch/ingressclasses
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the 

event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
IngressClassSpec [networking/v1beta1]
Field
apiGroup
string
kind
string
name
string
namespace
string
scope
string
Group
Version
Kind
v1beta1
v1
Field
host
string
Host is the fully qualified domain name of a network host, as defined by RFC 3986. Note the following deviations from the "host" part of the URI as defined in RFC 3986: 1. IPs are not allowed. Currently an IngressRuleValue can only apply to    the IP in the Spec of the parent Ingress. 2. The `:` delimiter is not respected because ports are not allowed. 	  Currently the port of an Ingress is implicitly :80 for http and 	  :443 for https. Both these may change in the future. Incoming requests are matched against the host before the IngressRuleValue. If the host is unspecified, the Ingress routes all traffic based on the specified IngressRuleValue.  Host can be "precise" which is a domain name without the terminating dot of a network host (e.g. "foo.bar.com") or "wildcard", which is a domain name prefixed with a single wildcard label (e.g. "*.foo.com"). The wildcard character '\*' mus

t appear by itself as the first DNS label and matches only a single label. You cannot have a wildcard label by itself (e.g. Host == "*"). Requests will be matched against the Host field in the following way: 1. If Host is precise, the request matches this rule if the http host header is equal to Host. 2. If Host is a wildcard, then the request matches this rule if the http host header is to equal to the suffix (removing the first label) of the wildcard rule.
http
Group
Version
Kind
v1beta1
v1
Field
host
string
http
Group
Version
Kind
v1beta1
v1
Field
hosts
string
SecretName is the name of the secret used to terminate SSL traffic on 443. Field is left optional to allow SSL routing based on SNI hostname alone. If the SNI host in a listener conflicts with the "Host" header field used by an IngressRule, the SNI host is used for termination and value of the Host header is used for routing.
Group
Version
Kind
v1beta1
v1
Field
hosts
string
Group
Version
Kind
v1beta1
JSON
v1
Field
Group
Version
Kind
v1beta1
JSONSchemaP

rops
v1
CustomResourceValidation [apiextensions/v1beta1]
Field
$ref
string
$schema
string
allOf
array
anyOf
array
default
JSON
default is a default value for undefined object fields. Defaulting is a beta feature under the CustomResourceDefaulting feature gate. CustomResourceDefinitions with defaults must be created using the v1 (or newer) CustomResourceDefinition API.
object
object
string
enum
JSON
array
example
JSON
boolean
boolean
format
string
id
string
items
maxItems
integer
maxLength
integer
integer
maximum
number
minItems
integer
minLength
integer
integer
minimum
number
number
not
nullable
boolean
oneOf
array
pattern
string
object
object
required
title
string
type
string
boolean
boolean
boolean
x-kubernetes-int-or-string specifies that this value is either an integer or a string. If this is true, an empty type is allowed and typ

e as child of anyOf is permitted if following one of the following patterns:  1) anyOf:    - type: integer    - type: string 2) allOf:    - anyOf:      - type: integer      - type: string    - ... zero or more
string
string
boolean
Group
Version
Kind
v1beta1
v1
Field
Group
Version
Kind
v1beta1
v1
Field
Group
Version
Kind
batch
v1beta1
v1
CronJobSpec [batch/v1beta1]
Field
metadata
spec
JobSpec
Group
Version
Kind
v1beta1
Lease
v1
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/de

vel/sig-architecture/api-conventions.md#resources
kind
string
metadata
spec
LeaseSpec
LeaseSpec v1beta1 coordination
Field
MicroTime
string
integer
integer
renewTime
MicroTime
LeaseList v1beta1 coordination
Field
string
items
Lease
array
kind
string
metadata
ListMeta
Create
POST /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leases
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Lease
Response
Code
200
Lease
OK
201
Lease
Created
202
Lease
Accepted
Patch
PATCH /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leases/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (applicatio

n/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
Lease
OK
Replace
PUT /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leases/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Lease
Response
Code
200
Lease
OK
201
Lease
Created
Delete
DELETE /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leases/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leases
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 Resource

Expired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leases/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Lease
OK
List
GET /apis/coordination.k8s.io/v1beta1/namespaces/{namespace}/leases
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, mod

ified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
LeaseList
OK
GET /apis/coordination.k8s.io/v1beta1/leases
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is spe

cified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
LeaseList
OK
Watch
GET /apis/coordination.k8s.io/v1beta1/watch/namespaces/{namespace}/leases/{name}
Parameter
name
namespace
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/coordination.k8s.io/v1beta1/watch/namespaces/{namespace}/leases
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest

 snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/coordination.k8s.io/v1beta1/watch/leases
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due t

o expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Canno

t be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
spec
status
Create
POST /apis/authorization.k8s.io/v1beta1/namespaces/{namespace}/localsubjectaccessreviews
Parameter
namespace
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v2beta1
v2beta2
Field
external
object
pods
resource
type
string
Group
Version
Kind
v2beta1
v2beta2
Field
container resource refers to a resource metric (such as those specified in requests and limits) known to Kubernetes describing a single container in each pod in the current scale

 target (e.g. CPU or memory). Such metrics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
external
object
pods
resource
type
string
Group
Version
Kind
v1beta1
v1
Field
AdmissionReviewVersions is an ordered list of preferred `AdmissionReview` versions the Webhook expects. API server will try to use first version in the list which it supports. If none of the versions specified in this list supported by API server, validation will fail for this object. If a persisted webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail and be subject to the failure policy. Default to `['v1beta1']`.
string
FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowed values are Ignore or Fail. Defaults to Ignore.
string
matchPolicy defines how the "rules" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent".  - Exact: match a request only if it exactly matches a specified rule. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, but "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/v1beta1 would not be sent to the webhook.  - Equivalent: match a request if modifies a resource listed in rules, even via another API group or version. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, and "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/v1beta1 would be converted to apps/v1 and sent to the webhook.  Defaults to "Exact"
name
string
NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped re

source, it never skips the webhook.  For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "runlevel",       "operator": "NotIn",       "values": [         "0",         "1"       ]     }   ] }  If instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "environment",       "operator": "In",       "values": [         "prod",         "staging"       ]     }   ] }  See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.  Default to the empty LabelSelector, which matches everything.
string
rules
array
string
SideEffects states whether this webhook has side effects. Acceptable values are: Unknown, None, Some, NoneOnDryRun Webhooks with side effects MUST implement a reconciliation system, since a request may be rejected by a future step in the admission chain and the side effects therefore need to be undone. Requests with the dryRun attribute will be auto-rejected if they match a webhook with sideEffects == Unknown or Some. Defaults to Unknown.
integer
TimeoutSeconds specifies the timeout for this webhook. After the timeout passes, the webhook call will be ignored or the API call will fail based on the failure policy. The timeout value must be between 1 and 30 seconds. Default to 30 seconds.
Group
Version
Kind
v

1beta1
v1
Field
string
kind
string
metadata
webhooks
array
:
merge
:
name
MutatingWebhookConfigurationList v1beta1 admissionregistration
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
force
Force is going to "for

ce" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests.
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field.

 Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using

-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/admissionregistration.k8s.io/v1beta1/mutatingwebhookconfigurations
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using li

mit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
GET /apis/admissionregistration.k8s.io/v1beta1/watch/mutatingwebhookconfigurations/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose no

t to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/admissionregistration.k8s.io/v1beta1/watch/mutatingwebhookconfigurations
Parameter
continue
limit
limit is a maximum num

ber of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
SelfSubjectAccessReviewSpec [authorization/v1beta1]
SubjectAccessReviewSpec [authorization/v1beta1]
Field
path
string
verb
string
Group
Version
Kind
v1beta1
v1
SubjectRulesReviewStatus [authorization/v1beta1]
Field
verbs
Group
Version
Kind
v2beta1
v2beta2
Field
Quantity
string
selector
selector is the string-encoded form of a standard kubernetes label selector for the given metric When set, it is passed as an additional parameter to the metrics server for more specific metrics scoping When unset, just the metricName will be used to gather metrics.
target
target is the described Kubernetes object.
Quantity
targetValue is t

he target value of the metric (as a quantity).
Group
Version
Kind
v2beta1
v2beta2
Field
Quantity
Quantity
currentValue is the current value of the metric (as a quantity).
string
selector
selector is the string-encoded form of a standard kubernetes label selector for the given metric When set in the ObjectMetricSource, it is passed as an additional parameter to the metrics server for more specific metrics scoping. When unset, just the metricName will be used to gather metrics.
target
Group
Version
Kind
v1beta1
Overhead
v1
v1alpha1
Field
podFixed
object
Group
Version
Kind
v1alpha1
Overhead
v1
v1beta1
RuntimeClassSpec [node/v1alpha1]
Field
podFixed
object
Group
Version
Kind
policy
v1beta1
v1
Field
string
kind
string
metadata
spec
status
PodDisruptionBudgetSpec v1beta1 policy
Field
selector
:
replace
Label query over pods whose evictions are managed by the disruption budget. A null selector selects no pods. An empty selector ({}) also selects no pods, which differs from standard behavior of selecting all pods. In policy/v1, an empty selector will select all pods in the namespace.
PodDisruptionBudgetStatus v1beta1 policy
Field
Condition
array
:
merge
:
type
Conditions contain conditions for PDB. The disruption controller sets the DisruptionAllowed condition. The following are known values for the reason field (additional reasons could be added in the future): - SyncFailed: The controller encountered an error and wasn't able to compute              

 the number of allowed disruptions. Therefore no disruptions are               allowed and the status of the condition will be False. - InsufficientPods: The number of pods are either at or below the number                     required by the PodDisruptionBudget. No disruptions are                     allowed and the status of the condition will be False. - SufficientPods: There are more pods than required by the PodDisruptionBudget.                   The condition will be True, and the number of allowed                   disruptions are provided by the disruptionsAllowed property.
integer
integer
object
integer
integer
integer
PodDisruptionBudgetList v1beta1 policy
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets/{name}
Parameter
name
namespace
Parameter
pretty
If 'true', then 

the output is pretty printed.
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets
Path Pa

rameters
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Whether and how garbage collection will be performed. Either this field or OrphanDepe

ndents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list m

etadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
GET /apis/policy/v1beta1/poddisruptionbudgets
Parameter
continue
A selector

 to restrict the list of returned objects by their fields. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/policy/v1beta1/watch/namespaces/{namespace}/poddisruptionbudgets/{name}
Parameter
name
namespace
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/policy/v1beta1/watch/namespaces/{namespace}/poddisruptionbudgets
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this 

field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
GET /apis/policy/v1beta1/watch/poddisruptionbudgets
Parameter
Descriptio

n
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the call, regardless of any act

ivity or inactivity.
watch
Response
Code
200
OK
PATCH /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
GET /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets/{name}/status
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
PUT /apis/policy/v1beta1/namespaces/{namespace}/poddisruptionbudgets/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v2beta1
v2beta2
Field
string
metricName is the name of the metric in question
selector
Quantity
targetAverageValue is the target value of the average of the metric across all relevant pods (as a quantity)
Group
Version
Kind
v2beta1
v2beta2
Field
Quanti

ty
currentAverageValue is the current value of the average of the metric across all relevant pods (as a quantity)
string
selector
selector is the string-encoded form of a standard kubernetes label selector for the given metric When set in the PodsMetricSource, it is passed as an additional parameter to the metrics server for more specific metrics scoping. When unset, just the metricName will be used to gather metrics.
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
apiGroups
resources
Resources is a list of resources this rule applies to.  '\*' represents all resources in the specified apiGroups. '\*/foo' represents the subresource 'foo' for all resources in the specified apiGroups.
verbs
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
apiGroups
resources
verbs
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
string
string
boolean
globalDefault specifies whether this PriorityClass should be considered as the 

default priority for pods that do not have any priority class. Only one PriorityClass can be marked as `globalDefault`. However, if more than one PriorityClasses exists with their `globalDefault` field set to true, the smallest value of such global default PriorityClasses will be used as the default priority.
kind
string
metadata
string
value
integer
PriorityClassList v1beta1 scheduling
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/scheduling.k8s.io/v1beta1/priorityclasses
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/scheduling.k8s.io/v1beta1/priorityclasses/{name}
Parameter
name
Parameter
pretty
dryRun
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for n

on-apply patch requests.
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/scheduling.k8s.io/v1beta1/priorityclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/scheduling.k8s.io/v1beta1/priorityclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/scheduling.k8s.io/v1beta1/priorityclasses
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be in

cluded in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read
read the specifie

d PriorityClass
GET /apis/scheduling.k8s.io/v1beta1/priorityclasses/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/scheduling.k8s.io/v1beta1/priorityclasses
Parameter
pretty
continue
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for deta

ils.  Defaults to unset
watch
Response
Code
200
OK
Watch
GET /apis/scheduling.k8s.io/v1beta1/watch/priorityclasses/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This

 is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/scheduling.k8s.io/v1beta1/watch/priorityclasses
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the a

vailable results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
string
string
boolean
globalDefault specifies whether this PriorityClass should be considered as the default priority for pods that do not have any priority class. Only one PriorityClass can be marked as `globalDefault`. However, if more than one PriorityClasses exists with their `globalDefault` field set to true, the smallest value of such global default PriorityClasses will be used as the default priority.
kind
string
metadata
string
value
integer
PriorityClassList v1alpha1 scheduling
Field
string
items
P

riorityClass
array
kind
string
metadata
ListMeta
Create
POST /apis/scheduling.k8s.io/v1alpha1/priorityclasses
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stag

es will be processed
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/scheduling.k8s.io/v1alpha1/priorityclasses
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. 

Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/scheduling.k8s.io/v1alpha1/priorityclasses/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/scheduling.k8s.io/v1alpha1/priorityclasses
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error togethe

r with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/scheduling.k8s.io/v1alpha1/watch/priorityclasses/{name}
Parameter
name
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/scheduling.k8s.io/v1alpha1/watch/priorityclasses
Parameter
allowWatchBookmark

s requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for cha

nges to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
Field
group
string
name
string
namespace
string
resource
string
string
verb
string
version
string
Group
Version
Kind
v2beta1
v2beta2
Field
name
string
integer
Quantity
Group
Version
Kind
v2beta1
v2beta2
Field
integer
Quantity
name
string
Group
Version
Kind
v1beta1
v1
Field
apiGroups
resources
verbs
Role

 v1beta1 rbac.authorization.k8s.io
Group
Version
Kind
v1beta1
Role
v1
v1alpha1
Field
string
kind
string
metadata
rules
array
RoleList v1beta1 rbac
Field
string
items
Role
array
kind
string
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/roles
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
202
Role
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Role
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{nam

espace}/roles/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/roles
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request wi

ll be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read
read the 

specified Role
GET /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Role
OK
List
GET /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/roles
Parameter
namespace
Parameter
pretty
continue
limit
resourc

eVersion
watch
Response
Code
200
RoleList
OK
GET /apis/rbac.authorization.k8s.io/v1beta1/roles
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a cons

istent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
RoleList
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use 

the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/namespaces/{namespace}/roles
Parameter
namespace
Parameter
continue
A selector to restrict the list of returned 

objects by their fields. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/roles
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from th

e latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
Role
v1
v1beta1
Field
string
kind
string
metadata
rules
array
RoleList v1alpha1 rbac
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schem

as to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
items
Role
array
kind
string
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
202
Role
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Role
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Que

ry Parameters
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return f

or a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Role
OK
List
GET /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/roles
Parameter
namespace
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the 

feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
watch
Response
Code
200
RoleList
OK
GET /apis/rbac.authorization.k8s.io/v1alpha1/roles
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers

 that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a 

stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
RoleList
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/roles/{name}
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a reques

t may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/roles
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or delet

ed after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/roles
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more res

ults are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
RoleBindingList v1beta1 rbac
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
pr

etty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Deprecated: please use the Propagati

onPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server 

guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/rbac.authorization.k8s.io/v1beta1/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs 

a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1beta1/rolebindings
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continu

e) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
allowWatchBookmarks requests watch events

 with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described r

esources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
Watch List
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kube

rnetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1beta1/watch/rolebindings
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent sn

apshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
RoleBindingList v1alpha1 rbac
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
pretty
dryRun
Parameter

body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - 

allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculat

ed is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
GET /apis/rbac.authorization.k8s.io/v1alpha1/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value

 returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1alpha1/rolebindings
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/rolebindings/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue valu

e from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/namespaces/{namespace}/rolebindings
Parameter
namespace
Parameter
allowWatchBookmarks requests watch event

s with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described 

resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
GET /apis/rbac.authorization.k8s.io/v1alpha1/watch/rolebindings
Parameter
continue
limit
pretty

nMatch
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
RoleRef
v1
v1alpha1
Field
apiGroup
string
kind
string
name
string
Group
Version
Kind
v1alpha1
RoleRef
v1
v1beta1
Field
apiGroup
string
kind
string
name
string
Group
Version
Kind
v1beta1
v1
Field
apiGroups
resources
scope
string
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
string
handler
string
Handler specifies the underlying runtime and configuration that the CRI implementation will use to handle pods of this class. The possible values ar

e specific to the node &amp; CRI configuration.  It is assumed that all handlers are available on every node, and handlers of the same name are equivalent on every node. For example, a handler called "runc" might specify that the runc OCI runtime (using native Linux containers) will be used to run the containers in a pod. The Handler must be lowercase, conform to the DNS Label (RFC 1123) requirements, and is immutable.
kind
string
metadata
overhead
Overhead
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead This field is alpha-level as of Kubernetes v1.15, and is only honored by servers that enable the PodOverhead feature.
RuntimeClassList v1beta1 node
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/node.k8s.io/v1beta1/runtimeclasses
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/node.k8s.io/v1beta1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-appl

y patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/node.k8s.io/v1beta1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/node.k8s.io/v1beta1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/node.k8s.io/v1beta1/runtimeclasses
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the n

ext key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Timeout for the list/watch call. This limits the duration of the cal

l, regardless of any activity or inactivity.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/node.k8s.io/v1beta1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/node.k8s.io/v1beta1/runtimeclasses
Parameter
pretty
continue
limit
resourceVersion set

s a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
GET /apis/node.k8s.io/v1beta1/watch/runtimeclasses/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a

 limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/node.k8s.io/v1beta1/watch/runtimeclasses
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the c

ontinue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
string
kind
string
metadata
spec
RuntimeClassSpec
Specification of the RuntimeClass More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
RuntimeClassSpec v1alpha1 node
Field
overhead
Overhead
runtimeHandler
string
RuntimeHandler specifies the underlying runtime and configuration that the CRI implementation will use to handle pods of this class. The possible values are specific to the node &amp; CRI configuration.  It is assumed that all handlers are available on every node, and handlers of the same name are equivalent on every node. For example, a handler called "runc" might specify that the runc OCI runtime (using native Linux containers) will be used to run the containers in a pod. The RuntimeHandler must be lowercase, conform to the DNS Label (RFC 1123) requirements, and is immutable.
Scheduling hold

s the scheduling constraints to ensure that pods running with this RuntimeClass are scheduled to nodes that support it. If scheduling is nil, this RuntimeClass is assumed to be supported by all nodes.
RuntimeClassList v1alpha1 node
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/node.k8s.io/v1alpha1/runtimeclasses
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/node.k8s.io/v1alpha1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/node.k8s.io/v1alpha1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
O

K
201
Created
Delete
DELETE /apis/node.k8s.io/v1alpha1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/node.k8s.io/v1alpha1/runtimeclasses
Parameter
pretty
continue
dryRun
A selector to restrict the list of returned objects by their labels. Defaults to

 everything.
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/node.k8s.io/v1alpha1/runtimeclasses/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/node.k8s.io/v1alpha1/runtimeclasses
Parameter
pretty
continue
The continue option should be set when ret

rieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/node.k8s.io/v1alpha1/watch/runtimeclasses/{name}
Parameter
name
Parameter
allowWa

tchBookmarks
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or 

inactivity.
watch
Response
Code
200
OK
Watch List
GET /apis/node.k8s.io/v1alpha1/watch/runtimeclasses
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defau

lts to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
object
array
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
object
array
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
spec
status
SelfSubjectAccessReviewSpec v1beta1 authorization
Field
Create
POST /apis/authorization.k8s.io/v1beta1/selfsubjectaccessreviews
Parameter
dryRun
fieldM

anager
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
spec
status
SelfSubjectRulesReviewSpec v1beta1 authorization
Field
namespace
string
Create
POST /apis/authorization.k8s.io/v1beta1/selfsubjectrulesreviews
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1
v1beta1
v1beta1
v1beta1
WebhookClientConfig [apiextensions/v1]
Field
name
string
namespace
string
path
string
port
integer
port is an optional service port at which the webhook will be contacted. `port` should be a valid port number (1-65535, inclusive). Defaults to 443 for backward compatibility.
Group
Version
Kind
v1
v1beta1
v1beta1
v1beta1
APIServiceSpec [apiregistration/v1]
Field
name
string
namespace
string
port
integer
Group
Version
Kind
v1beta1
v1
v1
v1
WebhookClientConfig [admissionregistration/v

1beta1]
Field
name
string
namespace
string
path
string
port
integer
Group
Version
Kind
v1beta1
v1
v1
v1
WebhookClientConfig [apiextensions/v1beta1]
Field
name
string
namespace
string
path
string
port
integer
Group
Version
Kind
v1beta1
v1
v1
v1
APIServiceSpec [apiregistration/v1beta1]
Field
name
string
namespace
string
port
integer
Group
Version
Kind
v1beta1
v1
Field
boolean
array
string
kind
string
metadata
object
string
string
string
StorageClassList v1beta1 storage
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unreco

gnized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
items
array
kind
string
metadata
ListMeta
Create
POST /apis/storage.k8s.io/v1beta1/storageclasses
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/storage.k8s.io/v1beta1/storageclasses/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/storage.k8s.io/v1beta1/storageclasses/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/storage.k8s.io/v1beta1/storageclasses/{name}
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error re

sponse and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
202
Accepted
DELETE /apis/storage.k8s.io/v1beta1/storageclasses
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients shou

ld only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/storage.k8s.io/v1beta1/storageclasses/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/storage.k8s.io/v1beta1/storageclasses
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration chan

ge on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/storage.k8s.io/v1beta1/watch/storageclasses/{name}
Parameter
name
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate W

atchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/storage.k8s.io/v1beta1/watch/storageclasses
Query Par

ameters
Parameter
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the

 call, regardless of any activity or inactivity.
watch
Response
Code
200
OK
Group
Version
Kind
v1
Subject
v1beta1
v1beta1
v1alpha1
Field
apiGroup
string
kind
string
name
string
namespace
string
Namespace of the referenced object.  If the object kind is non-namespace, such as "User" or "Group", and this value is not empty the Authorizer should report an error.
Group
Version
Kind
v1beta1
Subject
v1
v1alpha1
Field
apiGroup
string
kind
string
name
string
namespace
string
Group
Version
Kind
v1alpha1
Subject
v1beta1
v1
v1beta1
Field
string
APIVersion holds the API group and version of the referenced subject. Defaults to "v1" for ServiceAccount subjects. Defaults to "rbac.authorization.k8s.io/v1alpha1" for User and Group subjects.
kind
string
name
string
namespace
string
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
spec
status
SubjectAccessReviewSpec v1beta1 authorization
LocalSubjectAcces

sReview [authorization/v1beta1]
Field
extra
object
group
uid
string
user
string
User is the user you're testing for. If you specify "User" but not "Group", then is it interpreted as "What if User were not a member of any groups
SubjectAccessReviewStatus v1beta1 authorization
Field
allowed
boolean
denied
boolean
string
reason
string
Create
POST /apis/authorization.k8s.io/v1beta1/subjectaccessreviews
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1beta1
v1
Field
string
boolean
array
array
Group
Version
Kind
v1
v1beta1
CSIDriverSpec [storage/v1]
Field
audience
stri

ng
integer
Group
Version
Kind
v1beta1
v1
v1
CSIDriverSpec [storage/v1beta1]
Field
audience
string
integer
ExpirationSeconds is the duration of validity of the token in "TokenRequestSpec". It has the same default value of "ExpirationSeconds" in "TokenRequestSpec"
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
spec
status
TokenReviewSpec v1beta1 authentication
Field
audiences
token
string
TokenReviewStatus v1beta1 authentication
Field
audiences
boolean
error
string
user
UserInfo
Create
POST /apis/authentication.k8s.io/v1beta1/tokenreviews
Parameter
dryRun
pretty
If 'true', then the output is pretty 

printed.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1beta1
UserInfo
v1
TokenReviewStatus [authentication/v1beta1]
Field
extra
object
groups
uid
string
username
string
Group
Version
Kind
v1beta1
v1
Field
string
string
name
string
NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.  For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "runlevel",       "operator": "NotIn",       "values": [         "0",         "1"       ]     }   ] }  If instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "environment",       "operator": "In",       "values": [         "prod",         "staging"       ]     }   ] }  See https://kubernetes.io/docs/concepts

/overview/working-with-objects/labels for more examples of label selectors.  Default to the empty LabelSelector, which matches everything.
rules
array
string
integer
Group
Version
Kind
v1beta1
v1
Field
string
kind
string
metadata
webhooks
array
:
merge
:
name
ValidatingWebhookConfigurationList v1beta1 admissionregistration
Field
string
items
array
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: ht

tps://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Create
POST /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an 

error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations
Parameter
pretty
continue
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/admissionregistration.k8s.io/v1beta1/validatingwebhookconfigurations
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reje

ct a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/admissionregistration.k8s.io/v1beta1/watch/validatingwebhookconfigurations/{name}
Parameter
name
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and book

marks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resou

rceVersion.
Response
Code
200
OK
Watch List
GET /apis/admissionregistration.k8s.io/v1beta1/watch/validatingwebhookconfigurations
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applie

d to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
v1alpha1
Field
string
kind
string
metadata
spec
status
VolumeAttachmentSpec v1beta1 storage
Field
attacher
string
nodeName
string
source
VolumeAttachmentStatus v1beta1 storage
Field
attached
boolean
object
VolumeAttachmentList v1beta1 storage
Field
string
items
array
kind
string
metadata
ListMeta
Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-archite

cture/api-conventions.md#metadata
Create
POST /apis/storage.k8s.io/v1beta1/volumeattachments
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/storage.k8s.io/v1beta1/volumeattachments/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/storage.k8s.io/v1beta1/volumeattachments/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/storage.k8s.io/v1beta1/volumeattachments/{name}
Parameter
name
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be depre

cated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
OK
202
Accepted
DELETE /apis/storage.k8s.io/v1beta1/volumeattachments
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects cr

eated, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/storage.k8s.io/v1beta1/volumeattachments/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/storage.k8s.io/v1beta1/volumeattachments
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or d

eleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
GET /apis/storage.k8s.io/v1beta1/watch/volumeattachments/{name}
Parameter
name
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/storage.k8s.io/v1beta1/watch/volumeattachments
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored. If the feature gate WatchBookmarks is 

not enabled in apiserver, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
v1
v1beta1
Field
ap

iVersion
string
kind
string
metadata
spec
status
VolumeAttachmentSpec v1alpha1 storage
Field
attacher
string
nodeName
string
source
VolumeAttachmentStatus v1alpha1 storage
Field
attached
boolean
object
VolumeAttachmentList v1alpha1 storage
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/storage.k8s.io/v1alpha1/volumeattachments
Parameter
pretty
dryRun
Paramete

r
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/storage.k8s.io/v1alpha1/volumeattachments/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
Replace
PUT /apis/storage.k8s.io/v1alpha1/volumeattachments/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/storage.k8s.io/v1alpha1/volumeattachments/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body

Response
Code
200
OK
202
Accepted
DELETE /apis/storage.k8s.io/v1alpha1/volumeattachments
Parameter
pretty
continue
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/storage.k8s.io/v1alpha1/volumeattachments/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/storage.k8s.io/v1alpha1/volumeattachments
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit m

ay return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
GET /apis/storage.k8s.io/v1alpha1/watch/volumeattachments/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVer

sion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/storage.k8s.io/v1alpha1/watch/volumeattachments
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list requ

est with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1beta1
v1
v1alpha1
VolumeAttachmentSpec [storage/v1beta1]
Field
string
Group
Version
Kind
v1alpha1
v1
v1beta1
VolumeAttachmentSpec [storage/v1alpha1]
Field
inl

ineVolumeSpec contains all the information necessary to attach a persistent volume defined by a pod's inline VolumeSource. This field is populated only for the CSIMigration feature. It contains translated fields from a pod's inline VolumeSource to a PersistentVolumeSpec. This field is alpha-level and is only honored by servers that enabled the CSIMigration feature.
string
Group
Version
Kind
v1beta1
v1
v1alpha1
VolumeAttachmentStatus [storage/v1beta1]
Field
message
string
time
Time
Group
Version
Kind
v1alpha1
v1
v1beta1
VolumeAttachmentStatus [storage/v1alpha1]
Field
message
string
String detailing the error encountered during Attach or Detach operation. This string maybe logged, so it should not contain sensitive information.
time
Time
Group
Version
Kind
v1beta1
v1
CSINodeDriver [storage/v1beta1]
Field
count
integer
Maximum number of unique volumes managed by the CSI driver that can be used on a node. A volume that is both attached and mounted on a node is considered to be used once, not twice. The same rule applies for a unique volume that is shared among multiple pods on the same node. If this field is nil, then the supported number of volumes on this node is unbounded.
Group
Version
Kind
v1
v1beta1
v1beta1
WebhookConversion [apiextensions/v1]
Field
caBundle
string
service
service is a reference to the service for this webhook. Either service or url must be specified.  If the webhook is running within the cluster, then you should use `service`.
url
string
url gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be specified.  The `host` should not refer to a service running in the cluster; use the `service` field instead. The host might be resolved via external DNS in some apiservers (e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would be a layering violation). `host` may also be an IP address.  Please note that using `localhost` or `127.0.0.1` as a `host` is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster.  The scheme must be "https"; the URL must begin with "https://".  A path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier.  Attempting to use a user or basic auth e.g. "user:password@" is not allowed. Fragments ("#...") and query parameters ("?...") are not allowed, either.
Group
Version
Kind
v1beta1
v1
v1
Field
caBundle
string
`caBundle` is a PEM encoded CA bundle which will be used to validate the w

ebhook's server certificate. If unspecified, system trust roots on the apiserver are used.
service
url
string
Group
Version
Kind
v1beta1
v1
v1
CustomResourceConversion [apiextensions/v1beta1]
Field
caBundle
string
service
url
string

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Using kubectl to Create a Deployment
cluster.
Summary:
Kubectl
Kubectl
5
and
6
).
›
2 - Interactive Tutorial - Deploying an App
.
Home
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-proxy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on m

acOS
Debug Pods
Troub

leshooting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to Res

ources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

 Configuration (v1alpha1)
Reference
Synopsis
Options
--cleanup
Mode to 

use to detect local traffic. This parameter is ignored if a config file is specified by --config.
Nod

eInclusionPolicyInPodTopologySpread=true|false (BETA - default=true)
-h, --help
The maximum interval of how often ipvs rules are 

refreshed (e.g. '5s', '1m', '2h22m'). Must be greater than 0.
The previous version for which you want to show hidden metrics. Only th

e previous minor version is meaningful, other values will not be allowed. The format is &lt;major&gt;.&lt;minor&gt;, e.g.: '1.16'. The purpose of this format is make sure you have the opportunity to notice if the next release hides additional metrics, rather than being surprised when they are permanently removed in the release after that. This parameter is ignored if a config file is specified by --config.
-v, --v int
Feedback
Yes
No
.
or
.
Synopsis
Options
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
1 - Managing Secrets using kubectl
This page shows you how to create, edit, manage, and delete Kubernetes
Secrets
using the
kubectl
command-line tool.
minikube
A
Secret
object stores sensitive data such as credentials
used by Pods to access services. For example, you might need a Secret to store
the username and password needed to access a database.
You can create the Secret by passing the raw data in the command, or by storing
the credentials in files that you pass in the command. The following commands
create a Secret that stores the username
admin
S!B\*d$zDsb=
.
Use raw data
kubectl create secret generic db-user-pass
\
=
username
=
admin
\
=
password
=
'S!B\*d$zDsb='
You must use single quotes
''
to escape special characters such as
$
,
\
,
*
,
=
, and
!
in your strings. If you don't, your shell will interpret these
characters.
Use source files
Store the credentials in files:
echo
-n
'admin'
echo
-n
The
-n
flag ensures that the generated files do not have an extra newline
character at the end of the text. This is important because when
kubectl
reads a file and encodes the content into a base64 string, the extra
newline character gets encoded too. You do not need to escape special
characters in strings that you include in a file.
Pass the file paths in the
kubectl
command:
\
=
\
=
The default key name is the file name. You can optionally set the key name
using
--from-file=[key=]source
\
=
username
=
\
=
password
=
With either method, the output is similar to:
secret/db-user-pass created
Verify the Secret
Check that the Secret was created:
NAME              TYPE       DATA      AGE
db-user-pass      Opaque     2         51s
View the details of the Secret:
kubectl describe secret db-user-pass
Name:            db-user-pass
Namespace:       default
Labels:          &lt;none&gt;
Annotations:     &lt;none&gt;

Type:            Opaque

Data
====
password:    12 bytes
username:    5 bytes
and
avoid showing the contents
of a
Secret
by default. This is to protect the
Secret
from being exposed
accidentally, or from being stored in a terminal log.
Decode the Secret
View the contents of the Secret you created:
kubectl get secret db-user-pass -o
jsonpath
=
'{.data}'
{
:
"UyFCXCpkJHpEc2I9"
,
:
"YWRtaW4="
}
Decode the
password
data:
echo
'UyFCXCpkJHpEc2I9'
Caution:
This is an example for documentation purposes. In practice,
this method could cause the command with the encoded data to

 be stored in
your shell history. Anyone with access to your computer could find the
command and decode the secret. A better approach is to combine the view and
decode commands.
jsonpath
=
'{.data.password}'
You can edit an existing
Secret
immutable
. To edit a
Secret, run the following command:
kubectl edit secrets &lt;secret-name&gt;
This opens your default editor and allows you to update the base64 encoded
Secret values in the
data
field, such as in the following example:
# and an empty file will abort the edit. If an error occurs while saving this file, it will be
#
:
v1
data
:
password
:
UyFCXCpkJHpEc2I9
username
:
YWRtaW4=
kind
:
Secret
metadata
:
:
"2022-06-28T17:44:13Z"
name
:
namespace
:
default
:
"12708504"
uid
:
91becd59-78fa-4c85-823f-6d44436242ac
type
:
Opaque
Clean up
To delete a Secret, run the following command:
2 - Managing Secrets using Configuration File
minikube
Secret
Secret
data
and
.
The
data
The keys of
data
and
-
,
_
or
.
.
data
field.
echo
-n
'admin'
| base64
echo
-n
| base64
Note:
base64
-b
should
-w 0
to
base64
if the
-w
YWRtaW4=
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
password
:
.
:
.
that uses a Secret to store a configuration file, and you want to populate
apiUrl
:
username
:
password
:
You could sto

re this in a Secret using the following definition:
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
:
:
|
.
:
v1
data
:
:
kind
:
Secret
metadata
:
:
name
:
mysecret
namespace
:
default
:
"7225"
uid
:
type
:
Opaque
data
and
data
and
is used.
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
:
username
:
The
Secret
:
v1
data
:
username
:
kind
:
Secret
metadata
:
:
name
:
mysecret
namespace
:
default
:
"7579"
uid
:
type
:
Opaque
decodes to
.
data
or
Secret
immutable
.
echo
-n
| base64
Update the
data
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
password
:
Secret
kubectl
tool
Secret
kubectl
Secret
instead,
kubectl
uses
instead.
Clean up
3 - Managing Secrets using Kustomize
kubectl
kubectl
.
minikube
in a
.env
files, or
admin
.
Literals
Files
.env files
:
-
name
:
literals
:
Store the credentials in files with the values

 encoded in base64:
echo
-n
'admin'
echo
-n
The
-n
files.
Create the
file:
:
-
name
:
files
:
file by
providing
.env
file
file:
:
-
name
:
envs
:
file
must
be
or
.
.
In your
file, modify the data, such as the
password
.
Secret
existing
Secret
your Pods.
Clean up
kubectl
:
Home
Blog
Training
Partners
Community
CC BY 4.0

Monitor Node Health | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
DaemonSet
Condition
s
or as
Event
s.
.
minikube
.
Addon
.
kubectl
kubectl
:
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
Note:
Start n

ode problem detector with
kubectl
:
Create
directory
The
ConfigMap
config/
Create the
ConfigMap
:
=
config/
Change the
to use the
ConfigMap
:
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
-
name
:
config
mountPath
:
/config
readOnly
:
true
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
-
name
:
config
configMap
:
name
:
Note:
kubectl
.
ConfigMap
.
A
such as
filelog
,
kmsg
,
kernel
,
abrt
,
and
systemd
.
A
.
A
.
A
.
.
Exporter
An exporter reports the node pr

oblems and/or metrics to certain backends.
.
.
Feedback
Yes
No
.
or
.
Last modified December 17, 2022 at 12:16 PM PST:
Revise the NPD task page (b940dcf2f2)
Exporter
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Service Accounts for Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

Tasks
API server
.
A
user
not
have a User
API.
minikube
default
namespace
.
default
.
default
.
See
in use.
kubelet
on the ServiceA

ccount:
:
v1
kind
:
metadata
:
name
:
:
false
...
:
v1
kind
:
Pod
metadata
:
name
:
my-pod
spec
:
:
:
false
...
.spec
default
with:
You can create additional ServiceAccount objects like this:
metadata:
EOF
.
:
v1
kind
:
metadata
:
:
name
:
namespace
:
default
:
"272500"
uid
:
.
field
Note:
The
.
on the
.
Cleanup
kubectl
:
Note:
API,
.
.
metadata:
EOF
Because o

f the annotation you set, the control plane automatically generates a token for that

Data
====
token:          ...
Note:
token
Secret somewhere that your terminal / computer screen could be seen by an onlooker.
First,
.
.
=
\
=
=
\
=
sa.yaml
:
v1
kind
:
metadata
:
:
name
:
default
namespace
:
default
:
"243024"
uid
:
uid
:
v1
kind
:
metadata
:
:
name
:
default
namespace
:
default
uid
:
:
-
name
:
=
=
Never
=
jsonpath
=
Note:
:
(RSA or ECDSA), used to verify ServiceAccount tokens. The specified file can c

ontain
If
command line argument but you don't set
not
spec
.
vault
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
-
image
:
nginx
name
:
nginx
:
-
mountPath
:
name
:
:
volumes
:
-
name
:
projected
:
sources
:
-
:
path
:
:
7200
audience
:
vault
,
.
Note:
. In
https
.
.
.
.
Note:
and
RBAC
include a
default Cl

usterRole called
.
group,
or
jwks_uri
In many cases, Kubernetes API servers are not available on the public internet,
jwks_uri
https
scheme.
See also:
Read the
Read about
Read about
Secrets
or learn to
.
Read about
.
Read the
Feedback
Yes
No
.
or
.
Cleanup
Home
Blog
Training
Partners
Community
CC BY 4.0

Windows debugging tips | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

Tasks
See
Note:
or
this guide
.
enabled
on all
commands.
cni.conf
(see
) is
from the
in
cni.conf
            ]
NodePort
it, users need to 

pass the hostname to kube-proxy as follows:
C:\k\
| ? Name
-ieq
"cbr0"
| ? Name
-Like
start.ps1
section
.
.
If you would like to build your own pause infrastructure container be sure to include
wincat
.
,
,
,
,
issue
;
start.ps1
,
)
-file
to restart
=
=
=
1500
=
true
topic
Feedback
Yes
No
.
or
.
Last modified August 02, 2022 at 12:25 AM PST:
fix an orphan link (c54e53de33)
Further investigatio

n
Home
Blog
Training
Partners
Community
CC BY 4.0

Exposing an External IP Address to Access an Application in a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Cust

Tutorials
Install
kubectl
.
,
Configure
kubectl
:
apps/v1
kind
:
metadata
:
labels
:
:
name
:
spec
:
replicas
:
5
selector
:
:
:
template
:
metadata
:
labels
:
:
spec
:
:
-
image
:
name
:
ports
:
-
:
8080
.
Pods
=
=
Note:
The
this page
Note:
Display detailed info

rmation about the Service:
Port
and
NodePort
Port
NodePort
is 32377.
verify these are pod addresses, enter this command:
=
wide
where
)
Port
will
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Enforcing Pod Security Standards | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kube

ctl on Linux
Debug Pods
Debug Running P

ods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespac

e Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiser

ver Encryption Configuration (v1)
This page provides an overview of best practices when it comes to enforcing
.
Using the built-in Pod Security Admission Controller
The
Pod Security Admission Controller
intends to replace the deprecated PodSecurityPolicies.
Configure all cluster namespaces
Namespaces that lack any configuration at all should be considered significant gaps in your cluster
security model. We recommend taking the time to analyze the types of workloads occurring in each
namespace, and by referencing the Pod Security Standards, decide on an appropriate level for
each of them. Unlabeled namespaces should only indicate that they've yet to be evaluated.
In the scenario that all workloads in all namespaces have the same security requirements,
we provide an
example
that illustrates how the PodSecurity labels can be applied in bulk.
Embrace the principle of least privilege
In an ideal world, every pod in every namespace would meet the requirements of the
policy. However, this is not possible nor practical, as some workloads will require elevated
privileges for legitimate reasons.
Namespaces allowing
workloads should establish and enforce appropriate access controls.
For workloads running in those permissive namespaces, maintain documentation about their unique
security requirements. If at all possible, consider how those requirements could be further
constrained.
Adopt a multi-mode strategy
The
audit
and
warn
modes of the Pod Security Standards admission controller make it easy to
collect important security insights about your pods without breaking existing workloads.
It is good practice to enable these modes for all namespaces, setting them to the
desired
level
and version you would eventually like to
enforce
. The warnings and audit annotations generated in
this phase can guide you toward that state. If you expect workload authors to make changes to fit
within the desired level, enable the
warn
mode. If you expect to use audit logs to monitor/drive
changes to fit within the desired level, enable the
audit
mode.
When you have the
enforce
mode set to your desired value, these modes can still be useful in a
few different ways:
By setting
warn
to the same level as
enforce
, clients will receive warnings when attempting
to create Pods (or resources that have Pod templates) that do not pass validation. This will help
them update those resources to become compliant.
In Namespaces that pin

enforce
to a specific non-latest version, setting the
audit
and
warn
modes to the same level as
enforce
, but to the
latest
version, gives visibility into settings
that were allowed by previous versions but are not allowed per current best practices.
Third-party alternatives
Note:
Other alternatives for enforcing security profiles are being developed in the Kubernetes
ecosystem:
Kubewarden
.
Kyverno
.
OPA Gatekeeper
.
The decision to go with a
built-in
solution (e.g. PodSecurity admission controller) versus a
third-party tool is entirely dependent on your own situation. When evaluating any solution,
trust of your supply chain is crucial. Ultimately, using
any
of the aforementioned approaches
will be better than doing nothing.
Feedback
Yes
No
.
or
.
Last modified September 22, 2022 at 3:12 PM PST:
Bump the feature state of the Pod Security plugin to stable (1ab76fba82)
Home
Blog
Training
Partners
Community
CC BY 4.0

ResourceClass v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Reference
kind
metadata
(
)
(string)
(string)
Required. A list of node selector t

erms. The terms are ORed.
([]
)
([]
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
prop

agationPolicy
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Concepts
1:
Overview
1.1:
1.2:
1.3:
1.3.1:
1.3.2:
1.3.3:
1.3.4:
1.3.5:
1.3.6:
1.3.7:
1.3.8:
1.3.9:
1.3.10:
2:
2.1:
Nodes
2.2:
2.3:
2.4:
Leases
2.5:
2.6:
2.7:
2.8:
3:
3.1:
Images
3.2:
3.3:
3.4:
4:
4.1:
4.2:
5:
Workloads
5.1:
Pods
5.1.1:
5.1.2:
5.1.3:
5.1.4:
5.1.5:
5.1.6:
5.2:
5.2.1:
5.2.2:
5.2.3:
5.2.4:
DaemonSet
5.2.5:
Jobs
5.2.6:
5.2.7:
CronJob
5.2.8:
6:
6.1:
Service
6.2:
Ingress
6.3:
6.4:
6.5:
6.6:
6.7:
6.8:
6.9:
6.10:
6.11:
6.12:
7:
Storage
7.1:
Volumes
7.2:
7.3:
7.4:
7.5:
7.6:
7.7:
7.8:
7.9:
7.10:
7.11:
7.12:
7.13:
8:
8.1:
8.2:
8.3:
Secrets
8.4:
8.5:
8.6:
9:
Security
9.1:
9.2:
9.3:
9.4:
9.5:
9.6:
9.7:
9.8:
9.9:
9.10:
9.11:
10:
Policies
10.1:
10.2:
10.3:
10.4:
11:
11.1:
11.2:
11.3:
11.4:
11.5:
11.6:
11.7:
11.8:
11.9:
11.10:
11.11:
11.12:
11.13:
12:
12.1:
12.2:
12.3:
12.4:
12.5:
12.6:
12.7:
12.8:
12.9:
12.10:
13:
13.1:
13.1.1:
13.1.2:
Dev

ice Plugins
13.2:
13.2.1:
13.2.2:
13.3:
cluster
1 - Overview
running
Environmental consistency across development, testing, and

 production: runs
Delivery, and Deployment (CI/CD) workflows are dete

rmined by organization cultures and
.
Ready to
?
1.1 - Kubernetes Components
nodes
,
Pods
that are
pod
replicas
.
etcd
back up
plan
.
Pods
node
to run on.
Fa

ctors taken into account for scheduling decisions include:
component
kubelet
node
Pod
.
node
Service
concept.
,
CRI-O
,
.
Addons
DaemonSet
,
, etc)
Selected addons are described below; for an extended list of available addons, please
see
Addons
.
DNS
Dashboard
is a gener

al purpose, web-based UI for Kubernetes clusters. It allows users to manage and troubleshoot applications running in the cluster, as well as the cluster itself.
A
Nodes
1.2 - The Kubernetes API
is the
API server
kubectl
kubeadm
OpenAPI
.
OpenAPI V2
Header
Notes
gzip
Accept
default
*
serves
OpenAPI V3
named
OpenAPIV3
{
"paths":
{
...,
"api/v1":
{
},
{
},
....
}
}
Expires
to 1 year in
to
immutable
endpoint.
Valid r

equest header values for OpenAPI v3 queries
Header
Notes
gzip
Accept
default
*
serves
etcd
.
as
/api/v1
or
.
API groups
that can be
.
v1
and
v1beta1
v1beta1
v1beta1
or the
v1
v1beta1
v1
API.
not
.
v1
beta
Note:
alpha
Refer to
.
.
describes
Learn about API endpoi

nts, resource types and samples by reading
.
.
1.3 - Working with Kubernetes Objects
1.3.1 - Understanding Kubernetes Objects
.yaml
format.
cluster's
.
kubectl
.
spec
status
.
spec
its
.
The
status
supplied.
spec
.
kubectl
kubectl
kubectl
request.
.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.yaml
com

mand
in the
kubectl
.yaml
In the
.yaml
kind
metadata
name
string,
UID
namespace
spec
spec
spec
field
.spec
spec
field
.spec
Within the
.spec
template
.status
.status
Pods
objects.
kubectl
and
.
1.3.2 - Kubernetes Object Management
The
kubectl
for
Warning:
1+
Lowest
1
Moderate
1+
Highest
the
kubectl
Examples
See the
Warning:
replace
com

mand replaces the existing
Examples
kubectl
Note:
patch
replace
Examples
configs
diff
1.3.3 - Object Names and IDs
Name
UID
namespace
.
For non-unique user-provided att

ributes, Kubernetes provides
labels
and
.
Names
.
Note:
RFC 1123
.
RFC 1123
.
RFC 1035
.
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Note:
UIDs
Read about
labels
and
See the
1.3.4 - Labels and Selectors
Labels
:
{
"labels"
: {
"key1"
:
"value1"
,
"key2"
:
"value2"
}
}
.
,
,
,
"environmen

t" : "production"
,
,
,
,
Labels
/
-
_
), dots (
.
.
/
).
,
,
,
kubectl
The
and
k8s.io/
reserved
),
-
_
), dots (
.
and
app: nginx
:
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Unlike
Via a
and
set-based
.
AND
(
them.
Note:
Caution:
OR
(
||
Equality-
or
=
,
==
,
!=
equality
.
tier
frontend
tier
key.
excluding
frontend
the label "
".
:
v1
kind
:
Pod
metadata
:
name
:
cuda-test
spec
:
:
-
name
:
cuda-test
image
:
"registry.k8

s.io/cuda-vector-add:v0.1"
resources
:
limits
:
:
1
:
:
Set-based
Set-based
in
,
notin
and
exists
partition
or
qa
.
tier
frontend
and
backend
tier
key.
partition
partition
AND
partition
qa
.
The
set-based
!=
and
notin
.
Set-based
.
API
set-based
apiserver
with
kubectl
and using
=
=
frontend
or using
set-based
set-based
OR
exists
operator:
services
and
,
pods
.
service
json
or
yaml
:
{
:
"redis"
,
}
or
selector
:
component
:
redis
json
or
yaml
or
.
Job
,
,
, and
DaemonSet
,
support
set-based
selector
:
:
component
:
redis
:
- {
:
[cache]}
- {
:
[dev]}
is a map of
in the
, whose
key
operator
values
and
are ANDed together -- they

 must all be satisfied in order to match.
Find
See
1.3.5 - Namespaces
.
).
labels
Note:
not
using the
default
default
Lease
all
.
Note:
kube-
flag.
=
=
=
context.
=
Service
,
DNS entry
.
multiple name

spaces such as Development, Staging and Production. If you want to reach
.
Warning:
domains
will
as
admission
webhooks
,
public
TLDs
.
nodes
and
=
true
=
false
label
.
.
1.3.6 - Annotations
:
{
: {
"key1"
:
"value1"
,
"key2"
:
"value2"
}
}
Note:
Directives from the end-user to the implementations to modify behavior or
/
). The name seg

ment is required and must be 63 characters or less, beginning and ending with an alphanumeric character (
-
_
), dots (
.
.
/
).
,
,
,
kubectl
The
and
k8s.io/
:
:
v1
kind
:
Pod
metadata
:
name
:
:
:
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.
1.3.7 - Field Selectors
let you
This
kubectl
field is
Running
:
=
Running
Note:
filters
kubectl
queries
and
and
=
baz
=
,
==
, and
!=
=
and
==
kubectl
default
=
default
As with
label
kubectl
Running
and the
Always
:
=
=
=
Always
kubectl
default
=
default
1.3.8 - Finalizers
,
202
Kubernetes considers the deletion

 complete and deletes the object.
your own.
the
field
Returns a
202
field set
using the
Like
labels
,
Job
Note:
to resources for a reason, so forcefully removing them can lead to issues in
Read
1.3.9 - Owners and Dependents
owners

creates
field that
field to
true
if a
Note:
must
,
and an
.
has the
Kubernetes also adds finalizers to an owner resource when you use either
.
orphan
object.
Learn 

more about
.
.
.
1.3.10 - Recommended Labels
Note:
. Labels
Labels
Key
Example
Type
mysql
string
string
SemVer 1.0
5.7.21
string
database
string
wordpress
string
helm
string
object:
:
apps/v1
kind
:
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
database
:
wordpress
:
helm
of
wordpress
Examples
and
Service
The
:
apps/v1
kind
:
metadata
:
labels
:
:
myservice
:
...
The
Service
:
v1
kind
:
Service
metadata
:
labels
:
:
myservice
:
...
:
apps/v1
kind
:
metadata
:
labels
:
:
wordpress
app.kub

ernetes.io/instance
:
:
"4.9.4"
:
helm
:
server
:
wordpress
...
The
Service
:
v1
kind
:
Service
metadata
:
labels
:
:
wordpress
:
:
"4.9.4"
:
helm
:
server
:
wordpress
...
:
apps/v1
kind
:
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
helm
:
database
:
wordpress
...
The
Service
:
v1
kind
:
Service
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
helm
:
database
:
wordpress
...
and
Service
2 - Cluster Architecture
2.1 - Nodes
Kubernetes runs your workload by placing containers into Pods to run on
Nodes
.
A node may be a virtual or physical machine, depending on the cluster. Each node
is managed by the
and contains the services necessary to run
Pods
.
Typically you have several nodes in a cluster; in a learning or resource-limited
environment, you might have only one node.
The
on a node include the
kubelet
, a
, and the
.
Management
There are two main ways to have Nodes added to the
API server
:
The kubelet on a node self-registers to the control plane
You (or another human user) manually add a Node object
After you create a Node
object
,
or the kubelet on a node self-registers, the control plane checks whether the new Node object is
valid. For example, if you try to create a Node from the following JSON manifest:
{
"kind"
:
"Node"
,
:
"v1"
,
: {
"name"
:
"10.240.79.157"
,
"labels"
: {
"name"
:
"my-first-k8s-node"
}
}
}
Kubernetes creates a Node object internally (the representation). Kubernetes checks
that a kubelet has registered to the API server that matches the
field of the Node. If the node is healthy (i.e. all necessary services are running),
then it is eligible to run a Pod. Otherwise, that node is ignored for any cluster activity
until it becomes healthy.
Note:
Kubernetes keeps the object for the invalid Node and continues checking to see whether
it becomes healthy.
You, or a
, must explicitly
delete the Node object to stop that health checking.
The name of a Node object must be a valid
.
Node name uniqueness
The
name
identifies a Node. Two Nodes
cannot have the same name at the same time. Kubernetes also assumes that a resource with the same
name is the same object. In case of a Node, it is implicitly assumed that an instance using the
same name will have the same state (e.g. network settings, root disk contents)
and attributes like node labels. This may lead to
inconsistencies if an instance was modified without changing its name. If the Node needs to be
replaced or updated significantly, the existing Node object needs to be removed from API server
first and re-added after the update.
Self-registration of Nodes
When the kubelet flag
is true (the default), the kubelet will attempt to
register itself with the API server. This is the preferred pattern, used by most distros.
For self-registration, the kubelet is started with the following options:
- Path to credent

ials to authenticate itself to the API server.
- How to talk to a
cloud provider
to read metadata about itself.
- Automatically register with the API server.
--register-with-taints
- Register the node with the given list of
taints
(comma separated
).
No-op if
register-node
is false.
--node-ip
- IP address of the node.
--node-labels
-
Labels
to add when registering the node
in the cluster (see label restrictions enforced by the
).
- Specifies how often kubelet posts its node status to the API server.
When the
Node authorization mode
and
are enabled, kubelets are only authorized to create/modify their own Node resource.
Note:
section,
when Node configuration needs to be updated, it is a good practice to re-register
the node with the API server. For example, if the kubelet being restarted with
the new set of
, but the same Node name is used, the change will
not take an effect, as labels are being set on the Node registration.
Pods already scheduled on the Node may misbehave or cause issues if the Node
configuration will be changed on kubelet restart. For example, already running
Pod may be tainted against the new labels assigned to the Node, while other
Pods, that are incompatible with that Pod will be scheduled based on this new
label. Node re-registration ensures all Pods will be drained and properly
re-scheduled.
Manual Node administration
You can create and modify Node objects using
kubectl
.
When you want to create Node objects manually, set the kubelet flag
--register-node=false
.
You can modify Node objects regardless of the setting of
.
For example, you can set labels on an existing Node or mark it unschedulable.
You can use labels on Nodes in conjunction with node selectors on Pods to control
scheduling. For example, you can constrain a Pod to only be eligible to run on
a subset of the available nodes.
Marking a node as unschedulable prevents the scheduler from placing new pods onto
that Node but does not affect existing Pods on the Node. This is useful as a
preparatory step before a node reboot or other maintenance.
To mark a Node unschedulable, run:
$NODENAME
See
Note:
Pods that are part of a
DaemonSet
tolerate
being run on an unschedulable Node. DaemonSets typically provide node-local services
that should run on the Node even if it is being drained of workload applications.
Node status
A Node's status contains the following information:
Addresses
Conditions
Capacity and Allocatable
Info
You can use
kubectl
to view a Node's status and other details:
kubectl describe node &lt;insert-node-name-here&gt;
Each section of the output is described below.
Addresses
The usage of these fields varies depending on your cloud provider or bare metal configuration.
HostName: The hostname as reported by the node's kernel. Can be overridden via the kubelet
ExternalIP: Typically the IP address of the node that is externally routable (available from
outside the cluster).
InternalIP: Typically the IP address of the node that is routable only within the cluster.
The
field describes the status of all
Running
nodes. Examples of conditions include:
Node conditions, and a description of when each condition applies.
Node Condition
Ready
True
if the node is healthy and ready to accept pods,
False
if the node is not healthy and is not accepting pods, and
Unknown
if the node controller has not heard from the node in the last
node-monitor-grace-period
(default is 40 seconds)
True
if pressure exists on the disk size—that is, if the disk capacity is low; otherwise
False
True
if pressure exists on the node memory—that is, if the node memory is low; otherwise
False
PIDPres

sure
True
if pressure exists on the processes—that is, if there are too many processes on the node; otherwise
False
True
if the network for the node is not correctly configured, otherwise
False
Note:
If you use command-line tools to print details of a cordoned Node, the Condition includes
SchedulingDisabled
.
is not a Condition in the Kubernetes API; instead,
cordoned nodes are marked Unschedulable in their spec.
In the Kubernetes API, a node's condition is represented as part of the
.status
of the Node resource. For example, the following JSON structure describes a healthy node:
"conditions"
:
[
{
"type"
:
"Ready"
,
"status"
:
"True"
,
"reason"
:
"KubeletReady"
,
"message"
:
"kubelet is posting ready status"
,
"lastHeartbeatTime"
:
"2019-06-05T18:38:35Z"
,
"lastTransitionTime"
:
"2019-06-05T11:41:27Z"
}
]
If the
status
of the Ready condition remains
Unknown
or
False
for longer
than the
pod-eviction-timeout
(an argument passed to the
), then the
node controller
triggers
for all Pods assigned to that node. The default eviction timeout duration is
five minutes
.
In some cases when the node is unreachable, the API server is unable to communicate
with the kubelet on the node. The decision to delete the pods cannot be communicated to
the kubelet until communication with the API server is re-established. In the meantime,
the pods that are scheduled for deletion may continue to run on the partitioned node.
The node controller does not force delete pods until it is confirmed that they have stopped
running in the cluster. You can see the pods that might be running on an unreachable node as
or
Unknown
state. In cases where Kubernetes cannot deduce from the
underlying infrastructure if a node has permanently left a cluster, the cluster administrator
may need to delete the node object by hand. Deleting the node object from Kubernetes causes
all the Pod objects running on the node to be deleted from the API server and frees up their
names.
When problems occur on nodes, the Kubernetes control plane automatically creates
taints
that match the conditions
affecting the node.
The scheduler takes the Node's taints into consideration when assigning a Pod to a Node.
Pods can also have
that let
them run on a Node even though it has a specific taint.
See
Taint Nodes by Condition
Describes the resources available on the node: CPU, memory, and the maximum
number of pods that can be scheduled onto the node.
The fields in the capacity block indicate the total amount of resources that a
Node has. The allocatable block indicates the amount of resources on a
Node that is available to be consumed by normal Pods.
You may read more about capacity and allocatable resources while learning how
to
reserve compute resources
on a Node.
Info
Describes general information about the node, such as kernel version, Kubernetes
version (kubelet and kube-proxy version), container runtime details, and which
operating system the node uses.
The kubelet gathers this information from the node and publishes it into
Heartbeats
Heartbeats, sent by Kubernetes nodes, help your cluster determine the
availability of each node, and to take action when failures are detected.
For nodes there are two forms of heartbeats:
updates to the
.status
of a Node
Lease
objects
within the
namespace
.
Each Node has an associated Lease object.
Compared to updates to
.status
of a Node, a Lease is a lightweight resource.
Using Leases for heartbeats reduces the performance impact of these updates
for large clusters.
The kubelet is responsible for creating and updating the
.status
of Nodes,
and for updating their related Leases.
The kubelet updates the node's
.status
either when there is change in status
or if there has been no update for a configured interval. The default interval
for
.status
updates to Node

s is 5 minutes, which is much longer than the 40
second default timeout for unreachable nodes.
The kubelet creates and then updates its Lease object every 10 seconds
(the default update interval). Lease updates occur independently from
updates to the Node's
.status
. If the Lease update fails, the kubelet retries,
using exponential backoff that starts at 200 milliseconds and capped at 7 seconds.
Node controller
The node
is a
Kubernetes control plane component that manages various aspects of nodes.
The node controller has multiple roles in a node's life. The first is assigning a
CIDR block to the node when it is registered (if CIDR assignment is turned on).
The second is keeping the node controller's internal list of nodes up to date with
the cloud provider's list of available machines. When running in a cloud
environment and whenever a node is unhealthy, the node controller asks the cloud
provider if the VM for that node is still available. If not, the node
controller deletes the node from its list of nodes.
The third is monitoring the nodes' health. The node controller is
responsible for:
In the case that a node becomes unreachable, updating the
Ready
condition
in the Node's
.status
field. In this case the node controller sets the
Ready
condition to
Unknown
.
If a node remains unreachable: triggering
for all of the Pods on the unreachable node. By default, the node controller
waits 5 minutes between marking the node as
Unknown
and submitting
the first eviction request.
By default, the node controller checks the state of each node every 5 seconds.
This period can be configured using the
--node-monitor-period
flag on the
Rate limits on eviction
In most cases, the node controller limits the eviction rate to
--node-eviction-rate
(default 0.1) per second, meaning it won't evict pods
from more than 1 node per 10 seconds.
The node eviction behavior changes when a node in a given availability zone
becomes unhealthy. The node controller checks what percentage of nodes in the zone
are unhealthy (the
Ready
condition is
Unknown
or
False
) at
the same time:
If the fraction of unhealthy nodes is at least
--unhealthy-zone-threshold
(default 0.55), then the eviction rate is reduced.
If the cluster is small (i.e. has less than or equal to
--large-cluster-size-threshold
nodes - default 50), then evictions are stopped.
Otherwise, the eviction rate is reduced to
--secondary-node-eviction-rate
(default 0.01) per second.
The reason these policies are implemented per availability zone is because one
availability zone might become partitioned from the control plane while the others remain
connected. If your cluster does not span multiple cloud provider availability zones,
then the eviction mechanism does not take per-zone unavailability into account.
A key reason for spreading your nodes across availability zones is so that the
workload can be shifted to healthy zones when one entire zone goes down.
Therefore, if all nodes in a zone are unhealthy, then the node controller evicts at
the normal rate of
. The corner case is when all zones are
completely unhealthy (none of the nodes in the cluster are healthy). In such a
case, the node controller assumes that there is some problem with connectivity
between the control plane and the nodes, and doesn't perform any evictions.
(If there has been an outage and some nodes reappear, the node controller does
evict pods from the remaining nodes that are unhealthy or unreachable).
The node controller is also responsible for evicting pods running on nodes with
NoExecute
taints, unless those pods tolerate that taint.
The node controller also adds
taints
corresponding to node problems like node unreachable or not ready. This means
that the scheduler won't place Pods onto unhealthy nodes.
Resource capacity tracking
Node objects track information about the Node's resource capacity: for example, the amount
of memory available and 

the number of CPUs.
Nodes that
self register
report their capacity during
registration. If you
manually
add a Node, then
you need to set the node's capacity information when you add it.
scheduler
ensures that
there are enough resources for all the Pods on a Node. The scheduler checks that the sum
of the requests of containers on the node is no greater than the node's capacity.
That sum of requests includes all containers managed by the kubelet, but excludes any
containers started directly by the container runtime, and also excludes any
processes running outside of the kubelet's control.
Note:
If you want to explicitly reserve resources for non-Pod processes, see
reserve resources for system daemons
.
Node topology
If you have enabled the
, then
the kubelet can use topology hints when making resource assignment decisions.
See
Control Topology Management Policies on a Node
Graceful node shutdown
The kubelet attempts to detect node system shutdown and terminates pods running on the node.
Kubelet ensures that pods follow the normal
pod termination process
during the node shutdown.
The Graceful node shutdown feature depends on systemd since it takes advantage of
systemd inhibitor locks
to
delay the node shutdown with a given duration.
Graceful node shutdown is controlled with the
which is
enabled by default in 1.21.
Note that by default, both configuration options described below,
and
are set to zero,
thus not activating the graceful node shutdown functionality.
To activate the feature, the two kubelet config settings should be configured appropriately and
set to non-zero values.
During a graceful shutdown, kubelet terminates pods in two phases:
Terminate regular pods running on the node.
Terminate
critical pods
running on the node.
Graceful node shutdown feature is configured with two
options:
:
Specifies the total duration that the node should delay the shutdown by. This is the total
grace period for pod termination for both regular and
.
:
Specifies the duration used to terminate
during a node shutdown. This value should be less than
.
shutdownGracePeriod=30s
, and
shutdownGracePeriodCriticalPods=10s
, kubelet will delay the node shutdown by
30 seconds. During the shutdown, the first 20 (30-10) seconds would be reserved
for gracefully terminating normal pods, and the last 10 seconds would be
reserved for terminating
.
Note:
When pods were evicted during the graceful node shutdown, they are marked as shutdown.
Running
shows the status of the evicted pods as
.
And
indicates that the pod was evicted because of node shutdown:
Reason:         Terminated
Message:        Pod was terminated in response to imminent node shutdown.
Pod Priority based graceful node shutdown
To provide more flexibility during graceful node shutdown around the ordering
of pods during shutdown, graceful node shutdown honors the PriorityClass for
Pods, provided that you enabled this feature in your cluster. The feature
allows cluster administers to explicitly define the ordering of pods
during graceful node shutdown based on
priority classes
.
The
feature, as described
above, shuts down pods in two phases, non-critical pods, followed by critical
pods. If additional flexibility is needed to explicitly define the ordering of
pods during shutdown in a more granular way, pod priority based graceful
shutdown can be used.
When graceful node shutdown honors pod priorities, this makes it possible to do
graceful node shutdown in multiple phases, each phase shutting down a
particular pri

ority class of pods. The kubelet can be configured with the exact
phases and shutdown time per phase.
Assuming the following custom pod
in a cluster,
Pod priority class name
Pod priority class value
custom-class-a
100000
custom-class-b
10000
custom-class-c
1000
regular/unset
0
Within the
the settings for
could look like:
Shutdown period
100000
10 seconds
10000
180 seconds
1000
120 seconds
0
60 seconds
The corresponding kubelet config YAML configuration would be:
:
-
priority
:
100000
:
10
-
priority
:
10000
:
180
-
priority
:
1000
:
120
-
priority
:
0
:
60
The above table implies that any pod with
priority
value &gt;= 100000 will get
just 10 seconds to stop, any pod with value &gt;= 10000 and &lt; 100000 will get 180
seconds to stop, any pod with value &gt;= 1000 and &lt; 10000 will get 120 seconds to stop.
Finally, all other pods will get 60 seconds to stop.
One doesn't have to specify values corresponding to all of the classes. For
example, you could instead use these settings:
100000
300 seconds
1000
0
60 seconds
In the above case, the pods with
will go into the same bucket
as
for shutdown.
If there are no pods in a particular range, then the kubelet does not wait
for pods in that priority range. Instead, the kubelet immediately skips to the
next priority class value range.
If this feature is enabled and no configuration is provided, then no ordering
action will be taken.
Using this feature requires enabling the
, and setting
in the
to the desired configuration containing the pod priority class values and
their respective shutdown periods.
Note:
The ability to take Pod priority into account during graceful node shutdown was introduced
as an Alpha feature in Kubernetes v1.23. In Kubernetes 1.26
the feature is Beta and is enabled by default.
Metrics
graceful_shutdown_start_time_seconds
and
graceful_shutdown_end_time_seconds
are emitted under the kubelet subsystem to monitor node shutdowns.
Non Graceful node shutdown
A node shutdown action may not be detected by kubelet's Node Shutdown Manager,
either because the command does not trigger the inhibitor locks mechanism used by
kubelet or because of a user error, i.e., the ShutdownGracePeriod and
ShutdownGracePeriodCriticalPods are not configured properly. Please refer to above
section
When a node is shutdown but not detected by kubelet's Node Shutdown Manager, the pods
that are part of a StatefulSet will be stuck in terminating status on
the shutdown node and cannot move to a new running node. This is because kubelet on
the shutdown node is not available to delete the pods so the StatefulSet cannot
create a new pod with the same name. If there are volumes used by the pods, the
VolumeAttachments will not be deleted from the original shutdown node so the volumes
used by these pods cannot be attached to a new running node. As a result, the
application running on the StatefulSet cannot function properly. If the original
shutdown node comes up, the pods will be deleted by kubelet and new pods will be
created on a different running node. If the original shutdown node does not come up,
these pods will be stuck in terminating status on the shutdown node forever.
To mitigate the above situation, a user can manually add the taint
with either
NoExecute
or
effect to a Node marking it out-of-service.
If the
, and a Node is marked out-of-service with this taint, the
pods on the 

node will be forcefully deleted if there are no matching tolerations on it and volume
detach operations for the pods terminating on the node will happen immediately. This allows the
Pods on the out-of-service node to recover quickly on a different node.
During a non-graceful shutdown, Pods are terminated in the two phases:
Force delete the Pods that do not have matching
out-of-service
tolerations.
Immediately perform detach volume operation for such pods.
Note:
Before adding the taint
, it should be verified
that the node is already in shutdown or power off state (not in the middle of
restarting).
The user is required to manually remove the out-of-service taint after the pods are
moved to a new node and the user has checked that the shutdown node has been
recovered since the user was the one who originally added the taint.
Swap memory management
Prior to Kubernetes 1.22, nodes did not support the use of swap memory, and a
kubelet would by default fail to start if swap was detected on a node. In 1.22
onwards, swap memory support can be enabled on a per-node basis.
To enable swap on a node, the
NodeSwap
feature gate must be enabled on
the kubelet, and the
--fail-swap-on
command line flag or
configuration setting
must be set to false.
Warning:
When the memory swap feature is turned on, Kubernetes data such as the content
of Secret objects that were written to tmpfs now could be swapped to disk.
A user can also optionally configure
memorySwap.swapBehavior
in order to
specify how a node will use swap memory. For example,
:
:
LimitedSwap
The available configuration options for
are:
: Kubernetes workloads are limited in how much swap they can
use. Workloads on the node not managed by Kubernetes can still swap.
UnlimitedSwap
: Kubernetes workloads can use as much swap memory as they
request, up to the system limit.
If configuration for
is not specified and the feature gate is
enabled, by default the kubelet will apply the same behaviour as the
setting.
The behaviour of the
setting depends if the node is running with
v1 or v2 of control groups (also known as "cgroups"):
cgroupsv1:
Kubernetes workloads can use any combination of memory and
swap, up to the pod's memory limit, if set.
cgroupsv2:
Kubernetes workloads cannot use swap memory.
For more information, and to assist with testing and provide feedback, please
see
KEP-2400
and its
.
that make up a node.
Read the
API definition for Node
.
Read the
Node
section of the architecture design document.
Read about
.
2.2 - Communication between Nodes and the Control Plane
This document catalogs the communication paths between the API server and the Kubernetes cluster.
The intent is to allow users to customize their installation to harden the network configuration
such that the cluster can be run on an untrusted network (or on fully public IPs on a cloud
provider).
Node to Control Plane
Kubernetes has a "hub-and-spoke" API pattern. All API usage from nodes (or the pods they run)
terminates at the API server. None of the other control plane components are designed to expose
remote services. The API server is configured to listen for remote connections on a secure HTTPS
port (typically 443) with one or more forms of client
enabled.
One or more forms of
should be
enabled, especially if
or
Nodes should be provisioned with the public root certificate for the cluster such that they can
connect securely to the API server along with valid client credentials. A good approach is that the
client credentials provided to the kubelet are in the form of a client certificate. See
for automated provisioning of kubelet client certificates.
Pods that wish to co

nnect to the API server can do so securely by leveraging a service account so
that Kubernetes will automatically inject the public root certificate and a valid bearer token
into the pod when it is instantiated.
The
service (in
default
namespace) is configured with a virtual IP address that is
redirected (via
) to the HTTPS endpoint on the API server.
The control plane components also communicate with the API server over the secure port.
As a result, the default operating mode for connections from the nodes and pods running on the
nodes to the control plane is secured by default and can run over untrusted and/or public
networks.
Control plane to node
There are two primary communication paths from the control plane (the API server) to the nodes.
The first is from the API server to the kubelet process which runs on each node in the cluster.
The second is from the API server to any node, pod, or service through the API server's
proxy
functionality.
API server to kubelet
The connections from the API server to the kubelet are used for:
Fetching logs for pods.
Attaching (usually through
kubectl
) to running pods.
Providing the kubelet's port-forwarding functionality.
These connections terminate at the kubelet's HTTPS endpoint. By default, the API server does not
verify the kubelet's serving certificate, which makes the connection subject to man-in-the-middle
attacks and
unsafe
to run over untrusted and/or public networks.
To verify this connection, use the
flag to provide the API
server with a root certificate bundle to use to verify the kubelet's serving certificate.
If that is not possible, use
SSH tunneling
between the API server and kubelet if
required to avoid connecting over an
untrusted or public network.
Finally,
Kubelet authentication and/or authorization
should be enabled to secure the kubelet API.
API server to nodes, pods, and services
The connections from the API server to a node, pod, or service default to plain HTTP connections
and are therefore neither authenticated nor encrypted. They can be run over a secure HTTPS
connection by prefixing
https:
to the node, pod, or service name in the API URL, but they will
not validate the certificate provided by the HTTPS endpoint nor provide client credentials. So
while the connection will be encrypted, it will not provide any guarantees of integrity. These
connections
are not currently safe
to run over untrusted or public networks.
SSH tunnels
Kubernetes supports SSH tunnels to protect the control plane to nodes communication paths. In this
configuration, the API server initiates an SSH tunnel to each node in the cluster (connecting to
the SSH server listening on port 22) and passes all traffic destined for a kubelet, node, pod, or
service through the tunnel.
This tunnel ensures that the traffic is not exposed outside of the network in which the nodes are
running.
Note:
SSH tunnels are currently deprecated, so you shouldn't opt to use them unless you know what you
are doing. The
Konnectivity service
is a replacement for this
communication channel.
As a replacement to the SSH tunnels, the Konnectivity service provides TCP level proxy for the
control plane to cluster communication. The Konnectivity service consists of two parts: the
Konnectivity server in the control plane network and the Konnectivity agents in the nodes network.
The Konnectivity agents initiate connections to the Konnectivity server and maintain the network
connections.
After enabling the Konnectivity service, all control plane to nodes traffic goes through these
Follow the
Konnectivity service task
to set
up the Konnectivity service in your cluster.
2.3 - Controllers
In robotics and automation, a
control loop
is
a non-terminating loop that regulates the state of a system.
Here is one example of a control loop: a thermostat in a room.
When you set the temperature, that's telling the therm

ostat
about your
. The actual room temperature is the
. The thermostat acts to bring the current state
closer to the desired state, by turning equipment on or off.
cluster
state.
Controller pattern
A controller tracks at least one Kubernetes resource type.
These
objects
have a spec field that represents the desired state. The
controller(s) for that resource are responsible for making the current
state come closer to that desired state.
The controller might carry the action out itself; more commonly, in Kubernetes,
a controller will send messages to the
API server
that have
useful side effects. You'll see examples of this below.
Control via API server
The
Job
controller is an example of a
Kubernetes built-in controller. Built-in controllers manage state by
interacting with the cluster API server.
Job is a Kubernetes resource that runs a
Pod
, or perhaps several Pods, to carry out
a task and then stop.
(Once
scheduled
, Pod objects become part of the
desired state for a kubelet).
When the Job controller sees a new task it makes sure that, somewhere
in your cluster, the kubelets on a set of Nodes are running the right
number of Pods to get the work done.
The Job controller does not run any Pods or containers
itself. Instead, the Job controller tells the API server to create or remove
Pods.
Other components in the
act on the new information (there are new Pods to schedule and run),
and eventually the work is done.
After you create a new Job, the desired state is for that Job to be completed.
The Job controller makes the current state for that Job be nearer to your
desired state: creating Pods that do the work you wanted for that Job, so that
the Job is closer to completion.
Controllers also update the objects that configure them.
For example: once the work is done for a Job, the Job controller
updates that Job object to mark it
Finished
.
(This is a bit like how some thermostats turn a light off to
indicate that your room is now at the temperature you set).
Direct control
In contrast with Job, some controllers need to make changes to
things outside of your cluster.
For example, if you use a control loop to make sure there
are enough
Nodes
in your cluster, then that controller needs something outside the
current cluster to set up new Nodes when needed.
Controllers that interact with external state find their desired state from
the API server, then communicate directly with an external system to bring
the current state closer in line.
(There actually is a
that horizontally scales the nodes in your cluster.)
The important point here is that the controller makes some changes to bring about
your desired state, and then reports the current state back to your cluster's API server.
Other control loops can observe that reported data and take their own actions.
In the thermostat example, if the room is very cold then a different controller
might also turn on a frost protection heater. With Kubernetes clusters, the control
plane indirectly works with IP address management tools, storage services,
cloud provider APIs, and other services by
extending Kubernetes
to implement that.
Desired versus current state
Kubernetes takes a cloud-native view of systems, and is able to handle
constant change.
Your cluster could be changing at any point as work happens and
control loops automatically fix failures. This means that,
potentially, your cluster never reaches a stable state.
As long as the controllers for your cluster are running and able to make
useful changes, it doesn't matter if the overall state is stable or not.
Design
As a tenet of its design, Kubernetes uses lots of controllers that each manage
a particular aspect of cluster state. Most commonly, a particular control loop
(controller) uses one k

ind of resource as its desired state, and has a different
kind of resource that it manages to make that desired state happen. For example,
a controller for Jobs tracks Job objects (to discover new work) and Pod objects
(to run the Jobs, and then to see when the work is finished). In this case
something else creates the Jobs, whereas the Job controller creates Pods.
It's useful to have simple controllers rather than one, monolithic set of control
loops that are interlinked. Controllers can fail, so Kubernetes is designed to
allow for that.
Note:
There can be several controllers that create or update the same kind of object.
Behind the scenes, Kubernetes controllers make sure that they only pay attention
to the resources linked to their controlling resource.
For example, you can have Deployments and Jobs; these both create Pods.
The Job controller does not delete the Pods that your Deployment created,
because there is information (
labels
)
the controllers can use to tell those Pods apart.
Ways of running controllers
Kubernetes comes with a set of built-in controllers that run inside
the
. These
built-in controllers provide important core behaviors.
The Deployment controller and Job controller are examples of controllers that
come as part of Kubernetes itself ("built-in" controllers).
Kubernetes lets you run a resilient control plane, so that if any of the built-in
controllers were to fail, another part of the control plane will take over the work.
You can find controllers that run outside the control plane, to extend Kubernetes.
Or, if you want, you can write a new controller yourself.
You can run your own controller as a set of Pods,
or externally to Kubernetes. What fits best will depend on what that particular
controller does.
Kubernetes control plane
Discover some of the basic
If you want to write your own controller, see
Extension Patterns
in Extending Kubernetes.
2.4 - Leases
Distributed systems often have a need for "leases", which provides a mechanism to lock shared resources and coordinate activity between nodes.
In Kubernetes, the "lease" concept is represented by
Lease
objects in the
API group, which are used for system-critical
capabilities like node heart beats and component-level leader election.
Node Heart Beats
Kubernetes uses the Lease API to communicate kubelet node heart beats to the Kubernetes API server.
For every
Node
, there is a
Lease
object with a matching name in the
namespace. Under the hood, every kubelet heart beat is an UPDATE request to this
Lease
object, updating
the
spec.renewTime
field for the Lease. The Kubernetes control plane uses the time stamp of this field
to determine the availability of this
Node
.
See
Node Lease objects
Leader Election
Leases are also used in Kubernetes to ensure only one instance of a component is running at any given time.
This is used by control plane components like
and
in
HA configurations, where only one instance of the component should be actively running while the other
instances are on stand-by.
API Server Identity
Starting in Kubernetes v1.26, each
uses the Lease API to publish its identity to the
rest of the system. While not particularly useful on its own, this provides a mechanism for clients to
discover how many instances of
are operating the Kubernetes control plane.
Existence of kube-apiserver leases enables future capabilities that may require coordination between
each kube-apiserver.
You can inspect Leases owned by each kube-apiserver by checking for lease objects in the
namespace
kube-apiserver-&lt;sha256-hash&gt;
. Alternatively you can use the label selector
k8s.io/component=kube-apiserver
:
$ kubectl -n kube-system get lease -l k8s.io/component
=

er
NAME                                        HOLDER                                                                           AGE
kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a   kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a_9cbf54e5-1136-44bd-8f9a-1dcd15c346b4   5m33s
kube-apiserver-dz2dqprdpsgnm756t5rnov7yka   kube-apiserver-dz2dqprdpsgnm756t5rnov7yka_84f2a85d-37c1-4b14-b6b9-603e62e4896f   4m23s
kube-apiserver-fyloo45sdenffw2ugwaz3likua   kube-apiserver-fyloo45sdenffw2ugwaz3likua_c5ffa286-8a9a-45d4-91e7-61118ed58d2e   4m43s
The SHA256 hash used in the lease name is based on the OS hostname as seen by kube-apiserver. Each kube-apiserver should be
configured to use a hostname that is unique within the cluster. New instances of kube-apiserver that use the same hostname
will take over existing Leases using a new holder identity, as opposed to instantiating new lease objects. You can check the
hostname used by kube-apisever by checking the value of the
label:
$ kubectl -n kube-system get lease kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a -o yaml
:
kind
:
Lease
metadata
:
:
"2022-11-30T15:37:15Z"
labels
:
k8s.io/component
:
:
name
:
kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a
namespace
:
:
"18171"
uid
:
d6c68901-4ec5-4385-b1ef-2d783738da6c
spec
:
:
kube-apiserver-c4vwjftbvpc5os2vvzle4qg27a_9cbf54e5-1136-44bd-8f9a-1dcd15c346b4
:
3600
renewTime
:
"2022-11-30T18:04:27.912073Z"
Expired leases from kube-apiservers that no longer exist are garbage collected by new kube-apiservers after 1 hour.
2.5 - Cloud Controller Manager
Cloud infrastructure technologies let you run Kubernetes on public, private, and hybrid clouds.
Kubernetes believes in automated, API-driven infrastructure without tight coupling between
component
The cloud-controller-manager is structured using a plugin
mechanism that allows different cloud providers to integrate their platforms with Kubernetes.
Design
The cloud controller manager runs in the control plane as a replicated set of processes
(usually, these are containers in Pods). Each cloud-controller-manager implements
multiple
in a single
process.
Note:
You can also run the cloud controller manager as a Kubernetes
addon
rather than as part
of the control plane.
Cloud controller manager functions
The controllers inside the cloud controller manager include:
The node controller is responsible for updating
Node
objects
when new servers are created in your cloud infrastructure. The node controller obtains information about the
hosts running inside your tenancy with the cloud provider. The node controller performs the following functions:
Update a Node object with the corresponding server's unique identifier obtained from the cloud provider API.
Annotating and labelling the Node object with cloud-specific information, such as the region the node
is deployed into and the resources (CPU, memory, etc) that it has available.
Obtain the node's hostname and network addresses.
Verifying the node's health. In case a node becomes unresponsive, this controller checks with
your cloud provider's API to see if the server has been deactivated / deleted / terminated.
If the node has been deleted from the cloud, the controller de

letes the Node object from your Kubernetes
cluster.
Some cloud provider implementations split this into a node controller and a separate node
lifecycle controller.
Route controller
The route controller is responsible for configuring routes in the cloud
appropriately so that containers on different nodes in your Kubernetes
cluster can communicate with each other.
Depending on the cloud provider, the route controller might also allocate blocks
of IP addresses for the Pod network.
Service controller
Services
integrate with cloud
infrastructure components such as managed load balancers, IP addresses, network
packet filtering, and target health checking. The service controller interacts with your
cloud provider's APIs to set up load balancers and other infrastructure components
when you declare a Service resource that requires them.
This section breaks down the access that the cloud controller manager requires
on various API objects, in order to perform its operations.
The Node controller only works with Node objects. It requires full access
to read and modify Node objects.
v1/Node
:
Get
List
Create
Update
Patch
Watch
Delete
The route controller listens to Node object creation and configures
routes appropriately. It requires Get access to Node objects.
v1/Node
:
Get
The service controller listens to Service object Create, Update and Delete events and then configures Endpoints for those Services appropriately (for EndpointSlices, the kube-controller-manager manages these on demand).
To access Services, it requires List, and Watch access. To update Services, it requires Patch and Update access.
To set up Endpoints resources for the Services, it requires access to Create, List, Get, Watch, and Update.
v1/Service
:
List
Get
Watch
Patch
Update
Others
The implementation of the core of the cloud controller manager requires access to create Event objects, and to ensure secure operation, it requires access to create ServiceAccounts.
v1/Event
:
Create
Patch
Update
v1/ServiceAccount
:
Create
The
RBAC
ClusterRole for the cloud
controller manager looks like:
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
-
""
resources
:
- events
verbs
:
- create
- patch
- update
-
apiGroups
:
-
""
resources
:
- nodes
verbs
:
-
'*'
-
apiGroups
:
-
""
resources
:
- nodes/status
verbs
:
- patch
-
apiGroups
:
-
""
resources
:
- services
verbs
:
- list
- patch
- update
- watch
-
apiGroups
:
-
""
resources
:
- serviceaccounts
verbs
:
- create
-
apiGroups
:
-
""
resources
:
- persistentvolumes
verbs
:
- get
- list
- update
- watch
-
apiGroups
:
-
""
resources
:
verbs
:
- create
- get
- list
- watch
- update
has instructions on running and managing the cloud controller manager.
To upgrade a HA control plane to use the cloud controller manager, see
.
Want to know how to implement your own cloud controller manager, or extend an existing project?
The cloud controller manager uses Go interfaces to allow implementations from any cloud to be plugged in. Specifically, it uses the
CloudProvider
interface defined in
cloud.go
from
kubernetes/cloud-provider
.
The implementation of the shared controllers highlighted in this document (Node, Route, and Service), and some scaffolding along with the shared cloudprovider interface, is part of the Kubernetes core. Implementations specific to cloud providers are outside the core of Kubernetes and implement the
For more information about developing plugins, see
.
2.6 - About cgroup v2
On Linux,
control groups
constrain resources that are allocated to processes.
The
kubelet
and the
underlying container runtime need to interface with cgroups to enforce
resource management for pods and containers
which
includes 

cpu/memory requests and limits for containerized workloads.
There are two versions of cgroups in Linux: cgroup v1 and cgroup v2. cgroup v2 is
the new generation of the
cgroup
API.
What is cgroup v2?
cgroup v2 is the next version of the Linux
cgroup
API. cgroup v2 provides a
unified control system with enhanced resource management
capabilities.
cgroup v2 offers several improvements over cgroup v1, such as the following:
Single unified hierarchy design in API
Safer sub-tree delegation to containers
Newer features like
Pressure Stall Information
Enhanced resource allocation management and isolation across multiple resources
Unified accounting for different types of memory allocations (network memory, kernel memory, etc)
Accounting for non-immediate resource changes such as page cache write backs
Some Kubernetes features exclusively use cgroup v2 for enhanced resource
management and isolation. For example, the
MemoryQoS
feature improves memory QoS
and relies on cgroup v2 primitives.
Using cgroup v2
The recommended way to use cgroup v2 is to use a Linux distribution that
enables and uses cgroup v2 by default.
To check if your distribution uses cgroup v2, refer to
Identify cgroup version on Linux nodes
.
cgroup v2 has the following requirements:
OS distribution enables cgroup v2
Linux Kernel version is 5.8 or later
Container runtime supports cgroup v2. For example:
v1.4 and later
cri-o
v1.20 and later
The kubelet and the container runtime are configured to use the
systemd cgroup driver
Linux Distribution cgroup v2 support
For a list of Linux distributions that use cgroup v2, refer to the
cgroup v2 documentation
Container Optimized OS (since M97)
Ubuntu (since 21.10, 22.04+ recommended)
Debian GNU/Linux (since Debian 11 bullseye)
Fedora (since 31)
Arch Linux (since April 2021)
RHEL and RHEL-like distributions (since 9)
To check if your distribution is using cgroup v2, refer to your distribution's
documentation or follow the instructions in
Identify the cgroup version on Linux nodes
.
You can also enable cgroup v2 manually on your Linux distribution by modifying
the kernel cmdline boot arguments. If your distribution uses GRUB,
systemd.unified_cgroup_hierarchy=1
should be added in
GRUB_CMDLINE_LINUX
under
/etc/default/grub
sudo update-grub
. However, the
recommended approach is to use a distribution that already enables cgroup v2 by
default.
Migrating to cgroup v2
To migrate to cgroup v2, ensure that you meet the
, then upgrade
to a kernel version that enables cgroup v2 by default.
The kubelet automatically detects that the OS is running on cgroup v2 and
performs accordingly with no additional configuration required.
There should not be any noticeable difference in the user experience when
switching to cgroup v2, unless users are accessing the cgroup file system
directly, either on the node or from within the containers.
cgroup v2 uses a different API than cgroup v1, so if there are any
applications that directly access the cgroup file system, they need to be
updated to newer versions that support cgroup v2. For example:
Some third-party monitoring and security agents may depend on the cgroup filesystem.
Update these agents to versions that support cgroup v2.
If you run
cAdvisor
as a stand-alone
DaemonSet for monitoring pods and containers, update it to v0.43.0 or later.
If you use JDK, prefer to use JDK 11.0.16 and later or JDK 15 and later, which
fully support cgroup v2
.
Identify the cgroup version on Linux Nodes
The cgroup version depends on the Linux distribution being used and the
default cgroup version configured on the OS. To check which cgroup version your
distribution uses, run the
stat -fc %T /sys/fs/cgroup/
command on
the node:
For cgroup v2, the output is
cgroup2fs
.
For cgroup v1, the output is
tmpfs.
cgroups
cgroup drivers

2.7 - Container Runtime Interface (CRI)
The CRI is a plugin interface which enables the kubelet to use a wide variety of
container runtimes, without having a need to recompile the cluster components.
You need a working
on
each Node in your cluster, so that the
kubelet
can launch
Pods
and their containers.
The Container Runtime Interface (CRI) is the main protocol for the communication between the kubelet and Container Runtime.
gRPC
kubelet
and
.
The API
The kubelet acts as a client when connecting to the container runtime via gRPC.
The runtime and image service endpoints have to be available in the container
runtime, which can be configured separately within the kubelet by using the
--image-service-endpoint
and
flags
For Kubernetes v1.26, the kubelet prefers to use CRI
v1
.
If a container runtime does not support
v1
of the CRI, then the kubelet tries to
negotiate any older supported version.
The v1.26 kubelet can also negotiate CRI
v1alpha2
, but
this version is considered as deprecated.
If the kubelet cannot negotiate a supported CRI version, the kubelet gives up
and doesn't register as a node.
Upgrading
When upgrading Kubernetes, the kubelet tries to automatically select the
latest CRI version on restart of the component. If that fails, then the fallback
will take place as mentioned above. If a gRPC re-dial was required because the
container runtime has been upgraded, then the container runtime must also
support the initially selected version or the redial is expected to fail. This
requires a restart of the kubelet.
Learn more about the CRI
protocol definition
2.8 - Garbage Collection
cluster resources. This
allows the clean up of resources like the following:
Terminated pods
Completed Jobs
Objects without owner references
Unused containers and container images
Dynamically provisioned PersistentVolumes with a StorageClass reclaim policy of Delete
Stale or expired CertificateSigningRequests (CSRs)
Nodes
deleted in the following scenarios:
On a cloud when the cluster uses a
cloud controller manager
On-premises when the cluster uses an addon similar to a cloud controller
manager
Owners and dependents
Many objects in Kubernetes link to each other through
.
Owner references tell the control plane which objects are dependent on others.
Kubernetes uses owner references to give the control plane, and other API
clients, the opportunity to clean up related resources before deleting an
object. In most cases, Kubernetes manages owner references automatically.
mechanism that some resources also use. For example, consider a
Service
that creates
objects. The Service uses
labels
to allow the control plane to
Note:
must
In v1.20+, if the garbage collector

 detects an invalid cross-namespace
,
and an
.
Cascading deletion
Kubernetes checks for and deletes objects that no longer have owner
references, like the pods left behind when you delete a ReplicaSet. When you
delete an object, you can control whether Kubernetes deletes the object's
dependents automatically, in a process called
. There are
two types of cascading deletion, as follows:
Foreground cascading deletion
Background cascading deletion
You can also control how and when garbage collection deletes resources that have
owner references using Kubernetes
.
In foreground cascading deletion, the owner object you're deleting first enters
a
deletion in progress
state. In this state, the following happens to the
owner object:
The Kubernetes API server sets the object's
field to the time the object was marked for deletion.
The Kubernetes API server also sets the
field to
.
The object remains visible through the Kubernetes API until the deletion
process is complete.
After the owner object enters the deletion in progress state, the controller
deletes the dependents. After deleting all the dependent objects, the controller
deletes the owner object. At this point, the object is no longer visible in the
During foreground cascading deletion, the only dependents that block owner
deletion are those that have the
ownerReference.blockOwnerDeletion=true
field.
See
In background cascading deletion, the Kubernetes API server deletes the owner
object immediately and the controller cleans up the dependent objects in
the background. By default, Kubernetes uses background cascading deletion unless
you manually use foreground deletion or choose to orphan the dependent objects.
See
Orphaned dependents
When Kubernetes deletes an owner object, the dependents left behind are called
orphan
objects. By default, Kubernetes deletes dependent objects. To learn how
to override this behaviour, see
.
Garbage collection of unused containers and images
The
kubelet
performs garbage
collection on unused images every five minutes and on unused containers every
minute. You should avoid using external garbage collection tools, as these can
break the kubelet behavior and remove containers that should exist.
To configure options for unused container and image garbage collection, tune the
kubelet using a
and change the parameters related to garbage collection using the
resource type.
Container image lifecycle
Kubernetes manages the lifecycle of all images through its
image manager
,
which is part of the kubelet, with the cooperation of
cadvisor
. The kubelet
considers the following disk usage limits when making garbage collection
decisions:
HighThresholdPercent
LowThresholdPercent
Disk usage above the configured
value triggers garbage
collection, which deletes images in order based on the last time they were used,
starting with the oldest first. The kubelet deletes images
until disk usage reaches the
value.
Container garbage collection
The kubelet garbage collects unused containers based on the following variables,
which you can define:
MinAge
: the minimum age at which the kubelet can garbage collect a
container. Disable by setting to
0
.
MaxPerPodContainer
: the maximum number of dead containers each Pod pair
can have. Disable by setting to less than
0
.
MaxC

ontainers
: the maximum number of dead containers the cluster can have.
Disable by setting to less than
0
.
In addition to these variables, the kubelet garbage collects unidentified and
deleted containers, typically starting with the oldest first.
and
MaxContainers
may potentially conflict with each other
in situations where retaining the maximum number of containers per Pod
(
) would go outside the allowable total of global dead
containers (
). In this situation, the kubelet adjusts
to address the conflict. A worst-case scenario would be to
downgrade
to
1
and evict the oldest containers.
Additionally, containers owned by pods that have been deleted are removed once
they are older than
MinAge
.
Note:
The kubelet only garbage collects the containers it manages.
Configuring garbage collection
You can tune garbage collection of resources by configuring options specific to
the controllers managing those resources. The following pages show you how to
configure garbage collection:
Configuring cascading deletion of Kubernetes objects
Configuring cleanup of finished Jobs
ownership of Kubernetes objects
.
Learn more about Kubernetes
.
(beta) that cleans up finished Jobs.
3 - Containers
Each container that you run is repeatable; the standardization from having
dependencies included means that you get the same behavior wherever you
run it.
Containers decouple applications from underlying host infrastructure.
This makes deployment easier in different cloud or OS environments.
Each
node
cluster runs the containers that form the
Pods
assigned to that node.
Containers in a Pod are co-located and co-scheduled to run on the same node.
Container images
A
is a ready-to-run
software package, containing everything needed to run an application:
the code and any runtime it requires, application and system libraries,
and default values for any essential settings.
Containers are intended to be stateless and
immutable
:
you should not change
the code of a container that is already running. If you have a containerized
application and want to make changes, the correct process is to build a new
image that includes the change, then recreate the container to start from the
updated image.
,
CRI-O
,
.
Usually, you can allow your cluster to pick the default container runtime
for a Pod. If you need to use more than one container runtime in your cluster,
you can specify the
for a Pod to make sure that Kubernetes runs those containers using a
particular container runtime.
You can also use RuntimeClass to run different Pods with the same container
runtime but with different settings.
3.1 - Images
A container image represents binary data that encapsulates an application and all its
software dependencies. Container images are executable software bundles that can run
standalone and that make very well defined assumptions about their runtime environment.
You typically create a container image of your application and push it to a registry
before referring to it in a
Pod
This page provides an outline of the container image concept.
Note:
If you are looking for the container images for a Kubernetes
release (such as v1.26, the latest minor release),
visit
.
Image names
Container images are usually given a name such as
pause
,
example/mycontainer
, or
.
Images can also include a registry hostname; for example:
fictional.registry.example/imagename
,
and possibly a port number as well; for example:
fictional.registry.example:10

443/imagename
.
If you don't specify a registry hostname, Kubernetes assumes that you mean the Docker public registry.
After the image name part you can add a
tag
(in the same way you would when using with commands like
docker
or
podman
).
Tags let you identify different versions of the same series of images.
Image tags consist of lowercase and uppercase letters, digits, underscores (
_
),
periods (
.
), and dashes (
-
).
There are additional rules about where you can place the separator
characters (
_
,
-
, and
.
) inside an image tag.
If you don't specify a tag, Kubernetes assumes you mean the tag
latest
.
Updating images
When you first create a
,
, Pod, or other
object that includes a Pod template, then by default the pull policy of all
containers in that pod will be set to
if it is not explicitly
specified. This policy causes the
kubelet
to skip pulling an
image if it already exists.
Image pull policy
The
for a container and the tag of the image affect when the
kubelet
attempts to pull (download) the specified image.
Here's a list of the values you can set for
and the effects
these values have:
the image is pulled only if it is not already present locally.
Always
every time the kubelet launches a container, the kubelet queries the container
image registry to resolve the name to an image
digest
. If the kubelet has a
container image with that exact digest cached locally, the kubelet uses its cached
image; otherwise, the kubelet pulls the image with the resolved digest,
and uses that image to launch the container.
Never
the kubelet does not try fetching the image. If the image is somehow already present
locally, the kubelet attempts to start the container; otherwise, startup fails.
See
pre-pulled images
The caching semantics of the underlying image provider make even
efficient, as long as the registry is reliably accessible.
Your container runtime can notice that the image layers already exist on the node
so that they don't need to be downloaded again.
Note:
You should avoid using the
:latest
tag when deploying containers in production as
it is harder to track which version of the image is running and more difficult to
roll back properly.
Instead, specify a meaningful tag such as
v1.42.0
.
To make sure the Pod always uses the same version of a container image, you can specify
the image's digest;
replace
&lt;image-name&gt;:&lt;tag&gt;
with
&lt;image-name&gt;@&lt;digest&gt;
image@sha256:45b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5cb2
).
When using image tags, if the image registry were to change the code that the tag on that image represents, you might end up with a mix of Pods running the old and new code. An image digest uniquely identifies a specific version of the image, so Kubernetes runs the same code every time it starts a container with that image name and digest specified. Specifying an image by digest fixes the code that you run so that a change at the registry cannot lead to that mix of versions.
There are third-party
that mutate Pods (and pod templates) when they are created, so that the
running workload is defined based on an image digest rather than a tag.
That might be useful if you want to make sure that all your workload is
running the same code no matter what tag changes happen at the registry.
Default image pull policy
When you (or a controller) submit a new Pod to the API server, your cluster sets the
field when specific conditions are met:
if you omit the
field, and the tag for the container image is
:latest
,
is automatically set to
Always
;
field, and you don't specify the tag for the
container image,
Always
;
field, and you specify the tag for the
container image that isn't
:latest

, the
.
Note:
of the container is always set when the object is
first
created
, and is not updated if the image's tag later changes.
For example, if you create a Deployment with an image whose tag is
not
:latest
, and later update that Deployment's image to a
:latest
tag, the
field will
not
change to
Always
. You must manually change
the pull policy of any object after its initial creation.
Required image pull
If you would like to always force a pull, you can do one of the following:
Set the
of the container to
Always
.
Omit the
and use
:latest
as the tag for the image to use;
Kubernetes will set the policy to
Always
when you submit the Pod.
Omit the
and the tag for the image to use;
Always
Enable the
When a kubelet starts creating containers for a Pod using a container runtime,
it might be possible the container is in
Waiting
state because of
.
The status
means that a container could not start because Kubernetes
could not pull a container image (for reasons such as invalid image name, or pulling
from a private registry without
imagePullSecret
). The
BackOff
part indicates
that Kubernetes will keep trying to pull the image, with an increasing back-off delay.
Kubernetes raises the delay between each attempt until it reaches a compiled-in limit,
which is 300 seconds (5 minutes).
Multi-architecture images with image indexes
As well as providing binary images, a container registry can also serve a
container image index
. An image index can point to multiple
image manifests
for architecture-specific versions of a container. The idea is that you can have a name for an image (for example:
pause
,
,
) and allow different systems to fetch the right binary image for the machine architecture they are using.
Kubernetes itself typically names container images with a suffix
-$(ARCH)
. For backward compatibility, please generate the older images with suffixes. The idea is to generate say
pause
image which has the manifest for all the arch(es) and say
pause-amd64
which is backwards compatible for older configurations or YAML files which may have hard coded the images with suffixes.
Using a private registry
Private registries may require keys to read images from them.
Credentials can be provided in several ways:
Configuring Nodes to Authenticate to a Private Registry
all pods can read any configured private registries
requires node configuration by cluster administrator
Kubelet Credential Provider to dynamically fetch credentials for private registries
kubelet can be configured to use credential provider exec plugin
for the respective private registry.
Pre-pulled Images
all pods can use any images cached on a node
requires root access to all nodes to set up
only pods which provide own keys can access the private registry
Vendor-specific or local extensions
if you're using a custom node configuration, you (or your cloud
provider) can implement your mechanism for authenticating the node
to the container registry.
These options are explained in more detail below.
Configuring nodes to authenticate to a private registry
Specific instructions for setting credentials depends on the container runtime and registry you chose to use. You should refer to your solution's documentation for the most accurate information.
For an example of configuring a private container image registry, see the
task. That example uses a private registry in Docker Hub.
Kubelet credential provider for authenticated image pulls
Note:
This approach is especially suitable when kubelet needs to fetch registry credentials dynamically.
Most commonly used for registries provided by clou

d providers where auth tokens are short-lived.
You can configure the kubelet to invoke a plugin binary to dynamically fetch registry credentials for a container image.
This is the most robust and versatile way to fetch credentials for private registries, but also requires kubelet-level configuration to enable.
See
Interpretation of config.json
The interpretation of
varies between the original Docker
implementation and the Kubernetes interpretation. In Docker, the
auths
keys
can only specify root URLs, whereas Kubernetes allows glob URLs as well as
prefix-matched paths. This means that a
like this is valid:
{
"auths"
: {
"*my-registry.io/images"
: {
"auth"
:
"…"
}
}
}
The root URL (
*my-registry.io
) is matched by using the following syntax:
pattern:
    { term }

term:
    '*'         matches any sequence of non-Separator characters
    '?'         matches any single non-Separator character
    '[' [ '^' ] { character-range } ']'
                character class (must be non-empty)
    c           matches character c (c != '*', '?', '\\', '[')
    '\\' c      matches character c

character-range:
    c           matches character c (c != '\\', '-', ']')
    lo '-' hi   matches character c for lo &lt;= c &lt;= hi
Image pull operations would now pass the credentials to the CRI container
runtime for every valid pattern. For example the following container image names
would match successfully:
my-registry.io/images
my-registry.io/images/my-image
my-registry.io/images/another-image
sub.my-registry.io/images/my-image
a.sub.my-registry.io/images/my-image
The kubelet performs image pulls sequentially for every found credential. This
means, that multiple entries in
are possible, too:
{
"auths"
: {
"my-registry.io/images"
: {
"auth"
:
"…"
},
"my-registry.io/images/subpath"
: {
"auth"
:
"…"
}
}
}
If now a container specifies an image
my-registry.io/images/subpath/my-image
to be pulled, then the kubelet will try to download them from both
authentication sources if one of them fails.
Pre-pulled images
Note:
This approach is suitable if you can control node configuration. It
will not work reliably if your cloud provider manages nodes and replaces
them automatically.
By default, the kubelet tries to pull each image from the specified registry.
However, if the
property of the container is set to
or
Never
,
then a local image is used (preferentially or exclusively, respectively).
If you want to rely on pre-pulled images as a substitute for registry authentication,
you must ensure all nodes in the cluster have the same pre-pulled images.
This can be used to preload certain images for speed or as an alternative to authenticating to a private registry.
All pods will have read access to any pre-pulled images.
Specifying imagePullSecrets on a Pod
Note:
This is the recommended approach to run containers based on images
in private registries.
Kubernetes supports specifying container image registry keys on a Pod.
must all be in the same namespace as the Pod. The referenced
Secrets must be of type
kubernetes.io/dockercfg
or
.
Creating a Secret with a Docker config
You need to know the username, registry password and client email address for authenticating
to the registry, as well as its hostname.
Run the following command, substituting the appropriate uppercase values:
kubectl create secret docker-registry &lt;name&gt; --docker-server
=
DOCKER_REGISTRY_SERVER --docker-username
=
DOCKER_USER --docker-password
=
DOCKER_PASSWORD --docker-email
=
DOCKER_EMAIL
If you already have a Docker credentials file then, rather than using the above
command, you can import the credentials file as a Kubernetes
Secrets
.
Create a Secret based on existing Docker credentials
explains how to set this up.
This is particularly useful if you are using multiple private

 container
registries, as
creates a Secret that
only works with a single private registry.
Note:
Pods can only reference image pull secrets in their own namespace,
so this process needs to be done one time per namespace.
Referring to an imagePullSecrets on a Pod
Now, you can create pods which reference that secret by adding an
section to a Pod definition. Each item in the
array can only
reference a Secret in the same namespace.
cat
&lt;&lt;EOF &gt; pod.yaml
kind: Pod
metadata:
name: foo
namespace: awesomeapps
spec:
- name: foo
image: janedoe/awesomeapp:v1
- name: myregistrykey
EOF
cat
&lt;&lt;EOF &gt;&gt; ./kustomization.yaml
- pod.yaml
EOF
This needs to be done for each pod that is using a private registry.
However, setting of this field can be automated by setting the imagePullSecrets
in a
resource.
Check
Add ImagePullSecrets to a Service Account
for detailed instructions.
You can use this in conjunction with a per-node
. The credentials
will be merged.
Use cases
There are a number of solutions for configuring private registries. Here are some
common use cases and suggested solutions.
Cluster running only non-proprietary (e.g. open-source) images. No need to hide images.
Use public images from a public registry
No configuration required.
Some cloud providers automatically cache or mirror public images, which improves availability and reduces the time to pull images.
Cluster running some proprietary images which should be hidden to those outside the company, but
visible to all cluster users.
Use a hosted private registry
Manual configuration may be required on the nodes that need to access to private registry
Or, run an internal private registry behind your firewall with open read access.
No Kubernetes configuration is required.
Use a hosted container image registry service that controls image access
It will work better with cluster autoscaling than manual node configuration.
Or, on a cluster where changing the node configuration is inconvenient, use
.
Cluster with proprietary images, a few of which require stricter access control.
Ensure
AlwaysPullImages admission controller
is active. Otherwise, all Pods potentially have access to all images.
Move sensitive data into a "Secret" resource, instead of packaging it in an image.
A multi-tenant cluster where each tenant needs own private registry.
Ensure
is active. Otherwise, all Pods of all tenants potentially have access to all images.
Run a private registry with authorization required.
Generate registry credential for each tenant, put into secret, and populate secret to each tenant namespace.
The tenant adds that secret to imagePullSecrets of each namespace.
If you need access to multiple registries, you can create one secret for each registry.
Read the
OCI Image Manifest Specification
.
container image garbage collection
.
pulling an Image from a Private Registry
.
3.2 - Container Environment
This page describes the resources available to Containers in the Container environment.
Container environment
The Kubernetes Container environment provides several important resources to Containers:
A filesystem, which is a combination of an
image
and one or more
volumes
.
Information about the Container itself.
Information about other objects in the cluster.
Container information
The
hostname
of a Container is the name of the Pod in which the Container is running.
It is available through the
hostname
command or the
gethostname
function call in libc.
The Pod name and namespace are available as environment variables through the
.
User defined environment variables from the Pod definition are also available to the Container,
as are any environment variables specified statically in the container image.
Cluster 

information
A list of all services that were running when a Container was created is available to that Container as environment variables.
This list is limited to services within the same namespace as the new Container's Pod and Kubernetes control plane services.
For a service named
foo
that maps to a Container named
bar
,
the following variables are defined:
FOO_SERVICE_HOST
=
&lt;the host the service is running on&gt;
FOO_SERVICE_PORT
=
&lt;the port the service is running on&gt;
Services have dedicated IP addresses and are available to the Container via DNS,
if
DNS addon
.
Get hands-on experience
attaching handlers to Container lifecycle events
.
3.3 - Runtime Class
This page describes the RuntimeClass resource and runtime selection mechanism.
RuntimeClass is a feature for selecting the container runtime configuration. The container runtime
configuration is used to run a Pod's containers.
You can set a different RuntimeClass between different Pods to provide a balance of
performance versus security. For example, if part of your workload deserves a high
level of information security assurance, you might choose to schedule those Pods so
that they run in a container runtime that uses hardware virtualization. You'd then
benefit from the extra isolation of the alternative runtime, at the expense of some
additional overhead.
You can also use RuntimeClass to run different Pods with the same container runtime
but with different settings.
Setup
Configure the CRI implementation on nodes (runtime dependent)
Create the corresponding RuntimeClass resources
1. Configure the CRI implementation on nodes
The configurations available through RuntimeClass are Container Runtime Interface (CRI)
implementation dependent. See the corresponding documentation (
below
) for your
CRI implementation for how to configure.
Note:
RuntimeClass assumes a homogeneous node configuration across the cluster by default (which means
that all nodes are configured the same way with respect to container runtimes). To support
heterogeneous node configurations, see
below.
The configurations have a corresponding
handler
name, referenced by the RuntimeClass. The
handler must be a valid
DNS label name
.
2. Create the corresponding RuntimeClass resources
The configurations setup in step 1 should each have an associated
handler
name, which identifies
the configuration. For each handler, create a corresponding RuntimeClass object.
The RuntimeClass resource currently only has 2 significant fields: the RuntimeClass name
(
) and the handler (
handler
). The object definition looks like this:
# RuntimeClass is defined in the node.k8s.io API group
:
kind
:
metadata
:
# The name the RuntimeClass will be referenced by.
# RuntimeClass is a non-namespaced resource.
name
:
myclass
# The name of the corresponding CRI configuration
handler
:
myconfiguration
The name of a RuntimeClass object must be a valid
.
Note:
It is recommended that RuntimeClass write operations (create/update/patch/delete) be
restricted to the cluster administrator. This is typically the default. See
Usage
Once RuntimeClasses are configured for the cluster, you can specify a
in the Pod spec to use it. For example:
:
v1
kind
:
Pod
metadata
:
name
:
mypod
spec
:
:
myclass
# ...
This will instruct the kubelet to use the named RuntimeClass to run this pod. If the named
RuntimeClass does not exist, or the CRI cannot run the corresponding handler, the pod will enter the
Failed
terminal
phase
. Look for a
event
for an
error message.
If no
is specified, the default RuntimeHandler will be used, which is equivalent
to the behavior when the RuntimeClass feature is disabled.
CRI Configuration
For more details on 

setting up CRI runtimes, see
CRI installation
.
Runtime handlers are configured through containerd's configuration at
. Valid handlers are configured under the runtimes section:
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.${HANDLER_NAME}]
See containerd's
config documentation
for more details:
CRI-O
Runtime handlers are configured through CRI-O's configuration at
. Valid
handlers are configured under the
crio.runtime table
:
[crio.runtime.runtimes.${HANDLER_NAME}]
  runtime_path = "${PATH_TO_BINARY}"
See CRI-O's
Kubernetes v1.16 [beta]
By specifying the
field for a RuntimeClass, you can set constraints to
ensure that Pods running with this RuntimeClass are scheduled to nodes that support it.
If
is not set, this RuntimeClass is assumed to be supported by all nodes.
To ensure pods land on nodes supporting a specific RuntimeClass, that set of nodes should have a
common label which is then selected by the
runtimeclass.scheduling.nodeSelector
field. The
RuntimeClass's nodeSelector is merged with the pod's nodeSelector in admission, effectively taking
the intersection of the set of nodes selected by each. If there is a conflict, the pod will be
rejected.
If the supported nodes are tainted to prevent other RuntimeClass pods from running on the node, you
can add
to the RuntimeClass. As with the
, the tolerations are merged
with the pod's tolerations in admission, effectively taking the union of the set of nodes tolerated
by each.
To learn more about configuring the node selector and tolerations, see
.
You can specify
overhead
resources that are associated with running a Pod. Declaring overhead allows
the cluster (including the scheduler) to account for it when making decisions about Pods and resources.
Pod overhead is defined in RuntimeClass through the
overhead
field. Through the use of this field,
you can specify the overhead of running pods utilizing this RuntimeClass and ensure these overheads
are accounted for in Kubernetes.
RuntimeClass Design
RuntimeClass Scheduling Design
concept
PodOverhead Feature Design
3.4 - Container Lifecycle Hooks
This page describes how kubelet managed Containers can use the Container lifecycle hook framework
to run code triggered by events during their management lifecycle.
Overview
Analogous to many programming language frameworks that have component lifecycle hooks, such as Angular,
Kubernetes provides Containers with lifecycle hooks.
The hooks enable Containers to be aware of events in their management lifecycle
and run code implemented in a handler when the corresponding lifecycle hook is executed.
There are two hooks that are exposed to Containers:
PostStart
This hook is executed immediately after a container is created.
However, there is no guarantee that the hook will execute before the container ENTRYPOINT.
No parameters are passed to the handler.
PreStop
This hook is called immediately before a container is terminated due to an API request or management
event such as a liveness/startup probe failure, preemption, resource contention and others. A call
to the
PreStop
hook fails if the container is already in a terminated or completed state and the
hook must complete before the TERM signal to stop the container can be sent. The Pod's termination
grace period countdown begins before the
PreStop
hook is executed, so regardless of the outcome of
the handler, the container will eventually terminate within the Pod's termination grace period. No
parameters are passed to the handler.
A more detailed description of the termination behavior can be found in
Termination of Pods
.
Hook handler implementations
Containers can access a hook by implementing and registering a handler for that hook.
T

here are two types of hook handlers that can be implemented for Containers:
Exec - Executes a specific command, such as
pre-stop.sh
, inside the cgroups and namespaces of the Container.
Resources consumed by the command are counted against the Container.
HTTP - Executes an HTTP request against a specific endpoint on the Container.
Hook handler execution
When a Container lifecycle management hook is called,
the Kubernetes management system executes the handler according to the hook action,
httpGet
and
tcpSocket
are executed by the kubelet process, and
exec
is executed in the container.
Hook handler calls are synchronous within the context of the Pod containing the Container.
This means that for a
PostStart
hook,
the Container ENTRYPOINT and hook fire asynchronously.
However, if the hook takes too long to run or hangs,
the Container cannot reach a
running
state.
PreStop
hooks are not executed asynchronously from the signal to stop the Container; the hook must
complete its execution before the TERM signal can be sent. If a
PreStop
hook hangs during
execution, the Pod's phase will be
and remain there until the Pod is killed after its
expires. This grace period applies to the total time it takes for
both the
PreStop
hook to execute and for the Container to stop normally. If, for example,
is 60, and the hook takes 55 seconds to complete, and the Container
takes 10 seconds to stop normally after receiving the signal, then the Container will be killed
before it can stop normally, since
is less than the total time
(55+10) it takes for these two things to happen.
If either a
PostStart
or
PreStop
hook fails,
it kills the Container.
Users should make their hook handlers as lightweight as possible.
There are cases, however, when long running commands make sense,
such as when saving state prior to stopping a Container.
Hook delivery guarantees
Hook delivery is intended to be
at least once
,
which means that a hook may be called multiple times for any given event,
such as for
PostStart
or
PreStop
.
It is up to the hook implementation to handle this correctly.
Generally, only single deliveries are made.
If, for example, an HTTP hook receiver is down and is unable to take traffic,
there is no attempt to resend.
In some rare cases, however, double delivery may occur.
For instance, if a kubelet restarts in the middle of sending a hook,
the hook might be resent after the kubelet comes back up.
Debugging Hook handlers
The logs for a Hook handler are not exposed in Pod events.
If a handler fails for some reason, it broadcasts an event.
For
PostStart
, this is the
FailedPostStartHook
event,
and for
PreStop
FailedPreStopHook
event.
To generate a failed
event yourself, modify the
lifecycle-events.yaml
file to change the postStart command to "badcommand" and apply it.
Here is some example output of the resulting events you see from running
kubectl describe pod lifecycle-demo
:
Events:
  Type     Reason               Age              From               Message
  ----     ------               ----             ----               -------
  Normal   Scheduled            7s               default-scheduler  Successfully assigned default/lifecycle-demo to ip-XXX-XXX-XX-XX.us-east-2...
  Normal   Pulled               6s               kubelet            Successfully pulled image "nginx" in 229.604315ms
  Normal   Pulling              4s (x2 over 6s)  kubelet            Pulling image "nginx"
  Normal   Created              4s (x2 over 5s)  kubelet            Created container lifecycle-demo-container
  Normal   Started              4s (x2 over 5s)  kubelet            Started container lifecycle-demo-container
  Warning  FailedPostStartHook  4s (x2 over 5s)  kubelet            Exec lifecycle hook ([badcommand]) for Container "lifecycle-demo-container" in Pod "lifecycle-demo_default(30229739-9651-4e5a-9a32-a8f1688862db)" failed - e

rror: command 'badcommand' exited with 126: , message: "OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: \"badcommand\": executable file not found in $PATH: unknown\r\n"
  Normal   Killing              4s (x2 over 5s)  kubelet            FailedPostStartHook
  Normal   Pulled               4s               kubelet            Successfully pulled image "nginx" in 215.66395ms
  Warning  BackOff              2s (x2 over 3s)  kubelet            Back-off restarting failed container
.
.
4 - Windows in Kubernetes
4.1 - Windows containers in Kubernetes
Windows applications constitute a large portion of the services and applications that
run in many organizations.
Windows containers
provide a way to encapsulate processes and package dependencies, making it easier
to use DevOps practices and follow cloud native patterns for Windows applications.
Organizations with investments in Windows-based applications and Linux-based
applications don't have to look for separate orchestrators to manage their workloads,
leading to increased operational efficiencies across their deployments, regardless
of operating system.
Windows nodes in Kubernetes
To enable the orchestration of Windows containers in Kubernetes, include Windows nodes
in your existing Linux cluster. Scheduling Windows containers in
Pods
on Kubernetes is similar to
scheduling Linux-based containers.
In order to run Windows containers, your Kubernetes cluster must include
multiple operating systems.
While you can only run the
on Linux,
you can deploy worker nodes running either Windows or Linux.
Windows
nodes
are
supported
provided that the operating system is
Windows Server 2019.
This document uses the term
to mean Windows containers with
process isolation. Kubernetes does not support running Windows containers with
.
Compatibility and limitations
Some node features are only available if you use a specific
; others are not available on Windows nodes,
including:
HugePages: not supported for Windows containers
Privileged containers: not supported for Windows containers.
HostProcess Containers
offer similar functionality.
TerminationGracePeriod: requires containerD
Not all features of shared namespaces are supported. See
API compatibility
See
Windows OS version compatibility
the Windows versions that Kubernetes is tested against.
From an API and kubectl perspective, Windows containers behave in much the same
way as Linux-based containers. However, there are some notable differences in key
functionality which are outlined in this section.
Comparison with Linux
Key Kubernetes elements work the same way in Windows as they do in Linux. This
section refers to several key workload abstractions and how they map to Windows.
Pods
A Pod is the basic building block of Kubernetes–the smallest and simplest unit in
the Kubernetes object model that you create or deploy. You may not deploy Windows and
Linux containers in the same Pod. All containers in a Pod are scheduled onto a single
Node where each Node represents a specific platform and architecture. The following
Pod capabilities, properties and events are supported with Windows containers:
Single or multiple containers per Pod with process isolation and volume sharing
Pod
status
fields
Readiness, liveness, and startup probes
postStart &amp; preStop container lifecycle hooks
ConfigMap, Secrets: as environment variables or volumes
emptyDir
volumes
Named pipe host mounts
OS field:
The
.spec.os.name
field should be set to
windows
to indicate that the current Pod uses Windows containers.
Note:
Starting from 1.25, the
feature gate is in GA stage and defaults to be enabled.
If you set the
field to
windows
,
you must not set the followi

ng fields in the
.spec
of that Pod:
spec.hostPID
spec.hostIPC
spec.securityContext.seLinuxOptions
spec.securityContext.seccompProfile
spec.securityContext.fsGroup
spec.securityContext.fsGroupChangePolicy
spec.securityContext.sysctls
spec.shareProcessNamespace
spec.securityContext.runAsUser
spec.securityContext.runAsGroup
spec.securityContext.supplementalGroups
spec.containers[*].securityContext.seLinuxOptions
spec.containers[*].securityContext.seccompProfile
spec.containers[*].securityContext.capabilities
spec.containers[*].securityContext.readOnlyRootFilesystem
spec.containers[*].securityContext.privileged
spec.containers[*].securityContext.allowPrivilegeEscalation
spec.containers[*].securityContext.procMount
spec.containers[*].securityContext.runAsUser
spec.containers[*].securityContext.runAsGroup
In the above list, wildcards (
*
) indicate all elements in a list.
spec.containers[*].securityContext
refers to the SecurityContext object
for all containers. If any of these fields is specified, the Pod will
not be admitted by the API server.
Workload resources
DaemonSet
Job
CronJob
Services
See
Load balancing and Services
Pods, workload resources, and Services are critical elements to managing Windows
workloads on Kubernetes. However, on their own they are not enough to enable
the proper lifecycle management of Windows workloads in a dynamic cloud native
environment.
Pod and container metrics
Horizontal pod autoscaling
Resource quotas
Scheduler preemption
Command line options for the kubelet
Some kubelet command line options behave differently on Windows, as described below:
The
--windows-priorityclass
lets you set the scheduling priority of the kubelet process
(see
CPU resource management
)
The
,
, and
flags update
NodeAllocatable
Eviction by using
--enforce-node-allocable
is not implemented
and
are not implemented
When running on a Windows node the kubelet does not have memory or CPU
restrictions.
and
only subtract from
and do not guarantee resource provided for workloads.
See
The
Condition is not implemented
The kubelet does not take OOM eviction actions
There are subtle differences in the way the Kubernetes APIs work for Windows due to the OS
and container runtime. Some workload properties were designed for Linux, and fail to run on Windows.
At a high level, these OS concepts are different:
Identity - Linux uses userID (UID) and groupID (GID) which
are represented as integer types. User and group names
are not canonical - they are just an alias in
/etc/groups
or
/etc/passwd
back to UID+GID. Windows uses a larger binary
security identifier
(SID)
which is stored in the Windows Security Access Manager (SAM) database. This
database is not shared between the host and containers, or between containers.
File permissions - Windows uses an access control list based on (SIDs), whereas
POSIX systems such as Linux use a bitmask based on object permissions and UID+GID,
plus
optional
access control lists.
File paths - the convention on Windows is to use
\
instead of
/
. The Go IO
libraries typically accept both and just make it work, but when you're setting a
path or command line that's interpreted inside a container,
\
may be needed.
Signals - Windows interactive apps handle termination differently, and can
implement one or more of these:
A UI thread handles well-defined messages including
WM_CLOSE
.
Console apps handle Ctrl-C or Ctrl-break using a Control Handler.
Services register a Service Control Handler function that can accept
SERVICE_CONTROL_STOP
control codes.
Container exit codes follow the same convention where 0 is success, and nonzero is failure.
The specific error codes may dif

fer across Windows and Linux. However, exit codes
passed from the Kubernetes components (kubelet, kube-proxy) are unchanged.
Field compatibility for container specifications
The following list documents differences between how Pod container specifications
work between Windows and Linux:
Huge pages are not implemented in the Windows container
runtime, and are not available. They require
asserting a user
privilege
that's not configurable for containers.
and
- requests are subtracted
from node available resources, so they can be used to avoid overprovisioning a
node. However, they cannot be used to guarantee resources in an overprovisioned
node. They should be applied to all containers as a best practice if the operator
wants to avoid overprovisioning entirely.
-
not possible on Windows; none of the capabilities are hooked up
-
POSIX capabilities are not implemented on Windows
-
Windows doesn't support privileged containers, use
instead
-
Windows doesn't have a
/proc
filesystem
-
not possible on Windows; write access is required for registry &amp; system
processes to run inside the container
-
not possible on Windows as there is no GID support
-
this setting will prevent containers from running as
which is the closest equivalent to a root user on Windows.
-
use
instead
-
not possible on Windows as SELinux is Linux-specific
-
this has some limitations in that Windows doesn't support mapping single files. The
default value is
, which does work because it does not
exist on Windows by default.
Field compatibility for Pod specifications
The following list documents differences between how Pod specifications work between Windows and Linux:
hostIPC
and
hostpid
- host namespace sharing is not possible on Windows
-
see below
dnsPolicy
- setting the Pod
dnsPolicy
to
ClusterFirstWithHostNet
is
not supported on Windows because host networking is not provided. Pods always
run with a container network.
podSecurityContext
see below
- this is a beta feature, and depends on Linux namespaces
which are not implemented on Windows. Windows cannot share process namespaces or
the container's root filesystem. Only the network can be shared.
- this is not fully implemented in Docker on Windows,
see the
.
The behavior today is that the ENTRYPOINT process is sent CTRL_SHUTDOWN_EVENT,
then Windows waits 5 seconds by default, and finally shuts down
all processes using the normal Windows shutdown behavior. The 5
second default is actually in the Windows registry
inside the container
,
so it can be overridden when the container is built.
- this is a beta feature, and is not implemented on Windows.
Windows cannot attach raw block devices to pods.
volumes
If you define an
emptyDir
volume, you cannot set its volume source to
memory
.
You cannot enable
for volume mounts as this is not
supported on Windows.
Field compatibility for hostNetwork
The kubelet can now request that pods running on Windows nodes use the host's network namespace instead
of creating a new pod network namespace. To enable this functionality pass
--feature-gates=WindowsHostNetwork=true
to the kubelet.
Note:
This functionality requires a container runtime that supports this functionality.
Field compatibility for Pod security context
None of the Pod
fields work on Windows.
Node problem detector
The node problem detector (see
)
has preliminary support for Windows.
For more information, visit the project's
GitHub pag

e
.
In a Kubernetes Pod, an infrastructure or “pause” container is first created
to host the container. In Linux, the cgroups and namespaces that make up a pod
need a process to maintain their continued existence; the pause process provides
this. Containers that belong to the same pod, including infrastructure and worker
containers, share a common network endpoint (same IPv4 and / or IPv6 address, same
network port spaces). Kubernetes uses pause containers to allow for worker containers
crashing or restarting without losing any of the networking configuration.
Kubernetes maintains a multi-architecture image that includes support for Windows.
For Kubernetes v1.26 the recommended pause image is
.
The
source code
is available on GitHub.
Microsoft maintains a different multi-architecture image, with Linux and Windows
amd64 support, that you can find as
.
This image is built from the same source as the Kubernetes maintained image but
all of the Windows binaries are
authenticode signed
by Microsoft.
The Kubernetes project recommends using the Microsoft maintained image if you are
deploying to a production or production-like environment that requires signed
binaries.
You need to install a
into each node in the cluster so that Pods can run there.
The following container runtimes work with Windows:
Note:
ContainerD
You can use
1.4.0+
as the container runtime for Kubernetes nodes that run Windows.
install ContainerD on a Windows node
.
Note:
There is a
known limitation
when using GMSA with containerd to access Windows network shares, which requires a
kernel patch.
Mirantis Container Runtime
(MCR)
is available as a container runtime for all Windows Server 2019 and later versions.
See
Install MCR on Windows Servers
On Windows nodes, strict compatibility rules apply where the host OS version must
match the container base image OS version. Only Windows containers with a container
operating system of Windows Server 2019 are fully supported.
For Kubernetes v1.26, operating system compatibility for Windows nodes (and Pods)
is as follows:
Windows Server LTSC release
Windows Server 2019
Windows Server 2022
Windows Server SAC release
Windows Server version 20H2
version-skew policy
also applies.
Getting help and troubleshooting
Your main source of help for troubleshooting your Kubernetes cluster should start
with the
page.
Some additional, Windows-specific troubleshooting help is included
in this section. Logs are an important element of troubleshooting
issues in Kubernetes. Make sure to include them any time you seek
troubleshooting assistance from other contributors. Follow the
instructions in the
SIG Windows
contributing guide on gathering logs
.
Reporting issues and feature requests
If you have what looks like a bug, or you would like to
make a feature request, please follow the
SIG Windows contributing guide
to create a new issue.
You should first search the list of issues in case it was
reported previously and comment with your experience on the issue and add additional
logs. SIG Windows channel on the Kubernetes Slack is also a great avenue to get some initial support and
troubleshooting ideas prior to creating a ticket.
Deployment tools
The kubeadm tool helps you to deploy a Kubernetes cluster, providing the control
plane to manage the cluster it, and nodes to run your workloads.
cluster API
project also provides means to automate deployme

nt of Windows nodes.
Windows distribution channels
For a detailed explanation of Windows distribution channels see the
Microsoft documentation
.
Information on the different Windows Server servicing channels
including their support models can be found at
Windows Server servicing channels
.
4.2 - Guide for scheduling Windows containers in Kubernetes
Windows applications constitute a large portion of the services and applications that run in many organizations.
This guide walks you through the steps to configure and deploy Windows containers in Kubernetes.
Configure an example deployment to run Windows containers on the Windows node
Highlight Windows specific functionality in Kubernetes
Create a Kubernetes cluster that includes a control plane and a worker node running Windows Server
It is important to note that creating and deploying services and workloads on Kubernetes
behaves in much the same way for Linux and Windows containers.
Kubectl commands
to interface with the cluster are identical.
The example in the section below is provided to jumpstart your experience with Windows containers.
Getting Started: Deploying a Windows container
The example YAML file below deploys a simple webserver application running inside a Windows container.
Create a service spec named
win-webserver.yaml
with the contents below:
:
v1
kind
:
Service
metadata
:
name
:
win-webserver
labels
:
app
:
spec
:
ports
:
-
port
:
80
:
80
selector
:
app
:
type
:
NodePort
---
:
apps/v1
kind
:
metadata
:
labels
:
app
:
name
:
spec
:
replicas
:
2
selector
:
:
app
:
template
:
metadata
:
labels
:
app
:
name
:
spec
:
:
-
name
:
windowswebserver
image
:
command
:
- powershell.exe
- -command
-
"&lt;#code used from https://gist.github.com/19WAS85/5424431#&gt; ; $$listener = New-Object System.Net.HttpListener ; $$listener.Prefixes.Add('http://*:80/') ; $$listener.Start() ; $$callerCounts = @{} ; Write-Host('Listening at http://*:80/') ; while ($$listener.IsListening) { ;$$context = $$listener.GetContext() ;$$requestUrl = $$context.Request.Url ;$$clientIP = $$context.Request.RemoteEndPoint.Address ;$$response = $$context.Response ;Write-Host '' ;Write-Host('&gt; {0}' -f $$requestUrl) ;  ;$$count = 1 ;$$k=$$callerCounts.Get_Item($$clientIP) ;if ($$k -ne $$null) { $$count += $$k } ;$$callerCounts.Set_Item($$clientIP, $$count) ;$$ip=(Get-NetAdapter | Get-NetIpAddress); $$header='&lt;html&gt;&lt;body&gt;&lt;H1&gt;Windows Container Web Server&lt;/H1&gt;' ;$$callerCountsString='' ;$$callerCounts.Keys | % { $$callerCountsString+='&lt;p&gt;IP {0} callerCount {1} ' -f $$ip[1].IPAddress,$$callerCounts.Item($$_) } ;$$footer='&lt;/body&gt;&lt;/html&gt;' ;$$content='{0}{1}{2}' -f $$header,$$callerCountsString,$$footer ;Write-Output $$content ;$$buffer = [System.Text.Encoding]::UTF8.GetBytes($$content) ;$$response.ContentLength64 = $$buffer.Length ;$$response.OutputStream.Write($$buffer, 0, $$buffer.Length) ;$$response.Close() ;$$responseStatus = $$response.StatusCode ;Write-Host('&lt; {0}' -f $$responseStatus)  } ; "
:
:
windows
Note:
Port mapping is also supported, but for simplicity this example exposes
port 80 of the container directly to the Service.
Check that all nodes are healthy:
Deploy the service and watch for pod updates:
kubectl apply -f win-webserver.yaml
kubectl get pods -o wide -w
When the service is deployed correctly both Pods are marked as Ready. To exit the watch command, press Ctrl+C.
Check that the deployment succeeded. To verify:
Two pods listed from the Linux control plane node, use
Node-to-pod communication across the network,
curl
port 80 of your pod IPs from the Linux control plane node
to check for a web server resp

onse
Pod-to-pod communication, ping between pods (and across hosts, if you have more than one Windows node)
using
or
Service-to-pod communication,
curl
the virtual service IP (seen under
)
from the Linux control plane node and from individual pods
Service discovery,
curl
the service name with the Kubernetes
default DNS suffix
Inbound connectivity,
curl
the NodePort from the Linux control plane node or machines outside of the cluster
Outbound connectivity,
curl
external IPs from inside the pod using
Note:
Windows container hosts are not able to access the IP of services scheduled on them due to current platform limitations of the Windows networking stack.
Only Windows pods are able to access service IPs.
Capturing logs from workloads
Logs are an important element of observability; they enable users to gain insights
into the operational aspect of workloads and are a key ingredient to troubleshooting issues.
Because Windows containers and workloads inside Windows containers behave differently from Linux containers,
users had a hard time collecting logs, limiting operational visibility.
Windows workloads for example are usually configured to log to ETW (Event Tracing for Windows)
or push entries to the application event log.
LogMonitor
, an open source tool by Microsoft,
is the recommended way to monitor configured log sources inside a Windows container.
LogMonitor supports monitoring event logs, ETW providers, and custom application logs,
piping them to STDOUT for consumption by
kubectl logs &lt;pod&gt;
.
Follow the instructions in the LogMonitor GitHub page to copy its binaries and configuration files
to all your containers and add the necessary entrypoints for LogMonitor to push your logs to STDOUT.
Configuring container user
Using configurable Container usernames
Windows containers can be configured to run their entrypoints and processes
with different usernames than the image defaults.
Learn more about it
here
.
Managing Workload Identity with Group Managed Service Accounts
Windows container workloads can be configured to use Group Managed Service Accounts (GMSA).
Group Managed Service Accounts are a specific type of Active Directory account that provide automatic password management,
simplified service principal name (SPN) management, and the ability to delegate the management to other administrators across multiple servers.
Containers configured with a GMSA can access external Active Directory Domain resources while carrying the identity configured with the GMSA.
Learn more about configuring and using GMSA for Windows containers
here
.
Users need to use some combination of taints and node selectors in order to
schedule Linux and Windows workloads to their respective OS-specific nodes.
The recommended approach is outlined below,
with one of its main goals being that this approach should not break compatibility for existing Linux workloads.
Starting from 1.25, you can (and should) set
for each Pod, to indicate the operating system
that the containers in that Pod are designed for. For Pods that run Linux containers, set
to
linux
. For Pods that run Windows containers, set
to
windows
.
Note:
feature is in GA stage and defaults to be enabled.
The scheduler does not use the value of
when assigning Pods to nodes. You should
use normal Kubernetes mechanisms for
assigning pods to nodes
to ensure that the control plane for your cluster places pods onto nodes that are running the
appropriate operating system.
The
value has no effect on the scheduling of the Windows pods,
so taints and tolerations and node selectors are still required
to ensure that the Windows pods land onto appropriate Windows nodes.
Ensuring OS-specific workloads land on the appropriate container host
Users can ensure Windows containers can be scheduled on the appro

priate host using Taints and Tolerations.
All Kubernetes nodes today have the following default labels:
kubernetes.io/os = [windows|linux]
kubernetes.io/arch = [amd64|arm64|...]
If a Pod specification does not specify a nodeSelector like
"kubernetes.io/os": windows
,
it is possible the Pod can be scheduled on any host, Windows or Linux.
This can be problematic since a Windows container can only run on Windows and a Linux container can only run on Linux.
The best practice is to use a nodeSelector.
However, we understand that in many cases users have a pre-existing large number of deployments for Linux containers,
as well as an ecosystem of off-the-shelf configurations, such as community Helm charts, and programmatic Pod generation cases, such as with Operators.
In those situations, you may be hesitant to make the configuration change to add nodeSelectors.
The alternative is to use Taints. Because the kubelet can set Taints during registration,
it could easily be modified to automatically add a taint when running on Windows only.
--register-with-taints='os=windows:NoSchedule'
By adding a taint to all Windows nodes, nothing will be scheduled on them (that includes existing Linux Pods).
In order for a Windows Pod to be scheduled on a Windows node,
it would need both the nodeSelector and the appropriate matching toleration to choose Windows.
:
:
windows
:
'10.0.17763'
:
-
key
:
"os"
operator
:
"Equal"
value
:
"windows"
effect
:
Handling multiple Windows versions in the same cluster
The Windows Server version used by each pod must match that of the node. If you want to use multiple Windows
Server versions in the same cluster, then you should set additional node labels and nodeSelectors.
Kubernetes 1.17 automatically adds a new label
to simplify this.
If you're running an older version, then it's recommended to add this label manually to Windows nodes.
This label reflects the Windows major, minor, and build number that need to match for compatibility.
Here are values used today for each Windows Server version.
Product Name
Build Number(s)
10.0.17763
Windows Server, Version 20H2
10.0.19042
10.0.20348
Simplifying with RuntimeClass
can be used to simplify the process of using taints and tolerations.
A cluster administrator can create a
object which is used to encapsulate these taints and tolerations.
Save this file to
runtimeClasses.yml
. It includes the appropriate
for the Windows OS, architecture, and version.
:
kind
:
metadata
:
name
:
windows-2019
handler
:
'docker'
:
:
:
'windows'
:
'amd64'
:
:
-
effect
:
key
:
os
operator
:
Equal
value
:
"windows"
Run
kubectl create -f runtimeClasses.yml
using as a cluster administrator
Add
runtimeClassName: windows-2019
as appropriate to Pod specs
:
apps/v1
kind
:
metadata
:
name
:
iis-2019
labels
:
app
:
iis-2019
spec
:
replicas
:
1
template
:
metadata
:
name
:
iis-2019
labels
:
app
:
iis-2019
spec
:
:
:
-
name
:
iis
image
:
resources
:
limits
:
cpu
:
1
memory
:
800Mi
requests
:
cpu
:
.1
memory
:
300Mi
ports
:
-
:
80
selector
:
:
app
:
iis-2019
---
:
v1
kind
:
Service
metadata
:
name
:
iis
spec
:
type
:
ports
:
-
protocol
:
TCP
port
:
80
selector
:
app
:
iis-2019
5 - Workloads
Whether your workload is a single component or several that work together, 

on Kubernetes you run
it inside a set of
pods
.
Pod
represents a set of running
Kubernetes pods have a
defined lifecycle
.
For example, once a pod is running in your cluster then a critical fault on the
node
where that pod is running means that
all the pods on that node fail. Kubernetes treats that level of failure as final: you
would need to create a new
Pod
to recover, even if the node later becomes healthy.
However, to make life considerably easier, you don't need to manage each
Pod
directly.
Instead, you can use
that manage a set of pods on your behalf.
These resources configure
that make sure the right number of the right kind of pod are running, to match the state
Kubernetes provides several built-in workload resources:
and
(replacing the legacy resource
).
is a good fit for managing a stateless application workload on your cluster,
where any
Pod
in the
is interchangeable and can be replaced if needed.
lets you
run one or more related Pods that do track state somehow. For example, if your workload
records data persistently, you can run a
that matches each
Pod
with a
. Your code, running in the
Pods
for that
, can replicate data to other
Pods
in the same
to improve overall resilience.
DaemonSet
defines
Pods
that provide
node-local facilities. These might be fundamental to the operation of your cluster, such
as a networking helper tool, or be part of an
add-on
.
Every time you add a node to your cluster that matches the specification in a
DaemonSet
,
the control plane schedules a
Pod
for that
DaemonSet
onto the new node.
Job
and
CronJob
define tasks that run to completion and then stop. Jobs represent one-off tasks, whereas
CronJobs
recur according to a schedule.
In the wider Kubernetes ecosystem, you can find third-party workload resources that provide
additional behaviors. Using a
custom resource definition
,
you can add in a third-party workload resource if you want a specific behavior that's not part
of Kubernetes' core. For example, if you wanted to run a group of
Pods
for your application but
stop work unless
all
the Pods are available (perhaps for some high-throughput distributed task),
then you can implement or install an extension that does provide that feature.
As well as reading about each resource, you can learn about specific tasks that relate to them:
Run a stateless application using a
Run a stateful application either as a
single instance
or as a
replicated set
Run automated tasks with a
CronJob
To learn about Kubernetes' mechanisms for separating code from configuration,
visit
.
There are two supporting concepts that provide backgrounds about how Kubernetes manages pods
for applications:
Garbage collection
tidies up objects
from your cluster after their
owning resource
has been removed.
The
time-to-live after finished
removes Jobs once a defined time has passed since they completed.
Once your application is running, you might want to make it available on the internet as
a
Service
or, for web application only,
using an
Ingress
.
5.1 - Pods
Pods
are the smallest deployable units of computing that you can create and manage in Kubernetes.
A
Pod
(as in a pod of whales or pea pod) is a group of one or more
, with shared storage and network resources, and a specification for how to run the containers. A Pod's contents are always co-located and
co-scheduled, and run in a shared context. A Pod models an
application-specific "logical host": it contains one or more application
containers which are relatively tightly coupled.
In non-cloud contexts, applications executed on the same physical or virtual machine are analogous to cloud applications executed on the same logical host.
As well as application containers, a Pod can contain
init contai

ners
that run
during Pod startup. You can also inject
for debugging if your cluster offers this.
What is a Pod?
Note:
While Kubernetes supports more
than just Docker,
Docker
is the most commonly known
runtime, and it helps to describe Pods using some terminology from Docker.
The shared context of a Pod is a set of Linux namespaces, cgroups, and
potentially other facets of isolation - the same things that isolate a
container
. Within a Pod's context, the individual applications may have
further sub-isolations applied.
A Pod is similar to a set of containers with shared namespaces and shared filesystem volumes.
Using Pods
The following is an example of a Pod which consists of a container running the image
.
pods/simple-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
To create the Pod shown above, run the following command:
kubectl apply -f https://k8s.io/examples/pods/simple-pod.yaml
Pods are generally not created directly and are created using workload resources.
See
Working with Pods
for more information on how Pods are used
with workload resources.
Workload resources for managing pods
Usually you don't need to create Pods directly, even singleton Pods. Instead, create them using workload resources such as
or
Job
.
If your Pods need to track state, consider the
resource.
Pods in a Kubernetes cluster are used in two main ways:
Pods that run a single container
. The "one-container-per-Pod" model is the
most common Kubernetes use case; in this case, you can think of a Pod as a
wrapper around a single container; Kubernetes manages Pods rather than managing
the containers directly.
Pods that run multiple containers that need to work together
. A Pod can
encapsulate an application composed of multiple co-located containers that are
tightly coupled and need to share resources. These co-located containers
form a single cohesive unit of service—for example, one container serving data
stored in a shared volume to the public, while a separate
sidecar
container
refreshes or updates those files.
The Pod wraps these containers, storage resources, and an ephemeral network
identity together as a single unit.
Note:
Grouping multiple co-located and co-managed containers in a single Pod is a
relatively advanced use case. You should use this pattern only in specific
instances in which your containers are tightly coupled.
Each Pod is meant to run a single instance of a given application. If you want to
scale your application horizontally (to provide more overall resources by running
more instances), you should use multiple Pods, one for each instance. In
Kubernetes, this is typically referred to as
replication
.
Replicated Pods are usually created and managed as a group by a workload resource
and its
.
See
Pods and controllers
for more information on how
Kubernetes uses workload resources, and their controllers, to implement application
scaling and auto-healing.
How Pods manage multiple containers
Pods are designed to support multiple cooperating processes (as containers) that form
a cohesive unit of service. The containers in a Pod are automatically co-located and
co-scheduled on the same physical or virtual machine in the cluster. The containers
can share resources and dependencies, communicate with one another, and coordinate
when and how they are terminated.
For example, you might have a container that
acts as a web server for files in a shared volume, and a separate "sidecar" container
that updates those files from a remote source, as in the following diagram:
Some Pods have
as well as
app containers
. Init containers run and complete before the app containers are started.
Pods natively provide two kinds of shared resources for their constituent containers:
and
storage
.
You'll rarely create individual Pods directly i

n Kubernetes—even singleton Pods. This
is because Pods are designed as relatively ephemeral, disposable entities. When
a Pod gets created (directly by you, or indirectly by a
), the new Pod is
scheduled to run on a
Node
The Pod remains on that node until the Pod finishes execution, the Pod object is deleted,
the Pod is
evicted
for lack of resources, or the node fails.
Note:
Restarting a container in a Pod should not be confused with restarting a Pod. A Pod
is not a process, but an environment for running container(s). A Pod persists until
The name of a Pod must be a valid
DNS subdomain
value, but this can produce unexpected results for the Pod hostname. For best compatibility,
the name should follow the more restrictive rules for a
DNS label
.
Pod OS
You should set the
field to either
windows
or
linux
to indicate the OS on
which you want the pod to run. These two are the only operating systems supported for now by
Kubernetes. In future, this list may be expanded.
In Kubernetes v1.26, the value you set for this field has no
effect on
of the pods.
helps to identify the pod OS
authoratitively and is used for validation. The kubelet refuses to run a Pod where you have
specified a Pod OS, if this isn't the same as the operating system for the node where
that kubelet is running.
The
Pod security standards
also use this
field to avoid enforcing policies that aren't relevant to that operating system.
You can use workload resources to create and manage multiple Pods for you. A controller
for the resource handles replication and rollout and automatic healing in case of
Pod failure. For example, if a Node fails, a controller notices that Pods on that
Node have stopped working and creates a replacement Pod. The scheduler places the
replacement Pod onto a healthy Node.
Here are some examples of workload resources that manage one or more Pods:
DaemonSet
Pod templates
Controllers for
workload
resources create Pods
from a
and manage those Pods on your behalf.
PodTemplates are specifications for creating Pods, and are included in workload resources such as
,
Jobs
, and
.
Each controller for a workload resource uses the
inside the workload
object to make actual Pods. The
is part of the desired state of whatever
workload resource you used to run your app.
The sample below is a manifest for a simple Job with a
template
that starts one
container. The container in that Pod prints a message then pauses.
:
batch/v1
kind
:
Job
metadata
:
name
:
hello
spec
:
template
:
# This is the pod template
spec
:
:
-
name
:
hello
image
:
command
:
[
'sh'
,
'-c'
,
'echo "Hello, Kubernetes!" &amp;&amp; sleep 3600'
]
:
OnFailure
# The pod template ends here
Modifying the pod template or switching to a new pod template has no direct effect
on the Pods that already exist. If you change the pod template for a workload
resource, that resource needs to create replacement Pods that use the updated template.
For example, the StatefulSet controller ensures that the running Pods match the current
pod template for each StatefulSet object. If you edit the StatefulSet to change its pod
template, the StatefulSet starts to create new Pods based on the updated template.
Eventually, all of the old Pods are replaced with new Pods, and the update is complete.
Each workload resource implements its own rules for handling changes to the Pod template.
If you want to read more about StatefulSet specifically, read
Update strategy
in the StatefulSet Basics tutorial.
On Nodes, the
kubelet
does not
directly observe or manage any of the details around pod templates and updates; those
details are abstracted away. That abstraction and separation of concerns simplifies
system semantics, and makes it feasible to ex

tend the cluster's behavior without
changing existing code.
Pod update and replacement
As mentioned in the previous section, when the Pod template for a workload
resource is changed, the controller creates new Pods based on the updated
template instead of updating or patching the existing Pods.
Kubernetes doesn't prevent you from managing Pods directly. It is possible to
update some fields of a running Pod, in place. However, Pod update operations
like
patch
, and
replace
have some limitations:
Most of the metadata about a Pod is immutable. For example, you cannot
change the
namespace
,
name
,
uid
, or
fields;
the
field is unique. It only accepts updates that increment the
field's current value.
If the
is set, no new entry can be added to the
list.
Pod updates may not change fields other than
spec.containers[*].image
,
spec.initContainers[*].image
,
spec.activeDeadlineSeconds
or
spec.tolerations
. For
, you can only add new entries.
When updating the
field, two types of updates
are allowed:
setting the unassigned field to a positive number;
updating the field from a positive number to a smaller, non-negative
number.
Resource sharing and communication
Pods enable data sharing and communication among their constituent
Storage in Pods
A Pod can specify a set of shared storage
volumes
. All containers
in the Pod can access the shared volumes, allowing those containers to
share data. Volumes also allow persistent data in a Pod to survive
in case one of the containers within needs to be restarted. See
Storage
Kubernetes implements shared storage and makes it available to Pods.
Pod networking
Each Pod is assigned a unique IP address for each address family. Every
container in a Pod shares the network namespace, including the IP address and
network ports. Inside a Pod (and
only
then), the containers that belong to the Pod
can communicate with one another using
localhost
. When containers in a Pod communicate
with entities
outside the Pod
,
they must coordinate how they use the shared network resources (such as ports).
Within a Pod, containers share an IP address and port space, and
can find each other via
localhost
. The containers in a Pod can also communicate
with each other using standard inter-process communications like SystemV semaphores
or POSIX shared memory. Containers in different Pods have distinct IP addresses
and can not communicate by OS-level IPC without special configuration.
Containers that want to interact with a container running in a different Pod can
use IP networking to communicate.
Containers within the Pod see the system hostname as being the same as the configured
name
for the Pod. There's more about this in the
section.
Privileged mode for containers
In Linux, any container in a Pod can enable privileged mode using the
(Linux) flag on the
of the container spec. This is useful for containers that want to use operating system administrative capabilities such as manipulating the network stack or accessing hardware devices.
If your cluster has the
feature enabled, you can create a
Windows HostProcess pod
windowsOptions.hostProcess
flag on the security context of the pod spec. All containers in these pods must run as Windows HostProcess containers. HostProcess pods run directly on the host and can also be used to perform administrative tasks as is done with Linux privileged containers.
Note:
Your
must support the concept of a privileged container for this setting to be relevant.
API server
Whereas most Pods are managed by the control plane (for example, a
), for static
Pods, the kubelet directly supervises each static Pod (and restart

s it if it fails).
Kubelet
The main use for static Pods is to run a self-hosted control plane: in other words,
using the kubelet to supervise the individual
.
mirror Pod
Note:
The
spec
(e.g.,
,
ConfigMap
,
Secret
, etc).
Container probes
A
probe
is a diagnostic performed periodically by the kubelet on a container. To perform a diagnostic, the kubelet can invoke different actions:
(performed with the help of the container runtime)
(checked directly by the kubelet)
probes
in the Pod Lifecycle documentation.
.
and how you can use it to
configure different Pods with different container runtime configurations.
Read about
and how you can use it to manage application availability during disruptions.
Pod is a top-level resource in the Kubernetes REST API.
The
Pod
object definition describes the object in detail.
The Distributed System Toolkit: Patterns for Composite Containers
explains common layouts for Pods with more than one container.
Read about
To understand the context for why Kubernetes wraps a common Pod API in other resources (such as
or
), you can read about the prior art, including:
Aurora
Borg
Marathon
Omega
Tupperware
.
5.1.1 - Pod Lifecycle
This page describes the lifecycle of a Pod. Pods follow a defined lifecycle, starting
in the
Pending
phase
, moving through
Running
if at least one
of its primary containers starts OK, and then through either the
Succeeded
or
Failed
phases depending on whether any container in the Pod terminated in failure.
Whilst a Pod is running, the kubelet is able to restart containers to handle some
kind of faults. Within a Pod, Kubernetes tracks different container
states
and determines what action to take to make the Pod
healthy again.
In the Kubernetes API, Pods have both a specification and an actual status. The
status for a Pod object consists of a set of
Pod conditions
.
You can also inject
custom readiness information
into the
condition data for a Pod, if that is useful to your application.
Pods are only
scheduled
once in their lifetime.
Once a Pod is scheduled (assigned) to a Node, the Pod runs on that Node until it stops
or is
.
Pod lifetime
Like individual application containers, Pods are considered to be relatively
ephemeral (rather than durable) entities. Pods are created, assigned a unique
ID (
UID
), and scheduled
to nodes where they remain until termination (according to restart policy) or
deletion.
If a
Node
dies, the Pods scheduled to that node
are
scheduled for deletion
after a timeout period.
Pods do not, by themselves, self-heal. If a Pod is scheduled to a
node
that then fails, the Pod is deleted; likewise, a Pod won't
survive an eviction due to a lack of resources or Node maintenance. Kubernetes uses a
higher-level abstraction, called a
, that handles the work of
managing the relatively disposable Pod instances.
A given Pod (as defined by a UID) is never "rescheduled" to a different node; instead,
that Pod can be replaced by a new, near-identical Pod, with even the same name if
desired, but with a different UID.
When something is said to have the same lifetime as a Pod, such as a
volume
,
that means that the thing exists as long as that specific Pod (with that exact UID)
exists. If that Pod is deleted for any reason, and even if an identical replacement
is created, the related thing (a volume, in this example) is also destroyed and
created anew.
Pod diagram
A multi-c

ontainer Pod that contains a file puller and a
web server that uses a persistent volume for shared storage between the containers.
Pod phase
A Pod's
status
field is a
PodStatus
object, which has a
phase
field.
The phase of a Pod is a simple, high-level summary of where the Pod is in its
lifecycle. The phase is not intended to be a comprehensive rollup of observations
of container or Pod state, nor is it intended to be a comprehensive state machine.
The number and meanings of Pod phase values are tightly guarded.
Other than what is documented here, nothing should be assumed about Pods that
have a given
phase
value.
Here are the possible values for
phase
:
Value
Pending
The Pod has been accepted by the Kubernetes cluster, but one or more of the containers has not been set up and made ready to run. This includes time a Pod spends waiting to be scheduled as well as the time spent downloading container images over the network.
Running
The Pod has been bound to a node, and all of the containers have been created. At least one container is still running, or is in the process of starting or restarting.
Succeeded
All containers in the Pod have terminated in success, and will not be restarted.
Failed
All containers in the Pod have terminated, and at least one container has terminated in failure. That is, the container either exited with non-zero status or was terminated by the system.
Unknown
For some reason the state of the Pod could not be obtained. This phase typically occurs due to an error in communicating with the node where the Pod should be running.
Note:
When a Pod is being deleted, it is shown as
by some kubectl commands.
This
status is not one of the Pod phases.
A Pod is granted a term to terminate gracefully, which defaults to 30 seconds.
You can use the flag
--force
to
terminate a Pod by force
.
If a node dies or is disconnected from the rest of the cluster, Kubernetes
applies a policy for setting the
phase
of all Pods on the lost node to Failed.
Container states
As well as the
phase
of the Pod overall, Kubernetes tracks the state of
each container inside a Pod. You can use
container lifecycle hooks
to
trigger events to run at certain points in a container's lifecycle.
Once the
scheduler
assigns a Pod to a Node, the kubelet starts creating containers for that Pod
using a
.
There are three possible container states:
Waiting
,
Running
, and
.
To check the state of a Pod's containers, you can use
kubectl describe pod &lt;name-of-pod&gt;
. The output shows the state for each container
within that Pod.
Each state has a specific meaning:
Waiting
If a container is not in either the
Running
or
state, it is
Waiting
.
A container in the
Waiting
state is still running the operations it requires in
order to complete start up: for example, pulling the container image from a container
image registry, or applying
Secret
data.
When you use
kubectl
to query a Pod with a container that is
Waiting
, you also see
a Reason field to summarize why the container is in that state.
Running
The
Running
status indicates that a container is executing without issues. If there
was a
postStart
hook configured, it has already executed and finished. When you use
kubectl
Running
, you also see information
about when the container entered the
Running
state.
state began execution and then either ran to
completion or failed for some reason. When you use
kubectl
to query a Pod with
a container that is
, you see a reason, an exit code, and the start and
finish time for that container's period of execution.
If a container has a
preStop
hook configured, this hook runs before the container enters
the
state.
Container restart policy
The
spec
of a Pod has a
field with possible values Always, OnFailure,
and Never. The default value is Always.
The
applies to all cont

ainers in the Pod.
only
refers to restarts of the containers by the kubelet on the same node. After containers
in a Pod exit, the kubelet restarts them with an exponential back-off delay (10s, 20s,
40s, …), that is capped at five minutes. Once a container has executed for 10 minutes
without any problems, the kubelet resets the restart backoff timer for that container.
A Pod has a PodStatus, which has an array of
PodConditions
through which the Pod has or has not passed. Kubelet manages the following
PodConditions:
: the Pod has been scheduled to a node.
: (alpha feature; must be
enabled explicitly
) the
Pod sandbox has been successfully created and networking configured.
: all containers in the Pod are ready.
: all
have completed successfully.
Ready
: the Pod is able to serve requests and should be added to the load
balancing pools of all matching Services.
Field name
type
Name of this Pod condition.
status
Indicates whether that condition is applicable, with possible values "
True
", "
False
", or "
Unknown
".
Timestamp of when the Pod condition was last probed.
Timestamp for when the Pod last transitioned from one status to another.
reason
Machine-readable, UpperCamelCase text indicating the reason for the condition's last transition.
message
Human-readable message indicating details about the last status transition.
Pod readiness
Kubernetes v1.14 [stable]
Your application can inject extra feedback or signals into PodStatus:
. To use this, set
in the Pod's
spec
to
specify a list of additional conditions that the kubelet evaluates for Pod readiness.
Readiness gates are determined by the current state of
status.condition
fields for the Pod. If Kubernetes cannot find such a condition in the
field of a Pod, the status of the condition
is defaulted to "
False
".
kind
:
Pod
...
spec
:
:
-
:
"www.example.com/feature-1"
status
:
:
-
type
:
Ready
# a built in PodCondition
status
:
"False"
:
null
:
2018-01-01T00:00:00Z
-
type
:
# an extra PodCondition
status
:
"False"
:
null
:
:
-
:
docker://abcd...
ready
:
true
...
The Pod conditions you add must have names that meet the Kubernetes
label key format
.
Status for Pod readiness
The
command does not support patching object status.
To set these
for the pod, applications and
operators
should use
the
PATCH
action.
Kubernetes client library
to
write code that sets custom Pod conditions for Pod readiness.
For a Pod that uses custom conditions, that Pod is evaluated to be ready
only
when both the following statements apply:
All containers in the Pod are ready.
All conditions specified in
are
True
.
When a Pod's containers are Ready but at least one custom condition is missing or
False
, the kubelet sets the Pod's
condition
to
.
Pod network readiness
After a Pod gets scheduled on a node, it needs to be admitted by the Kubelet and
have any volumes mounted. Once these phases are complete, the Kubelet works with
a container runtime (using
) to set up a
runtime sandbox and configure networking for the Pod. If the
is enabled,
Kubelet reports whether a pod has reached this initialization milestone through
the
field of a Pod.
The
condition is set to
False
by the Kubelet when it detects a
Pod does not have a runtime sandbox with networking configured. This occurs in
the following scenarios:
Early in the lifecycle of the Pod, when the kubelet has not yet

 begun to set up a sandbox for the Pod using the container runtime.
Later in the lifecycle of the Pod, when the Pod sandbox has been destroyed due
to either:
the node rebooting, without the Pod getting evicted
for container runtimes that use virtual machines for isolation, the Pod
sandbox virtual machine rebooting, which then requires creating a new sandbox and fresh container network configuration.
The
True
by the kubelet after the
successful completion of sandbox creation and network configuration for the Pod
by the runtime plugin. The kubelet can start pulling container images and create
containers after
condition has been set to
True
.
For a Pod with init containers, the kubelet sets the
True
after the init containers have successfully completed (which happens
after successful sandbox creation and network configuration by the runtime
plugin). For a Pod without init containers, the kubelet sets the
True
before sandbox creation and network configuration starts.
A
probe
is a diagnostic
performed periodically by the
kubelet
on a container. To perform a diagnostic,
the kubelet either executes code within the container, or makes
a network request.
Check mechanisms
There are four different ways to check a container using a probe.
Each probe must define exactly one of these four mechanisms:
exec
Executes a specified command inside the container. The diagnostic
is considered successful if the command exits with a status code of 0.
grpc
Performs a remote procedure call using
gRPC
.
The target should implement
gRPC health checks
.
The diagnostic is considered successful if the
status
of the response is
SERVING
.
gRPC probes are an alpha feature and are only available if you
enable the
.
httpGet
Performs an HTTP
GET
request against the Pod's IP
address on a specified port and path. The diagnostic is
considered successful if the response has a status code
greater than or equal to 200 and less than 400.
tcpSocket
Performs a TCP check against the Pod's IP address on
a specified port. The diagnostic is considered successful if
the port is open. If the remote system (the container) closes
the connection immediately after it opens, this counts as healthy.
Probe outcome
Each probe has one of three results:
Success
The container passed the diagnostic.
Failure
The container failed the diagnostic.
Unknown
The diagnostic failed (no action should be taken, and the kubelet
will make further checks).
Types of probe
The kubelet can optionally perform and react to three kinds of probes on running
Indicates whether the container is running. If
the liveness probe fails, the kubelet kills the container, and the container
is subjected to its
restart policy
. If a container does not
provide a liveness probe, the default state is
Success
.
Indicates whether the container is ready to respond to requests.
If the readiness probe fails, the endpoints controller removes the Pod's IP
address from the endpoints of all Services that match the Pod. The default
state of readiness before the initial delay is
Failure
. If a container does
not provide a readiness probe, the default state is
Success
.
Indicates whether the application within the container is started.
All other probes are disabled if a startup probe is provided, until it succeeds.
If the startup probe fails, the kubelet kills the container, and the container
provide a startup probe, the default state is
Success
.
For more information about how to set up a liveness, readiness, or startup probe,
see
.
When should you use a liveness probe?
Kubernetes v1.0 [stable]
If the process in your container is able to crash on its own whenever it
encounters an issue or becomes unhealthy, you do not

 necessarily need a liveness
probe; the kubelet will automatically perform the correct action in accordance
with the Pod's
.
If you'd like your container to be killed and restarted if a probe fails, then
specify a liveness probe, and specify a
of Always or OnFailure.
When should you use a readiness probe?
If you'd like to start sending traffic to a Pod only when a probe succeeds,
specify a readiness probe. In this case, the readiness probe might be the same
as the liveness probe, but the existence of the readiness probe in the spec means
that the Pod will start without receiving any traffic and only start receiving
traffic after the probe starts succeeding.
If you want your container to be able to take itself down for maintenance, you
can specify a readiness probe that checks an endpoint specific to readiness that
is different from the liveness probe.
If your app has a strict dependency on back-end services, you can implement both
a liveness and a readiness probe. The liveness probe passes when the app itself
is healthy, but the readiness probe additionally checks that each required
back-end service is available. This helps you avoid directing traffic to Pods
that can only respond with error messages.
If your container needs to work on loading large data, configuration files, or
migrations during startup, you can use a
startup probe
. However, if you want to
detect the difference between an app that has failed and an app that is still
processing its startup data, you might prefer a readiness probe.
Note:
If you want to be able to drain requests when the Pod is deleted, you do not
necessarily need a readiness probe; on deletion, the Pod automatically puts itself
into an unready state regardless of whether the readiness probe exists.
The Pod remains in the unready state while it waits for the containers in the Pod
to stop.
When should you use a startup probe?
Startup probes are useful for Pods that have containers that take a long time to
come into service. Rather than set a long liveness interval, you can configure
a separate configuration for probing the container as it starts up, allowing
a time longer than the liveness interval would allow.
If your container usually starts in more than
initialDelaySeconds + failureThreshold × periodSeconds
, you should specify a
startup probe that checks the same endpoint as the liveness probe. The default for
is 10s. You should then set its
high enough to
allow the container to start, without changing the default values of the liveness
probe. This helps to protect against deadlocks.
Because Pods represent processes running on nodes in the cluster, it is important to
allow those processes to gracefully terminate when they are no longer needed (rather
than being abruptly stopped with a
KILL
signal and having no chance to clean up).
The design aim is for you to be able to request deletion and know when processes
terminate, but also be able to ensure that deletes eventually complete.
When you request deletion of a Pod, the cluster records and tracks the intended grace period
before the Pod is allowed to be forcefully killed. With that forceful shutdown tracking in
place, the
kubelet
attempts graceful
shutdown.
Typically, the container runtime sends a TERM signal to the main process in each
container. Many container runtimes respect the
STOPSIGNAL
value defined in the container
image and send this instead of TERM.
Once the grace period has expired, the KILL signal is sent to any remaining
processes, and the Pod is then deleted from the
API Server
. If the kubelet or the
container runtime's management service is restarted while waiting for processes to terminate, the
cluster retries from the start including the full original grace period.
An example flow:
You use the
kubectl
tool to manually delete a specific Pod, with the default grace pe

riod
(30 seconds).
The Pod in the API server is updated with the time beyond which the Pod is considered "dead"
along with the grace period.
If you use
to check on the Pod you're deleting, that Pod shows up as
"Terminating".
On the node where the Pod is running: as soon as the kubelet sees that a Pod has been marked
as terminating (a graceful shutdown duration has been set), the kubelet begins the local Pod
shutdown process.
If one of the Pod's containers has defined a
preStop
hook
, the kubelet
runs that hook inside of the container. If the
preStop
hook is still running after the
grace period expires, the kubelet requests a small, one-off grace period extension of 2
seconds.
Note:
If the
preStop
hook needs longer to complete than the default grace period allows,
you must modify
to suit this.
The kubelet triggers the container runtime to send a TERM signal to process 1 inside each
Note:
The containers in the Pod receive the TERM signal at different times and in an arbitrary
order. If the order of shutdowns matters, consider using a
preStop
hook to synchronize.
At the same time as the kubelet is starting graceful shutdown, the control plane removes that
shutting-down Pod from EndpointSlice (and Endpoints) objects where these represent
a
Service
with a configured
selector
.
and other workload resources
no longer treat the shutting-down Pod as a valid, in-service replica. Pods that shut down slowly
cannot continue to serve traffic as load balancers (like the service proxy) remove the Pod from
the list of endpoints as soon as the termination grace period
begins
.
When the grace period expires, the kubelet triggers forcible shutdown. The container runtime sends
SIGKILL
to any processes still running in any container in the Pod.
The kubelet also cleans up a hidden
pause
container if that container runtime uses one.
The kubelet triggers forcible removal of Pod object from the API server, by setting grace period
to 0 (immediate deletion).
The API server deletes the Pod's API object, which is then no longer visible from any client.
Forced Pod termination
Caution:
Forced deletions can be potentially disruptive for some workloads and their Pods.
By default, all deletes are graceful within 30 seconds. The
command supports
the
--grace-period=&lt;seconds&gt;
option which allows you to override the default and specify your
own value.
Setting the grace period to
0
forcibly and immediately deletes the Pod from the API
server. If the pod was still running on a node, that forcible deletion triggers the kubelet to
begin immediate cleanup.
Note:
You must specify an additional flag
--force
along with
--grace-period=0
in order to perform force deletions.
When a force deletion is performed, the API server does not wait for confirmation
from the kubelet that the Pod has been terminated on the node it was running on. It
removes the Pod in the API immediately so a new Pod can be created with the same
name. On the node, Pods that are set to terminate immediately will still be given
a small grace period before being force killed.
Caution:
Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
If you need to force-delete Pods that are part of a StatefulSet, refer to the task
deleting Pods from a StatefulSet
.
Garbage collection of Pods
For failed Pods, the API objects remain in the cluster's API until a human or
process
explicitly removes them.
The Pod garbage collector (PodGC), which is a controller in the control plane, cleans up terminated Pods (with a phase of
Succeeded
or
Failed
), when the number of Pods exceeds the configured threshold
(determined by
terminated-pod-gc-threshold
in the kube-controller-manager).
This avoids a resource leak as Pods are created and terminated over time.
Additionally, PodGC cleans up any Pods which satisfy 

any of the following conditions:
are orphan pods - bound to a node which no longer exists,
are unscheduled terminating pods,
are terminating pods, bound to a non-ready node tainted with
, when the
feature gate is enabled.
When the
feature gate is enabled, along with
cleaning up the pods, PodGC will also mark them as failed if they are in a non-terminal
phase. Also, PodGC adds a pod disruption condition when cleaning up an orphan
pod (see also:
Pod disruption conditions
).
attaching handlers to container lifecycle events
.
configuring Liveness, Readiness and Startup Probes
.
.
For detailed information about Pod and container status in the API, see
the API reference documentation covering
.status
for Pod.
5.1.2 - Init Containers
This page provides an overview of init containers: specialized containers that run
before app containers in a
Pod
.
Init containers can contain utilities or setup scripts not present in an app image.
You can specify init containers in the Pod specification alongside the
array (which describes app containers).
Understanding init containers
A
Pod
can have multiple containers
running apps within it, but it can also have one or more init containers, which are run
before the app containers are started.
Init containers are exactly like regular containers, except:
Init containers always run to completion.
Each init container must complete successfully before the next one starts.
If a Pod's init container fails, the kubelet repeatedly restarts that init container until it succeeds.
However, if the Pod has a
of Never, and an init container fails during startup of that Pod, Kubernetes treats the overall Pod as failed.
To specify an init container for a Pod, add the
field into
the
Pod specification
,
as an array of
container
items (similar to the app
field and its contents).
See
Container
in the
API reference for more details.
The status of the init containers is returned in
.status.initContainerStatuses
field as an array of the container statuses (similar to the
.status.containerStatuses
field).
Differences from regular containers
Init containers support all the fields and features of app containers,
including resource limits, volumes, and security settings. However, the
resource requests and limits for an init container are handled differently,
as documented in
Resources
.
Also, init containers do not support
lifecycle
,
,
, or
because they must run to completion before the Pod can be ready.
If you specify multiple init containers for a Pod, kubelet runs each init
container sequentially. Each init container must succeed before the next can run.
When all of the init containers have run to completion, kubelet initializes
the application containers for the Pod and runs them as usual.
Using init containers
Because init containers have separate images from app containers, they
have some advantages for start-up related code:
Init containers can contain utilities or custom code for setup that are not present in an app
image. For example, there is no need to make an image
FROM
another image just to use a tool like
sed
,
awk
,
python
, or
dig
during setup.
The application image builder and deployer roles can work independently without
the need to jointly build a single app image.
Init containers can run with a different view of the filesystem than app containers in the
same Pod. Consequently, they can be given access to
Secrets
that app containers cannot access.
Because init containers run to completion before any app containers start, init containers offer
a mechanism to block or delay app container startup until a set of preconditions are met. Once
preconditions are met, all of the app containers in a Pod can start in parallel.
Init container

s can securely run utilities or custom code that would otherwise make an app
container image less secure. By keeping unnecessary tools separate you can limit the attack
surface of your app container image.
Examples
Here are some ideas for how to use init containers:
Wait for a
Service
to
be created, using a shell one-line command like:
for
i in
{
1..100
}
;
do
sleep 1;
if
dig myservice;
then
exit
0;
fi
;
done
;
exit
1
Register this Pod with a remote server from the downward API with a command like:
curl -X POST http://
$MANAGEMENT_SERVICE_HOST
:
$MANAGEMENT_SERVICE_PORT
/register -d
'instance=$(&lt;POD_NAME&gt;)&amp;ip=$(&lt;POD_IP&gt;)'
Wait for some time before starting the app container with a command like
sleep
60
Clone a Git repository into a
Volume
Place values into a configuration file and run a template tool to dynamically
generate a configuration file for the main app container. For example,
place the
POD_IP
value in a configuration and generate the main app
configuration file using Jinja.
Init containers in use
This example defines a simple Pod that has two init containers.
The first waits for
myservice
, and the second waits for
mydb
. Once both
init containers complete, the Pod runs the app container from its
spec
section.
:
v1
kind
:
Pod
metadata
:
name
:
myapp-pod
labels
:
:
MyApp
spec
:
:
-
name
:
myapp-container
image
:
command
:
[
'sh'
,
'-c'
,
'echo The app is running! &amp;&amp; sleep 3600'
]
:
-
name
:
init-myservice
image
:
command
:
[
'sh'
,
'-c'
,
"until nslookup myservice.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for myservice; sleep 2; done"
]
-
name
:
init-mydb
image
:
command
:
[
'sh'
,
'-c'
,
"until nslookup mydb.$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace).svc.cluster.local; do echo waiting for mydb; sleep 2; done"
]
You can start this Pod by running:
kubectl apply -f myapp.yaml
pod/myapp-pod created
And check on its status with:
kubectl get -f myapp.yaml
NAME        READY     STATUS     RESTARTS   AGE
myapp-pod   0/1       Init:0/2   0          6m
or for more details:
kubectl describe -f myapp.yaml
Name:          myapp-pod
[...]
Labels:        app.kubernetes.io/name=MyApp
[...]
  init-myservice:
[...]
    State:         Running
[...]
  init-mydb:
[...]
    State:         Waiting
      Reason:      PodInitializing
    Ready:         False
[...]
  myapp-container:
[...]
[...]
Events:
  FirstSeen    LastSeen    Count    From                      SubObjectPath                           Type          Reason        Message
  ---------    --------    -----    ----                      -------------                           --------      ------        -------
  16s          16s         1        {default-scheduler }                                              Normal        Scheduled     Successfully assigned myapp-pod to 172.17.4.201
  16s          16s         1        {kubelet 172.17.4.201}    spec.initContainers{init-myservice}     Normal        Pulling       pulling image "busybox"
  13s          13s         1        {kubelet 172.17.4.201}    spec.initContainers{init-myservice}     Normal        Pulled        Successfully pulled image "busybox"
  13s          13s         1        {kubelet 172.17.4.201}    spec.initContainers{init-myservice}     Normal        Created       Created container init-myservice
  13s          13s         1        {kubelet 172.17.4.201}    spec.initContainers{init-myservice}     Normal        Started       Started container init-myservice
To see logs for the init containers in this Pod, run:
kubectl logs myapp-pod -c init-myservice
# Inspect the first init

 container
kubectl logs myapp-pod -c init-mydb
# Inspect the second init container
At this point, those init containers will be waiting to discover Services named
mydb
and
myservice
.
Here's a configuration you can use to make those Services appear:
---
:
v1
kind
:
Service
metadata
:
name
:
myservice
spec
:
ports
:
-
protocol
:
TCP
port
:
80
:
9376
---
:
v1
kind
:
Service
metadata
:
name
:
mydb
spec
:
ports
:
-
protocol
:
TCP
port
:
80
:
9377
To create the
mydb
and
myservice
services:
kubectl apply -f services.yaml
service/myservice created
service/mydb created
You'll then see that those init containers complete, and that the
myapp-pod
Pod moves into the Running state:
myapp-pod   1/1       Running   0          9m
This simple example should provide some inspiration for you to create your own
init containers.
contains a link to a more detailed example.
Detailed behavior
During Pod startup, the kubelet delays running init containers until the networking
and storage are ready. Then the kubelet runs the Pod's init containers in the order
they appear in the Pod's spec.
Each init container must exit successfully before
the next container starts. If a container fails to start due to the runtime or
exits with failure, it is retried according to the Pod
. However,
if the Pod
is set to Always, the init containers use
OnFailure.
A Pod cannot be
Ready
until all init containers have succeeded. The ports on an
init container are not aggregated under a Service. A Pod that is initializing
is in the
Pending
state but should have a condition
If the Pod
restarts
, or is restarted, all init containers
must execute again.
Changes to the init container spec are limited to the container image field.
Altering an init container image field is equivalent to restarting the Pod.
Because init containers can be restarted, retried, or re-executed, init container
code should be idempotent. In particular, code that writes to files on
EmptyDirs
should be prepared for the possibility that an output file already exists.
Init containers have all of the fields of an app container. However, Kubernetes
prohibits
from being used because init containers cannot
define readiness distinct from completion. This is enforced during validation.
Use
on the Pod to prevent init containers from failing forever.
The active deadline includes init containers.
However it is recommended to use
only if teams deploy their application
as a Job, because
has an effect even after initContainer finished.
The Pod which is already running correctly would be killed by
if you set.
The name of each app and init container in a Pod must be unique; a
validation error is thrown for any container sharing a name with another.
Resources
Given the ordering and execution for init containers, the following rules
for resource usage apply:
The highest of any particular resource request or limit defined on all init
containers is the
effective init request/limit
. If any resource has no
resource limit specified this is considered as the highest limit.
The Pod's
effective request/limit
for a resource is the higher of:
the sum of all app containers request/limit for a resource
the effective init request/limit for a resource
Scheduling is done based on effective requests/limits, which means
init containers can reserve resources for initialization that are not used
during the life of the Pod.
The QoS (quality of service) tier of the Pod's
effective QoS tier
is the
QoS tier for init containers and app containers alike.
Quota and limits are applied based on the effective Pod request and
limit.
Pod level control groups (cgroups) are based on the effective Pod r

equest and
limit, the same as the scheduler.
Pod restart reasons
A Pod can restart, causing re-execution of init containers, for the following
reasons:
The Pod infrastructure container is restarted. This is uncommon and would
have to be done by someone with root access to nodes.
All containers in a Pod are terminated while
is set to Always,
forcing a restart, and the init container completion record has been lost due
to garbage collection.
The Pod will not be restarted when the init container image is changed, or the
init container completion record has been lost due to garbage collection. This
applies for Kubernetes v1.20 and later. If you are using an earlier version of
Kubernetes, consult the documentation for the version you are using.
Read about
creating a Pod that has an init container
debug init containers
5.1.3 - Disruptions
This guide is for application owners who want to build
highly available applications, and thus need to understand
what types of disruptions can happen to Pods.
It is also for cluster administrators who want to perform automated
cluster actions, like upgrading and autoscaling clusters.
Voluntary and involuntary disruptions
Pods do not disappear until someone (a person or a controller) destroys them, or
there is an unavoidable hardware or system software error.
We call these unavoidable cases
involuntary disruptions
to
an application. Examples are:
a hardware failure of the physical machine backing the node
cluster administrator deletes VM (instance) by mistake
cloud provider or hypervisor failure makes VM disappear
a kernel panic
the node disappears from the cluster due to cluster network partition
eviction of a pod due to the node being
out-of-resources
.
Except for the out-of-resources condition, all these conditions
should be familiar to most users; they are not specific
to Kubernetes.
We call other cases
voluntary disruptions
. These include both
actions initiated by the application owner and those initiated by a Cluster
Administrator. Typical application owner actions include:
deleting the deployment or other controller that manages the pod
updating a deployment's pod template causing a restart
directly deleting a pod (e.g. by accident)
Cluster administrator actions include:
Draining a node
for repair or upgrade.
Draining a node from a cluster to scale the cluster down (learn about
Cluster Autoscaling
).
Removing a pod from a node to permit something else to fit on that node.
These actions might be taken directly by the cluster administrator, or by automation
run by the cluster administrator, or by your cluster hosting provider.
Ask your cluster administrator or consult your cloud provider or distribution documentation
to determine if any sources of voluntary disruptions are enabled for your cluster.
If none are enabled, you can skip creating Pod Disruption Budgets.
Caution:
Not all voluntary disruptions are constrained by Pod Disruption Budgets. For example,
deleting deployments or pods bypasses Pod Disruption Budgets.
Dealing with disruptions
Here are some ways to mitigate involuntary disruptions:
Ensure your pod
requests the resources
it needs.
Replicate your application if you need higher availability. (Learn about running replicated
stateless
and
stateful
applications.)
For even higher availability when running replicated applications,
spread applications across racks (using
anti-affinity
)
or across zones (if using a
multi-zone cluster
.)
The frequency of voluntary disruptions varies. On a basic Kubernetes cluster, there are
no automated voluntary disruptions (only user-triggered ones). However, your cluster administrator or hosting provider
may run some additional services which cause voluntary disruptions. For example,
rolling out node software updates can cause voluntary disruptions. Also, some implementations
of cluster (node) autoscaling may cause voluntary disruptions to defragment and compact nodes.
Your cluster administrator or hosting provid

er should have documented what level of voluntary
disruptions, if any, to expect. Certain configuration options, such as
using PriorityClasses
in your pod spec can also cause voluntary (and involuntary) disruptions.
Pod disruption budgets
Kubernetes offers features to help you run highly available applications even when you
introduce frequent voluntary disruptions.
As an application owner, you can create a PodDisruptionBudget (PDB) for each application.
A PDB limits the number of Pods of a replicated application that are down simultaneously from
voluntary disruptions. For example, a quorum-based application would
like to ensure that the number of replicas running is never brought below the
number needed for a quorum. A web front end might want to
ensure that the number of replicas serving load never falls below a certain
percentage of the total.
Cluster managers and hosting providers should use tools which
respect PodDisruptionBudgets by calling the
instead of directly deleting pods or deployments.
subcommand lets you mark a node as going out of
service. When you run
, the tool tries to evict all of the Pods on
the Node you're taking out of service. The eviction request that
kubectl
submits on
your behalf may be temporarily rejected, so the tool periodically retries all failed
requests until all Pods on the target node are terminated, or until a configurable timeout
is reached.
A PDB specifies the number of replicas that an application can tolerate having, relative to how
many it is intended to have. For example, a Deployment which has a
.spec.replicas: 5
is
supposed to have 5 pods at any given time. If its PDB allows for there to be 4 at a time,
then the Eviction API will allow voluntary disruption of one (but not two) pods at a time.
The group of pods that comprise the application is specified using a label selector, the same
as the one used by the application's controller (deployment, stateful-set, etc).
The "intended" number of pods is computed from the
of the workload resource
that is managing those pods. The control plane discovers the owning workload resource by
examining the
.metadata.ownerReferences
of the Pod.
Involuntary disruptions
cannot be prevented by PDBs; however they
Pods which are deleted or unavailable due to a rolling upgrade to an application do count
against the disruption budget, but workload resources (such as Deployment and StatefulSet)
are not limited by PDBs when doing rolling upgrades. Instead, the handling of failures
during application updates is configured in the spec for the specific workload resource.
When a pod is evicted using the eviction API, it is gracefully
, honoring the
setting in its
PodSpec
.
PodDisruptionBudget example
Consider a cluster with 3 nodes,
node-1
through
node-3
.
The cluster is running several applications. One of them has 3 replicas initially called
pod-a
,
pod-b
, and
pod-c
. Another, unrelated pod without a PDB, called
pod-x
, is also shown.
Initially, the pods are laid out as follows:
node-1
node-2
node-3
pod-a
available
pod-b
available
pod-c
available
pod-x
available
All 3 pods are part of a deployment, and they collectively have a PDB which requires
there be at least 2 of the 3 pods to be available at all times.
For example, assume the cluster administrator wants to reboot into a new kernel version to fix a bug in the kernel.
The cluster administrator first tries to drain
node-1
using the
command.
That tool tries to evict
pod-a
and
pod-x
. This succeeds immediately.
Both pods go into the
state at the same time.
This puts the cluster in this state:
node-1
draining
node-2
node-3
pod-a
pod-b
available
pod-c
available
pod-x
The deployment notices that one of the pods is terminating, so it creates a replacement
called
pod-d
. Since
node-1
i

s cordoned, it lands on another node. Something has
also created
pod-y
as a replacement for
pod-x
.
(Note: for a StatefulSet,
pod-a
, which would be called something like
pod-0
, would need
to terminate completely before its replacement, which is also called
pod-0
but has a
different UID, could be created. Otherwise, the example applies to a StatefulSet as well.)
Now the cluster is in this state:
node-1
draining
node-2
node-3
pod-a
pod-b
available
pod-c
available
pod-x
pod-d
starting
pod-y
At some point, the pods terminate, and the cluster looks like this:
node-1
drained
node-2
node-3
pod-b
available
pod-c
available
pod-d
starting
pod-y
At this point, if an impatient cluster administrator tries to drain
node-2
or
node-3
, the drain command will block, because there are only 2 available
pods for the deployment, and its PDB requires at least 2. After some time passes,
pod-d
becomes available.
The cluster state now looks like this:
node-1
drained
node-2
node-3
pod-b
available
pod-c
available
pod-d
available
pod-y
Now, the cluster administrator tries to drain
node-2
.
The drain command will try to evict the two pods in some order, say
pod-b
first and then
pod-d
. It will succeed at evicting
pod-b
.
But, when it tries to evict
pod-d
, it will be refused because that would leave only
one pod available for the deployment.
The deployment creates a replacement for
pod-b
called
pod-e
.
Because there are not enough resources in the cluster to schedule
pod-e
the drain will again block. The cluster may end up in this
state:
node-1
drained
node-2
node-3
no node
pod-b
pod-c
available
pod-e
pending
pod-d
available
pod-y
At this point, the cluster administrator needs to
add a node back to the cluster to proceed with the upgrade.
You can see how Kubernetes varies the rate at which disruptions
can happen, according to:
how many replicas an application needs
how long it takes to gracefully shutdown an instance
how long it takes a new instance to start up
the type of controller
the cluster's resource capacity
Note:
If you are using an older version of Kubernetes than 1.26
please refer to the corresponding version of the documentation.
Note:
In order to use this behavior, you must have the
enabled in your cluster.
When enabled, a dedicated Pod
condition
is added to indicate
that the Pod is about to be deleted due to a
disruption
.
The
reason
field of the condition additionally
indicates one of the following reasons for the Pod termination:
PreemptionByKubeScheduler
Pod is due to be
preempted
by a scheduler in order to accommodate a new Pod with a higher priority. For more information, see
Pod priority preemption
.
DeletionByTaintManager
Pod is due to be deleted by Taint Manager (which is part of the node lifecycle controller within
) due to a
NoExecute
taint that the Pod does not tolerate; see
taint
-based evictions.
EvictionByEvictionAPI
Pod has been marked for
eviction using the Kubernetes API
.
DeletionByPodGC
Pod, that is bound to a no longer existing Node, is due to be deleted by
Pod garbage collection
.
TerminationByKubelet
Pod has been terminated by the kubelet, because of either
or the
graceful node shutdown
.
Note:
A Pod disruption might be interrupted. The control plane might re-attempt to
continue the disruption of the same Pod, but it is not guaranteed. As a result,
the
condition might be added to a Pod, but that Pod might then not actually be
deleted. In such a situation, after some time, the
Pod disruption condition will be cleared.
When the
feature gate is enabled,
along with cleaning up the pods, the Pod garbage collector (PodGC) will also mark them as failed if they are in a non-terminal
phase (see also
).
When using a Job (or CronJob), you may want to use these Pod disr

uption conditions as part of your Job's
.
Separating Cluster Owner and Application Owner Roles
Often, it is useful to think of the Cluster Manager
and Application Owner as separate roles with limited knowledge
of each other. This separation of responsibilities
may make sense in these scenarios:
when there are many application teams sharing a Kubernetes cluster, and
there is natural specialization of roles
when third-party tools or services are used to automate cluster management
Pod Disruption Budgets support this separation of roles by providing an
interface between the roles.
If you do not have such a separation of responsibilities in your organization,
you may not need to use Pod Disruption Budgets.
How to perform Disruptive Actions on your Cluster
If you are a Cluster Administrator, and you need to perform a disruptive action on all
the nodes in your cluster, such as a node or system software upgrade, here are some options:
Accept downtime during the upgrade.
Failover to another complete replica cluster.
No downtime, but may be costly both for the duplicated nodes
and for human effort to orchestrate the switchover.
Write disruption tolerant applications and use PDBs.
No downtime.
Minimal resource duplication.
Allows more automation of cluster administration.
Writing disruption-tolerant applications is tricky, but the work to tolerate voluntary
disruptions largely overlaps with work to support autoscaling and tolerating
involuntary disruptions.
.
draining nodes
updating a deployment
including steps to maintain its availability during the rollout.
5.1.4 - Ephemeral Containers
This page provides an overview of ephemeral containers: a special type of container
that runs temporarily in an existing
Pod
to
accomplish user-initiated actions such as troubleshooting. You use ephemeral
containers to inspect services rather than to build applications.
Understanding ephemeral containers
Pods
are the fundamental building
block of Kubernetes applications. Since Pods are intended to be disposable and
replaceable, you cannot add a container to a Pod once it has been created.
Instead, you usually delete and replace Pods in a controlled fashion using
.
Sometimes it's necessary to inspect the state of an existing Pod, however, for
example to troubleshoot a hard-to-reproduce bug. In these cases you can run
an ephemeral container in an existing Pod to inspect its state and run
arbitrary commands.
What is an ephemeral container?
Ephemeral containers differ from other containers in that they lack guarantees
for resources or execution, and they will never be automatically restarted, so
they are not appropriate for building applications. Ephemeral containers are
described using the same
ContainerSpec
as regular containers, but many fields
are incompatible and disallowed for ephemeral containers.
Ephemeral containers may not have ports, so fields such as
ports
,
,
are disallowed.
Pod resource allocations are immutable, so setting
resources
is disallowed.
For a complete list of allowed fields, see the
EphemeralContainer reference
.
Ephemeral containers are created using a special
ephemeralcontainers
handler
in the API rather than by adding them directly to
pod.spec
, so it's not
possible to add an ephemeral container using
.
Like regular containers, you may not change or remove an ephemeral container
after you have added it to a Pod.
Note:
.
Uses for ephemeral containers
Ephemeral containers are useful for interactive troubleshooting when
is insufficient because a container has crashed or a container image
doesn't include debugging utilities.
In particular,
enable you to deploy minimal container images that reduce attack s

urface
and exposure to bugs and vulnerabilities. Since distroless images do not include a
shell or any debugging utilities, it's difficult to troubleshoot distroless
images using
alone.
When using ephemeral containers, it's helpful to enable
process namespace
sharing
so
you can view processes in other containers.
debug pods using ephemeral containers
.
5.1.5 - User Namespaces
This page explains how user namespaces are used in Kubernetes pods. A user
namespace allows to isolate the user running inside the container from the one
in the host.
HIGH
or
CRITICAL
This is a Linux only feature. In addition, support is needed in the
containerd: support is planned for the 1.7 release. See containerd
issue
#7063
Support for this in
cri-dockerd is not planned
yet.
User namespaces is a Linux feature that allows to map users in the container to
different users in the host. Furthermore, the capabilities granted to a pod in
a user namespace are valid only in the namespace and void outside of it.
A pod can opt-in to use user namespaces by setting the
pod.spec.hostUsers
field
to
false
.
The kubelet will pick host UIDs/GIDs a pod is mapped to, and will do so in a way
to guarantee that no two stateless pods on the same node use the same mapping.
The
runAsUser
,
,
fsGroup
, etc. fields in the
pod.spec
always
refer to the user inside the container.
The valid UIDs/GIDs when this feature is enabled is the range 0-65535. This
applies to files and processes (
runAsUser
,
, etc.).
Files using a UID/GID outside this range will be seen as belonging to the
overflow ID, usually 65534 (configured in
/proc/sys/kernel/overflowuid
and
/proc/sys/kernel/overflowgid
). However, it is not possible to modify those
files, even by running as the 65534 user/group.
Most applications that need to run as root but don't access other host
namespaces or resources, should continue to run fine without any changes needed
if user namespaces is activated.
Understanding user namespaces for stateless pods
Several container runtimes with their default configuration (like Docker Engine,
containerd, CRI-O) use Linux namespaces for isolation. Other technologies exist
and can be used with those runtimes too (e.g. Kata Containers uses VMs instead of
Linux namespaces). This page is applicable for container runtimes using Linux
namespaces for isolation.
When creating a pod, by default, several new namespaces are used for isolation:
a network namespace to isolate the network of the container, a PID namespace to
isolate the view of processes, etc. If a user namespace is used, this will
isolate the users in the container from the users in the node.
This means containers can run as root and be mapped to a non-root user on the
host. Inside the container the process will think it is running as root (and
therefore tools like
apt
,
yum
, etc. work fine), while in reality the process
doesn't have privileges on the host. You can verify this, for example, if you
check which user the container process is running by executing
ps aux
from
the host. The user
ps
shows is not the same as the user y

ou see if you
execute inside the container the command
id
.
This abstraction limits what can happen, for example, if the container manages
to escape to the host. Given that the container is running as a non-privileged
user on the host, it is limited what it can do to the host.
Furthermore, as users on each pod will be mapped to different non-overlapping
users in the host, it is limited what they can do to other pods too.
Capabilities granted to a pod are also limited to the pod user namespace and
mostly invalid out of it, some are even completely void. Here are two examples:
CAP_SYS_MODULE
does not have any effect if granted to a pod using user
namespaces, the pod isn't able to load kernel modules.
is limited to the pod's user namespace and invalid outside
of it.
this is true when we use user namespaces.
If you want to know more details about what changes when user namespaces are in
use, see
man 7 user_namespaces
.
Set up a node to support user namespaces
It is recommended that the host's files and host's processes use UIDs/GIDs in
the range of 0-65535.
The kubelet will assign UIDs/GIDs higher than that to pods. Therefore, to
guarantee as much isolation as possible, the UIDs/GIDs used by the host's files
and host's processes should be in the range 0-65535.
Note that this recommendation is important to mitigate the impact of CVEs like
, where a pod can potentially read arbitrary
files in the hosts. If the UIDs/GIDs of the pod and the host don't overlap, it
is limited what a pod would be able to do: the pod UID/GID won't match the
host's file owner/group.
When using a user namespace for the pod, it is disallowed to use other host
namespaces. In particular, if you set
hostUsers: false
then you are not
allowed to set any of:
hostIPC: true
hostPID: true
The pod is allowed to use no volumes at all or, if using volumes, only these
volume types are allowed:
configmap
secret
projected
emptyDir
To guarantee that the pod can read the files of such volumes, volumes are
created as if you specified
as
0
for the Pod.
If it is specified to a different value, this other value will of course be
honored instead.
As a by-product of this, folders and files for these volumes will have
permissions for the group, even if
or
mode
to specific items of
the volumes were specified without permissions to groups. For example, it is not
possible to mount these volumes in a way that its files have permissions only
for the owner.
5.1.6 - Downward API
There are two ways to expose Pod and container fields to a running container: environment variables, and as files that are populated by a special volume type. Together, these two ways of exposing Pod and container fields are called the downward API.
being overly coupled to Kubernetes. The
allows containers to consume
information about themselves or the cluster without using the Kubernetes client
or API server.
An example is an existing application that assumes a particular well-known
environment variable holds a unique identifier. One possibility is to wrap the
application, but that is tedious and error-prone, and it violates the goal of low
coupling. A better option would be to use the Pod's name as an identifier, and
inject the Pod's name into the well-known environment variable.
as
as
files in a
volume
.
Available fields
Only some Kubernetes API fields are availab

le through the downward API. This
section lists which fields you can make available.
You can pass information from available Pod-level fields using
fieldRef
.
At the API level, the
spec
for a Pod always defines at least one
Container
.
You can pass information from available Container-level fields using
.
Information available via
fieldRef
For most Pod-level fields, you can provide them to a container either as
an environment variable or using a
volume. The fields available
via either mechanism are:
the pod's name
the pod's
namespace
metadata.uid
the pod's unique ID
metadata.annotations['&lt;KEY&gt;']
the value of the pod's
named
&lt;KEY&gt;
metadata.annotations['myannotation']
)
metadata.labels['&lt;KEY&gt;']
the text value of the pod's
label
named
metadata.labels['mylabel']
)
the name of the pod's
the name of the
node
where the Pod is executing
status.hostIP
the primary IP address of the node to which the Pod is assigned
the pod's primary IP address (usually, its IPv4 address)
In addition, the following information is available through
a
volume
fieldRef
, but
not as environment variables
:
all of the pod's labels, formatted as
label-key="escaped-label-value"
with one label per line
all of the pod's annotations, formatted as
annotation-key="escaped-annotation-value"
with one annotation per line
These container-level fields allow you to provide information about
for resources such as CPU and memory.
resource: limits.cpu
A container's CPU limit
resource: requests.cpu
A container's CPU request
resource: limits.memory
A container's memory limit
resource: requests.memory
A container's memory request
resource: limits.hugepages-*
A container's hugepages limit (provided that the
is enabled)
resource: requests.hugepages-*
A container's hugepages request (provided that the
resource: limits.ephemeral-storage
A container's ephemeral-storage limit
resource: requests.ephemeral-storage
A container's ephemeral-storage request
Fallback information for resource limits
If CPU and memory limits are not specified for a container, and you use the
downward API to try to expose that information, then the
kubelet defaults to exposing the maximum allocatable value for CPU and memory
based on the
calculation.
You can read about
volumes
.
You can try using the downward API to expose container- or Pod-level information:
as
as
files in
volume
5.2 - Workload Resources
5.2.1 - Deployments
A
provides declarative updates for
Pods
and
.
You describe a
in a Deployment, and the Deployment
changes the actual state to the desired state at a controlled rate. You can define Deployments to create new ReplicaSets, or to remove existing Deployments and adopt all their resources with new Deployments.
Note:
Do not manage ReplicaSets owned by a Deployment. Consider opening an issue in the main Kubernetes repository if your use case is not covered below.
Use Case
The following are typical use cases for Deployments:
Create a Deployment to rollout a ReplicaSet
. The ReplicaSet creates Pods in the background. Check the status of the rollout to see if it succeeds or not.
Declare the new state of the Pods
by updating the PodTemplateSpec of the Deployment. A new ReplicaSet is created and the Deployment manages moving the Pods from the old ReplicaSet to the new one at a controlled rate. Each new ReplicaSet updates the revision of the Deployment.
Rollback to an earlier Deployment revision
if the current state of the Deployment is not stable. Each rollback updates the revision of the Deployment.

Scale up the Deployment to facilitate more load
.
Pause the rollout of a Deployment
to apply multiple fixes to its PodTemplateSpec and then resume it to start a new rollout.
Use the status of the Deployment
as an indicator that a rollout has stuck.
Clean up older ReplicaSets
that you don't need anymore.
Creating a Deployment
The following is an example of a Deployment. It creates a ReplicaSet to bring up three
nginx
Pods:
controllers/nginx-deployment.yaml
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
nginx
spec
:
replicas
:
3
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
In this example:
A Deployment named
is created, indicated by the
field. This name will become the basis for the ReplicaSets
and Pods which are created later. See
Writing a Deployment Spec
The Deployment creates a ReplicaSet that creates three replicated Pods, indicated by the
field.
The
field defines how the created ReplicaSet finds which Pods to manage.
In this case, you select a label that is defined in the Pod template (
app: nginx
).
However, more sophisticated selection rules are possible,
as long as the Pod template itself satisfies the rule.
Note:
The
.spec.selector.matchLabels
field is a map of {key,value} pairs.
A single {key,value} in the
,
whose
key
operator
values
All of the requirements, from both
and
, must be satisfied in order to match.
The
template
field contains the following sub-fields:
The Pods are labeled
app: nginx
using the
.metadata.labels
field.
The Pod template's specification, or
.template.spec
field, indicates that
the Pods run one container,
nginx
, which runs the
nginx
Docker Hub
image at version 1.14.2.
Create one container and name it
nginx
using the
.spec.template.spec.containers[0].name
field.
Before you begin, make sure your Kubernetes cluster is up and running.
Follow the steps given below to create the above Deployment:
Create the Deployment by running the following command:
kubectl apply -f https://k8s.io/examples/controllers/nginx-deployment.yaml
Run
to check if the Deployment was created.
If the Deployment is still being created, the output is similar to the following:
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   0/3     0            0           1s
When you inspect the Deployments in your cluster, the following fields are displayed:
NAME
lists the names of the Deployments in the namespace.
READY
displays how many replicas of the application are available to your users. It follows the pattern ready/desired.
UP-TO-DATE
displays the number of replicas that have been updated to achieve the desired state.
AVAILABLE
displays how many replicas of the application are available to your users.
AGE
displays the amount of time that the application has been running.
Notice how the number of desired replicas is 3 according to
field.
To see the Deployment rollout status, run
kubectl rollout status deployment/nginx-deployment
.
Waiting for rollout to finish: 2 out of 3 new replicas have been updated...
deployment "nginx-deployment" successfully rolled out
Run the
again a few seconds later.
nginx-deployment   3/3     3            3           18s
Notice that the Deployment has created all three replicas, and all replicas are up-to-date (they contain the latest Pod template) and available.
To see the ReplicaSet (
rs
) created by the Deployment, run
kubectl get rs
. The output is similar to this:
NAME                          DESIRED   CURRENT  

 READY   AGE
nginx-deployment-75675f5897   3         3         3       18s
ReplicaSet output shows the following fields:
NAME
lists the names of the ReplicaSets in the namespace.
DESIRED
displays the desired number of
replicas
of the application, which you define when you create the Deployment. This is the
.
CURRENT
displays how many replicas are currently running.
READY
AGE
Notice that the name of the ReplicaSet is always formatted as
[DEPLOYMENT-NAME]-[HASH]
. This name will become the basis for the Pods
which are created.
The
HASH
string is the same as the
label on the ReplicaSet.
To see the labels automatically generated for each Pod, run
.
NAME                                READY     STATUS    RESTARTS   AGE       LABELS
nginx-deployment-75675f5897-7ci7o   1/1       Running   0          18s       app=nginx,pod-template-hash=3123191453
nginx-deployment-75675f5897-kzszj   1/1       Running   0          18s       app=nginx,pod-template-hash=3123191453
nginx-deployment-75675f5897-qqcnn   1/1       Running   0          18s       app=nginx,pod-template-hash=3123191453
The created ReplicaSet ensures that there are three
nginx
Pods.
Note:
You must specify an appropriate selector and Pod template labels in a Deployment
app: nginx
).
Do not overlap labels or selectors with other controllers (including other Deployments and StatefulSets). Kubernetes doesn't stop you from overlapping, and if multiple controllers have overlapping selectors those controllers might conflict and behave unexpectedly.
Pod-template-hash label
Caution:
Do not change this label.
The
label is added by the Deployment controller to every ReplicaSet that a Deployment creates or adopts.
This label ensures that child ReplicaSets of a Deployment do not overlap. It is generated by hashing the
of the ReplicaSet and using the resulting hash as the label value that is added to the ReplicaSet selector, Pod template labels,
and in any existing Pods that the ReplicaSet might have.
Updating a Deployment
Note:
A Deployment's rollout is triggered if and only if the Deployment's Pod template (that is,
)
is changed, for example if the labels or container images of the template are updated. Other updates, such as scaling the Deployment, do not trigger a rollout.
Follow the steps given below to update your Deployment:
Let's update the nginx Pods to use the
image instead of the
image.
kubectl
set
image deployment.v1.apps/nginx-deployment
nginx
=
or use the following command:
kubectl
set
image deployment/nginx-deployment
nginx
=
deployment.apps/nginx-deployment image updated
Alternatively, you can
edit
the Deployment and change
from
to
:
kubectl edit deployment/nginx-deployment
deployment.apps/nginx-deployment edited
To see the rollout status, run:
or
Get more details on your updated Deployment:
After the rollout succeeds, you can view the Deployment by running
.
nginx-deployment   3/3     3            3           36s
Run
to see that the Deployment updated the Pods by creating a new ReplicaSet and scaling it
up to 3 replicas, as well as scaling down the old ReplicaSet to 0 replicas.
NAME                          DESIRED   CURRENT   READY   AGE
nginx-de

ployment-1564180365   3         3         3       6s
nginx-deployment-2035384211   0         0         0       36s
Running
get pods
should now show only the new Pods:
nginx-deployment-1564180365-khku8   1/1       Running   0          14s
nginx-deployment-1564180365-nacti   1/1       Running   0          14s
nginx-deployment-1564180365-z9gth   1/1       Running   0          14s
Next time you want to update these Pods, you only need to update the Deployment's Pod template again.
Deployment ensures that only a certain number of Pods are down while they are being updated. By default,
it ensures that at least 75% of the desired number of Pods are up (25% max unavailable).
Deployment also ensures that only a certain number of Pods are created above the desired number of Pods.
By default, it ensures that at most 125% of the desired number of Pods are up (25% max surge).
For example, if you look at the above Deployment closely, you will see that it first creates a new Pod,
then deletes an old Pod, and creates another new one. It does not kill old Pods until a sufficient number of
new Pods have come up, and does not create new Pods until a sufficient number of old Pods have been killed.
It makes sure that at least 3 Pods are available and that at max 4 Pods in total are available. In case of
a Deployment with 4 replicas, the number of Pods would be between 3 and 5.
Get details of your Deployment:
kubectl describe deployments
Name:                   nginx-deployment
CreationTimestamp:      Thu, 30 Nov 2017 10:56:25 +0000
Labels:                 app=nginx
Annotations:            deployment.kubernetes.io/revision=2
Selector:               app=nginx
Replicas:               3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
  Labels:  app=nginx
    nginx:
      Image:        nginx:1.16.1
      Port:         80/TCP
      Environment:  &lt;none&gt;
      Mounts:       &lt;none&gt;
    Volumes:        &lt;none&gt;
  Conditions:
    Type           Status  Reason
    ----           ------  ------
    Available      True    MinimumReplicasAvailable
    Progressing    True    NewReplicaSetAvailable
  OldReplicaSets:  &lt;none&gt;
  NewReplicaSet:   nginx-deployment-1564180365 (3/3 replicas created)
  Events:
    Type    Reason             Age   From                   Message
    ----    ------             ----  ----                   -------
    Normal  ScalingReplicaSet  2m    deployment-controller  Scaled up replica set nginx-deployment-2035384211 to 3
    Normal  ScalingReplicaSet  24s   deployment-controller  Scaled up replica set nginx-deployment-1564180365 to 1
    Normal  ScalingReplicaSet  22s   deployment-controller  Scaled down replica set nginx-deployment-2035384211 to 2
    Normal  ScalingReplicaSet  22s   deployment-controller  Scaled up replica set nginx-deployment-1564180365 to 2
    Normal  ScalingReplicaSet  19s   deployment-controller  Scaled down replica set nginx-deployment-2035384211 to 1
    Normal  ScalingReplicaSet  19s   deployment-controller  Scaled up replica set nginx-deployment-1564180365 to 3
    Normal  ScalingReplicaSet  14s   deployment-controller  Scaled down replica set nginx-deployment-2035384211 to 0
Here you see that when you first created the Deployment, it created a ReplicaSet (nginx-deployment-2035384211)
and scaled it up to 3 replicas directly. When you updated the Deployment, it created a new ReplicaSet
(nginx-deployment-1564180365) and scaled it up to 1 and waited for it to come up. Then it scaled down the old ReplicaSet
to 2 and scaled up the new ReplicaSet to 2 so that at least 3 Pods were available and at most 4 Pods were created at all times.
It then continued s

caling up and down the new and the old ReplicaSet, with the same rolling update strategy.
Finally, you'll have 3 available replicas in the new ReplicaSet, and the old ReplicaSet is scaled down to 0.
Note:
Kubernetes doesn't count terminating Pods when calculating the number of
, which must be between
replicas - maxUnavailable
and
replicas + maxSurge
. As a result, you might notice that there are more Pods than
expected during a rollout, and that the total resources consumed by the Deployment is more than
until the
of the terminating Pods expires.
Rollover (aka multiple updates in-flight)
Each time a new Deployment is observed by the Deployment controller, a ReplicaSet is created to bring up
the desired Pods. If the Deployment is updated, the existing ReplicaSet that controls Pods whose labels
match
but whose template does not match
are scaled down. Eventually, the new
ReplicaSet is scaled to
and all old ReplicaSets is scaled to 0.
If you update a Deployment while an existing rollout is in progress, the Deployment creates a new ReplicaSet
as per the update and start scaling that up, and rolls over the ReplicaSet that it was scaling up previously
-- it will add it to its list of old ReplicaSets and start scaling it down.
For example, suppose you create a Deployment to create 5 replicas of
,
but then update the Deployment to create 5 replicas of
, when only 3
replicas of
had been created. In that case, the Deployment immediately starts
killing the 3
Pods that it had created, and starts creating
Pods. It does not wait for the 5 replicas of
to be created
before changing course.
Label selector updates
It is generally discouraged to make label selector updates and it is suggested to plan your selectors up front.
In any case, if you need to perform a label selector update, exercise great caution and make sure you have grasped
all of the implications.
Note:
In API version
apps/v1
, a Deployment's label selector is immutable after it gets created.
Selector additions require the Pod template labels in the Deployment spec to be updated with the new label too,
otherwise a validation error is returned. This change is a non-overlapping one, meaning that the new selector does
not select ReplicaSets and Pods created with the old selector, resulting in orphaning all old ReplicaSets and
creating a new ReplicaSet.
Selector updates changes the existing value in a selector key -- result in the same behavior as additions.
Selector removals removes an existing key from the Deployment selector -- do not require any changes in the
Pod template labels. Existing ReplicaSets are not orphaned, and a new ReplicaSet is not created, but note that the
removed label still exists in any existing Pods and ReplicaSets.
Rolling Back a Deployment
Sometimes, you may want to rollback a Deployment; for example, when the Deployment is not stable, such as crash looping.
By default, all of the Deployment's rollout history is kept in the system so that you can rollback anytime you want
(you can change that by modifying revision history limit).
Note:
A Deployment's revision is created when a Deployment's rollout is triggered. This means that the
new revision is created if and only if the Deployment's Pod template (
) is changed,
for example if you update the labels or container images of the template. Other updates, such as scaling the Deployment,
do not create a Deployment revision, so that you can facilitate simultaneous manual- or auto-scaling.
This means that when you roll back to an earlier revision, only the Deployment's Pod template part is
rolled back.
Suppose that you made a typo while updating the Deployment, by putting the image name as
nginx:1.161
instead of
:
kubectl
set
nginx
=
The output is similar 

to this:
The rollout gets stuck. You can verify it by checking the rollout status:
Waiting for rollout to finish: 1 out of 3 new replicas have been updated...
Press Ctrl-C to stop the above rollout status watch. For more information on stuck rollouts,
read more here
.
You see that the number of old replicas (
nginx-deployment-1564180365
and
nginx-deployment-2035384211
) is 2, and new replicas (nginx-deployment-3066724191) is 1.
nginx-deployment-1564180365   3         3         3       25s
nginx-deployment-3066724191   1         1         0       6s
Looking at the Pods created, you see that 1 Pod created by new ReplicaSet is stuck in an image pull loop.
NAME                                READY     STATUS             RESTARTS   AGE
nginx-deployment-1564180365-70iae   1/1       Running            0          25s
nginx-deployment-1564180365-jbqqo   1/1       Running            0          25s
nginx-deployment-1564180365-hysrc   1/1       Running            0          25s
nginx-deployment-3066724191-08mng   0/1       ImagePullBackOff   0          6s
Note:
The Deployment controller stops the bad rollout automatically, and stops scaling up the new ReplicaSet. This depends on the rollingUpdate parameters (
specifically) that you have specified. Kubernetes by default sets the value to 25%.
Get the description of the Deployment:
kubectl describe deployment
Name:           nginx-deployment
CreationTimestamp:  Tue, 15 Mar 2016 14:48:04 -0700
Labels:         app=nginx
Selector:       app=nginx
Replicas:       3 desired | 1 updated | 4 total | 3 available | 1 unavailable
StrategyType:       RollingUpdate
MinReadySeconds:    0
   nginx:
    Image:        nginx:1.161
    Port:         80/TCP
    Host Port:    0/TCP
    Mounts:       &lt;none&gt;
  Volumes:        &lt;none&gt;
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:     nginx-deployment-1564180365 (3/3 replicas created)
NewReplicaSet:      nginx-deployment-3066724191 (1/1 replicas created)
Events:
  FirstSeen LastSeen    Count   From                    SubObjectPath   Type        Reason              Message
  --------- --------    -----   ----                    -------------   --------    ------              -------
  1m        1m          1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled up replica set nginx-deployment-2035384211 to 3
  22s       22s         1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled up replica set nginx-deployment-1564180365 to 1
  22s       22s         1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled down replica set nginx-deployment-2035384211 to 2
  22s       22s         1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled up replica set nginx-deployment-1564180365 to 2
  21s       21s         1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled down replica set nginx-deployment-2035384211 to 1
  21s       21s         1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled up replica set nginx-deployment-1564180365 to 3
  13s       13s         1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled down replica set nginx-deployment-20353842

11 to 0
  13s       13s         1       {deployment-controller }                Normal      ScalingReplicaSet   Scaled up replica set nginx-deployment-3066724191 to 1
To fix this, you need to rollback to a previous revision of Deployment that is stable.
Checking Rollout History of a Deployment
Follow the steps given below to check the rollout history:
First, check the revisions of this Deployment:
history
deployment/nginx-deployment
deployments "nginx-deployment"
REVISION    CHANGE-CAUSE
1           kubectl apply --filename=https://k8s.io/examples/controllers/nginx-deployment.yaml
2           kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
3           kubectl set image deployment/nginx-deployment nginx=nginx:1.161
CHANGE-CAUSE
is copied from the Deployment annotation
to its revisions upon creation. You can specify the
message by:
Annotating the Deployment with
kubectl annotate deployment/nginx-deployment kubernetes.io/change-cause="image updated to 1.16.1"
Manually editing the manifest of the resource.
To see the details of each revision, run:
history
deployment/nginx-deployment --revision
=
2
deployments "nginx-deployment" revision 2
  Labels:       app=nginx
          pod-template-hash=1159050644
  Annotations:  kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
   nginx:
    Image:      nginx:1.16.1
    Port:       80/TCP
     QoS Tier:
        cpu:      BestEffort
        memory:   BestEffort
    Environment Variables:      &lt;none&gt;
  No volumes.
Rolling Back to a Previous Revision
Follow the steps given below to rollback the Deployment from the current version to the previous version, which is version 2.
Now you've decided to undo the current rollout and rollback to the previous revision:
kubectl rollout undo deployment/nginx-deployment
deployment.apps/nginx-deployment rolled back
Alternatively, you can rollback to a specific revision by specifying it with
:
kubectl rollout undo deployment/nginx-deployment --to-revision
=
2
For more details about rollout related commands, read
.
The Deployment is now rolled back to a previous stable revision. As you can see, a
event
for rolling back to revision 2 is generated from Deployment controller.
Check if the rollback was successful and the Deployment is running as expected, run:
kubectl get deployment nginx-deployment
nginx-deployment   3/3     3            3           30m
kubectl describe deployment nginx-deployment
CreationTimestamp:      Sun, 02 Sep 2018 18:17:55 -0500
Annotations:            deployment.kubernetes.io/revision=4
                        kubernetes.io/change-cause=kubectl set image deployment/nginx-deployment nginx=nginx:1.16.1
   nginx:
    Image:        nginx:1.16.1
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  &lt;none&gt;
NewReplicaSet:   nginx-deployment-c4747d96c (3/3 repl

icas created)
Events:
  Type    Reason              Age   From                   Message
  ----    ------              ----  ----                   -------
  Normal  ScalingReplicaSet   12m   deployment-controller  Scaled up replica set nginx-deployment-75675f5897 to 3
  Normal  ScalingReplicaSet   11m   deployment-controller  Scaled up replica set nginx-deployment-c4747d96c to 1
  Normal  ScalingReplicaSet   11m   deployment-controller  Scaled down replica set nginx-deployment-75675f5897 to 2
  Normal  ScalingReplicaSet   11m   deployment-controller  Scaled up replica set nginx-deployment-c4747d96c to 2
  Normal  ScalingReplicaSet   11m   deployment-controller  Scaled down replica set nginx-deployment-75675f5897 to 1
  Normal  ScalingReplicaSet   11m   deployment-controller  Scaled up replica set nginx-deployment-c4747d96c to 3
  Normal  ScalingReplicaSet   11m   deployment-controller  Scaled down replica set nginx-deployment-75675f5897 to 0
  Normal  ScalingReplicaSet   11m   deployment-controller  Scaled up replica set nginx-deployment-595696685f to 1
  Normal  DeploymentRollback  15s   deployment-controller  Rolled back deployment "nginx-deployment" to revision 2
  Normal  ScalingReplicaSet   15s   deployment-controller  Scaled down replica set nginx-deployment-595696685f to 0
You can scale a Deployment by using the following command:
=
10
deployment.apps/nginx-deployment scaled
Assuming
horizontal Pod autoscaling
is enabled
in your cluster, you can set up an autoscaler for your Deployment and choose the minimum and maximum number of
Pods you want to run based on the CPU utilization of your existing Pods.
kubectl autoscale deployment/nginx-deployment --min
=
10
--max
=
15
=
80
Proportional scaling
RollingUpdate Deployments support running multiple versions of an application at the same time. When you
or an autoscaler scales a RollingUpdate Deployment that is in the middle of a rollout (either in progress
or paused), the Deployment controller balances the additional replicas in the existing active
ReplicaSets (ReplicaSets with Pods) in order to mitigate risk. This is called
proportional scaling
.
For example, you are running a Deployment with 10 replicas,
maxSurge
=3, and
=2.
Ensure that the 10 replicas in your Deployment are running.
kubectl get deploy
NAME                 DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment     10        10        10           10          50s
You update to a new image which happens to be unresolvable from inside the cluster.
kubectl
set
nginx
=
nginx:sometag
The image update starts a new rollout with ReplicaSet nginx-deployment-1989198191, but it's blocked due to the
requirement that you mentioned above. Check out the rollout status:
NAME                          DESIRED   CURRENT   READY     AGE
nginx-deployment-1989198191   5         5         0         9s
nginx-deployment-618515232    8         8         8         1m
Then a new scaling request for the Deployment comes along. The autoscaler increments the Deployment replicas
to 15. The Deployment controller needs to decide where to add these new 5 replicas. If you weren't using
proportional scaling, all 5 of them would be added in the new ReplicaSet. With proportional scaling, you
spread the additional replicas across all ReplicaSets. Bigger proportions go to the ReplicaSets with the
most replicas and lower proportions go to ReplicaSets with less replicas. Any leftovers are added to the
ReplicaSet with the most replicas. ReplicaSets with zero replicas are not scaled up.
In our example above, 3 replicas are added to the old

 ReplicaSet and 2 replicas are added to the
new ReplicaSet. The rollout process should eventually move all replicas to the new ReplicaSet, assuming
the new replicas become healthy. To confirm this, run:
nginx-deployment     15        18        7            8           7m
The rollout status confirms how the replicas were added to each ReplicaSet.
nginx-deployment-1989198191   7         7         0         7m
nginx-deployment-618515232    11        11        11        7m
Pausing and Resuming a rollout of a Deployment
When you update a Deployment, or plan to, you can pause rollouts
for that Deployment before you trigger one or more updates. When
you're ready to apply those changes, you resume rollouts for the
Deployment. This approach allows you to
apply multiple fixes in between pausing and resuming without triggering unnecessary rollouts.
For example, with a Deployment that was created:
Get the Deployment details:
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx     3         3         3            3           1m
Get the rollout status:
NAME               DESIRED   CURRENT   READY     AGE
nginx-2142116321   3         3         3         1m
Pause by running the following command:
kubectl rollout pause deployment/nginx-deployment
deployment.apps/nginx-deployment paused
Then update the image of the Deployment:
kubectl
set
nginx
=
Notice that no new rollout started:
history
deployments "nginx"
REVISION  CHANGE-CAUSE
1   &lt;none&gt;
Get the rollout status to verify that the existing ReplicaSet has not changed:
nginx-2142116321   3         3         3         2m
You can make as many updates as you wish, for example, update the resources that will be used:
kubectl
set
resources deployment/nginx-deployment -c
=
nginx --limits
=
cpu
=
200m,memory
=
512Mi
deployment.apps/nginx-deployment resource requirements updated
The initial state of the Deployment prior to pausing its rollout will continue its function, but new updates to
the Deployment will not have any effect as long as the Deployment rollout is paused.
Eventually, resume the Deployment rollout and observe a new ReplicaSet coming up with all the new updates:
kubectl rollout resume deployment/nginx-deployment
deployment.apps/nginx-deployment resumed
kubectl get rs -w
nginx-2142116321   2         2         2         2m
nginx-3926361531   2         2         0         6s
nginx-3926361531   2         2         1         18s
nginx-2142116321   1         2         2         2m
nginx-3926361531   3         2         1         18s
nginx-2142116321   1         1         1         2m
nginx-3926361531   3         3         1         18s
nginx-3926361531   3         3         2         19s
nginx-2142116321   0         1         1         2m
nginx-2142116321   0         0         0         2m
nginx-3926361531   3         3         3         20s
Get the status of the latest rollout:
nginx-214

2116321   0         0         0         2m
nginx-3926361531   3         3         3         28s
Note:
You cannot rollback a paused Deployment until you resume it.
Deployment status
A Deployment enters various states during its lifecycle. It can be
progressing
while
rolling out a new ReplicaSet, it can be
complete
, or it can
fail to progress
.
Progressing Deployment
Kubernetes marks a Deployment as
when one of the following tasks is performed:
The Deployment creates a new ReplicaSet.
The Deployment is scaling up its newest ReplicaSet.
The Deployment is scaling down its older ReplicaSet(s).
New Pods become ready or available (ready for at least
MinReadySeconds
).
When the rollout becomes “progressing”, the Deployment controller adds a condition with the following
attributes to the Deployment's
:
type: Progressing
status: "True"
reason: NewReplicaSetCreated
|
reason: FoundNewReplicaSet
|
reason: ReplicaSetUpdated
You can monitor the progress for a Deployment by using
.
Complete Deployment
complete
when it has the following characteristics:
All of the replicas associated with the Deployment have been updated to the latest version you've specified, meaning any
updates you've requested have been completed.
All of the replicas associated with the Deployment are available.
No old replicas for the Deployment are running.
When the rollout becomes “complete”, the Deployment controller sets a condition with the following
:
reason: NewReplicaSetAvailable
This
Progressing
condition will retain a status value of
"True"
until a new rollout
is initiated. The condition holds even when availability of replicas changes (which
does instead affect the
Available
condition).
You can check if a Deployment has completed by using
. If the rollout completed
successfully,
returns a zero exit code.
Waiting for rollout to finish: 2 of 3 updated replicas are available...
and the exit status from
is 0 (success):
echo
$?
0
Failed Deployment
Your Deployment may get stuck trying to deploy its newest ReplicaSet without ever completing. This can occur
due to some of the following factors:
Insufficient quota
Readiness probe failures
Image pull errors
Insufficient permissions
Application runtime misconfiguration
One way you can detect this condition is to specify a deadline parameter in your Deployment spec:
(
.spec.progressDeadlineSeconds
).
denotes the
number of seconds the Deployment controller waits before indicating (in the Deployment status) that the
Deployment progress has stalled.
kubectl
command sets the spec with
to make the controller report
lack of progress of a rollout for a Deployment after 10 minutes:
kubectl patch deployment/nginx-deployment -p
'{"spec":{"progressDeadlineSeconds":600}}'
deployment.apps/nginx-deployment patched
Once the deadline has been exceeded, the Deployment controller adds a DeploymentCondition with the following
:
status: "False"
reason: ProgressDeadlineExceeded
This condition can also fail early and is then set to status value of
"False"
due to reasons as
ReplicaSetCreateError
.
Also, the deadline is not taken into account anymore once the Deployment rollout completes.
See the
for more information on status conditions.
Note:
Kubernetes takes no action on a stalled Deployment other than to report a status condition with
. Higher level orchestrators can take advantage of it and act accordingly, for
example, rollback the Deployment t

o its previous version.
Note:
If you pause a Deployment rollout, Kubernetes does not check progress against your specified deadline.
You can safely pause a Deployment rollout in the middle of a rollout and resume without triggering
the condition for exceeding the deadline.
You may experience transient errors with your Deployments, either due to a low timeout that you have set or
due to any other kind of error that can be treated as transient. For example, let's suppose you have
insufficient quota. If you describe the Deployment you will notice the following section:
&lt;...&gt;
  Type            Status  Reason
  ----            ------  ------
  Available       True    MinimumReplicasAvailable
  Progressing     True    ReplicaSetUpdated
  ReplicaFailure  True    FailedCreate
If you run
, the Deployment status is similar to this:
status:
  availableReplicas: 2
  conditions:
  - lastTransitionTime: 2016-10-04T12:25:39Z
    lastUpdateTime: 2016-10-04T12:25:39Z
    message: Replica set "nginx-deployment-4262182780" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  - lastTransitionTime: 2016-10-04T12:25:42Z
    lastUpdateTime: 2016-10-04T12:25:42Z
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    type: Available
    message: 'Error creating: pods "nginx-deployment-4262182780-" is forbidden: exceeded quota:
      object-counts, requested: pods=1, used: pods=3, limited: pods=2'
    reason: FailedCreate
    type: ReplicaFailure
  observedGeneration: 3
  replicas: 2
  unavailableReplicas: 2
Eventually, once the Deployment progress deadline is exceeded, Kubernetes updates the status and the
reason for the Progressing condition:
  Progressing     False   ProgressDeadlineExceeded
You can address an issue of insufficient quota by scaling down your Deployment, by scaling down other
controllers you may be running, or by increasing quota in your namespace. If you satisfy the quota
conditions and the Deployment controller then completes the Deployment rollout, you'll see the
Deployment's status update with a successful condition (
and
).
type: Available
with
means that your Deployment has minimum availability. Minimum availability is dictated
by the parameters specified in the deployment strategy.
with
means that your Deployment
is either in the middle of a rollout and it is progressing or that it has successfully completed its progress and the minimum
required new replicas are available (see the Reason of the condition for the particulars - in our case
means that the Deployment is complete).
You can check if a Deployment has failed to progress by using
.
returns a non-zero exit code if the Deployment has exceeded the progression deadline.
error: deployment "nginx" exceeded its progress deadline
is 1 (indicating an error):
echo
$?
1
Operating on a failed deployment
All actions that apply to a complete Deployment also apply to a failed Deployment. You can scale it up/down, roll back
to a previous revision,

 or even pause it if you need to apply multiple tweaks in the Deployment Pod template.
Clean up Policy
You can set
.spec.revisionHistoryLimit
field in a Deployment to specify how many old ReplicaSets for
this Deployment you want to retain. The rest will be garbage-collected in the background. By default,
it is 10.
Note:
Explicitly setting this field to 0, will result in cleaning up all the history of your Deployment
thus that Deployment will not be able to roll back.
Canary Deployment
If you want to roll out releases to a subset of users or servers using the Deployment, you
can create multiple Deployments, one for each release, following the canary pattern described in
.
As with all other Kubernetes configs, a Deployment needs
.apiVersion
,
.kind
, and
.metadata
fields.
For general information about working with config files, see
deploying applications
,
configuring containers, and
When the control plane creates new Pods for a Deployment, the
of the
Deployment is part of the basis for naming those Pods. The name of a Deployment must be a valid
value, but this can produce unexpected results for the Pod hostnames. For best compatibility,
DNS label
.
A Deployment also needs a
.spec
section
.
Pod Template
The
and
are the only required fields of the
.spec
.
The
is a
Pod template
. It has exactly the same schema as a
Pod
, except it is nested and does not have an
or
kind
.
In addition to required fields for a Pod, a Pod template in a Deployment must specify appropriate
labels and an appropriate restart policy. For labels, make sure not to overlap with other controllers. See
selector
.
Only a
.spec.template.spec.restartPolicy
equal to
Always
is
allowed, which is the default if not specified.
Replicas
is an optional field that specifies the number of desired Pods. It defaults to 1.
Should you manually scale a Deployment, example via
kubectl scale deployment deployment --replicas=X
, and then you update that Deployment based on a manifest
(for example: by running
),
then applying that manifest overwrites the manual scaling that you previously did.
If a
(or any
similar API for horizontal scaling) is managing scaling for a Deployment, don't set
.
Instead, allow the Kubernetes
to manage the
field automatically.
Selector
is a required field that specifies a
for the Pods targeted by this Deployment.
must match
.spec.template.metadata.labels
, or it will be rejected by the API.
apps/v1
,
and
do not default to
if not set. So they must be set explicitly. Also note that
is immutable after creation of the Deployment in
apps/v1
.
A Deployment may terminate Pods whose labels match the selector if their template is different
from
or if the total number of such Pods exceeds
. It brings up new
Pods with
if the number of Pods is less than the desired number.
Note:
You should not create other Pods whose labels match this selector, either directly, by creating
another Deployment, or by creating another controller such as a ReplicaSet or a ReplicationController. If you
do so, the first Deployment thinks that it created these other Pods. Kubernetes does not stop you from doing this.
If you have multiple controllers that have overlapping selectors, the controllers will fight with each
other and won't behave correctly.
Strategy
.spec.strategy
specifies the strategy used to replace old Pods by new ones.
.spec.strategy.type
can be "Recreate" or "RollingUpdate". "RollingUpdate" is
the default value.
Recreate Deployment
All existing Pods are

 killed before new ones are created when
.spec.strategy.type==Recreate
.
Note:
This will only guarantee Pod termination previous to creation for upgrades. If you upgrade a Deployment, all Pods
of the old revision will be terminated immediately. Successful removal is awaited before any Pod of the new
revision is created. If you manually delete a Pod, the lifecycle is controlled by the ReplicaSet and the
replacement will be created immediately (even if the old Pod is still in a Terminating state). If you need an
"at most" guarantee for your Pods, you should consider using a
.
Rolling Update Deployment
The Deployment updates Pods in a rolling update
fashion when
.spec.strategy.type==RollingUpdate
. You can specify
and
maxSurge
to control
the rolling update process.
Max Unavailable
.spec.strategy.rollingUpdate.maxUnavailable
is an optional field that specifies the maximum number
of Pods that can be unavailable during the update process. The value can be an absolute number (for example, 5)
or a percentage of desired Pods (for example, 10%). The absolute number is calculated from percentage by
rounding down. The value cannot be 0 if
.spec.strategy.rollingUpdate.maxSurge
is 0. The default value is 25%.
For example, when this value is set to 30%, the old ReplicaSet can be scaled down to 70% of desired
Pods immediately when the rolling update starts. Once new Pods are ready, old ReplicaSet can be scaled
down further, followed by scaling up the new ReplicaSet, ensuring that the total number of Pods available
at all times during the update is at least 70% of the desired Pods.
Max Surge
is an optional field that specifies the maximum number of Pods
that can be created over the desired number of Pods. The value can be an absolute number (for example, 5) or a
percentage of desired Pods (for example, 10%). The value cannot be 0 if
MaxUnavailable
is 0. The absolute number
is calculated from the percentage by rounding up. The default value is 25%.
For example, when this value is set to 30%, the new ReplicaSet can be scaled up immediately when the
rolling update starts, such that the total number of old and new Pods does not exceed 130% of desired
Pods. Once old Pods have been killed, the new ReplicaSet can be scaled up further, ensuring that the
total number of Pods running at any time during the update is at most 130% of desired Pods.
Progress Deadline Seconds
is an optional field that specifies the number of seconds you want
to wait for your Deployment to progress before the system reports back that the Deployment has
failed progressing
- surfaced as a condition with
,
.
and
in the status of the resource. The Deployment controller will keep
retrying the Deployment. This defaults to 600. In the future, once automatic rollback will be implemented, the Deployment
controller will roll back a Deployment as soon as it observes such a condition.
If specified, this field needs to be greater than
.
Min Ready Seconds
is an optional field that specifies the minimum number of seconds for which a newly
created Pod should be ready without any of its containers crashing, for it to be considered available.
This defaults to 0 (the Pod will be considered available as soon as it is ready). To learn more about when
a Pod is considered ready, see
.
Revision History Limit
A Deployment's revision history is stored in the ReplicaSets it controls.
is an optional field that specifies the number of old ReplicaSets to retain
to allow rollback. These old ReplicaSets consume resources in
etcd
and crowd the output of
. The configuration of each Deployment revision is stored in its ReplicaSets; therefore, once an old ReplicaSet is deleted, you lose the ability to rollback to that revision of Deployment. By defaul

t, 10 old ReplicaSets will be kept, however its ideal value depends on the frequency and stability of new Deployments.
More specifically, setting this field to zero means that all old ReplicaSets with 0 replicas will be cleaned up.
In this case, a new Deployment rollout cannot be undone, since its revision history is cleaned up.
Paused
.spec.paused
is an optional boolean field for pausing and resuming a Deployment. The only difference between
a paused Deployment and one that is not paused, is that any changes into the PodTemplateSpec of the paused
Deployment will not trigger new rollouts as long as it is paused. A Deployment is not paused by default when
it is created.
Pods
.
.
is a top-level resource in the Kubernetes REST API.
Read the
object definition to understand the API for deployments.
Read about
and how
you can use it to manage application availability during disruptions.
5.2.2 - ReplicaSet
A ReplicaSet's purpose is to maintain a stable set of replica Pods running at any given time. As such, it is often
used to guarantee the availability of a specified number of identical Pods.
How a ReplicaSet works
A ReplicaSet is defined with fields, including a selector that specifies how to identify Pods it can acquire, a number
of replicas indicating how many Pods it should be maintaining, and a pod template specifying the data of new Pods
it should create to meet the number of replicas criteria. A ReplicaSet then fulfills its purpose by creating
and deleting Pods as needed to reach the desired number. When a ReplicaSet needs to create new Pods, it uses its Pod
template.
A ReplicaSet is linked to its Pods via the Pods'
field, which specifies what resource the current object is owned by. All Pods acquired by a ReplicaSet have their owning
ReplicaSet's identifying information within their ownerReferences field. It's through this link that the ReplicaSet
knows of the state of the Pods it is maintaining and plans accordingly.
A ReplicaSet identifies new Pods to acquire by using its selector. If there is a Pod that has no
OwnerReference or the OwnerReference is not a
and it
matches a ReplicaSet's selector, it will be immediately acquired by said ReplicaSet.
When to use a ReplicaSet
A ReplicaSet ensures that a specified number of pod replicas are running at any given
time. However, a Deployment is a higher-level concept that manages ReplicaSets and
provides declarative updates to Pods along with a lot of other useful features.
Therefore, we recommend using Deployments instead of directly using ReplicaSets, unless
you require custom update orchestration or don't require updates at all.
This actually means that you may never need to manipulate ReplicaSet objects:
use a Deployment instead, and define your application in the spec section.
Example
controllers/frontend.yaml
:
apps/v1
kind
:
metadata
:
name
:
frontend
labels
:
app
:
guestbook
tier
:
frontend
spec
:
# modify replicas according to your case
replicas
:
3
selector
:
:
tier
:
frontend
template
:
metadata
:
labels
:
tier
:
frontend
spec
:
:
-
name
:
php-redis
image
:
gcr.io/google_samples/gb-frontend:v3
Saving this manifest into
frontend.yaml
and submitting it to a Kubernetes cluster will
create the defined ReplicaSet and the Pods that it manages.
kubectl apply -f https://kubernetes.io/examples/controllers/frontend.yaml
You can then get the current ReplicaSets deployed:
And see the frontend one you created:
NAME       DESIRED   CURRENT   READY   AGE
frontend
3
3
3
6s
You can also check on the state of the ReplicaSet:
kubectl describe rs/frontend
And you will see output similar to:
Name:         frontend
Selector:     tier=frontend
Labels:       app=guestbook
              tier=frontend
     
           {"apiVersion":"apps/v1","kind":"ReplicaSet","metadata":{"annotations":{},"labels":{"app":"guestbook","tier":"frontend"},"name":"frontend",...
Replicas:     3 current / 3 desired
Pods Status:  3 Running / 0 Waiting / 0 Succeeded / 0 Failed
  Labels:  tier=frontend
   php-redis:
    Image:        gcr.io/google_samples/gb-frontend:v3
    Port:         &lt;none&gt;
    Host Port:    &lt;none&gt;
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  117s  replicaset-controller  Created pod: frontend-wtsmm
  Normal  SuccessfulCreate  116s  replicaset-controller  Created pod: frontend-b2zdv
  Normal  SuccessfulCreate  116s  replicaset-controller  Created pod: frontend-vcmts
And lastly you can check for the Pods brought up:
You should see Pod information similar to:
NAME             READY   STATUS    RESTARTS   AGE
frontend-b2zdv   1/1     Running
0
6m36s
frontend-vcmts   1/1     Running
0
6m36s
frontend-wtsmm   1/1     Running
0
6m36s
You can also verify that the owner reference of these pods is set to the frontend ReplicaSet.
To do this, get the yaml of one of the Pods running:
kubectl get pods frontend-b2zdv -o yaml
The output will look similar to this, with the frontend ReplicaSet's info set in the metadata's ownerReferences field:
:
v1
kind
:
Pod
metadata
:
:
"2020-02-12T07:06:16Z"
:
frontend-
labels
:
tier
:
frontend
name
:
frontend-b2zdv
namespace
:
default
:
-
:
apps/v1
:
true
:
true
kind
:
name
:
frontend
uid
:
f391f6db-bb9b-4c09-ae74-6a1f77f3d5cf
...
Non-Template Pod acquisitions
While you can create bare Pods with no problems, it is strongly recommended to make sure that the bare Pods do not have
labels which match the selector of one of your ReplicaSets. The reason for this is because a ReplicaSet is not limited
to owning Pods specified by its template-- it can acquire other Pods in the manner specified in the previous sections.
Take the previous frontend ReplicaSet example, and the Pods specified in the following manifest:
pods/pod-rs.yaml
:
v1
kind
:
Pod
metadata
:
name
:
pod1
labels
:
tier
:
frontend
spec
:
:
-
name
:
hello1
image
:
gcr.io/google-samples/hello-app:2.0
---
:
v1
kind
:
Pod
metadata
:
name
:
pod2
labels
:
tier
:
frontend
spec
:
:
-
name
:
hello2
image
:
gcr.io/google-samples/hello-app:1.0
As those Pods do not have a Controller (or any object) as their owner reference and match the selector of the frontend
ReplicaSet, they will immediately be acquired by it.
Suppose you create the Pods after the frontend ReplicaSet has been deployed and has set up its initial Pod replicas to
fulfill its replica count requirement:
kubectl apply -f https://kubernetes.io/examples/pods/pod-rs.yaml
The new Pods will be acquired by the ReplicaSet, and then immediately terminated as the ReplicaSet would be over
its desired count.
Fetching the Pods:
The output shows that the new Pods are either already terminated, or in the process of being terminated:
NAME             READY   STATUS        RESTARTS   AGE
frontend-b2zdv   1/1     Running       0          10m
frontend-vcmts   1/1     Running       0          10m
frontend-wtsmm   1/1     Running       0          10m
pod1             0/1     Terminating   0          1s
pod2             0/1     Terminating   0          1s
If you create the Pods first:
And then create the ReplicaSet however:
You shall see that the ReplicaSet has acquired the Pods and has only created new ones according to its spec until the
number of its new P

ods and the original matches its desired count. As fetching the Pods:
Will reveal in its output:
frontend-hmmj2   1/1     Running   0          9s
pod1             1/1     Running   0          36s
pod2             1/1     Running   0          36s
In this manner, a ReplicaSet can own a non-homogenous set of Pods
Writing a ReplicaSet manifest
As with all other Kubernetes API objects, a ReplicaSet needs the
,
kind
, and
metadata
fields.
For ReplicaSets, the
kind
is always a ReplicaSet.
When the control plane creates new Pods for a ReplicaSet, the
of the
ReplicaSet is part of the basis for naming those Pods. The name of a ReplicaSet must be a valid
DNS label
.
A ReplicaSet also needs a
.spec
section
.
The
is a
which is also
required to have labels in place. In our
example we had one label:
tier: frontend
.
Be careful not to overlap with the selectors of other controllers, lest they try to adopt this Pod.
For the template's
field,
, the only allowed value is
Always
, which is the default.
Pod Selector
The
field is a
. As discussed
earlier
these are the labels used to identify potential Pods to acquire. In our
example, the selector was:
:
tier
:
frontend
In the ReplicaSet,
must match
, or it will
be rejected by the API.
Note:
For 2 ReplicaSets specifying the same
but different
and
.spec.template.spec
fields, each ReplicaSet ignores the
Pods created by the other ReplicaSet.
Replicas
You can specify how many Pods should run concurrently by setting
. The ReplicaSet will create/delete
its Pods to match this number.
If you do not specify
, then it defaults to 1.
Working with ReplicaSets
Deleting a ReplicaSet and its Pods
To delete a ReplicaSet and all of its Pods, use
. The
Garbage collector
automatically deletes all of
the dependent Pods by default.
When using the REST API or the
client-go
library, you must set
to
or
Foreground
in the
-d
option. For example:
=
8080
curl -X DELETE
'localhost:8080/apis/apps/v1/namespaces/default/replicasets/frontend'
\
&gt; -d
\
&gt; -H
Deleting just a ReplicaSet
You can delete a ReplicaSet without affecting any of its Pods using
with the
option.
client-go
to
Orphan
.
=
8080
\
&gt; -d
\
&gt; -H
Once the original is deleted, you can create a new ReplicaSet to replace it. As long
as the old and new
are the same, then the new one will adopt the old Pods.
However, it will not make any effort to make existing Pods match a new, different pod template.
To update Pods to a new spec in a controlled way, use a
, as
ReplicaSets do not support a rolling update directly.
Isolating Pods from a ReplicaSet
You can remove Pods from a ReplicaSet by changing their labels. This technique may be used to remove Pods
from service for debugging, data recovery, etc. Pods that are removed in this way will be replaced automatically (
assuming that the number of replicas is not also changed).
Scaling a ReplicaSet
A ReplicaSet can be easily scaled up or down by simply updating the
.spec

.replicas
field. The ReplicaSet controller
ensures that a desired number of Pods with a matching label selector are available and operational.
When scaling down, the ReplicaSet controller chooses which pods to delete by sorting the available pods to
prioritize scaling down pods based on the following general algorithm:
Pending (and unschedulable) pods are scaled down first
If
annotation is set, then
the pod with the lower value will come first.
Pods on nodes with more replicas come before pods on nodes with fewer replicas.
If the pods' creation times differ, the pod that was created more recently
comes before the older pod (the creation times are bucketed on an integer log scale
when the
If all of the above match, then selection is random.
Pod deletion cost
Using the
annotation, users can set a preference regarding which pods to remove first when downscaling a ReplicaSet.
The annotation should be set on the pod, the range is [-2147483647, 2147483647]. It represents the cost of
deleting a pod compared to other pods belonging to the same ReplicaSet. Pods with lower deletion
cost are preferred to be deleted before pods with higher deletion cost.
The implicit value for this annotation for pods that don't set it is 0; negative values are permitted.
Invalid values will be rejected by the API server.
This feature is beta and enabled by default. You can disable it using the
in both kube-apiserver and kube-controller-manager.
Note:
This is honored on a best-effort basis, so it does not offer any guarantees on pod deletion order.
Users should avoid updating the annotation frequently, such as updating it based on a metric value,
because doing so will generate a significant number of pod updates on the apiserver.
Example Use Case
The different pods of an application could have different utilization levels. On scale down, the application
may prefer to remove the pods with lower utilization. To avoid frequently updating the pods, the application
should update
once before issuing a scale down (setting the
annotation to a value proportional to pod utilization level). This works if the application itself controls
the down scaling; for example, the driver pod of a Spark deployment.
ReplicaSet as a Horizontal Pod Autoscaler Target
A ReplicaSet can also be a target for
Horizontal Pod Autoscalers (HPA)
. That is,
a ReplicaSet can be auto-scaled by an HPA. Here is an example HPA targeting
the ReplicaSet we created in the previous example.
controllers/hpa-rs.yaml
:
kind
:
metadata
:
name
:
frontend-scaler
spec
:
:
kind
:
name
:
frontend
:
3
:
10
:
50
hpa-rs.yaml
and submitting it to a Kubernetes cluster should
create the defined HPA that autoscales the target ReplicaSet depending on the CPU usage
of the replicated Pods.
kubectl apply -f https://k8s.io/examples/controllers/hpa-rs.yaml
command to accomplish the same
(and it's easier!)
kubectl autoscale rs frontend --max
=
10
--min
=
3
=
50
Alternatives to ReplicaSet
Deployment (recommended)
is an object which can own ReplicaSets and update
them and their Pods via declarative, server-side rolling updates.
While ReplicaSets can be used independently, today they're mainly used by Deployments as a mechanism to orchestrate Pod
creation, deletion and updates. When you use Deployments you don't have to worry about managing the ReplicaSets that
they create. Deployments own and manage their ReplicaSets.
As such, it is recommended to use Deployments when you want ReplicaSets.
Bare Pods
Unlike the case where a user directly created Pods, a Replica

Set replaces Pods that are deleted or
terminated for any reason, such as in the case of node failure or disruptive node maintenance,
such as a kernel upgrade. For this reason, we recommend that you use a ReplicaSet even if your
application requires only a single Pod. Think of it similarly to a process supervisor, only it
supervises multiple Pods across multiple nodes instead of individual processes on a single node. A
ReplicaSet delegates local container restarts to some agent on the node such as Kubelet.
Job
Use a
Job
instead of a ReplicaSet for Pods that are
expected to terminate on their own (that is, batch jobs).
DaemonSet
Use a
DaemonSet
instead of a ReplicaSet for Pods that provide a
machine-level function, such as machine monitoring or machine logging. These Pods have a lifetime that is tied
to a machine lifetime: the Pod needs to be running on the machine before other Pods start, and are
safe to terminate when the machine is otherwise ready to be rebooted/shutdown.
ReplicaSets are the successors to
.
The two serve the same purpose, and behave similarly, except that a ReplicationController does not support set-based
selector requirements as described in the
labels user guide
.
As such, ReplicaSets are preferred over ReplicationControllers
Pods
.
.
,
which relies on ReplicaSets to work.
Read the
object definition to understand the API for replica sets.
Read about
and how
5.2.3 - StatefulSets
StatefulSet is the workload API object used to manage stateful applications.
Pods
,
Like a
Using StatefulSets
StatefulSets are valuable for applications that require one or more of the
following.
Stable, unique network identifiers.
Stable, persistent storage.
Ordered, graceful deployment and scaling.
Ordered, automated rolling updates.
In the above, stable is synonymous with persistence across Pod (re)scheduling.
If an application doesn't require any stable identifiers or ordered deployment,
deletion, or scaling, you should deploy your application using a workload object
that provides a set of stateless replicas.
or
may be better suited to your stateless needs.
The storage for a given Pod must either be provisioned by a
PersistentVolume Provisioner
based on the requested
storage class
, or pre-provisioned by an admin.
Deleting and/or scaling a StatefulSet down will
not
delete the volumes associated with the
StatefulSet. This is done to ensure data safety, which is generally more valuable than an
automatic purge of all related StatefulSet resources.
StatefulSets currently require a
to be responsible for the network identity of the Pods. You are responsible for creating this
Service.
StatefulSets do not provide any guarantees on the termination of pods when a StatefulSet is
deleted. To achieve ordered and graceful termination of the pods in the StatefulSet, it is
possible to scale the StatefulSet down to 0 prior to deletion.
When using
Rol

ling Updates
with the default
(
),
it's possible to get into a broken state that requires
manual intervention to repair
.
The example below demonstrates the components of a StatefulSet.
:
v1
kind
:
Service
metadata
:
name
:
nginx
labels
:
app
:
nginx
spec
:
ports
:
-
port
:
80
name
:
web
clusterIP
:
None
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
web
spec
:
selector
:
:
app
:
nginx
# has to match .spec.template.metadata.labels
:
"nginx"
replicas
:
3
# by default is 1
:
10
# by default is 0
template
:
metadata
:
labels
:
app
:
nginx
# has to match .spec.selector.matchLabels
spec
:
:
10
:
-
name
:
nginx
image
:
ports
:
-
:
80
name
:
web
:
-
name
:
www
mountPath
:
:
-
metadata
:
name
:
www
spec
:
:
[
]
:
"my-storage-class"
resources
:
requests
:
storage
:
1Gi
In the above example:
A Headless Service, named
nginx
, is used to control the network domain.
The StatefulSet, named
web
, has a Spec that indicates that 3 replicas of the nginx container will be launched in unique Pods.
The
will provide stable storage using
provisioned by a
PersistentVolume Provisioner.
The name of a StatefulSet object must be a valid
DNS label
.
You must set the
field of a StatefulSet to match the labels of its
. Failing to specify a matching Pod Selector will result in a
validation error during StatefulSet creation.
Volume Claim Templates
.spec.volumeClaimTemplates
which can provide stable storage using
provisioned by a PersistentVolume
Provisioner.
Minimum ready seconds
created Pod should be running and ready without any of its containers crashing, for it to be considered available.
This is used to check progression of a rollout when using a
strategy.
This field defaults to 0 (the Pod will be considered available as soon as it is ready). To learn more about when
.
Pod Identity
StatefulSet Pods have a unique identity that consists of an ordinal, a
stable network identity, and stable storage. The identity sticks to the Pod,
regardless of which node it's (re)scheduled on.
Ordinal Index
For a StatefulSet with N
replicas
, each Pod in the StatefulSet
will be assigned an integer ordinal, that is unique over the Set. By default,
pods will be assigned ordinals from 0 up through N-1.
.spec.ordinals
is an optional field that allows you to configure the integer
ordinals assigned to each Pod. It defaults to nil. You must enable the
to
use this field. Once enabled, you can configure the following options:
.spec.ordinals.start
: If the
field is set, Pods will
be assigned ordinals from
up through
.spec.ordinals.start + .spec.replicas - 1
.
Stable Network ID
Each Pod in a StatefulSet derives its hostname from the name of the StatefulSet
and the ordinal of the Pod. The pattern for the constructed hostname
is
$(statefulset name)-$(ordinal)
. The example above will create three Pods
named
web-0,web-1,web-2
.
A StatefulSet can use a
to control the domain of its Pods. The domain managed by this Service takes the form:
$(service name).$(namespace).svc.cluster.local
, where "cluster.local" is the
cluster domain.
As each Pod is created, it gets a matching DNS subdomain, taking the form:
$(podname).$(governing service domain)
, where the governing service is defined
by the
se

rviceName
field on the StatefulSet.
Depending on how DNS is configured in your cluster, you may not be able to look up the DNS
name for a newly-run Pod immediately. This behavior can occur when other clients in the
cluster have already sent queries for the hostname of the Pod before it was created.
Negative caching (normal in DNS) means that the results of previous failed lookups are
remembered and reused, even after the Pod is running, for at least a few seconds.
If you need to discover Pods promptly after they are created, you have a few options:
Query the Kubernetes API directly (for example, using a watch) rather than relying on DNS lookups.
Decrease the time of caching in your Kubernetes DNS provider (typically this means editing the
config map for CoreDNS, which currently caches for 30 seconds).
limitations
section, you are responsible for
creating the
responsible for the network identity of the pods.
Here are some examples of choices for Cluster Domain, Service name,
StatefulSet name, and how that affects the DNS names for the StatefulSet's Pods.
Cluster Domain
Service (ns/name)
StatefulSet (ns/name)
StatefulSet Domain
Pod DNS
Pod Hostname
default/nginx
default/web
web-{0..N-1}.nginx.default.svc.cluster.local
web-{0..N-1}
foo/nginx
foo/web
nginx.foo.svc.cluster.local
web-{0..N-1}.nginx.foo.svc.cluster.local
kube.local
foo/nginx
foo/web
nginx.foo.svc.kube.local
web-{0..N-1}.nginx.foo.svc.kube.local
Note:
Cluster Domain will be set to
unless
otherwise configured
.
Stable Storage
For each VolumeClaimTemplate entry defined in a StatefulSet, each Pod receives one
PersistentVolumeClaim. In the nginx example above, each Pod receives a single PersistentVolume
with a StorageClass of
my-storage-class
and 1 Gib of provisioned storage. If no StorageClass
is specified, then the default StorageClass will be used. When a Pod is (re)scheduled
onto a node, its
mount the PersistentVolumes associated with its
PersistentVolume Claims. Note that, the PersistentVolumes associated with the
Pods' PersistentVolume Claims are not deleted when the Pods, or StatefulSet are deleted.
This must be done manually.
When the StatefulSet
creates a Pod,
it adds a label,
, that is set to the name of
the Pod. This label allows you to attach a Service to a specific Pod in
the StatefulSet.
Deployment and Scaling Guarantees
For a StatefulSet with N replicas, when Pods are being deployed, they are created sequentially, in order from {0..N-1}.
When Pods are being deleted, they are terminated in reverse order, from {N-1..0}.
Before a scaling operation is applied to a Pod, all of its predecessors must be Running and Ready.
Before a Pod is terminated, all of its successors must be completely shutdown.
The StatefulSet should not specify a
pod.Spec.TerminationGracePeriodSeconds
of 0. This practice
is unsafe and strongly discouraged. For further explanation, please refer to
.
When the nginx example above is created, three Pods will be deployed in the order
web-0, web-1, web-2. web-1 will not be deployed before web-0 is
Running and Ready
, and web-2 will not be deployed until
web-1 is Running and Ready. If web-0 should fail, after web-1 is Running and Ready, but before
web-2 is launched, web-2 will not be launched until web-0 is successfully relaunched and
becomes Running and Ready.
If a user were to scale the deployed example by patching the StatefulSet such that
replicas=1
, web-2 would be terminated first. web-1 would not be terminated until web-2
is fully shutdown and deleted. If web-0 were to fail after web-2 has been terminated and
is completely shutdown, but prior to web-1's termination, web-1 would not be terminated
until web-0 is Running and Ready.
Pod Management Policies
StatefulSet allows you to relax its ordering guaran

tees while
preserving its uniqueness and identity guarantees via its
field.
pod management is the default for StatefulSets. It implements the behavior
described
above
.
Parallel
terminate all Pods in parallel, and to not wait for Pods to become Running
Pod. This option only affects the behavior for scaling operations. Updates are not
affected.
Update strategies
A StatefulSet's
.spec.updateStrategy
field allows you to configure
and disable automated rolling updates for containers, labels, resource request/limits, and
annotations for the Pods in a StatefulSet. There are two possible values:
OnDelete
When a StatefulSet's
is set to
OnDelete
,
the StatefulSet controller will not automatically update the Pods in a
StatefulSet. Users must manually delete Pods to cause the controller to
create new Pods that reflect modifications made to a StatefulSet's
.
The
update strategy implements automated, rolling updates for the Pods in a
StatefulSet. This is the default update strategy.
Rolling Updates
is set to
, the
StatefulSet controller will delete and recreate each Pod in the StatefulSet. It will proceed
in the same order as Pod termination (from the largest ordinal to the smallest), updating
each Pod one at a time.
The Kubernetes control plane waits until an updated Pod is Running and Ready prior
to updating its predecessor. If you have set
(see
Minimum Ready Seconds
), the control plane additionally waits that
amount of time after the Pod turns ready, before moving on.
Partitioned rolling updates
The
update strategy can be partitioned, by specifying a
.spec.updateStrategy.rollingUpdate.partition
. If a partition is specified, all Pods with an
ordinal that is greater than or equal to the partition will be updated when the StatefulSet's
is updated. All Pods with an ordinal that is less than the partition will not
be updated, and, even if they are deleted, they will be recreated at the previous version. If a
is greater than its
,
updates to its
will not be propagated to its Pods.
In most cases you will not need to use a partition, but they are useful if you want to stage an
update, roll out a canary, or perform a phased roll out.
Maximum unavailable Pods
You can control the maximum number of Pods that can be unavailable during an update
by specifying the
field.
The value can be an absolute number (for example,
5
) or a percentage of desired
Pods (for example,
10%
). Absolute number is calculated from the percentage value
by rounding it up. This field cannot be 0. The default setting is 1.
This field applies to all Pods in the range
0
to
replicas - 1
. If there is any
unavailable Pod in the range
0
to
, it will be counted towards
.
Note:
The
field is in Alpha stage and it is honored only by API servers
that are running with the
enabled.
Forced rollback
When using
(
),
it's possible to get into a broken state that requires manual intervention to repair.
If you update the Pod template to a configuration that never becomes Running and
Ready (for example, due to a bad binary or application-level configuration error),
StatefulSet will stop the rollout and wait.
In this state, it's not enough to revert the Pod template to a good configuration.
Due to a
known issue
,
StatefulSet will continue to wait for the broken Pod to 

become Ready
(which never happens) before it will attempt to revert it back to the working
After reverting the template, you must also delete any Pods that StatefulSet had
already attempted to run with the bad configuration.
StatefulSet will then begin to recreate the Pods using the reverted template.
PersistentVolumeClaim retention
The optional
.spec.persistentVolumeClaimRetentionPolicy
field controls if
and how PVCs are deleted during the lifecycle of a StatefulSet. You must enable the
on the API server and the controller manager to use this field.
Once enabled, there are two policies you can configure for each StatefulSet:
whenDeleted
configures the volume retention behavior that applies when the StatefulSet is deleted
whenScaled
configures the volume retention behavior that applies when the replica count of
the StatefulSet is reduced; for example, when scaling down the set.
For each policy that you can configure, you can set the value to either
Delete
or
Retain
.
Delete
The PVCs created from the StatefulSet
are deleted for each Pod
affected by the policy. With the
policy all PVCs from the
are deleted after their Pods have been deleted. With the
policy, only PVCs corresponding to Pod replicas being scaled down are
deleted, after their Pods have been deleted.
Retain
(default)
PVCs from the
are not affected when their Pod is
deleted. This is the behavior before this new feature.
Bear in mind that these policies
only
apply when Pods are being removed due to the
StatefulSet being deleted or scaled down. For example, if a Pod associated with a StatefulSet
fails due to node failure, and the control plane creates a replacement Pod, the StatefulSet
retains the existing PVC. The existing volume is unaffected, and the cluster will attach it to
the node where the new Pod is about to launch.
The default for policies is
Retain
, matching the StatefulSet behavior before this new feature.
Here is an example policy.
:
apps/v1
kind
:
...
spec
:
:
:
Retain
:
Delete
...
The StatefulSet
adds
to its PVCs, which are then deleted by the
garbage collector
after the Pod is terminated. This enables the Pod to
cleanly unmount all volumes before the PVCs are deleted (and before the backing PV and
volume are deleted, depending on the retain policy). When you set the
policy to
Delete
, an owner reference to the StatefulSet instance is placed on all PVCs
associated with that StatefulSet.
The
policy must delete PVCs only when a Pod is scaled down, and not when a
Pod is deleted for another reason. When reconciling, the StatefulSet controller compares
its desired replica count to the actual Pods present on the cluster. Any StatefulSet Pod
whose id greater than the replica count is condemned and marked for deletion. If the
policy is
Delete
, the condemned Pods are first set as owners to the
associated StatefulSet template PVCs, before the Pod is deleted. This causes the PVCs
to be garbage collected after only the condemned Pods have terminated.
This means that if the controller crashes and restarts, no Pod will be deleted before its
owner reference has been updated appropriate to the policy. If a condemned Pod is
force-deleted while the controller is down, the owner reference may or may not have been
set up, depending on when the controller crashed. It may take several reconcile loops to
update the owner references, so some condemned Pods may have set up owner references and
others may not. For this reason we recommend waiting for the controller to come back up,
which will verify owner references before terminating Pods. If that is not possible, the
operator should verify the owner references on PVCs to ensure the expected objects are
deleted 

when Pods are force-deleted.
Replicas
Should you manually scale a deployment, example via
kubectl scale statefulset statefulset --replicas=X
, and then you update that StatefulSet
based on a manifest (for example: by running
kubectl apply -f statefulset.yaml
), then applying that manifest overwrites the manual scaling
that you previously did.
If a
(or any similar API for horizontal scaling) is managing scaling for a
Statefulset, don't set
. Instead, allow the Kubernetes
to manage
the
Pods
.
Find out how to use StatefulSets
Follow an example of
deploying a stateful application
.
deploying Cassandra with Stateful Sets
.
running a replicated stateful application
.
scale a StatefulSet
.
Learn what's involved when you
.
configure a Pod to use a volume for storage
.
configure a Pod to use a PersistentVolume for storage
.
Read the
object definition to understand the API for stateful sets.
Read about
and how
5.2.4 - DaemonSet
A
DaemonSet
ensures that all (or some) Nodes run a copy of a Pod. As nodes are added to the
cluster, Pods are added to them. As nodes are removed from the cluster, those Pods are garbage
collected. Deleting a DaemonSet will clean up the Pods it created.
Some typical uses of a DaemonSet are:
running a cluster storage daemon on every node
running a logs collection daemon on every node
running a node monitoring daemon on every node
In a simple case, one DaemonSet, covering all nodes, would be used for each type of daemon.
A more complex setup might use multiple DaemonSets for a single type of daemon, but with
different flags and/or different memory and cpu requests for different hardware types.
Writing a DaemonSet Spec
Create a DaemonSet
You can describe a DaemonSet in a YAML file. For example, the
daemonset.yaml
file below
describes a DaemonSet that runs the fluentd-elasticsearch Docker image:
controllers/daemonset.yaml
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
spec
:
selector
:
:
name
:
template
:
metadata
:
labels
:
name
:
spec
:
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
-
name
:
image
:
resources
:
limits
:
memory
:
200Mi
requests
:
cpu
:
100m
memory
:
200Mi
:
-
name
:
varlog
mountPath
:
/var/log
:
30
volumes
:
-
name
:
varlog
hostPath
:
path
:
/var/log
Create a DaemonSet based on the YAML file:
kubectl apply -f https://k8s.io/examples/controllers/daemonset.yaml
Required Fields
As with all other Kubernetes config, a DaemonSet needs
,
kind
, and
metadata
fields. For
general information about working with config files, see
running stateless applications
and
object management using kubectl
.
The name of a DaemonSet object must be a valid
.
A DaemonSet also needs a
.spec
section.
The
is one of the required fields in
.spec
.
The
is a
.
Pod
,
except it is nested and does not have an
or
kind
.
In addition to required fields for a Pod, a Pod template in a DaemonSet has to specify appr

opriate
labels (see
pod selector
).
A Pod Template in a DaemonSet must have a
equal to
Always
, or be unspecified, which defaults to
Always
.
The
field is a pod selector. It works the same as the
of
a
Job
.
You must specify a pod selector that matches the labels of the
.
Also, once a DaemonSet is created,
its
can not be mutated. Mutating the pod selector can lead to the
unintentional orphaning of Pods, and it was found to be confusing to users.
The
is an object consisting of two fields:
- works the same as the
of a
.
- allows to build more sophisticated selectors by specifying key,
list of values and an operator that relates the key and values.
When the two are specified the result is ANDed.
The
must match the
.
Config with these two not matching will be rejected by the API.
Running Pods on select Nodes
If you specify a
.spec.template.spec.nodeSelector
, then the DaemonSet controller will
create Pods on nodes which match that
.
Likewise if you specify a
.spec.template.spec.affinity
,
then DaemonSet controller will create Pods on nodes which match that
.
If you do not specify either, then the DaemonSet controller will create Pods on all nodes.
How Daemon Pods are scheduled
Scheduled by default scheduler
Kubernetes 1.17 [stable]
A DaemonSet ensures that all eligible nodes run a copy of a Pod. Normally, the
node that a Pod runs on is selected by the Kubernetes scheduler. However,
DaemonSet pods are created and scheduled by the DaemonSet controller instead.
That introduces the following issues:
Inconsistent Pod behavior: Normal Pods waiting to be scheduled are created
and in
Pending
state, but DaemonSet pods are not created in
Pending
state. This is confusing to the user.
Pod preemption
is handled by default scheduler. When preemption is enabled, the DaemonSet controller
will make scheduling decisions without considering pod priority and preemption.
allows you to schedule DaemonSets using the default
scheduler instead of the DaemonSet controller, by adding the
term
to the DaemonSet pods, instead of the
term. The default
scheduler is then used to bind the pod to the target host. If node affinity of
the DaemonSet pod already exists, it is replaced (the original node affinity was
taken into account before selecting the target host). The DaemonSet controller only
performs these operations when creating or modifying DaemonSet pods, and no
changes are made to the
spec.template
of the DaemonSet.
:
:
:
-
:
-
key
:
operator
:
In
values
:
- target-host-name
In addition,
node.kubernetes.io/unschedulable:NoSchedule
toleration is added
automatically to DaemonSet Pods. The default scheduler ignores
Nodes when scheduling DaemonSet Pods.
Although Daemon Pods respect
,
the following tolerations are added to DaemonSet Pods automatically according to
the related features.
Toleration Key
Effect
Version
NoExecute
1.13+
DaemonSet pods will not be evicted when there are node problems such as a network partition.
NoExecute
1.13+
1.8+
DaemonSet pods tolerate disk-pressure attributes by default scheduler.
1.8+
DaemonSet pods tolerate memory-pressure attributes by default scheduler.
1.12+
DaemonSet pods tolerate unschedulable attributes by default scheduler.
NoSch

edule
1.12+
DaemonSet pods, who uses host network, tolerate network-unavailable attributes by default scheduler.
Communicating with Daemon Pods
Some possible patterns for communicating with Pods in a DaemonSet are:
Push
: Pods in the DaemonSet are configured to send updates to another service, such
as a stats database. They do not have clients.
NodeIP and Known Port
: Pods in the DaemonSet can use a
hostPort
, so that the pods
are reachable via the node IPs.
Clients know the list of node IPs somehow, and know the port by convention.
DNS
: Create a
with the same pod selector, and then discover DaemonSets using the
endpoints
resource or retrieve multiple A records from DNS.
Service
: Create a service with the same Pod selector, and use the service to reach a
daemon on a random node. (No way to reach specific node.)
Updating a DaemonSet
If node labels are changed, the DaemonSet will promptly add Pods to newly matching nodes and delete
Pods from newly not-matching nodes.
You can modify the Pods that a DaemonSet creates. However, Pods do not allow all
fields to be updated. Also, the DaemonSet controller will use the original template the next
time a node (even with the same name) is created.
You can delete a DaemonSet. If you specify
with
kubectl
, then the Pods
will be left on the nodes. If you subsequently create a new DaemonSet with the same selector,
the new DaemonSet adopts the existing Pods. If any Pods need replacing the DaemonSet replaces
them according to its
.
You can
perform a rolling update
on a DaemonSet.
Alternatives to DaemonSet
Init scripts
It is certainly possible to run daemon processes by directly starting them on a node (e.g. using
init
,
upstartd
, or
systemd
). This is perfectly fine. However, there are several advantages to
running such processes via a DaemonSet:
Ability to monitor and manage logs for daemons in the same way as applications.
Same config language and tools (e.g. Pod templates,
kubectl
) for daemons and applications.
Running daemons in containers with resource limits increases isolation between daemons from app
containers. However, this can also be accomplished by running the daemons in a container but not in a Pod.
Bare Pods
It is possible to create Pods directly which specify a particular node to run on. However,
a DaemonSet replaces Pods that are deleted or terminated for any reason, such as in the case of
node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, you should
use a DaemonSet rather than creating individual Pods.
It is possible to create Pods by writing a file to a certain directory watched by Kubelet. These
are called
.
Unlike DaemonSet, static Pods cannot be managed with kubectl
or other Kubernetes API clients. Static Pods do not depend on the apiserver, making them useful
in cluster bootstrapping cases. Also, static Pods may be deprecated in the future.
DaemonSets are similar to
in that
they both create Pods, and those Pods have processes which are not expected to terminate (e.g. web servers,
storage servers).
Use a Deployment for stateless services, like frontends, where scaling up and down the
number of replicas and rolling out updates are more important than controlling exactly which host
the Pod runs on. Use a DaemonSet when it is important that a copy of a Pod always run on
all or certain hosts, if the DaemonSet provides node-level functionality that allows other Pods to run correctly on that particular node.
network plugins
often include a component that runs as a DaemonSet. The DaemonSet component makes sure that the node where it's running has working cluster networking.
Pods
.
, which are useful for running Kubernetes
Find out how to use DaemonSets
Perform a rolling update on a DaemonSet
Perform a rollback on a DaemonSet
(for example, if a roll out didn't 

work how you expected).
how Kubernetes assigns Pods to Nodes
.
and
add ons
, which often run as DaemonSets.
DaemonSet
Read the
DaemonSet
object definition to understand the API for daemon sets.
5.2.5 - Jobs
A Job creates one or more Pods and will continue to retry execution of the Pods until a specified number of them successfully terminate.
As pods successfully complete, the Job tracks the successful completions. When a specified number
of successful completions is reached, the task (ie, Job) is complete. Deleting a Job will clean up
the Pods it created. Suspending a Job will delete its active Pods until the Job
is resumed again.
A simple case is to create one Job object in order to reliably run one Pod to completion.
The Job object will start a new Pod if the first Pod fails or is deleted (for example
due to a node hardware failure or a node reboot).
You can also use a Job to run multiple Pods in parallel.
If you want to run a Job (either a single task, or several in parallel) on a schedule,
see
CronJob
.
Running an example Job
Here is an example Job config. It computes π to 2000 places and prints it out.
It takes around 10s to complete.
controllers/job.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
pi
spec
:
template
:
spec
:
:
-
name
:
pi
image
:
perl:5.34.0
command
:
[
"perl"
,
,
"-wle"
,
]
:
Never
:
4
You can run the example with this command:
kubectl apply -f https://kubernetes.io/examples/controllers/job.yaml
job.batch/pi created
Check on the status of the Job with
kubectl
:
kubectl describe job pi
kubectl get job pi -o yaml
Name:             pi
=
0cd26dd5-88a2-4a5f-a203-ea19a1d5d578
=
job-name
=
pi
Annotations:      batch.kubernetes.io/job-tracking:
1
Completions:
1
Completion Mode:  NonIndexed
Start Time:       Fri,
28
Oct
2022
13:05:18 +0530
Completed At:     Fri,
28
Oct
2022
13:05:21 +0530
Duration:         3s
0
Active /
1
0
Failed
Labels:  controller-uid
=
job-name
=
pi
pi:
Image:      perl:5.34.0
Port:       &lt;none&gt;
Host Port:  &lt;none&gt;
Command:
perl
-Mbignum
=
bpi
-wle
print bpi
(
2000
)
Environment:  &lt;none&gt;
Mounts:       &lt;none&gt;
Volumes:        &lt;none&gt;
Events:
Type    Reason            Age   From            Message
----    ------            ----  ----            -------
Normal  SuccessfulCreate  21s   job-controller  Created pod: pi-xf9p4
Normal  Completed         18s   job-controller  Job completed
kind: Job
metadata:
batch.kubernetes.io/job-tracking:
""
kubectl.kubernetes.io/last-applied-configuration: |
{
:
"batch/v1"
,
"kind"
:
"Job"
,
:
{
:
{}
,
"name"
:
"pi"
,
:
"default"
}
,
"spec"
:
{
"backoffLimit"
:4,
:
{
"spec"
:
{
:
[{
"command"
:
[
"perl"
,
,
"-wle"
,
]
,
"image"
:
"perl:5.34.0"
,
"name"
:
"pi"
}]
,
"restartPolicy"
:
"Never"
}}}}
creationTimestamp:
"2022-11-10T17:53:53Z"
generation:
1
labels:
controller-uid: 204fb678-040b-497f-9266-35ffa8716d14
job-name: pi
name: pi
namespace: default
resourceVersion:
"4751"
uid: 204fb678-040b-497f-9266-35ffa8716d14
spec:
backoffLimit:
4
completionMode: NonIndexed
completions:
1
parallelism:
1
selector:
suspend:
false
template:
metadata:
creationTimestamp: null
labels:
spec:
- command:
- perl
- -Mbignum
=
bpi
- -wle
- print bpi
(
2000
)
image: perl:5.34.0
imagePullPolicy: IfNotPresent
name: pi
{}
terminationMessagePath: /dev/

termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Never
schedulerName: default-scheduler
securityContext:
{}
terminationGracePeriodSeconds:
30
status:
active:
1
ready:
0
startTime:
"2022-11-10T17:53:57Z"
uncountedTerminatedPods:
{}
To view completed Pods of a Job, use
.
To list all the Pods that belong to a Job in a machine readable form, you can use a command like this:
pods
=
$(
=
job-name
=
pi --output
=
jsonpath
=
)
echo
$pods
pi-5rwd7
Here, the selector is the same as the selector for the Job. The
--output=jsonpath
option specifies an expression
with the name from each Pod in the returned list.
View the standard output of one of the pods:
$pods
3.1415926535897932384626433832795028841971693993751058209749445923078164062862089986280348253421170679821480865132823066470938446095505822317253594081284811174502841027019385211055596446229489549303819644288109756659334461284756482337867831652712019091456485669234603486104543266482133936072602491412737245870066063155881748815209209628292540917153643678925903600113305305488204665213841469519415116094330572703657595919530921861173819326117931051185480744623799627495673518857527248912279381830119491298336733624406566430860213949463952247371907021798609437027705392171762931767523846748184676694051320005681271452635608277857713427577896091736371787214684409012249534301465495853710507922796892589235420199561121290219608640344181598136297747713099605187072113499999983729780499510597317328160963185950244594553469083026425223082533446850352619311881710100031378387528865875332083814206171776691473035982534904287554687311595628638823537875937519577818577805321712268066130019278766111959092164201989380952572010654858632788659361533818279682303019520353018529689957736225994138912497217752834791315155748572424541506959508295331168617278558890750983817546374649393192550604009277016711390098488240128583616035637076601047101819429555961989467678374494482553797747268471040475346462080466842590694912933136770289891521047521620569660240580381501935112533824300355876402474964732639141992726042699227967823547816360093417216412199245863150302861829745557067498385054945885869269956909272107975093029553211653449872027559602364806654991198818347977535663698074265425278625518184175746728909777727938000816470600161452491921732172147723501414419735685481613611573525521334757418494684385233239073941433345477624168625189835694855620992192221842725502542568876717904946016534668049886272327917860857843838279679766814541009538837863609506800642251252051173929848960841284886269456042419652850222106611863067442786220391949450471237137869609563643719172874677646575739624138908658326459958133904780275901
Writing a Job spec
As with all other Kubernetes config, a Job needs
,
kind
, and
metadata
fields.
When the control plane creates new Pods for a Job, the
of the
Job is part of the basis for naming those Pods. The name of a Job must be a valid
DNS label
.
Even when the name is a DNS subdomain, the name must be no longer than 63
A Job also needs a
.spec
section
.
The
is the only required field of the
.spec
.
The
is a
Pod
or
kind
.
In addition to required fields for a Pod, a pod template in a Job must specify appropriate
) and an appropriate restart policy.
Only a
equal to
Never
or
OnFailure
Pod selector
The
field is optional. In almost all cases you should not specify it.
See section
specifying your own pod selector
.
Parallel execut

ion for Jobs
There are three main types of task suitable to run as a Job:
Non-parallel Jobs
normally, only one Pod is started, unless the Pod fails.
the Job is complete as soon as its Pod terminates successfully.
Parallel Jobs with a
fixed completion count
:
specify a non-zero positive value for
.
the Job represents the overall task, and is complete when there are
successful Pods.
when using
.spec.completionMode="Indexed"
, each Pod gets a different index in the range 0 to
.spec.completions-1
.
work queue
:
do not specify
, default to
.
the Pods must coordinate amongst themselves or an external service to determine what each should work on. For example, a Pod might fetch a batch of up to N items from the work queue.
each Pod is independently capable of determining whether or not all its peers are done, and thus that the entire Job is done.
when
any
Pod from the Job terminates with success, no new Pods are created.
once at least one Pod has terminated with success and all Pods are terminated, then the Job is completed with success.
once any Pod has exited with success, no other Pod should still be doing any work for this task or writing any output. They should all be in the process of exiting.
For a
non-parallel
Job, you can leave both
and
unset. When both are
unset, both are defaulted to 1.
For a
Job, you should set
to the number of completions needed.
You can set
, or leave it unset and it will default to 1.
For a
work queue
Job, you must leave
unset, and set
to
a non-negative integer.
For more information about how to make use of the different types of job, see the
section.
Controlling parallelism
The requested parallelism (
) can be set to any non-negative value.
If it is unspecified, it defaults to 1.
If it is specified as 0, then the Job is effectively paused until it is increased.
Actual parallelism (number of pods running at any instant) may be more or less than requested
parallelism, for a variety of reasons:
For
Jobs, the actual number of pods running in parallel will not exceed the number of
remaining completions. Higher values of
are effectively ignored.
For
work queue
Jobs, no new Pods are started after any Pod has succeeded -- remaining Pods are allowed to complete, however.
If the Job
has not had time to react.
If the Job controller failed to create Pods for any reason (lack of
, lack of permission, etc.),
then there may be fewer pods than requested.
The Job controller may throttle new Pod creation due to excessive previous pod failures in the same Job.
When a Pod is gracefully shut down, it takes time to stop.
Completion mode
Jobs with
- that is, jobs that have non null
- can have a completion mode that is specified in
.spec.completionMode
:
(default): the Job is considered complete when there have been
successfully completed Pods. In other words, each Pod
completion is homologous to each other. Note that Jobs that have null
are implicitly
.
Indexed
: the Pods of a Job get an associated completion index from 0 to
. The index is available through three mechanisms:
The Pod annotation
.
As part of the Pod hostname, following the pattern
.
When you use an Indexed Job in combination with a
Service
, Pods within the Job can use
the deterministic hostnames to address each other via DNS. For more information about
how to configure this, see
.
From the containerized task, in the environment variable
.
The Job is considered complete wh

en there is one successfully completed Pod
for each index. For more information about how to use this mode, see
.
Note that, although rare, more than one Pod could be started for the same
index, but only one of them will count towards the completion count.
Handling Pod and container failures
A container in a Pod may fail for a number of reasons, such as because the process in it exited with
a non-zero exit code, or the container was killed for exceeding a memory limit, etc. If this
happens, and the
.spec.template.spec.restartPolicy = "OnFailure"
, then the Pod stays
on the node, but the container is re-run. Therefore, your program needs to handle the case when it is
restarted locally, or else specify
.spec.template.spec.restartPolicy = "Never"
.
See
pod lifecycle
for more information on
.
An entire Pod can also fail, for a number of reasons, such as when the pod is kicked off the node
(node is upgraded, rebooted, deleted, etc.), or if a container of the Pod fails and the
. When a Pod fails, then the Job controller
starts a new Pod. This means that your application needs to handle the case when it is restarted in a new
pod. In particular, it needs to handle temporary files, locks, incomplete output and the like
caused by previous runs.
By default, each pod failure is counted towards the
limit,
see
pod backoff failure policy
. However, you can
customize handling of pod failures by setting the Job's
pod failure policy
.
Note that even if you specify
.spec.parallelism = 1
and
.spec.completions = 1
and
, the same program may
sometimes be started twice.
If you do specify
and
both greater than 1, then there may be
multiple pods running at once. Therefore, your pods must also be tolerant of concurrency.
When the
and
are both enabled,
and the
.spec.podFailurePolicy
field is set, the Job controller does not consider a terminating
Pod (a pod that has a
field set) as a failure until that Pod is
terminal (its
.status.phase
is
Failed
or
Succeeded
). However, the Job controller
creates a replacement Pod as soon as the termination becomes apparent. Once the
pod terminates, the Job controller evaluates
.backoffLimit
and
.podFailurePolicy
for the relevant Job, taking this now-terminated Pod into consideration.
If either of these requirements is not satisfied, the Job controller counts
a terminating Pod as an immediate failure, even if that Pod later terminates
with
phase: "Succeeded"
.
There are situations where you want to fail a Job after some amount of retries
due to a logical error in configuration etc.
To do so, set
to specify the number of retries before
considering a Job as failed. The back-off limit is set by default to 6. Failed
Pods associated with the Job are recreated by the Job controller with an
exponential back-off delay (10s, 20s, 40s ...) capped at six minutes.
The number of retries is calculated in two ways:
The number of Pods with
.status.phase = "Failed"
.
When using
restartPolicy = "OnFailure"
, the number of retries in all the
containers of Pods with
equal to
Pending
or
Running
.
If either of the calculations reaches the
, the Job is
considered failed.
Note:
If your job has
, keep in mind that your Pod running the Job
will be terminated once the job backoff limit has been reached. This can make debugging the Job's executable more difficult. We suggest setting
restartPolicy = "Never"
when debugging the Job or using a logging system to ensure output
from failed Jobs is not lost inadvertently.
Job termination and cleanup
When a Job completes, no more Pods are created, but the Pods are
usually
not deleted either.
Keeping

 them around
allows you to still view the logs of completed pods to check for errors, warnings, or other diagnostic output.
The job object also remains after it is completed so that you can view its status. It is up to the user to delete
old jobs after noting their status. Delete the job with
kubectl
(e.g.
kubectl delete jobs/pi
or
kubectl delete -f ./job.yaml
). When you delete the job using
kubectl
, all the pods it created are deleted too.
By default, a Job will run uninterrupted unless a Pod fails (
restartPolicy=Never
) or a Container exits in error (
restartPolicy=OnFailure
), at which point the Job defers to the
described above. Once
has been reached the Job will be marked as failed and any running Pods will be terminated.
Another way to terminate a Job is by setting an active deadline.
Do this by setting the
.spec.activeDeadlineSeconds
field of the Job to a number of seconds.
The
applies to the duration of the job, no matter how many Pods are created.
Once a Job reaches
, all of its running Pods are terminated and the Job status will become
type: Failed
with
reason: DeadlineExceeded
.
Note that a Job's
takes precedence over its
. Therefore, a Job that is retrying one or more failed Pods will not deploy additional Pods once it reaches the time limit specified by
, even if the
is not yet reached.
Example:
:
batch/v1
kind
:
Job
metadata
:
name
:
pi-with-timeout
spec
:
:
5
:
100
template
:
spec
:
:
-
name
:
pi
image
:
command
:
[
"perl"
,
,
"-wle"
,
]
:
Never
Note that both the Job spec and the
Pod template spec
within the Job have an
field. Ensure that you set this field at the proper level.
applies to the Pod, and not to the Job itself: there is no automatic Job restart once the Job status is
.
That is, the Job termination mechanisms activated with
and
result in a permanent Job failure that requires manual intervention to resolve.
Clean up finished jobs automatically
Finished Jobs are usually no longer needed in the system. Keeping them around in
the system will put pressure on the API server. If the Jobs are managed directly
by a higher level controller, such as
CronJobs
, the Jobs can be
cleaned up by CronJobs based on the specified capacity-based cleanup policy.
TTL mechanism for finished Jobs
Another way to clean up finished Jobs (either
Complete
or
Failed
)
automatically is to use a TTL mechanism provided by a
for
finished resources, by specifying the
.spec.ttlSecondsAfterFinished
field of
the Job.
When the TTL controller cleans up the Job, it will delete the Job cascadingly,
i.e. delete its dependent objects, such as Pods, together with the Job. Note
that when the Job is deleted, its lifecycle guarantees, such as finalizers, will
be honored.
:
batch/v1
kind
:
Job
metadata
:
name
:
pi-with-ttl
spec
:
:
100
template
:
spec
:
:
-
name
:
pi
image
:
command
:
[
"perl"
,
,
"-wle"
,
]
:
Never
The Job
will be eligible to be automatically deleted,
100
seconds after it finishes.
If the field is set to
0
, the Job will be eligible to be automatically deleted
immediately after it finishes. If the field is unset, this Job won't be cleaned
up by the TTL controller after it finishes.
Note:
It is recommended to set
field because unmanaged jobs
(Jobs that you created directly, and not indirectly through other workload APIs
such as CronJob) have a default deletion
policy of
causing Pods created by an unmanaged Job to be

 left around
after that Job is fully deleted.
Even though the
eventually
garbage collects
the Pods from a deleted Job after they either fail or complete, sometimes those
lingering pods may cause cluster performance degradation or in worst case cause the
cluster to go offline due to this degradation.
You can use
and
ResourceQuotas
to place a
cap on the amount of resources that a particular namespace can
consume.
Job patterns
The Job object can be used to support reliable parallel execution of Pods. The Job object is not
designed to support closely-communicating parallel processes, as commonly found in scientific
computing. It does support parallel processing of a set of independent but related
work items
.
These might be emails to be sent, frames to be rendered, files to be transcoded, ranges of keys in a
NoSQL database to scan, and so on.
In a complex system, there may be multiple different sets of work items. Here we are just
considering one set of work items that the user wants to manage together — a
batch job
.
There are several different patterns for parallel computation, each with strengths and weaknesses.
The tradeoffs are:
One Job object for each work item, vs. a single Job object for all work items. The latter is
better for large numbers of work items. The former creates some overhead for the user and for the
system to manage large numbers of Job objects.
Number of pods created equals number of work items, vs. each Pod can process multiple work items.
The former typically requires less modification to existing code and containers. The latter
is better for large numbers of work items, for similar reasons to the previous bullet.
Several approaches use a work queue. This requires running a queue service,
and modifications to the existing program or container to make it use the work queue.
Other approaches are easier to adapt to an existing containerised application.
The tradeoffs are summarized here, with columns 2 to 4 corresponding to the above tradeoffs.
The pattern names are also links to examples and more detailed description.
Pattern
Single Job object
Fewer pods than work items?
Use app unmodified?
Queue with Pod Per Work Item
✓
sometimes
Queue with Variable Pod Count
✓
✓
Indexed Job with Static Work Assignment
✓
✓
Job Template Expansion
✓
✓
sometimes
sometimes
When you specify completions with
, each Pod created by the Job controller
has an identical
spec
. This means that
all pods for a task will have the same command line and the same
image, the same volumes, and (almost) the same environment variables. These patterns
are different ways to arrange for pods to work on different things.
This table shows the required settings for
and
for each of the patterns.
Here,
W
is the number of work items.
Pattern
W
any
null
any
W
any
1
should be 1
W
W
Advanced usage
Suspending a Job
When a Job is created, the Job controller will immediately begin creating Pods
to satisfy the Job's requirements and will continue to do so until the Job is
complete. However, you may want to temporarily suspend a Job's execution and
resume it later, or start Jobs in suspended state and have a custom controller
decide later when to start them.
To suspend a Job, you can update the
field of
the Job to true; later, when you want to resume it again, update it to false.
Creating a Job with
set to true will create it in the suspended
state.
When a Job is resumed from suspension, its
.status.startTime
reset to the current time. This means that the
timer will be stopped and reset when a Job is suspended and resumed.
When you suspend a Job, any ru

nning Pods that don't have a status of
Completed
will be
.
with a SIGTERM signal. The Pod's graceful termination period will be honored and
your Pod must handle this signal in this period. This may involve saving
progress for later or undoing changes. Pods terminated this way will not count
towards the Job's
count.
An example Job definition in the suspended state can be like so:
kubectl get job myjob -o yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
myjob
spec
:
suspend
:
true
:
1
:
5
template
:
spec
:
...
You can also toggle Job suspension by patching the Job using the command line.
Suspend an active Job:
kubectl patch job/myjob --type
=
strategic --patch
'{"spec":{"suspend":true}}'
Resume a suspended Job:
=
'{"spec":{"suspend":false}}'
The Job's status can be used to determine if a Job is suspended or has been
suspended in the past:
kubectl get jobs/myjob -o yaml
:
batch/v1
kind
:
Job
# .metadata and .spec omitted
status
:
:
-
:
"2021-02-05T13:14:33Z"
:
status
:
"True"
type
:
Suspended
startTime
:
"2021-02-05T13:13:48Z"
The Job condition of type "Suspended" with status "True" means the Job is
suspended; the
field can be used to determine how long the
Job has been suspended for. If the status of that condition is "False", then the
Job was previously suspended and is now running. If such a condition does not
exist in the Job's status, the Job has never been stopped.
Events are also created when the Job is suspended and resumed:
kubectl describe jobs/myjob
Name:           myjob
...
Events:
  Normal  SuccessfulCreate  12m   job-controller  Created pod: myjob-hlrpl
  Normal  SuccessfulDelete  11m   job-controller  Deleted pod: myjob-hlrpl
  Normal  Suspended         11m   job-controller  Job suspended
  Normal  SuccessfulCreate  3s    job-controller  Created pod: myjob-jvb44
  Normal  Resumed           3s    job-controller  Job resumed
The last four events, particularly the "Suspended" and "Resumed" events, are
directly a result of toggling the
field. In the time between
these two events, we see that no Pods were created, but Pod creation restarted
as soon as the Job was resumed.
Mutable Scheduling Directives
Note:
on the
API server
.
It is enabled by default.
In most cases a parallel job will want the pods to run with constraints,
like all in the same zone, or all either on GPU model x or y but not a mix of both.
The
suspend
field is the first step towards achieving those semantics. Suspend allows a
custom queue controller to decide when a job should start; However, once a job is unsuspended,
a custom queue controller has no influence on where the pods of a job will actually land.
This feature allows updating a Job's scheduling directives before it starts, which gives custom queue
controllers the ability to influence pod placement while at the same time offloading actual
pod-to-node assignment to kube-scheduler. This is allowed only for suspended Jobs that have never
been unsuspended before.
The fields in a Job's pod template that can be updated are node affinity, node selector,
tolerations, labels and annotations.
Specifying your own Pod selector
Normally, when you create a Job object, you do not specify
.
The system defaulting logic adds this field when the Job is created.
It picks a selector value that will not overlap with any other jobs.
However, in some cases, you might need to override this automatically set selector.
To do this, you can specify the
of the Job.
Be very careful when doing this. If you specify a label selector which 

is not
unique to the pods of that Job, and which matches unrelated Pods, then pods of the unrelated
job may be deleted, or this Job may count other Pods as completing it, or one or both
Jobs may refuse to create Pods or run to completion. If a non-unique selector is
chosen, then other controllers (e.g. ReplicationController) and their Pods may behave
in unpredictable ways too. Kubernetes will not stop you from making a mistake when
.
Here is an example of a case when you might want to use this feature.
Say Job
old
is already running. You want existing Pods
to keep running, but you want the rest of the Pods it creates
to use a different pod template and for the Job to have a new name.
You cannot update the Job because these fields are not updatable.
Therefore, you delete Job
old
but
leave its pods
running
, using
kubectl delete jobs/old --cascade=orphan
.
Before deleting it, you make a note of what selector it uses:
kubectl get job old -o yaml
kind
:
Job
metadata
:
name
:
old
...
spec
:
selector
:
:
controller-uid
:
a8f3d00d-c6d2-11e5-9f87-42010af00002
...
Then you create a new Job with name
new
and you explicitly specify the same selector.
Since the existing Pods have label
controller-uid=a8f3d00d-c6d2-11e5-9f87-42010af00002
,
they are controlled by Job
new
as well.
You need to specify
manualSelector: true
in the new Job since you are not using
the selector that the system normally generates for you automatically.
kind
:
Job
metadata
:
name
:
new
...
spec
:
:
true
selector
:
:
:
...
The new Job itself will have a different uid from
. Setting
tells the system that you know what you are doing and to allow this
mismatch.
Note:
You can only configure a Pod failure policy for a Job if you have the
enabled in your cluster. Additionally, it is recommended
feature gate in order to be able to detect and handle
Pod disruption conditions in the Pod failure policy (see also:
). Both feature gates are
available in Kubernetes 1.26.
A Pod failure policy, defined with the
field, enables
your cluster to handle Pod failures based on the container exit codes and the
Pod conditions.
In some situations, you may want to have a better control when handling Pod
failures than the control provided by the
,
which is based on the Job's
. These are some examples of use cases:
To optimize costs of running workloads by avoiding unnecessary Pod restarts,
you can terminate a Job as soon as one of its Pods fails with an exit code
indicating a software bug.
To guarantee that your Job finishes even if there are disruptions, you can
ignore Pod failures caused by disruptions (such
,
or
taint
-based eviction) so
that they don't count towards the
limit of retries.
You can configure a Pod failure policy, in the
field,
to meet the above use cases. This policy can handle Pod failures based on the
container exit codes and the Pod conditions.
Here is a manifest for a Job that defines a
:
/controllers/job-pod-failure-policy-example.yaml
:
batch/v1
kind
:
Job
metadata
:
name
:
job-pod-failure-policy-example
spec
:
:
12
:
3
template
:
spec
:
:
Never
:
-
name
:
main
image
:
command
:
[
"bash"
]
# example command simulating a bug which triggers the FailJob action
args
:
- -c
- echo "Hello world!" &amp;&amp; sleep 5 &amp;&amp; exit 42
:
6
:
rules
:
-
action
:
FailJob
:
:
main
# optional
operator: In             # one of
:
In,

 NotIn
values
:
[
42
]
-
action: Ignore             # one of
:
Ignore, FailJob, Count
:
-
type
:
# indicates Pod disruption
In the example above, the first rule of the Pod failure policy specifies that
the Job should be marked failed if the
main
container fails with the 42 exit
code. The following are the rules for the
main
container specifically:
an exit code of 0 means that the container succeeded
an exit code of 42 means that the
entire Job
failed
any other exit code represents that the container failed, and hence the entire
Pod. The Pod will be re-created if the total number of restarts is
below
. If the
is reached the
entire Job
failed.
Note:
Because the Pod template specifies a
,
the kubelet does not restart the
main
container in that particular Pod.
The second rule of the Pod failure policy, specifying the
Ignore
action for
failed Pods with condition
excludes Pod disruptions from
being counted towards the
Note:
If the Job failed, either by the Pod failure policy or Pod backoff
failure policy, and the Job is running multiple Pods, Kubernetes terminates all
the Pods in that Job that are still Pending or Running.
These are some requirements and semantics of the API:
if you want to use a
field for a Job, you must
also define that Job's pod template with
.spec.restartPolicy
set to
Never
.
the Pod failure policy rules you specify under
spec.podFailurePolicy.rules
are evaluated in order. Once a rule matches a Pod failure, the remaining rules
are ignored. When no rule matches the Pod failure, the default
handling applies.
you may want to restrict a rule to a specific container by specifing its name
in
spec.podFailurePolicy.rules[*].containerName
. When not specified the rule
applies to all containers. When specified, it should match one the container
or
initContainer
names in the Pod template.
you may specify the action taken when a Pod failure policy is matched by
spec.podFailurePolicy.rules[*].action
. Possible values are:
FailJob
: use to indicate that the Pod's job should be marked as Failed and
all running Pods should be terminated.
Ignore
: use to indicate that the counter towards the
should not be incremented and a replacement Pod should be created.
Count
: use to indicate that the Pod should be handled in the default way.
The counter towards the
should be incremented.
Job tracking with finalizers
Note:
The control plane doesn't track Jobs using finalizers, if the Jobs were created
when the feature gate
was disabled, even after you
upgrade the control plane to 1.26.
The control plane keeps track of the Pods that belong to any Job and notices if
any such Pod is removed from the API server. To do that, the Job controller
creates Pods with the finalizer
batch.kubernetes.io/job-tracking
. The
controller removes the finalizer only after the Pod has been accounted for in
the Job status, allowing the Pod to be removed by other controllers or users.
Jobs created before upgrading to Kubernetes 1.26 or before the feature gate
is enabled are tracked without the use of Pod
finalizers.
The Job
updates
the status counters for
succeeded
and
failed
Pods based only on the Pods
that exist in the cluster. The contol plane can lose track of the progress of
the Job if Pods are deleted from the cluster.
You can determine if the control plane is tracking a Job using Pod finalizers by
checking if the Job has the annotation
. You should
not
manually add or remove
this annotation from Jobs. Instead, you can recreate the Jobs to ensure they
are tracked using Pod finalizers.
Bare Pods
When the node that a Pod is running on reboots or fails, the pod is terminated
and will not be restarted. However, a Job will c

reate new Pods to replace terminated ones.
For this reason, we recommend that you use a Job rather than a bare Pod, even if your application
requires only a single Pod.
Replication Controller
Jobs are complementary to
.
A Replication Controller manages Pods which are not expected to terminate (e.g. web servers), and a Job
manages Pods that are expected to terminate (e.g. batch tasks).
As discussed in
,
Job
is
only
for pods with
equal to
OnFailure
or
Never
.
(Note: If
is not set, the default value is
Always
.)
Single Job starts controller Pod
Another pattern is for a single Job to create a Pod which then creates other Pods, acting as a sort
of custom controller for those Pods. This allows the most flexibility, but may be somewhat
complicated to get started with and offers less integration with Kubernetes.
One example of this pattern would be a Job which starts a Pod which runs a script that in turn
starts a Spark master controller (see
spark example
), runs a spark
driver, and then cleans up.
An advantage of this approach is that the overall process gets the completion guarantee of a Job
object, but maintains complete control over what Pods are created and how work is assigned to them.
Pods
.
Read about different ways of running Jobs:
Use an
indexed Job for parallel processing with static work assignment
Create multiple Jobs based on a template:
Follow the links within
to learn more about how your cluster can clean up completed and / or failed tasks.
Job
is part of the Kubernetes REST API.
Read the
Job
object definition to understand the API for jobs.
Read about
CronJob
, which you
can use to define a series of Jobs that will run based on a schedule, similar to
the UNIX tool
cron
.
Practice how to configure handling of retriable and non-retriable pod failures
using
, based on the step-by-step
examples
.
5.2.6 - Automatic Clean-up for Finished Jobs
TTL-after-finished
provides a
TTL (time to live) mechanism to limit the lifetime of resource objects that
have finished execution. TTL controller only handles
Jobs
.
TTL-after-finished Controller
The TTL-after-finished controller is only supported for Jobs. A cluster operator can use this feature to clean
up finished Jobs (either
Complete
or
Failed
) automatically by specifying the
field of a Job, as in this
example
.
The TTL-after-finished controller will assume that a job is eligible to be cleaned up
TTL seconds after the job has finished, in other words, when the TTL has expired. When the
TTL-after-finished controller cleans up a job, it will delete it cascadingly, that is to say it will delete
its dependent objects together with it. Note that when the job is deleted,
its lifecycle guarantees, such as finalizers, will be honored.
The TTL seconds can be set at any time. Here are some examples for setting the
field of a Job:
Specify this field in the job manifest, so that a Job can be cleaned up
automatically some time after it finishes.
Set this field of existing, already finished jobs, to adopt this new
feature.
Use a
to set this field dynamically at job creation time. Cluster administrators can
use this to enforce a TTL policy for finished jobs.
Use a
to set this field dynamically after the job has finished, and choose
different TTL values based on job status, labels, etc.
Caveat
Updating TTL Seconds
Note that the TTL period, e.g.
field of Jobs,
can be modified after the job is created or has finished. However, once the
Job becomes eligible to be deleted (when the TTL has expired), the system won't
guarantee th

at the Jobs will be kept, even if an update to extend the TTL
returns a successful API response.
Time Skew
Because TTL-after-finished controller uses timestamps stored in the Kubernetes jobs to
determine whether the TTL has expired or not, this feature is sensitive to time
skew in the cluster, which may cause TTL-after-finish controller to clean up job objects
at the wrong time.
Clocks aren't always correct, but the difference should be
very small. Please be aware of this risk when setting a non-zero TTL.
Clean up Jobs automatically
Design doc
5.2.7 - CronJob
A
CronJob
creates
Jobs
on a repeating schedule.
One CronJob object is like one line of a
crontab
(cron table) file. It runs a job periodically
on a given schedule, written in
Cron
format.
Caution:
All
CronJob
schedule:
times are based on the timezone of the
.
If your control plane runs the kube-controller-manager in Pods or bare
containers, the timezone set for the kube-controller-manager container determines the timezone
that the cron job controller uses.
Caution:
The
v1 CronJob API
does not officially support setting timezone as explained above.
Setting variables such as
CRON_TZ
or
TZ
is not officially supported by the Kubernetes project.
CRON_TZ
or
TZ
is an implementation detail of the internal library being used
for parsing and calculating the next Job creation time. Any usage of it is not
recommended in a production cluster.
When the control plane creates new Jobs and (indirectly) Pods for a CronJob, the
of the CronJob is part of the basis for naming those Pods. The name of a CronJob must be a valid
DNS label
.
Even when the name is a DNS subdomain, the name must be no longer than 52
characters. This is because the CronJob controller will automatically append
11 characters to the name you provide and there is a constraint that the
length of a Job name is no more than 63 characters.
CronJob
CronJobs are meant for performing regular scheduled actions such as backups,
report generation, and so on. Each of those tasks should be configured to recur
indefinitely (for example: once a day / week / month); you can define the point
in time within that interval when the job should start.
Example
This example CronJob manifest prints the current time and a hello message every minute:
:
batch/v1
kind
:
CronJob
metadata
:
name
:
hello
spec
:
schedule
:
"* * * * *"
:
spec
:
template
:
spec
:
:
-
name
:
hello
image
:
:
command
:
- /bin/sh
- -c
:
OnFailure
(
takes you through this example in more detail).
Cron schedule syntax
# ┌───────────── minute (0 - 59)
# │ ┌───────────── hour (0 - 23)
# │ │ ┌───────────── day of the month (1 - 31)
# │ │ │ ┌───────────── month (1 - 12)
# │ │ │ │ ┌───────────── day of the week (0 - 6) (Sunday to Saturday;
# │ │ │ │ │                                   7 is also Sunday on some systems)
# │ │ │ │ │                                   OR sun, mon, tue, wed, thu, fri, sat
# │ │ │ │ │
# * * * * *
Entry
Equivalent to
@yearly (or @annually)
Run once a year at midnight of 1 January
0 0 1 1 *
@monthly
Run once a month at midnight of the first day of the month
0 0 1 * *
@weekly
Run once a week at midnight on Sunday morning
0 0 * * 0
@daily (or @midnight)
Run once a day at midnight
0 0 * * *
@hourly
Run once an hour at the beginning of the hour
0 * * * *
For example, the line below states that the task must be started every Friday at midnight, as well as on the 13th of each month at midnight:
0 0 13 * 5
To generate CronJob schedule expressions, you can also use web tools like
crontab.guru
.
Time z

ones
For CronJobs with no time zone specified, the kube-controller-manager interprets schedules relative to its local time zone.
,
you can specify a time zone for a CronJob (if you don't enable that feature gate, or if you are using a version of
Kubernetes that does not have experimental time zone support, all CronJobs in your cluster have an unspecified
timezone).
When you have the feature enabled, you can set
spec.timeZone
to the name of a valid
time zone
. For example, setting
spec.timeZone: "Etc/UTC"
instructs Kubernetes to interpret the schedule relative to Coordinated Universal Time.
A time zone database from the Go standard library is included in the binaries and used as a fallback in case an external database is not available on the system.
CronJob limitations
A cron job creates a job object
about
once per execution time of its schedule. We say "about" because there
are certain circumstances where two jobs might be created, or no job might be created. We attempt to make these rare,
but do not completely prevent them. Therefore, jobs should be
.
If
is set to a large value or left unset (the default)
and if
is set to
Allow
, the jobs will always run
at least once.
Caution:
If
is set to a value less than 10 seconds, the CronJob may not be scheduled. This is because the CronJob controller checks things every 10 seconds.
For every CronJob, the CronJob
checks how many schedules it missed in the duration from its last scheduled time until now. If there are more than 100 missed schedules, then it does not start the job and logs the error.
Cannot determine if job needs to be started. Too many missed start time (&gt; 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.
It is important to note that if the
field is set (not
nil
), the controller counts how many missed jobs occurred from the value of
until now rather than from the last scheduled time until now. For example, if
is
200
, the controller counts how many missed jobs occurred in the last 200 seconds.
A CronJob is counted as missed if it has failed to be created at its scheduled time. For example, if
is set to
Forbid
and a CronJob was attempted to be scheduled when there was a previous schedule still running, then it would count as missed.
For example, suppose a CronJob is set to schedule a new Job every one minute beginning at
08:30:00
, and its
field is not set. If the CronJob controller happens to
be down from
08:29:00
to
10:21:00
, the job will not start as the number of missed jobs which missed their schedule is greater than 100.
To illustrate this concept further, suppose a CronJob is set to schedule a new Job every one minute beginning at
08:30:00
, and its
is set to 200 seconds. If the CronJob controller happens to
be down for the same period as the previous example (
08:29:00
to
10:21:00
,) the Job will still start at 10:22:00. This happens as the controller now checks how many missed schedules happened in the last 200 seconds (i.e., 3 missed schedules), rather than from the last scheduled time until now.
The CronJob is only responsible for creating Jobs that match its schedule, and
the Job in turn is responsible for the management of the Pods it represents.
Controller version
Starting with Kubernetes v1.21 the second version of the CronJob controller
is the default implementation. To disable the default CronJob controller
and use the original CronJob controller instead, pass the
flag to the
,
and set this flag to
false
--feature-gates="CronJobControllerV2=false"
Pods
and
Jobs
, two concepts
that CronJobs rely upon.
format
of Cr

onJob
fields.
For instructions on creating and working with CronJobs, and for an example
of a CronJob manifest,
see
Running automated tasks with CronJobs
.
For instructions to clean up failed or completed jobs automatically,
see
CronJob
Read the
CronJob
object definition to understand the API for Kubernetes cron jobs.
5.2.8 - ReplicationController
Note:
A
that configures a
is now the recommended way to set up replication.
A
ensures that a specified number of pod replicas are running at any one
time. In other words, a ReplicationController makes sure that a pod or a homogeneous set of pods is
always up and available.
How a ReplicationController Works
If there are too many pods, the ReplicationController terminates the extra pods. If there are too few, the
ReplicationController starts more pods. Unlike manually created pods, the pods maintained by a
ReplicationController are automatically replaced if they fail, are deleted, or are terminated.
For example, your pods are re-created on a node after disruptive maintenance such as a kernel upgrade.
For this reason, you should use a ReplicationController even if your application requires
only a single pod. A ReplicationController is similar to a process supervisor,
but instead of supervising individual processes on a single node, the ReplicationController supervises multiple pods
across multiple nodes.
ReplicationController is often abbreviated to "rc" in discussion, and as a shortcut in
kubectl commands.
A simple case is to create one ReplicationController object to reliably run one instance of
a Pod indefinitely. A more complex use case is to run several identical replicas of a replicated
service, such as web servers.
Running an example ReplicationController
This example ReplicationController config runs three copies of the nginx web server.
controllers/replication.yaml
:
v1
kind
:
metadata
:
name
:
nginx
spec
:
replicas
:
3
selector
:
app
:
nginx
template
:
metadata
:
name
:
nginx
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
nginx
ports
:
-
:
80
Run the example job by downloading the example file and then running this command:
kubectl apply -f https://k8s.io/examples/controllers/replication.yaml
replicationcontroller/nginx created
Check on the status of the ReplicationController using this command:
kubectl describe replicationcontrollers/nginx
Name:        nginx
Namespace:   default
Selector:    app=nginx
Labels:      app=nginx
Replicas:    3 current / 3 desired
Pods Status: 0 Running / 3 Waiting / 0 Succeeded / 0 Failed
   nginx:
    Image:              nginx
Events:
  FirstSeen       LastSeen     Count    From                        SubobjectPath    Type      Reason              Message
  ---------       --------     -----    ----                        -------------    ----      ------              -------
  20s             20s          1        {replication-controller }                    Normal    SuccessfulCreate    Created pod: nginx-qrm3m
  20s             20s          1        {replication-controller }                    Normal    SuccessfulCreate    Created pod: nginx-3ntk0
  20s             20s          1        {replication-controller }                    Normal    SuccessfulCreate    Created pod: nginx-4ok8v
Here, three pods are created, but none is running yet, perhaps because the image is being pulled.
A little later, the same command may show:
Pods Status:
3
Running /
0
Waiting /
0
0
Failed
To list all the pods that belong to the ReplicationController in a machine readable form, you can 

use a command like this:
pods
=
$(
=
app
=
=
jsonpath
={
}
)
echo
$pods
nginx-3ntk0 nginx-4ok8v nginx-qrm3m
Here, the selector is the same as the selector for the ReplicationController (seen in the
output), and in a different form in
replication.yaml
. The
option
specifies an expression with the name from each pod in the returned list.
Writing a ReplicationController Manifest
As with all other Kubernetes config, a ReplicationController needs
,
kind
, and
metadata
fields.
When the control plane creates new Pods for a ReplicationController, the
of the
ReplicationController is part of the basis for naming those Pods. The name of a ReplicationController must be a valid
DNS label
.
For general information about working with configuration files, see
object management
.
A ReplicationController also needs a
.spec
section
.
The
.spec
.
The
is a
Pod
or
kind
.
In addition to required fields for a Pod, a pod template in a ReplicationController must specify appropriate
.
Only a
equal to
Always
is allowed, which is the default if not specified.
For local container restarts, ReplicationControllers delegate to an agent on the node,
for example the
Kubelet
.
Labels on the ReplicationController
The ReplicationController can itself have labels (
). Typically, you
would set these the same as the
; if
is not specified
then it defaults to
. However, they are allowed to be
different, and the
do not affect the behavior of the ReplicationController.
The
field is a
. A ReplicationController
manages all the pods with labels that match the selector. It does not distinguish
between pods that it created or deleted and pods that another person or process created or
deleted. This allows the ReplicationController to be replaced without affecting the running pods.
If specified, the
must be equal to the
, or it will
be rejected by the API. If
is unspecified, it will be defaulted to
.
Also you should not normally create any pods whose labels match this selector, either directly, with
another ReplicationController, or with another controller such as Job. If you do so, the
ReplicationController thinks that it created the other pods. Kubernetes does not stop you
from doing this.
If you do end up with multiple controllers that have overlapping selectors, you
will have to manage the deletion yourself (see
below
).
Multiple Replicas
You can specify how many pods should run concurrently by setting
to the number
of pods you would like to have running concurrently. The number running at any time may be higher
or lower, such as if the replicas were just increased or decreased, or if a pod is gracefully
shutdown, and a replacement starts early.
Working with ReplicationControllers
Deleting a ReplicationController and its Pods
To delete a ReplicationController and all its pods, use
. Kubectl will scale the ReplicationController to zero and wait
for it to delete each pod before deleting the ReplicationController itself. If this kubectl
command is interrupted, it can be restarted.
When using the RES

T API or
client library
, you need to do the steps explicitly (scale replicas to
0, wait for pod deletions, then delete the ReplicationController).
Deleting only a ReplicationController
You can delete a ReplicationController without affecting any of its pods.
Using kubectl, specify the
option to
.
When using the REST API or
, you can delete the ReplicationController object.
Once the original is deleted, you can create a new ReplicationController to replace it. As long
are the same, then the new one will adopt the old pods.
However, it will not make any effort to make existing pods match a new, different pod template.
To update pods to a new spec in a controlled way, use a
rolling update
.
Isolating pods from a ReplicationController
Pods may be removed from a ReplicationController's target set by changing their labels. This technique may be used to remove pods from service for debugging and data recovery. Pods that are removed in this way will be replaced automatically (assuming that the number of replicas is not also changed).
Common usage patterns
Rescheduling
As mentioned above, whether you have 1 pod you want to keep running, or 1000, a ReplicationController will ensure that the specified number of pods exists, even in the event of node failure or pod termination (for example, due to an action by another control agent).
Scaling
The ReplicationController enables scaling the number of replicas up or down, either manually or by an auto-scaling control agent, by updating the
replicas
field.
The ReplicationController is designed to facilitate rolling updates to a service by replacing pods one-by-one.
#1353
, the recommended approach is to create a new ReplicationController with 1 replica, scale the new (+1) and old (-1) controllers one by one, and then delete the old controller after it reaches 0 replicas. This predictably updates the set of pods regardless of unexpected failures.
Ideally, the rolling update controller would take application readiness into account, and would ensure that a sufficient number of pods were productively serving at any given time.
The two ReplicationControllers would need to create pods with at least one differentiating label, such as the image tag of the primary container of the pod, since it is typically image updates that motivate rolling updates.
Multiple release tracks
In addition to running multiple releases of an application while a rolling update is in progress, it's common to run multiple releases for an extended period of time, or even continuously, using multiple release tracks. The tracks would be differentiated by labels.
For instance, a service might target all pods with
tier in (frontend), environment in (prod)
. Now say you have 10 replicated pods that make up this tier. But you want to be able to 'canary' a new version of this component. You could set up a ReplicationController with
replicas
set to 9 for the bulk of the replicas, with labels
tier=frontend, environment=prod, track=stable
, and another ReplicationController with
replicas
set to 1 for the canary, with labels
tier=frontend, environment=prod, track=canary
. Now the service is covering both the canary and non-canary pods. But you can mess with the ReplicationControllers separately to test things out, monitor the results, etc.
Using ReplicationControllers with Services
Multiple ReplicationControllers can sit behind a single service, so that, for example, some traffic
goes to the old version, and some goes to the new version.
A ReplicationController will never terminate on its own, but it isn't expected to be as long-lived as services. Services may be composed of pods controlled by multiple ReplicationControllers, and it is expected that many ReplicationControllers may be created and destroyed over the lifetime of a service (for instance, to perform an update of pods that run the service). Both services themselves 

and their clients should remain oblivious to the ReplicationControllers that maintain the pods of the services.
Writing programs for Replication
Pods created by a ReplicationController are intended to be fungible and semantically identical, though their configurations may become heterogeneous over time. This is an obvious fit for replicated stateless servers, but ReplicationControllers can also be used to maintain availability of master-elected, sharded, and worker-pool applications. Such applications should use dynamic work assignment mechanisms, such as the
RabbitMQ work queues
, as opposed to static/one-time customization of the configuration of each pod, which is considered an anti-pattern. Any pod customization performed, such as vertical auto-sizing of resources (for example, cpu or memory), should be performed by another online controller process, not unlike the ReplicationController itself.
Responsibilities of the ReplicationController
The ReplicationController ensures that the desired number of pods matches its label selector and are operational. Currently, only terminated pods are excluded from its count. In the future,
readiness
and other information available from the system may be taken into account, we may add more controls over the replacement policy, and we plan to emit events that could be used by external clients to implement arbitrarily sophisticated replacement and/or scale-down policies.
The ReplicationController is forever constrained to this narrow responsibility. It itself will not perform readiness nor liveness probes. Rather than performing auto-scaling, it is intended to be controlled by an external auto-scaler (as discussed in
#492
), which would change its
replicas
field. We will not add scheduling policies (for example,
spreading
) to the ReplicationController. Nor should it verify that the pods controlled match the currently specified template, as that would obstruct auto-sizing and other automated processes. Similarly, completion deadlines, ordering dependencies, configuration expansion, and other features belong elsewhere. We even plan to factor out the mechanism for bulk pod creation (
#170
).
The ReplicationController is intended to be a composable building-block primitive. We expect higher-level APIs and/or tools to be built on top of it and other complementary primitives for user convenience in the future. The "macro" operations currently supported by kubectl (run, scale) are proof-of-concept examples of this. For instance, we could imagine something like
Asgard
managing ReplicationControllers, auto-scalers, services, scheduling policies, canaries, etc.
API Object
Replication controller is a top-level resource in the Kubernetes REST API. More details about the
API object can be found at:
ReplicationController API object
.
Alternatives to ReplicationController
is the next-generation ReplicationController that supports the new
set-based label selector
.
It's mainly used by
as a mechanism to orchestrate pod creation, deletion and updates.
Note that we recommend using Deployments instead of directly using Replica Sets, unless you require custom update orchestration or don't require updates at all.
Deployment (Recommended)
is a higher-level API object that updates its underlying Replica Sets and their Pods. Deployments are recommended if you want the rolling update functionality, because they are declarative, server-side, and have additional features.
Bare Pods
Unlike in the case where a user directly created pods, a ReplicationController replaces pods that are deleted or terminated for any reason, such as in the case of node failure or disruptive node maintenance, such as a kernel upgrade. For this reason, we recommend that you use a ReplicationController even if your application requires only a single pod. Think of it similarly to a process supervisor, only it supervises multiple pods across multiple nodes instead of individual processes on a single 

node. A ReplicationController delegates local container restarts to some agent on the node, such as the kubelet.
Job
Use a
Job
instead of a ReplicationController for pods that are expected to terminate on their own
(that is, batch jobs).
DaemonSet
Use a
DaemonSet
instead of a ReplicationController for pods that provide a
machine-level function, such as machine monitoring or machine logging. These pods have a lifetime that is tied
to a machine lifetime: the pod needs to be running on the machine before other pods start, and are
Pods
.
, the replacement
for ReplicationController.
Read the
object definition to understand the API for replication controllers.
6 - Services, Load Balancing, and Networking
The Kubernetes network model
Every
Pod
in a cluster gets its own unique cluster-wide IP address.
This means you do not need to explicitly create links between
Pods
and you
almost never need to deal with mapping container ports to host ports.
This creates a clean, backwards-compatible model where
Pods
can be treated
much like VMs or physical hosts from the perspectives of port allocation,
naming, service discovery,
load balancing
,
application configuration, and migration.
Kubernetes imposes the following fundamental requirements on any networking
implementation (barring any intentional network segmentation policies):
pods can communicate with all other pods on any other
node
without NAT
agents on a node (e.g. system daemons, kubelet) can communicate with all
pods on that node
Note: For those platforms that support
Pods
running in the host network (e.g.
Linux), when pods are attached to the host network of a node they can still communicate
with all pods on all nodes without NAT.
This model is not only less complex overall, but it is principally compatible
with the desire for Kubernetes to enable low-friction porting of apps from VMs
to containers. If your job previously ran in a VM, your VM had an IP and could
talk to other VMs in your project. This is the same basic model.
Kubernetes IP addresses exist at the
Pod
scope - containers within a
Pod
share their network namespaces - including their IP address and MAC address.
This means that containers within a
Pod
can all reach each other's ports on
localhost
. This also means that containers within a
Pod
must coordinate port
usage, but this is no different from processes in a VM. This is called the
"IP-per-pod" model.
How this is implemented is a detail of the particular container runtime in use.
It is possible to request ports on the
Node
itself which forward to your
Pod
(called host ports), but this is a very niche operation. How that forwarding is
implemented is also a detail of the container runtime. The
Pod
itself is
blind to the existence or non-existence of host ports.
Kubernetes networking addresses four concerns:
Containers within a Pod
use networking to communicate
via loopback.
Cluster networking provides communication between different Pods.
The
Service
API lets you
expose an application running in Pods
to be reachable from outside your cluster.
Ingress
provides extra functionality
specifically for exposing HTTP applications, websites and APIs.
You can also use Services to
publish services only for consumption inside your cluster
.
The
tutorial lets you learn about Services and Kubernetes networking with a hands-on example.
explains how to set
up networking for your cluster, and also provides an overview of the technologies involved.
6.1 - Service
Expose an application running in your cluster behind a single outward-facing endpoint, even when the workload is split across multiple backends.
Pods
as a ne

twork service.
With Kubernetes you don't need to modify your application to use an unfamiliar service discovery mechanism.
Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods,
and can load-balance across them.
Pods
are created and destroyed
to match the desired state of your cluster. Pods are nonpermanent resources.
If you use a
to run your app,
it can create and destroy Pods dynamically.
Each Pod gets its own IP address, however in a Deployment, the set of Pods
running in one moment in time could be different from
the set of Pods running that application a moment later.
This leads to a problem: if some set of Pods (call them "backends") provides
functionality to other Pods (call them "frontends") inside your cluster,
how do the frontends find out and keep track of which IP address to connect
to, so that the frontend can use the backend part of the workload?
Enter
Services
.
Service resources
In Kubernetes, a Service is an abstraction which defines a logical set of Pods
and a policy by which to access them (sometimes this pattern is called
a micro-service). The set of Pods targeted by a Service is usually determined
by a
selector
.
To learn about other ways to define Service endpoints,
see
Services
without
selectors
.
For example, consider a stateless image-processing backend which is running with
3 replicas. Those replicas are fungible—frontends do not care which backend
they use. While the actual Pods that compose the backend set may change, the
frontend clients should not need to be aware of that, nor should they need to keep
track of the set of backends themselves.
The Service abstraction enables this decoupling.
Cloud-native service discovery
If you're able to use Kubernetes APIs for service discovery in your application,
you can query the
API server
for matching EndpointSlices. Kubernetes updates the EndpointSlices for a Service
whenever the set of Pods in a Service changes.
For non-native applications, Kubernetes offers ways to place a network port or load
balancer in between your application and the backend Pods.
Defining a Service
A Service in Kubernetes is a REST object, similar to a Pod. Like all of the
REST objects, you can
POST
a Service definition to the API server to create
a new instance.
The name of a Service object must be a valid
RFC 1035 label name
.
For example, suppose you have a set of Pods where each listens on TCP port 9376
and contains a label
:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
:
9376
This specification creates a new Service object named "my-service", which
targets TCP port 9376 on any Pod with the
label.
Kubernetes assigns this Service an IP address (sometimes called the "cluster IP"),
which is used by the Service proxies
(see
Virtual IP addressing mechanism
below).
The controller for the Service selector continuously scans for Pods that
match its selector, and then POSTs any updates to an Endpoint object
also named "my-service".
Note:
A Service can map
any
incoming
port
to a
. By default and
for convenience, the
is set to the same value as the
port
field.
Port definitions in Pods have names, and you can reference these names in the
attribute of a Service. For example, we can bind the
of the Service to the Pod port in the following way:
:
v1
kind
:
Pod
metadata
:
name
:
nginx
labels
:
:
proxy
spec
:
:
-
name
:
nginx
image
:
nginx:stable
ports
:
-
:
80
name
:
http-web-svc
---
:
v1
kind
:
Service
metadata
:
name
:
nginx-service
spec
:
selector
:
:
proxy
ports
:
-
name
:
name-of-service-port
protocol
:
TCP
port
:
80
:
This works even if there is a mixture of Pods in the Service using a single
configured 

name, with the same network protocol available via different
port numbers. This offers a lot of flexibility for deploying and evolving
your Services. For example, you can change the port numbers that Pods expose
in the next version of your backend software, without breaking clients.
The default protocol for Services is
TCP
; you can also
use any other
supported protocol
.
As many Services need to expose more than one port, Kubernetes supports multiple
port definitions on a Service object.
Each port definition can have the same
protocol
, or a different one.
Services without selectors
Services most commonly abstract access to Kubernetes Pods thanks to the selector,
but when used with a corresponding set of
objects and without a selector, the Service can abstract other kinds of backends,
including ones that run outside the cluster.
You want to have an external database cluster in production, but in your
test environment you use your own databases.
You want to point your Service to a Service in a different
Namespace
or on another cluster.
You are migrating a workload to Kubernetes. While evaluating the approach,
you run only a portion of your backends in Kubernetes.
In any of these scenarios you can define a Service
without
a Pod selector.
:
v1
kind
:
Service
metadata
:
name
:
spec
:
ports
:
-
protocol
:
TCP
port
:
80
:
9376
Because this Service has no selector, the corresponding EndpointSlice (and
legacy Endpoints) objects are not created automatically. You can map the Service
to the network address and port where it's running, by adding an EndpointSlice
object manually. For example:
:
kind
:
metadata
:
name
:
my-service-1
# by convention, use the name of the Service
# as a prefix for the name of the EndpointSlice
labels
:
# You should set the "kubernetes.io/service-name" label.
# Set its value to match the name of the Service
:
:
IPv4
ports
:
-
name
:
''
# empty because port 9376 is not assigned as a well-known
# port (by IANA)
:
http
protocol
:
TCP
port
:
9376
endpoints
:
-
addresses
:
-
"10.4.5.6"
# the IP addresses in this list can appear in any order
-
"10.1.2.3"
Custom EndpointSlices
When you create an
object for a Service, you can
use any name for the EndpointSlice. Each EndpointSlice in a namespace must have a
unique name. You link an EndpointSlice to a Service by setting the
label
on that EndpointSlice.
Note:
The endpoint IPs
must not
be: loopback (127.0.0.0/8 for IPv4, ::1/128 for IPv6), or
link-local (169.254.0.0/16 and 224.0.0.0/24 for IPv4, fe80::/64 for IPv6).
The endpoint IP addresses cannot be the cluster IPs of other Kubernetes Services,
because
doesn't support virtual IPs
as a destination.
For an EndpointSlice that you create yourself, or in your own code,
you should also pick a value to use for the
label.
If you create your own controller code to manage EndpointSlices, consider using a
value similar to
"my-domain.example/name-of-controller"
. If you are using a third
party tool, use the name of the tool in all-lowercase and change spaces and other
punctuation to dashes (
-
).
If people are directly using a tool such as
kubectl
to manage EndpointSlices,
use a name that describes this manual management, such as
"staff"
or
"cluster-admins"
avoid using the reserved value
"controller"
, which identifies EndpointSlices
managed by Kubernetes' own control plane.
Accessing a Service without a selector
Accessing a Service without a selector works the same as if it had a selector.
In the
example
for a Service without a selector, traffic is routed to one of the two endpoints defined in
the EndpointSlice manifest: a TCP connection to 10.1.2.3 or 10.4.5.6, on port 9376.
Note:
The Kubernetes API server does not allow proxying to endpoints that a

re not mapped to
pods. Actions such as
kubectl proxy &lt;service-name&gt;
where the service has no
selector will fail due to this constraint. This prevents the Kubernetes API server
from being used as a proxy to endpoints the caller may not be authorized to access.
An ExternalName Service is a special case of Service that does not have
selectors and uses DNS names instead. For more information, see the
section later in this document.
are objects that
represent a subset (a
slice
) of the backing network endpoints for a Service.
Your Kubernetes cluster tracks how many endpoints each EndpointSlice represents.
If there are so many endpoints for a Service that a threshold is reached, then
Kubernetes adds another empty EndpointSlice and stores new endpoint information
there.
By default, Kubernetes makes a new EndpointSlice once the existing EndpointSlices
all contain at least 100 endpoints. Kubernetes does not make the new EndpointSlice
until an extra endpoint needs to be added.
See
for more
information about this API.
Endpoints
In the Kubernetes API, an
Endpoints
(the resource kind is plural) defines a list of network endpoints, typically
referenced by a Service to define which Pods the traffic can be sent to.
The EndpointSlice API is the recommended replacement for Endpoints.
Over-capacity endpoints
Kubernetes limits the number of endpoints that can fit in a single Endpoints
object. When there are over 1000 backing endpoints for a Service, Kubernetes
truncates the data in the Endpoints object. Because a Service can be linked
with more than one EndpointSlice, the 1000 backing endpoint limit only
affects the legacy Endpoints API.
In that case, Kubernetes selects at most 1000 possible backend endpoints to store
into the Endpoints object, and sets an
on the
Endpoints:
endpoints.kubernetes.io/over-capacity: truncated
.
The control plane also removes that annotation if the number of backend Pods drops below 1000.
Traffic is still sent to backends, but any load balancing mechanism that relies on the
legacy Endpoints API only sends traffic to at most 1000 of the available backing endpoints.
The same API limit means that you cannot manually update an Endpoints to have more than 1000 endpoints.
Application protocol
The
field provides a way to specify an application protocol for
each Service port. The value of this field is mirrored by the corresponding
Endpoints and EndpointSlice objects.
This field follows standard Kubernetes label syntax. Values should either be
IANA standard service names
or
domain prefixed names such as
mycompany.com/my-custom-protocol
.
Multi-Port Services
For some Services, you need to expose more than one port.
Kubernetes lets you configure multiple port definitions on a Service object.
When using multiple ports for a Service, you must give all of your ports names
so that these are unambiguous.
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
:
MyApp
ports
:
-
name
:
http
protocol
:
TCP
port
:
80
:
9376
-
name
:
https
protocol
:
TCP
port
:
443
:
9377
Note:
As with Kubernetes
names
in general, names for ports
must only contain lowercase alphanumeric characters and
-
. Port names must
also start and end with an alphanumeric character.
For example, the names
123-abc
and
web
are valid, but
123_abc
and
-web
are not.
Choosing your own IP address
You can specify your own cluster IP address as part of a
Service
creation
request. To do this, set the
field. For example, if you
already have an existing DNS entry that you wish to reuse, or legacy systems
that are configured for a specific IP address and difficult to re-configure.
The IP address that you choose must be a valid IPv4 or IPv6 address from within the
CIDR range that is configured f

or the API server.
If you try to create a Service with an invalid clusterIP address value, the API
server will return a 422 HTTP status code to indicate that there's a problem.
Discovering services
Kubernetes supports 2 primary modes of finding a Service - environment
variables and DNS.
When a Pod is run on a Node, the kubelet adds a set of environment variables
for each active Service. It adds
{SVCNAME}_SERVICE_HOST
and
{SVCNAME}_SERVICE_PORT
variables,
where the Service name is upper-cased and dashes are converted to underscores.
It also supports variables (see
makeLinkVariables
)
that are compatible with Docker Engine's
"
legacy container links
" feature.
For example, the Service
redis-primary
which exposes TCP port 6379 and has been
allocated cluster IP address 10.0.0.11, produces the following environment
REDIS_PRIMARY_SERVICE_HOST
=
10.0.0.11
REDIS_PRIMARY_SERVICE_PORT
=
6379
REDIS_PRIMARY_PORT
=
tcp://10.0.0.11:6379
REDIS_PRIMARY_PORT_6379_TCP
=
REDIS_PRIMARY_PORT_6379_TCP_PROTO
=
tcp
REDIS_PRIMARY_PORT_6379_TCP_PORT
=
6379
REDIS_PRIMARY_PORT_6379_TCP_ADDR
=
10.0.0.11
Note:
When you have a Pod that needs to access a Service, and you are using
the environment variable method to publish the port and cluster IP to the client
Pods, you must create the Service
before
the client Pods come into existence.
Otherwise, those client Pods won't have their environment variables populated.
If you only use DNS to discover the cluster IP for a Service, you don't need to
worry about this ordering issue.
DNS
You can (and almost always should) set up a DNS service for your Kubernetes
cluster using an
add-on
.
A cluster-aware DNS server, such as CoreDNS, watches the Kubernetes API for new
Services and creates a set of DNS records for each one. If DNS has been enabled
throughout your cluster then all Pods should automatically be able to resolve
Services by their DNS name.
For example, if you have a Service called
namespace
my-ns
, the control plane and the DNS Service acting together
create a DNS record for
my-service.my-ns
. Pods in the
my-ns
namespace
should be able to find the service by doing a name lookup for
(
would also work).
Pods in other namespaces must qualify the name as
. These names
will resolve to the cluster IP assigned for the Service.
Kubernetes also supports DNS SRV (Service) records for named ports. If the
Service has a port named
http
with the protocol set to
TCP
, you can do a DNS SRV query for
_http._tcp.my-service.my-ns
to discover
the port number for
http
, as well as the IP address.
The Kubernetes DNS server is the only way to access
Services.
You can find more information about
resolution in
DNS Pods and Services
.
Sometimes you don't need load-balancing and a single Service IP. In
this case, you can create what are termed "headless" Services, by explicitly
"None"
for the cluster IP (
).
You can use a headless Service to interface with other service discovery mechanisms,
without being tied to Kubernetes' implementation.
For headless
Services
, a cluster IP is not allocated, kube-proxy does not handle
these Services, and there is no load balancing or proxying done by the platform
for them. How DNS is automatically configured depends on whether the Service has
selectors defined:
With selectors
For headless Services that define selectors, the Kubernetes control plane creates
EndpointSlice objects in the Kubernetes API, and modifies the DNS configuration to return
A or AAAA records (IPv4 or IPv6 addresses) that point directly to the Pods backing
the Service.
Without selectors
For headless Services that do not define selectors, the control plane does
not create EndpointSlice objects. However, the DNS system looks for and configures
either:
DNS CNAME records for
Services.
DNS A / AAAA record

s for all IP addresses of the Service's ready endpoints,
for all Service types other than
.
For IPv4 endpoints, the DNS system creates A records.
For IPv6 endpoints, the DNS system creates AAAA records.
Publishing Services (ServiceTypes)
For some parts of your application (for example, frontends) you may want to expose a
Service onto an external IP address, that's outside of your cluster.
ServiceTypes
allow you to specify what kind of Service you want.
Type
values and their behaviors are:
ClusterIP
: Exposes the Service on a cluster-internal IP. Choosing this value
makes the Service only reachable from within the cluster. This is the
default that is used if you don't explicitly specify a
type
for a Service.
NodePort
: Exposes the Service on each Node's IP at a static port
(the
NodePort
).
To make the node port available, Kubernetes sets up a cluster IP address,
the same as if you had requested a Service of
.
: Exposes the Service externally using a cloud
provider's load balancer.
: Maps the Service to the contents of the
CNAME
record
with its value. No proxying of any kind is set up.
Note:
You need either
kube-dns
version 1.7 or CoreDNS version 0.0.8 or higher
to use the
type.
The
type
field was designed as nested functionality - each level adds to the
previous. This is not strictly required on all cloud providers (for example: Google
Compute Engine does not need to allocate a node port to make
work,
but another cloud provider integration might do). Although strict nesting is not required,
but the Kubernetes API design for Service requires it anyway.
Ingress
to expose your Service.
Ingress is not a Service type, but it acts as the entry point for your cluster.
It lets you consolidate your routing rules into a single resource as it can expose multiple
services under the same IP address.
Type NodePort
type
field to
NodePort
, the Kubernetes control plane
allocates a port from a range specified by
flag (default: 30000-32767).
Each node proxies that port (the same port number on every Node) into your Service.
Your Service reports the allocated port in its
.spec.ports[*].nodePort
field.
Using a NodePort gives you the freedom to set up your own load balancing solution,
to configure environments that are not fully supported by Kubernetes, or even
to expose one or more nodes' IP addresses directly.
For a node port Service, Kubernetes additionally allocates a port (TCP, UDP or
SCTP to match the protocol of the Service). Every node in the cluster configures
itself to listen on that assigned port and to forward traffic to one of the ready
endpoints associated with that Service. You'll be able to contact the
type: NodePort
Service, from outside the cluster, by connecting to any node using the appropriate
protocol (for example: TCP), and the appropriate port (as assigned to that Service).
Choosing your own port
If you want a specific port number, you can specify a value in the
nodePort
field. The control plane will either allocate you that port or report that
the API transaction failed.
This means that you need to take care of possible port collisions yourself.
You also have to use a valid port number, one that's inside the range configured
for NodePort use.
Here is an example manifest for a Service of
that specifies
a NodePort value (30007, in this example).
:
v1
kind
:
Service
metadata
:
name
:
spec
:
type
:
NodePort
selector
:
:
MyApp
ports
:
# By default and for convenience, the `targetPort` is set to the same value as the `port` field.
-
port
:
80
:
80
# Optional field
# By default and for convenience, the Kubernetes control plane will allocate a port from a range (default: 30000-32767)
nodePort
:
30007
Custom IP address configuration for
type: NodeP

ort
Services
You can set up nodes in your cluster to use a particular IP address for serving node port
services. You might want to do this if each node is connected to multiple networks (for example:
one network for application traffic, and another network for traffic between nodes and the
control plane).
If you want to specify particular IP address(es) to proxy the port, you can set the
--nodeport-addresses
flag for kube-proxy or the equivalent
kube-proxy configuration file
to particular IP block(s).
This flag takes a comma-delimited list of IP blocks (e.g.
10.0.0.0/8
,
192.0.2.0/25
)
to specify IP address ranges that kube-proxy should consider as local to this node.
For example, if you start kube-proxy with the
--nodeport-addresses=127.0.0.0/8
flag,
kube-proxy only selects the loopback interface for NodePort Services.
The default for
is an empty list.
This means that kube-proxy should consider all available network interfaces for NodePort.
(That's also compatible with earlier Kubernetes releases.)
Note:
This Service is visible as
&lt;NodeIP&gt;:spec.ports[*].nodePort
and
.spec.clusterIP:spec.ports[*].port
.
If the
flag for kube-proxy or the equivalent field
in the kube-proxy configuration file is set,
&lt;NodeIP&gt;
would be a filtered node IP address (or possibly IP addresses).
Type LoadBalancer
On cloud providers which support external load balancers, setting the
type
field to
provisions a load balancer for your Service.
The actual creation of the load balancer happens asynchronously, and
information about the provisioned balancer is published in the Service's
.status.loadBalancer
field.
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
:
9376
clusterIP
:
10.0.171.239
type
:
status
:
:
ingress
:
-
ip
:
192.0.2.127
Traffic from the external load balancer is directed at the backend Pods.
The cloud provider decides how it is load balanced.
Some cloud providers allow you to specify the
. In those cases, the load-balancer is created
with the user-specified
. If the
field is not specified,
the loadBalancer is set up with an ephemeral IP address. If you specify a
but your cloud provider does not support the feature, the
loadbalancerIP
field that you
set is ignored.
To implement a Service of
, Kubernetes typically starts off
by making the changes that are equivalent to you requesting a Service of
. The cloud-controller-manager component then configures the external load balancer to
forward traffic to that assigned node port.
As an alpha feature
, you can configure a load balanced Service to
omit
assigning a node port, provided that the
cloud provider implementation supports this.
Note:
On
Azure
, if you want to use a user-specified public type
, you first need
to create a static type public IP address resource. This public IP address resource should
be in the same resource group of the other automatically created resources of the cluster.
MC_myResourceGroup_myAKSCluster_eastus
.
Specify the assigned IP address as loadBalancerIP. Ensure that you have updated the
securityGroupName
in the cloud provider configuration file.
For information about troubleshooting
CreatingLoadBalancerFailed
permission issues see,
Use a static IP address with the Azure Kubernetes Service (AKS) load balancer
or
CreatingLoadBalancerFailed on AKS cluster with advanced networking
.
Load balancers with mixed protocol types
By default, for LoadBalancer type of Services, when there is more than one port defined, all
ports must have the same protocol, and the protocol must be one which is supported
by the cloud provider.
The feature gate
(enabled by default

 for the kube-apiserver as of v1.24) allows the use of
different protocols for LoadBalancer type of Services, when there is more than one port defined.
Note:
The set of protocols that can be used for LoadBalancer type of Services is still defined by the cloud provider. If a
cloud provider does not support mixed protocols they will provide only a single protocol.
Disabling load balancer NodePort allocation
You can optionally disable node port allocation for a Service of
, by setting
the field
spec.allocateLoadBalancerNodePorts
to
false
. This should only be used for load balancer implementations
that route traffic directly to pods as opposed to using node ports. By default,
is
true
and type LoadBalancer Services will continue to allocate node ports. If
is set to
false
on an existing Service with allocated node ports, those node ports will
not
be de-allocated automatically.
You must explicitly remove the
nodePorts
entry in every Service port to de-allocate those node ports.
spec.loadBalancerClass
enables you to use a load balancer implementation other than the cloud provider default.
is
nil
and a
type of Service uses
the cloud provider's default load balancer implementation if the cluster is configured with
a cloud provider using the
If
is specified, it is assumed that a load balancer
implementation that matches the specified class is watching for Services.
Any default load balancer implementation (for example, the one provided by
the cloud provider) will ignore Services that have this field set.
can be set on a Service of type
only.
Once set, it cannot be changed.
must be a label-style identifier,
with an optional prefix such as "
internal-vip
" or "
example.com/internal-vip
".
Unprefixed names are reserved for end-users.
Internal load balancer
In a mixed environment it is sometimes necessary to route traffic from Services inside the same
(virtual) network address block.
In a split-horizon DNS environment you would need two Services to be able to route both external
and internal traffic to your endpoints.
To set an internal load balancer, add one of the following annotations to your Service
depending on the cloud Service provider you're using.
Default
GCP
AWS
Azure
IBM Cloud
OpenStack
Baidu Cloud
Tencent Cloud
Alibaba Cloud
OCI
Select one of the tabs.
[...]
metadata
:
name
:
:
cloud.google.com/load-balancer-type
:
"Internal"
[...]
[...]
metadata
:
name
:
:
service.beta.kubernetes.io/aws-load-balancer-internal
:
"true"
[...]
[...]
metadata
:
name
:
:
service.beta.kubernetes.io/azure-load-balancer-internal
:
"true"
[...]
[...]
metadata
:
name
:
:
service.kubernetes.io/ibm-load-balancer-cloud-provider-ip-type
:
"private"
[...]
[...]
metadata
:
name
:
:
service.beta.kubernetes.io/openstack-internal-load-balancer
:
"true"
[...]
[...]
metadata
:
name
:
:
service.beta.kubernetes.io/cce-load-balancer-internal-vpc
:
"true"
[...]
[...]
metadata
:
:
service.kubernetes.io/qcloud-loadbalancer-internal-subnetid
:
subnet-xxxxx
[...]
[...]
metadata
:
:
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type
:
"intranet"
[...]
[...]
metadata
:
name
:
:
service.beta.kubernetes.io/oci-load-balancer-internal
:
true
[...]
TLS support on AWS
For partial TLS / SSL support on clusters running on AWS, you can add three
annotations to a
service:
metadata
:
name
:
:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert
:
arn:aws:acm:us-eas

t-1:123456789012:certificate/12345678-1234-1234-1234-123456789012
The first specifies the ARN of the certificate to use. It can be either a
certificate from a third party issuer that was uploaded to IAM or one created
within AWS Certificate Manager.
metadata
:
name
:
:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol
:
(https|http|ssl|tcp)
The second annotation specifies which protocol a Pod speaks. For HTTPS and
SSL, the ELB expects the Pod to authenticate itself over the encrypted
connection, using a certificate.
HTTP and HTTPS selects layer 7 proxying: the ELB terminates
the connection with the user, parses headers, and injects the
X-Forwarded-For
header with the user's IP address (Pods only see the IP address of the
ELB at the other end of its connection) when forwarding requests.
TCP and SSL selects layer 4 proxying: the ELB forwards traffic without
modifying the headers.
In a mixed-use environment where some ports are secured and others are left unencrypted,
you can use the following annotations:
metadata
:
name
:
:
:
http
service.beta.kubernetes.io/aws-load-balancer-ssl-ports
:
"443,8443"
In the above example, if the Service contained three ports,
80
,
443
, and
8443
, then
443
and
8443
would use the SSL certificate, but
80
would be proxied HTTP.
From Kubernetes v1.9 onwards you can use
predefined AWS SSL policies
with HTTPS or SSL listeners for your Services.
To see which policies are available for use, you can use the
aws
command line tool:
aws elb describe-load-balancer-policies --query
'PolicyDescriptions[].PolicyName'
You can then specify any one of those policies using the
"
service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy
"
annotation; for example:
metadata
:
name
:
:
:
"ELBSecurityPolicy-TLS-1-2-2017-01"
PROXY protocol support on AWS
To enable
support for clusters running on AWS, you can use the following service
annotation:
metadata
:
name
:
:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol
:
"*"
Since version 1.3.0, the use of this annotation applies to all ports proxied by the ELB
and cannot be configured otherwise.
ELB Access Logs on AWS
There are several annotations to manage access logs for ELB Services on AWS.
service.beta.kubernetes.io/aws-load-balancer-access-log-enabled
controls whether access logs are enabled.
service.beta.kubernetes.io/aws-load-balancer-access-log-emit-interval
controls the interval in minutes for publishing the access logs. You can specify
an interval of either 5 or 60 minutes.
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-name
controls the name of the Amazon S3 bucket where load balancer access logs are
stored.
service.beta.kubernetes.io/aws-load-balancer-access-log-s3-bucket-prefix
specifies the logical hierarchy you created for your Amazon S3 bucket.
metadata
:
name
:
:
# Specifies whether access logs are enabled for the load balancer
:
"true"
# The interval for publishing the access logs. You can specify an interval of either 5 or 60 (minutes).
:
"60"
# The name of the Amazon S3 bucket where the access logs are stored
:
"my-bucket"
# The logical hierarchy you created for your Amazon S3 bucket, for example `my-bucket-prefix/prod`
:
"my-bucket-prefix/prod"
Connection Draining on AWS
Connection draining for Classic ELBs can be managed with the annotation
service.beta.kubernetes.io/aws-load-balancer-connection-draining-enabled
set
to

 the value of
"true"
. The annotation
service.beta.kubernetes.io/aws-load-balancer-connection-draining-timeout
can
also be used to set maximum time, in seconds, to keep the existing connections open before
deregistering the instances.
metadata
:
name
:
:
:
"true"
:
"60"
Other ELB annotations
There are other annotations to manage Classic Elastic Load Balancers that are described below.
metadata
:
name
:
:
# The time, in seconds, that the connection is allowed to be idle (no data has been sent
# over the connection) before it is closed by the load balancer
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout
:
"60"
# Specifies whether cross-zone load balancing is enabled for the load balancer
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled
:
"true"
# A comma-separated list of key-value pairs which will be recorded as
# additional tags in the ELB.
service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags
:
"environment=prod,owner=devops"
# The number of successive successful health checks required for a backend to
# be considered healthy for traffic. Defaults to 2, must be between 2 and 10
service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold
:
""
# The number of unsuccessful health checks required for a backend to be
# considered unhealthy for traffic. Defaults to 6, must be between 2 and 10
service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold
:
"3"
# The approximate interval, in seconds, between health checks of an
# individual instance. Defaults to 10, must be between 5 and 300
service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
:
"20"
# The amount of time, in seconds, during which no response means a failed
# health check. This value must be less than the service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval
# value. Defaults to 5, must be between 2 and 60
service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout
:
"5"
# A list of existing security groups to be configured on the ELB created. Unlike the annotation
# service.beta.kubernetes.io/aws-load-balancer-extra-security-groups, this replaces all other
# security groups previously assigned to the ELB and also overrides the creation
# of a uniquely generated security group for this ELB.
# The first security group ID on this list is used as a source to permit incoming traffic to
# target worker nodes (service traffic and health checks).
# If multiple ELBs are configured with the same security group ID, only a single permit line
# will be added to the worker node security groups, that means if you delete any
# of those ELBs it will remove the single permit line and block access for all ELBs that shared the same security group ID.
# This can cause a cross-service outage if not used properly
service.beta.kubernetes.io/aws-load-balancer-security-groups
:
"sg-53fae93f"
# A list of additional security groups to be added to the created ELB, this leaves the uniquely
# generated security group in place, this ensures that every ELB
# has a unique security group ID and a matching permit line to allow traffic to the target worker nodes
# (service traffic and health checks).
# Security groups defined here can be shared between services.
service.beta.kubernetes.io/aws-load-balancer-extra-security-groups
:
"sg-53fae93f,sg-42efd82e"
# A comma separated list of key-value pairs which are used
# to select the target nodes for the load balancer
service.beta.kubernetes.io/aws-load-balancer-target-node-labels
:
"ingress-gw,gw-name=public-api"
Network Load Balancer support on AWS
Kubernetes v1.15 [beta]
To use a Network Load Balancer on AWS, use the annotation
service.beta.kubernetes.io/aws-load-balancer-type
with the value set to
nlb
.
metadata
:
name
:
my-se

rvice
:
:
"nlb"
Note:
NLB only works with certain instance classes; see the
AWS documentation
on Elastic Load Balancing for a list of supported instance types.
Unlike Classic Elastic Load Balancers, Network Load Balancers (NLBs) forward the
client's IP address through to the node. If a Service's
is set to
Cluster
, the client's IP address is not propagated to the end
Pods.
By setting
to
Local
, the client IP addresses is
propagated to the end Pods, but this could result in uneven distribution of
traffic. Nodes without any Pods for a particular LoadBalancer Service will fail
the NLB Target Group's health check on the auto-assigned
and not receive any traffic.
In order to achieve even traffic, either use a DaemonSet or specify a
pod anti-affinity
to not locate on the same node.
You can also use NLB Services with the
internal load balancer
annotation.
In order for client traffic to reach instances behind an NLB, the Node security
groups are modified with the following IP rules:
Rule
Protocol
Port(s)
IpRange(s)
IpRange Description
Health Check
TCP
NodePort(s) (
for
.spec.externalTrafficPolicy = Local
)
Subnet CIDR
kubernetes.io/rule/nlb/health=&lt;loadBalancerName&gt;
Client Traffic
TCP
NodePort(s)
.spec.loadBalancerSourceRanges
(defaults to
0.0.0.0/0
)
kubernetes.io/rule/nlb/client=&lt;loadBalancerName&gt;
MTU Discovery
ICMP
3,4
0.0.0.0/0
)
kubernetes.io/rule/nlb/mtu=&lt;loadBalancerName&gt;
In order to limit which client IP's can access the Network Load Balancer,
specify
.
spec
:
:
-
"143.231.0.0/16"
Note:
If
is not set, Kubernetes
allows traffic from
0.0.0.0/0
to the Node Security Group(s). If nodes have
public IP addresses, be aware that non-NLB traffic can also reach all instances
in those modified security groups.
Further documentation on annotations for Elastic IPs and other common use-cases may be found
in the
AWS Load Balancer Controller documentation
.
Type ExternalName
Services of type ExternalName map a Service to a DNS name, not to a typical selector such as
or
cassandra
. You specify these Services with the
spec.externalName
This Service definition, for example, maps
the
Service in the
prod
namespace to
my.database.example.com
:
:
v1
kind
:
Service
metadata
:
name
:
namespace
:
prod
spec
:
type
:
:
Note:
ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address.
ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName
is intended to specify a canonical DNS name. To hardcode an IP address, consider using
headless Services
.
When looking up the host
my-service.prod.svc.cluster.local
, the cluster DNS Service
returns a
CNAME
record with the value
. Accessing
works in the same way as other Services but with the crucial
difference that redirection happens at the DNS level rather than via proxying or
forwarding. Should you later decide to move your database into your cluster, you
can start its Pods, add appropriate selectors or endpoints, and change the
Service's
type
.
Warning:
You may have trouble using ExternalName for some common protocols, including HTTP and HTTPS.
If you use ExternalName then the hostname used by clients inside your cluster is different from
the name that the ExternalName references.
For protocols that use hostnames this difference may lead to errors or unexpected responses.
HTTP requests will have a
Host:
header that the origin server does not recognize;
TLS servers will not be able to provide a certificate matching the hostname that the client connected to.
Note:
This sectio

n is indebted to the
Kubernetes Tips - Part
1
blog post from
Alen Komljen
.
External IPs
If there are external IPs that route to one or more cluster nodes, Kubernetes Services can be exposed on those
. Traffic that ingresses into the cluster with the external IP (as destination IP), on the Service port,
will be routed to one of the Service endpoints.
are not managed by Kubernetes and are the responsibility
of the cluster administrator.
In the Service spec,
can be specified along with any of the
.
In the example below, "
" can be accessed by clients on "
80.11.12.10:80
" (
externalIP:port
)
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
:
MyApp
ports
:
-
name
:
http
protocol
:
TCP
port
:
80
:
9376
:
-
80.11.12.10
Session stickiness
If you want to make sure that connections from a particular client are passed to
the same Pod each time, you can configure session affinity based on the client's
IP address. Read
session affinity
API Object
Service is a top-level resource in the Kubernetes REST API. You can find more details
about the
Service API object
.
Read
to learn about the
mechanism Kubernetes provides to expose a Service with a virtual IP address.
Follow the
tutorial
Read about
Ingress
Read about
For more context:
Read
Read the
Read the
for the Endpoints API
Read the
for the EndpointSlice API
6.2 - Ingress
Make your HTTP (or HTTPS) network service available using a protocol-aware configuration mechanism, that understands web concepts like URIs, hostnames, paths, and more. The Ingress concept lets you map traffic to different backends based on rules you define via the Kubernetes API.
For clarity, this guide defines the following terms:
Node: A worker machine in Kubernetes, part of a cluster.
Cluster: A set of Nodes that run containerized applications managed by Kubernetes. For this example, and in most common Kubernetes deployments, nodes in the cluster are not part of the public internet.
Edge router: A router that enforces the firewall policy for your cluster. This could be a gateway managed by a cloud provider or a physical piece of hardware.
Cluster network: A set of links, logical or physical, that facilitate communication within a cluster according to the Kubernetes
.
Service: A Kubernetes
Service
that identifies a set of Pods using
label
selectors. Unless mentioned otherwise, Services are assumed to have virtual IPs only routable within the cluster network.
What is Ingress?
Ingress
exposes HTTP and HTTPS routes from outside the cluster to
services
within the cluster.
Traffic routing is controlled by rules defined on the Ingress resource.
Here is a simple example where an Ingress sends all its traffic to one Service:
Figure. Ingress
An Ingress may be configured to give Services externally-reachable URLs, load balance traffic, terminate SSL / TLS, and offer name-based virtual hosting. An
Ingress controller
is responsible for fulfilling the Ingress, usually with a load balancer, though it may also configure your edge router or additional frontends to help handle the traffic.
An Ingress does not expose arbitrary ports or protocols. Exposing services other than HTTP and HTTPS to the internet typically
uses a service of type
Service.Type=NodePort
or
Service.Type=LoadBalancer
.
You must have an
to satisfy an Ingress. Only creating an Ingress resource has no effect.
You may need to deploy a

n Ingress controller such as
ingress-nginx
. You can choose from a number of
.
Ideally, all Ingress controllers should fit the reference specification. In reality, the various Ingress
controllers operate slightly differently.
Note:
Make sure you review your Ingress controller's documentation to understand the caveats of choosing it.
The Ingress resource
A minimal Ingress resource example:
service/networking/minimal-ingress.yaml
:
kind
:
Ingress
metadata
:
name
:
minimal-ingress
:
nginx.ingress.kubernetes.io/rewrite-target
:
/
spec
:
:
nginx-example
rules
:
-
http
:
paths
:
-
path
:
/testpath
pathType
:
Prefix
backend
:
service
:
name
:
test
port
:
number
:
80
An Ingress needs
,
kind
,
metadata
and
spec
fields.
The name of an Ingress object must be a valid
.
,
configuring containers
,
.
Ingress frequently uses annotations to configure some options depending on the Ingress controller, an example of which
is the
rewrite-target annotation
.
Different
support different annotations. Review the documentation for
your choice of Ingress controller to learn which annotations are supported.
The Ingress
spec
has all the information needed to configure a load balancer or proxy server. Most importantly, it
contains a list of rules matched against all incoming requests. Ingress resource only supports rules
for directing HTTP(S) traffic.
If the
is omitted, a
default Ingress class
should be defined.
There are some ingress controllers, that work without the definition of a
default
. For example, the Ingress-NGINX controller can be
configured with a
flag
--watch-ingress-without-class
. It is
though, to specify the
default
as shown
below
.
Ingress rules
Each HTTP rule contains the following information:
An optional host. In this example, no host is specified, so the rule applies to all inbound
HTTP traffic through the IP address specified. If a host is provided (for example,
foo.bar.com), the rules apply to that host.
A list of paths (for example,
/testpath
), each of which has an associated
backend defined with a
and a
or
. Both the host and path must match the content of an
incoming request before the load balancer directs traffic to the referenced
Service.
A backend is a combination of Service and port names as described in the
Service doc
or a
custom resource backend
by way of a
CRD
. HTTP (and HTTPS) requests to the
Ingress that match the host and path of the rule are sent to the listed backend.
A
is often configured in an Ingress controller to service any requests that do not
match a path in the spec.
DefaultBackend
An Ingress with no rules sends all traffic to a single default backend and
.spec.defaultBackend
is the backend that should handle requests in that case.
The
is conventionally a configuration option of the
and
is not specified in your Ingress resources.
If no
.spec.rules
are specified,
If
is not set, the handling of requests that do not match any of the rules will be up to the
ingress controller (consult the documentation for your ingress controller to find out how it handles this case).
If none of the hosts or paths match the HTTP request in the Ingress objects, the traffic is
routed to your default backend.
Resource backends
A
Resource
backend is an ObjectRef to another Kubernetes resource within the
same namespace as the Ingress object. A
Resource
is a mutually exclusive
setting with Service, and will fail validation if both are specified. A common
usage for a
Resource
backend is to ingress data to an object storage backend
with static assets.
service/networking/ingress-resource-backend.yaml

:
kind
:
Ingress
metadata
:
name
:
ingress-resource-backend
spec
:
:
resource
:
apiGroup
:
k8s.example.com
kind
:
StorageBucket
name
:
static-assets
rules
:
-
http
:
paths
:
-
path
:
/icons
pathType
:
backend
:
resource
:
apiGroup
:
kind
:
name
:
icon-assets
After creating the Ingress above, you can view it with the following command:
kubectl describe ingress ingress-resource-backend
Name:             ingress-resource-backend
Address:
Default backend:  APIGroup: k8s.example.com, Kind: StorageBucket, Name: static-assets
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /icons   APIGroup: k8s.example.com, Kind: StorageBucket, Name: icon-assets
Events:       &lt;none&gt;
Path types
Each path in an Ingress is required to have a corresponding path type. Paths
that do not include an explicit
pathType
will fail validation. There are three
supported path types:
: With this path type, matching is up to the
IngressClass. Implementations can treat this as a separate
pathType
or treat
it identically to
Prefix
or
Exact
path types.
Exact
: Matches the URL path exactly and with case sensitivity.
Prefix
: Matches based on a URL path prefix split by
/
. Matching is case
sensitive and done on a path element by element basis. A path element refers
to the list of labels in the path split by the
/
separator. A request is a
match for path
p
if every
p
is an element-wise prefix of
p
of the
request path.
Note:
If the last element of the path is a substring of the last
element in request path, it is not a match (for example:
/foo/bar
matches
/foo/bar/baz
, but does not match
/foo/barbaz
).
Examples
Kind
Path(s)
Request path(s)
Matches?
Prefix
/
(all paths)
Yes
Exact
/foo
/foo
Yes
Exact
/foo
/bar
No
Exact
/foo
/foo/
No
Exact
/foo/
/foo
No
Prefix
/foo
/foo
,
/foo/
Yes
Prefix
/foo/
/foo
,
/foo/
Yes
Prefix
/aaa/bb
/aaa/bbb
No
Prefix
/aaa/bbb
/aaa/bbb
Yes
Prefix
/aaa/bbb/
/aaa/bbb
Yes, ignores trailing slash
Prefix
/aaa/bbb
/aaa/bbb/
Yes, matches trailing slash
Prefix
/aaa/bbb
/aaa/bbb/ccc
Yes, matches subpath
Prefix
/aaa/bbb
/aaa/bbbxyz
No, does not match string prefix
Prefix
/
,
/aaa
/aaa/ccc
Yes, matches
/aaa
prefix
Prefix
/
,
/aaa
,
/aaa/bbb
/aaa/bbb
/aaa/bbb
prefix
Prefix
/
,
/aaa
,
/aaa/bbb
/ccc
/
prefix
Prefix
/aaa
/ccc
No, uses default backend
Mixed
/foo
(Prefix),
/foo
(Exact)
/foo
Yes, prefers Exact
Multiple matches
In some cases, multiple paths within an Ingress will match a request. In those
cases precedence will be given first to the longest matching path. If two paths
are still equally matched, precedence will be given to paths with an exact path
type over prefix path type.
Hostname wildcards
Hosts can be precise matches (for example “
foo.bar.com
”) or a wildcard (for
example “
*.foo.com
”). Precise matches require that the HTTP
host
header
matches the
host
field. Wildcard matches require the HTTP
host
header is
equal to the suffix of the wildcard rule.
Host
Host header
Match?
*.foo.com
bar.foo.com
Matches based on shared suffix
*.foo.com
baz.bar.foo.com
No match, wildcard only covers a single DNS label
*.foo.com
foo.com
service/networking/ingress-wildcard-host.yaml
:
kind
:
Ingress
metadata
:
name
:
ingress-wildcard-host
spec
:
rules
:
-
host
:
"foo.bar.com"
http
:
paths
:
-
pathType
:
Prefix
path
:
"/bar"
backend
:
service
:
name
:
service1
port
:
number
:
80
-
host
:
"*.foo.com"
http
:
paths
:
-
pathType
:
Prefix
path
:
"/foo"
backend
:
service
:
name
:
service2
port
:
number
:
80
Ingress class
Ingresses can be implemented by different controllers, often with different
configuration. Each Ingress should specify a class, a reference to an
IngressClass resource that contains additional configuration including the name
of the cont

roller that should implement the class.
service/networking/external-lb.yaml
:
kind
:
metadata
:
name
:
external-lb
spec
:
:
example.com/ingress-controller
:
apiGroup
:
kind
:
IngressParameters
name
:
The
.spec.parameters
field of an IngressClass lets you reference another
resource that provides configuration related to that IngressClass.
The specific type of parameters to use depends on the ingress controller
that you specify in the
.spec.controller
field of the IngressClass.
IngressClass scope
Depending on your ingress controller, you may be able to use parameters
that you set cluster-wide, or just for one namespace.
Cluster
The default scope for IngressClass parameters is cluster-wide.
field and don't set
.spec.parameters.scope
, or if you set
to
Cluster
, then the IngressClass refers to a cluster-scoped resource.
The
kind
(in combination the
apiGroup
) of the parameters
refers to a cluster-scoped API (possibly a custom resource), and
the
name
of the parameters identifies a specific cluster scoped
resource for that API.
---
:
kind
:
metadata
:
name
:
external-lb-1
spec
:
:
:
# The parameters for this IngressClass are specified in a
# ClusterIngressParameter (API group k8s.example.net) named
# "external-config-1". This definition tells Kubernetes to
# look for a cluster-scoped parameter resource.
scope
:
Cluster
apiGroup
:
k8s.example.net
kind
:
ClusterIngressParameter
name
:
external-config-1
field and set
to
Namespace
, then the IngressClass refers
to a namespaced-scoped resource. You must also set the
namespace
field within
to the namespace that contains
the parameters you want to use.
The
kind
apiGroup
refers to a namespaced API (for example: ConfigMap), and
the
name
of the parameters identifies a specific resource
in the namespace you specified in
namespace
.
Namespace-scoped parameters help the cluster operator delegate control over the
configuration (for example: load balancer settings, API gateway definition)
that is used for a workload. If you used a cluster-scoped parameter then either:
the cluster operator team needs to approve a different team's changes every
time there's a new configuration change being applied.
the cluster operator must define specific access controls, such as
RBAC
roles and bindings, that let
the application team make changes to the cluster-scoped parameters resource.
The IngressClass API itself is always cluster-scoped.
Here is an example of an IngressClass that refers to parameters that are
namespaced:
---
:
kind
:
metadata
:
name
:
external-lb-2
spec
:
:
:
# The parameters for this IngressClass are specified in an
# IngressParameter (API group k8s.example.com) named "external-config",
# that's in the "external-configuration" namespace.
scope
:
Namespace
apiGroup
:
kind
:
IngressParameter
namespace
:
external-configuration
name
:
external-config
Deprecated annotation
Before the IngressClass resource and
field were added in
Kubernetes 1.18, Ingress classes were specified with a
annotation on the Ingress. This annotation was
never formally defined, but was widely supported by Ingress controllers.
The newer
field on Ingresses is a replacement for that
annotation, but is not a direct equivalent. While the annotation was generally
used to reference the name of the Ingress controller that should implement the
Ingress, the field is a reference to an IngressClass resource that contains
additional Ingress config

uration, including the name of the Ingress controller.
Default IngressClass
You can mark a particular IngressClass as default for your cluster. Setting the
annotation to
true
on an
IngressClass resource will ensure that new Ingresses without an
field specified will be assigned this default IngressClass.
Caution:
If you have more than one IngressClass marked as the default for your cluster,
the admission controller prevents creating new Ingress objects that don't have
an
specified. You can resolve this by ensuring that at most 1
IngressClass is marked as default in your cluster.
default
flag
. It is
default
:
service/networking/default-ingressclass.yaml
:
kind
:
metadata
:
labels
:
:
name
:
:
:
"true"
spec
:
:
k8s.io/ingress-nginx
Types of Ingress
Ingress backed by a single Service
There are existing Kubernetes concepts that allow you to expose a single Service
(see
alternatives
). You can also do this with an Ingress by specifying a
default backend
with no rules.
service/networking/test-ingress.yaml
:
kind
:
Ingress
metadata
:
name
:
test-ingress
spec
:
:
service
:
name
:
test
port
:
number
:
80
If you create it using
you should be able to view the state
of the Ingress you added:
kubectl get ingress test-ingress
NAME           CLASS         HOSTS   ADDRESS         PORTS   AGE
test-ingress   external-lb   *       203.0.113.123   80      59s
Where
203.0.113.123
is the IP allocated by the Ingress controller to satisfy
this Ingress.
Note:
Ingress controllers and load balancers may take a minute or two to allocate an IP address.
Until that time, you often see the address listed as
.
Simple fanout
A fanout configuration routes traffic from a single IP address to more than one Service,
based on the HTTP URI being requested. An Ingress allows you to keep the number of load balancers
down to a minimum. For example, a setup like:
Figure. Ingress Fan Out
would require an Ingress such as:
service/networking/simple-fanout-example.yaml
:
kind
:
Ingress
metadata
:
name
:
simple-fanout-example
spec
:
rules
:
-
host
:
http
:
paths
:
-
path
:
/foo
pathType
:
Prefix
backend
:
service
:
name
:
service1
port
:
number
:
4200
-
path
:
/bar
pathType
:
Prefix
backend
:
service
:
name
:
service2
port
:
number
:
8080
When you create the Ingress with
:
kubectl describe ingress simple-fanout-example
Name:             simple-fanout-example
Address:          178.91.123.132
Default backend:  default-http-backend:80 (10.8.2.3:8080)
Rules:
  Host         Path  Backends
  ----         ----  --------
  foo.bar.com
               /foo   service1:4200 (10.8.0.90:4200)
               /bar   service2:8080 (10.8.0.91:8080)
Events:
  Type     Reason  Age                From                     Message
  ----     ------  ----               ----                     -------
  Normal   ADD     22s                loadbalancer-controller  default/test
The Ingress controller provisions an implementation-specific load balancer
that satisfies the Ingress, as long as the Services (
service1
,
service2
) exist.
When it has done so, you can see the address of the load balancer at the
Address field.
Note:
you are using, you may need to create a default-http-backend
Service
.
Name based virtual hosting
Name-based virtual hosts support routing HTTP traffic to multiple host names at the same IP address.
Figure. Ingre

ss Name Based Virtual hosting
The following Ingress tells the backing load balancer to route requests based on
the
.
service/networking/name-virtual-host-ingress.yaml
:
kind
:
Ingress
metadata
:
name
:
name-virtual-host-ingress
spec
:
rules
:
-
host
:
http
:
paths
:
-
pathType
:
Prefix
path
:
"/"
backend
:
service
:
name
:
service1
port
:
number
:
80
-
host
:
http
:
paths
:
-
pathType
:
Prefix
path
:
"/"
backend
:
service
:
name
:
service2
port
:
number
:
80
If you create an Ingress resource without any hosts defined in the rules, then any
web traffic to the IP address of your Ingress controller can be matched without a name based
virtual host being required.
For example, the following Ingress routes traffic
requested for
first.bar.com
to
service1
,
second.bar.com
to
service2
, and any traffic whose request host header doesn't match
and
to
service3
.
service/networking/name-virtual-host-ingress-no-third-host.yaml
:
kind
:
Ingress
metadata
:
name
:
name-virtual-host-ingress-no-third-host
spec
:
rules
:
-
host
:
http
:
paths
:
-
pathType
:
Prefix
path
:
"/"
backend
:
service
:
name
:
service1
port
:
number
:
80
-
host
:
http
:
paths
:
-
pathType
:
Prefix
path
:
"/"
backend
:
service
:
name
:
service2
port
:
number
:
80
-
http
:
paths
:
-
pathType
:
Prefix
path
:
"/"
backend
:
service
:
name
:
service3
port
:
number
:
80
TLS
You can secure an Ingress by specifying a
Secret
that contains a TLS private key and certificate. The Ingress resource only
supports a single TLS port, 443, and assumes TLS termination at the ingress point
(traffic to the Service and its Pods is in plaintext).
If the TLS configuration section in an Ingress specifies different hosts, they are
multiplexed on the same port according to the hostname specified through the
SNI TLS extension (provided the Ingress controller supports SNI). The TLS secret
must contain keys named
tls.crt
and
tls.key
that contain the certificate
and private key to use for TLS. For example:
:
v1
kind
:
Secret
metadata
:
name
:
testsecret-tls
namespace
:
default
data
:
tls.crt
:
base64 encoded cert
tls.key
:
base64 encoded key
type
:
kubernetes.io/tls
Referencing this secret in an Ingress tells the Ingress controller to
secure the channel from the client to the load balancer using TLS. You need to make
sure the TLS secret you created came from a certificate that contains a Common
Name (CN), also known as a Fully Qualified Domain Name (FQDN) for
https-example.foo.com
.
Note:
Keep in mind that TLS will not work on the default rule because the
certificates would have to be issued for all the possible sub-domains. Therefore,
hosts
in the
tls
section need to explicitly match the
host
in the
rules
section.
service/networking/tls-example-ingress.yaml
:
kind
:
Ingress
metadata
:
name
:
tls-example-ingress
spec
:
tls
:
-
hosts
:
- https-example.foo.com
:
rules
:
-
host
:
http
:
paths
:
-
path
:
/
pathType
:
Prefix
backend
:
service
:
name
:
service1
port
:
number
:
80
Note:
There is a gap between TLS features supported by various Ingress
controllers. Please refer to documentation on
nginx
,
GCE
, or any other
platform specific Ingress controller to understand how TLS works in your environment.
Load balancing
An Ingress controller is bootstrapped with some load balancing policy settings
that it applies to all Ingress, such as the load balancing algorithm, backend
weight scheme, and others. More advanced load balancing concepts
(e.g. persistent sessions, dynamic weights) are not yet exposed through the
Ingress. You can instead get these features through the load balancer used for
a Service.
It's also worth noting that even though health checks are not exposed directly
through the Ingress, there exist parallel concepts in Kubernetes such as

s
that allow you to achieve the same end result. Please review the controller
specific documentation to see how they handle health checks (for example:
nginx
, or
GCE
).
Updating an Ingress
To update an existing Ingress to add a new Host, you can update it by editing the resource:
kubectl describe ingress
test
Name:             test
Rules:
               /foo   service1:80 (10.8.0.90:80)
Annotations:
  nginx.ingress.kubernetes.io/rewrite-target:  /
Events:
  Normal   ADD     35s                loadbalancer-controller  default/test
kubectl edit ingress
test
This pops up an editor with the existing configuration in YAML format.
Modify it to include the new Host:
spec
:
rules
:
-
host
:
http
:
paths
:
-
backend
:
service
:
name
:
service1
port
:
number
:
80
path
:
/foo
pathType
:
Prefix
-
host
:
bar.baz.com
http
:
paths
:
-
backend
:
service
:
name
:
service2
port
:
number
:
80
path
:
/foo
pathType
:
Prefix
..
After you save your changes, kubectl updates the resource in the API server, which tells the
Ingress controller to reconfigure the load balancer.
Verify this:
test
Name:             test
Rules:
  bar.baz.com
               /foo   service2:80 (10.8.0.91:80)
Events:
  Normal   ADD     45s                loadbalancer-controller  default/test
You can achieve the same outcome by invoking
on a modified Ingress YAML file.
Failing across availability zones
Techniques for spreading traffic across failure domains differ between cloud providers.
Please check the documentation of the relevant
You can expose a Service in multiple ways that don't directly involve the Ingress resource:
Use
Use
Ingress
API
Set up Ingress on Minikube with the NGINX Controller
6.3 - Ingress Controllers
In order for an
Ingress
to work in your cluster, there must be an
ingress controller
running. You need to select at least one ingress controller and make sure it is set up in your cluster. This page lists common ingress controllers that you can deploy.
In order for the Ingress resource to work, the cluster must have an ingress controller running.
Unlike other types of controllers which run as part of the
binary, Ingress controllers
are not started automatically with a cluster. Use this page to choose the ingress controller implementation
that best fits your cluster.
Kubernetes as a project supports and maintains
AWS
,
GCE
, and
nginx
ingress controllers.
Additional controllers
Note:
AKS Application Gateway Ingress Controller
is an ingress controller that configures the
Azure Application Gateway
.
Ambassador
API Gateway is an
Envoy
-based ingress
Apache APISIX ingress controller
is an
Apache APISIX
-based ingress controller.
Avi Kubernetes Operator
provides L4-L7 load

-balancing using
VMware NSX Advanced Load Balancer
.
BFE Ingress Controller
is a
BFE
The
Citrix ingress controller
works with
Citrix Application Delivery Controller.
Contour
is an
Envoy
based ingress controller.
EnRoute
is an
Envoy
based API gateway that can run as an ingress controller.
Easegress IngressController
is an
Easegress
F5 BIG-IP
Container Ingress Services for Kubernetes
lets you use an Ingress to configure F5 BIG-IP virtual servers.
Gloo
is an open-source ingress controller based on
Envoy
,
which offers API gateway functionality.
HAProxy Ingress
is an ingress controller for
HAProxy
.
The
HAProxy Ingress Controller for Kubernetes
is also an ingress controller for
HAProxy
.
Istio Ingress
is an
Istio
The
Kong Ingress Controller for Kubernetes
is an ingress controller driving
Kong Gateway
.
Kusk Gateway
is an OpenAPI-driven ingress controller based on
Envoy
.
The
NGINX Ingress Controller for Kubernetes
works with the
NGINX
webserver (as a proxy).
The
Pomerium Ingress Controller
is based on
Pomerium
, which offers context-aware access policy.
Skipper
HTTP router and reverse proxy for service composition, including use cases like Kubernetes Ingress, designed as a library to build your custom proxy.
The
Traefik Kubernetes Ingress provider
is an
ingress controller for the
Traefik
proxy.
Tyk Operator
extends Ingress with Custom Resources to bring API Management capabilities to Ingress. Tyk Operator works with the Open Source Tyk Gateway &amp; Tyk Cloud control plane.
Voyager
HAProxy
.
Using multiple Ingress controllers
You may deploy any number of ingress controllers using
ingress class
within a cluster. Note the
of your ingress class resource. When you create an ingress you would need that name to specify the
field on your Ingress object (refer to
IngressSpec v1 reference
).
is a replacement of the older
annotation method
.
If you do not specify an IngressClass for an Ingress, and your cluster has exactly one IngressClass marked as default, then Kubernetes
applies
the cluster's default IngressClass to the Ingress.
You mark an IngressClass as default by setting the
on that IngressClass, with the string value
"true"
.
Ideally, all ingress controllers should fulfill this specification, but the various ingress
Note:
Make sure you review your ingress controller's documentation to understand the caveats of choosing it.
Ingress
.
.
6.4 - EndpointSlices
The EndpointSlice API is the mechanism that Kubernetes uses to let your Service scale to handle large numbers of backends, and allows the cluster to update its list of healthy backends efficiently.
API provides a way to track network endpoints
within a Kubernetes cluster. EndpointSlices offer a more scalable and extensible
alternative to
Endpoints
.
EndpointSlice API
In Kubernetes, an EndpointSlice contains references to a set of network
endpoints. The control plane automatically creates EndpointSlices
for any Kubernetes Service that has a
selector
specified. These EndpointSlices include
references to all the Pods that match the Service selector. EndpointSlices group
network endpoints together by unique combinations of protocol, port number, and
Service name.
The name of a EndpointSlice object must be a valid
.
As an example, here's a sample EndpointSlice object, that's owned by the
example
Kubernetes Service.
:
kind
:
metadata
:
name
:
example-abc
labels
:
:
example
:
IPv4
ports
:
-
name
:
http
protocol
:
TCP
port
:
80
endpoints
:
-
addresses
:
-
"10.1.2

.3"
:
ready
:
true
hostname
:
pod-1
nodeName
:
node-1
zone
:
us-west2-a
By default, the control plane creates and manages EndpointSlices to have no
more than 100 endpoints each. You can configure this with the
--max-endpoints-per-slice
flag, up to a maximum of 1000.
EndpointSlices can act as the source of truth for
when it comes to
how to route internal traffic.
Address types
EndpointSlices support three address types:
IPv4
IPv6
FQDN (Fully Qualified Domain Name)
Each
object represents a specific IP address type. If you have
a Service that is available via IPv4 and IPv6, there will be at least two
objects (one for IPv4, and one for IPv6).
The EndpointSlice API stores conditions about endpoints that may be useful for consumers.
The three conditions are
ready
,
serving
, and
.
Ready
ready
is a condition that maps to a Pod's
Ready
condition. A running Pod with the
Ready
condition set to
True
should have this EndpointSlice condition also set to
true
. For
compatibility reasons,
ready
is NEVER
true
when a Pod is terminating. Consumers should refer
to the
serving
condition to inspect the readiness of terminating Pods. The only exception to
this rule is for Services with
spec.publishNotReadyAddresses
set to
true
. Endpoints for these
Services will always have the
ready
true
.
Serving
serving
is identical to the
ready
condition, except it does not account for terminating states.
Consumers of the EndpointSlice API should check this condition if they care about pod readiness while
the pod is also terminating.
Note:
Although
serving
is almost identical to
ready
, it was added to prevent break the existing meaning
of
ready
. It may be unexpected for existing clients if
ready
could be
true
for terminating
endpoints, since historically terminating endpoints were never included in the Endpoints or
EndpointSlice API to begin with. For this reason,
ready
is
always
false
endpoints, and a new condition
serving
was added in v1.20 so that clients can track readiness
for terminating pods independent of the existing semantics for
ready
.
is a condition that indicates whether an endpoint is terminating.
For pods, this is any pod that has a deletion timestamp set.
Topology information
Each endpoint within an EndpointSlice can contain relevant topology information.
The topology information includes the location of the endpoint and information
about the corresponding Node and zone. These are available in the following
per endpoint fields on EndpointSlices:
nodeName
- The name of the Node this endpoint is on.
zone
- The zone this endpoint is in.
Note:
In the v1 API, the per endpoint
topology
was effectively removed in favor of
the dedicated fields
nodeName
and
zone
.
Setting arbitrary topology fields on the
endpoint
field of an
resource has been deprecated and is not supported in the v1 API.
Instead, the v1 API supports setting individual
nodeName
and
zone
fields.
These fields are automatically translated between API versions. For example, the
value of the
"topology.kubernetes.io/zone"
key in the
topology
field in
the v1beta1 API is accessible as the
zone
field in the v1 API.
Most often, the control plane (specifically, the endpoint slice
) creates and
manages EndpointSlice objects. There are a variety of other use cases for
EndpointSlices, such as service mesh implementations, that could result in other
entities or controllers managing additional sets of EndpointSlices.
To ensure that multiple entities can manage EndpointSlices without interfering
with each other, Kubernetes defines the
label
, which indicates the entity managing
an EndpointSlice.
The endpoint slice controller sets
endpointslice-controller.k8s.io
as the value
for this label on all Endp

ointSlices it manages. Other entities managing
EndpointSlices should also set a unique value for this label.
Ownership
In most use cases, EndpointSlices are owned by the Service that the endpoint
slice object tracks endpoints for. This ownership is indicated by an owner
reference on each EndpointSlice as well as a
label that enables simple lookups of all EndpointSlices belonging to a Service.
EndpointSlice mirroring
In some cases, applications create custom Endpoints resources. To ensure that
these applications do not need to concurrently write to both Endpoints and
EndpointSlice resources, the cluster's control plane mirrors most Endpoints
resources to corresponding EndpointSlices.
The control plane mirrors Endpoints resources unless:
the Endpoints resource has a
label
set to
true
.
control-plane.alpha.kubernetes.io/leader
the corresponding Service resource does not exist.
the corresponding Service resource has a non-nil selector.
Individual Endpoints resources may translate into multiple EndpointSlices. This
will occur if an Endpoints resource has multiple subsets or includes endpoints
with multiple IP families (IPv4 and IPv6). A maximum of 1000 addresses per
subset will be mirrored to EndpointSlices.
Distribution of EndpointSlices
Each EndpointSlice has a set of ports that applies to all endpoints within the
resource. When named ports are used for a Service, Pods may end up with
different target port numbers for the same named port, requiring different
EndpointSlices. This is similar to the logic behind how subsets are grouped
with Endpoints.
The control plane tries to fill EndpointSlices as full as possible, but does not
actively rebalance them. The logic is fairly straightforward:
Iterate through existing EndpointSlices, remove endpoints that are no longer
desired and update matching endpoints that have changed.
Iterate through EndpointSlices that have been modified in the first step and
fill them up with any new endpoints needed.
If there's still new endpoints left to add, try to fit them into a previously
unchanged slice and/or create new ones.
Importantly, the third step prioritizes limiting EndpointSlice updates over a
perfectly full distribution of EndpointSlices. As an example, if there are 10
new endpoints to add and 2 EndpointSlices with room for 5 more endpoints each,
this approach will create a new EndpointSlice instead of filling up the 2
existing EndpointSlices. In other words, a single EndpointSlice creation is
preferrable to multiple EndpointSlice updates.
With kube-proxy running on each Node and watching EndpointSlices, every change
to an EndpointSlice becomes relatively expensive since it will be transmitted to
every Node in the cluster. This approach is intended to limit the number of
changes that need to be sent to every Node, even if it may result with multiple
EndpointSlices that are not full.
In practice, this less than ideal distribution should be rare. Most changes
processed by the EndpointSlice controller will be small enough to fit in an
existing EndpointSlice, and if not, a new EndpointSlice is likely going to be
necessary soon anyway. Rolling updates of Deployments also provide a natural
repacking of EndpointSlices with all Pods and their corresponding endpoints
getting replaced.
Duplicate endpoints
Due to the nature of EndpointSlice changes, endpoints may be represented in more
than one EndpointSlice at the same time. This naturally occurs as changes to
different EndpointSlice objects can arrive at the Kubernetes client watch / cache
at different times.
Note:
Clients of the EndpointSlice API must iterate through all the existing EndpointSlices
associated to a Service and build a complete list of unique network endpoints. It is
important to mention that endpoints may be duplicated in different EndointSlices.
You can find a reference implementation for how to perform this endpoint aggregati

on
and deduplication as part of the
EndpointSliceCache
code within
.
Comparison with Endpoints
The original Endpoints API provided a simple and straightforward way of
tracking network endpoints in Kubernetes. As Kubernetes clusters
and
Services
grew to handle
more traffic and to send more traffic to more backend Pods, the
limitations of that original API became more visible.
Most notably, those included challenges with scaling to larger numbers of
network endpoints.
Since all network endpoints for a Service were stored in a single Endpoints
object, those Endpoints objects could get quite large. For Services that stayed
stable (the same set of endpoints over a long period of time) the impact was
less noticeable; even then, some use cases of Kubernetes weren't well served.
When a Service had a lot of backend endpoints and the workload was either
scaling frequently, or rolling out new changes frequently, each update to
the single Endpoints object for that Service meant a lot of traffic between
Kubernetes cluster components (within the control plane, and also between
nodes and the API server). This extra traffic also had a cost in terms of
CPU use.
With EndpointSlices, adding or removing a single Pod triggers the same
number
of updates to clients that are watching for changes, but the size of those
update message is much smaller at large scale.
EndpointSlices also enabled innovation around new features such dual-stack
networking and topology-aware routing.
Follow the
tutorial
Read the
Read the
6.5 - Network Policies
If you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), NetworkPolicies allow you to specify rules for traffic flow within your cluster, and also between Pods and the outside world. Your cluster must use a network plugin that supports NetworkPolicy enforcement.
If you want to control traffic flow at the IP address or port level (OSI layer 3 or 4), then you
might consider using Kubernetes NetworkPolicies for particular applications in your cluster.
NetworkPolicies are an application-centric construct which allow you to specify how a
pod
is allowed to communicate with various network
"entities" (we use the word "entity" here to avoid overloading the more common terms such as
"endpoints" and "services", which have specific Kubernetes connotations) over the network.
NetworkPolicies apply to a connection with a pod on one or both ends, and are not relevant to
other connections.
The entities that a Pod can communicate with are identified through a combination of the following
3 identifiers:
Other pods that are allowed (exception: a pod cannot block access to itself)
Namespaces that are allowed
IP blocks (exception: traffic to and from the node where a Pod is running is always allowed,
regardless of the IP address of the Pod or the node)
When defining a pod- or namespace- based NetworkPolicy, you use a
selector
to specify what traffic is allowed to
and from the Pod(s) that match the selector.
Meanwhile, when IP based NetworkPolicies are created, we define policies based on IP blocks (CIDR ranges).
Network policies are implemented by the
.
To use network policies, you must be using a networking solution which supports NetworkPolicy.
Creating a NetworkPolicy resource without a controller that implements it will have no effect.
The Two Sorts of Pod Isolation
There are two sorts of isolation for a pod: isolation for egress, and isolation for ingress.
They concern what connections may be established. "Isolation" here is not absolute, rather it
means "some restrictions apply". The alternative, "non-isolated for $direction", means that no
restrictions apply in the stated direction. The two sorts of isolation (or not) are declared
independently, and are both relevant for a connection from one pod to another.
By default, a pod is non-isolat

ed for egress; all outbound connections are allowed.
A pod is isolated for egress if there is any NetworkPolicy that both selects the pod and has
"Egress" in its
; we say that such a policy applies to the pod for egress.
When a pod is isolated for egress, the only allowed connections from the pod are those allowed by
the
egress
list of some NetworkPolicy that applies to the pod for egress.
The effects of those
egress
lists combine additively.
By default, a pod is non-isolated for ingress; all inbound connections are allowed.
A pod is isolated for ingress if there is any NetworkPolicy that both selects the pod and
has "Ingress" in its
; we say that such a policy applies to the pod for ingress.
When a pod is isolated for ingress, the only allowed connections into the pod are those from
the pod's node and those allowed by the
ingress
list of some NetworkPolicy that applies to
the pod for ingress. The effects of those
ingress
Network policies do not conflict; they are additive. If any policy or policies apply to a given
pod for a given direction, the connections allowed in that direction from that pod is the union of
what the applicable policies allow. Thus, order of evaluation does not affect the policy result.
For a connection from a source pod to a destination pod to be allowed, both the egress policy on
the source pod and the ingress policy on the destination pod need to allow the connection. If
either side does not allow the connection, it will not happen.
The NetworkPolicy resource
See the
reference for a full definition of the resource.
An example NetworkPolicy might look like this:
service/networking/networkpolicy.yaml
:
kind
:
metadata
:
name
:
test-network-policy
namespace
:
default
spec
:
:
:
role
:
db
:
- Ingress
- Egress
ingress
:
-
from
:
-
ipBlock
:
cidr
:
172.17.0.0
/16
except
:
-
172.17.1.0
/24
-
:
:
project
:
myproject
-
:
:
role
:
frontend
ports
:
-
protocol
:
TCP
port
:
6379
egress
:
-
to
:
-
ipBlock
:
cidr
:
10.0.0.0
/24
ports
:
-
protocol
:
TCP
port
:
5978
Note:
POSTing this to the API server for your cluster will have no effect unless your chosen networking
solution supports network policy.
Mandatory Fields
: As with all other Kubernetes config, a NetworkPolicy needs
,
kind
, and
metadata
fields. For general information about working with config files, see
,
and
Object Management
.
spec
spec
has all the information needed to define a particular network policy in the given namespace.
: Each NetworkPolicy includes a
which selects the grouping of pods to
which the policy applies. The example policy selects pods with the label "role=db". An empty
list which may include either
Ingress
,
Egress
, or both. The
field indicates whether or not the given policy
applies to ingress traffic to selected pod, egress traffic from selected pods, or both. If no
are specified on a NetworkPolicy then by default
Ingress
will always be set and
Egress
will be set if the NetworkPolicy has any egress rules.
ingress
: Each NetworkPolicy may include a list of allowed
ingress
rules. Each rule allows
traffic which matches both the
from
and
ports
sections. The example policy contains a single
rule, which matches traffic on a single port, from one of three sources, the first specified via
an
ipBlock
, the second via a
and the third via a
.
egress
egress
to
and
ports
rule, which matches traffic on a single port to any destination in
10.0.0.0/24
.
So, the e

xample NetworkPolicy:
isolates
role=db
pods in the
default
namespace for both ingress and egress traffic
(if they weren't already isolated)
(Ingress rules) allows connections to all pods in the
default
namespace with the label
role=db
on TCP port 6379 from:
any pod in the
default
role=frontend
any pod in a namespace with the label
project=myproject
IP addresses in the ranges
–
172.17.0.255
and
172.17.2.0
–
172.17.255.255
(ie, all of
172.17.0.0/16
except
172.17.1.0/24
)
(Egress rules) allows connections from any pod in the
default
role=db
to CIDR
on TCP port 5978
See the
walkthrough for further examples.
Behavior of
to
and
from
selectors
There are four kinds of selectors that can be specified in an
ingress
from
section or
egress
to
section:
: This selects particular Pods in the same namespace as the NetworkPolicy which
should be allowed as ingress sources or egress destinations.
: This selects particular namespaces for which all Pods should be allowed as
ingress sources or egress destinations.
and
: A single
to
/
from
entry that specifies both
and
selects particular Pods within particular namespaces. Be
careful to use correct YAML syntax. For example:
...
ingress
:
-
from
:
-
:
:
user
:
alice
:
:
role
:
client
...
This policy contains a single
from
element allowing connections from Pods with the label
role=client
in namespaces with the label
user=alice
. But the following policy is different:
...
ingress
:
-
from
:
-
:
:
user
:
alice
-
:
:
role
:
client
...
It contains two elements in the
from
array, and allows connections from Pods in the local
Namespace with the label
,
or
from any Pod in any namespace with the label
.
When in doubt, use
to see how Kubernetes has interpreted the policy.
ipBlock
: This selects particular IP CIDR ranges to allow as ingress sources or egress
destinations. These should be cluster-external IPs, since Pod IPs are ephemeral and unpredictable.
Cluster ingress and egress mechanisms often require rewriting the source or destination IP
of packets. In cases where this happens, it is not defined whether this happens before or
after NetworkPolicy processing, and the behavior may be different for different
combinations of network plugin, cloud provider,
Service
implementation, etc.
In the case of ingress, this means that in some cases you may be able to filter incoming
packets based on the actual original source IP, while in other cases, the "source IP" that
the NetworkPolicy acts on may be the IP of a
or of the Pod's node, etc.
For egress, this means that connections from pods to
Service
IPs that get rewritten to
cluster-external IPs may or may not be subject to
ipBlock
-based policies.
Default policies
By default, if no policies exist in a namespace, then all ingress and egress traffic is allowed to
and from pods in that namespace. The following examples let you change the default behavior
Default deny all ingress traffic
You can create a "default" ingress isolation policy for a namespace by creating a NetworkPolicy
that selects all pods but does not allow any ingress traffic to those pods.
service/networking/network-policy-default-deny-ingress.yaml
---
:
kind
:
metadata
:
name
:
default-deny-ingress
spec
:
:
{}
:
- Ingress
This ensures that even pods that aren't selected by any other NetworkPolicy will still be isolated
for ingress. This policy does not affect isolation for egress from any pod.
Allow all ingress traffic
If you want to allow all incoming connections to all pods in a namespace, you can create a policy
that explicitly allows that.
service/networking/network-polic

y-allow-all-ingress.yaml
---
:
kind
:
metadata
:
name
:
allow-all-ingress
spec
:
:
{}
ingress
:
- {}
:
- Ingress
With this policy in place, no additional policy or policies can cause any incoming connection to
those pods to be denied. This policy has no effect on isolation for egress from any pod.
Default deny all egress traffic
You can create a "default" egress isolation policy for a namespace by creating a NetworkPolicy
that selects all pods but does not allow any egress traffic from those pods.
service/networking/network-policy-default-deny-egress.yaml
---
:
kind
:
metadata
:
name
:
default-deny-egress
spec
:
:
{}
:
- Egress
This ensures that even pods that aren't selected by any other NetworkPolicy will not be allowed
egress traffic. This policy does not change the ingress isolation behavior of any pod.
Allow all egress traffic
If you want to allow all connections from all pods in a namespace, you can create a policy that
explicitly allows all outgoing connections from pods in that namespace.
service/networking/network-policy-allow-all-egress.yaml
---
:
kind
:
metadata
:
name
:
allow-all-egress
spec
:
:
{}
egress
:
- {}
:
- Egress
With this policy in place, no additional policy or policies can cause any outgoing connection from
those pods to be denied. This policy has no effect on isolation for ingress to any pod.
Default deny all ingress and all egress traffic
You can create a "default" policy for a namespace which prevents all ingress AND egress traffic by
creating the following NetworkPolicy in that namespace.
service/networking/network-policy-default-deny-all.yaml
---
:
kind
:
metadata
:
name
:
default-deny-all
spec
:
:
{}
:
- Ingress
- Egress
ingress or egress traffic.
SCTP support
As a stable feature, this is enabled by default. To disable SCTP at a cluster level, you (or your
cluster administrator) will need to disable the
for the API server with
--feature-gates=SCTPSupport=false,…
.
When the feature gate is enabled, you can set the
protocol
field of a NetworkPolicy to
SCTP
.
Note:
You must be using a
CNI
plugin that supports SCTP
protocol NetworkPolicies.
Targeting a range of ports
When writing a NetworkPolicy, you can target a range of ports instead of a single port.
This is achievable with the usage of the
endPort
field, as the following example:
service/networking/networkpolicy-multiport-egress.yaml
:
kind
:
metadata
:
name
:
multi-port-egress
namespace
:
default
spec
:
:
:
role
:
db
:
- Egress
egress
:
-
to
:
-
ipBlock
:
cidr
:
10.0.0.0
/24
ports
:
-
protocol
:
TCP
port
:
32000
endPort
:
32768
The above rule allows any Pod with label
role=db
on the namespace
default
to communicate
with any IP within the range
over TCP, provided that the target
port is between the range 32000 and 32768.
The following restrictions apply when using this field:
The
endPort
field must be equal to or greater than the
port
field.
endPort
can only be defined if
port
is also defined.
Both ports must be numeric.
Note:
Your cluster must be using a
CNI
plugin that
endPort
field in NetworkPolicy specifications.
If your
does not support the
endPort
field and you specify a NetworkPolicy with that,
the policy will be applied only for the single
port
field.
Targeting a Namespace by its name
Kubernetes 1.22 [stable]
The Kubernetes control plane sets an immutable label
on all
namespaces, provided that the
Names

paceDefaultLabelName
While NetworkPolicy cannot target a namespace by its name with some object field, you can use the
standardized label to target a specific namespace.
What you can't do with network policies (at least, not yet)
As of Kubernetes 1.26, the following functionality does not exist in the
NetworkPolicy API, but you might be able to implement workarounds using Operating System
components (such as SELinux, OpenVSwitch, IPTables, and so on) or Layer 7 technologies (Ingress
controllers, Service Mesh implementations) or admission controllers. In case you are new to
network security in Kubernetes, its worth noting that the following User Stories cannot (yet) be
implemented using the NetworkPolicy API.
Forcing internal cluster traffic to go through a common gateway (this might be best served with
a service mesh or other proxy).
Anything TLS related (use a service mesh or ingress controller for this).
Node specific policies (you can use CIDR notation for these, but you cannot target nodes by
their Kubernetes identities specifically).
Targeting of services by name (you can, however, target pods or namespaces by their
labels
, which is often a viable workaround).
Creation or management of "Policy requests" that are fulfilled by a third party.
Default policies which are applied to all namespaces or pods (there are some third party
Kubernetes distributions and projects which can do this).
Advanced policy querying and reachability tooling.
The ability to log network security events (for example connections that are blocked or accepted).
The ability to explicitly deny policies (currently the model for NetworkPolicies are deny by
default, with only the ability to add allow rules).
The ability to prevent loopback or incoming host traffic (Pods cannot currently block localhost
access, nor do they have the ability to block access from their resident node).
See the
See more
recipes
for common
scenarios enabled by the NetworkPolicy resource.
6.6 - DNS for Services and Pods
Your workload can discover Services within your cluster using DNS; this page explains how that works.
Kubernetes creates DNS records for Services and Pods. You can contact
Services with consistent DNS names instead of IP addresses.
Kubernetes publishes information about Pods and Services which is used
to program DNS. Kubelet configures Pods' DNS so that running containers
can lookup Services by name rather than IP.
Services defined in the cluster are assigned DNS names. By default, a
client Pod's DNS search list includes the Pod's own namespace and the
cluster's default domain.
Namespaces of Services
A DNS query may return different results based on the namespace of the Pod making
it. DNS queries that don't specify a namespace are limited to the Pod's
namespace. Access Services in other namespaces by specifying it in the DNS query.
For example, consider a Pod in a
test
namespace. A
data
Service is in
the
prod
A query for
data
returns no results, because it uses the Pod's
test
A query for
data.prod
returns the intended result, because it specifies the
DNS queries may be expanded using the Pod's
. Kubelet
configures this file for each Pod. For example, a query for just
data
may be
expanded to
data.test.svc.cluster.local
. The values of the
search
option
are used to expand queries. To learn more about DNS queries, see
the
manual page.
nameserver 10.32.0.10
search &lt;namespace&gt;.svc.cluster.local svc.cluster.local cluster.local
In summary, a Pod in the
test
namespace can successfully resolve either
data.prod
or
data.prod.svc.cluster.local
.
DNS Records
What objects get DNS records?
Services
Pods
The following sections detail the supported DNS record types and layout that is
supported. Any other layout or names or queries that happen to work are
conside

red implementation details and are subject to change without warning.
For more up-to-date specification, see
Kubernetes DNS-Based Service Discovery
.
Services
A/AAAA records
"Normal" (not headless) Services are assigned DNS A and/or AAAA records,
depending on the IP family or families of the Service, with a name of the form
my-svc.my-namespace.svc.cluster-domain.example
. This resolves to the cluster IP
of the Service.
(without a cluster IP) Services are also assigned DNS A and/or AAAA records,
with a name of the form
. Unlike normal
Services, this resolves to the set of IPs of all of the Pods selected by the Service.
Clients are expected to consume the set or else use standard round-robin
selection from the set.
SRV records
SRV Records are created for named ports that are part of normal or headless
services. For each named port, the SRV record has the form
_port-name._port-protocol.my-svc.my-namespace.svc.cluster-domain.example
.
For a regular Service, this resolves to the port number and the domain name:
.
For a headless Service, this resolves to multiple answers, one for each Pod
that is backing the Service, and contains the port number and the domain name of the Pod
of the form
hostname.my-svc.my-namespace.svc.cluster-domain.example
.
Pods
In general a Pod has the following DNS resolution:
pod-ip-address.my-namespace.pod.cluster-domain.example
.
For example, if a Pod in the
default
namespace has the IP address 172.17.0.3,
and the domain name for your cluster is
, then the Pod has a DNS name:
172-17-0-3.default.pod.cluster.local
.
Any Pods exposed by a Service have the following DNS resolution available:
pod-ip-address.service-name.my-namespace.svc.cluster-domain.example
.
Pod's hostname and subdomain fields
Currently when a Pod is created, its hostname (as observed from within the Pod)
is the Pod's
value.
The Pod spec has an optional
hostname
field, which can be used to specify a
different hostname. When specified, it takes precedence over the Pod's name to be
the hostname of the Pod (again, as observed from within the Pod). For example,
given a Pod with
spec.hostname
set to
"my-host"
, the Pod will have its
hostname set to
"my-host"
.
The Pod spec also has an optional
subdomain
field which can be used to indicate
that the pod is part of sub-group of the namespace. For example, a Pod with
set to
"foo"
, and
spec.subdomain
set to
"bar"
, in namespace
"my-namespace"
, will
have its hostname set to
"foo"
and its fully qualified domain name (FQDN) set to
"foo.bar.my-namespace.svc.cluster.local"
(once more, as observed from within
the Pod).
If there exists a headless Service in the same namespace as the Pod, with
the same name as the subdomain, the cluster's DNS Server also returns A and/or AAAA
records for the Pod's fully qualified hostname.
Example:
:
v1
kind
:
Service
metadata
:
name
:
busybox-subdomain
spec
:
selector
:
name
:
busybox
clusterIP
:
None
ports
:
-
name
:
foo
# name is not required for single-port Services
port
:
1234
---
:
v1
kind
:
Pod
metadata
:
name
:
busybox1
labels
:
name
:
busybox
spec
:
hostname
:
busybox-1
subdomain
:
:
-
image
:
command
:
- sleep
-
"3600"
name
:
busybox
---
:
v1
kind
:
Pod
metadata
:
name
:
busybox2
labels
:
name
:
busybox
spec
:
hostname
:
busybox-2
subdomain
:
:
-
image
:
command
:
- sleep
-
"3600"
name
:
busybox
Given the above Service
"busybox-subdomain"
and the Pods which set
to
, the first Pod will see its own FQDN as
"busybox-1.busybox-subdomain.my-namespace.svc.cluster-domain.example"
. DNS serves
A and/or AAAA records at that name, pointing to the Pod's IP. Both Pods "
busybox1
" and
"
busybox2
" will have their own address records.
An
can specify

the DNS hostname for any endpoint addresses, along with its IP.
Note:
Because A and AAAA records are not created for Pod names,
hostname
is required for the Pod's A or AAAA
record to be created. A Pod with no
hostname
but with
subdomain
will only create the
A or AAAA record for the headless Service (
busybox-subdomain.my-namespace.svc.cluster-domain.example
),
pointing to the Pods' IP addresses. Also, the Pod needs to be ready in order to have a
record unless
publishNotReadyAddresses=True
is set on the Service.
Pod's setHostnameAsFQDN field
When a Pod is configured to have fully qualified domain name (FQDN), its
hostname is the short hostname. For example, if you have a Pod with the fully
qualified domain name
busybox-1.busybox-subdomain.my-namespace.svc.cluster-domain.example
,
then by default the
hostname
command inside that Pod returns
busybox-1
and the
hostname --fqdn
command returns the FQDN.
setHostnameAsFQDN: true
in the Pod spec, the kubelet writes the Pod's FQDN into the hostname for that Pod's namespace. In this case, both
hostname
and
return the Pod's FQDN.
Note:
In Linux, the hostname field of the kernel (the
nodename
field of
struct utsname
) is limited to 64 characters.
If a Pod enables this feature and its FQDN is longer than 64 character, it will fail to start. The Pod will remain in
Pending
status (
ContainerCreating
as seen by
kubectl
) generating error events, such as Failed to construct FQDN from Pod hostname and cluster domain, FQDN
long-FQDN
is too long (64 characters is the max, 70 characters requested). One way of improving user experience for this scenario is to create an
admission webhook controller
to control FQDN size when users create top level objects, for example, Deployment.
Pod's DNS Policy
DNS policies can be set on a per-Pod basis. Currently Kubernetes supports the
following Pod-specific DNS policies. These policies are specified in the
dnsPolicy
field of a Pod Spec.
"
Default
": The Pod inherits the name resolution configuration from the node
that the Pods run on.
See
related discussion
"
": Any DNS query that does not match the configured cluster
domain suffix, such as "
www.kubernetes.io
", is forwarded to an upstream
nameserver by the DNS server. Cluster administrators may have extra
stub-domain and upstream DNS servers configured.
See
for details on how DNS queries are handled in those cases.
"
": For Pods running with hostNetwork, you should
explicitly set its DNS policy to "
". Otherwise, Pods
running with hostNetwork and
"ClusterFirst"
will fallback to the behavior
of the
"Default"
policy.
Note: This is not supported on Windows. See
below
for details
"
None
": It allows a Pod to ignore DNS settings from the Kubernetes
environment. All DNS settings are supposed to be provided using the
dnsConfig
field in the Pod Spec.
See
Pod's DNS config
subsection below.
Note:
"Default" is not the default DNS policy. If
dnsPolicy
is not
explicitly specified, then "ClusterFirst" is used.
The example below shows a Pod with its DNS policy set to
"
" because it has
set to
true
.
:
v1
kind
:
Pod
metadata
:
name
:
busybox
namespace
:
default
spec
:
:
-
image
:
command
:
- sleep
-
"3600"
:
name
:
busybox
:
Always
:
true
dnsPolicy
:
Pod's DNS Config allows users more control on the DNS settings for a Pod.
The
dnsConfig
field is optional and it can work with any
dnsPolicy
settings.
However, when a Pod's
dnsPolicy
is set to "
None
", the
dnsConfig
field has
Below are the properties a user can specify in the
dnsConfig
field:
: a list of IP addresses that will be used as DNS servers for the
Pod. There can be at mo

st 3 IP addresses specified. When the Pod's
dnsPolicy
is set to "
None
", the list must contain at least one IP address, otherwise
this property is optional.
The servers listed will be combined to the base nameservers generated from the
specified DNS policy with duplicate addresses removed.
searches
: a list of DNS search domains for hostname lookup in the Pod.
This property is optional. When specified, the provided list will be merged
into the base search domain names generated from the chosen DNS policy.
Duplicate domain names are removed.
Kubernetes allows for at most 6 search domains.
options
: an optional list of objects where each object may have a
name
property (required) and a
value
property (optional). The contents in this
property will be merged to the options generated from the specified DNS policy.
Duplicate entries are removed.
The following is an example Pod with custom DNS settings:
service/networking/custom-dns.yaml
:
v1
kind
:
Pod
metadata
:
namespace
:
default
name
:
dns-example
spec
:
:
-
name
:
test
image
:
nginx
dnsPolicy
:
"None"
dnsConfig
:
:
-
1.2.3.4
searches
:
- ns1.svc.cluster-domain.example
- my.dns.search.suffix
options
:
-
name
:
ndots
value
:
"2"
-
name
:
edns0
When the Pod above is created, the container
test
gets the following contents
in its
file:
nameserver 1.2.3.4
search ns1.svc.cluster-domain.example my.dns.search.suffix
options ndots:2 edns0
For IPv6 setup, search path and name server should be set up like this:
kubectl
exec
-it dns-example -- cat /etc/resolv.conf
nameserver 2001:db8:30::a
search default.svc.cluster-domain.example svc.cluster-domain.example cluster-domain.example
DNS search domain list limits
Kubernetes 1.26 [beta]
Kubernetes itself does not limit the DNS Config until the length of the search
domain list exceeds 32 or the total length of all search domains exceeds 2048.
This limit applies to the node's resolver configuration file, the Pod's DNS
Config, and the merged DNS Config respectively.
Note:
Some container runtimes of earlier versions may have their own restrictions on
the number of DNS search domains. Depending on the container runtime
environment, the pods with a large number of DNS search domains may get stuck in
the pending state.
It is known that containerd v1.5.5 or earlier and CRI-O v1.21 or earlier have
this problem.
DNS resolution on Windows nodes
ClusterFirstWithHostNet is not supported for Pods that run on Windows nodes.
Windows treats all names with a
.
as a FQDN and skips FQDN resolution.
On Windows, there are multiple DNS resolvers that can be used. As these come with
slightly different behaviors, using the
Resolve-DNSName
powershell cmdlet for name query resolutions is recommended.
On Linux, you have a DNS suffix list, which is used after resolution of a name as fully
qualified has failed.
On Windows, you can only have 1 DNS suffix, which is the DNS suffix associated with that
Pod's namespace (example:
mydns.svc.cluster.local
). Windows can resolve FQDNs, Services,
or network name which can be resolved with this single suffix. For example, a Pod spawned
in the
default
namespace, will have the DNS suffix
default.svc.cluster.local
.
Inside a Windows Pod, you can resolve both
and
, but not the partially qualified names (
or
).
For guidance on administering DNS configurations, check
Configure DNS Service
6.7 - IPv4/IPv6 dual-stack
Kubernetes lets you configure single-stack IPv4 networking, single-stack IPv6 networking, or dual stack networking with both network families active. This page explains how.
IPv4/IPv6 dual-stack networking enables the allocation of both IPv4 and IPv6 addresses to
Pods
and
Services
.
IPv4/IPv6 dual-stack networking is enabled by default for your Kubernetes cluster starting in
1.2

1, allowing the simultaneous assignment of both IPv4 and IPv6 addresses.
Supported Features
IPv4/IPv6 dual-stack on your Kubernetes cluster provides the following features:
Dual-stack Pod networking (a single IPv4 and IPv6 address assignment per Pod)
IPv4 and IPv6 enabled Services
Pod off-cluster egress routing (eg. the Internet) via both IPv4 and IPv6 interfaces
The following prerequisites are needed in order to utilize IPv4/IPv6 dual-stack Kubernetes clusters:
Kubernetes 1.20 or later
For information about using dual-stack services with earlier
Kubernetes versions, refer to the documentation for that version
of Kubernetes.
Provider support for dual-stack networking (Cloud provider or otherwise must be able to provide
Kubernetes nodes with routable IPv4/IPv6 network interfaces)
A
that
supports dual-stack networking.
Configure IPv4/IPv6 dual-stack
To configure IPv4/IPv6 dual-stack, set dual-stack cluster network assignments:
kube-apiserver:
--service-cluster-ip-range=&lt;IPv4 CIDR&gt;,&lt;IPv6 CIDR&gt;
kube-controller-manager:
--cluster-cidr=&lt;IPv4 CIDR&gt;,&lt;IPv6 CIDR&gt;
--node-cidr-mask-size-ipv4|--node-cidr-mask-size-ipv6
defaults to /24 for IPv4 and /64 for IPv6
kube-proxy:
kubelet:
when there is no
the administrator can pass a comma-separated pair of IP
addresses via
--node-ip
to manually configure dual-stack
.status.addresses
for that Node.
If a Pod runs on that node in HostNetwork mode, the Pod reports these IP addresses in its
.status.podIPs
field.
All
podIPs
in a node match the IP family preference defined by the
field for that Node.
Note:
An example of an IPv4 CIDR:
(though you would supply your own address range)
An example of an IPv6 CIDR:
fdXY:IJKL:MNOP:15::/64
(this shows the format but is not a valid
address - see
RFC 4193
)
Services
You can create
Services
which can use IPv4, IPv6, or both.
The address family of a Service defaults to the address family of the first service cluster IP
range (configured via the
flag to the kube-apiserver).
When you define a Service you can optionally configure it as dual stack. To specify the behavior you want, you
set the
.spec.ipFamilyPolicy
field to one of the following values:
SingleStack
: Single-stack service. The control plane allocates a cluster IP for the Service,
using the first configured service cluster IP range.
PreferDualStack
:
Allocates IPv4 and IPv6 cluster IPs for the Service.
RequireDualStack
: Allocates Service
.spec.ClusterIPs
from both IPv4 and IPv6 address ranges.
Selects the
.spec.ClusterIP
based on the address family
of the first element in the
.spec.ipFamilies
array.
If you would like to define which IP family to use for single stack or define the order of IP
families for dual-stack, you can choose the address families by setting an optional field,
, on the Service.
Note:
The
field is immutable because the
cannot be reallocated on a
Service that already exists. If you want to change
, delete and recreate the
Service.
You can set
to any of the following array values:
["IPv4"]
["IPv6"]
["IPv4","IPv6"]
(dual stack)
["IPv6","IPv4"]
The first family you list is used for the legacy
field.
Dual-stack Service configuration scenarios
These examples demonstrate the behavior of various dual-stack Service configuration scenarios.
Dual-stack options on new Services
This Service specification does not explicitly define
. When you create
this Service, Kubernetes assigns a cluster IP for the Service from the first configured
and sets the
to
. (
Services
without selectors
and
with selectors
will behave in this s

ame way.)
service/networking/dual-stack-default-svc.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
This Service specification explicitly defines
in
. When
you create this Service on a dual-stack cluster, Kubernetes assigns both IPv4 and IPv6
addresses for the service. The control plane updates the
.spec
for the Service to record the IP
address assignments. The field
is the primary field, and contains both assigned
IP addresses;
is a secondary field with its value calculated from
.
For the
field, the control plane records the IP address that is from the
same address family as the first service cluster IP range.
On a single-stack cluster, the
and
fields both only list
one address.
On a cluster with dual-stack enabled, specifying
in
behaves the same as
.
service/networking/dual-stack-preferred-svc.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
IPv6
and
IPv4
in
as well
as defining
in
. When Kubernetes assigns an IPv6 and
IPv4 address in
,
is set to the IPv6 address because that is
the first element in the
array, overriding the default.
service/networking/dual-stack-preferred-ipfamilies-svc.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
:
- IPv6
- IPv4
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Dual-stack defaults on existing Services
These examples demonstrate the default behavior when dual-stack is newly enabled on a cluster
where Services already exist. (Upgrading an existing cluster to 1.21 or beyond will enable
dual-stack.)
When dual-stack is enabled on a cluster, existing Services (whether
IPv4
or
IPv6
) are
configured by the control plane to set
to
and set
to the address family of the existing Service. The existing Service cluster IP
will be stored in
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
You can validate this behavior by using kubectl to inspect an existing service.
kubectl get svc my-service -o yaml
:
v1
kind
:
Service
metadata
:
labels
:
:
MyApp
name
:
spec
:
clusterIP
:
10.0.197.123
:
-
:
- IPv4
:
ports
:
-
port
:
80
protocol
:
TCP
:
80
selector
:
:
MyApp
type
:
ClusterIP
status
:
:
{}
When dual-stack is enabled on a cluster, existing
with selectors are
to
and set
to the address family of the first service cluster IP range (configured via the
flag to the kube-apiserver) even though
is set to
None
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
You can validate this behavior by using kubectl to inspect an existing headless service with selectors.
:
v1
kind

:
Service
metadata
:
labels
:
:
MyApp
name
:
spec
:
clusterIP
:
None
:
- None
:
- IPv4
:
ports
:
-
port
:
80
protocol
:
TCP
:
80
selector
:
:
MyApp
Switching Services between single-stack and dual-stack
Services can be changed from single-stack to dual-stack and from dual-stack to single-stack.
To change a Service from single-stack to dual-stack, change
from
to
or
as desired. When you change this
Service from single-stack to dual-stack, Kubernetes assigns the missing address family so that the
Service now has IPv4 and IPv6 addresses.
Edit the Service specification updating the
from
to
.
Before:
spec
:
:
After:
spec
:
:
To change a Service from dual-stack to single-stack, change
from
or
to
. When you change this Service from
dual-stack to single-stack, Kubernetes retains only the first element in the
array, and sets
to that IP address and sets
to the address
family of
.
Headless Services without selector
For
Headless Services without selectors
and without
explicitly set, the
field defaults to
.
Service type LoadBalancer
To provision a dual-stack load balancer for your Service:
Set the
.spec.type
field to
Set
field to
or
Note:
To use a dual-stack
type Service, your cloud provider must support IPv4 and IPv6
load balancers.
Egress traffic
If you want to enable egress traffic in order to reach off-cluster destinations (eg. the public
Internet) from a Pod that uses non-publicly routable IPv6 addresses, you need to enable the Pod to
use a publicly routed IPv6 address via a mechanism such as transparent proxying or IP
masquerading. The
project
supports IP masquerading on dual-stack clusters.
Note:
Ensure your
CNI
provider supports IPv6.
Windows support
Kubernetes on Windows does not support single-stack "IPv6-only" networking. However,
dual-stack IPv4/IPv6 networking for pods and nodes with single-family services
is supported.
You can use IPv4/IPv6 dual-stack networking with
l2bridge
networks.
Note:
Overlay (VXLAN) networks on Windows
do not
support dual-stack networking.
You can read more about the different network modes for Windows within the
topic.
Enable dual-stack networking using kubeadm
6.8 - Topology Aware Hints
provides a mechanism to help keep network traffic within the zone where it originated. Preferring same-zone traffic between Pods in your cluster can help with reliability, performance (network latency and throughput), or cost.
enable topology aware routing by including suggestions
for how clients should consume endpoints. This approach adds metadata to enable
consumers of EndpointSlice (or Endpoints) objects, so that traffic to
those network endpoints can be routed closer to where it originated.
For example, you can route traffic within a locality to reduce
costs, or to improve network performance.
Kubernetes clusters are increasingly deployed in multi-zone environments.
provides a mechanism to help keep traffic within the zone
it originated from. This concept is commonly referred to as "Topology Aware
Routing". When calculating the endpoints for a
Service
,
the EndpointSlice controller considers the topology (region and zone) of each endpoint
and populates the hints field to allocate it to a zone.
Cluster components such as the
can then consume those hints

, and use them to influence how the traffic is routed
(favoring topologically closer endpoints).
Using Topology Aware Hints
You can activate Topology Aware Hints for a Service by setting the
service.kubernetes.io/topology-aware-hints
auto
. This tells
the EndpointSlice controller to set topology hints if it is deemed safe.
Importantly, this does not guarantee that hints will always be set.
The functionality enabling this feature is split into two components: The
EndpointSlice controller and the kube-proxy. This section provides a high level overview
of how each component implements this feature.
EndpointSlice controller
The EndpointSlice controller is responsible for setting hints on EndpointSlices
when this feature is enabled. The controller allocates a proportional amount of
endpoints to each zone. This proportion is based on the
CPU cores for nodes running in that zone. For example, if one zone had 2 CPU
cores and another zone only had 1 CPU core, the controller would allocate twice
as many endpoints to the zone with 2 CPU cores.
The following example shows what an EndpointSlice looks like when hints have
been populated:
:
kind
:
metadata
:
name
:
example-hints
labels
:
:
example-svc
:
IPv4
ports
:
-
name
:
http
protocol
:
TCP
port
:
80
endpoints
:
-
addresses
:
-
:
ready
:
true
hostname
:
pod-1
zone
:
zone-a
hints
:
forZones
:
-
name
:
"zone-a"
The kube-proxy component filters the endpoints it routes to based on the hints set by
the EndpointSlice controller. In most cases, this means that the kube-proxy is able
to route traffic to endpoints in the same zone. Sometimes the controller allocates endpoints
from a different zone to ensure more even distribution of endpoints between zones.
This would result in some traffic being routed to other zones.
Safeguards
The Kubernetes control plane and the kube-proxy on each node apply some
safeguard rules before using Topology Aware Hints. If these don't check out,
the kube-proxy selects endpoints from anywhere in your cluster, regardless of the
zone.
Insufficient number of endpoints:
If there are less endpoints than zones
in a cluster, the controller will not assign any hints.
Impossible to achieve balanced allocation:
In some cases, it will be
impossible to achieve a balanced allocation of endpoints among zones. For
example, if zone-a is twice as large as zone-b, but there are only 2
endpoints, an endpoint allocated to zone-a may receive twice as much traffic
as zone-b. The controller does not assign hints if it can't get this "expected
overload" value below an acceptable threshold for each zone. Importantly this
is not based on real-time feedback. It is still possible for individual
endpoints to become overloaded.
One or more Nodes has insufficient information:
If any node does not have
a
label or is not reporting a value for
allocatable CPU, the control plane does not set any topology-aware endpoint
hints and so kube-proxy does not filter endpoints by zone.
One or more endpoints does not have a zone hint:
When this happens,
the kube-proxy assumes that a transition from or to Topology Aware Hints is
underway. Filtering endpoints for a Service in this state would be dangerous
so the kube-proxy falls back to using all endpoints.
A zone is not represented in hints:
If the kube-proxy is unable to find
at least one endpoint with a hint targeting the zone it is running in, it falls
to using endpoints from all zones. This is most likely to happen as you add
a new zone into your existing cluster.
Constraints
Topology Aware Hints are not used when either
or
is set to
Local
on a Service. It is possible to use
both features in the same cluster on different Services, just not on the same
Service.
This approach will not work well for Services that have a large proportion 

of
traffic originating from a subset of zones. Instead this assumes that incoming
traffic will be roughly proportional to the capacity of the Nodes in each
zone.
The EndpointSlice controller ignores unready nodes as it calculates the
proportions of each zone. This could have unintended consequences if a large
portion of nodes are unready.
The EndpointSlice controller does not take into account
when deploying or calculating the
proportions of each zone. If the Pods backing a Service are limited to a
subset of Nodes in the cluster, this will not be taken into account.
This may not work well with autoscaling. For example, if a lot of traffic is
originating from a single zone, only the endpoints allocated to that zone will
be handling that traffic. That could result in
either not picking up on this event, or newly added pods starting in a
different zone.
Follow the
tutorial
6.9 - Networking on Windows
Kubernetes supports running nodes on either Linux or Windows. You can mix both kinds of node
within a single cluster.
This page provides an overview to networking specific to the Windows operating system.
Container networking on Windows
Networking for Windows containers is exposed through
CNI plugins
.
Windows containers function similarly to virtual machines in regards to
networking. Each container has a virtual network adapter (vNIC) which is connected
to a Hyper-V virtual switch (vSwitch). The Host Networking Service (HNS) and the
Host Compute Service (HCS) work together to create containers and attach container
vNICs to networks. HCS is responsible for the management of containers whereas HNS
is responsible for the management of networking resources such as:
Virtual networks (including creation of vSwitches)
Endpoints / vNICs
Policies including packet encapsulations, load-balancing rules, ACLs, and NAT rules.
The Windows HNS and vSwitch implement namespacing and can
create virtual NICs as needed for a pod or container. However, many configurations such
as DNS, routes, and metrics are stored in the Windows registry database rather than as
files inside
/etc
, which is how Linux stores those configurations. The Windows registry for the container
is separate from that of the host, so concepts like mapping
from
the host into a container don't have the same effect they would on Linux. These must
be configured using Windows APIs run in the context of that container. Therefore
CNI implementations need to call the HNS instead of relying on file mappings to pass
network details into the pod or container.
Network modes
Windows supports five different networking drivers/modes: L2bridge, L2tunnel,
Overlay (Beta), Transparent, and NAT. In a heterogeneous cluster with Windows and Linux
worker nodes, you need to select a networking solution that is compatible on both
Windows and Linux. The following table lists the out-of-tree plugins are supported on Windows,
with recommendations on when to use each CNI:
Network Driver
Container Packet Modifications
Network Plugin Characteristics
L2bridge
Containers are attached to an external vSwitch. Containers are attached to the underlay network, although the physical network doesn't need to learn the container MACs because they are rewritten on ingress/egress.
MAC is rewritten to host MAC, IP may be rewritten to host IP using HNS OutboundNAT policy.
win-bridge
,
Azure-CNI
, Flannel host-gateway uses win-bridge
win-bridge uses L2bridge network mode, connects containers to the underlay of hosts, offering best performance. Requires user-defined routes (UDR) for inter-node connectivity.
L2Tunnel
This is a special case of l2bridge, but only used on Azure. All packets are sent to the virtualization host where SDN policy is applied.
MAC rewritten, IP visible on the underlay network
Azure-CNI
Azure-CNI allows integration of containers with Azure vNET, and allows them to le

verage the set of capabilities that
Azure Virtual Network provides
. For example, securely connect to Azure services or use Azure NSGs. See
azure-cni for some examples
Overlay
Containers are given a vNIC connected to an external vSwitch. Each overlay network gets its own IP subnet, defined by a custom IP prefix.The overlay network driver uses VXLAN encapsulation.
Encapsulated with an outer header.
win-overlay
, Flannel VXLAN (uses win-overlay)
win-overlay should be used when virtual container networks are desired to be isolated from underlay of hosts (e.g. for security reasons). Allows for IPs to be re-used for different overlay networks (which have different VNID tags) if you are restricted on IPs in your datacenter. This option requires
KB4489899
on Windows Server 2019.
Transparent (special use case for
ovn-kubernetes
)
Requires an external vSwitch. Containers are attached to an external vSwitch which enables intra-pod communication via logical networks (logical switches and routers).
Packet is encapsulated either via
GENEVE
or
STT
tunneling to reach pods which are not on the same host.
Packets are forwarded or dropped via the tunnel metadata information supplied by the ovn network controller.
NAT is done for north-south communication.
Deploy via ansible
. Distributed ACLs can be applied via Kubernetes policies. IPAM support. Load-balancing can be achieved without kube-proxy. NATing is done without using iptables/netsh.
NAT (
not used in Kubernetes
)
Containers are given a vNIC connected to an internal vSwitch. DNS/DHCP is provided using an internal component called
WinNAT
MAC and IP is rewritten to host MAC/IP.
nat
Included here for completeness
As outlined above, the
Flannel
CNI plugin
is also
supported
on Windows via the
VXLAN network backend
(
Beta support
; delegates to win-overlay)
and
host-gateway network backend
(stable support; delegates to win-bridge).
This plugin supports delegating to one of the reference CNI plugins (win-overlay,
win-bridge), to work in conjunction with Flannel daemon on Windows (Flanneld) for
automatic node subnet lease assignment and HNS network creation. This plugin reads
in its own configuration file (cni.conf), and aggregates it with the environment
variables from the FlannelD generated subnet.env file. It then delegates to one of
the reference CNI plugins for network plumbing, and sends the correct configuration
containing the node-assigned subnet to the IPAM plugin (for example:
host-local
).
For Node, Pod, and Service objects, the following network flows are supported for
TCP/UDP traffic:
Pod → Pod (IP)
Pod → Pod (Name)
Pod → Service (Cluster IP)
Pod → Service (PQDN, but only if there are no ".")
Pod → Service (FQDN)
Pod → external (IP)
Pod → external (DNS)
Node → Pod
Pod → Node
IP address management (IPAM)
The following IPAM options are supported on Windows:
azure-vnet-ipam
(for azure-cni only)
Windows Server IPAM
(fallback option if no IPAM is set)
Service
is an abstraction
that defines a logical set of Pods and a means to access them over a network.
In a cluster that includes Windows nodes, you can use the following types of Service:
NodePort
ClusterIP
Windows container networking differs in some important ways from Linux networking.
The
Microsoft documentation for Windows Container Networking
provides additional details and background.
On Windows, you can use the following settings to configure Services and load
balancing behavior:
Windows Service Settings
Feature
Minimum Supported Windows OS build
How to enable
Ensures that connections from a particular client are passed to the same Pod each time.
Set
service.spec.sessionAffinity
to "ClientIP"
Direct Server Return (DSR)
Load balancing mode where the IP address fixups and the LBNAT occurs at the container vSwitch port directly; service traffic arrives with the source IP set as the originating p

od IP.
Set the following flags in kube-proxy:
--feature-gates="WinDSR=true" --enable-dsr=true
Preserve-Destination
Skips DNAT of service traffic, thereby preserving the virtual IP of the target service in packets reaching the backend Pod. Also disables node-node forwarding.
Windows Server, version 1903
Set
"preserve-destination": "true"
in service annotations and enable DSR in kube-proxy.
IPv4/IPv6 dual-stack networking
Native IPv4-to-IPv4 in parallel with IPv6-to-IPv6 communications to, from, and within a cluster
See
Client IP preservation
Ensures that source IP of incoming ingress traffic gets preserved. Also disables node-node forwarding.
Set
to "Local" and enable DSR in kube-proxy
Warning:
There are known issue with NodePort Services on overlay networking, if the destination node is running Windows Server 2022.
To avoid the issue entirely, you can configure the service with
.
There are known issues with Pod to Pod connectivity on l2bridge network on Windows Server 2022 with KB5005619 or higher installed.
To workaround the issue and restore Pod to Pod connectivity, you can disable the WinDSR feature in kube-proxy.
These issues require OS fixes.
Please follow
https://github.com/microsoft/Windows-Containers/issues/204
for updates.
The following networking functionality is
not
supported on Windows nodes:
Host networking mode
Local NodePort access from the node itself (works for other nodes or external clients)
More than 64 backend pods (or unique destination addresses) for a single Service
IPv6 communication between Windows pods connected to overlay networks
Local Traffic Policy in non-DSR mode
Outbound communication using the ICMP protocol via the
,
, or using the Azure-CNI plugin.
Specifically, the Windows data plane (
VFP
)
doesn't support ICMP packet transpositions, and this means:
ICMP packets directed to destinations within the same network (such as pod to pod communication via ping)
work as expected;
TCP/UDP packets work as expected;
ICMP packets directed to pass through a remote network (e.g. pod to external internet communication via ping)
cannot be transposed and thus will not be routed back to their source;
Since TCP/UDP packets can still be transposed, you can substitute
ping &lt;destination&gt;
with
curl &lt;destination&gt;
when debugging connectivity with the outside world.
Other limitations:
Windows reference network plugins win-bridge and win-overlay do not implement
CNI spec
v0.4.0,
due to a missing
CHECK
The Flannel VXLAN CNI plugin has the following limitations on Windows:
Node-pod connectivity is only possible for local pods with Flannel v0.12.0 (or higher).
Flannel is restricted to using VNI 4096 and UDP port 4789. See the official
Flannel VXLAN
backend docs for more details on these parameters.
6.10 - Service ClusterIP allocation
Services
are an abstract way to expose
an application running on a set of Pods. Services
can have a cluster-scoped virtual IP address (using a Service of
).
Clients can connect using that virtual IP address, and Kubernetes then load-balances traffic to that
Service across the different backing Pods.
How Service ClusterIPs are allocated?
When Kubernetes needs to assign a virtual IP address for a Service,
that assignment happens one of two ways:
dynamically
the cluster's control plane automatically picks a free IP address from within the configured IP range for
Services.
statically
you specify an IP address of your choice, from within the configured IP range for Services.
Across your whole cluster, every Service
ClusterIP
must be unique.
Trying to create a Service with a specific
ClusterIP
that has already
been allocated will return an error.
Why do you need to reserve Service Cluster IPs?
Sometimes you may want to have Services running in wel

l-known IP addresses, so other components and
users in the cluster can use them.
The best example is the DNS Service for the cluster. As a soft convention, some Kubernetes installers assign the 10th IP address from
the Service IP range to the DNS service. Assuming you configured your cluster with Service IP range
10.96.0.0/16 and you want your DNS Service IP to be 10.96.0.10, you'd have to create a Service like
this:
:
v1
kind
:
Service
metadata
:
labels
:
k8s-app
:
kube-dns
:
"true"
:
CoreDNS
name
:
kube-dns
namespace
:
spec
:
clusterIP
:
10.96.0.10
ports
:
-
name
:
dns
port
:
53
protocol
:
UDP
:
53
-
name
:
dns-tcp
port
:
53
protocol
:
TCP
:
53
selector
:
k8s-app
:
kube-dns
type
:
ClusterIP
but as it was explained before, the IP address 10.96.0.10 has not been reserved; if other Services are created
before or in parallel with dynamic allocation, there is a chance they can allocate this IP, hence,
you will not be able to create the DNS Service because it will fail with a conflict error.
How can you avoid Service ClusterIP conflicts?
The allocation strategy implemented in Kubernetes to allocate ClusterIPs to Services reduces the
risk of collision.
The
ClusterIP
range is divided, based on the formula
,
described as
never less than 16 or more than 256 with a graduated step between them
.
Dynamic IP assignment uses the upper band by default, once this has been exhausted it will
use the lower range. This will allow users to use static allocations on the lower band with a low
Examples
Example 1
This example uses the IP address range: 10.96.0.0/24 (CIDR notation) for the IP addresses
of Services.
Range Size: 2
8
- 2 = 254
Band Offset:
min(max(16, 256/16), 256)
=
min(16, 256)
= 16
Static band start: 10.96.0.1
Static band end: 10.96.0.16
Range end: 10.96.0.254
pie showData
title 10.96.0.0/24
"Static" : 16
"Dynamic" : 238
enabled
Example 2
This example uses the IP address range: 10.96.0.0/20 (CIDR notation) for the IP addresses
12
- 2 = 4094
min(max(16, 4096/16), 256)
=
min(256, 256)
= 256
Static band end: 10.96.1.0
Range end: 10.96.15.254
title 10.96.0.0/20
"Static" : 256
"Dynamic" : 3838
enabled
Example 3
This example uses the IP address range: 10.96.0.0/16 (CIDR notation) for the IP addresses
16
- 2 = 65534
min(max(16, 65536/16), 256)
=
min(4096, 256)
= 256
Static band ends: 10.96.1.0
Range end: 10.96.255.254
title 10.96.0.0/16
"Dynamic" : 65278
enabled
Read about
Service External Traffic Policy
Read about
Read about
Services
6.11 - Service Internal Traffic Policy
If two Pods in your cluster want to communicate, and both Pods are actually running on the same node, use
to keep network traffic within that node. Avoiding a round trip via the cluster network can help with reliability, performance (network latency and throughput), or cost.
enables internal traffic restrictions to only route
internal traffic to endpoints within the node the traffic originated from. The
"internal" traffic here refers to traffic originated from Pods in the current
cluster. This can help to reduce costs and improve performance.
Using Service Internal Traffic Policy
You can enable the internal-only traffic policy for a
Service
, by setting its
to
Local
. This tells kube-proxy to only use node local
endpoints for cluster internal traffic.
Note:
For pods on nodes with no endpoints for a given Service, the Service
behaves as if it has zero endpoi

nts (for Pods on this node) even if the service
does have endpoints on other nodes.
The following example shows what a Service looks like when you set
to
Local
:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
:
9376
:
Local
The kube-proxy filters the endpoints it routes to based on the
spec.internalTrafficPolicy
setting. When it's set to
Local
, only node local
endpoints are considered. When it's
Cluster
(the default), or is not set,
Kubernetes considers all endpoints.
Read about
Read about
Follow the
tutorial
6.12 - Topology-aware traffic routing with topology keys
Note:
This feature, specifically the alpha
API, is deprecated since
Kubernetes v1.21.
,
introduced in Kubernetes v1.21, provide similar functionality.
Service Topology
enables a service to route traffic based upon the Node
topology of the cluster. For example, a service can specify that traffic be
preferentially routed to endpoints that are on the same Node as the client, or
in the same availability zone.
Topology-aware traffic routing
By default, traffic sent to a
ClusterIP
or
NodePort
Service may be routed to
any backend address for the Service. Kubernetes 1.7 made it possible to
route "external" traffic to the Pods running on the same Node that received the
traffic. For
ClusterIP
Services, the equivalent same-node preference for
routing wasn't possible; nor could you configure your cluster to favor routing
to endpoints within the same zone.
By setting
on a Service, you're able to define a policy for routing
traffic based upon the Node labels for the originating and destination Nodes.
The label matching between the source and destination lets you, as a cluster
operator, designate sets of Nodes that are "closer" and "farther" from one another.
You can define labels to represent whatever metric makes sense for your own
requirements.
In public clouds, for example, you might prefer to keep network traffic within the
same zone, because interzonal traffic has a cost associated with it (and intrazonal
traffic typically does not). Other common needs include being able to route traffic
to a local Pod managed by a DaemonSet, or directing traffic to Nodes connected to the
same top-of-rack switch for the lowest latency.
Using Service Topology
enabled, you can control Service traffic
routing by specifying the
field on the Service spec. This field
is a preference-order list of Node labels which will be used to sort endpoints
when accessing this Service. Traffic will be directed to a Node whose value for
the first label matches the originating Node's value for that label. If there is
no backend for the Service on a matching Node, then the second label will be
considered, and so forth, until no labels remain.
If no match is found, the traffic will be rejected, as if there were no
backends for the Service at all. That is, endpoints are chosen based on the first
topology key with available backends. If this field is specified and all entries
have no backends that match the topology of the client, the service has no
backends for that client and connections should fail. The special value
"*"
may
be used to mean "any topology". This catch-all value, if used, only makes sense
as the last value in the list.
If
is not specified or empty, no topology constraints will be applied.
Consider a cluster with Nodes that are labeled with their hostname, zone name,
and region name. Then you can set the
values of a service to direct
traffic as follows.
Only to endpoints on the same node, failing if no endpoint exists on the node:
["kubernetes.io/ho

stname"]
.
Preferentially to endpoints on the same node, falling back to endpoints in the
same zone, followed by the same region, and failing otherwise:
["kubernetes.io/hostname", "topology.kubernetes.io/zone", "topology.kubernetes.io/region"]
.
This may be useful, for example, in cases where data locality is critical.
Preferentially to the same zone, but fallback on any available endpoint if
none are available within this zone:
["topology.kubernetes.io/zone", "*"]
.
Service topology is not compatible with
externalTrafficPolicy=Local
, and
therefore a Service cannot use both of these features. It is possible to use
both features in the same cluster on different Services, only not on the same
Service.
Valid topology keys are currently limited to
,
, and
, but will
be generalized to other node labels in the future.
Topology keys must be valid label keys and at most 16 keys may be specified.
The catch-all value,
"*"
, must be the last value in the topology keys, if
it is used.
Examples
The following are common examples of using the Service Topology feature.
Only Node Local Endpoints
A Service that only routes to node local endpoints. If no endpoints exist on the node, traffic is dropped:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
my-app
ports
:
-
protocol
:
TCP
port
:
80
:
9376
:
-
Prefer Node Local Endpoints
A Service that prefers node local Endpoints but falls back to cluster wide endpoints if node local endpoints do not exist:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
my-app
ports
:
-
protocol
:
TCP
port
:
80
:
9376
:
-
-
"*"
Only Zonal or Regional Endpoints
A Service that prefers zonal then regional endpoints. If no endpoints exist in either, traffic is dropped.
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
my-app
ports
:
-
protocol
:
TCP
port
:
80
:
9376
:
-
-
"topology.kubernetes.io/region"
Prefer Node Local, Zonal, then Regional Endpoints
A Service that prefers node local, zonal, then regional endpoints but falls back to cluster wide endpoints.
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
my-app
ports
:
-
protocol
:
TCP
port
:
80
:
9376
:
-
-
-
-
"*"
Read about
Read
7 - Storage
7.1 - Volumes
On-disk files in a container are ephemeral, which presents some problems for
non-trivial applications when running in containers. One problem
is the loss of files when a container crashes. The kubelet restarts the container
but with a clean state. A second problem occurs when sharing files
between containers running together in a
Pod
.
volume
abstraction
solves both of these problems.
Familiarity with
Pods
is suggested.
Docker has a concept of
volumes
, though it is
somewhat looser and less managed. A Docker volume is a directory on
disk or in another container. Docker provides volume
drivers, but the functionality is somewhat limited.
Kubernetes supports many types of volumes. A
Pod
can use any number of volume types simultaneously.
Ephemeral volume types have a lifetime of a pod, but persistent volumes exist beyond
the lifetime of a pod. When a pod ceases to exist, Kubernetes destroys ephemeral volumes;
however, Kubernetes does not destroy persistent volumes.
For any kind of volume in a given pod, data is preserved across container restarts.
At its core, a volume is a directory, possibly with some data in it, which
is accessible to the containers in a pod.

 How that directory comes to be, the
medium that backs it, and the contents of it are determined by the particular
volume type used.
To use a volume, specify the volumes to provide for the Pod in
.spec.volumes
and declare where to mount those volumes into containers in
.spec.containers[*].volumeMounts
.
A process in a container sees a filesystem view composed from the initial contents of
the
, plus volumes
(if defined) mounted inside the container.
The process sees a root filesystem that initially matches the contents of the container
image.
Any writes to within that filesystem hierarchy, if allowed, affect what that process views
when it performs a subsequent filesystem access.
Volumes mount at the
specified paths
within
the image.
For each container defined within a Pod, you must independently specify where
to mount each volume that the container uses.
Volumes cannot mount within other volumes (but see
Using subPath
for a related mechanism). Also, a volume cannot contain a hard link to anything in
a different volume.
Types of volumes
Kubernetes supports several types of volumes.
awsElasticBlockStore (deprecated)
Kubernetes v1.17 [deprecated]
An
volume mounts an Amazon Web Services (AWS)
EBS volume
into your pod. Unlike
emptyDir
, which is erased when a pod is removed, the contents of an EBS
volume are persisted and the volume is unmounted. This means that an
EBS volume can be pre-populated with data, and that data can be shared between pods.
Note:
You must create an EBS volume by using
aws ec2 create-volume
or the AWS API before you can use it.
There are some restrictions when using an
volume:
the nodes on which pods are running must be AWS EC2 instances
those instances need to be in the same region and availability zone as the EBS volume
EBS only supports a single EC2 instance mounting a volume
Creating an AWS EBS volume
Before you can use an EBS volume with a pod, you need to create it.
aws ec2 create-volume --availability-zone
=
eu-west-1a --size
=
10
--volume-type
=
gp2
Make sure the zone matches the zone you brought up your cluster in. Check that the size and EBS volume
type are suitable for your use.
AWS EBS configuration example
:
v1
kind
:
Pod
metadata
:
name
:
test-ebs
spec
:
:
-
image
:
registry.k8s.io/test-webserver
name
:
:
-
mountPath
:
/test-ebs
name
:
test-volume
volumes
:
-
name
:
# This AWS EBS volume must already exist.
:
volumeID
:
"&lt;volume id&gt;"
fsType
:
ext4
If the EBS volume is partitioned, you can supply the optional field
partition: "&lt;partition number&gt;"
to specify which partition to mount on.
AWS EBS CSI migration
The
, when enabled, redirects
all plugin operations from the existing in-tree plugin to the
ebs.csi.aws.com
Container
Storage Interface (CSI) driver. In order to use this feature, the
AWS EBS CSI
driver
must be installed on the cluster.
AWS EBS CSI migration complete
Kubernetes v1.17 [alpha]
To disable the
storage plugin from being loaded by the controller manager
and the kubelet, set the
flag to
true
.
azureDisk (deprecated)
Kubernetes v1.19 [deprecated]
The
azureDisk
volume type mounts a Microsoft Azure
Data Disk
into a pod.
For more details, see the
azureDisk
volume plugin
.
azureDisk CSI migration
The
azureDisk
, when enabled, redirects all plugin operations
from the existing in-tree plugin to the
disk.csi.azure.com
Container
Storage Interface (CSI) Driver. In order to use this feature, the
Azure Disk CSI Driver
azureDisk CSI migration complete
Kubernetes v1.21 [alpha]
azureDisk
storage plugin from being loaded by the controller manage

r
flag to
true
.
azureFile (deprecated)
The
azureFile
volume type mounts a Microsoft Azure File volume (SMB 2.1 and 3.0)
into a pod.
azureFile
.
azureFile CSI migration
The
azureFile
file.csi.azure.com
Container
Azure File CSI
Driver
must be installed on the cluster and the
must be enabled.
Azure File CSI driver does not support using same volume with different fsgroups. If
is enabled, using same volume with different fsgroups won't be supported at all.
azureFile CSI migration complete
azureFile
flag to
true
.
cephfs
A
cephfs
volume allows an existing CephFS volume to be
mounted into your Pod. Unlike
emptyDir
, which is erased when a pod is
removed, the contents of a
cephfs
volume are preserved and the volume is merely
unmounted. This means that a
cephfs
volume can be pre-populated with data, and
that data can be shared between pods. The
cephfs
volume can be mounted by multiple
writers simultaneously.
Note:
You must have your own Ceph server running with the share exported before you can use it.
See the
CephFS example
cinder (deprecated)
Kubernetes v1.18 [deprecated]
Note:
Kubernetes must be configured with the OpenStack cloud provider.
The
cinder
volume type is used to mount the OpenStack Cinder volume into your pod.
Cinder volume configuration example
:
v1
kind
:
Pod
metadata
:
name
:
test-cinder
spec
:
:
-
image
:
name
:
test-cinder-container
:
-
mountPath
:
/test-cinder
name
:
volumes
:
-
name
:
# This OpenStack volume must already exist.
cinder
:
volumeID
:
fsType
:
ext4
OpenStack CSI migration
The
feature for Cinder is enabled by default since Kubernetes 1.21.
It redirects all plugin operations from the existing in-tree plugin to the
cinder.csi.openstack.org
Container Storage Interface (CSI) Driver.
OpenStack Cinder CSI Driver
To disable the in-tree Cinder plugin from being loaded by the controller manager
and the kubelet, you can enable the
.
configMap
A
ConfigMap
provides a way to inject configuration data into pods.
The data stored in a ConfigMap can be referenced in a volume of type
configMap
and then consumed by containerized applications running in a pod.
When referencing a ConfigMap, you provide the name of the ConfigMap in the
volume. You can customize the path to use for a specific
entry in the ConfigMap. The following configuration shows how to mount
the
log-config
ConfigMap onto a Pod called
configmap-pod
:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
test
image
:
:
-
name
:
config-vol
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
items
:
-
key
:
log_level
path
:
log_level
The
ConfigMap is mounted as a volume, and all contents stored in
its
log_level
entry are mounted into the Pod at path
/etc/config/log_level
.
Note that this path is derived from the volume's
mountPath
and the
path
keyed with
log_level
.
Note:
You must create a
ConfigMap
before you can use it.
subPath
receive ConfigMap updates.
Text data is exposed as files using the UTF-8 character encoding. For other character encoding

s, use
.
A
volume makes
data available to applications. Within the volume, you can find the exposed
data as read-only files in plain text format.
Note:
A container using the downward API as a
subPath
volume mount does not
receive updates when field values change.
See
emptyDir
An
emptyDir
volume is first created when a Pod is assigned to a node, and
exists as long as that Pod is running on that node. As the name says, the
emptyDir
volume is initially empty. All containers in the Pod can read and write the same
emptyDir
volume, though that volume can be mounted at the same
or different paths in each container. When a Pod is removed from a node for
any reason, the data in the
emptyDir
is deleted permanently.
Note:
A container crashing does
not
remove a Pod from a node. The data in an
emptyDir
volume
is safe across container crashes.
Some uses for an
emptyDir
are:
scratch space, such as for a disk-based merge sort
checkpointing a long computation for recovery from crashes
holding files that a content-manager container fetches while a webserver
container serves the data
The
field controls where
emptyDir
volumes are stored. By
default
emptyDir
volumes are stored on whatever medium that backs the node
such as disk, SSD, or network storage, depending on your environment. If you set
the
field to
"Memory"
, Kubernetes mounts a tmpfs (RAM-backed
filesystem) for you instead. While tmpfs is very fast, be aware that unlike
disks, tmpfs is cleared on node reboot and any files you write count against
your container's memory limit.
A size limit can be specified for the default medium, which limits the capacity
of the
emptyDir
volume. The storage is allocated from
node ephemeral
storage
.
If that is filled up from another source (for example, log files or image
overlays), the
emptyDir
may run out of capacity before this limit.
Note:
If the
you can specify a size for memory backed volumes. If no size is specified, memory
backed volumes are sized to 50% of the memory on a Linux host.
emptyDir configuration example
:
v1
kind
:
Pod
metadata
:
name
:
test-pd
spec
:
:
-
image
:
name
:
:
-
mountPath
:
/cache
name
:
cache-volume
volumes
:
-
name
:
emptyDir
:
sizeLimit
:
500Mi
fc (fibre channel)
An
fc
volume type allows an existing fibre channel block storage volume
to mount in a Pod. You can specify single or multiple target world wide names (WWNs)
using the parameter
in your Volume configuration. If multiple WWNs are specified,
targetWWNs expect that those WWNs are from multi-path connections.
Note:
You must configure FC SAN Zoning to allocate and mask those LUNs (volumes) to the target WWNs
beforehand so that Kubernetes hosts can access them.
See the
fibre channel example
gcePersistentDisk (deprecated)
A
volume mounts a Google Compute Engine (GCE)
persistent disk
(PD) into your Pod.
Unlike
emptyDir
, which is erased when a pod is removed, the contents of a PD are
preserved and the volume is merely unmounted. This means that a PD can be
pre-populated with data, and that data can be shared between pods.
Note:
You must create a PD using
gcloud
or the GCE API or UI before you can use it.
There are some restrictions when using a
:
the nodes on which Pods are running must be GCE VMs
those VMs need to be in the same GCE project and zone as the persistent disk
One feature of GCE persistent disk is concurrent read-only access to a persistent disk.
A
volume permits multiple consumers to simultaneously
mount a persistent disk as read-only. This means that you can pre-populate a PD with your dataset
and then serve it in parallel from as man

y Pods as you need. Unfortunately,
PDs can only be mounted by a single consumer in read-write mode. Simultaneous
writers are not allowed.
Using a GCE persistent disk with a Pod controlled by a ReplicaSet will fail unless
the PD is read-only or the replica count is 0 or 1.
Creating a GCE persistent disk
Before you can use a GCE persistent disk with a Pod, you need to create it.
gcloud compute disks create --size
=
500GB --zone
=
us-central1-a my-data-disk
GCE persistent disk configuration example
:
v1
kind
:
Pod
metadata
:
name
:
test-pd
spec
:
:
-
image
:
name
:
:
-
mountPath
:
/test-pd
name
:
volumes
:
-
name
:
# This GCE PD must already exist.
:
pdName
:
my-data-disk
fsType
:
ext4
Regional persistent disks
The
feature allows the creation of persistent disks that are available in two zones
within the same region. In order to use this feature, the volume must be provisioned
as a PersistentVolume; referencing the volume directly from a pod is not supported.
Manually provisioning a Regional PD PersistentVolume
Dynamic provisioning is possible using a
StorageClass for GCE PD
.
Before creating a PersistentVolume, you must create the persistent disk:
=
500GB my-data-disk
--region us-central1
--replica-zones us-central1-a,us-central1-b
Regional persistent disk configuration example
:
v1
kind
:
metadata
:
name
:
spec
:
capacity
:
storage
:
400Gi
:
:
pdName
:
fsType
:
ext4
:
required
:
:
-
:
# failure-domain.beta.kubernetes.io/zone should be used prior to 1.21
-
key
:
operator
:
In
values
:
- us-central1-a
- us-central1-b
GCE CSI migration
The
feature for GCE PD, when enabled, redirects all plugin operations
pd.csi.storage.gke.io
Container
GCE PD CSI
Driver
GCE CSI migration complete
flag to
true
.
gitRepo (deprecated)
Warning:
The
gitRepo
volume type is deprecated. To provision a container with a git repo, mount an
EmptyDir
into an InitContainer that clones the repo using git, then mount the
EmptyDir
into the Pod's container.
A
gitRepo
volume is an example of a volume plugin. This plugin
mounts an empty directory and clones a git repository into this directory
for your Pod to use.
Here is an example of a
gitRepo
volume:
:
v1
kind
:
Pod
metadata
:
name
:
server
spec
:
:
-
image
:
nginx
name
:
nginx
:
-
mountPath
:
/mypath
name
:
git-volume
volumes
:
-
name
:
gitRepo
:
:
"git@somewhere:me/my-git-repository.git"
revision
:
"22f1d8406d464b0c0874075539c1f2e96c253775"
glusterfs (removed)
The GlusterFS in-tree storage driver was deprecated in the Kubernetes v1.25 release
and then removed entirely in the v1.26 release.
hostPath
Warning:
HostPath volumes present many security risks, and it is a best practice to avoid the use of
HostPaths when possible. When a HostPath volume must be used, it should be scoped to only the
required file or directory, and mounted as ReadOnly.
If restricting HostPath access to specific directories through AdmissionPolicy,
MUST
be required to use
readOnly
mounts for the policy to be effective.
A
hostPath
volume mounts a file or directory from the host node's filesystem
into your Pod. This is not something that most Pods will need, but it offers a
powerful escape hatch for some applications.
For example, some uses for a
hostP

ath
are:
running a container that needs access to Docker internals; use a
hostPath
of
/var/lib/docker
running cAdvisor in a container; use a
hostPath
of
/sys
allowing a Pod to specify whether a given
hostPath
should exist prior to the
Pod running, whether it should be created, and what it should exist as
In addition to the required
path
property, you can optionally specify a
type
for a
hostPath
volume.
The supported values for field
type
are:
Value
Behavior
Empty string (default) is for backward compatibility, which means that no checks will be performed before mounting the hostPath volume.
If nothing exists at the given path, an empty directory will be created there as needed with permission set to 0755, having the same group and ownership with Kubelet.
Directory
A directory must exist at the given path
If nothing exists at the given path, an empty file will be created there as needed with permission set to 0644, having the same group and ownership with Kubelet.
File
A file must exist at the given path
Socket
A UNIX socket must exist at the given path
CharDevice
A character device must exist at the given path
BlockDevice
A block device must exist at the given path
Watch out when using this type of volume, because:
HostPaths can expose privileged system credentials (such as for the Kubelet) or privileged APIs
(such as container runtime socket), which can be used for container escape or to attack other
parts of the cluster.
Pods with identical configuration (such as created from a PodTemplate) may
behave differently on different nodes due to different files on the nodes
The files or directories created on the underlying hosts are only writable by root. You
either need to run your process as root in a
privileged Container
or modify the file
permissions on the host to be able to write to a
hostPath
volume
hostPath configuration example
:
v1
kind
:
Pod
metadata
:
name
:
test-pd
spec
:
:
-
image
:
name
:
:
-
mountPath
:
/test-pd
name
:
volumes
:
-
name
:
hostPath
:
# directory location on host
path
:
/data
# this field is optional
type
:
Directory
Caution:
The
mode does not create the parent directory of the file. If the parent directory
of the mounted file does not exist, the pod fails to start. To ensure that this mode works,
you can try to mount directories and files separately, as shown in the
.
hostPath FileOrCreate configuration example
:
v1
kind
:
Pod
metadata
:
name
:
test-webserver
spec
:
:
-
name
:
image
:
registry.k8s.io/test-webserver:latest
:
-
mountPath
:
/var/local/aaa
name
:
mydir
-
mountPath
:
/var/local/aaa/1.txt
name
:
myfile
volumes
:
-
name
:
mydir
hostPath
:
# Ensure the file directory is created.
path
:
type
:
-
name
:
myfile
hostPath
:
path
:
type
:
iscsi
An
iscsi
volume allows an existing iSCSI (SCSI over IP) volume to be mounted
into your Pod. Unlike
emptyDir
, which is erased when a Pod is removed, the
contents of an
iscsi
unmounted. This means that an iscsi volume can be pre-populated with data, and
that data can be shared between pods.
Note:
You must have your own iSCSI server running with the volume created before you can use it.
A feature of iSCSI is that it can be mounted as read-only by multiple consumers
simultaneously. This means that you can pre-populate a volume with your dataset
and then serve it in parallel from as many Pods as you need. Unfortunately,
iSCSI volumes can only be mounted by a single consumer in read-write mode.
Simultaneous writers are not allowed.
See the
iSCSI example
local
A
local
volume represents a mounted local storage device such as a disk,
partition or directory.
Local volumes can only be used as a statically created Per

sistentVolume. Dynamic
provisioning is not supported.
Compared to
hostPath
volumes,
local
volumes are used in a durable and
portable manner without manually scheduling pods to nodes. The system is aware
of the volume's node constraints by looking at the node affinity on the PersistentVolume.
However,
local
volumes are subject to the availability of the underlying
node and are not suitable for all applications. If a node becomes unhealthy,
then the
local
volume becomes inaccessible by the pod. The pod using this volume
is unable to run. Applications using
local
volumes must be able to tolerate this
reduced availability, as well as potential data loss, depending on the
durability characteristics of the underlying disk.
The following example shows a PersistentVolume using a
local
volume and
:
:
v1
kind
:
metadata
:
name
:
example-pv
spec
:
capacity
:
storage
:
100Gi
:
:
:
Delete
:
local-storage
local
:
path
:
/mnt/disks/ssd1
:
required
:
:
-
:
-
key
:
operator
:
In
values
:
- example-node
You must set a PersistentVolume
when using
local
volumes.
The Kubernetes scheduler uses the PersistentVolume
to schedule
these Pods to the correct node.
can be set to "Block" (instead of the default
value "Filesystem") to expose the local volume as a raw block device.
When using local volumes, it is recommended to create a StorageClass with
set to
WaitForFirstConsumer
. For more details, see the
local
example.
Delaying volume binding ensures that the PersistentVolumeClaim binding decision
will also be evaluated with any other node constraints the Pod may have,
such as node resource requirements, node selectors, Pod affinity, and Pod anti-affinity.
An external static provisioner can be run separately for improved management of
the local volume lifecycle. Note that this provisioner does not support dynamic
provisioning yet. For an example on how to run an external local provisioner,
see the
local volume provisioner user
guide
.
Note:
The local PersistentVolume requires manual cleanup and deletion by the
user if the external static provisioner is not used to manage the volume
lifecycle.
nfs
An
nfs
volume allows an existing NFS (Network File System) share to be
mounted into a Pod. Unlike
emptyDir
, which is erased when a Pod is
removed, the contents of an
nfs
unmounted. This means that an NFS volume can be pre-populated with data, and
that data can be shared between pods. NFS can be mounted by multiple
:
v1
kind
:
Pod
metadata
:
name
:
test-pd
spec
:
:
-
image
:
name
:
:
-
mountPath
:
/my-nfs-data
name
:
volumes
:
-
name
:
nfs
:
server
:
my-nfs-server.example.com
path
:
/my-nfs-volume
readOnly
:
true
Note:
You must have your own NFS server running with the share exported before you can use it.
Also note that you can't specify NFS mount options in a Pod spec. You can either set mount options server-side or
use
/etc/nfsmount.conf
.
You can also mount NFS volumes via PersistentVolumes which do allow you to set mount options.
See the
NFS example
for an example of mounting NFS volumes with PersistentVolumes.
A
volume is used to mount a
into a Pod. PersistentVolumeClaims
are a way for users to "claim" durable storage (such as a GCE PersistentDisk or an
iSCSI volume) without knowing the details of the particular cloud environment.
See the information about
for more
details.
portworxVolume (deprecated)
Kubernetes v1.25 [deprecated]
A
is an elastic block storage layer that runs hyperconverge

d with
Portworx
fingerprints storage
in a server, tiers based on capabilities, and aggregates capacity across multiple servers.
Portworx runs in-guest in virtual machines or on bare metal Linux nodes.
A
can be dynamically created through Kubernetes or it can also
be pre-provisioned and referenced inside a Pod.
Here is an example Pod referencing a pre-provisioned Portworx volume:
:
v1
kind
:
Pod
metadata
:
name
:
test-portworx-volume-pod
spec
:
:
-
image
:
name
:
:
-
mountPath
:
/mnt
name
:
pxvol
volumes
:
-
name
:
pxvol
# This Portworx volume must already exist.
:
volumeID
:
"pxvol"
fsType
:
"&lt;fs-type&gt;"
Note:
Make sure you have an existing PortworxVolume with name
pxvol
before using it in the Pod.
Portworx volume
examples.
Portworx CSI migration
The
feature for Portworx has been added but disabled by default in Kubernetes 1.23 since it's in alpha state.
It has been beta now since v1.25 but it is still turned off by default.
pxd.portworx.com
Portworx CSI Driver
To enable the feature, set
CSIMigrationPortworx=true
in kube-controller-manager and kubelet.
projected
A projected volume maps several existing volume sources into the same
directory. For more details, see
.
rbd
An
rbd
volume allows a
Rados Block Device
(RBD) volume to mount
emptyDir
, which is erased when a pod is removed, the
rbd
volume are preserved and the volume is unmounted. This
means that a RBD volume can be pre-populated with data, and that data can be
shared between pods.
Note:
You must have a Ceph installation running before you can use RBD.
A feature of RBD is that it can be mounted as read-only by multiple consumers
and then serve it in parallel from as many pods as you need. Unfortunately,
RBD volumes can only be mounted by a single consumer in read-write mode.
See the
RBD example
RBD CSI migration
The
RBD
, when enabled, redirects all plugin
operations from the existing in-tree plugin to the
rbd.csi.ceph.com
CSI
driver. In order to use this
feature, the
Ceph CSI driver
must be enabled. (Note that the
csiMigrationRBD
flag has been removed and
replaced with
in release v1.24)
Note:
As a Kubernetes cluster operator that administers storage, here are the
prerequisites that you must complete before you attempt migration to the
RBD CSI driver:
You must install the Ceph CSI driver (
), v3.5.0 or above,
into your Kubernetes cluster.
considering the
clusterID
field is a required parameter for CSI driver for
its operations, but in-tree StorageClass has
monitors
field as a required
parameter, a Kubernetes storage admin has to create a clusterID based on the
monitors hash ( ex:
#echo -n '&lt;monitors_string&gt;' | md5sum
) in the CSI config map and keep the monitors
under this clusterID configuration.
Also, if the value of
adminId
in the in-tree Storageclass is different from
admin
, the
adminSecretName
mentioned in the in-tree Storageclass has to be
patched with the base64 value of the
adminId
parameter value, otherwise this
step can be skipped.
secret
A
secret
volume is used to pass sensitive information, such as passwords, to
Pods. You can store secrets in the Kubernetes API and mount them as files for
use by pods without coupling to Kubernetes directly.
secret
volumes are
backed by tmpfs (a RAM-backed filesystem) so they are never written to
non-volatile storage

.
Note:
You must create a Secret in the Kubernetes API before you can use it.
Note:
A container using a Secret as a
subPath
receive Secret updates.
Configuring Secrets
.
vsphereVolume (deprecated)
Note:
We recommend to use vSphere CSI out-of-tree driver instead.
A
is used to mount a vSphere VMDK volume into your Pod. The contents
of a volume are preserved when it is unmounted. It supports both VMFS and VSAN datastore.
vSphere volume
examples.
vSphere CSI migration
In Kubernetes 1.26, all operations for the in-tree
type
are redirected to the
csi.vsphere.vmware.com
CSI
driver.
vSphere CSI driver
must be installed on the cluster. You can find additional advice on how to migrate in-tree
in VMware's documentation page
Migrating In-Tree vSphere Volumes to vSphere Container Storage lug-in
.
If vSphere CSI Driver is not installed volume operations can not be performed on the PV created with the in-tree
type.
You must run vSphere 7.0u2 or later in order to migrate to the vSphere CSI driver.
If you are running a version of Kubernetes other than v1.26, consult
the documentation for that version of Kubernetes.
Note:
The following StorageClass parameters from the built-in
plugin are not supported by the vSphere CSI driver:
diskformat
hostfailurestotolerate
forceprovisioning
cachereservation
diskstripes
objectspacereservation
iopslimit
Existing volumes created using these parameters will be migrated to the vSphere CSI driver,
but new volumes created by the vSphere CSI driver will not be honoring these parameters.
vSphere CSI migration complete
Kubernetes v1.19 [beta]
To turn off the
plugin from being loaded by the controller manager and the kubelet, you need to set
feature flag to
true
. You must install a
CSI
driver on all worker nodes.
Sometimes, it is useful to share one volume for multiple uses in a single pod.
The
property specifies a sub-path inside the referenced volume
instead of its root.
The following example shows how to configure a Pod with a LAMP stack (Linux Apache MySQL PHP)
using a single, shared volume. This sample
subPath
configuration is not recommended
for production use.
The PHP application's code and assets map to the volume's
html
folder and
the MySQL database is stored in the volume's
mysql
folder. For example:
:
v1
kind
:
Pod
metadata
:
name
:
my-lamp-site
spec
:
:
-
name
:
mysql
image
:
mysql
env
:
-
name
:
value
:
"rootpasswd"
:
-
mountPath
:
name
:
site-data
subPath
:
mysql
-
name
:
php
image
:
php:7.0-apache
:
-
mountPath
:
name
:
site-data
subPath
:
html
volumes
:
-
name
:
site-data
:
claimName
:
my-lamp-site-data
Using subPath with expanded environment variables
Use the
field to construct
subPath
directory names from
downward API environment variables.
The
subPath
and
properties are mutually exclusive.
In this example, a
Pod
uses
to create a directory
pod1
within
the
hostPath
volume
.
The
hostPath
volume takes the
Pod
name from the
.
The host directory
/var/log/pods/pod1
is mounted at
/logs
in the container.
:
v1
kind
:
Pod
metadata
:
name
:
pod1
spec
:
:
-
name
:
container1
env
:
-
name
:
POD_NAME
valueFrom
:
fieldRef
:
:
v1
fieldPath
:
image
:
command
:
[
"sh"
,
"-c"
,
"while [ true ]; do echo 'Hello'; sleep 10; done | tee -a /logs/hello.txt"
]
:
-
name
:
workdir1
mountPath
:
/logs
# The variable expansion uses round brackets (not curly brackets).
:
$(POD_NAME)
:
Never
volumes
:
-
name
:
workdir1

hostPath
:
path
:
Resources
The storage media (such as Disk or SSD) of an
emptyDir
volume is determined by the
medium of the filesystem holding the kubelet root dir (typically
). There is no limit on how much space an
emptyDir
or
hostPath
volume can consume, and no isolation between containers or between
pods.
To learn about requesting space using a resource specification, see
how to manage resources
.
Out-of-tree volume plugins
The out-of-tree volume plugins include
(CSI), and also FlexVolume (which is deprecated). These plugins enable storage vendors to create custom storage plugins
without adding their plugin source code to the Kubernetes repository.
Previously, all volume plugins were "in-tree". The "in-tree" plugins were built, linked, compiled,
and shipped with the core Kubernetes binaries. This meant that adding a new storage system to
Kubernetes (a volume plugin) required checking code into the core Kubernetes code repository.
Both CSI and FlexVolume allow volume plugins to be developed independent of
the Kubernetes code base, and deployed (installed) on Kubernetes clusters as
extensions.
For storage vendors looking to create an out-of-tree volume plugin, please refer
to the
volume plugin FAQ
.
csi
(CSI) defines a standard interface for container orchestration systems (like
Kubernetes) to expose arbitrary storage systems to their container workloads.
Please read the
CSI design proposal
Note:
Support for CSI spec versions 0.2 and 0.3 are deprecated in Kubernetes
v1.13 and will be removed in a future release.
Note:
CSI drivers may not be compatible across all Kubernetes releases.
Please check the specific CSI driver's documentation for supported
deployments steps for each Kubernetes release and a compatibility matrix.
Once a CSI compatible volume driver is deployed on a Kubernetes cluster, users
may use the
csi
volume type to attach or mount the volumes exposed by the
CSI driver.
A
csi
volume can be used in a Pod in three different ways:
through a reference to a
with a
generic ephemeral volume
with a
CSI ephemeral volume
if the driver supports that
The following fields are available to storage administrators to configure a CSI
persistent volume:
driver
: A string value that specifies the name of the volume driver to use.
This value must correspond to the value returned in the
GetPluginInfoResponse
by the CSI driver as defined in the
CSI spec
.
It is used by Kubernetes to identify which CSI driver to call out to, and by
CSI driver components to identify which PV objects belong to the CSI driver.
: A string value that uniquely identifies the volume. This value
must correspond to the value returned in the
volume.id
CreateVolumeResponse
CSI spec
.
The value is passed as
volume_id
on all calls to the CSI volume driver when
referencing the volume.
readOnly
: An optional boolean value indicating whether the volume is to be
"ControllerPublished" (attached) as read only. Default is false. This value is
passed to the CSI driver via the
readonly
ControllerPublishVolumeRequest
.
fsType
: If the PV's
VolumeMode
is
then this field may be used
to specify the filesystem that should be used to mount the volume. If the
volume has not been formatted and formatting is supported, this value will be
used to format the volume.
This value is passed to the CSI driver via the
VolumeCapability
field of
,
NodeStageVolumeRequest
, and
NodePublishVolumeRequest
.
: A map of string to string that specifies static properties
of a volume. This map must correspond to the map returned in the
volume.attributes
by the CSI driver as
CSI spec
.
The map is passed to the CSI driver via the
volume_context
,
Node

StageVolumeRequest
, and
.
: A reference to the secret object containing
sensitive information to pass to the CSI driver to complete the CSI
ControllerPublishVolume
and
ControllerUnpublishVolume
calls. This field is
optional, and may be empty if no secret is required. If the Secret
contains more than one secret, all secrets are passed.
nodeExpandSecretRef
: A reference to the secret containing sensitive
information to pass to the CSI driver to complete the CSI
call. This field is optional, and may be empty if no
secret is required. If the object contains more than one secret, all
secrets are passed. When you have configured secret data for node-initiated
volume expansion, the kubelet passes that data via the
NodeExpandVolume()
call to the CSI driver. In order to use the
field, your
cluster should be running Kubernetes version 1.25 or later and you must enable
the
named
for each kube-apiserver and for the kubelet on every
node. You must also be using a CSI driver that supports or requires secret data during
node-initiated storage resize operations.
secret is required. If the secret object contains more than one secret, all
secrets are passed.
NodeStageVolume
call. This field is optional, and may be empty if no secret
is required. If the Secret contains more than one secret, all secrets
are passed.
CSI raw block volume support
Vendors with external CSI drivers can implement raw block volume support
in Kubernetes workloads.
You can set up your
PersistentVolume/PersistentVolumeClaim with raw block volume support
as usual, without any CSI specific changes.
CSI ephemeral volumes
You can directly configure CSI volumes within the Pod
specification. Volumes specified in this way are ephemeral and do not
persist across pod restarts. See
Ephemeral
Volumes
For more information on how to develop a CSI driver, refer to the
kubernetes-csi documentation
Windows CSI proxy
CSI node plugins need to perform various privileged
operations like scanning of disk devices and mounting of file systems. These operations
differ for each host operating system. For Linux worker nodes, containerized CSI node
node plugins are typically deployed as privileged containers. For Windows worker nodes,
privileged operations for containerized CSI node plugins is supported using
csi-proxy
, a community-managed,
stand-alone binary that needs to be pre-installed on each Windows node.
For more details, refer to the deployment guide of the CSI plugin you wish to deploy.
Migrating to CSI drivers from in-tree plugins
The
feature directs operations against existing in-tree
plugins to corresponding CSI plugins (which are expected to be installed and configured).
As a result, operators do not have to make any
configuration changes to existing Storage Classes, PersistentVolumes or PersistentVolumeClaims
(referring to in-tree plugins) when transitioning to a CSI driver that supersedes an in-tree plugin.
The operations and features that are supported include:
provisioning/delete, attach/detach, mount/unmount and resizing of volumes.
In-tree plugins that support
and have a corresponding CSI driver implemented
are listed in
Types of Volumes
.
The following in-tree plugins support persistent storage on Windows nodes:
azureDisk
azureFile
flexVolume (deprecated)
FEATURE ST

ATE:
Kubernetes v1.23 [deprecated]
FlexVolume is an out-of-tree plugin interface that uses an exec-based model to interface
with storage drivers. The FlexVolume driver binaries must be installed in a pre-defined
volume plugin path on each node and in some cases the control plane nodes as well.
Pods interact with FlexVolume drivers through the
in-tree volume plugin.
For more details, see the FlexVolume
README
document.
The following FlexVolume
plugins
,
deployed as PowerShell scripts on the host, support Windows nodes:
SMB
iSCSI
Note:
FlexVolume is deprecated. Using an out-of-tree CSI driver is the recommended way to integrate external storage with Kubernetes.
Maintainers of FlexVolume driver should implement a CSI Driver and help to migrate users of FlexVolume drivers to CSI.
Users of FlexVolume should move their workloads to use the equivalent CSI Driver.
Mount propagation
Mount propagation allows for sharing volumes mounted by a container to
other containers in the same pod, or even to other pods on the same node.
Mount propagation of a volume is controlled by the
field
in
Container.volumeMounts
. Its values are:
None
- This volume mount will not receive any subsequent mounts
that are mounted to this volume or any of its subdirectories by the host.
In similar fashion, no mounts created by the container will be visible on
the host. This is the default mode.
This mode is equal to
private
mount propagation as described in the
Linux kernel documentation
HostToContainer
- This volume mount will receive all subsequent mounts
that are mounted to this volume or any of its subdirectories.
In other words, if the host mounts anything inside the volume mount, the
container will see it mounted there.
Similarly, if any Pod with
Bidirectional
mount propagation to the same
volume mounts anything there, the container with
mount
propagation will see it.
rslave
- This volume mount behaves the same the
mount.
In addition, all volume mounts created by the container will be propagated
back to the host and to all containers of all pods that use the same volume.
A typical use case for this mode is a Pod with a FlexVolume or CSI driver or
a Pod that needs to mount something on the host using a
hostPath
volume.
rshared
Warning:
mount propagation can be dangerous. It can damage
the host operating system and therefore it is allowed only in privileged
containers. Familiarity with Linux kernel behavior is strongly recommended.
In addition, any volume mounts created by containers in pods must be destroyed
(unmounted) by the containers on termination.
Before mount propagation can work properly on some deployments (CoreOS,
RedHat/Centos, Ubuntu) mount share must be configured correctly in
Docker as shown below.
Edit your Docker's
systemd
service file. Set
MountFlags
as follows:
=
shared
Or, remove
MountFlags=slave
if present. Then restart the Docker daemon:
sudo systemctl restart docker
deploying WordPress and MySQL with Persistent Volumes
.
7.2 - Persistent Volumes
This document describes
persistent volumes
in Kubernetes. Familiarity with
volumes
Managing storage is a distinct problem from managing compute instances. The PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this, we introduce two new API resources: PersistentVolume and PersistentVolumeClaim.
A
(PV) is a piece of storage in the cluster that has been provisioned by an administrator or dynamically provisioned using
. It is a resource in the cluster just like a node is a cluster resource. PVs ar

e volume plugins like Volumes, but have a lifecycle independent of any individual Pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.
A
(PVC) is a request for storage by a user. It is similar to a Pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., they can be mounted ReadWriteOnce, ReadOnlyMany or ReadWriteMany, see
AccessModes
).
While PersistentVolumeClaims allow a user to consume abstract storage resources, it is common that users need PersistentVolumes with varying properties, such as performance, for different problems. Cluster administrators need to be able to offer a variety of PersistentVolumes that differ in more ways than size and access modes, without exposing users to the details of how those volumes are implemented. For these needs, there is the
resource.
See the
detailed walkthrough with working examples
.
Lifecycle of a volume and claim
PVs are resources in the cluster. PVCs are requests for those resources and also act as claim checks to the resource. The interaction between PVs and PVCs follows this lifecycle:
Provisioning
There are two ways PVs may be provisioned: statically or dynamically.
Static
A cluster administrator creates a number of PVs. They carry the details of the real storage, which is available for use by cluster users. They exist in the Kubernetes API and are available for consumption.
Dynamic
When none of the static PVs the administrator created match a user's PersistentVolumeClaim,
the cluster may try to dynamically provision a volume specially for the PVC.
This provisioning is based on StorageClasses: the PVC must request a
and
the administrator must have created and configured that class for dynamic
provisioning to occur. Claims that request the class
""
effectively disable
dynamic provisioning for themselves.
To enable dynamic storage provisioning based on storage class, the cluster administrator
needs to enable the
on the API server. This can be done, for example, by ensuring that
is
among the comma-delimited, ordered list of values for the
flag of
the API server component. For more information on API server command-line flags,
check
Binding
A user creates, or in the case of dynamic provisioning, has already created, a PersistentVolumeClaim with a specific amount of storage requested and with certain access modes. A control loop in the master watches for new PVCs, finds a matching PV (if possible), and binds them together. If a PV was dynamically provisioned for a new PVC, the loop will always bind that PV to the PVC. Otherwise, the user will always get at least what they asked for, but the volume may be in excess of what was requested. Once bound, PersistentVolumeClaim binds are exclusive, regardless of how they were bound. A PVC to PV binding is a one-to-one mapping, using a ClaimRef which is a bi-directional binding between the PersistentVolume and the PersistentVolumeClaim.
Claims will remain unbound indefinitely if a matching volume does not exist. Claims will be bound as matching volumes become available. For example, a cluster provisioned with many 50Gi PVs would not match a PVC requesting 100Gi. The PVC can be bound when a 100Gi PV is added to the cluster.
Using
Pods use claims as volumes. The cluster inspects the claim to find the bound volume and mounts that volume for a Pod. For volumes that support multiple access modes, the user specifies which mode is desired when using their claim as a volume in a Pod.
Once a user has a claim and that claim is bound, the bound PV belongs to the user for as long as they need it. Users schedule Pods and access their claimed PVs by including

 a
section in a Pod's
volumes
block. See
Claims As Volumes
for more details on this.
The purpose of the Storage Object in Use Protection feature is to ensure that PersistentVolumeClaims (PVCs) in active use by a Pod and PersistentVolume (PVs) that are bound to PVCs are not removed from the system, as this may result in data loss.
Note:
PVC is in active use by a Pod when a Pod object exists that is using the PVC.
If a user deletes a PVC in active use by a Pod, the PVC is not removed immediately. PVC removal is postponed until the PVC is no longer actively used by any Pods. Also, if an admin deletes a PV that is bound to a PVC, the PV is not removed immediately. PV removal is postponed until the PV is no longer bound to a PVC.
You can see that a PVC is protected when the PVC's status is
and the
list includes
:
kubectl describe pvc hostpath
Name:          hostpath
StorageClass:  example-hostpath
Status:        Terminating
Volume:
Annotations:   volume.beta.kubernetes.io/storage-class
=
example-hostpath
volume.beta.kubernetes.io/storage-provisioner
=
example.com/hostpath
Finalizers:
[
]
...
You can see that a PV is protected when the PV's status is
and the
too:
kubectl describe pv task-pv-volume
Name:            task-pv-volume
Labels:
type
=
local
[
]
StorageClass:    standard
Status:          Terminating
Claim:
Reclaim Policy:  Delete
Access Modes:    RWO
Capacity:        1Gi
Message:
Source:
Type:          HostPath
(
bare host directory volume
)
Path:          /tmp/data
HostPathType:
Events:            &lt;none&gt;
Reclaiming
When a user is done with their volume, they can delete the PVC objects from the API that allows reclamation of the resource. The reclaim policy for a PersistentVolume tells the cluster what to do with the volume after it has been released of its claim. Currently, volumes can either be Retained, Recycled, or Deleted.
Retain
The
Retain
reclaim policy allows for manual reclamation of the resource. When the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered "released". But it is not yet available for another claim because the previous claimant's data remains on the volume. An administrator can manually reclaim the volume with the following steps.
Delete the PersistentVolume. The associated storage asset in external infrastructure (such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume) still exists after the PV is deleted.
Manually clean up the data on the associated storage asset accordingly.
Manually delete the associated storage asset.
If you want to reuse the same storage asset, create a new PersistentVolume with the same storage asset definition.
Delete
For volume plugins that support the
Delete
reclaim policy, deletion removes both the PersistentVolume object from Kubernetes, as well as the associated storage asset in the external infrastructure, such as an AWS EBS, GCE PD, Azure Disk, or Cinder volume. Volumes that were dynamically provisioned inherit the
reclaim policy of their StorageClass
, which defaults to
Delete
. The administrator should configure the StorageClass according to users' expectations; otherwise, the PV must be edited or patched after it is created. See
.
Recycle
Warning:
The
Recycle
reclaim policy is deprecated. Instead, the recommended approach is to use dynamic provisioning.
If supported by the underlying volume plugin, the
Recycle
reclaim policy performs a basic scrub (
rm -rf /thevolume/*
) on the volume and makes it available again for a new claim.
However, an administrator can configure a custom recycler Pod template using
the Kubernetes controller manager command line

 arguments as described in the
reference
.
The custom recycler Pod template must contain a
volumes
specification, as
shown in the example below:
:
v1
kind
:
Pod
metadata
:
name
:
pv-recycler
namespace
:
default
spec
:
:
Never
volumes
:
-
name
:
vol
hostPath
:
path
:
/any/path/it/will/be/replaced
:
-
name
:
image
:
"registry.k8s.io/busybox"
command
:
[
"/bin/sh"
,
"-c"
,
"test -e /scrub &amp;&amp; rm -rf /scrub/..?* /scrub/.[!.]* /scrub/*  &amp;&amp; test -z \"$(ls -A /scrub)\" || exit 1"
]
:
-
name
:
vol
mountPath
:
/scrub
However, the particular path specified in the custom recycler Pod template in the
volumes
part is replaced with the particular path of the volume that is being recycled.
Finalizers can be added on a PersistentVolume to ensure that PersistentVolumes
having
Delete
reclaim policy are deleted only after the backing storage are deleted.
The newly introduced finalizers
kubernetes.io/pv-controller
and
external-provisioner.volume.kubernetes.io/finalizer
are only added to dynamically provisioned volumes.
The finalizer
is added to in-tree plugin volumes. The following is an example
kubectl describe pv pvc-74a498d6-3929-47e8-8c02-078c1ece4d78
Name:            pvc-74a498d6-3929-47e8-8c02-078c1ece4d78
Annotations:     kubernetes.io/createdby: vsphere-volume-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: yes
pv.kubernetes.io/provisioned-by: kubernetes.io/vsphere-volume
[
kubernetes.io/pv-protection kubernetes.io/pv-controller
]
StorageClass:    vcp-sc
Status:          Bound
Claim:           default/vcp-pvc-1
VolumeMode:      Filesystem
Node Affinity:   &lt;none&gt;
Message:
Source:
Type:               vSphereVolume
(
a Persistent Disk resource in vSphere
)
VolumePath:
[
vsanDatastore
]
d49c4a62-166f-ce12-c464-020077ba5d46/kubernetes-dynamic-pvc-74a498d6-3929-47e8-8c02-078c1ece4d78.vmdk
FSType:             ext4
StoragePolicyName:  vSAN Default Storage Policy
Events:                 &lt;none&gt;
is added for CSI volumes.
The following is an example:
Name:            pvc-2f0bab97-85a8-4552-8044-eb8be45cf48d
Annotations:     pv.kubernetes.io/provisioned-by: csi.vsphere.vmware.com
[
kubernetes.io/pv-protection external-provisioner.volume.kubernetes.io/finalizer
]
StorageClass:    fast
Claim:           demo-app/nginx-logs
Capacity:        200Mi
Message:
Source:
Type:              CSI
(
a Container Storage Interface
(
CSI
)
volume
source
)
Driver:            csi.vsphere.vmware.com
FSType:            ext4
VolumeHandle:      44830fa8-79b4-406b-8b58-621ba25353fd
ReadOnly:
false
VolumeAttributes:      storage.kubernetes.io/csiProvisionerIdentity
=
1648442357185-8081-csi.vsphere.vmware.com
type
=
vSphere CNS Block Volume
Events:                &lt;none&gt;
When the
CSIMigration{provider}
feature flag is enabled for a specific in-tree volume plugin,
the
finalizer is replaced by the
finalizer.
Reserving a PersistentVolume
The control plane can
bind PersistentVolumeClaims to matching PersistentVolumes
in the
cluster. However, if you want a PVC to bind to a specific PV, you need to pre-bind them.
By specifying a PersistentVolume in a PersistentVolumeClaim, you declare a binding between that specific PV and PVC.
If the PersistentVolume exists and has not reserved PersistentVolumeClaims through its
claimRef
field, then the PersistentVolume and PersistentVolumeClaim will be bound.
The binding happens regardless of some volume matching criteria

, including node affinity.
The control plane still checks that
, access modes, and requested storage size are valid.
:
v1
kind
:
metadata
:
name
:
foo-pvc
namespace
:
foo
spec
:
:
""
# Empty string must be explicitly set otherwise default StorageClass will be set
:
foo-pv
...
This method does not guarantee any binding privileges to the PersistentVolume. If other PersistentVolumeClaims could use the PV that you specify, you first need to reserve that storage volume. Specify the relevant PersistentVolumeClaim in the
claimRef
field of the PV so that other PVCs can not bind to it.
:
v1
kind
:
metadata
:
name
:
foo-pv
spec
:
:
""
claimRef
:
name
:
foo-pvc
namespace
:
foo
...
This is useful if you want to consume PersistentVolumes that have their
claimPolicy
set
to
Retain
, including cases where you are reusing an existing PV.
Support for expanding PersistentVolumeClaims (PVCs) is enabled by default. You can expand
the following types of volumes:
azureDisk
azureFile
csi
rbd
You can only expand a PVC if its storage class's
field is set to true.
:
kind
:
metadata
:
name
:
example-vol-default
:
vendor-name.example/magicstorage
:
resturl
:
restuser
:
""
:
""
:
""
:
true
To request a larger volume for a PVC, edit the PVC object and specify a larger
size. This triggers expansion of the volume that backs the underlying PersistentVolume. A
new PersistentVolume is never created to satisfy the claim. Instead, an existing volume is resized.
Warning:
Directly editing the size of a PersistentVolume can prevent an automatic resize of that volume.
If you edit the capacity of a PersistentVolume, and then edit the
.spec
of a matching
PersistentVolumeClaim to make the size of the PersistentVolumeClaim match the PersistentVolume,
then no storage resize happens.
The Kubernetes control plane will see that the desired state of both resources matches,
conclude that the backing volume size has been manually
increased and that no resize is necessary.
CSI Volume expansion
Support for expanding CSI volumes is enabled by default but it also requires a specific CSI driver to support volume expansion. Refer to documentation of the specific CSI driver for more information.
Resizing a volume containing a file system
You can only resize volumes containing a file system if the file system is XFS, Ext3, or Ext4.
When a volume contains a file system, the file system is only resized when a new Pod is using
the PersistentVolumeClaim in
ReadWrite
mode. File system expansion is either done when a Pod is starting up
or when a Pod is running and the underlying file system supports online expansion.
FlexVolumes (deprecated since Kubernetes v1.23) allow resize if the driver is configured with the
RequiresFSResize
capability to
true
. The FlexVolume can be resized on Pod restart.
In this case, you don't need to delete and recreate a Pod or deployment that is using an existing PVC.
Any in-use PVC automatically becomes available to its Pod as soon as its file system has been expanded.
This feature has no effect on PVCs that are not in use by a Pod or deployment. You must create a Pod that
uses the PVC before the expansion can complete.
Similar to other volume types - FlexVolume volumes can also be expanded when in-use by a Pod.
Note:
FlexVolume resize is possible only when the underlying driver supports resize.
Note:
Expanding EBS volumes is a time-consuming operation. Also, there is a per-volume quota of one

 modification every 6 hours.
If a user specifies a new size that is too big to be satisfied by underlying storage system, expansion of PVC will be continuously retried until user or cluster administrator takes some action. This can be undesirable and hence Kubernetes provides following methods of recovering from such failures.
Manually with Cluster Administrator access
By requesting expansion to smaller size
If expanding underlying storage fails, the cluster administrator can manually recover the Persistent Volume Claim (PVC) state and cancel the resize requests. Otherwise, the resize requests are continuously retried by the controller without administrator intervention.
Mark the PersistentVolume(PV) that is bound to the PersistentVolumeClaim(PVC) with
Retain
reclaim policy.
Delete the PVC. Since PV has
Retain
reclaim policy - we will not lose any data when we recreate the PVC.
Delete the
claimRef
entry from PV specs, so as new PVC can bind to it. This should make the PV
Available
.
Re-create the PVC with smaller size than PV and set
field of the PVC to the name of the PV. This should bind new PVC to existing PV.
Don't forget to restore the reclaim policy of the PV.
Note:
Recovery from failing PVC expansion by users is available as an alpha feature since Kubernetes 1.23. The
feature must be enabled for this feature to work. Refer to the
If the feature gates
is
enabled in your cluster, and expansion has failed for a PVC, you can retry expansion with a
smaller size than the previously requested value. To request a new expansion attempt with a
smaller proposed size, edit
.spec.resources
for that PVC and choose a value that is less than the
value you previously tried.
This is useful if expansion to a higher value did not succeed because of capacity constraint.
If that has happened, or you suspect that it might have, you can retry expansion by specifying a
size that is within the capacity limits of underlying storage provider. You can monitor status of resize operation by watching
.status.resizeStatus
and events on the PVC.
Note that,
although you can specify a lower amount of storage than what was requested previously,
the new value must still be higher than
.status.capacity
.
Kubernetes does not support shrinking a PVC to less than its current size.
Types of Persistent Volumes
PersistentVolume types are implemented as plugins. Kubernetes currently supports the following plugins:
cephfs
- CephFS volume
csi
- Container Storage Interface (CSI)
fc
- Fibre Channel (FC) storage
hostPath
- HostPath volume
(for single node testing only; WILL NOT WORK in a multi-node cluster;
consider using
local
volume instead)
iscsi
- iSCSI (SCSI over IP) storage
local
- local storage devices
mounted on nodes.
nfs
- Network File System (NFS) storage
rbd
- Rados Block Device (RBD) volume
The following types of PersistentVolume are deprecated. This means that support is still available but will be removed in a future Kubernetes release.
- AWS Elastic Block Store (EBS)
(
in v1.17)
azureDisk
- Azure Disk
(
in v1.19)
azureFile
- Azure File
(
in v1.21)
cinder
- Cinder (OpenStack block storage)
(
in v1.18)
- FlexVolume
(
in v1.23)
- GCE Persistent Disk
(
in v1.17)
- Portworx volume
(
in v1.25)
- vSphere VMDK volume
(
in v1.19)
Older versions of Kubernetes also supported the following in-tree PersistentVolume types:
- Photon controller persistent disk.
(
not available
starting v1.15)
scaleIO
- ScaleIO volume
(
starting v1.21)
flocker
- Flocker storage
(
starting v1.25)
quobyte
- Quobyte volume
(
storageos
- StorageOS

 volume
(
Each PV contains a spec and status, which is the specification and status of the volume.
The name of a PersistentVolume object must be a valid
.
:
v1
kind
:
metadata
:
name
:
pv0003
spec
:
capacity
:
storage
:
5Gi
:
:
:
Recycle
:
slow
:
- hard
- nfsvers=4.1
nfs
:
path
:
/tmp
server
:
172.17.0.2
Note:
Helper programs relating to the volume type may be required for consumption of a PersistentVolume within a cluster. In this example, the PersistentVolume is of type NFS and the helper program /sbin/mount.nfs is required to support the mounting of NFS filesystems.
Capacity
Generally, a PV will have a specific storage capacity. This is set using the PV's
capacity
attribute. Read the glossary term
Quantity
to understand the units expected by
capacity
.
Currently, storage size is the only resource that can be set or requested. Future attributes may include IOPS, throughput, etc.
Volume Mode
Kubernetes supports two
volumeModes
of PersistentVolumes:
and
Block
.
is an optional API parameter.
is the default mode used when
parameter is omitted.
A volume with
volumeMode: Filesystem
is
mounted
into Pods into a directory. If the volume
is backed by a block device and the device is empty, Kubernetes creates a filesystem
on the device before mounting it for the first time.
You can set the value of
to
Block
to use a volume as a raw block device.
Such volume is presented into a Pod as a block device, without any filesystem on it.
This mode is useful to provide a Pod the fastest possible way to access a volume, without
any filesystem layer between the Pod and the volume. On the other hand, the application
running in the Pod must know how to handle a raw block device.
See
for an example on how to use a volume with
volumeMode: Block
in a Pod.
Access Modes
A PersistentVolume can be mounted on a host in any way supported by the resource provider. As shown in the table below, providers will have different capabilities and each PV's access modes are set to the specific modes supported by that particular volume. For example, NFS can support multiple read/write clients, but a specific NFS PV might be exported on the server as read-only. Each PV gets its own set of access modes describing that specific PV's capabilities.
The access modes are:
the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can allow multiple pods to access the volume when the pods are running on the same node.
ReadOnlyMany
the volume can be mounted as read-only by many nodes.
ReadWriteMany
the volume can be mounted as read-write by many nodes.
the volume can be mounted as read-write by a single Pod. Use ReadWriteOncePod access mode if you want to ensure that only one pod across whole cluster can read that PVC or write to it. This is only supported for CSI volumes and Kubernetes version 1.22+.
The blog article
Introducing Single Pod Access Mode for PersistentVolumes
covers this in more detail.
In the CLI, the access modes are abbreviated to:
RWO - ReadWriteOnce
ROX - ReadOnlyMany
RWX - ReadWriteMany
RWOP - ReadWriteOncePod
Note:
Kubernetes uses volume access modes to match PersistentVolumeClaims and PersistentVolumes.
In some cases, the volume access modes also constrain where the PersistentVolume can be mounted.
Volume access modes do
not
enforce write protection once the storage has been mounted.
Even if the access modes are specified as ReadWriteOnce, ReadOnlyMany, or ReadWriteMany, they don't set any constraints on the volume.
For example, even if a PersistentVolume is created as ReadOnlyMany, it is no guarantee that it will be read-only.
If the access modes are specified as ReadWrite

OncePod, the volume is constrained and can be mounted on only a single Pod.
Important!
A volume can only be mounted using one access mode at a time, even if it supports many. For example, a GCEPersistentDisk can be mounted as ReadWriteOnce by a single node or ReadOnlyMany by many nodes, but not at the same time.
AWSElasticBlockStore
✓
-
-
-
AzureFile
✓
✓
✓
-
AzureDisk
✓
-
-
-
CephFS
✓
✓
✓
-
Cinder
✓
-
(
if multi-attach volumes are available
)
-
CSI
depends on the driver
FC
✓
✓
-
-
✓
✓
-
GCEPersistentDisk
✓
✓
-
-
Glusterfs
✓
✓
✓
-
HostPath
✓
-
-
-
iSCSI
✓
✓
-
-
NFS
✓
✓
✓
-
RBD
✓
✓
-
-
VsphereVolume
✓
-
- (works when Pods are collocated)
-
PortworxVolume
✓
-
✓
-
Class
A PV can have a class, which is specified by setting the
attribute to the name of a
.
A PV of a particular class can only be bound to PVCs requesting
that class. A PV with no
has no class and can only be bound
to PVCs that request no particular class.
In the past, the annotation
volume.beta.kubernetes.io/storage-class
was used instead
of the
attribute. This annotation is still working; however,
it will become fully deprecated in a future Kubernetes release.
Reclaim Policy
Current reclaim policies are:
Retain -- manual reclamation
Recycle -- basic scrub (
)
Delete -- associated storage asset such as AWS EBS, GCE PD, Azure Disk, or OpenStack Cinder volume is deleted
Currently, only NFS and HostPath support recycling. AWS EBS, GCE PD, Azure Disk, and Cinder volumes support deletion.
Mount Options
A Kubernetes administrator can specify additional mount options for when a Persistent Volume is mounted on a node.
Note:
Not all Persistent Volume types support mount options.
The following volume types support mount options:
azureDisk
azureFile
cephfs
cinder
(
in v1.18)
iscsi
nfs
rbd
Mount options are not validated. If a mount option is invalid, the mount fails.
volume.beta.kubernetes.io/mount-options
of the
Note:
For most volume types, you do not need to set this field. It is automatically populated for
AWS EBS
,
GCE PD
and
Azure Disk
volume block types. You need to explicitly set this for
local
volumes.
A PV can specify node affinity to define constraints that limit what nodes this volume can be accessed from. Pods that use a PV will only be scheduled to nodes that are selected by the node affinity. To specify node affinity, set
in the
.spec
of a PV. The
API reference has more details on this field.
Phase
A volume will be in one of the following phases:
Available -- a free resource that is not yet bound to a claim
Bound -- the volume is bound to a claim
Released -- the claim has been deleted, but the resource is not yet reclaimed by the cluster
Failed -- the volume has failed its automatic reclamation
The CLI will show the name of the PVC bound to the PV.
Each PVC contains a spec and status, which is the specification and status of the claim.
The name of a PersistentVolumeClaim object must be a valid
.
:
v1
kind
:
metadata
:
name
:
myclaim
spec
:
:
:
resources
:
requests
:
storage
:
8Gi
:
slow
selector
:
:
release
:
"stable"
:
- {
key: environment, operator: In, values
:
[dev]}
Claims use
the same conventions as volumes
when requesting storage with specific access modes.
Volume Modes
Claims use
the same convention as volumes
to indicate the consumption of the volu

me as either a filesystem or block device.
Resources
Claims, like Pods, can request specific quantities of a resource. In this case, the request is for storage. The same
resource model
applies to both volumes and claims.
Selector
Claims can specify a
to further filter the set of volumes. Only the volumes whose labels match the selector can be bound to the claim. The selector can consist of two fields:
- the volume must have a label with this value
- a list of requirements made by specifying key, list of values, and operator that relates the key and values. Valid operators include In, NotIn, Exists, and DoesNotExist.
and
, are ANDed together – they must all be satisfied in order to match.
Class
A claim can request a particular class by specifying the name of a
using the attribute
.
Only PVs of the requested class, ones with the same
as the PVC, can
be bound to the PVC.
PVCs don't necessarily have to request a class. A PVC with its
set
equal to
""
is always interpreted to be requesting a PV with no class, so it
can only be bound to PVs with no class (no annotation or one set equal to
""
). A PVC with no
is not quite the same and is treated differently
by the cluster, depending on whether the
is turned on.
If the admission plugin is turned on, the administrator may specify a
default StorageClass. All PVCs that have no
can be bound only to
PVs of that default. Specifying a default StorageClass is done by setting the
equal to
true
in
a StorageClass object. If the administrator does not specify a default, the
cluster responds to PVC creation as if the admission plugin were turned off. If
more than one default is specified, the admission plugin forbids the creation of
all PVCs.
If the admission plugin is turned off, there is no notion of a default
StorageClass. All PVCs that have
set to
""
can be
bound only to PVs that have
also set to
""
.
However, PVCs with missing
can be updated later once
default StorageClass becomes available. If the PVC gets updated it will no
longer bind to PVs that have
also set to
""
.
See
retroactive default StorageClass assignment
Depending on installation method, a default StorageClass may be deployed
to a Kubernetes cluster by addon manager during installation.
When a PVC specifies a
selector
in addition to requesting a StorageClass,
the requirements are ANDed together: only a PV of the requested class and with
the requested labels may be bound to the PVC.
Note:
Currently, a PVC with a non-empty
selector
can't have a PV dynamically provisioned for it.
of
it won't be supported in a future Kubernetes release.
Retroactive default StorageClass assignment
You can create a PersistentVolumeClaim without specifying a
for the new PVC, and you can do so even when no default StorageClass exists in your cluster. In this case, the new PVC creates as you defined it, and the
of that PVC remains unset until default becomes available.
When a default StorageClass becomes available, the control plane identifies any existing PVCs without
. For the PVCs that either have an empty value for
or do not have this key, the control plane then updates those PVCs to set
to match the new default StorageClass. If you have an existing PVC where the
is
""
, and you configure a default StorageClass, then this PVC will not get updated.
In order to keep binding to PVs with

set to
""
(while a default StorageClass is present), you need to set the
of the associated PVC to
""
.
This behavior helps administrators change default StorageClass by removing the old one first and then creating or setting another one. This brief window while there is no default causes PVCs without
created at that time to not have any default, but due to the retroactive default StorageClass assignment this way of changing defaults is safe.
Pods access storage by using the claim as a volume. Claims must exist in the same namespace as the Pod using the claim. The cluster finds the claim in the Pod's namespace and uses it to get the PersistentVolume backing the claim. The volume is then mounted to the host and into the Pod.
:
v1
kind
:
Pod
metadata
:
name
:
mypod
spec
:
:
-
name
:
myfrontend
image
:
nginx
:
-
mountPath
:
"/var/www/html"
name
:
mypd
volumes
:
-
name
:
mypd
:
claimName
:
myclaim
A Note on Namespaces
PersistentVolumes binds are exclusive, and since PersistentVolumeClaims are namespaced objects, mounting claims with "Many" modes (
ROX
,
RWX
) is only possible within one namespace.
PersistentVolumes typed
hostPath
A
hostPath
See
an example of
hostPath
typed volume
.
The following volume plugins support raw block volumes, including dynamic provisioning where
applicable:
AzureDisk
CSI
FC (Fibre Channel)
iSCSI
Local volume
RBD (Ceph Block Device)
PersistentVolume using a Raw Block Volume
:
v1
kind
:
metadata
:
name
:
block-pv
spec
:
capacity
:
storage
:
10Gi
:
:
Block
:
Retain
fc
:
:
[
"50060e801049cfd1"
]
lun
:
0
readOnly
:
false
PersistentVolumeClaim requesting a Raw Block Volume
:
v1
kind
:
metadata
:
name
:
block-pvc
spec
:
:
:
Block
resources
:
requests
:
storage
:
10Gi
Pod specification adding Raw Block Device path in container
:
v1
kind
:
Pod
metadata
:
name
:
pod-with-block-volume
spec
:
:
-
name
:
fc-container
image
:
fedora:26
command
:
[
"/bin/sh"
,
"-c"
]
args
:
[
"tail -f /dev/null"
]
:
-
name
:
data
:
/dev/xvda
volumes
:
-
name
:
data
:
claimName
:
block-pvc
Note:
When adding a raw block device for a Pod, you specify the device path in the container instead of a mount path.
Binding Block Volumes
If a user requests a raw block volume by indicating this using the
field in the PersistentVolumeClaim spec, the binding rules differ slightly from previous releases that didn't consider this mode as part of the spec.
Listed is a table of possible combinations the user and admin might specify for requesting a raw block device. The table indicates if the volume will be bound or not given the combinations:
Volume binding matrix for statically provisioned volumes:
PV volumeMode
PVC volumeMode
Result
unspecified
BIND
Block
NO BIND
BIND
Block
NO BIND
Block
Block
BIND
Block
NO BIND
BIND
Block
NO BIND
BIND
Note:
Only statically provisioned volumes are supported for alpha release. Administrators should take care to consider these values when working with raw block devices.
Volume Snapshot and Restore Volume from Snapshot Support
Volume snapshots only support the out-of-tree CSI volume plugins. For details, see
.
In-tree volume plugins are deprecated. You can read about the deprecated volume plugins in the
Volume Plugin FAQ
.
Create a Persis

tentVolumeClaim from a Volume Snapshot
:
v1
kind
:
metadata
:
name
:
restore-pvc
spec
:
:
csi-hostpath-sc
:
name
:
new-snapshot-test
kind
:
VolumeSnapshot
apiGroup
:
snapshot.storage.k8s.io
:
resources
:
requests
:
storage
:
10Gi
Volume Cloning
only available for CSI volume plugins.
Create PersistentVolumeClaim from an existing PVC
:
v1
kind
:
metadata
:
name
:
cloned-pvc
spec
:
:
my-csi-plugin
:
name
:
existing-src-pvc-name
kind
:
:
resources
:
requests
:
storage
:
10Gi
Volume populators and data sources
Kubernetes supports custom volume populators.
To use custom volume populators, you must enable the
for
the kube-apiserver and kube-controller-manager.
Volume populators take advantage of a PVC spec field called
. Unlike the
field, which can only contain either a reference to another PersistentVolumeClaim
or to a VolumeSnapshot, the
field can contain a reference to any object in the
same namespace, except for core objects other than PVCs. For clusters that have the feature
gate enabled, use of the
is preferred over
.
Cross namespace data sources
Kubernetes supports cross namespace volume data sources.
To use cross namespace volume data sources, you must enable the
and
for
the kube-apiserver, kube-controller-manager.
Also, you must enable the
feature gate for the csi-provisioner.
feature gate allow you to specify a namespace in the dataSourceRef field.
Note:
When you specify a namespace for a volume data source, Kubernetes checks for a
ReferenceGrant in the other namespace before accepting the reference.
ReferenceGrant is part of the
gateway.networking.k8s.io
extension APIs.
See
ReferenceGrant
in the Gateway API documentation for details.
This means that you must extend your Kubernetes cluster with at least ReferenceGrant from the
Gateway API before you can use this mechanism.
Data source references
The
field behaves almost the same as the
field. If either one is
specified while the other is not, the API server will give both fields the same value. Neither
field can be changed after creation, and attempting to specify different values for the two
fields will result in a validation error. Therefore the two fields will always have the same
contents.
There are two differences between the
field and the
field that
users should be aware of:
The
field ignores invalid values (as if the field was blank) while the
field never ignores values and will cause an error if an invalid value is
used. Invalid values are any core object (objects with no apiGroup) except for PVCs.
The
field may contain different types of objects, while the
field
only allows PVCs and VolumeSnapshots.
When the
feature is enabled, there are additional differences:
The
field only allows local objects, while the
field allows objects in any namespaces.
When namespace is specified,
and
are not synced.
Users should always use
on clusters that have the feature gate enabled, and
fall back to
on clusters that do not. It is not necessary to look at both fields
under any circumstance. The duplicated values with slightly different semantics exist only for
backwards compatibility. In particular, a mixture of older and newer controllers are able to
interoperate because the fields are the same.
Using volume populators
Volume populators are
that ca

n
create non-empty volumes, where the contents of the volume are determined by a Custom Resource.
Users create a populated volume by referring to a Custom Resource using the
field:
:
v1
kind
:
metadata
:
name
:
populated-pvc
spec
:
:
name
:
example-name
kind
:
ExampleDataSource
apiGroup
:
example.storage.k8s.io
:
resources
:
requests
:
storage
:
10Gi
Because volume populators are external components, attempts to create a PVC that uses one
can fail if not all the correct components are installed. External controllers should generate
events on the PVC to provide feedback on the status of the creation, including warnings if
the PVC cannot be created due to some missing component.
You can install the alpha
volume data source validator
controller into your cluster. That controller generates warning Events on a PVC in the case that no populator
is registered to handle that kind of data source. When a suitable populator is installed for a PVC, it's the
responsibility of that populator controller to report Events that relate to volume creation and issues during
the process.
Using a cross-namespace volume data source
Create a ReferenceGrant to allow the namespace owner to accept the reference.
You define a populated volume by specifying a cross namespace volume data source using the
field. You must already have a valid ReferenceGrant in the source namespace:
:
gateway.networking.k8s.io/v1beta1
kind
:
metadata
:
name
:
allow-ns1-pvc
namespace
:
default
spec
:
from
:
-
group
:
""
kind
:
namespace
:
ns1
to
:
-
group
:
kind
:
name
:
new-snapshot-demo
:
v1
kind
:
metadata
:
name
:
foo-pvc
namespace
:
ns1
spec
:
:
example
:
resources
:
requests
:
storage
:
1Gi
:
apiGroup
:
kind
:
name
:
namespace
:
default
:
Writing Portable Configuration
If you're writing configuration templates or examples that run on a wide range of clusters
and need persistent storage, it is recommended that you use the following pattern:
Include PersistentVolumeClaim objects in your bundle of config (alongside
Deployments, ConfigMaps, etc).
Do not include PersistentVolume objects in the config, since the user instantiating
the config may not have permission to create PersistentVolumes.
Give the user the option of providing a storage class name when instantiating
the template.
If the user provides a storage class name, put that value into the
persistentVolumeClaim.storageClassName
field.
This will cause the PVC to match the right storage
class if the cluster has StorageClasses enabled by the admin.
If the user does not provide a storage class name, leave the
field as nil. This will cause a
PV to be automatically provisioned for the user with the default StorageClass
in the cluster. Many cluster environments have a default StorageClass installed,
or administrators can create their own default StorageClass.
In your tooling, watch for PVCs that are not getting bound after some time
and surface this to the user, as this may indicate that the cluster has no
dynamic storage support (in which case the user should create a matching PV)
or the cluster has no storage system (in which case the user cannot deploy
config requiring PVCs).
Creating a PersistentVolume
.
Creating a PersistentVolumeClaim
.
Read the
.
API references
Read about the APIs described in this page:
7.3 - Projected Volumes
volumes
A
projected
volume maps several ex

isting volume sources into the same directory.
Currently, the following types of volume sources can be projected:
secret
configMap
All sources are required to be in the same namespace as the Pod. For more details,
see the
Example configuration with a secret, a downwardAPI, and a configMap
pods/storage/projected-secret-downwardapi-configmap.yaml
:
v1
kind
:
Pod
metadata
:
name
:
volume-test
spec
:
:
-
name
:
container-test
image
:
:
-
name
:
mountPath
:
readOnly
:
true
volumes
:
-
name
:
projected
:
sources
:
-
secret
:
name
:
mysecret
items
:
-
key
:
username
path
:
my-group/my-username
-
:
items
:
-
path
:
"labels"
fieldRef
:
fieldPath
:
-
path
:
:
:
resource
:
-
configMap
:
name
:
myconfigmap
items
:
-
key
:
config
path
:
my-group/my-config
Example configuration: secrets with a non-default permission mode set
pods/storage/projected-secrets-nondefault-permission-mode.yaml
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
:
-
name
:
mountPath
:
readOnly
:
true
volumes
:
-
name
:
projected
:
sources
:
-
secret
:
name
:
mysecret
items
:
-
key
:
username
path
:
-
secret
:
name
:
mysecret2
items
:
-
key
:
password
path
:
my-group/my-password
mode
:
511
Each projected volume source is listed in the spec under
sources
. The
parameters are nearly the same with two exceptions:
For secrets, the
field has been changed to
name
to be consistent
with ConfigMap naming.
The
can only be specified at the projected level and not for each
volume source. However, as illustrated above, you can explicitly set the
mode
for each individual projection.
serviceAccountToken projected volumes
You can inject the token for the current
into a Pod at a specified path. For example:
pods/storage/projected-service-account-token.yaml
:
v1
kind
:
Pod
metadata
:
name
:
sa-token-test
spec
:
:
-
name
:
image
:
:
-
name
:
token-vol
mountPath
:
"/service-account"
readOnly
:
true
:
default
volumes
:
-
name
:
token-vol
projected
:
sources
:
-
:
audience
:
api
:
3600
path
:
token
The example Pod has a projected volume containing the injected service account
token. Containers in this Pod can use that token to access the Kubernetes API
server, authenticating with the identity of
the pod's ServiceAccount
.
The
audience
field contains the intended audience of the
token. A recipient of the token must identify itself with an identifier specified
in the audience of the token, and otherwise should reject the token. This field
is optional and it defaults to the identifier of the API server.
The
is the expected duration of validity of the service account
token. It defaults to 1 hour and must be at least 10 minutes (600 seconds). An administrator
can also limit its maximum value by specifying the
--service-account-max-token-expiration
option for the API server. The
path
field specifies a relative path to the mount point
of the projected volume.
Note:
A container using a projected volume source as a
subPath
volume mount will not receive updates for those volume sources.
SecurityContext interactions
The
proposal
for file permission handling in projected service account volume enhancement introduced the projected files having the correct owner permissions set.
Linux
In Linux pods that have a projected volume and
RunAsUser
set in the Pod
,
the projected files have the correct ownership set including container user
ownership.
When all containers in a pod have the same
runAsUser
set in their
PodSecurityC

ontext
or container
,
then the kubelet ensures that the contents of the
volume are owned by that user,
and the token file has its permission mode set to
0600
.
Note:
added to a Pod after it is created do
not
change volume permissions that were
set when the pod was created.
If a Pod's
volume permissions were set to
0600
because
all other containers in the Pod have the same
runAsUser
, ephemeral
containers must use the same
runAsUser
to be able to read the token.
Windows
In Windows pods that have a projected volume and
RunAsUsername
set in the
Pod
, the ownership is not enforced due to the way user
accounts are managed in Windows. Windows stores and manages local user and group
accounts in a database file called Security Account Manager (SAM). Each
container maintains its own instance of the SAM database, to which the host has
no visibility into while the container is running. Windows containers are
designed to run the user mode portion of the OS in isolation from the host,
hence the maintenance of a virtual SAM database. As a result, the kubelet running
on the host does not have the ability to dynamically configure host file
ownership for virtualized container accounts. It is recommended that if files on
the host machine are to be shared with the container then they should be placed
into their own volume mount outside of
C:\
.
By default, the projected files will have the following ownership as shown for
an example projected volume file:
PS
C:\&gt;
Get-Acl
C:\var\run\secrets\kubernetes.io\serviceaccount\..2021_08_31_22_22_18.318230061\ca.crt |
Format-List
Path
:
Microsoft.PowerShell.Core\FileSystem::C:\var\run\secrets\kubernetes.io\serviceaccount\..2021_08_31_22_22_18.318230061\ca.crt
Owner
:
BUILTIN\Administrators
Group
:
Access
:
NT AUTHORITY\SYSTEM Allow  FullControl
BUILTIN\Administrators Allow  FullControl
BUILTIN\Users Allow  ReadAndExecute, Synchronize
Audit
:
Sddl
:
O:BAG
:
SYD
:
AI(A;ID;FA;;;SY)(A;ID;FA;;;BA)(A;ID;0x1200a9;;;BU)
This implies all administrator users like
will have
read, write and execute access while, non-administrator users will have read and
execute access.
Note:
In general, granting the container access to the host is discouraged as it can
open the door for potential security exploits.
Creating a Windows Pod with
RunAsUser
in it's
will result in
the Pod being stuck at
forever. So it is advised to not use
the Linux only
RunAsUser
option with Windows Pods.
7.4 - Ephemeral Volumes
ephemeral volumes
in Kubernetes. Familiarity
with
volumes
is suggested, in
particular PersistentVolumeClaim and PersistentVolume.
Some application need additional storage but don't care whether that
data is stored persistently across restarts. For example, caching
services are often limited by memory size and can move infrequently
used data into storage that is slower than memory with little impact
on overall performance.
Other applications expect some read-only input data to be present in
files, like configuration data or secret keys.
Ephemeral volumes
are designed for these use cases. Because volumes
follow the Pod's lifetime and get created and deleted along with the
Pod, Pods can be stopped and restarted without being limited to where
some persistent volume is available.
Ephemeral volumes are specified
inline
in the Pod spec, which
simplifies application deployment and management.
Types of ephemeral volumes
Kubernetes supports several different kinds of ephemeral volumes for
different purposes:
emptyDir
: empty at Pod startup,
with storage coming locally from the kubelet base directory (usually
the root disk) or RAM
configMap
,
,
secret
: inject different
kinds of Kubernetes data into a Pod
:
similar to the previous volume kinds, but provided by special
CSI drivers
which specifically
support this fe

ature
generic ephemeral volumes
, which
can be provided by all storage drivers that also support persistent volumes
emptyDir
,
configMap
,
,
secret
are provided as
local ephemeral
storage
.
They are managed by kubelet on each node.
must
be provided by third-party CSI storage
drivers.
Generic ephemeral volumes
can
drivers, but also by any other storage driver that supports dynamic
provisioning. Some CSI drivers are written specifically for CSI
ephemeral volumes and do not support dynamic provisioning: those then
cannot be used for generic ephemeral volumes.
The advantage of using third-party drivers is that they can offer
functionality that Kubernetes itself does not support, for example
storage with different performance characteristics than the disk that
is managed by kubelet, or injecting different data.
Note:
CSI ephemeral volumes are only supported by a subset of CSI drivers.
The Kubernetes CSI
Drivers list
shows which drivers support ephemeral volumes.
Conceptually, CSI ephemeral volumes are similar to
configMap
,
and
secret
volume types: the storage is managed locally on each
node and is created together with other local resources after a Pod has been
scheduled onto a node. Kubernetes has no concept of rescheduling Pods
anymore at this stage. Volume creation has to be unlikely to fail,
otherwise Pod startup gets stuck. In particular,
storage capacity
aware Pod scheduling
is
not
supported for these volumes. They are currently also not covered by
the storage resource usage limits of a Pod, because that is something
that kubelet can only enforce for storage that it manages itself.
Here's an example manifest for a Pod that uses CSI ephemeral storage:
kind
:
Pod
:
v1
metadata
:
name
:
my-csi-app
spec
:
:
-
name
:
my-frontend
image
:
:
-
mountPath
:
"/data"
name
:
my-csi-inline-vol
command
:
[
"sleep"
,
"1000000"
]
volumes
:
-
name
:
csi
:
driver
:
inline.storage.kubernetes.io
:
foo
:
bar
The
determine what volume is prepared by the
driver. These attributes are specific to each driver and not
standardized. See the documentation of each CSI driver for further
instructions.
CSI driver restrictions
CSI ephemeral volumes allow users to provide
directly to the CSI driver as part of the Pod spec. A CSI driver
allowing
that are typically restricted to
administrators is NOT suitable for use in an inline ephemeral volume.
For example, parameters that are normally defined in the StorageClass
should not be exposed to users through the use of inline ephemeral volumes.
Cluster administrators who need to restrict the CSI drivers that are
allowed to be used as inline volumes within a Pod spec may do so by:
Removing
Ephemeral
from
in the CSIDriver spec, which prevents the
driver from being used as an inline ephemeral volume.
Using an
to restrict how this driver is used.
Generic ephemeral volumes are similar to
emptyDir
volumes in the
sense that they provide a per-pod directory for scratch data that is
usually empty after provisioning. But they may also have additional
features:
Storage can be local or network-attached.
Volumes can have a fixed size that Pods are not able to exceed.
Volumes may have some initial data, depending on the driver and
parameters.
Typical operations on volumes are supported assuming that the driver
supports them, including
snapshotting
,
cloning
,
resizing
,
and
storage capacity tracking
.
Example:
kind
:
Pod
:
v1
metadata
:
name
:
my-app
spec
:
:
-
name
:
image
:
:
-
mountPath
:
"/scratch"
name
:
scratch-volume
command
:
[
"sleep"
,
"1000000"
]
volumes
:
-
name
:
scratch-volum

e
ephemeral
:
:
metadata
:
labels
:
type
:
my-frontend-volume
spec
:
:
[
]
:
"scratch-storage-class"
resources
:
requests
:
storage
:
1Gi
Lifecycle and PersistentVolumeClaim
The key design idea is that the
parameters for a volume claim
are allowed inside a volume source of the Pod. Labels, annotations and
the whole set of fields for a PersistentVolumeClaim are supported. When such a Pod gets
created, the ephemeral volume controller then creates an actual PersistentVolumeClaim
object in the same namespace as the Pod and ensures that the PersistentVolumeClaim
gets deleted when the Pod gets deleted.
That triggers volume binding and/or provisioning, either immediately if
the
uses immediate volume binding or when the Pod is
tentatively scheduled onto a node (
volume
binding mode). The latter is recommended for generic ephemeral volumes
because then the scheduler is free to choose a suitable node for
the Pod. With immediate binding, the scheduler is forced to select a node that has
access to the volume once it is available.
In terms of
resource ownership
,
a Pod that has generic ephemeral storage is the owner of the PersistentVolumeClaim(s)
that provide that ephemeral storage. When the Pod is deleted,
the Kubernetes garbage collector deletes the PVC, which then usually
triggers deletion of the volume because the default reclaim policy of
storage classes is to delete volumes. You can create quasi-ephemeral local storage
using a StorageClass with a reclaim policy of
retain
: the storage outlives the Pod,
and in this case you need to ensure that volume clean up happens separately.
While these PVCs exist, they can be used like any other PVC. In
particular, they can be referenced as data source in volume cloning or
snapshotting. The PVC object also holds the current status of the
volume.
PersistentVolumeClaim naming
Naming of the automatically created PVCs is deterministic: the name is
a combination of Pod name and volume name, with a hyphen (
-
) in the
middle. In the example above, the PVC name will be
my-app-scratch-volume
. This deterministic naming makes it easier to
interact with the PVC because one does not have to search for it once
the Pod name and volume name are known.
The deterministic naming also introduces a potential conflict between different
Pods (a Pod "pod-a" with volume "scratch" and another Pod with name
"pod" and volume "a-scratch" both end up with the same PVC name
"pod-a-scratch") and between Pods and manually created PVCs.
Such conflicts are detected: a PVC is only used for an ephemeral
volume if it was created for the Pod. This check is based on the
ownership relationship. An existing PVC is not overwritten or
modified. But this does not resolve the conflict because without the
right PVC, the Pod cannot start.
Caution:
Take care when naming Pods and volumes inside the
same namespace, so that these conflicts can't occur.
Security
Enabling the GenericEphemeralVolume feature allows users to create
PVCs indirectly if they can create Pods, even if they do not have
permission to create PVCs directly. Cluster administrators must be
aware of this. If this does not fit their security model, they should
use an
that rejects objects like Pods that have a generic ephemeral volume.
The normal
namespace quota for PVCs
still applies, so even if users are allowed to use this new mechanism, they cannot use
it to circumvent other policies.
Ephemeral volumes managed by kubelet
See
.
For more information on the design, see the
Ephemeral Inline CSI volumes KEP
.
For more information on further development of this feature, see the
enhancement tracking issue #596
.
Generic ephemeral inline volumes KEP
.
7.5 - Storage Classes
This document describes the concept of a StorageClass in Kubernetes. Fa

miliarity
with
volumes
and
A StorageClass provides a way for administrators to describe the "classes" of
storage they offer. Different classes might map to quality-of-service levels,
or to backup policies, or to arbitrary policies determined by the cluster
administrators. Kubernetes itself is unopinionated about what classes
represent. This concept is sometimes called "profiles" in other storage
systems.
The StorageClass Resource
Each StorageClass contains the fields
,
, and
, which are used when a PersistentVolume belonging to the
class needs to be dynamically provisioned.
The name of a StorageClass object is significant, and is how users can
request a particular class. Administrators set the name and other parameters
of a class when first creating StorageClass objects, and the objects cannot
be updated once they are created.
Administrators can specify a default StorageClass only for PVCs that don't
request any particular class to bind to: see the
PersistentVolumeClaim section
:
kind
:
metadata
:
name
:
standard
:
kubernetes.io/aws-ebs
:
type
:
gp2
:
Retain
:
true
:
- debug
:
Immediate
Provisioner
Each StorageClass has a provisioner that determines what volume plugin is used
for provisioning PVs. This field must be specified.
Internal Provisioner
Config Example
✓
AWS EBS
AzureFile
✓
Azure File
AzureDisk
✓
Azure Disk
CephFS
-
-
Cinder
✓
FC
-
-
-
-
✓
GCE PD
iSCSI
-
-
NFS
-
NFS
RBD
✓
Ceph RBD
✓
vSphere
✓
Portworx Volume
Local
-
Local
You are not restricted to specifying the "internal" provisioners
listed here (whose names are prefixed with "kubernetes.io" and shipped
alongside Kubernetes). You can also run and specify external provisioners,
which are independent programs that follow a
defined by Kubernetes. Authors of external provisioners have full discretion
over where their code lives, how the provisioner is shipped, how it needs to be
run, what volume plugin it uses (including Flex), etc. The repository
kubernetes-sigs/sig-storage-lib-external-provisioner
houses a library for writing external provisioners that implements the bulk of
the specification. Some external provisioners are listed under the repository
.
For example, NFS doesn't provide an internal provisioner, but an external
provisioner can be used. There are also cases when 3rd party storage
vendors provide their own external provisioner.
PersistentVolumes that are dynamically created by a StorageClass will have the
reclaim policy specified in the
field of the class, which can be
either
Delete
or
Retain
. If no
is specified when a
StorageClass object is created, it will default to
Delete
.
PersistentVolumes that are created manually and managed via a StorageClass will have
whatever reclaim policy they were assigned at creation.
Allow Volume Expansion
PersistentVolumes can be configured to be expandable. This feature when set to
true
,
allows the users to resize the volume by editing the corresponding PVC object.
The following types of volumes support volume expansion, when the underlying
StorageClass has the field
set to true.
Table of Volume types and the version of Kubernetes they require
Volume type
Required Kubernetes version
1.11
1.11
Cinder
1.11
rbd
1.11
Azure File
1.11
Azure Disk
1.11
Portworx
1.11
1.13
CSI
1.14 (alpha), 1.16 (beta)
Note:
You can only use the volume expansion feature to grow a Volume, not to shrink it.
m

ount options specified in the
field of the class.
If the volume plugin does not support mount options but mount options are
specified, provisioning will fail. Mount options are not validated on either
the class or PV. If a mount option is invalid, the PV mount fails.
Volume Binding Mode
The
field controls when
volume binding and dynamic
provisioning
should occur. When unset, "Immediate" mode is used by default.
The
Immediate
mode indicates that volume binding and dynamic
provisioning occurs once the PersistentVolumeClaim is created. For storage
backends that are topology-constrained and not globally accessible from all Nodes
in the cluster, PersistentVolumes will be bound or provisioned without knowledge of the Pod's scheduling
requirements. This may result in unschedulable Pods.
A cluster administrator can address this issue by specifying the
mode which
will delay the binding and provisioning of a PersistentVolume until a Pod using the PersistentVolumeClaim is created.
PersistentVolumes will be selected or provisioned conforming to the topology that is
specified by the Pod's scheduling constraints. These include, but are not limited to,
resource
,
,
pod affinity and
,
and
.
The following plugins support
with dynamic provisioning:
AzureDisk
with pre-created PersistentVolume binding:
All of the above
Local
CSI volumes
are also supported with dynamic provisioning
and pre-created PVs, but you'll need to look at the documentation for a specific CSI driver
to see its supported topology keys and examples.
Note:
If you choose to use
, do not use
nodeName
in the Pod spec
to specify node affinity. If
nodeName
is used in this case, the scheduler will be bypassed and PVC will remain in
pending
state.
Instead, you can use node selector for hostname in this case as shown below.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
:
kube-01
volumes
:
-
name
:
:
claimName
:
:
-
name
:
image
:
nginx
ports
:
-
:
80
name
:
:
-
mountPath
:
name
:
Allowed Topologies
When a cluster operator specifies the
volume binding mode, it is no longer necessary
to restrict provisioning to specific topologies in most situations. However,
if still required,
can be specified.
This example demonstrates how to restrict the topology of provisioned volumes to specific
zones and should be used as a replacement for the
zone
and
zones
parameters for the
supported plugins.
:
kind
:
metadata
:
name
:
standard
:
kubernetes.io/gce-pd
:
type
:
pd-standard
:
:
-
:
-
key
:
values
:
- us-central-1a
- us-central-1b
Storage Classes have parameters that describe volumes belonging to the storage
class. Different parameters may be accepted depending on the
. For
example, the value
io1
, for the parameter
type
, and the parameter
iopsPerGB
are specific to EBS. When a parameter is omitted, some default is
used.
There can be at most 512 parameters defined for a StorageClass.
The total length of the parameters object including its keys and values cannot
exceed 256 KiB.
AWS EBS
:
kind
:
metadata
:
name
:
slow
:
:
type
:
io1
iopsPerGB
:
"10"
fsType
:
ext4
type
:
io1
,
gp2
,
sc1
,
st1
. See
AWS docs
for details. Default:
gp2
.
zone
(Deprecated): AWS zone. If neither
zone

nor
zones
is specified, volumes are
generally round-robin-ed across all active zones where Kubernetes cluster
has a node.
zone
and
zones
parameters must not be used at the same time.
zones
(Deprecated): A comma separated list of AWS zone(s). If neither
zone
nor
zones
is specified, volumes are generally round-robin-ed across all active zones
where Kubernetes cluster has a node.
zone
and
zones
parameters must not
be used at the same time.
iopsPerGB
: only for
io1
volumes. I/O operations per second per GiB. AWS
volume plugin multiplies this with size of requested volume to compute IOPS
of the volume and caps it at 20 000 IOPS (maximum supported by AWS, see
AWS docs
).
A string is expected here, i.e.
"10"
, not
10
.
fsType
: fsType that is supported by kubernetes. Default:
"ext4"
.
encrypted
: denotes whether the EBS volume should be encrypted or not.
"true"
or
"false"
. A string is expected here,
i.e.
"true"
, not
true
.
kmsKeyId
: optional. The full Amazon Resource Name of the key to use when
encrypting the volume. If none is supplied but
encrypted
is true, a key is
generated by AWS. See AWS docs for valid ARN value.
Note:
zone
and
zones
parameters are deprecated and replaced with
GCE PD
:
kind
:
metadata
:
name
:
slow
:
:
type
:
fstype
:
ext4
replication-type
:
none
type
:
or
pd-ssd
. Default:
zone
(Deprecated): GCE zone. If neither
zone
nor
zones
generally round-robin-ed across all active zones where Kubernetes cluster has
a node.
zone
and
zones
zones
(Deprecated): A comma separated list of GCE zone(s). If neither
zone
nor
zones
zone
and
zones
fstype
:
ext4
or
xfs
. Default:
ext4
. The defined filesystem type must be supported by the host operating system.
:
none
or
regional-pd
. Default:
none
.
If
is set to
none
, a regular (zonal) PD will be provisioned.
If
is set to
, a
Regional Persistent Disk
will be provisioned. It's highly recommended to have
volumeBindingMode: WaitForFirstConsumer
set, in which case when you create
a Pod that consumes a PersistentVolumeClaim which uses this StorageClass, a
Regional Persistent Disk is provisioned with two zones. One zone is the same
as the zone that the Pod is scheduled in. The other zone is randomly picked
from the zones available to the cluster. Disk zones can be further constrained
using
.
Note:
zone
and
zones
NFS
:
kind
:
metadata
:
name
:
example-nfs
:
example.com/external-nfs
:
server
:
nfs-server.example.com
path
:
/share
readOnly
:
"false"
server
: Server is the hostname or IP address of the NFS server.
path
: Path that is exported by the NFS server.
readOnly
: A flag indicating whether the storage will be mounted as read only (default false).
Kubernetes doesn't include an internal NFS provisioner. You need to use an external provisioner to create a StorageClass for NFS.
Here are some examples:
NFS Ganesha server and external provisioner
NFS subdir external provisioner
:
kind
:
metadata
:
name
:
gold
:
kubernetes.io/cinder
:
availability
:
nova
: Availability Zone. If not specified, volumes are generally
round-robin-ed across all active zones where Kubernetes cluster has a node.
Note:
Kubernetes v1.11 [deprecated]
This internal provisioner of OpenStack is deprecated. Please use
the external cloud provider for OpenStack
.
vSphere
There are two types of provisioners for vSphere storage cl

asses:
CSI provisioner
:
vCP provisioner
:
kubernetes.io/vsphere-volume
In-tree provisioners are
. For more information on the CSI provisioner, see
Kubernetes vSphere CSI Driver
and
vSphereVolume CSI migration
.
CSI Provisioner
The vSphere CSI StorageClass provisioner works with Tanzu Kubernetes clusters. For an example, refer to the
vSphere CSI repository
.
vCP Provisioner
The following examples use the VMware Cloud Provider (vCP) StorageClass provisioner.
Create a StorageClass with a user specified disk format.
:
kind
:
metadata
:
name
:
fast
:
:
:
zeroedthick
:
thin
,
and
eagerzeroedthick
. Default:
"thin"
.
Create a StorageClass with a disk format on a user specified datastore.
:
kind
:
metadata
:
name
:
fast
:
:
:
datastore
:
VSANDatastore
datastore
: The user can also specify the datastore in the StorageClass.
The volume will be created on the datastore specified in the StorageClass,
which in this case is
. This field is optional. If the
datastore is not specified, then the volume will be created on the datastore
specified in the vSphere config file used to initialize the vSphere Cloud
Provider.
Storage Policy Management inside kubernetes
Using existing vCenter SPBM policy
One of the most important features of vSphere for Storage Management is
policy based Management. Storage Policy Based Management (SPBM) is a
storage policy framework that provides a single unified control plane
across a broad range of data services and storage solutions. SPBM enables
vSphere administrators to overcome upfront storage provisioning challenges,
such as capacity planning, differentiated service levels and managing
capacity headroom.
The SPBM policies can be specified in the StorageClass using the
Virtual SAN policy support inside Kubernetes
Vsphere Infrastructure (VI) Admins will have the ability to specify custom
Virtual SAN Storage Capabilities during dynamic volume provisioning. You
can now define storage requirements, such as performance and availability,
in the form of storage capabilities during dynamic volume provisioning.
The storage capability requirements are converted into a Virtual SAN
policy which are then pushed down to the Virtual SAN layer when a
persistent volume (virtual disk) is being created. The virtual disk is
distributed across the Virtual SAN datastore to meet the requirements.
You can see
Storage Policy Based Management for dynamic provisioning of volumes
for more details on how to use storage policies for persistent volumes
management.
There are few
vSphere examples
which you try out for persistent volume management inside Kubernetes for vSphere.
Ceph RBD
:
kind
:
metadata
:
name
:
fast
:
kubernetes.io/rbd
:
monitors
:
10.16.153.105
:
6789
adminId
:
kube
:
ceph-secret
adminSecretNamespace
:
pool
:
kube
userId
:
kube
userSecretName
:
ceph-secret-user
userSecretNamespace
:
default
fsType
:
ext4
imageFormat
:
"2"
imageFeatures
:
"layering"
monitors
: Ceph monitors, comma delimited. This parameter is required.
adminId
: Ceph client ID that is capable of creating images in the pool.
Default is "admin".
: Secret Name for
adminId
. This parameter is required.
The provided secret must have type "kubernetes.io/rbd".
: The namespace for
. Default is "default".
pool
: Ceph RBD pool. Default is "rbd".
userId
: Ceph client ID that is used to map the RBD image. Default is the
same as
adminId
.
: The name of Ceph Secret for
userId
to map RBD image. It
must exist in the same namespace as PVCs. This parameter is required.
The provided secret must have type 

"kubernetes.io/rbd", for example created in this
way:
kubectl create secret generic ceph-secret --type
=
"kubernetes.io/rbd"
\
=
key
=
'QVFEQ1pMdFhPUnQrSmhBQUFYaERWNHJsZ3BsMmNjcDR6RFZST0E9PQ=='
\
=
.
fsType
"ext4"
.
: Ceph RBD image format, "1" or "2". Default is "2".
: This parameter is optional and should only be used if you
set
to "2". Currently supported features are
layering
only.
Default is "", and no features are turned on.
Azure Disk
Azure Unmanaged Disk storage class
:
kind
:
metadata
:
name
:
slow
:
kubernetes.io/azure-disk
:
skuName
:
Standard_LRS
location
:
eastus
storageAccount
:
azure_storage_account_name
skuName
: Azure storage account Sku tier. Default is empty.
location
: Azure storage account location. Default is empty.
: Azure storage account name. If a storage account is provided,
it must reside in the same resource group as the cluster, and
location
is
ignored. If a storage account is not provided, a new storage account will be
created in the same resource group as the cluster.
Azure Disk storage class (starting from v1.7.2)
:
kind
:
metadata
:
name
:
slow
:
:
storageaccounttype
:
kind
:
managed
kind
: Possible values are
shared
,
dedicated
, and
managed
(default).
When
kind
is
shared
, all unmanaged disks are created in a few shared
storage accounts in the same resource group as the cluster. When
kind
is
dedicated
, a new dedicated storage account will be created for the new
unmanaged disk in the same resource group as the cluster. When
kind
is
managed
, all managed disks are created in the same resource group as
resourceGroup
: Specify the resource group in which the Azure disk will be created.
It must be an existing resource group name. If it is unspecified, the disk will be
placed in the same resource group as the current Kubernetes cluster.
Premium VM can attach both Standard_LRS and Premium_LRS disks, while Standard
VM can only attach Standard_LRS disks.
Managed VM can only attach managed disks and unmanaged VM can only attach
unmanaged disks.
Azure File
:
kind
:
metadata
:
name
:
azurefile
:
kubernetes.io/azure-file
:
skuName
:
location
:
eastus
:
skuName
location
: Azure storage account name. Default is empty. If a storage
account is not provided, all storage accounts associated with the resource
group are searched to find one that matches
skuName
and
location
. If a
storage account is provided, it must reside in the same resource group as the
cluster, and
skuName
and
location
are ignored.
: the namespace of the secret that contains the Azure Storage
Account Name and Key. Default is the same as the Pod.
: the name of the secret that contains the Azure Storage Account Name and
Key. Default is
azure-storage-account-&lt;accountName&gt;-secret
readOnly
: a flag indicating whether the storage will be mounted as read only.
Defaults to false which means a read/write mount. This setting will impact the
ReadOnly
setting in VolumeMounts as well.
During storage provisioning, a secret named by
is created for the
mounting credentials. If the cluster has enabled both
RBAC
and
Controller Roles
,
add the
create
permission of resource
secret
for clusterrole
.
In a multi-tenancy context, it is strongly recommended to set the value for

, otherwise the storage account credentials may
be read by other users.
:
kind
:
metadata
:
name
:
portworx-io-priority-high
:
kubernetes.io/portworx-volume
:
repl
:
"1"
snap_interval
:
"70"
priority_io
:
"high"
fs
: filesystem to be laid out:
none/xfs/ext4
(default:
ext4
).
block_size
: block size in Kbytes (default:
32
).
repl
: number of synchronous replicas to be provided in the form of
replication factor
1..3
(default:
1
) A string is expected here i.e.
"1"
and not
1
.
: determines whether the volume will be created from higher
performance or a lower priority storage
high/medium/low
(default:
low
).
: clock/time interval in minutes for when to trigger snapshots.
Snapshots are incremental based on difference with the prior snapshot, 0
disables snaps (default:
0
). A string is expected here i.e.
"70"
and not
70
.
aggregation_level
: specifies the number of chunks the volume would be
distributed into, 0 indicates a non-aggregated volume (default:
0
). A string
is expected here i.e.
"0"
and not
0
ephemeral
: specifies whether the volume should be cleaned-up after unmount
or should be persistent.
emptyDir
use case can set this value to true and
use case such as for databases like Cassandra should set
to false,
true/false
(default
false
"true"
and not
true
.
Local
:
kind
:
metadata
:
name
:
:
kubernetes.io/no-provisioner
:
Local volumes do not currently support dynamic provisioning, however a StorageClass
should still be created to delay volume binding until Pod scheduling. This is
volume binding mode.
Delaying volume binding allows the scheduler to consider all of a Pod's
scheduling constraints when choosing an appropriate PersistentVolume for a
7.6 - Dynamic Volume Provisioning
Dynamic volume provisioning allows storage volumes to be created on-demand.
Without dynamic provisioning, cluster administrators have to manually make
calls to their cloud or storage provider to create new storage volumes, and
then create
objects
to represent them in Kubernetes. The dynamic provisioning feature eliminates
the need for cluster administrators to pre-provision storage. Instead, it
automatically provisions storage when it is requested by users.
The implementation of dynamic volume provisioning is based on the API object
from the API group
. A cluster administrator can define as many
objects as needed, each specifying a
(aka
) that provisions a volume and the set of parameters to pass to
that provisioner when provisioning.
A cluster administrator can define and expose multiple flavors of storage (from
the same or different storage systems) within a cluster, each with a custom set
of parameters. This design also ensures that end users don't have to worry
about the complexity and nuances of how storage is provisioned, but still
have the ability to select from multiple storage options.
More information on storage classes can be found
here
.
Enabling Dynamic Provisioning
To enable dynamic provisioning, a cluster administrator needs to pre-create
one or more StorageClass objects for users.
StorageClass objects define which provisioner should be used and what parameters
should be passed to that provisioner when dynamic provisioning is invoked.
The name of a StorageClass object must be a valid
.
The following manifest creates a storage class "slow" which provisions standard
disk-like persistent disks.
:
kind
:
metadata
:
name
:
slow
:
:
type
:
The following manifest creates a s

torage class "fast" which provisions
SSD-like persistent disks.
:
kind
:
metadata
:
name
:
fast
:
:
type
:
pd-ssd
Using Dynamic Provisioning
Users request dynamically provisioned storage by including a storage class in
their
. Before Kubernetes v1.6, this was done via the
annotation. However, this annotation
is deprecated since v1.9. Users now can and should instead use the
object. The value of
this field must match the name of a
configured by the
administrator (see
below
).
To select the "fast" storage class, for example, a user would create the
following PersistentVolumeClaim:
:
v1
kind
:
metadata
:
name
:
claim1
spec
:
:
:
fast
resources
:
requests
:
storage
:
30Gi
This claim results in an SSD-like Persistent Disk being automatically
provisioned. When the claim is deleted, the volume is destroyed.
Defaulting Behavior
Dynamic provisioning can be enabled on a cluster such that all claims are
dynamically provisioned if no storage class is specified. A cluster administrator
can enable this behavior by:
Marking one
object as
default
;
Making sure that the
is enabled on the API server.
An administrator can mark a specific
as default by adding the
to it.
When a default
exists in a cluster and a user creates a
with
unspecified, the
admission controller automatically adds the
field pointing to the default storage class.
Note that there can be at most one
default
storage class on a cluster, or
a
without
explicitly specified cannot
be created.
Topology Awareness
In
Multi-Zone
clusters, Pods can be spread across
Zones in a Region. Single-Zone storage backends should be provisioned in the Zones where
Pods are scheduled. This can be accomplished by setting the
.
7.7 - Volume Snapshots
represents a snapshot of a volume on a storage
system. This document assumes that you are already familiar with Kubernetes
.
Similar to how API resources
and
are
used to provision volumes for users and administrators,
VolumeSnapshotContent
and
API resources are provided to create volume snapshots for
users and administrators.
A
is a snapshot taken from a volume in the cluster that
has been provisioned by an administrator. It is a resource in the cluster just
like a PersistentVolume is a cluster resource.
A
is a request for snapshot of a volume by a user. It is similar
to a PersistentVolumeClaim.
VolumeSnapshotClass
allows you to specify different attributes belonging to a
. These attributes may differ among snapshots taken from the same
volume on the storage system and therefore cannot be expressed by using the same
of a
.
Volume snapshots provide Kubernetes users with a standardized way to copy a volume's
contents at a particular point in time without creating an entirely new volume. This
functionality enables, for example, database administrators to backup databases before
performing edit or delete modifications.
Users need to be aware of the following when using this feature:
API Objects
,
, and
are
CRDs
, not
part of the core API.
support is only available for CSI drivers.
As part of the deployment process of
, the Kubernetes team provides
a snapshot controller to be deployed into the control plane, and a sidecar hel

per
container called csi-snapshotter to be deployed together with the CSI driver.
The snapshot controller watches
and
objects
and is responsible for the creation and deletion of
object.
The sidecar csi-snapshotter watches
objects and triggers
CreateSnapshot
and
DeleteSnapshot
operations against a CSI endpoint.
There is also a validating webhook server which provides tightened validation on
snapshot objects. This should be installed by the Kubernetes distros along with
the snapshot controller and CRDs, not CSI drivers. It should be installed in all
Kubernetes clusters that has the snapshot feature enabled.
CSI drivers may or may not have implemented the volume snapshot functionality.
The CSI drivers that have provided support for volume snapshot will likely use
the csi-snapshotter. See
CSI Driver documentation
The CRDs and snapshot controller installations are the responsibility of the Kubernetes distribution.
Lifecycle of a volume snapshot and volume snapshot content
VolumeSnapshotContents
are resources in the cluster.
VolumeSnapshots
are requests
for those resources. The interaction between
and
follow this lifecycle:
Provisioning Volume Snapshot
There are two ways snapshots may be provisioned: pre-provisioned or dynamically provisioned.
Pre-provisioned
A cluster administrator creates a number of
. They carry the details
of the real volume snapshot on the storage system which is available for use by cluster users.
They exist in the Kubernetes API and are available for consumption.
Dynamic
Instead of using a pre-existing snapshot, you can request that a snapshot to be dynamically
taken from a PersistentVolumeClaim. The
specifies storage provider-specific parameters to use when taking a snapshot.
Binding
The snapshot controller handles the binding of a
object with an appropriate
object, in both pre-provisioned and dynamically provisioned scenarios.
The binding is a one-to-one mapping.
In the case of pre-provisioned binding, the VolumeSnapshot will remain unbound until the
requested VolumeSnapshotContent object is created.
Persistent Volume Claim as Snapshot Source Protection
The purpose of this protection is to ensure that in-use
API objects are not removed from the system while a snapshot is being taken from it
(as this may result in data loss).
While a snapshot is being taken of a PersistentVolumeClaim, that PersistentVolumeClaim
is in-use. If you delete a PersistentVolumeClaim API object in active use as a snapshot
source, the PersistentVolumeClaim object is not removed immediately. Instead, removal of
the PersistentVolumeClaim object is postponed until the snapshot is readyToUse or aborted.
Delete
Deletion is triggered by deleting the
object, and the
DeletionPolicy
will be followed. If the
is
Delete
, then the underlying storage snapshot
will be deleted along with the
object. If the
is
Retain
, then both the underlying snapshot and
remain.
Each VolumeSnapshot contains a spec and a status.
:
snapshot.storage.k8s.io/v1
kind
:
metadata
:
name
:
spec
:
volumeSnapshotClassName
:
csi-hostpath-snapclass
source
:
persistentVolumeClaimName
:
pvc-test
is the name of the PersistentVolumeClaim data source
for the snapshot. This field is required for dynamically provisioning a snapshot.
A volume snapshot can request a particular class by specifying the name of a
. If nothing is set, then the
default class is used if available.
For pre-provisioned snapshots, you need to specify a
volumeSnapshotContentName
as the source for the snapshot as shown in the following example. The

source field is required for pre-provisioned snapshots.
:
kind
:
metadata
:
name
:
test-snapshot
spec
:
source
:
:
test-content
Volume Snapshot Contents
Each VolumeSnapshotContent contains a spec and status. In dynamic provisioning,
the snapshot common controller creates
objects. Here is an example:
:
kind
:
metadata
:
name
:
snapcontent-72d9a349-aacd-42d2-a240-d775650d2455
spec
:
deletionPolicy
:
Delete
driver
:
hostpath.csi.k8s.io
source
:
:
ee0cfb94-f8d4-11e9-b2d8-0242ac110002
sourceVolumeMode
:
:
volumeSnapshotRef
:
name
:
namespace
:
default
uid
:
72d9a349-aacd-42d2-a240-d775650d2455
is the unique identifier of the volume created on the storage
backend and returned by the CSI driver during the volume creation. This field
is required for dynamically provisioning a snapshot.
It specifies the volume source of the snapshot.
For pre-provisioned snapshots, you (as cluster administrator) are responsible
for creating the
object as follows.
:
kind
:
metadata
:
name
:
new-snapshot-content-test
spec
:
:
Delete
driver
:
source
:
snapshotHandle
:
7bdd0de3-aaeb-11e8-9aae-0242ac110002
:
:
name
:
namespace
:
default
is the unique identifier of the volume snapshot created on
the storage backend. This field is required for the pre-provisioned snapshots.
It specifies the CSI snapshot id on the storage system that this
represents.
is the mode of the volume whose snapshot is taken. The value
of the
field can be either
or
Block
. If the
source volume mode is not specified, Kubernetes treats the snapshot as if the
source volume's mode is unknown.
is the reference of the corresponding
. Note that
when the
is being created as a pre-provisioned snapshot, the
referenced in
might not exist yet.
If the
API installed on your cluster supports the
field, then the API has the capability to prevent unauthorized users from converting
the mode of a volume.
To check if your cluster has capability for this feature, run the following command:
$ kubectl get crd volumesnapshotcontent -o yaml
If you want to allow users to create a
from an existing
, but with a different volume mode than the source, the annotation
needs to be added to
the
that corresponds to the
.
For pre-provisioned snapshots,
spec.sourceVolumeMode
needs to be populated
by the cluster administrator.
An example
resource with this feature enabled would look like:
:
kind
:
metadata
:
name
:
:
-
:
"true"
spec
:
:
Delete
driver
:
source
:
:
:
:
name
:
namespace
:
default
Provisioning Volumes from Snapshots
You can provision a new volume, pre-populated with data from a snapshot, by using
the
object.
Volume Snapshot and Restore Volume from Snapshot
.
7.8 - Volume Snapshot Classes
This document describes the concept of VolumeSnapshotClass in Kubernetes. Familiarity
with
volume snapshots

and
storage classes
Just like StorageClass provides a way for administrators to describe the "classes"
of storage they offer when provisioning a volume, VolumeSnapshotClass provides a
way to describe the "classes" of storage when provisioning a volume snapshot.
The VolumeSnapshotClass Resource
Each VolumeSnapshotClass contains the fields
driver
,
, and
,
which are used when a VolumeSnapshot belonging to the class needs to be
dynamically provisioned.
The name of a VolumeSnapshotClass object is significant, and is how users can
of a class when first creating VolumeSnapshotClass objects, and the objects cannot
Note:
Installation of the CRDs is the responsibility of the Kubernetes distribution. Without the required CRDs present, the creation of a VolumeSnapshotClass fails.
:
kind
:
metadata
:
name
:
driver
:
:
Delete
:
Administrators can specify a default VolumeSnapshotClass for VolumeSnapshots
that don't request any particular class to bind to by adding the
snapshot.storage.kubernetes.io/is-default-class: "true"
:
kind
:
metadata
:
name
:
:
snapshot.storage.kubernetes.io/is-default-class
:
"true"
driver
:
:
Delete
:
Driver
Volume snapshot classes have a driver that determines what CSI volume plugin is
used for provisioning VolumeSnapshots. This field must be specified.
Volume snapshot classes have a deletionPolicy. It enables you to configure what happens to a VolumeSnapshotContent when the VolumeSnapshot object it is bound to is to be deleted. The deletionPolicy of a volume snapshot class can either be
Retain
or
Delete
. This field must be specified.
If the deletionPolicy is
Delete
, then the underlying storage snapshot will be deleted along with the VolumeSnapshotContent object. If the deletionPolicy is
Retain
, then both the underlying snapshot and VolumeSnapshotContent remain.
Volume snapshot classes have parameters that describe volume snapshots belonging to
the volume snapshot class. Different parameters may be accepted depending on the
driver
.
7.9 - CSI Volume Cloning
This document describes the concept of cloning existing CSI Volumes in Kubernetes. Familiarity with
Volumes
The
CSI
Volume Cloning feature adds support for specifying existing
PVC
s in the
field to indicate a user would like to clone a
Volume
.
A Clone is defined as a duplicate of an existing Kubernetes Volume that can be consumed as any standard Volume would be. The only difference is that upon provisioning, rather than creating a "new" empty Volume, the back end device creates an exact duplicate of the specified Volume.
The implementation of cloning, from the perspective of the Kubernetes API, adds the ability to specify an existing PVC as a dataSource during new PVC creation. The source PVC must be bound and available (not in use).
Cloning support (
) is only available for CSI drivers.
Cloning support is only available for dynamic provisioners.
CSI drivers may or may not have implemented the volume cloning functionality.
You can only clone a PVC when it exists in the same namespace as the destination PVC (source and destination must be in the same namespace).
Cloning is supported with a different Storage Class.
Destination volume can be the same or a different storage class as the source.
Default storage class can be used and storageClassName omitted in the spec.
Cloning can only be performed between two volumes that use the same VolumeMode setting (if you request a block mo

de volume, the source MUST also be block mode)
Clones are provisioned like any other PVC with the exception of adding a dataSource that references an existing PVC in the same namespace.
:
v1
kind
:
metadata
:
name
:
clone-of-pvc-1
namespace
:
myns
spec
:
:
:
cloning
resources
:
requests
:
storage
:
5Gi
:
kind
:
name
:
pvc-1
Note:
You must specify a capacity value for
spec.resources.requests.storage
, and the value you specify must be the same or larger than the capacity of the source volume.
The result is a new PVC with the name
that has the exact same content as the specified source
pvc-1
.
Usage
Upon availability of the new PVC, the cloned PVC is consumed the same as other PVC. It's also expected at this point that the newly created PVC is an independent object. It can be consumed, cloned, snapshotted, or deleted independently and without consideration for it's original dataSource PVC. This also implies that the source is not linked in any way to the newly created clone, it may also be modified or deleted without affecting the newly created clone.
7.10 - Storage Capacity
Storage capacity is limited and may vary depending on the node on
which a pod runs: network-attached storage might not be accessible by
all nodes, or storage is local to a node to begin with.
This page describes how Kubernetes keeps track of storage capacity and
how the scheduler uses that information to
schedule Pods
onto nodes
that have access to enough storage capacity for the remaining missing
volumes. Without storage capacity tracking, the scheduler may choose a
node that doesn't have enough capacity to provision a volume and
multiple scheduling retries will be needed.
Kubernetes v1.26 includes cluster-level API support for
storage capacity tracking. To use this you must also be using a CSI driver that
supports capacity tracking. Consult the documentation for the CSI drivers that
you use to find out whether this support is available and, if so, how to use
it. If you are not running Kubernetes v1.26, check the
documentation for that version of Kubernetes.
API
There are two API extensions for this feature:
objects:
these get produced by a CSI driver in the namespace
where the driver is installed. Each object contains capacity
information for one storage class and defines which nodes have
access to that storage.
The
CSIDriverSpec.StorageCapacity
field
:
when set to
true
, the Kubernetes scheduler will consider storage
capacity for volumes that use the CSI driver.
Storage capacity information is used by the Kubernetes scheduler if:
a Pod uses a volume that has not been created yet,
that volume uses a
which references a CSI driver and
uses
volume binding
mode
,
and
the
CSIDriver
object for the driver has
StorageCapacity
set to
true.
In that case, the scheduler only considers nodes for the Pod which
have enough storage available to them. This check is very
simplistic and only compares the size of the volume against the
capacity listed in
objects with a topology that
includes the node.
For volumes with
Immediate
volume binding mode, the storage driver
decides where to create the volume, independently of Pods that will
use the volume. The scheduler then schedules Pods onto nodes where the
volume is available after the volume has been created.
For
,
scheduling always happens without considering storage capacity. This
is based on the assumption that this volume type is only used by
special CSI drivers which are local to a node and do not need
significant resources there.
When a node has been selected for a Pod with
volumes, that decision is still tentative. The next step is that the
CSI storage driver gets asked to create the volume 

with a hint that the
volume is supposed to be available on the selected node.
Because Kubernetes might have chosen a node based on out-dated
capacity information, it is possible that the volume cannot really be
created. The node selection is then reset and the Kubernetes scheduler
tries again to find a node for the Pod.
Storage capacity tracking increases the chance that scheduling works
on the first try, but cannot guarantee this because the scheduler has
to decide based on potentially out-dated information. Usually, the
same retry mechanism as for scheduling without any storage capacity
information handles scheduling failures.
One situation where scheduling can fail permanently is when a Pod uses
multiple volumes: one volume might have been created already in a
topology segment which then does not have enough capacity left for
another volume. Manual intervention is necessary to recover from this,
for example by increasing capacity or deleting the volume that was
already created.
Storage Capacity Constraints for Pod Scheduling KEP
.
7.11 - Node-specific Volume Limits
This page describes the maximum number of volumes that can be attached
to a Node for various cloud providers.
Cloud providers like Google, Amazon, and Microsoft typically have a limit on
how many volumes can be attached to a Node. It is important for Kubernetes to
respect those limits. Otherwise, Pods scheduled on a Node could get stuck
waiting for volumes to attach.
Kubernetes default limits
The Kubernetes scheduler has default limits on the number of volumes
that can be attached to a Node:
Cloud service
Maximum volumes per Node
Amazon Elastic Block Store (EBS)
39
Google Persistent Disk
16
Microsoft Azure Disk Storage
16
Custom limits
You can change these limits by setting the value of the
KUBE_MAX_PD_VOLS
environment variable, and then starting the scheduler.
CSI drivers might have a different procedure, see their documentation
on how to customize their limits.
Use caution if you set a limit that is higher than the default limit. Consult
the cloud provider's documentation to make sure that Nodes can actually support
the limit you set.
The limit applies to the entire cluster, so it affects all Nodes.
Dynamic volume limits
Dynamic volume limits are supported for following volume types.
Amazon EBS
Azure Disk
CSI
For volumes managed by in-tree volume plugins, Kubernetes automatically determines the Node
type and enforces the appropriate maximum number of volumes for the node. For example:
On
Google Compute Engine
,
up to 127 volumes can be attached to a node,
depending on the node
type
.
For Amazon EBS disks on M5,C5,R5,T3 and Z1D instance types, Kubernetes allows only 25
volumes to be attached to a Node. For other instance types on
Amazon Elastic Compute Cloud (EC2)
,
Kubernetes allows 39 volumes to be attached to a Node.
On Azure, up to 64 disks can be attached to a node, depending on the node type. For more details, refer to
Sizes for virtual machines in Azure
.
If a CSI storage driver advertises a maximum number of volumes for a Node (using
NodeGetInfo
), the
honors that limit.
CSI specifications
For volumes managed by in-tree plugins that have been migrated to a CSI driver, the maximum number of volumes will be the one reported by the CSI driver.
7.12 - Volume Health Monitoring
CSI
volume health monitoring allows CSI Drivers to detect abnormal volume conditions from the underlying storage systems and report them as events on
PVCs
or
Pods
.
Volume health monitoring
volume health monitoring
is part of how Kubernetes implements the Container Storage Interface (CSI). Volume health monitoring feature is implemented in two components: an External Health Monitor controller, and the
kubelet
.
If a CSI Driver supports Volume Health Monitoring featur

e from the controller side, an event will be reported on the related
(PVC) when an abnormal volume condition is detected on a CSI volume.
The External Health Monitor
also watches for node failure events. You can enable node failure monitoring by setting the
enable-node-watcher
flag to true. When the external health monitor detects a node failure event, the controller reports an Event will be reported on the PVC to indicate that pods using this PVC are on a failed node.
If a CSI Driver supports Volume Health Monitoring feature from the node side, an Event will be reported on every Pod using the PVC when an abnormal volume condition is detected on a CSI volume. In addition, Volume Health information is exposed as Kubelet VolumeStats metrics. A new metric kubelet_volume_stats_health_status_abnormal is added. This metric includes two labels:
namespace
and
. The count is either 1 or 0. 1 indicates the volume is unhealthy, 0 indicates volume is healthy. For more information, please check
KEP
.
Note:
You need to enable the
to use this feature from the node side.
See the
CSI driver documentation
to find out which CSI drivers have implemented this feature.
7.13 - Windows Storage
This page provides an storage overview specific to the Windows operating system.
Persistent storage
Windows has a layered filesystem driver to mount container layers and create a copy
filesystem based on NTFS. All file paths in the container are resolved only within
the context of that container.
With Docker, volume mounts can only target a directory in the container, and not
an individual file. This limitation does not apply to containerd.
Volume mounts cannot project files or directories back to the host filesystem.
Read-only filesystems are not supported because write access is always required
for the Windows registry and SAM database. However, read-only volumes are supported.
Volume user-masks and permissions are not available. Because the SAM is not shared
between the host &amp; container, there's no mapping between them. All permissions are
resolved within the context of the container.
As a result, the following storage functionality is not supported on Windows nodes:
Volume subpath mounts: only the entire volume can be mounted in a Windows container
Subpath volume mounting for Secrets
Host mount projection
Read-only root filesystem (mapped volumes still support
readOnly
)
Block device mapping
Memory as the storage medium (for example,
set to
Memory
)
File system features like uid/gid; per-user Linux filesystem permissions
Setting
secret permissions with DefaultMode
(due to UID/GID dependency)
NFS based storage/volume support
Expanding the mounted volume (resizefs)
volumes
enable complex
applications, with data persistence and Pod volume sharing requirements, to be deployed
on Kubernetes. Management of persistent volumes associated with a specific storage
back-end or protocol includes actions such as provisioning/de-provisioning/resizing
of volumes, attaching/detaching a volume to/from a Kubernetes node and
mounting/dismounting a volume to/from individual containers in a pod that needs to
persist data.
Volume management components are shipped as Kubernetes volume
plugin
.
The following broad classes of Kubernetes volume plugins are supported on Windows:
FlexVolume plugins
Please note that FlexVolumes have been deprecated as of 1.23
CSI Plugins
In-tree volume plugins
azureDisk
azureFile
8 - Configuration
8.1 - Configuration Best Practices
This document highlights and consolidates configuration best practices that are introduced
throughout the user guide, Getting Started documentation, and examples.
This is a living document. If you think of something tha

t is not on this list but might be useful
to others, please don't hesitate to file an issue or submit a PR.
General Configuration Tips
When defining configurations, specify the latest stable API version.
Configuration files should be stored in version control before being pushed to the cluster. This
allows you to quickly roll back a configuration change if necessary. It also aids cluster
re-creation and restoration.
Write your configuration files using YAML rather than JSON. Though these formats can be used
interchangeably in almost all scenarios, YAML tends to be more user-friendly.
Group related objects into a single file whenever it makes sense. One file is often easier to
manage than several. See the
guestbook-all-in-one.yaml
file as an example of this syntax.
Note also that many
kubectl
commands can be called on a directory. For example, you can call
on a directory of config files.
Don't specify default values unnecessarily: simple, minimal configuration will make errors less likely.
Put object descriptions in annotations, to allow better introspection.
"Naked" Pods versus ReplicaSets, Deployments, and Jobs
Don't use naked Pods (that is, Pods not bound to a
or
) if you can avoid it. Naked Pods
will not be rescheduled in the event of a node failure.
A Deployment, which both creates a ReplicaSet to ensure that the desired number of Pods is
always available, and specifies a strategy to replace Pods (such as
), is
almost always preferable to creating Pods directly, except for some explicit
scenarios.
A
Job
may also be appropriate.
Services
Create a
Service
before its corresponding backend
workloads (Deployments or ReplicaSets), and before any workloads that need to access it.
When Kubernetes starts a container, it provides environment variables pointing to all the Services
which were running when the container was started. For example, if a Service named
foo
exists,
all containers will get the following variables in their initial environment:
=
&lt;the host the Service is running on&gt;
=
&lt;the port the Service is running on&gt;
This does imply an ordering requirement
- any
Service
that a
Pod
wants to access must be
created before the
Pod
itself, or else the environment variables will not be populated.
DNS does not have this restriction.
An optional (though strongly recommended)
is a DNS server. The DNS server watches the Kubernetes API for new
Services
and creates a set
of DNS records for each. If DNS has been enabled throughout the cluster then all
Pods
should be
able to do name resolution of
Services
automatically.
Don't specify a
hostPort
for a Pod unless it is absolutely necessary. When you bind a Pod to a
hostPort
, it limits the number of places the Pod can be scheduled, because each &lt;
hostIP
,
hostPort
,
protocol
&gt; combination must be unique. If you don't specify the
hostIP
and
protocol
explicitly, Kubernetes will use
0.0.0.0
hostIP
and
TCP
as the
default
protocol
.
If you only need access to the port for debugging purposes, you can use the
or
.
If you explicitly need to expose a Pod's port on the node, consider using a
NodePort
Service before resorting to
hostPort
.
Avoid using
, for the same reasons as
hostPort
.
Use
(which have a
ClusterIP
of
None
) for service discovery when you don't need
load balancing.
Using Labels
Define and use
labels
that identify
semantic attributes
of your application or Deployment, such as
{ app.kubernetes.io/name: MyApp, tier: frontend, phase: test, deployment: v3 }
. You can use these labels to select the
appropriate Pods for other resources; for example, a Service that selects all
Pods, or all
phase: test
components of
app.kubernetes.io/name: MyApp
.
See the
guestbook
app
for examples of this approach.
A Service can be made to span multiple Deployments b

y omitting release-specific labels from its
selector. When you need to update a running service without downtime, use a
.
A desired state of an object is described by a Deployment, and if changes to that spec are
applied
, the deployment controller changes the actual state to the desired state at a controlled
rate.
Use the
Kubernetes common labels
for common use cases. These standardized labels enrich the metadata in a way that allows tools,
including
kubectl
and
dashboard
, to
work in an interoperable way.
You can manipulate labels for debugging. Because Kubernetes controllers (such as ReplicaSet) and
Services match to Pods using selector labels, removing the relevant labels from a Pod will stop
it from being considered by a controller or from being served traffic by a Service. If you remove
the labels of an existing Pod, its controller will create a new Pod to take its place. This is a
useful way to debug a previously "live" Pod in a "quarantine" environment. To interactively remove
or add labels, use
.
Use
. This looks for Kubernetes configuration in all
.yaml
,
.yml
, and
.json
files in
&lt;directory&gt;
and passes it to
apply
.
Use label selectors for
get
and
delete
operations instead of specific object names. See the
sections on
label selectors
and
using labels effectively
.
Use
and
to quickly create single-container
Deployments and Services.
See
8.2 - ConfigMaps
A ConfigMap is an API object used to store non-confidential data in key-value pairs.
Pods
volume
.
Caution:
ConfigMap does not provide secrecy or encryption.
If the data you want to store are confidential, use a
Secret
rather than a ConfigMap,
or use additional (third party) tools to keep your data private.
Use a ConfigMap for setting configuration data separately from application code.
For example, imagine that you are developing an application that you can run on your
own computer (for development) and in the cloud (to handle real traffic).
You write the code to look in an environment variable named
DATABASE_HOST
.
Locally, you set that variable to
localhost
. In the cloud, you set it to
refer to a Kubernetes
Service
that exposes the database component to your cluster.
This lets you fetch a container image running in the cloud and
debug the exact same code locally if needed.
A ConfigMap is not designed to hold large chunks of data. The data stored in a
ConfigMap cannot exceed 1 MiB. If you need to store settings that are
larger than this limit, you may want to consider mounting a volume or use a
separate database or file service.
ConfigMap object
A ConfigMap is an API
object
that lets you store configuration for other objects to use. Unlike most
Kubernetes objects that have a
spec
, a ConfigMap has
data
and
fields. These fields accept key-value pairs as their values. Both the
data
are optional. The
data
field is designed to
contain UTF-8 strings while the
contain binary data as base64-encoded strings.
The name of a ConfigMap must be a valid
.
Each key under the
data
or the
field must consist of
alphanumeric characters,
-
,
_
or
.
. The keys stored in
data
must not
overlap with the keys in the
field.
Starting from v1.19, you can add an
immutable
field to a ConfigMap
definition to create an
immutable ConfigMap
.
ConfigMaps and Pods
You can write a Pod
spec
that refers to a ConfigMap and configures the container(s)
in that Pod based on the data in the ConfigMap. The Pod and the ConfigMap must be in
the same
namespace
.
N

ote:
The
spec
of a
static Pod
cannot refer to a ConfigMap
or any other API objects.
Here's an example ConfigMap that has some keys with single values,
and other keys where the value looks like a fragment of a configuration
format.
:
v1
kind
:
ConfigMap
metadata
:
name
:
game-demo
data
:
# property-like keys; each key maps to a simple value
player_initial_lives
:
"3"
ui_properties_file_name
:
"user-interface.properties"
# file-like keys
:
|
enemy.types=aliens,monsters
player.maximum-lives=5
user-interface.properties
:
|
There are four different ways that you can use a ConfigMap to configure
a container inside a Pod:
Inside a container command and args
Environment variables for a container
Add a file in read-only volume, for the application to read
Write code to run inside the Pod that uses the Kubernetes API to read a ConfigMap
These different methods lend themselves to different ways of modeling
the data being consumed.
For the first three methods, the
kubelet
uses the data from
the ConfigMap when it launches container(s) for a Pod.
The fourth method means you have to write code to read the ConfigMap and its data.
However, because you're using the Kubernetes API directly, your application can
subscribe to get updates whenever the ConfigMap changes, and react
when that happens. By accessing the Kubernetes API directly, this
technique also lets you access a ConfigMap in a different namespace.
Here's an example Pod that uses values from
game-demo
to configure a Pod:
configmap/configure-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
configmap-demo-pod
spec
:
:
-
name
:
demo
image
:
alpine
command
:
[
"sleep"
,
"3600"
]
env
:
-
name
:
PLAYER_INITIAL_LIVES
# Notice that the case is different here
# from the key name in the ConfigMap.
valueFrom
:
:
name
:
game-demo
# The ConfigMap this value comes from.
key
:
# The key to fetch.
-
name
:
UI_PROPERTIES_FILE_NAME
valueFrom
:
:
name
:
game-demo
key
:
:
-
name
:
config
mountPath
:
"/config"
readOnly
:
true
volumes
:
# You set volumes at the Pod level, then mount them into containers inside that Pod
-
name
:
config
configMap
:
# Provide the name of the ConfigMap you want to mount.
name
:
game-demo
# An array of keys from the ConfigMap to create as files
items
:
-
key
:
path
:
-
key
:
path
:
A ConfigMap doesn't differentiate between single line property values and
multi-line file-like values.
What matters is how Pods and other objects consume those values.
For this example, defining a volume and mounting it inside the
demo
container as
/config
creates two files,
/config/game.properties
and
/config/user-interface.properties
,
even though there are four keys in the ConfigMap. This is because the Pod
definition specifies an
items
array in the
volumes
section.
If you omit the
items
array entirely, every key in the ConfigMap becomes
a file with the same name as the key, and you get 4 files.
Using ConfigMaps
ConfigMaps can be mounted as data volumes. ConfigMaps can also be used by other
parts of the system, without being directly exposed to the Pod. For example,
ConfigMaps can hold data that other parts of the system should use for configuration.
The most common way to use ConfigMaps is to configure settings for
containers running in a Pod in the same namespace. You can also use a
ConfigMap separately.
For example, you
might encounter
addons
or
operators
that
adjust their behavior based on a ConfigMap.
Using ConfigMaps as files from a Pod
To consume a ConfigMap in a volume in a Pod:
Create a ConfigMap or use an existing one. Multiple Pods can reference the
same ConfigMap.
Modify your Pod definition to add a volume under
.spec.volumes[]
. Name
the volume anything, and have a
.spec.volum

es[].configMap.name
field set
to reference your ConfigMap object.
Add a
.spec.containers[].volumeMounts[]
to each container that needs the
ConfigMap. Specify
.spec.containers[].volumeMounts[].readOnly = true
and
.spec.containers[].volumeMounts[].mountPath
to an unused directory name
where you would like the ConfigMap to appear.
Modify your image or command line so that the program looks for files in
that directory. Each key in the ConfigMap
data
map becomes the filename
under
mountPath
.
This is an example of a Pod that mounts a ConfigMap in a volume:
:
v1
kind
:
Pod
metadata
:
name
:
mypod
spec
:
:
-
name
:
mypod
image
:
redis
:
-
name
:
foo
mountPath
:
"/etc/foo"
readOnly
:
true
volumes
:
-
name
:
foo
configMap
:
name
:
Each ConfigMap you want to use needs to be referred to in
.
If there are multiple containers in the Pod, then each container needs its
own
block, but only one
is needed per ConfigMap.
When a ConfigMap currently consumed in a volume is updated, projected keys are eventually updated as well.
The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync.
However, the kubelet uses its local cache for getting the current value of the ConfigMap.
The type of the cache is configurable using the
ConfigMapAndSecretChangeDetectionStrategy
field in
the
.
A ConfigMap can be either propagated by watch (default), ttl-based, or by redirecting
all requests directly to the API server.
As a result, the total delay from the moment when the ConfigMap is updated to the moment
when new keys are projected to the Pod can be as long as the kubelet sync period + cache
propagation delay, where the cache propagation delay depends on the chosen cache type
(it equals to watch propagation delay, ttl of cache, or zero correspondingly).
ConfigMaps consumed as environment variables are not updated automatically and require a pod restart.
Note:
subPath
volume mount will not receive ConfigMap updates.
Immutable ConfigMaps
The Kubernetes feature
Immutable Secrets and ConfigMaps
provides an option to set
individual Secrets and ConfigMaps as immutable. For clusters that extensively use ConfigMaps
(at least tens of thousands of unique ConfigMap to Pod mounts), preventing changes to their
data has the following advantages:
protects you from accidental (or unwanted) updates that could cause applications outages
improves performance of your cluster by significantly reducing load on kube-apiserver, by
closing watches for ConfigMaps marked as immutable.
This feature is controlled by the
.
You can create an immutable ConfigMap by setting the
immutable
field to
true
.
:
v1
kind
:
ConfigMap
metadata
:
...
data
:
...
immutable
:
true
Once a ConfigMap is marked as immutable, it is
not
possible to revert this change
nor to mutate the contents of the
data
or the
field. You can
only delete and recreate the ConfigMap. Because existing Pods maintain a mount point
to the deleted ConfigMap, it is recommended to recreate these pods.
Read about
Secrets
.
Read
.
Read about
changing a ConfigMap (or any other Kubernetes object)
Read
The Twelve-Factor App
to understand the motivation for
separating code from configuration.
8.3 - Secrets
A Secret is an object that contains a small amount of sensitive data such as
a password, a token, or a key. Such information might otherwise be put in a
Pod
specification or in a
. Using a
Secret means that you don't need to include confidential data in your
application code.
Because Secrets can be created independently of the Pods that use them, there
is less risk of the Secret (and its data) being exposed during the workflow of
creating, viewing, and e

diting Pods. Kubernetes, and applications that run in
your cluster, can also take additional precautions with Secrets, such as avoiding
writing secret data to nonvolatile storage.
Secrets are similar to
but are specifically intended to hold confidential data.
Caution:
Kubernetes Secrets are, by default, stored unencrypted in the API server's underlying data store
(etcd). Anyone with API access can retrieve or modify a Secret, and so can anyone with access to etcd.
Additionally, anyone who is authorized to create a Pod in a namespace can use that access to read
any Secret in that namespace; this includes indirect access such as the ability to create a
In order to safely use Secrets, take at least the following steps:
Enable Encryption at Rest
for Secrets.
Enable or configure RBAC rules
with
least-privilege access to Secrets.
Restrict Secret access to specific containers.
Consider using external Secret store providers
.
For more guidelines to manage and improve the security of your Secrets, refer to
.
See
Information security for Secrets
Uses for Secrets
There are three main ways for a Pod to use a Secret:
As
files
in a
volume
mounted on one or more of
its containers.
As
.
By the
kubelet when pulling images
The Kubernetes control plane also uses Secrets; for example,
bootstrap token Secrets
are a mechanism to
help automate node registration.
Alternatives to Secrets
Rather than using a Secret to protect confidential data, you can pick from alternatives.
Here are some of your options:
if your cloud-native component needs to authenticate to another application that you
know is running within the same Kubernetes cluster, you can use a
and its tokens to identify your client.
there are third-party tools that you can run, either within or outside your cluster,
that provide secrets management. For example, a service that Pods access over HTTPS,
that reveals a secret if the client correctly authenticates (for example, with a ServiceAccount
token).
for authentication, you can implement a custom signer for X.509 certificates, and use
CertificateSigningRequests
to let that custom signer issue certificates to Pods that need them.
you can use a
device plugin
to expose node-local encryption hardware to a specific Pod. For example, you can schedule
trusted Pods onto nodes that provide a Trusted Platform Module, configured out-of-band.
You can also combine two or more of those options, including the option to use Secret objects themselves.
For example: implement (or deploy) an
operator
that fetches short-lived session tokens from an external service, and then creates Secrets based
on those short-lived session tokens. Pods running in your cluster can make use of the session tokens,
and operator ensures they are valid. This separation means that you can run Pods that are unaware of
the exact mechanisms for issuing and refreshing those session tokens.
Working with Secrets
Creating a Secret
There are several options to create a Secret:
Use
kubectl
Use a configuration file
Use the Kustomize tool
Constraints on Secret names and data
The name of a Secret object must be a valid
.
data
and/or the
field when creating a
configuration file for a Secret. The
data
and the
The values for all keys in the
data
field have to be base64-encoded strings.
If the conversion to base64 string is not desirable, you can choose to specify
the
field instead, which accepts arbitrary strings as values.
The keys of
data
and
-
,
_
or
.
. All key-value pairs in the
field are internally
merged into the
data
field. If a key appears in both the
data
and the
field, the value specified in the
field takes
precedence.
Size limit
Individual secrets are limited to 1MiB in size.

 This is to discourage creation
of very large secrets that could exhaust the API server and kubelet memory.
However, creation of many smaller secrets could also exhaust memory. You can
use a
to limit the
number of Secrets (or other resources) in a namespace.
Editing a Secret
You can edit an existing Secret unless it is
immutable
. To
edit a Secret, use one of the following methods:
Use
kubectl
You can also edit the data in a Secret using the
Kustomize tool
. However, this
method creates a new
Secret
object with the edited data.
Depending on how you created the Secret, as well as how the Secret is used in
your Pods, updates to existing
Secret
objects are propagated automatically to
Pods that use the data. For more information, refer to
Mounted Secrets are updated automatically
.
Using a Secret
Secrets can be mounted as data volumes or exposed as
to be used by a container in a Pod. Secrets can also be used by other parts of the
system, without being directly exposed to the Pod. For example, Secrets can hold
credentials that other parts of the system should use to interact with external
systems on your behalf.
Secret volume sources are validated to ensure that the specified object
reference actually points to an object of type Secret. Therefore, a Secret
needs to be created before any Pods that depend on it.
If the Secret cannot be fetched (perhaps because it does not exist, or
due to a temporary lack of connection to the API server) the kubelet
periodically retries running that Pod. The kubelet also reports an Event
for that Pod, including details of the problem fetching the Secret.
Optional Secrets
When you define a container environment variable based on a Secret,
you can mark it as
optional
. The default is for the Secret to be
required.
None of a Pod's containers will start until all non-optional Secrets are
If a Pod references a specific key in a Secret and that Secret does exist, but
is missing the named key, the Pod fails during startup.
Using Secrets as files from a Pod
If you want to access data from a Secret in a Pod, one way to do that is to
have Kubernetes make the value of that Secret be available as a file inside
the filesystem of one or more of the Pod's containers.
To configure that, you:
Create a secret or use an existing one. Multiple Pods can reference the same secret.
. Name the volume anything,
and have a
.spec.volumes[].secret.secretName
field equal to the name of the Secret object.
Add a
to each container that needs the secret. Specify
and
to an unused directory name where you would like the
secrets to appear.
Modify your image or command line so that the program looks for files in that directory. Each
key in the secret
data
map becomes the filename under
mountPath
.
This is an example of a Pod that mounts a Secret named
mysecret
in a volume:
:
v1
kind
:
Pod
metadata
:
name
:
mypod
spec
:
:
-
name
:
mypod
image
:
redis
:
-
name
:
foo
mountPath
:
readOnly
:
true
volumes
:
-
name
:
foo
secret
:
:
mysecret
optional
:
false
# default setting; "mysecret" must exist
Each Secret you want to use needs to be referred to in
.
own
is needed per Secret.
Note:
are obtained directly by using the
API,
.
The tokens obtained using this met

hod have bounded lifetimes, and are automatically
a service account token Secret; for example, if you need a token that never expires.
However, using the
command to obtain a token from the
API.
Projection of Secret keys to specific paths
You can also control the paths within the volume where Secret keys are projected.
.spec.volumes[].secret.items
field to change the target path of each key:
:
v1
kind
:
Pod
metadata
:
name
:
mypod
spec
:
:
-
name
:
mypod
image
:
redis
:
-
name
:
foo
mountPath
:
readOnly
:
true
volumes
:
-
name
:
foo
secret
:
:
mysecret
items
:
-
key
:
username
path
:
What will happen:
the
username
key from
mysecret
is available to the container at the path
/etc/foo/my-group/my-username
instead of at
/etc/foo/username
.
the
password
key from that Secret object is not projected.
If
is used, only keys specified in
items
are projected.
To consume all keys from the Secret, all of them must be listed in the
items
field.
If you list keys explicitly, then all listed keys must exist in the corresponding Secret.
Otherwise, the volume is not created.
Secret files permissions
You can set the POSIX file access permission bits for a single Secret key.
If you don't specify any permissions,
0644
is used by default.
You can also set a default mode for the entire Secret volume and override per key if needed.
For example, you can specify a default mode like this:
:
v1
kind
:
Pod
metadata
:
name
:
mypod
spec
:
:
-
name
:
mypod
image
:
redis
:
-
name
:
foo
mountPath
:
volumes
:
-
name
:
foo
secret
:
:
mysecret
:
0400
The secret is mounted on
/etc/foo
; all the files created by the
secret volume mount have permission
0400
.
Note:
If you're defining a Pod or a Pod template using JSON, beware that the JSON
specification doesn't support octal notation. You can use the decimal value
for the
(for example, 0400 in octal is 256 in decimal) instead.
If you're writing YAML, you can write the
in octal.
Consuming Secret values from volumes
Inside the container that mounts a secret volume, the secret keys appear as
files. The secret values are base64 decoded and stored inside these files.
This is the result of commands executed inside the container from the example above:
ls /etc/foo/
username
password
cat /etc/foo/username
admin
cat /etc/foo/password
The program in a container is responsible for reading the secret data from these
files, as needed.
When a volume contains data from a Secret, and that Secret is updated, Kubernetes tracks
this and updates the data in the volume, using an eventually-consistent approach.
Note:
subPath
volume mount does not receive
automated Secret updates.
The kubelet keeps a cache of the current keys and values for the Secrets that are used in
volumes for pods on that node.
You can configure the way that the kubelet detects changes from the cached values. The
controls
which strategy the kubelet uses. The default strategy is
Watch
.
Updates to Secrets can be either propagated by an API watch mechanism (the default), based on
a cache with a defined time-to-live, or polled from the cluster API server on each kubelet
synchronisation loop.
As a result, the total delay from the moment when the Secret is updated to the moment
propagation delay, where the cache propa

gation delay depends on the chosen cache type
(following the same order listed in the previous paragraph, these are:
watch propagation delay, the configured cache TTL, or zero for direct polling).
Using Secrets as environment variables
To use a Secret in an
in a Pod:
Create a Secret (or use an existing one). Multiple Pods can reference the same Secret.
Modify your Pod definition in each container that you wish to consume the value of a secret
key to add an environment variable for each secret key you wish to consume. The environment
variable that consumes the secret key should populate the secret's name and key in
env[].valueFrom.secretKeyRef
.
Modify your image and/or command line so that the program looks for values in the specified
This is an example of a Pod that uses a Secret via environment variables:
:
v1
kind
:
Pod
metadata
:
name
:
secret-env-pod
spec
:
:
-
name
:
mycontainer
image
:
redis
env
:
-
name
:
valueFrom
:
:
name
:
mysecret
key
:
username
optional
:
false
# same as default; "mysecret" must exist
# and include a key named "username"
-
name
:
SECRET_PASSWORD
valueFrom
:
:
name
:
mysecret
key
:
password
optional
:
false
# and include a key named "password"
:
Never
Invalid environment variables
Secrets used to populate environment variables by the
envFrom
field that have keys
that are considered invalid environment variable names will have those keys
skipped. The Pod is allowed to start.
If you define a Pod with an invalid variable name, the failed Pod startup includes
an event with the reason set to
and a message that lists the
skipped invalid keys. The following example shows a Pod that refers to a Secret
named
mysecret
, where
mysecret
contains 2 invalid keys:
1badkey
and
2alsobad
.
LASTSEEN   FIRSTSEEN   COUNT     NAME            KIND      SUBOBJECT                         TYPE      REASON
0s         0s          1         dapi-test-pod   Pod                                         Warning   InvalidEnvironmentVariableNames   kubelet, 127.0.0.1      Keys [1badkey, 2alsobad] from the EnvFrom secret default/mysecret were skipped since they are considered invalid environment variable names.
Consuming Secret values from environment variables
Inside a container that consumes a Secret using environment variables, the secret keys appear
as normal environment variables. The values of those variables are the base64 decoded values
of the secret data.
echo
"
$SECRET_USERNAME
"
admin
echo
"
$SECRET_PASSWORD
"
Note:
If a container already consumes a Secret in an environment variable,
a Secret update will not be seen by the container unless it is
restarted. There are third party solutions for triggering restarts when
secrets change.
Container image pull secrets
If you want to fetch container images from a private repository, you need a way for
the kubelet on each node to authenticate to that repository. You can configure
image pull secrets
to make this possible. These secrets are configured at the Pod
level.
Using imagePullSecrets
The
field is a list of references to secrets in the same namespace.
You can use an
to pass a secret that contains a Docker (or other) image registry
password to the kubelet. The kubelet uses this information to pull a private image on behalf of your Pod.
See the
PodSpec API
for more information about the
field.
Manually specifying an imagePullSecret
You can learn how to specify
from the
Arranging for imagePullSecrets to be automatically attached
You can manually create
, and reference these from a ServiceAccoun

t. Any Pods
created with that ServiceAccount or created with that ServiceAccount by default, will get their
field set to that of the service account.
See
for a detailed explanation of that process.
Using Secrets with static Pods
You cannot use ConfigMaps or Secrets with
.
Use cases
Use case: As container environment variables
Create a secret
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
USER_NAME
:
YWRtaW4=
PASSWORD
:
Create the Secret:
kubectl apply -f mysecret.yaml
Use
envFrom
to define all of the Secret's data as container environment variables. The key from
the Secret becomes the environment variable name in the Pod.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
envFrom
:
-
secretRef
:
name
:
mysecret
:
Never
Use case: Pod with SSH keys
Create a Secret containing some SSH keys:
kubectl create secret generic ssh-key-secret --from-file
=
ssh-privatekey
=
/path/to/.ssh/id_rsa --from-file
=
ssh-publickey
=
/path/to/.ssh/id_rsa.pub
secret "ssh-key-secret" created
You can also create a
with a
field containing ssh keys.
Caution:
Think carefully before sending your own SSH keys: other users of the cluster may have access
to the Secret.
You could instead create an SSH private key representing a service identity that you want to be
accessible to all the users with whom you share the Kubernetes cluster, and that you can revoke
if the credentials are compromised.
Now you can create a Pod which references the secret with the SSH key and
consumes it in a volume:
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
name
:
secret-test
spec
:
volumes
:
-
name
:
secret
:
:
ssh-key-secret
:
-
name
:
ssh-test-container
image
:
mySshImage
:
-
name
:
readOnly
:
true
mountPath
:
"/etc/secret-volume"
When the container's command runs, the pieces of the key will be available in:
/etc/secret-volume/ssh-publickey
/etc/secret-volume/ssh-privatekey
The container is then free to use the secret data to establish an SSH connection.
Use case: Pods with prod / test credentials
This example illustrates a Pod which consumes a secret containing production credentials and
another Pod which consumes a secret with test environment credentials.
with a
field or run
.
kubectl create secret generic prod-db-secret --from-literal
=
username
=
produser --from-literal
=
password
=
Y4nys7f11
secret "prod-db-secret" created
You can also create a secret for test environment credentials.
kubectl create secret generic test-db-secret --from-literal
=
username
=
testuser --from-literal
=
password
=
iluvtests
secret "test-db-secret" created
Note:
Special characters such as
$
,
\
,
*
,
=
, and
!
will be interpreted by your
shell
and require escaping.
In most shells, the easiest way to escape the password is to surround it with single quotes (
'
).
For example, if your actual password is
, you should execute the command this way:
kubectl create secret generic dev-db-secret --from-literal
=
username
=
devuser --from-literal
=
password
=
You do not need to escape special characters in passwords from files (
).
Now make the Pods:
cat
kind: List
items:
- kind: Pod
metadata:
name: prod-db-client-pod
labels:
name: prod-db-client
spec:
volumes:
- name: secret-volume
secret:
secretName: prod-db-secret
- name: db-client-container
image: myClientImage
readOnly: true
mountPath: "/etc/secret-volume"
- kind: Pod
met

adata:
name: test-db-client-pod
labels:
name: test-db-client
spec:
volumes:
secret:
secretName: test-db-secret
EOF
Add the pods to the same
:
cat
&lt;&lt;EOF &gt;&gt; kustomization.yaml
- pod.yaml
EOF
Apply all those objects on the API server by running:
Both containers will have the following files present on their filesystems with the values
for each container's environment:
/etc/secret-volume/username
/etc/secret-volume/password
Note how the specs for the two Pods differ only in one field; this facilitates
creating Pods with different capabilities from a common Pod template.
You could further simplify the base Pod specification by using two service accounts:
prod-user
with the
prod-db-secret
test-user
with the
test-db-secret
The Pod specification is shortened to:
:
v1
kind
:
Pod
metadata
:
name
:
prod-db-client-pod
labels
:
name
:
prod-db-client
spec
:
:
:
-
name
:
db-client-container
image
:
myClientImage
Use case: dotfiles in a secret volume
You can make your data "hidden" by defining a key that begins with a dot.
This key represents a dotfile or "hidden" file. For example, when the following secret
is mounted into a volume,
:
:
v1
kind
:
Secret
metadata
:
name
:
dotfile-secret
data
:
.secret-file
:
dmFsdWUtMg0KDQo=
---
:
v1
kind
:
Pod
metadata
:
name
:
secret-dotfiles-pod
spec
:
volumes
:
-
name
:
secret
:
:
:
-
name
:
dotfile-test-container
image
:
command
:
- ls
-
"-l"
-
:
-
name
:
readOnly
:
true
mountPath
:
The volume will contain a single file, called
, and
the
will have this file present at the path
/etc/secret-volume/.secret-file
.
Note:
Files beginning with dot characters are hidden from the output of
ls -l
;
you must use
ls -la
to see them when listing directory contents.
Use case: Secret visible to one container in a Pod
Consider a program that needs to handle HTTP requests, do some complex business
logic, and then sign some messages with an HMAC. Because it has complex
application logic, there might be an unnoticed remote file reading exploit in
the server, which could expose the private key to an attacker.
This could be divided into two processes in two containers: a frontend container
which handles user interaction and business logic, but which cannot see the
private key; and a signer container that can see the private key, and responds
to simple signing requests from the frontend (for example, over localhost networking).
With this partitioned approach, an attacker now has to trick the application
server into doing something rather arbitrary, which may be harder than getting
it to read a file.
Types of Secret
When creating a Secret, you can specify its type using the
type
field of
the
Secret
resource, or certain equivalent
kubectl
command line flags (if available).
The Secret type is used to facilitate programmatic handling of the Secret data.
Kubernetes provides several built-in types for some common usage scenarios.
These types vary in terms of the validations performed and the constraints
Kubernetes imposes on them.
Built-in Type
Usage
Opaque
arbitrary user-defined data
ServiceAccount token
~/.dockercfg
file
file
kubernetes.io/basic-auth
credentials for basic authentication
kubernetes.io/ssh-auth
credentials for SSH authentication
data for a TLS client or server
bootstrap token data
You can define and use your own Secret type by assigning a non-empty s

tring as the
type
value for a Secret object (an empty string is treated as an
Opaque
type).
Kubernetes doesn't impose any constraints on the type name. However, if you
are using one of the built-in types, you must meet all the requirements defined
for that type.
If you are defining a type of secret that's for public use, follow the convention
and structure the secret type to have your domain name before the name, separated
by a
/
cloud-hosting.example.net/cloud-api-credentials
.
Opaque secrets
Opaque
is the default Secret type if omitted from a Secret configuration file.
When you create a Secret using
kubectl
, you will use the
generic
subcommand to indicate an
Opaque
Secret type. For example, the following
command creates an empty Secret of type
Opaque
.
kubectl create secret generic empty-secret
kubectl get secret empty-secret
The output looks like:
NAME           TYPE     DATA   AGE
empty-secret   Opaque   0      2m6s
The
DATA
column shows the number of data items stored in the Secret.
0
means you have created an empty Secret.
Service account token Secrets
A
type of Secret is used to store a
token credential that identifies a
.
Since 1.22, this type of Secret is no longer used to mount credentials into Pods,
and obtaining tokens via the
API is recommended instead of using service account token Secret objects.
Tokens obtained from the
API are more secure than ones stored in Secret objects,
because they have a bounded lifetime and are not readable by other API clients.
API.
You should only create a service account token Secret object
if you can't use the
API to obtain a token,
and the security exposure of persisting a non-expiring token credential
in a readable API object is acceptable to you.
When using this Secret type, you need to ensure that the
annotation is set to an existing
service account name. If you are creating both the ServiceAccount and
the Secret objects, you should create the ServiceAccount object first.
After the Secret is created, a Kubernetes
fills in some other fields such as the
annotation, and the
token
key in the
data
field, which is populated with an authentication token.
The following example configuration declares a service account token Secret:
:
v1
kind
:
Secret
metadata
:
name
:
secret-sa-sample
:
:
"sa-name"
type
:
data
:
# You can include additional key value pairs as you do with Opaque Secrets
extra
:
YmFyCg==
After creating the Secret, wait for Kubernetes to populate the
token
key in the
data
field.
See the
documentation for more information on how service accounts work.
You can also check the
Pod
for information on referencing service account credentials from within Pods.
Docker config Secrets
You can use one of the following
type
values to create a Secret to
store the credentials for accessing a container image registry:
The
type is reserved to store a serialized
which is the legacy format for configuring Docker command line.
When using this Secret type, you have to ensure the Secret
data
field
contains a
key whose value is content of a
file
encoded in the base64 format.
The
type is designed for storing a serialized
JSON that follows the same format rules as the
file
which is a new format for
.
When using this Secret type, the
data
field of the Secret object must
contain a
key, in which the content for the
file is

 provided as a base64 encoded string.
Below is an example for a
type of Secret:
:
v1
kind
:
Secret
metadata
:
name
:
secret-dockercfg
type
:
data
:
:
|
"&lt;base64 encoded ~/.dockercfg file&gt;"
Note:
If you do not want to perform the base64 encoding, you can choose to use the
When you create these types of Secrets using a manifest, the API
server checks whether the expected key exists in the
data
field, and
it verifies if the value provided can be parsed as a valid JSON. The API
server doesn't validate if the JSON actually is a Docker config file.
When you do not have a Docker config file, or you want to use
kubectl
to create a Secret for accessing a container registry, you can do:
kubectl create secret docker-registry secret-tiger-docker
\
=
tiger@acme.example
\
=
tiger
\
=
pass1234
\
=
my-registry.example:5000
That command creates a Secret of type
.
If you dump the
.data.dockerconfigjson
field from that new Secret and then
decode it from base64:
kubectl get secret secret-tiger-docker -o
jsonpath
=
'{.data.*}'
| base64 -d
then the output is equivalent to this JSON document (which is also a valid
Docker configuration file):
{
"auths"
: {
"my-registry.example:5000"
: {
:
"tiger"
,
:
"pass1234"
,
"email"
:
"tiger@acme.example"
,
"auth"
:
"dGlnZXI6cGFzczEyMzQ="
}
}
}
Note:
The
auth
value there is base64 encoded; it is obscured but not secret.
Anyone who can read that Secret can learn the registry access bearer token.
Basic authentication Secret
The
type is provided for storing credentials needed
for basic authentication. When using this Secret type, the
data
Secret must contain one of the following two keys:
username
: the user name for authentication
password
: the password or token for authentication
Both values for the above two keys are base64 encoded strings. You can, of
course, provide the clear text content using the
for Secret
creation.
The following manifest is an example of a basic authentication Secret:
:
v1
kind
:
Secret
metadata
:
name
:
secret-basic-auth
type
:
:
username
:
admin
# required field for kubernetes.io/basic-auth
password
:
t0p-Secret
The basic authentication Secret type is provided only for convenience.
You can create an
Opaque
type for credentials used for basic authentication.
However, using the defined and public Secret type (
) helps other
people to understand the purpose of your Secret, and sets a convention for what key names
to expect.
The Kubernetes API verifies that the required keys are set for a Secret of this type.
SSH authentication secrets
The builtin type
is provided for storing data used in
SSH authentication. When using this Secret type, you will have to specify a
key-value pair in the
data
(or
) field
as the SSH credential to use.
The following manifest is an example of a Secret used for SSH public/private
key authentication:
:
v1
kind
:
Secret
metadata
:
name
:
secret-ssh-auth
type
:
data
:
# the data is abbreviated in this example
:
|
MIIEpQIBAAKCAQEAulqb/Y ...
The SSH authentication Secret type is provided only for user's convenience.
You could instead create an
Opaque
type Secret for credentials used for SSH authentication.
to expect.
and the API server does verify if the required keys are provided in a Secret configuration.
Caution:
SSH private keys do not establish trusted communication between an SSH client and
host server on thei

r own. A secondary means of establishing trust is needed to
mitigate "man in the middle" attacks, such as a
known_hosts
file added to a ConfigMap.
TLS secrets
Kubernetes provides a builtin Secret type
for storing
a certificate and its associated key that are typically used for TLS.
One common use for TLS secrets is to configure encryption in transit for
an
Ingress
, but you can also use it
with other resources or directly in your workload.
When using this type of Secret, the
tls.key
and the
tls.crt
key must be provided
in the
data
(or
) field of the Secret configuration, although the API
server doesn't actually validate the values for each key.
The following YAML contains an example config for a TLS Secret:
:
v1
kind
:
Secret
metadata
:
name
:
secret-tls
type
:
data
:
tls.crt
:
|
MIIC2DCCAcCgAwIBAgIBATANBgkqh ...
tls.key
:
|
MIIEpgIBAAKCAQEA7yn3bRHQ5FHMQ ...
The TLS Secret type is provided for user's convenience. You can create an
Opaque
for credentials used for TLS server and/or client. However, using the builtin Secret
type helps ensure the consistency of Secret format in your project; the API server
does verify if the required keys are provided in a Secret configuration.
When creating a TLS Secret using
kubectl
, you can use the
tls
subcommand
as shown in the following example:
kubectl create secret tls my-tls-secret
\
--cert
=
path/to/cert/file
\
--key
=
path/to/key/file
The public/private key pair must exist before hand. The public key certificate
for
--cert
must be DER format as per
Section 5.1 of RFC 7468
,
and must match the given private key for
--key
(PKCS #8 in DER format;
Section 11 of RFC 7468
).
Note:
A kubernetes.io/tls Secret stores the Base64-encoded DER data for keys and
certificates. If you're familiar with PEM format for private keys and for certificates,
the base64 data are the same as that format except that you omit
the initial and the last lines that are used in PEM.
For example, for a certificate, you do
not
include
--------BEGIN CERTIFICATE-----
and
-------END CERTIFICATE----
.
Bootstrap token Secrets
A bootstrap token Secret can be created by explicitly specifying the Secret
type
to
. This type of Secret is designed for
tokens used during the node bootstrap process. It stores tokens used to sign
well-known ConfigMaps.
A bootstrap token Secret is usually created in the
namespace and
named in the form
where
&lt;token-id&gt;
is a 6 character
string of the token ID.
As a Kubernetes manifest, a bootstrap token Secret might look like the
:
v1
kind
:
Secret
metadata
:
name
:
bootstrap-token-5emitj
namespace
:
type
:
data
:
:
c3lzdGVtOmJvb3RzdHJhcHBlcnM6a3ViZWFkbTpkZWZhdWx0LW5vZGUtdG9rZW4=
:
MjAyMC0wOS0xM1QwNDozOToxMFo=
token-id
:
NWVtaXRq
:
a3E0Z2lodnN6emduMXAwcg==
:
dHJ1ZQ==
:
dHJ1ZQ==
A bootstrap type Secret has the following keys specified under
data
:
token-id
: A random 6 character string as the token identifier. Required.
: A random 16 character string as the actual token secret. Required.
: A human-readable string that describes what the token is
used for. Optional.
: An absolute UTC time using
RFC3339
specifying when the token
should be expired. Optional.
usage-bootstrap-&lt;usage&gt;
: A boolean flag indicating additional usage for
the bootstrap token.
: A comma-separated list of group names that will be
authenticated as in addition to the
group.
The above YAML may look confusing because the values are all in base64 encoded
strings. In fact, you can create an identical Secret using the following YAML:
:
v1
kind
:
Secret
metadata
:
# Note how the Secret is named
name
:
bootstrap-token-

5emitj
# A bootstrap token Secret usually resides in the kube-system namespace
namespace
:
type
:
:
:
"system:bootstrappers:kubeadm:default-node-token"
:
"2020-09-13T04:39:10Z"
# This token ID is used in the name
token-id
:
"5emitj"
:
"kq4gihvszzgn1p0r"
# This token can be used for authentication
:
"true"
# and it can be used for signing
:
"true"
Immutable Secrets
Kubernetes lets you mark specific Secrets (and ConfigMaps) as
immutable
.
Preventing changes to the data of an existing Secret has the following benefits:
(for clusters that extensively use Secrets - at least tens of thousands of unique Secret
to Pod mounts), switching to immutable Secrets improves the performance of your cluster
by significantly reducing load on kube-apiserver. The kubelet does not need to maintain
a [watch] on any Secrets that are marked as immutable.
Marking a Secret as immutable
You can create an immutable Secret by setting the
immutable
field to
true
:
v1
kind
:
Secret
metadata
:
...
data
:
...
immutable
:
true
You can also update any existing mutable Secret to make it immutable.
Note:
Once a Secret or ConfigMap is marked as immutable, it is
not
data
field. You can only delete and recreate the Secret.
Existing Pods maintain a mount point to the deleted Secret - it is recommended to recreate
these pods.
Although ConfigMap and Secret work similarly, Kubernetes applies some additional
protection for Secret objects.
Secrets often hold values that span a spectrum of importance, many of which can
cause escalations within Kubernetes (e.g. service account tokens) and to
external systems. Even if an individual app can reason about the power of the
Secrets it expects to interact with, other apps within the same namespace can
render those assumptions invalid.
A Secret is only sent to a node if a Pod on that node requires it.
For mounting secrets into Pods, the kubelet stores a copy of the data into a
tmpfs
so that the confidential data is not written to durable storage.
Once the Pod that depends on the Secret is deleted, the kubelet deletes its local copy
of the confidential data from the Secret.
There may be several containers in a Pod. By default, containers you define
only have access to the default ServiceAccount and its related Secret.
You must explicitly define environment variables or map a volume into a
container in order to provide access to any other Secret.
There may be Secrets for several Pods on the same node. However, only the
Secrets that a Pod requests are potentially visible within its containers.
Therefore, one Pod does not have access to the Secrets of another Pod.
Warning:
Any containers that run with
on a node can access all
Secrets used on that node.
For guidelines to manage and improve the security of your Secrets, refer to
.
manage Secrets using
kubectl
Read the
for
Secret
8.4 - Resource Management for Pods and Containers
When you specify a
Pod
, you can optionally specify how
much of each resource a
container
needs.
The most common resources to specify are CPU and memory (RAM); there are others.
When you specify the resource
request
for containers in a Pod, the
uses this
information to decide which node to place the Pod on. When you specify a resource
limit
for a container, the kubelet enforces those limits so that the running container is not
allowed to use more of that resource than the limit you set. The kubelet also reserves
at least th

e
request
amount of that system resource specifically for that container
to use.
Requests and limits
If the node where a Pod is running has enough of a resource available, it's possible (and
allowed) for a container to use more resource than its
request
for that resource specifies.
However, a container is not allowed to use more than its resource
limit
.
For example, if you set a
memory
request of 256 MiB for a container, and that container is in
a Pod scheduled to a Node with 8GiB of memory and no other Pods, then the container can try to use
more RAM.
If you set a
memory
limit of 4GiB for that container, the kubelet (and
) enforce the limit.
The runtime prevents the container from using more than the configured resource limit. For example:
when a process in the container tries to consume more than the allowed amount of memory,
the system kernel terminates the process that attempted the allocation, with an out of memory
(OOM) error.
Limits can be implemented either reactively (the system intervenes once it sees a violation)
or by enforcement (the system prevents the container from ever exceeding the limit). Different
runtimes can have different ways to implement the same restrictions.
Note:
If you specify a limit for a resource, but do not specify any request, and no admission-time
mechanism has applied a default request for that resource, then Kubernetes copies the limit
you specified and uses it as the requested value for the resource.
CPU
and
memory
are each a
. A resource type has a base unit.
CPU represents compute processing and is specified in units of
Kubernetes CPUs
.
Memory is specified in units of bytes.
For Linux workloads, you can specify
huge page
Huge pages are a Linux-specific feature where the node kernel allocates blocks of memory
that are much larger than the default page size.
For example, on a system where the default page size is 4KiB, you could specify a limit,
hugepages-2Mi: 80Mi
. If the container tries allocating over 40 2MiB huge pages (a
total of 80 MiB), that allocation fails.
Note:
You cannot overcommit
hugepages-*
This is different from the
memory
and
cpu
CPU and memory are collectively referred to as
compute resources
, or
resources
. Compute
resources are measurable quantities that can be requested, allocated, and
consumed. They are distinct from
API resources
. API resources, such as Pods and
Services
are objects that can be read and modified
through the Kubernetes API server.
Resource requests and limits of Pod and container
For each container, you can specify resource limits and requests,
including the following:
spec.containers[].resources.limits.cpu
spec.containers[].resources.limits.memory
spec.containers[].resources.limits.hugepages-&lt;size&gt;
spec.containers[].resources.requests.cpu
spec.containers[].resources.requests.memory
spec.containers[].resources.requests.hugepages-&lt;size&gt;
Although you can only specify requests and limits for individual containers,
it is also useful to think about the overall resource requests and limits for
a Pod.
For a particular resource, a
Pod resource request/limit
is the sum of the
resource requests/limits of that type for each container in the Pod.
Resource units in Kubernetes
CPU resource units
Limits and requests for CPU resources are measured in
cpu
units.
In Kubernetes, 1 CPU unit is equivalent to
1 physical CPU core
,
or
1 virtual core
, depending on whether the node is a physical host
or a virtual machine running inside a physical machine.
Fractional requests are allowed. When you define a container with
set to
0.5
, you are requesting half
as much CPU time compared to if you asked for
1.0
CPU.
For CPU resource units, the
quantity
0.1
is equivalent to the
100m
, which can be read as "one hundred millicpu". Some people say
"one hundred millicores", and this is understood to mean the same thing.
CPU reso

urce is always specified as an absolute amount of resource, never as a relative amount. For example,
500m
CPU represents the roughly same amount of computing power whether that container
runs on a single-core, dual-core, or 48-core machine.
Note:
Kubernetes doesn't allow you to specify CPU resources with a precision finer than
1m
. Because of this, it's useful to specify CPU units less than
1.0
or
1000m
using
the milliCPU form; for example,
5m
rather than
0.005
.
Memory resource units
Limits and requests for
memory
are measured in bytes. You can express memory as
a plain integer or as a fixed-point number using one of these
quantity
suffixes:
E, P, T, G, M, k. You can also use the power-of-two equivalents: Ei, Pi, Ti, Gi,
Mi, Ki. For example, the following represent roughly the same value:
128974848, 129e6, 129M,  128974848000m, 123Mi
Pay attention to the case of the suffixes. If you request
400m
of memory, this is a request
for 0.4 bytes. Someone who types that probably meant to ask for 400 mebibytes (
400Mi
)
or 400 megabytes (
400M
).
Container resources example
The following Pod has two containers. Both containers are defined with a request for
0.25 CPU
and 64MiB (2
26
bytes) of memory. Each container has a limit of 0.5
CPU and 128MiB of memory. You can say the Pod has a request of 0.5 CPU and 128
MiB of memory, and a limit of 1 CPU and 256MiB of memory.
---
:
v1
kind
:
Pod
metadata
:
name
:
frontend
spec
:
:
-
name
:
app
image
:
images.my-company.example/app:v4
resources
:
requests
:
memory
:
"64Mi"
cpu
:
"250m"
limits
:
memory
:
"128Mi"
cpu
:
"500m"
-
name
:
log-aggregator
image
:
images.my-company.example/log-aggregator:v6
resources
:
requests
:
memory
:
"64Mi"
cpu
:
"250m"
limits
:
memory
:
"128Mi"
cpu
:
"500m"
How Pods with resource requests are scheduled
When you create a Pod, the Kubernetes scheduler selects a node for the Pod to
run on. Each node has a maximum capacity for each of the resource types: the
amount of CPU and memory it can provide for Pods. The scheduler ensures that,
for each resource type, the sum of the resource requests of the scheduled
containers is less than the capacity of the node.
Note that although actual memory
or CPU resource usage on nodes is very low, the scheduler still refuses to place
a Pod on a node if the capacity check fails. This protects against a resource
shortage on a node when resource usage later increases, for example, during a
daily peak in request rate.
How Kubernetes applies resource requests and limits
When the kubelet starts a container as part of a Pod, the kubelet passes that container's
requests and limits for memory and CPU to the container runtime.
On Linux, the container runtime typically configures
kernel
cgroups
that apply and enforce the
limits you defined.
The CPU limit defines a hard ceiling on how much CPU time that the container can use.
During each scheduling interval (time slice), the Linux kernel checks to see if this
limit is exceeded; if so, the kernel waits before allowing that cgroup to resume execution.
The CPU request typically defines a weighting. If several different containers (cgroups)
want to run on a contended system, workloads with larger CPU requests are allocated more
CPU time than workloads with small requests.
The memory request is mainly used during (Kubernetes) Pod scheduling. On a node that uses
cgroups v2, the container runtime might use the memory request as a hint to set
memory.min
and
memory.low
.
The memory limit defines a memory limit for that cgroup. If the container tries to
allocate more memory than this limit, the Linux kernel out-of-memory subsystem activates
and, typically, intervenes by stopping one of the processes in the container that tried
to allocate memory. If that process is the container's PID 1, and the container is marked
as restartable, Kubernetes restarts the container.
The memory limit for the Pod or container can also apply to pages in memory backed
volumes, such as an
emptyDir
. The kub

elet tracks
tmpfs
emptyDir volumes as container
memory use, rather than as local ephemeral storage.
If a container exceeds its memory request and the node that it runs on becomes short of
memory overall, it is likely that the Pod the container belongs to will be
evicted
.
A container might or might not be allowed to exceed its CPU limit for extended periods of time.
However, container runtimes don't terminate Pods or containers for excessive CPU usage.
To determine whether a container cannot be scheduled or is being killed due to resource limits,
see the
section.
Monitoring compute &amp; memory resource usage
The kubelet reports the resource usage of a Pod as part of the Pod
status
.
If optional
tools for monitoring
are available in your cluster, then Pod resource usage can be retrieved either
from the
directly or from your monitoring tools.
Local ephemeral storage
Nodes have local ephemeral storage, backed by
locally-attached writeable devices or, sometimes, by RAM.
"Ephemeral" means that there is no long-term guarantee about durability.
Pods use ephemeral local storage for scratch space, caching, and for logs.
The kubelet can provide scratch space to Pods using local ephemeral storage to
mount
emptyDir
volumes
into containers.
The kubelet also uses this kind of storage to hold
node-level container logs
,
container images, and the writable layers of running containers.
Caution:
If a node fails, the data in its ephemeral storage can be lost.
Your applications cannot expect any performance SLAs (disk IOPS for example)
from local ephemeral storage.
As a beta feature, Kubernetes lets you track, reserve and limit the amount
of ephemeral local storage a Pod can consume.
Configurations for local ephemeral storage
Kubernetes supports two ways to configure local ephemeral storage on a node:
Single filesystem
Two filesystems
In this configuration, you place all different kinds of ephemeral local data
(
emptyDir
volumes, writeable layers, container images, logs) into one filesystem.
The most effective way to configure the kubelet means dedicating this filesystem
to Kubernetes (kubelet) data.
The kubelet also writes
and treats these similarly to ephemeral local storage.
The kubelet writes logs to files inside its configured log directory (
/var/log
by default); and has a base directory for other locally stored data
(
Typically, both
and
/var/log
are on the system root filesystem,
and the kubelet is designed with that layout in mind.
Your node can have as many other filesystems, not used for Kubernetes,
as you like.
You have a filesystem on the node that you're using for ephemeral data that
comes from running Pods: logs, and
emptyDir
volumes. You can use this filesystem
for other data (for example: system logs not related to Kubernetes); it can even
be the root filesystem.
into the first filesystem, and treats these similarly to ephemeral local storage.
You also use a separate filesystem, backed by a different logical storage device.
In this configuration, the directory where you tell the kubelet to place
container image layers and writeable layers is on this second filesystem.
The first filesystem does not hold any image layers or writeable layers.
The kubelet can measure how much local storage it is using. It does this provided
that you have set up the node using one of the supported configurations for local
ephemeral storage.
If you have a different configuration, then the kubelet does not apply resource
limits for ephemeral local storage.
Note:
The kubelet tracks
tmpfs
emptyDir volumes as container memory use, rather
than as local ephemeral storage.
Note:
The kubelet will only track the root filesystem for ephemeral storage. OS layouts that mount a separate disk t

o
or
will not report ephemeral storage correctly.
Setting requests and limits for local ephemeral storage
for managing local ephemeral storage. Each
container of a Pod can specify either or both of the following:
spec.containers[].resources.limits.ephemeral-storage
spec.containers[].resources.requests.ephemeral-storage
are measured in byte quantities.
You can express storage as a plain integer or as a fixed-point number using one of these suffixes:
Mi, Ki. For example, the following quantities all represent roughly the same value:
128974848
129e6
129M
123Mi
400m
of ephemeral-storage, this is a request
400Mi
)
400M
).
In the following example, the Pod has two containers. Each container has a request of
2GiB of local ephemeral storage. Each container has a limit of 4GiB of local ephemeral
storage. Therefore, the Pod has a request of 4GiB of local ephemeral storage, and
a limit of 8GiB of local ephemeral storage. 500Mi of that limit could be
consumed by the
emptyDir
volume.
:
v1
kind
:
Pod
metadata
:
name
:
frontend
spec
:
:
-
name
:
app
image
:
resources
:
requests
:
:
"2Gi"
limits
:
:
"4Gi"
:
-
name
:
ephemeral
mountPath
:
"/tmp"
-
name
:
image
:
resources
:
requests
:
:
"2Gi"
limits
:
:
"4Gi"
:
-
name
:
ephemeral
mountPath
:
"/tmp"
volumes
:
-
name
:
ephemeral
emptyDir
:
sizeLimit
:
500Mi
How Pods with ephemeral-storage requests are scheduled
run on. Each node has a maximum amount of local ephemeral storage it can provide for Pods.
.
The scheduler ensures that the sum of the resource requests of the scheduled containers is less than the capacity of the node.
Ephemeral storage consumption management
If the kubelet is managing local ephemeral storage as a resource, then the
kubelet measures storage use in:
emptyDir
volumes, except
tmpfs
emptyDir
volumes
directories holding node-level logs
writeable container layers
If a Pod is using more ephemeral storage than you allow it to, the kubelet
sets an eviction signal that triggers Pod eviction.
For container-level isolation, if a container's writable layer and log
usage exceeds its storage limit, the kubelet marks the Pod for eviction.
For pod-level isolation the kubelet works out an overall Pod storage limit by
summing the limits for the containers in that Pod. In this case, if the sum of
the local ephemeral storage usage from all containers and also the Pod's
emptyDir
volumes exceeds the overall Pod storage limit, then the kubelet also marks the Pod
for eviction.
Caution:
If the kubelet is not measuring local ephemeral storage, then a Pod
that exceeds its local storage limit will not be evicted for breaching
local storage resource limits.
However, if the filesystem space for writeable container layers, node-level logs,
or
emptyDir
volumes falls low, the node
taints
itself as short on local storage
and this taint triggers eviction for any Pods that don't specifically tolerate the taint.
See the supported
for ephemeral local storage.
The kubelet supports different ways to measure Pod storage use:
Periodic scanning
Filesystem project quota
The kubelet performs regular, scheduled checks that scan each
emptyDir
volume, container log directory, and writeable container layer.
The scan measures how much space is used.
Note:
In this mode, the kubelet does not track open file descriptors
for deleted files.

If you (or a container) create a file inside an
emptyDir
volume,
something then opens that file, and you delete the file while it is
still open, then the inode for the deleted file stays until you close
that file but the kubelet does not categorize the space as in use.
Kubernetes v1.15 [alpha]
Project quotas are an operating-system level feature for managing
storage use on filesystems. With Kubernetes, you can enable project
quotas for monitoring storage use. Make sure that the filesystem
backing the
emptyDir
volumes, on the node, provides project quota support.
For example, XFS and ext4fs offer project quotas.
Note:
Project quotas let you monitor storage use; they do not enforce limits.
Kubernetes uses project IDs starting from
1048576
. The IDs in use are
registered in
/etc/projects
and
/etc/projid
. If project IDs in
this range are used for other purposes on the system, those project
IDs must be registered in
and
so that
Kubernetes does not use them.
Quotas are faster and more accurate than directory scanning. When a
directory is assigned to a project, all files created under a
directory are created in that project, and the kernel merely has to
keep track of how many blocks are in use by files in that project.
If a file is created and deleted, but has an open file descriptor,
it continues to consume space. Quota tracking records that space accurately
whereas directory scans overlook the storage used by deleted files.
If you want to use project quotas, you should:
Enable the
LocalStorageCapacityIsolationFSQuotaMonitoring=true
using the
or the
Ensure that the root filesystem (or optional runtime filesystem)
has project quotas enabled. All XFS filesystems support project quotas.
For ext4 filesystems, you need to enable the project quota tracking feature
while the filesystem is not mounted.
# For ext4, with /dev/block-device not mounted
sudo tune2fs -O project -Q prjquota /dev/block-device
Ensure that the root filesystem (or optional runtime filesystem) is
mounted with project quotas enabled. For both XFS and ext4fs, the
mount option is named
prjquota
.
Extended resources
Extended resources are fully-qualified resource names outside the
domain. They allow cluster operators to advertise and users to
consume the non-Kubernetes-built-in resources.
There are two steps required to use Extended Resources. First, the cluster
operator must advertise an Extended Resource. Second, users must request the
Extended Resource in Pods.
Managing extended resources
Node-level extended resources
Node-level extended resources are tied to nodes.
Device plugin managed resources
See
Device
Plugin
for how to advertise device plugin managed resources on each node.
Other resources
To advertise a new node-level extended resource, the cluster operator can
submit a
PATCH
HTTP request to the API server to specify the available
quantity in the
status.capacity
for a node in the cluster. After this
operation, the node's
will include a new resource. The
status.allocatable
field is updated automatically with the new resource
asynchronously by the kubelet.
Because the scheduler uses the node's
value when
evaluating Pod fitness, the scheduler only takes account of the new value after
that asynchronous update. There may be a short delay between patching the
node capacity with a new resource and the time when the first Pod that requests
the resource can be scheduled on that node.
Example:
Here is an example showing how to use
curl
to form an HTTP request that
advertises five "example.com/foo" resources on node
whose master
is
k8s-master
.
\
\
--data
'[{"op": "add", "path": "/status/capacity/example.com~1foo", "value": "5"}]'
\
http://k8s-master:8080/api/v1/nodes/k8s-node-1/status
Note:
In the preced

ing request,
~1
is the encoding for the character
/
in the patch path. The operation path value in JSON-Patch is interpreted as a
IETF RFC 6901, section 3
.
Cluster-level extended resources
Cluster-level extended resources are not tied to nodes. They are usually managed
by scheduler extenders, which handle the resource consumption and resource quota.
You can specify the extended resources that are handled by scheduler extenders
in
scheduler configuration
Example:
The following configuration for a scheduler policy indicates that the
cluster-level extended resource "example.com/foo" is handled by the scheduler
extender.
The scheduler sends a Pod to the scheduler extender only if the Pod requests
"example.com/foo".
The
field specifies that the scheduler does not check
the "example.com/foo" resource in its
PodFitsResources
predicate.
{
"kind"
:
"Policy"
,
:
"v1"
,
"extenders"
: [
{
"urlPrefix"
:
"&lt;extender-endpoint&gt;"
,
"bindVerb"
:
"bind"
,
"managedResources"
: [
{
"name"
:
"example.com/foo"
,
"ignoredByScheduler"
:
true
}
]
}
]
}
Consuming extended resources
Users can consume extended resources in Pod specs like CPU and memory.
The scheduler takes care of the resource accounting so that no more than the
available amount is simultaneously allocated to Pods.
The API server restricts quantities of extended resources to whole numbers.
Examples of
valid
quantities are
3
,
3000m
and
3Ki
. Examples of
invalid
0.5
and
1500m
.
Note:
Extended resources replace Opaque Integer Resources.
Users can use any domain name prefix other than
which is reserved.
To consume an extended resource in a Pod, include the resource name as a key
in the
spec.containers[].resources.limits
map in the container spec.
Note:
Extended resources cannot be overcommitted, so request and limit
must be equal if both are present in a container spec.
A Pod is scheduled only if all of the resource requests are satisfied, including
CPU, memory and any extended resources. The Pod remains in the
PENDING
state
as long as the resource request cannot be satisfied.
Example:
The Pod below requests 2 CPUs and 1 "example.com/foo" (an extended resource).
:
v1
kind
:
Pod
metadata
:
name
:
my-pod
spec
:
:
-
name
:
my-container
image
:
myimage
resources
:
requests
:
cpu
:
2
:
1
limits
:
:
1
PID limiting
Process ID (PID) limits allow for the configuration of a kubelet
to limit the number of PIDs that a given Pod can consume. See
PID Limiting
My Pods are pending with event message
If the scheduler cannot find any node where a Pod can fit, the Pod remains
unscheduled until a place can be found. An
Event
is produced
each time the scheduler fails to find a place for the Pod. You can use
kubectl
to view the events for a Pod; for example:
kubectl describe pod frontend | grep -A
9999999999
Events
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  23s   default-scheduler  0/42 nodes available: insufficient cpu
In the preceding example, the Pod named "frontend" fails to be scheduled due to
insufficient CPU resource on any node. Similar error messages can also suggest
failure due to insufficient memory (PodExceedsFreeMemory). In general, if a Pod
is pending with a message of this type, there are several things to try:
Add more nodes to the cluster.
Terminate unneeded Pods to make room for pending Pods.
Check that the Pod is not larger than all the nodes. For example, if all the
nodes have a capacity of
cpu: 1
, then a Pod with a request of
cpu: 1.1
will
never be scheduled.
Check for node taints. If most of your nodes are tainted, and the new Pod does
not tolerate that taint, the scheduler only considers placements onto the
remaining nodes that don't have that taint.
You can check node c

apacities and amounts allocated with the
kubectl describe nodes e2e-test-node-pool-4lw4
Name:            e2e-test-node-pool-4lw4
[ ... lines removed for clarity ...]
Capacity:
 cpu:                               2
 memory:                            7679792Ki
 pods:                              110
 cpu:                               1800m
 memory:                            7474992Ki
 pods:                              110
Non-terminated Pods:        (5 in total)
  Namespace    Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------    ----                                  ------------  ----------  ---------------  -------------
  kube-system  fluentd-gcp-v1.38-28bv1               100m (5%)     0 (0%)      200Mi (2%)       200Mi (2%)
  kube-system  kube-dns-3297075139-61lj3             260m (13%)    0 (0%)      100Mi (1%)       170Mi (2%)
  kube-system  kube-proxy-e2e-test-...               100m (5%)     0 (0%)      0 (0%)           0 (0%)
  kube-system  monitoring-influxdb-grafana-v4-z1m12  200m (10%)    200m (10%)  600Mi (8%)       600Mi (8%)
  kube-system  node-problem-detector-v0.1-fj7m3      20m (1%)      200m (10%)  20Mi (0%)        100Mi (1%)
  CPU Requests    CPU Limits    Memory Requests    Memory Limits
  ------------    ----------    ---------------    -------------
  680m (34%)      400m (20%)    920Mi (11%)        1070Mi (13%)
In the preceding output, you can see that if a Pod requests more than 1.120 CPUs
or more than 6.23Gi of memory, that Pod will not fit on the node.
By looking at the “Pods” section, you can see which Pods are taking up space on
the node.
The amount of resources available to Pods is less than the node capacity because
system daemons use a portion of the available resources. Within the Kubernetes API,
each Node has a
.status.allocatable
field
(see
for details).
The
field describes the amount of resources that are available
to Pods on that node (for example: 15 virtual CPUs and 7538 MiB of memory).
For more information on node allocatable resources in Kubernetes, see
.
resource quotas
to limit the total amount of resources that a namespace can consume.
Kubernetes enforces quotas for objects in particular namespace when there is a
ResourceQuota in that namespace.
For example, if you assign specific namespaces to different teams, you
can add ResourceQuotas into those namespaces. Setting resource quotas helps to
prevent one team from using so much of any resource that this over-use affects other teams.
You should also consider what access you grant to that namespace:
full
write access to a namespace allows someone with that access to remove any
resource, including a configured ResourceQuota.
My container is terminated
Your container might get terminated because it is resource-starved. To check
whether a container is being killed because it is hitting a resource limit, call
on the Pod of interest:
kubectl describe pod simmemleak-hra99
Name:                           simmemleak-hra99
Image(s):                       saadali/simmemleak
Node:                           kubernetes-node-tf0f/10.240.216.66
Labels:                         name=simmemleak
Status:                         Running
Reason:
Message:
IP:                             10.244.2.75
  simmemleak:
    Image:  saadali/simmemleak:latest
    Limits:
      cpu:          100m
      memory:       50Mi
      Started:      Tue, 07 Jul 2019 12:54:41 -0700
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Fri, 07 Jul 2019 12:54:30 -0700
      Finished

:     Fri, 07 Jul 2019 12:54:33 -0700
    Restart Count:  5
  Type      Status
  Ready     False
Events:
  Normal  Scheduled  42s   default-scheduler  Successfully assigned simmemleak-hra99 to kubernetes-node-tf0f
  Normal  Pulled     41s   kubelet            Container image "saadali/simmemleak:latest" already present on machine
  Normal  Created    41s   kubelet            Created container simmemleak
  Normal  Started    40s   kubelet            Started container simmemleak
  Normal  Killing    32s   kubelet            Killing container with id ead3fb35-5cf5-44ed-9ae1-488115be66c6: Need to kill Pod
Restart Count: 5
indicates that the
simmemleak
container in the Pod was terminated and restarted five times (so far).
The
OOMKilled
reason shows that the container tried to use more memory than its limit.
Your next step might be to check the application code for a memory leak. If you
find that the application is behaving how you expect, consider setting a higher
memory limit (and possibly request) for that container.
assigning Memory resources to containers and Pods
.
assigning CPU resources to containers and Pods
.
Read how the API reference defines a
container
and its
resource requirements
Read about
project quotas
in XFS
kube-scheduler configuration reference (v1beta3)
8.5 - Organizing Cluster Access Using kubeconfig Files
Use kubeconfig files to organize information about clusters, users, namespaces, and
authentication mechanisms. The
kubectl
command-line tool uses kubeconfig files to
find the information it needs to choose a cluster and communicate with the API server
Note:
A file that is used to configure access to clusters is called
a
. This is a generic way of referring to configuration files.
It does not mean that there is a file named
.
Warning:
Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig file could result in malicious code execution or file exposure.
If you must use an untrusted kubeconfig file, inspect it carefully first, much as you would a shell script.
kubectl
config
in the
You can specify other kubeconfig files by setting the
variable or by setting the
flag.
For step-by-step instructions on creating and specifying kubeconfig files, see
.
Supporting multiple clusters, users, and authentication mechanisms
Suppose you have several clusters, and your users and components authenticate
in a variety of ways. For example:
A running kubelet might authenticate using certificates.
A user might authenticate using tokens.
Administrators might have sets of certificates that they provide to individual users.
With kubeconfig files, you can organize your clusters, users, and namespaces.
You can also define contexts to quickly and easily switch between
clusters and namespaces.
Context
A
context
element in a kubeconfig file is used to group access parameters
under a convenient name. Each context has three parameters: cluster, namespace, and user.
kubectl
command-line tool uses parameters from
the
current context
to communicate with the cluster.
To choose the current context:
kubectl config use-context
The KUBECONFIG environment variable
The
environment variable holds a list of kubeconfig files.
For Linux and Mac, the list is colon-delimited. For Windows, the list
is semicolon-delimited. The
environment variable is not
required. If the
environment variable doesn't exist,
kubectl
uses the default kubeconfig file,
$HOME/.kube/config
.
If the
environment variable does exist,
kubectl
uses
an effective configuration tha

t is the result of merging the files
listed in the
Merging kubeconfig files
To see your configuration, enter this command:
As described previously, the output might be from a single kubeconfig file,
or it might be the result of merging several kubeconfig files.
Here are the rules that
kubectl
uses when it merges kubeconfig files:
If the
flag is set, use only the specified file. Do not merge.
Only one instance of this flag is allowed.
Otherwise, if the
environment variable is set, use it as a
list of files that should be merged.
Merge the files listed in the
according to these rules:
Ignore empty filenames.
Produce errors for files with content that cannot be deserialized.
The first file to set a particular value or map key wins.
Never change the value or map key.
Example: Preserve the context of the first file to set
.
Example: If two files specify a
red-user
, use only values from the first file's
red-user
.
Even if the second file has non-conflicting entries under
red-user
, discard them.
For an example of setting the
environment variable, see
Setting the KUBECONFIG environment variable
.
Otherwise, use the default kubeconfig file,
, with no merging.
Determine the context to use based on the first hit in this chain:
Use the
--context
command-line flag if it exists.
Use the
from the merged kubeconfig files.
An empty context is allowed at this point.
Determine the cluster and user. At this point, there might or might not be a context.
Determine the cluster and user based on the first hit in this chain,
which is run twice: once for user and once for cluster:
Use a command-line flag if it exists:
--user
or
--cluster
.
If the context is non-empty, take the user or cluster from the context.
The user and cluster can be empty at this point.
Determine the actual cluster information to use. At this point, there might or
might not be cluster information.
Build each piece of the cluster information based on this chain; the first hit wins:
Use command line flags if they exist:
--server
,
,
.
If any cluster information attributes exist from the merged kubeconfig files, use them.
If there is no server location, fail.
Determine the actual user information to use. Build user information using the same
rules as cluster information, except allow only one authentication
technique per user:
,
,
,
,
--token
.
Use the
user
fields from the merged kubeconfig files.
If there are two conflicting techniques, fail.
For any information still missing, use default values and potentially
prompt for authentication information.
File references
File and path references in a kubeconfig file are relative to the location of the kubeconfig file.
File references on the command line are relative to the current working directory.
In
, relative paths are stored relatively, and absolute paths
are stored absolutely.
Proxy
kubectl
to use a proxy per cluster using
proxy-url
in your kubeconfig file, like this:
:
v1
kind
:
Config
clusters
:
-
cluster
:
proxy-url
:
http://proxy.example.org:3128
server
:
https://k8s.example.org/k8s/clusters/c-xxyyzz
name
:
users
:
-
name
:
developer
contexts
:
-
context
:
name
:
8.6 - Resource Management for Windows nodes
This page outlines the differences in how resources are managed between Linux and Windows.
On Linux nodes,
cgroups
are used
as a pod boundary for resource control. Containers are created within that boundary
for network, process and file system isolation. The Linux cgroup APIs can be used to
gather CPU, I/O, and memory use statistics.
In contrast, Windows uses a
job object
per con

tainer with a system namespace filter
to contain all processes in a container and provide logical isolation from the
host.
(Job objects are a Windows process isolation mechanism and are different from
what Kubernetes refers to as a
Job
).
There is no way to run a Windows container without the namespace filtering in
place. This means that system privileges cannot be asserted in the context of the
host, and thus privileged containers are not available on Windows.
Containers cannot assume an identity from the host because the Security Account Manager
(SAM) is separate.
Memory management
Windows does not have an out-of-memory process killer as Linux does. Windows always
treats all user-mode memory allocations as virtual, and pagefiles are mandatory.
Windows nodes do not overcommit memory for processes. The
net effect is that Windows won't reach out of memory conditions the same way Linux
does, and processes page to disk instead of being subject to out of memory (OOM)
termination. If memory is over-provisioned and all physical memory is exhausted,
then paging can slow down performance.
CPU management
Windows can limit the amount of CPU time allocated for different processes but cannot
guarantee a minimum amount of CPU time.
On Windows, the kubelet supports a command-line flag to set the
scheduling priority
of the
kubelet process:
. This flag allows the kubelet process to get
more CPU time slices when compared to other processes running on the Windows host.
More information on the allowable values and their meaning is available at
Windows Priority Classes
.
To ensure that running Pods do not starve the kubelet of CPU cycles, set this flag to
ABOVE_NORMAL_PRIORITY_CLASS
or above.
Resource reservation
To account for memory and CPU used by the operating system, the container runtime, and by
Kubernetes host processes such as the kubelet, you can (and should) reserve
memory and CPU resources with the
and/or
kubelet flags.
On Windows these values are only used to calculate the node's
Caution:
As you deploy workloads, set resource memory and CPU limits on containers.
This also subtracts from
and helps the cluster-wide scheduler in determining which pods to place on which nodes.
Scheduling pods without limits may over-provision the Windows nodes and in extreme
cases can cause the nodes to become unhealthy.
On Windows, a good practice is to reserve at least 2GiB of memory.
To determine how much CPU to reserve,
identify the maximum pod density for each node and monitor the CPU usage of
the system services running there, then choose a value that meets your workload needs.
9 - Security
9.1 - Overview of Cloud Native Security
A model for thinking about Kubernetes security in the context of Cloud Native security.
This overview defines a model for thinking about Kubernetes security in the context of Cloud Native security.
Warning:
This container security model provides suggestions, not proven information security policies.
The 4C's of Cloud Native security
You can think about security in layers. The 4C's of Cloud Native security are Cloud,
Clusters, Containers, and Code.
Note:
This layered approach augments the
defense in depth
computing approach to security, which is widely regarded as a best practice for securing
software systems.
The 4C's of Cloud Native Security
Each layer of the Cloud Native security model builds upon the next outermost layer.
The Code layer benefits from strong base (Cloud, Cluster, Container) security layers.
You cannot safeguard against poor security standards in the base layers by addressing
security at the Code level.
Cloud
In many ways, the Cloud (or co-located servers, or the corporate datacenter) is the
trusted computing base
of a Kubernetes cluster. If the Cloud layer is vulnerable (or
configured in a vulnerable way) then there is no guarantee that the components bui

lt
on top of this base are secure. Each cloud provider makes security recommendations
for running workloads securely in their environment.
Cloud provider security
If you are running a Kubernetes cluster on your own hardware or a different cloud provider,
consult your documentation for security best practices.
Here are links to some of the popular cloud providers' security documentation:
IaaS Provider
Link
https://www.alibabacloud.com/trust-center
Amazon Web Services
https://aws.amazon.com/security
Google Cloud Platform
https://cloud.google.com/security
Huawei Cloud
https://www.huaweicloud.com/securecenter/overallsafety
IBM Cloud
https://www.ibm.com/cloud/security
Microsoft Azure
https://docs.microsoft.com/en-us/azure/security/azure-security
Oracle Cloud Infrastructure
https://www.oracle.com/security
VMware vSphere
https://www.vmware.com/security/hardening-guides
Infrastructure security
Suggestions for securing your infrastructure in a Kubernetes cluster:
Area of Concern for Kubernetes Infrastructure
Recommendation
Network access to API Server (Control plane)
All access to the Kubernetes control plane is not allowed publicly on the internet and is controlled by network access control lists restricted to the set of IP addresses needed to administer the cluster.
Network access to Nodes (nodes)
Nodes should be configured to
only
accept connections (via network access control lists) from the control plane on the specified ports, and accept connections for services in Kubernetes of type NodePort and LoadBalancer. If possible, these nodes should not be exposed on the public internet entirely.
Kubernetes access to Cloud Provider API
Each cloud provider needs to grant a different set of permissions to the Kubernetes control plane and nodes. It is best to provide the cluster with cloud provider access that follows the
for the resources it needs to administer. The
Kops documentation
provides information about IAM policies and roles.
Access to etcd
Access to etcd (the datastore of Kubernetes) should be limited to the control plane only. Depending on your configuration, you should attempt to use etcd over TLS. More information can be found in the
.
etcd Encryption
Wherever possible it's a good practice to encrypt all storage at rest, and since etcd holds the state of the entire cluster (including Secrets) its disk should especially be encrypted at rest.
Cluster
There are two areas of concern for securing Kubernetes:
Securing the cluster components that are configurable
Securing the applications which run in the cluster
Components of the Cluster
If you want to protect your cluster from accidental or malicious access and adopt
good information practices, read and follow the advice about
securing your cluster
.
Components in the cluster (your application)
Depending on the attack surface of your application, you may want to focus on specific
aspects of security. For example: If you are running a service (Service A) that is critical
in a chain of other resources and a separate workload (Service B) which is
vulnerable to a resource exhaustion attack, then the risk of compromising Service A
is high if you do not limit the resources of Service B. The following table lists
areas of security concerns and recommendations for securing workloads running in Kubernetes:
Area of Concern for Workload Security
RBAC Authorization (Access to the Kubernetes API)
https://kubernetes.io/docs/reference/access-authn-authz/rbac/
https://kubernetes.io/docs/concepts/security/controlling-access/
Application secrets management (and encrypting them in etcd at rest)
https://kubernetes.io/docs/concepts/configuration/secret/
https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/
Ensuring that pods meet defined Pod Security Standards
https://kubernetes.io/docs/concepts/security/pod-security-standards/#policy-instantiation
Qual

ity of Service (and Cluster resource management)
https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/
https://kubernetes.io/docs/concepts/services-networking/network-policies/
TLS for Kubernetes Ingress
https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
Container
Container security is outside the scope of this guide. Here are general recommendations and
links to explore this topic:
Area of Concern for Containers
Container Vulnerability Scanning and OS Dependency Security
As part of an image build step, you should scan your containers for known vulnerabilities.
Image Signing and Enforcement
Sign container images to maintain a system of trust for the content of your containers.
Disallow privileged users
When constructing containers, consult your documentation for how to create users inside of the containers that have the least level of operating system privilege necessary in order to carry out the goal of the container.
Use container runtime with stronger isolation
Select
container runtime classes
that provide stronger isolation.
Code
Application code is one of the primary attack surfaces over which you have the most control.
While securing application code is outside of the Kubernetes security topic, here
are recommendations to protect application code:
Code security
Area of Concern for Code
Access over TLS only
If your code needs to communicate by TCP, perform a TLS handshake with the client ahead of time. With the exception of a few cases, encrypt everything in transit. Going one step further, it's a good idea to encrypt network traffic between services. This can be done through a process known as mutual TLS authentication or
mTLS
which performs a two sided verification of communication between two certificate holding services.
Limiting port ranges of communication
This recommendation may be a bit self-explanatory, but wherever possible you should only expose the ports on your service that are absolutely essential for communication or metric gathering.
3rd Party Dependency Security
It is a good practice to regularly scan your application's third party libraries for known security vulnerabilities. Each programming language has a tool for performing this check automatically.
Static Code Analysis
Most languages provide a way for a snippet of code to be analyzed for any potentially unsafe coding practices. Whenever possible you should perform checks using automated tooling that can scan codebases for common security errors. Some of the tools can be found at:
https://owasp.org/www-community/Source_Code_Analysis_Tools
Dynamic probing attacks
There are a few automated tools that you can run against your service to try some of the well known service attacks. These include SQL injection, CSRF, and XSS. One of the most popular dynamic analysis tools is the
OWASP Zed Attack proxy
tool.
Learn about related Kubernetes security topics:
Network policies for Pods
Securing your cluster
Data encryption in transit
for the control plane
Data encryption at rest
Secrets in Kubernetes
Runtime class
9.2 - Pod Security Standards
A detailed look at the different policy levels defined in the Pod Security Standards.
The Pod Security Standards define three different
policies
to broadly cover the security
spectrum. These policies are
cumulative
and range from highly-permissive to highly-restrictive.
This guide outlines the requirements of each policy.
Profile
Unrestricted policy, providing the widest possible level of permissions. This policy allows for known privilege escalations.
Baseline
Minimally restrictive policy which prevents known privilege escalations. Allows the default (minimally specified) Pod configuration.
Heavily restricted policy, following current Pod hardening best practices.
Profile Details
The
policy is

 purposely-open, and entirely unrestricted.
This type of policy is
typically aimed at system- and infrastructure-level workloads managed by privileged, trusted users.
The Privileged policy is defined by an absence of restrictions. Allow-by-default
mechanisms (such as gatekeeper) may be Privileged by default. In contrast, for a deny-by-default mechanism (such as Pod
Security Policy) the Privileged policy should disable all restrictions.
Baseline
The
Baseline
policy is aimed at ease of adoption for common containerized workloads while
preventing known privilege escalations.
This policy is targeted at application operators and
developers of non-critical applications. The following listed controls should be
enforced/disallowed:
Note:
In this table, wildcards (
*
) indicate all elements in a list. For example,
refers to the Security Context object for
all defined
. If any of the listed containers fails to meet the requirements, the entire pod will
fail validation.
Baseline policy specification
Control
Policy
HostProcess
which enables privileged access to the Windows node. Privileged access to the host is disallowed in the baseline policy.
Restricted Fields
spec.securityContext.windowsOptions.hostProcess
spec.containers[*].securityContext.windowsOptions.hostProcess
spec.initContainers[*].securityContext.windowsOptions.hostProcess
spec.ephemeralContainers[*].securityContext.windowsOptions.hostProcess
Undefined/nil
false
Host Namespaces
Sharing the host namespaces must be disallowed.
spec.hostNetwork
false
Privileged Containers
Privileged Pods disable most security mechanisms and must be disallowed.
spec.initContainers[*].securityContext.privileged
spec.ephemeralContainers[*].securityContext.privileged
false
Adding additional capabilities beyond those listed below must be disallowed.
spec.containers[*].securityContext.capabilities.add
spec.initContainers[*].securityContext.capabilities.add
spec.ephemeralContainers[*].securityContext.capabilities.add
CHOWN
FOWNER
FSETID
KILL
MKNOD
SETFCAP
SETGID
SETPCAP
SETUID
HostPath Volumes
HostPath volumes must be forbidden.
spec.volumes[*].hostPath
Host Ports
HostPorts should be disallowed, or at minimum restricted to a known list.
spec.containers[*].ports[*].hostPort
spec.initContainers[*].ports[*].hostPort
spec.ephemeralContainers[*].ports[*].hostPort
Known list
0
AppArmor
On supported hosts, the
AppArmor profile is applied by default. The baseline policy should prevent overriding or disabling the default AppArmor profile, or restrict overrides to an allowed set of profiles.
metadata.annotations["container.apparmor.security.beta.kubernetes.io/*"]
SELinux
Setting the SELinux type is restricted, and setting a custom SELinux user or role option is forbidden.
spec.securityContext.seLinuxOptions.type
spec.containers[*].securityContext.seLinuxOptions.type
spec.initContainers[*].securityContext.seLinuxOptions.type
spec.ephemeralContainers[*].securityContext.seLinuxOptions.type
Undefined/""
container_t
container_init_t
container_kvm_t
spec.securityContext.seLinuxOptions.user
spec.containers[*].securityContext.seLinuxOptions.user
spec.initContainers[*].securityContext.seLinuxOptions.user
spec.ephemeralContainers[*].securityContext.seLinuxOptions.user
spec.securityContext.seLinuxOptions.role
spec.containers[*].securityContext.seLinuxOptions.r

ole
spec.initContainers[*].securityContext.seLinuxOptions.role
spec.ephemeralContainers[*].securityContext.seLinuxOptions.role
/proc
Mount Type
/proc
masks are set up to reduce attack surface, and should be required.
spec.initContainers[*].securityContext.procMount
spec.ephemeralContainers[*].securityContext.procMount
Default
Seccomp
Seccomp profile must not be explicitly set to
.
spec.securityContext.seccompProfile.type
spec.containers[*].securityContext.seccompProfile.type
spec.initContainers[*].securityContext.seccompProfile.type
spec.ephemeralContainers[*].securityContext.seccompProfile.type
Localhost
Sysctls
Sysctls can disable security mechanisms or affect all containers on a host, and should be disallowed except for an allowed "safe" subset. A sysctl is considered safe if it is namespaced in the container or the Pod, and it is isolated from other Pods or processes on the same Node.
spec.securityContext.sysctls[*].name
The
policy is aimed at enforcing current Pod hardening best practices, at the
expense of some compatibility.
It is targeted at operators and developers of security-critical
applications, as well as lower-trust users. The following listed controls should be
Note:
*
Restricted policy specification
Control
Policy
Everything from the baseline profile.
Volume Types
The restricted policy only permits the following volume types.
spec.volumes[*]
Every item in the
list must set one of the following fields to a non-null value:
spec.volumes[*].configMap
spec.volumes[*].csi
spec.volumes[*].downwardAPI
spec.volumes[*].emptyDir
spec.volumes[*].ephemeral
spec.volumes[*].persistentVolumeClaim
spec.volumes[*].projected
spec.volumes[*].secret
Privilege Escalation (v1.8+)
Privilege escalation (such as via set-user-ID or set-group-ID file mode) should not be allowed.
This is Linux only policy
in v1.25+
(spec.os.name != windows)
spec.initContainers[*].securityContext.allowPrivilegeEscalation
spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation
false
Running as Non-root
Containers must be required to run as non-root users.
spec.securityContext.runAsNonRoot
spec.containers[*].securityContext.runAsNonRoot
spec.initContainers[*].securityContext.runAsNonRoot
spec.ephemeralContainers[*].securityContext.runAsNonRoot
true
The container fields may be undefined/
nil
if the pod-level
is set to
true
.
Running as Non-root user (v1.23+)
Containers must not set
runAsUser
to 0
spec.initContainers[*].securityContext.runAsUser
spec.ephemeralContainers[*].securityContext.runAsUser
any non-zero value
undefined/null
Seccomp (v1.19+)
Seccomp profile must be explicitly set to one of the allowed values. Both the
profile and the
absence
of a profile are prohibited.
in v1.25+
spec.ephemeralContai

ners[*].securityContext.seccompProfile.type
Localhost
nil
field is set appropriately.
Conversely, the pod-level field may be undefined/
nil
if _all_ container-
level fields are set.
Capabilities (v1.22+)
Containers must drop
ALL
capabilities, and are only permitted to add back
the
in v1.25+
(.spec.os.name != "windows")
spec.containers[*].securityContext.capabilities.drop
spec.initContainers[*].securityContext.capabilities.drop
spec.ephemeralContainers[*].securityContext.capabilities.drop
Any list of capabilities that includes
ALL
Policy Instantiation
Decoupling policy definition from policy instantiation allows for a common understanding and
consistent language of policies across clusters, independent of the underlying enforcement
As mechanisms mature, they will be defined below on a per-policy basis. The methods of enforcement
of individual policies are not defined here.
Privileged namespace
Baseline namespace
Restricted namespace
Note:
Other alternatives for enforcing policies are being developed in the Kubernetes ecosystem, such as:
Kyverno
Pod OS field
Kubernetes lets you use nodes that run either Linux or Windows. You can mix both kinds of
node in one cluster.
Windows in Kubernetes has some limitations and differentiators from Linux-based
workloads. Specifically, many of the Pod
fields
have no effect on Windows
.
Note:
Kubelets prior to v1.24 don't enforce the pod OS field, and if a cluster has nodes on versions earlier than v1.24 the restricted policies should be pinned to a version prior to v1.25.
Restricted Pod Security Standard changes
Another important change, made in Kubernetes v1.25 is that the
Pod security
has been updated to use the
field. Based on the OS name, certain policies that are specific
to a particular OS can be relaxed for the other OS.
OS-specific policy controls
Restrictions on the following controls are only required if
is not
windows
:
Privilege Escalation
Seccomp
FAQ
Why isn't there a profile between privileged and baseline?
The three profiles defined here have a clear linear progression from most secure (restricted) to least
secure (privileged), and cover a broad set of workloads. Privileges required above the baseline
policy are typically very application specific, so we do not offer a standard profile in this
niche. This is not to say that the privileged profile should always be used in this case, but that
policies in this space need to be defined on a case-by-case basis.
SIG Auth may reconsider this position in the future, should a clear need for other profiles arise.
What's the difference between a security profile and a security context?
Security Contexts
configure Pods and
Containers at runtime. Security contexts are defined as part of the Pod and container specifications
in the Pod manifest, and represent parameters to the container runtime.
Security profiles are control plane mechanisms to enforce specific settings in the Security Context,
as well as other related parameters outside the Security Context. As of July 2021,
are deprecated in favor of the
built-in
Pod Security Admiss

ion Controller
.
What about sandboxed Pods?
There is not currently an API standard that controls whether a Pod is considered sandboxed or
not. Sandbox Pods may be identified by the use of a sandboxed runtime (such as gVisor or Kata
Containers), but there is no standard definition of what a sandboxed runtime is.
The protections necessary for sandboxed workloads can differ from others. For example, the need to
restrict privileged permissions is lessened when the workload is isolated from the underlying
kernel. This allows for workloads requiring heightened permissions to still be isolated.
Additionally, the protection of sandboxed workloads is highly dependent on the method of
sandboxing. As such, no single recommended profile is recommended for all sandboxed workloads.
9.3 - Pod Security Admission
An overview of the Pod Security Admission Controller, which can enforce the Pod Security Standards.
define
different isolation levels for Pods. These standards let you define how you want to restrict the
behavior of pods in a clear, consistent fashion.
Kubernetes offers a built-in
to enforce the Pod Security Standards. Pod security restrictions
are applied at the
namespace
level when pods are
created.
Built-in Pod Security admission enforcement
Pod Security levels
Pod Security admission places requirements on a Pod's
Security
Context
and other related fields according
to the three levels defined by the
Standards
:
,
baseline
, and
. Refer to the
page for an in-depth look at those requirements.
Pod Security Admission labels for namespaces
Once the feature is enabled or the webhook is installed, you can configure namespaces to define the admission
control mode you want to use for pod security in each namespace. Kubernetes defines a set of
labels
that you can set to define which of the
predefined Pod Security Standard levels you want to use for a namespace. The label you select
defines what action the
takes if a potential violation is detected:
Pod Security Admission modes
Mode
enforce
Policy violations will cause the pod to be rejected.
audit
Policy violations will trigger the addition of an audit annotation to the event recorded in the
audit log
, but are otherwise allowed.
warn
Policy violations will trigger a user-facing warning, but are otherwise allowed.
A namespace can configure any or all modes, or even set a different level for different modes.
For each mode, there are two labels that determine the policy used:
# The per-mode level label indicates which policy level to apply for the mode.
#
# MODE must be one of `enforce`, `audit`, or `warn`.
# LEVEL must be one of `privileged`, `baseline`, or `restricted`.
pod-security.kubernetes.io/&lt;MODE&gt;
:
&lt;LEVEL&gt;
# Optional: per-mode version label that can be used to pin the policy to the
# version that shipped with a given Kubernetes minor version (for example v1.26).
#
# VERSION must be a valid Kubernetes minor version, or `latest`.
pod-security.kubernetes.io/&lt;MODE&gt;-version
:
&lt;VERSION&gt;
Check out
to see example usage.
Workload resources and Pod templates
Pods are often created indirectly, by creating a
workload
object
such as a
or
Job
. The workload object defines a
and a
for the
workload resource creates Pods based on that template. To help catch violations early, both the
audit and warning modes are applied to the workload resources. However, enforce mode is
not
applied to workload resources, only to the resulting pod objects.
Exemptions
You can define
from pod secur

ity enforcement in order to allow the creation of pods that
would have otherwise been prohibited due to the policy associated with a given namespace.
Exemptions can be statically configured in the
Admission Controller configuration
.
Exemptions must be explicitly enumerated. Requests meeting exemption criteria are
ignored
by the
Admission Controller (all
enforce
,
audit
and
warn
behaviors are skipped). Exemption dimensions include:
Usernames:
requests from users with an exempt authenticated (or impersonated) username are
ignored.
RuntimeClassNames:
pods and
specifying an exempt runtime class name are
ignored.
Namespaces:
pods and
in an exempt namespace are ignored.
Caution:
Most pods are created by a controller in response to a
workload
resource
, meaning that exempting an end user will only
exempt them from enforcement when creating pods directly, but not when creating a workload resource.
Controller service accounts (such as
system:serviceaccount:kube-system:replicaset-controller
)
should generally not be exempted, as doing so would implicitly exempt any user that can create the
corresponding workload resource.
Updates to the following pod fields are exempt from policy checks, meaning that if a pod update
request only changes these fields, it will not be denied even if the pod is in violation of the
current policy level:
Any metadata updates
except
changes to the seccomp or AppArmor annotations:
container.seccomp.security.alpha.kubernetes.io/*
container.apparmor.security.beta.kubernetes.io/*
Valid updates to
.spec.tolerations
9.4 - Pod Security Policies
Removed feature
PodSecurityPolicy was
in Kubernetes v1.21, and removed from Kubernetes in v1.25.
Instead of using PodSecurityPolicy, you can enforce similar restrictions on Pods using
either or both:
a 3rd party admission plugin, that you deploy and configure yourself
.
For more information on the removal of this API,
see
.
your version of Kubernetes.
9.5 - Security For Windows Nodes
This page describes security considerations and best practices specific to the Windows operating system.
Protection for Secret data on nodes
On Windows, data from Secrets are written out in clear text onto the node's local
storage (as compared to using tmpfs / in-memory filesystems on Linux). As a cluster
operator, you should take both of the following additional measures:
Use file ACLs to secure the Secrets' file location.
Apply volume-level encryption using
BitLocker
.
Container users
can be specified for Windows Pods or containers to execute the container
processes as specific user. This is roughly equivalent to
RunAsUser
.
Windows containers offer two default user accounts, ContainerUser and ContainerAdministrator.
The differences between these two user accounts are covered in
When to use ContainerAdmin and ContainerUser user accounts
within Microsoft's
Secure Windows containers
Local users can be added to container images during the container build process.
Note:
Nano Server
based images run as
by default
Server Core
by default
Windows containers can also run as Active Directory identities by utilizing
Pod-level security isolation
Linux-specific pod security context mechanisms (such as SELinu

x, AppArmor, Seccomp, or custom
POSIX capabilities) are not supported on Windows nodes.
Privileged containers are
not supported
on Windows.
Instead
can be used on Windows to perform many of the tasks performed by privileged containers on Linux.
9.6 - Controlling Access to the Kubernetes API
This page provides an overview of controlling access to the Kubernetes API.
Users access the
using
kubectl
,
client libraries, or by making REST requests. Both human users and
Kubernetes service accounts
can be
authorized for API access.
When a request reaches the API, it goes through several stages, illustrated in the
following diagram:
Transport security
By default, the Kubernetes API server listens on port 6443 on the first non-localhost network interface, protected by TLS. In a typical production Kubernetes cluster, the API serves on port 443. The port can be changed with the
--secure-port
, and the listening IP address with the
flag.
The API server presents a certificate. This certificate may be signed using
a private certificate authority (CA), or based on a public key infrastructure linked
to a generally recognized CA. The certificate and corresponding private key can be set by using the
and
flags.
If your cluster uses a private certificate authority, you need a copy of that CA
certificate configured into your
on the client, so that you can
trust the connection and be confident it was not intercepted.
Your client can present a TLS client certificate at this stage.
Once TLS is established, the HTTP request moves to the Authentication step.
This is shown as step
1
in the diagram.
The cluster creation script or cluster admin configures the API server to run
one or more Authenticator modules.
Authenticators are described in more detail in
.
The input to the authentication step is the entire HTTP request; however, it typically
examines the headers and/or client certificate.
Authentication modules include client certificates, password, and plain tokens,
bootstrap tokens, and JSON Web Tokens (used for service accounts).
Multiple authentication modules can be specified, in which case each one is tried in sequence,
until one of them succeeds.
If the request cannot be authenticated, it is rejected with HTTP status code 401.
Otherwise, the user is authenticated as a specific
username
, and the user name
is available to subsequent steps to use in their decisions. Some authenticators
also provide the group memberships of the user, while other authenticators
do not.
While Kubernetes uses usernames for access control decisions and in request logging,
it does not have a
User
object nor does it store usernames or other information about
users in its API.
After the request is authenticated as coming from a specific user, the request must be authorized. This is shown as step
2
A request must include the username of the requester, the requested action, and the object affected by the action. The request is authorized if an existing policy declares that the user has permissions to complete the requested action.
For example, if Bob has the policy below, then he can read pods only in the namespace
projectCaribou
:
{
:
,
"kind"
:
"Policy"
,
"spec"
: {
"user"
:
"bob"
,
:
,
:
"pods"
,
:
true
}
}
If Bob makes the following request, the request is authorized because he is allowed to read objects in the
{
:
,
"kind"
:
,
"spec"
: {
: {
:
,
"verb"
:
"get"
,
"group"
:
,
:
"pods"
}
}
}
If Bob makes a request to write (
create
or
update
) to the objects in the
namespace, his authorization is denied. If Bob makes 

a request to read (
get
) objects in a different namespace such as
projectFish
, then his authorization is denied.
Kubernetes authorization requires that you use common REST attributes to interact with existing organization-wide or cloud-provider-wide access control systems. It is important to use REST formatting because these control systems might interact with other APIs besides the Kubernetes API.
Kubernetes supports multiple authorization modules, such as ABAC mode, RBAC Mode, and Webhook mode. When an administrator creates a cluster, they configure the authorization modules that should be used in the API server. If more than one authorization modules are configured, Kubernetes checks each module, and if any module authorizes the request, then the request can proceed. If all of the modules deny the request, then the request is denied (HTTP status code 403).
To learn more about Kubernetes authorization, including details about creating policies using the supported authorization modules, see
.
Admission control
Admission Control modules are software modules that can modify or reject requests.
In addition to the attributes available to Authorization modules, Admission
Control modules can access the contents of the object that is being created or modified.
Admission controllers act on requests that create, modify, delete, or connect to (proxy) an object.
Admission controllers do not act on requests that merely read objects.
When multiple admission controllers are configured, they are called in order.
3
Unlike Authentication and Authorization modules, if any admission controller module
rejects, then the request is immediately rejected.
In addition to rejecting objects, admission controllers can also set complex defaults for
fields.
The available Admission Control modules are described in
.
Once a request passes all admission controllers, it is validated using the validation routines
for the corresponding API object, and then written to the object store (shown as step
4
).
Auditing
Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster.
The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself.
Auditing
.
Read more documentation on authentication, authorization and API access control:
including
and
You can learn about:
how Pods can use
Secrets
to obtain API credentials.
9.7 - Role Based Access Control Good Practices
Principles and practices for good RBAC design for cluster operators.
RBAC
is a key security control
to ensure that cluster users and workloads have only the access to resources required to
execute their roles. It is important to ensure that, when designing permissions for cluster
users, the cluster administrator understands the areas where privilege escalation could occur,
to reduce the risk of excessive access leading to security incidents.
The good practices laid out here should be read in conjunction with the general
RBAC documentation
.
General good practice
Least privilege
Ideally, minimal RBAC rights should be assigned to users and service accounts. Only permissions
explicitly required for their operation should be used. While each cluster will be different,
some general rules that can be applied are :
Assign permissions at the namespace level where possible. Use RoleBindings as opposed to
ClusterRoleBindings to give users rights only within a specific namespace.
Avoid providing wildcard permissions when possible, especially to all

 resources.
As Kubernetes is an extensible system, providing wildcard access gives rights
not just to all object types that currently exist in the cluster, but also to all object types
which are created in the future.
Administrators should not use
accounts except where specifically needed.
Providing a low privileged account with
impersonation rights
can avoid accidental modification of cluster resources.
Avoid adding users to the
group. Any user who is a member of this group
bypasses all RBAC rights checks and will always have unrestricted superuser access, which cannot be
revoked by removing RoleBindings or ClusterRoleBindings. As an aside, if a cluster is
using an authorization webhook, membership of this group also bypasses that webhook (requests
from users who are members of that group are never sent to the webhook)
Minimize distribution of privileged tokens
Ideally, pods shouldn't be assigned service accounts that have been granted powerful permissions
(for example, any of the rights listed under
privilege escalation risks
).
In cases where a workload requires powerful permissions, consider the following practices:
Limit the number of nodes running powerful pods. Ensure that any DaemonSets you run
are necessary and are run with least privilege to limit the blast radius of container escapes.
Avoid running powerful pods alongside untrusted or publicly-exposed ones. Consider using
Taints and Toleration
,
, or
to ensure pods don't run alongside untrusted or less-trusted Pods. Pay especial attention to
situations where less-trustworthy Pods are not meeting the
Hardening
Kubernetes defaults to providing access which may not be required in every cluster. Reviewing
the RBAC rights provided by default can provide opportunities for security hardening.
In general, changes should not be made to rights provided to
system:
accounts some options
to harden cluster rights exist:
Review bindings for the
group and remove them where possible, as this gives
access to anyone who can contact the API server at a network level.
Avoid the default auto-mounting of service account tokens by setting
. For more details, see
using default service account token
.
Setting this value for a Pod will overwrite the service account setting, workloads
which require service account tokens can still mount them.
Periodic review
It is vital to periodically review the Kubernetes RBAC settings for redundant entries and
possible privilege escalations.
If an attacker is able to create a user account with the same name as a deleted user,
they can automatically inherit all the rights of the deleted user, especially the
rights assigned to that user.
Kubernetes RBAC - privilege escalation risks
Within Kubernetes RBAC there are a number of privileges which, if granted, can allow a user or a service account
to escalate their privileges in the cluster or affect systems outside the cluster.
This section is intended to provide visibility of the areas where cluster operators
should take care, to ensure that they do not inadvertently allow for more access to clusters than intended.
Listing secrets
It is generally clear that allowing
get
access on Secrets will allow a user to read their contents.
It is also important to note that
list
and
watch
access also effectively allow for users to reveal the Secret contents.
For example, when a List response is returned (for example, via
kubectl get secrets -A -o yaml
), the response
includes the contents of all Secrets.
Workload creation
Permission to create workloads (either Pods, or
that manage Pods) in a namespace
implicitly grants access to many other resources in that namespace, such as Secrets, ConfigMaps, and
PersistentVolumes that can be mounted in Pods. Additionally, since Pods can run as any
, granting permission
to create workloads also implicitly

 grants the API access levels of any service account in that
Users who can run privileged Pods can use that access to gain node access and potentially to
further elevate their privileges. Where you do not fully trust a user or other principal
with the ability to create suitably secure and isolated Pods, you should enforce either the
Baseline
or
You can use
or other (third party) mechanisms to implement that enforcement.
For these reasons, namespaces should be used to separate resources requiring different levels of
trust or tenancy. It is still considered best practice to follow
least privilege
principles and assign the minimum set of permissions, but boundaries within a namespace should be
considered weak.
Persistent volume creation
documentation, access to create PersistentVolumes can allow for escalation of access to the underlying host.
Where access to persistent storage is required trusted administrators should create
PersistentVolumes, and constrained users should use PersistentVolumeClaims to access that storage.
Access to
proxy
subresource of Nodes
Users with access to the proxy sub-resource of node objects have rights to the Kubelet API,
which allows for command execution on every pod on the node(s) to which they have rights.
This access bypasses audit logging and admission control, so care should be taken before
granting rights to this resource.
Escalate verb
Generally, the RBAC system prevents users from creating clusterroles with more rights than the user possesses.
The exception to this is the
escalate
verb. As noted in the
,
users with this right can effectively escalate their privileges.
Bind verb
escalate
verb, granting users this right allows for the bypass of Kubernetes
in-built protections against privilege escalation, allowing users to create bindings to
roles with rights they do not already have.
Impersonate verb
This verb allows users to impersonate and gain the rights of other users in the cluster.
Care should be taken when granting it, to ensure that excessive permissions cannot be gained
via one of the impersonated accounts.
CSRs and certificate issuing
The CSR API allows for users with
create
rights to CSRs and
update
rights on
where the signer is
to create new client certificates
which allow users to authenticate to the cluster. Those client certificates can have arbitrary
names including duplicates of Kubernetes system components. This will effectively allow for privilege escalation.
Token request
Users with
create
rights on
serviceaccounts/token
can create TokenRequests to issue
tokens for existing service accounts.
Control admission webhooks
Users with control over
or
can control webhooks that can read any object admitted to the cluster, and in the case of
mutating webhooks, also mutate admitted objects.
Kubernetes RBAC - denial of service risks
Object creation denial-of-service
Users who have rights to create objects in a cluster may be able to create sufficient large
objects to create a denial of service condition either based on the size or number of objects, as discussed in
etcd used by Kubernetes is vulnerable to OOM attack
specifically relevant in multi-tenant clusters if semi-trusted or untrusted users
are allowed limited access to a system.
One option for mitigation of this issue would be to use
to limit the quantity of objects which can be created.
To learn more about RBAC, see the
.
9.8 - Good practices for Kubernetes Secrets
Principles and practices for good Secret management for cluster administrators and application developers.
In Kubernetes, a Secret is an object that stores sensitive information, such as passwords, OAuth tokens, and SSH keys.
Secrets give yo

u more control over how sensitive information is used and reduces
.
A
Pod
are
The following good practices are intended for both cluster administrators and
application developers. Use these guidelines to improve the security of your
sensitive information in Secret objects, as well as to more effectively manage
your Secrets.
Cluster administrators
This section provides good practices that cluster administrators can use to
improve the security of confidential information in the cluster.
Configure encryption at rest
By default, Secret objects are stored unencrypted in
etcd
. You should configure encryption of your Secret
data in
etcd
. For instructions, refer to
Encrypt Secret Data at Rest
.
Configure least-privilege access to Secrets
When planning your access control mechanism, such as Kubernetes
Role-based Access Control
(RBAC)
,
consider the following guidelines for access to
Secret
objects. You should
also follow the other guidelines in
RBAC good practices
.
: Restrict
watch
or
list
access to only the most
privileged, system-level components. Only grant
get
access for Secrets if
the component's normal behavior requires it.
Humans
: Restrict
get
,
watch
, or
list
access to Secrets. Only allow
cluster administrators to access
etcd
. This includes read-only access. For
more complex access control, such as restricting access to Secrets with
specific annotations, consider using third-party authorization mechanisms.
Caution:
Granting
list
access to Secrets implicitly lets the subject fetch the
contents of the Secrets.
A user who can create a Pod that uses a Secret can also see the value of that
Secret. Even if cluster policies do not allow a user to read the Secret
directly, the same user could have access to run a Pod that then exposes the
Secret. You can detect or limit the impact caused by Secret data being exposed,
either intentionally or unintentionally, by a user with this access. Some
recommendations include:
Use short-lived Secrets
Implement audit rules that alert on specific events, such as concurrent
reading of multiple Secrets by a single user
Improve etcd management policies
Consider wiping or shredding the durable storage used by
etcd
once it is
no longer in use.
If there are multiple
etcd
instances, configure encrypted SSL/TLS
communication between the instances to protect the Secret data in transit.
Configure access to external Secrets
Note:
You can use third-party Secrets store providers to keep your confidential data
outside your cluster and then configure Pods to access that information.
The
Kubernetes Secrets Store CSI Driver
is a DaemonSet that lets the kubelet retrieve Secrets from external stores, and
mount the Secrets as a volume into specific Pods that you authorize to access
the data.
For a list of supported providers, refer to
Providers for the Secret Store CSI Driver
.
Developers
This section provides good practices for developers to use to improve the
security of confidential data when building and deploying Kubernetes resources.
Restrict Secret access to specific containers
If you are defining multiple containers in a Pod, and only one of those
containers needs access to a Secret, define the volume mount or environment
variable configuration so that the other containers do not have access to that
Secret.
Protect Secret data after reading
Applicatio

ns still need to protect the value of confidential information after
reading it from an environment variable or volume. For example, your
application must avoid logging the secret data in the clear or transmitting it
to an untrusted party.
Avoid sharing Secret manifests
If you configure a Secret through a
manifest
, with the secret
data encoded as base64, sharing this file or checking it in to a source
repository means the secret is available to everyone who can read the manifest.
Caution:
Base64 encoding is
not
an encryption method, it provides no additional
confidentiality over plain text.
9.9 - Multi-tenancy
This page provides an overview of available configuration options and best practices for cluster
multi-tenancy.
Sharing clusters saves costs and simplifies administration. However, sharing clusters also
presents challenges such as security, fairness, and managing
noisy neighbors
.
Clusters can be shared in many ways. In some cases, different applications may run in the same
cluster. In other cases, multiple instances of the same application may run in the same cluster,
one for each end user. All these types of sharing are frequently described using the umbrella term
multi-tenancy
.
While Kubernetes does not have first-class concepts of end users or tenants, it provides several
features to help manage different tenancy requirements. These are discussed below.
Use cases
The first step to determining how to share your cluster is understanding your use case, so you can
evaluate the patterns and tools available. In general, multi-tenancy in Kubernetes clusters falls
into two broad categories, though many variations and hybrids are also possible.
Multiple teams
A common form of multi-tenancy is to share a cluster between multiple teams within an
organization, each of whom may operate one or more workloads. These workloads frequently need to
communicate with each other, and with other workloads located on the same or different clusters.
In this scenario, members of the teams often have direct access to Kubernetes resources via tools
such as
kubectl
, or indirect access through GitOps controllers or other types of release
automation tools. There is often some level of trust between members of different teams, but
Kubernetes policies such as RBAC, quotas, and network policies are essential to safely and fairly
share clusters.
Multiple customers
The other major form of multi-tenancy frequently involves a Software-as-a-Service (SaaS) vendor
running multiple instances of a workload for customers. This business model is so strongly
associated with this deployment style that many people call it "SaaS tenancy." However, a better
term might be "multi-customer tenancy," since SaaS vendors may also use other deployment models,
and this deployment model can also be used outside of SaaS.
In this scenario, the customers do not have access to the cluster; Kubernetes is invisible from
their perspective and is only used by the vendor to manage the workloads. Cost optimization is
frequently a critical concern, and Kubernetes policies are used to ensure that the workloads are
strongly isolated from each other.
Tenants
When discussing multi-tenancy in Kubernetes, there is no single definition for a "tenant".
Rather, the definition of a tenant will vary depending on whether multi-team or multi-customer
tenancy is being discussed.
In multi-team usage, a tenant is typically a team, where each team typically deploys a small
number of workloads that scales with the complexity of the service. However, the definition of
"team" may itself be fuzzy, as teams may be organized into higher-level divisions or subdivided
into smaller teams.
By contrast, if each team deploys dedicated workloads for each new client, they are using a
multi-customer model of tenancy. In this case, a "tenant" is simply a group of users who share a
single workload. This may be as large as an entire company, or as small as a single team at that
company.
In many cases, the

 same organization may use both definitions of "tenants" in different contexts.
For example, a platform team may offer shared services such as security tools and databases to
multiple internal “customers” and a SaaS vendor may also have multiple teams sharing a development
cluster. Finally, hybrid architectures are also possible, such as a SaaS provider using a
combination of per-customer workloads for sensitive data, combined with multi-tenant shared
services.
A cluster showing coexisting tenancy models
Isolation
There are several ways to design and build multi-tenant solutions with Kubernetes. Each of these
methods comes with its own set of tradeoffs that impact the isolation level, implementation
effort, operational complexity, and cost of service.
A Kubernetes cluster consists of a control plane which runs Kubernetes software, and a data plane
consisting of worker nodes where tenant workloads are executed as pods. Tenant isolation can be
applied in both the control plane and the data plane based on organizational requirements.
The level of isolation offered is sometimes described using terms like “hard” multi-tenancy, which
implies strong isolation, and “soft” multi-tenancy, which implies weaker isolation. In particular,
"hard" multi-tenancy is often used to describe cases where the tenants do not trust each other,
often from security and resource sharing perspectives (e.g. guarding against attacks such as data
exfiltration or DoS). Since data planes typically have much larger attack surfaces, "hard"
multi-tenancy often requires extra attention to isolating the data-plane, though control plane
isolation also remains critical.
However, the terms "hard" and "soft" can often be confusing, as there is no single definition that
will apply to all users. Rather, "hardness" or "softness" is better understood as a broad
spectrum, with many different techniques that can be used to maintain different types of isolation
in your clusters, based on your requirements.
In more extreme cases, it may be easier or necessary to forgo any cluster-level sharing at all and
assign each tenant their dedicated cluster, possibly even running on dedicated hardware if VMs are
not considered an adequate security boundary. This may be easier with managed Kubernetes clusters,
where the overhead of creating and operating clusters is at least somewhat taken on by a cloud
provider. The benefit of stronger tenant isolation must be evaluated against the cost and
complexity of managing multiple clusters. The
Multi-cluster SIG
is responsible for addressing these types of use cases.
The remainder of this page focuses on isolation techniques used for shared Kubernetes clusters.
However, even if you are considering dedicated clusters, it may be valuable to review these
recommendations, as it will give you the flexibility to shift to shared clusters in the future if
your needs or capabilities change.
Control plane isolation
Control plane isolation ensures that different tenants cannot access or affect each others'
Kubernetes API resources.
Namespace
provides a
mechanism for isolating groups of API resources within a single cluster. This isolation has two
key dimensions:
Object names within a namespace can overlap with names in other namespaces, similar to files in
folders. This allows tenants to name their resources without having to consider what other
tenants are doing.
Many Kubernetes security policies are scoped to namespaces. For example, RBAC Roles and Network
Policies are namespace-scoped resources. Using RBAC, Users and Service Accounts can be
restricted to a namespace.
In a multi-tenant environment, a Namespace helps segment a tenant's workload into a logical and
distinct management unit. In fact, a common practice is to isolate every workload in its own
namespace, even if multiple workloads are operated by the same tenant. This ensures that each
workload has its own identity and can be configured with an appropriate security po

licy.
The namespace isolation model requires configuration of several other Kubernetes resources,
networking plugins, and adherence to security best practices to properly isolate tenant workloads.
These considerations are discussed below.
Access controls
The most important type of isolation for the control plane is authorization. If teams or their
workloads can access or modify each others' API resources, they can change or disable all other
types of policies thereby negating any protection those policies may offer. As a result, it is
critical to ensure that each tenant has the appropriate access to only the namespaces they need,
and no more. This is known as the "Principle of Least Privilege."
Role-based access control (RBAC) is commonly used to enforce authorization in the Kubernetes
control plane, for both users and workloads (service accounts).
Roles
and
RoleBindings
are
Kubernetes objects that are used at a namespace level to enforce access control in your
application; similar objects exist for authorizing access to cluster-level objects, though these
are less useful for multi-tenant clusters.
In a multi-team environment, RBAC must be used to restrict tenants' access to the appropriate
namespaces, and ensure that cluster-wide resources can only be accessed or modified by privileged
users such as cluster administrators.
If a policy ends up granting a user more permissions than they need, this is likely a signal that
the namespace containing the affected resources should be refactored into finer-grained
namespaces. Namespace management tools may simplify the management of these finer-grained
namespaces by applying common RBAC policies to different namespaces, while still allowing
fine-grained policies where necessary.
Quotas
Kubernetes workloads consume node resources, like CPU and memory. In a multi-tenant environment,
you can use
to manage resource usage of
tenant workloads. For the multiple teams use case, where tenants have access to the Kubernetes
API, you can use resource quotas to limit the number of API resources (for example: the number of
Pods, or the number of ConfigMaps) that a tenant can create. Limits on object count ensure
fairness and aim to avoid
noisy neighbor
issues from affecting other tenants that share a
Resource quotas are namespaced objects. By mapping tenants to namespaces, cluster admins can use
quotas to ensure that a tenant cannot monopolize a cluster's resources or overwhelm its control
plane. Namespace management tools simplify the administration of quotas. In addition, while
Kubernetes quotas only apply within a single namespace, some namespace management tools allow
groups of namespaces to share quotas, giving administrators far more flexibility with less effort
than built-in quotas.
Quotas prevent a single tenant from consuming greater than their allocated share of resources
hence minimizing the “noisy neighbor” issue, where one tenant negatively impacts the performance
of other tenants' workloads.
When you apply a quota to namespace, Kubernetes requires you to also specify resource requests and
limits for each container. Limits are the upper bound for the amount of resources that a container
can consume. Containers that attempt to consume resources that exceed the configured limits will
either be throttled or killed, based on the resource type. When resource requests are set lower
than limits, each container is guaranteed the requested amount but there may still be some
potential for impact across workloads.
Quotas cannot protect against all kinds of resource sharing, such as network traffic.
Node isolation (described below) may be a better solution for this problem.
Data Plane Isolation
Data plane isolation ensures that pods and workloads for different tenants are sufficiently
isolated.
Network isolation
By default, all pods in a Kubernetes cluster are allowed to communicate with each other, and all
network traffic is unencrypted. This can lead to security vu

lnerabilities where traffic is
accidentally or maliciously sent to an unintended destination, or is intercepted by a workload on
a compromised node.
Pod-to-pod communication can be controlled using
,
which restrict communication between pods using namespace labels or IP address ranges.
In a multi-tenant environment where strict network isolation between tenants is required, starting
with a default policy that denies communication between pods is recommended with another rule that
allows all pods to query the DNS server for name resolution. With such a default policy in place,
you can begin adding more permissive rules that allow for communication within a namespace.
It is also recommended not to use empty label selector '{}' for namespaceSelector field in network policy definition,
in case traffic need to be allowed between namespaces.
This scheme can be further refined as required. Note that this only applies to pods within a single
control plane; pods that belong to different virtual control planes cannot talk to each other via
Kubernetes networking.
Namespace management tools may simplify the creation of default or common network policies.
In addition, some of these tools allow you to enforce a consistent set of namespace labels across
your cluster, ensuring that they are a trusted basis for your policies.
Warning:
Network policies require a
CNI plugin
that supports the implementation of network policies. Otherwise, NetworkPolicy resources will be ignored.
More advanced network isolation may be provided by service meshes, which provide OSI Layer 7
policies based on workload identity, in addition to namespaces. These higher-level policies can
make it easier to manage namespace-based multi-tenancy, especially when multiple namespaces are
dedicated to a single tenant. They frequently also offer encryption using mutual TLS, protecting
your data even in the presence of a compromised node, and work across dedicated or virtual clusters.
However, they can be significantly more complex to manage and may not be appropriate for all users.
Storage isolation
Kubernetes offers several types of volumes that can be used as persistent storage for workloads.
For security and data-isolation,
dynamic volume provisioning
is recommended and volume types that use node resources should be avoided.
allow you to describe custom "classes"
of storage offered by your cluster, based on quality-of-service levels, backup policies, or custom
policies determined by the cluster administrators.
Pods can request storage using a
.
A PersistentVolumeClaim is a namespaced resource, which enables isolating portions of the storage
system and dedicating it to tenants within the shared Kubernetes cluster.
However, it is important to note that a PersistentVolume is a cluster-wide resource and has a
lifecycle independent of workloads and namespaces.
For example, you can configure a separate StorageClass for each tenant and use this to strengthen isolation.
If a StorageClass is shared, you should set a
reclaim policy of
Delete
to ensure that a PersistentVolume cannot be reused across different namespaces.
Sandboxing containers
Note:
Kubernetes pods are composed of one or more containers that execute on worker nodes.
Containers utilize OS-level virtualization and hence offer a weaker isolation boundary than
virtual machines that utilize hardware-based virtualization.
In a shared environment, unpatched vulnerabilities in the application and system layers can be
exploited by attackers for container breakouts and remote code execution that allow access to host
resources. In some applications, like a Content Management System (CMS

), customers may be allowed
the ability to upload and execute untrusted scripts or code. In either case, mechanisms to further
isolate and protect workloads using strong isolation are desirable.
Sandboxing provides a way to isolate workloads running in a shared cluster. It typically involves
running each pod in a separate execution environment such as a virtual machine or a userspace
kernel. Sandboxing is often recommended when you are running untrusted code, where workloads are
assumed to be malicious. Part of the reason this type of isolation is necessary is because
containers are processes running on a shared kernel; they mount file systems like
/sys
and
/proc
from the underlying host, making them less secure than an application that runs on a virtual
machine which has its own kernel. While controls such as seccomp, AppArmor, and SELinux can be
used to strengthen the security of containers, it is hard to apply a universal set of rules to all
workloads running in a shared cluster. Running workloads in a sandbox environment helps to
insulate the host from container escapes, where an attacker exploits a vulnerability to gain
access to the host system and all the processes/files running on that host.
Virtual machines and userspace kernels are 2 popular approaches to sandboxing. The following
sandboxing implementations are available:
gVisor
intercepts syscalls from containers and runs them through a
userspace kernel, written in Go, with limited access to the underlying host.
Kata Containers
is an OCI compliant runtime that allows you to run
containers in a VM. The hardware virtualization available in Kata offers an added layer of
security for containers running untrusted code.
Node Isolation
Node isolation is another technique that you can use to isolate tenant workloads from each other.
With node isolation, a set of nodes is dedicated to running pods from a particular tenant and
co-mingling of tenant pods is prohibited. This configuration reduces the noisy tenant issue, as
all pods running on a node will belong to a single tenant. The risk of information disclosure is
slightly lower with node isolation because an attacker that manages to escape from a container
will only have access to the containers and volumes mounted to that node.
Although workloads from different tenants are running on different nodes, it is important to be
aware that the kubelet and (unless using virtual control planes) the API service are still shared
services. A skilled attacker could use the permissions assigned to the kubelet or other pods
running on the node to move laterally within the cluster and gain access to tenant workloads
running on other nodes. If this is a major concern, consider implementing compensating controls
such as seccomp, AppArmor or SELinux or explore using sandboxed containers or creating separate
clusters for each tenant.
Node isolation is a little easier to reason about from a billing standpoint than sandboxing
containers since you can charge back per node rather than per pod. It also has fewer compatibility
and performance issues and may be easier to implement than sandboxing containers.
For example, nodes for each tenant can be configured with taints so that only pods with the
corresponding toleration can run on them. A mutating webhook could then be used to automatically
add tolerations and node affinities to pods deployed into tenant namespaces so that they run on a
specific set of nodes designated for that tenant.
Node isolation can be implemented using an
pod node selectors
or a
Virtual Kubelet
.
Additional Considerations
This section discusses other Kubernetes constructs and patterns that are relevant for multi-tenancy.
API priority and fairness
is a Kubernetes
feature that allows you to assign a priority to certain pods running within the cluster.
When an application calls the Kubernetes API, the API server evaluates the priority assigned to pod.
Calls from pods with higher priority are fulfille

d before those with a lower priority.
When contention is high, lower priority calls can be queued until the server is less busy or you
can reject the requests.
Using API priority and fairness will not be very common in SaaS environments unless you are
allowing customers to run applications that interface with the Kubernetes API, for example,
a controller.
Quality-of-Service (QoS)
When you’re running a SaaS application, you may want the ability to offer different
Quality-of-Service (QoS) tiers of service to different tenants. For example, you may have freemium
service that comes with fewer performance guarantees and features and a for-fee service tier with
specific performance guarantees. Fortunately, there are several Kubernetes constructs that can
help you accomplish this within a shared cluster, including network QoS, storage classes, and pod
priority and preemption. The idea with each of these is to provide tenants with the quality of
service that they paid for. Let’s start by looking at networking QoS.
Typically, all pods on a node share a network interface. Without network QoS, some pods may
consume an unfair share of the available bandwidth at the expense of other pods.
bandwidth plugin
creates an
extended resource
for networking that allows you to use Kubernetes resources constructs, i.e. requests/limits, to
apply rate limits to pods by using Linux tc queues.
Be aware that the plugin is considered experimental as per the
documentation and should be thoroughly tested before use in production environments.
For storage QoS, you will likely want to create different storage classes or profiles with
different performance characteristics. Each storage profile can be associated with a different
tier of service that is optimized for different workloads such IO, redundancy, or throughput.
Additional logic might be necessary to allow the tenant to associate the appropriate storage
profile with their workload.
Finally, there’s
pod priority and preemption
where you can assign priority values to pods. When scheduling pods, the scheduler will try
evicting pods with lower priority when there are insufficient resources to schedule pods that are
assigned a higher priority. If you have a use case where tenants have different service tiers in a
shared cluster e.g. free and paid, you may want to give higher priority to certain tiers using
this feature.
DNS
Kubernetes clusters include a Domain Name System (DNS) service to provide translations from names
to IP addresses, for all Services and Pods. By default, the Kubernetes DNS service allows lookups
across all namespaces in the cluster.
In multi-tenant environments where tenants can access pods and other Kubernetes resources, or where
stronger isolation is required, it may be necessary to prevent pods from looking up services in other
Namespaces.
You can restrict cross-namespace DNS lookups by configuring security rules for the DNS service.
For example, CoreDNS (the default DNS service for Kubernetes) can leverage Kubernetes metadata
to restrict queries to Pods and Services within a namespace. For more information, read an
example
of
configuring this within the CoreDNS documentation.
When a
Virtual Control Plane per tenant
model is used, a DNS
service must be configured per tenant or a multi-tenant DNS service must be used.
customized version of CoreDNS
that supports multiple tenants.
Operators
Operators
are Kubernetes controllers that manage
applications. Operators can simplify the management of multiple instances of an application, like
a database service, which makes them a common building block in the multi-consumer (SaaS)
multi-tenancy use case.
Operators used in a multi-tenant environment should follow a stricter set of guidelines.
Specifically, the Operator should:
Support creating resources within different tenant namespaces, rather than just in the namespace
in which the Operator is deployed.
Ensure that the Pods are configured 

with resource requests and limits, to ensure scheduling and fairness.
Support configuration of Pods for data-plane isolation techniques such as node isolation and
sandboxed containers.
Implementations
Note:
There are two primary ways to share a Kubernetes cluster for multi-tenancy: using Namespaces
(that is, a Namespace per tenant) or by virtualizing the control plane (that is, virtual control
plane per tenant).
In both cases, data plane isolation, and management of additional considerations such as API
Priority and Fairness, is also recommended.
Namespace isolation is well-supported by Kubernetes, has a negligible resource cost, and provides
mechanisms to allow tenants to interact appropriately, such as by allowing service-to-service
communication. However, it can be difficult to configure, and doesn't apply to Kubernetes
resources that can't be namespaced, such as Custom Resource Definitions, Storage Classes, and Webhooks.
Control plane virtualization allows for isolation of non-namespaced resources at the cost of
somewhat higher resource usage and more difficult cross-tenant sharing. It is a good option when
namespace isolation is insufficient but dedicated clusters are undesirable, due to the high cost
of maintaining them (especially on-prem) or due to their higher overhead and lack of resource
sharing. However, even within a virtualized control plane, you will likely see benefits by using
namespaces as well.
The two options are discussed in more detail in the following sections.
Namespace per tenant
As previously mentioned, you should consider isolating each workload in its own namespace, even if
you are using dedicated clusters or virtualized control planes. This ensures that each workload
only has access to its own resources, such as Config Maps and Secrets, and allows you to tailor
dedicated security policies for each workload. In addition, it is a best practice to give each
namespace names that are unique across your entire fleet (that is, even if they are in separate
clusters), as this gives you the flexibility to switch between dedicated and shared clusters in
the future, or to use multi-cluster tooling such as service meshes.
Conversely, there are also advantages to assigning namespaces at the tenant level, not just the
workload level, since there are often policies that apply to all workloads owned by a single
tenant. However, this raises its own problems. Firstly, this makes it difficult or impossible to
customize policies to individual workloads, and secondly, it may be challenging to come up with a
single level of "tenancy" that should be given a namespace. For example, an organization may have
divisions, teams, and subteams - which should be assigned a namespace?
To solve this, Kubernetes provides the
Hierarchical Namespace Controller (HNC)
,
which allows you to organize your namespaces into hierarchies, and share certain policies and
resources between them. It also helps you manage namespace labels, namespace lifecycles, and
delegated management, and share resource quotas across related namespaces. These capabilities can
be useful in both multi-team and multi-customer scenarios.
Other projects that provide similar capabilities and aid in managing namespaced resources are
listed below.
Multi-team tenancy
Capsule
Kiosk
Multi-customer tenancy
Kubeplus
Policy engines
Policy engines provide features to validate and generate tenant configurations:
Kyverno
OPA/Gatekeeper
Virtual control plane per tenant
Another form of control-plane isolation is to use Kubernetes extensions to provide each tenant a
virtual control-plane that enables segmentation of cluster-wide API resources.
Data plane isolation
techniques can be used with 

this model to securely
manage worker nodes across tenants.
The virtual control plane based multi-tenancy model extends namespace-based multi-tenancy by
providing each tenant with dedicated control plane components, and hence complete control over
cluster-wide resources and add-on services. Worker nodes are shared across all tenants, and are
managed by a Kubernetes cluster that is normally inaccessible to tenants.
This cluster is often referred to as a
super-cluster
(or sometimes as a
host-cluster
).
Since a tenant’s control-plane is not directly associated with underlying compute resources it is
referred to as a
virtual control plane
.
A virtual control plane typically consists of the Kubernetes API server, the controller manager,
and the etcd data store. It interacts with the super cluster via a metadata synchronization
controller which coordinates changes across tenant control planes and the control plane of the
super-cluster.
By using per-tenant dedicated control planes, most of the isolation problems due to sharing one
API server among all tenants are solved. Examples include noisy neighbors in the control plane,
uncontrollable blast radius of policy misconfigurations, and conflicts between cluster scope
objects such as webhooks and CRDs. Hence, the virtual control plane model is particularly
suitable for cases where each tenant requires access to a Kubernetes API server and expects the
full cluster manageability.
The improved isolation comes at the cost of running and maintaining an individual virtual control
plane per tenant. In addition, per-tenant control planes do not solve isolation problems in the
data plane, such as node-level noisy neighbors or security threats. These must still be addressed
separately.
Cluster API - Nested (CAPN)
project provides an implementation of virtual control planes.
Other implementations
Kamaji
vcluster
9.10 - Kubernetes API Server Bypass Risks
Security architecture information relating to the API server and other components
The Kubernetes API server is the main point of entry to a cluster for external parties
(users and services) interacting with it.
As part of this role, the API server has several key built-in security controls, such as
audit logging and
.
However, there are ways to modify the configuration
or content of the cluster that bypass these controls.
This page describes the ways in which the security controls built into the
Kubernetes API server can be bypassed, so that cluster operators
and security architects can ensure that these bypasses are appropriately restricted.
The
kubelet
on each node loads and
directly manages any manifests that are stored in a named directory or fetched from
a specific URL as
in
your cluster. The API server doesn't manage these static Pods. An attacker with write
access to this location could modify the configuration of static pods loaded from that
source, or could introduce new static Pods.
Static Pods are restricted from accessing other objects in the Kubernetes API. For example,
you can't configure a static Pod to mount a Secret from the cluster. However, these Pods can
take other security sensitive actions, such as using
hostPath
mounts from the underlying
node.
By default, the kubelet creates a
mirror pod
so that the static Pods are visible in the Kubernetes API. However, if the attacker uses an invalid
namespace name when creating the Pod, it will not be visible in the Kubernetes API and can only
be discovered by tooling that has access to the affected host(s).
If a static Pod fails admission control, the kubelet won't register the Pod with the
API server. However, the Pod still runs on the node. For more information, refer to
kubeadm issue #1541
.
Only
enable the kubelet static Pod manifest functionality
if required by the node.
If a node uses the static Pod functionality, restrict filesystem access to the static Pod manifest directory
or URL to users who need the acce

ss.
Restrict access to kubelet configuration parameters and files to prevent an attacker setting
a static Pod path or URL.
Regularly audit and centrally report all access to directories or web storage locations that host
static Pod manifests and kubelet configuration files.
The kubelet API
The kubelet provides an HTTP API that is typically exposed on TCP port 10250 on cluster
worker nodes. The API might also be exposed on control plane nodes depending on the Kubernetes
distribution in use. Direct access to the API allows for disclosure of information about
the pods running on a node, the logs from those pods, and execution of commands in
every container running on the node.
When Kubernetes cluster users have RBAC access to
Node
object sub-resources, that access
serves as authorization to interact with the kubelet API. The exact access depends on
which sub-resource access has been granted, as detailed in
kubelet authorization
.
Direct access to the kubelet API is not subject to admission control and is not logged
by Kubernetes audit logging. An attacker with direct access to this API may be able to
bypass controls that detect or prevent certain actions.
The kubelet API can be configured to authenticate requests in a number of ways.
By default, the kubelet configuration allows anonymous access. Most Kubernetes providers
change the default to use webhook and certificate authentication. This lets the control plane
ensure that the caller is authorized to access the
nodes
API resource or sub-resources.
The default anonymous access doesn't make this assertion with the control plane.
Restrict access to sub-resources of the
nodes
API object using mechanisms such as
RBAC
. Only grant this access when required,
such as by monitoring services.
Restrict access to the kubelet port. Only allow specified and trusted IP address
ranges to access the port.
Ensure that
kubelet authentication
.
is set to webhook or certificate mode.
Ensure that the unauthenticated "read-only" Kubelet port is not enabled on the cluster.
The etcd API
Kubernetes clusters use etcd as a datastore. The
etcd
service listens on TCP port 2379.
The only clients that need access are the Kubernetes API server and any backup tooling
that you use. Direct access to this API allows for disclosure or modification of any
data held in the cluster.
Access to the etcd API is typically managed by client certificate authentication.
Any certificate issued by a certificate authority that etcd trusts allows full access
to the data stored inside etcd.
Direct access to etcd is not subject to Kubernetes admission control and is not logged
by Kubernetes audit logging. An attacker who has read access to the API server's
etcd client certificate private key (or can create a new trusted client certificate) can gain
cluster admin rights by accessing cluster secrets or modifying access rules. Even without
elevating their Kubernetes RBAC privileges, an attacker who can modify etcd can retrieve any API object
or create new workloads inside the cluster.
Many Kubernetes providers configure
etcd to use mutual TLS (both client and server verify each other's certificate for authentication).
There is no widely accepted implementation of authorization for the etcd API, although
the feature exists. Since there is no authorization model, any certificate
with client access to etcd can be used to gain full access to etcd. Typically, etcd client certificates
that are only used for health checking can also grant full read and write access.
Ensure that the certificate authority trusted by etcd is used only for the purposes of
authentication to that service.
Control access to the private key for the etcd server certificate, and to the API server's
client certificate and key.
Consider restricting access to the etcd port at a network level, to only allow access
from specified and trusted IP address ranges.
Container runtime socket
On each node in a Kubernetes cluster, access to interact with co

ntainers is controlled
by the container runtime (or runtimes, if you have configured more than one). Typically,
the container runtime exposes a Unix socket that the kubelet can access. An attacker with
access to this socket can launch new containers or interact with running containers.
At the cluster level, the impact of this access depends on whether the containers that
run on the compromised node have access to Secrets or other confidential
data that an attacker could use to escalate privileges to other worker nodes or to
control plane components.
Ensure that you tightly control filesystem access to container runtime sockets.
When possible, restrict this access to the
root
user.
Isolate the kubelet from other components running on the node, using
mechanisms such as Linux kernel namespaces.
Ensure that you restrict or forbid the use of
hostPath
mounts
that include the container runtime socket, either directly or by mounting a parent
directory. Also
hostPath
mounts must be set as read-only to mitigate risks
of attackers bypassing directory restrictions.
Restrict user access to nodes, and especially restrict superuser access to nodes.
9.11 - Security Checklist
Baseline checklist for ensuring security in Kubernetes clusters.
This checklist aims at providing a basic list of guidance with links to more
comprehensive documentation on each topic. It does not claim to be exhaustive
and is meant to evolve.
On how to read and use this document:
The order of topics does not reflect an order of priority.
Some checklist items are detailed in the paragraph below the list of each section.
Caution:
Checklists are
not
sufficient for attaining a good security posture on their
own. A good security posture requires constant attention and improvement, but a
checklist can be the first step on the never-ending journey towards security
preparedness. Some of the recommendations in this checklist may be too
restrictive or too lax for your specific security needs. Since Kubernetes
security is not "one size fits all", each category of checklist items should be
evaluated on its merits.
Authentication &amp; Authorization
group is not used for user or component authentication after bootstrapping.
The kube-controller-manager is running with
enabled.
The root certificate is protected (either an offline CA, or a managed
online CA with effective access controls).
Intermediate and leaf certificates have an expiry date no more than 3
years in the future.
A process exists for periodic access review, and reviews occur no more
than 24 months apart.
The
is followed for guidance related to authentication and authorization.
After bootstrapping, neither users nor components should authenticate to the
Kubernetes API as
. Similarly, running all of
kube-controller-manager as
should be avoided. In fact,
should only be used as a break-glass mechanism, as opposed to
an admin user.
Network security
CNI plugins in-use supports network policies.
Ingress and egress network policies are applied to all workloads in the
cluster.
Default network policies within each namespace, selecting all pods, denying
everything, are in place.
If appropriate, a service mesh is used to encrypt all communications inside of the cluster.
The Kubernetes API, kubelet API and etcd are not exposed publicly on Internet.
Access from the workloads to the cloud metadata API is filtered.
Use of LoadBalancer and ExternalIPs is restricted.
A number of
Container Network Interface (CNI) plugins
plugins provide the functionality to
restrict network resources that pods may communicate with. This is most commonly done
through
which provide a namespaced resource to define rules. Default network policies
blocking everything egress and ingress, in each namespace, selecting all the
pods, can be useful to adopt an allow list approach, ensuring that no workload

s
is missed.
Not all CNI plugins provide encryption in transit. If the chosen plugin lacks this
feature, an alternative solution could be to use a service mesh to provide that
The etcd datastore of the control plane should have controls to limit access and
not be publicly exposed on the Internet. Furthermore, mutual TLS (mTLS) should
be used to communicate securely with it. The certificate authority for this
should be unique to etcd.
External Internet access to the Kubernetes API server should be restricted to
not expose the API publicly. Be careful as many managed Kubernetes distribution
are publicly exposing the API server by default. You can then use a bastion host
to access the server.
The
kubelet
API access
should be restricted and not publicly exposed, the defaults authentication and
authorization settings, when no configuration file specified with the
--config
flag, are overly permissive.
If a cloud provider is used for hosting Kubernetes, the access from pods to the cloud
metadata API
169.254.169.254
should also be restricted or blocked if not needed
because it may leak information.
For restricted LoadBalancer and ExternalIPs use, see
CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs
and the
DenyServiceExternalIPs admission controller
RBAC rights to
create
,
update
,
patch
,
delete
workloads is only granted if necessary.
Appropriate Pod Security Standards policy is applied for all namespaces and enforced.
Memory limit is set for the workloads with a limit equal or inferior to the request.
CPU limit might be set on sensitive workloads.
For nodes that support it, Seccomp is enabled with appropriate syscalls
profile for programs.
For nodes that support it, AppArmor or SELinux is enabled with appropriate
RBAC authorization is crucial but
cannot be granular enough to have authorization on the Pods' resources
(or on any resource that manages Pods). The only granularity is the API verbs
on the resource itself, for example,
create
on Pods. Without
additional admission, the authorization to create these resources allows direct
unrestricted access to the schedulable nodes of a cluster.
The
define three different policies, privileged, baseline and restricted that limit
how fields can be set in the
PodSpec
regarding security.
These standards can be enforced at the namespace level with the new
admission,
enabled by default, or by third-party admission webhook. Please note that,
contrary to the removed PodSecurityPolicy admission it replaces,
admission can be easily combined with admission webhooks and external services.
policy, the most restrictive policy of the
set,
can operate in several modes
,
warn
,
audit
or
enforce
to gradually apply the most appropriate
according to security best practices. Nevertheless, pods'
should be separately investigated to limit the privileges and access pods may
have on top of the predefined security standards, for specific use cases.
For a hands-on tutorial on
,
see the blog post
Kubernetes 1.23: Pod Security Graduates to Beta
.
Memory and CPU limits
should be set in order to restrict the memory and CPU resources a pod can
consume on a node, and therefore prevent potential DoS attacks from malicious or
breached workloads. Such policy can be enforced by an admission controller.
Please note that CPU limits will throttle usage and thus can have unintended
effects on auto-scaling features or efficiency i.e. running the process in best
effort with the CPU resource available.
Caution:
Memory limit superior to request can expose the whole node to OOM issues.
Enabling Seccomp
Seccomp can improve the security of your workloads by reducing the Linux kernel
syscall attack surface available inside containers. The seccomp filter mode
leverages BPF to create an allow or de

ny list of specific syscalls, named
profiles. Those seccomp profiles can be enabled on individual workloads,
a security tutorial is available
. In
addition, the
Kubernetes Security Profiles Operator
is a project to facilitate the management and use of seccomp in clusters.
For historical context, please note that Docker has been using
a default seccomp profile
to only allow a restricted set of syscalls since 2016 from
Docker Engine 1.10
,
but Kubernetes is still not confining workloads by default. The default seccomp
profile can be found
in containerd
as well. Fortunately,
Seccomp Default
, a
new alpha feature to use a default seccomp profile for all workloads can now be
enabled and tested.
Note:
Seccomp is only available on Linux nodes.
Enabling AppArmor or SELinux
AppArmor
AppArmor
is a Linux kernel security module that can
provide an easy way to implement Mandatory Access Control (MAC) and better
auditing through system logs. To
enable AppArmor in Kubernetes
,
at least version 1.4 is required. Like seccomp, AppArmor is also configured
through profiles, where each profile is either running in enforcing mode, which
blocks access to disallowed resources or complain mode, which only reports
violations. AppArmor profiles are enforced on a per-container basis, with an
annotation, allowing for processes to gain just the right privileges.
Note:
AppArmor is only available on Linux nodes, and enabled in
some Linux distributions
.
SELinux
SELinux
is also a
Linux kernel security module that can provide a mechanism for supporting access
control security policies, including Mandatory Access Controls (MAC). SELinux
labels can be assigned to containers or pods
via their
section
.
Note:
SELinux is only available on Linux nodes, and enabled in
.
Pod placement
Pod placement is done in accordance with the tiers of sensitivity of the
application.
Sensitive applications are running isolated on nodes or with specific
sandboxed runtimes.
Pods that are on different tiers of sensitivity, for example, an application pod
and the Kubernetes API server, should be deployed onto separate nodes. The
purpose of node isolation is to prevent an application container breakout to
directly providing access to applications with higher level of sensitivity to easily
pivot within the cluster. This separation should be enforced to prevent pods
accidentally being deployed onto the same node. This could be enforced with the
following features:
Node Selectors
Key-value pairs, as part of the pod specification, that specify which nodes to
deploy onto. These can be enforced at the namespace and cluster level with the
An admission controller that allows administrators to restrict permitted
within a
namespace. Pods within a namespace may only utilize the tolerations specified on
the namespace object annotation keys that provide a set of default and allowed
RuntimeClass is a feature for selecting the container runtime configuration.
The container runtime configuration is used to run a Pod's containers and can
provide more or less isolation from the host at the cost of performance
overhead.
Secrets
ConfigMaps are not used to hold confidential data.
Encryption at rest is configured for the Secret API.
If appropriate, a mechanism to inject secrets stored in third-party storage
is deployed and available.
Service account tokens are not mounted in pods that don't require them.
Bound service account token volume
is in-use instead of non-expiring tokens.
Secrets required for pods should be stored within Kubernetes Secrets as opposed
to alternatives such as ConfigMap. Secret resources stored within etcd should
be
.
Pods needing secrets should have these automatically mounted through volumes,
preferably stored in memory like with the
option
.
Mechanism can be used to also inject secrets from third-party storages 

as
volume, like the
Secrets Store CSI Driver
.
This should be done preferentially as compared to providing the pods service
account RBAC access to secrets. This would allow adding secrets into the pod as
environment variables or files. Please note that the environment variable method
might be more prone to leakage due to crash dumps in logs and the
non-confidential nature of environment variable in Linux, as opposed to the
permission mechanism on files.
Service account tokens should not be mounted into pods that do not require them. This can be configured by setting
to
false
either within the service account to apply throughout the namespace
or specifically for a pod. For Kubernetes v1.22 and above, use
Bound Service Accounts
for time-bound service account credentials.
Images
Minimize unnecessary content in container images.
Container images are configured to be run as unprivileged user.
References to container images are made by sha256 digests (rather than
tags) or the provenance of the image is validated by verifying the image's
digital signature at deploy time
via admission control
.
Container images are regularly scanned during creation and in deployment, and
known vulnerable software is patched.
Container image should contain the bare minimum to run the program they
package. Preferably, only the program and its dependencies, building the image
from the minimal possible base. In particular, image used in production should not
contain shells or debugging utilities, as an
ephemeral debug container
can be used for troubleshooting.
Build images to directly start with an unprivileged user by using the
USER
instruction in Dockerfile
.
The
allows a container image to be started with a specific user and group with
runAsUser
and
, even if not specified in the image manifest.
However, the file permissions in the image layers might make it impossible to just
start the process with a new unprivileged user without image modification.
Avoid using image tags to reference an image, especially the
latest
tag, the
image behind a tag can be easily modified in a registry. Prefer using the
complete
sha256
digest which is unique to the image manifest. This policy can be
enforced via an
.
Image signatures can also be automatically
verified with an admission controller
at deploy time to validate their authenticity and integrity.
Scanning a container image can prevent critical vulnerabilities from being
deployed to the cluster alongside the container image. Image scanning should be
completed before deploying a container image to a cluster and is usually done
as part of the deployment process in a CI/CD pipeline. The purpose of an image
scan is to obtain information about possible vulnerabilities and their
prevention in the container image, such as a
Common Vulnerability Scoring System (CVSS)
score. If the result of the image scans is combined with the pipeline
compliance rules, only properly patched container images will end up in
Production.
Admission controllers
An appropriate selection of admission controllers is enabled.
A pod security policy is enforced by the Pod Security Admission or/and a
webhook admission controller.
The admission chain plugins and webhooks are securely configured.
Admission controllers can help to improve the security of the cluster. However,
they can present risks themselves as they extend the API server and
should be properly secured
.
The following lists present a number of admission controllers that could be
considered to enhance the security posture of your cluster and application. It
includes controllers that may be referenced in other parts of this document.
This first group of admission controllers includes plugins
enabled by default
,
consider to leave them enabled unless you know what you are doing:
Performs additional authorization checks to ensure the approving user has
permission to approve certificate request.
Ce

rtificateSigning
Performs additional authorization checks to ensure the signing user has
permission to sign certificate requests.
Rejects any certificate request that specifies a 'group' (or 'organization
attribute') of
.
Enforce the LimitRange API constraints.
Allows the use of custom controllers through webhooks, these controllers may
mutate requests that it reviews.
Replacement for Pod Security Policy, restricts security contexts of deployed
Pods.
Enforces resource quotas to prevent over-usage of resources.
Allows the use of custom controllers through webhooks, these controllers do
not mutate requests that it reviews.
The second group includes plugin that are not enabled by default but in general
availability state and recommended to improve your security posture:
Rejects all net-new usage of the
Service.spec.externalIPs
field. This is a mitigation for
.
Restricts kubelet's permissions to only modify the pods API resources they own
or the node API ressource that represent themselves. It also prevents kubelet
from using the
annotation, which can be used
by an attacker with access to the kubelet's credentials to influence pod
placement to the controlled node.
The third group includes plugins that are not enabled by default but could be
considered for certain use cases:
Enforces the usage of the latest version of a tagged image and ensures that the deployer
has permissions to use the image.
Allows enforcing additional controls for images through webhooks.
RBAC Good Practices
for
further information on authorization.
Cluster Multi-tenancy guide
for
configuration options recommendations and best practices on multi-tenancy.
Blog post "A Closer Look at NSA/CISA Kubernetes Hardening Guidance"
for complementary resource on hardening Kubernetes clusters.
10 - Policies
Note:
See
for documentation about NetworkPolicy in Kubernetes.
10.1 - Limit Ranges
By default, containers run with unbounded
on a Kubernetes cluster.
Using Kubernetes
,
administrators (also termed
cluster operators
) can restrict consumption and creation
of cluster resources (such as CPU time, memory, and persistent storage) within a specified
namespace
.
Within a namespace, a
Pod
can consume as much CPU and memory as is allowed by the ResourceQuotas that apply to that namespace. As a cluster operator, or as a namespace-level administrator, you might also be concerned about making sure that a single object cannot monopolize all available resources within a namespace.
A LimitRange is a policy to constrain the resource allocations (limits and requests) that you can specify for each applicable object kind (such as Pod or
) in a namespace.
A
provides constraints that can:
Enforce minimum and maximum compute resources usage per Pod or Container in a namespace.
Enforce minimum and maximum storage request per
Enforce a ratio between request and limit for a resource in a namespace.
Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime.
A LimitRange is enforced in a particular namespace when there is a
LimitRange object in that namespace.
The name of a LimitRange object must be a valid
.
The administrator creates a LimitRange in a namespace.
Users create (or try to create) objects in that namespace, such as Pods or PersistentVolumeClaims.
First, the
admission controller applies default request and limit values for all Pods (and their containers) th

at do not set compute resource requirements.
Second, the
tracks usage to ensure it does not exceed resource minimum, maximum and ratio defined in any
present in the namespace.
If you attempt to create or update an object (Pod or PersistentVolumeClaim) that violates a
constraint, your request to the API server will fail with an HTTP status code
and a message explaining the constraint that has been violated.
If you add a
in a namespace that applies to compute-related resources such as
cpu
and
memory
, you must specify
requests or limits for those values. Otherwise, the system may reject Pod creation.
validations occur only at Pod admission stage, not on running Pods.
If you add or modify a LimitRange, the Pods that already exist in that namespace
continue unchanged.
If two or more
objects exist in the namespace, it is not deterministic which default value will be applied.
LimitRange and admission checks for Pods
A
does
not
limit
request
For example, you define a
with this manifest:
concepts/policy/limit-range/problematic-limit-range.yaml
:
v1
kind
:
metadata
:
name
:
cpu-resource-constraint
spec
:
limits
:
-
default
:
# this section defines default limits
cpu
:
500m
:
# this section defines default requests
cpu
:
500m
max
:
# max and min define the limit range
cpu
:
"1"
min
:
cpu
:
100m
type
:
Container
along with a Pod that declares a CPU resource request of
700m
, but not a limit:
concepts/policy/limit-range/example-conflict-with-limitrange-cpu.yaml
:
v1
kind
:
Pod
metadata
:
name
:
example-conflict-with-limitrange-cpu
spec
:
:
-
name
:
demo
image
:
resources
:
requests
:
cpu
:
700m
then that Pod will not be scheduled, failing with an error similar to:
Pod "example-conflict-with-limitrange-cpu" is invalid: spec.containers[0].resources.requests: Invalid value: "700m": must be less than or equal to cpu limit
If you set both
request
and
limit
, then that new Pod will be scheduled successfully even with the same
in place:
concepts/policy/limit-range/example-no-conflict-with-limitrange-cpu.yaml
:
v1
kind
:
Pod
metadata
:
name
:
example-no-conflict-with-limitrange-cpu
spec
:
:
-
name
:
demo
image
:
resources
:
requests
:
cpu
:
700m
limits
:
cpu
:
700m
Example resource constraints
Examples of policies that could be created using
are:
In a 2 node cluster with a capacity of 8 GiB RAM and 16 cores, constrain Pods in a namespace to request 100m of CPU with a max limit of 500m for CPU and request 200Mi for Memory with a max limit of 600Mi for Memory.
Define default CPU limit and request to 150m and memory default request to 300Mi for Containers started with no cpu and memory requests in their specs.
In the case where the total limits of the namespace is less than the sum of the limits of the Pods/Containers,
there may be contention for resources. In this case, the Containers or Pods will not be created.
Neither contention nor changes to a LimitRange will affect already created resources.
For examples on using limits, see:
how to configure minimum and maximum CPU constraints per namespace
.
how to configure minimum and maximum Memory constraints per namespace
.
how to configure default CPU Requests and Limits per namespace
.
how to configure default Memory Requests and Limits per namespace
.
how to configure minimum and maximum Storage consumption per namespace
.
a
detailed example on configuring quota per namespace
.
LimitRanger design document
for context and historical informati

on.
10.2 - Resource Quotas
When several users or teams share a cluster with a fixed number of nodes,
there is a concern that one team could use more than its fair share of resources.
Resource quotas are a tool for administrators to address this concern.
A resource quota, defined by a
object, provides constraints that limit
aggregate resource consumption per namespace. It can limit the quantity of objects that can
be created in a namespace by type, as well as the total amount of compute resources that may
be consumed by resources in that namespace.
Resource quotas work like this:
Different teams work in different namespaces. This can be enforced with
RBAC
.
The administrator creates one ResourceQuota for each namespace.
Users create resources (pods, services, etc.) in the namespace, and the quota system
tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.
If creating or updating a resource violates a quota constraint, the request will fail with HTTP
status code
403 FORBIDDEN
with a message explaining the constraint that would have been violated.
If quota is enabled in a namespace for compute resources like
cpu
and
memory
, users must specify
requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use
the
admission controller to force defaults for pods that make no compute resource requirements.
See the
walkthrough
for an example of how to avoid this problem.
Note:
For
cpu
and
memory
resources, ResourceQuotas enforce that
every
(new) pod in that namespace sets a limit for that resource.
If you enforce a resource quota in a namespace for either
cpu
or
memory
,
you, and other clients,
must
specify either
requests
or
limits
for that resource,
for every new Pod you submit. If you don't, the control plane may reject admission
for that Pod.
For other resources: ResourceQuota works and will ignore pods in the namespace without setting a limit or request for that resource. It means that you can create a new pod without limit/request ephemeral storage if the resource quota limits the ephemeral storage of this namespace.
to automatically set
a default request for these resources.
The name of a ResourceQuota object must be a valid
.
Examples of policies that could be created using namespaces and quotas are:
In a cluster with a capacity of 32 GiB RAM, and 16 cores, let team A use 20 GiB and 10 cores,
let B use 10GiB and 4 cores, and hold 2GiB and 2 cores in reserve for future allocation.
Limit the "testing" namespace to using 1 core and 1GiB RAM. Let the "production" namespace
use any amount.
In the case where the total capacity of the cluster is less than the sum of the quotas of the namespaces,
there may be contention for resources. This is handled on a first-come-first-served basis.
Neither contention nor changes to quota will affect already created resources.
Enabling Resource Quota
Resource Quota support is enabled by default for many Kubernetes distributions. It is
enabled when the
API server
--enable-admission-plugins=
flag has
as
one of its arguments.
A resource quota is enforced in a particular namespace when there is a
Compute Resource Quota
You can limit the total sum of
that can be requested in a given namespace.
The following resource types are supported:
Resource Name
Across all pods in a non-terminal state, the sum of CPU limits cannot exceed this value.
Across all pods in a non-terminal state, the sum of memory limits cannot exceed this value.
Across all pods in a non-terminal state, the sum of CPU requests cannot exceed this value.
Across all pods in a non-terminal state, the sum of memory requests cannot exceed this value.
Across all pods in a non-terminal state, the number of huge page requests of the specified size cannot ex

ceed this value.
cpu
Same as
memory
Same as
Resource Quota For Extended Resources
In addition to the resources mentioned above, in release 1.10, quota support for
extended resources
is added.
As overcommit is not allowed for extended resources, it makes no sense to specify both
requests
and
limits
for the same extended resource in a quota. So for extended resources, only quota items
with prefix
requests.
is allowed for now.
Take the GPU resource as an example, if the resource name is
, and you want to
limit the total number of GPUs requested in a namespace to 4, you can define a quota as follows:
requests.nvidia.com/gpu: 4
See
Viewing and Setting Quotas
for more detail information.
Storage Resource Quota
storage resources
In addition, you can limit consumption of storage resources based on associated storage-class.
Across all persistent volume claims, the sum of storage requests cannot exceed this value.
The total number of
that can exist in the namespace.
&lt;storage-class-name&gt;.storageclass.storage.k8s.io/requests.storage
Across all persistent volume claims associated with the
&lt;storage-class-name&gt;
, the sum of storage requests cannot exceed this value.
&lt;storage-class-name&gt;.storageclass.storage.k8s.io/persistentvolumeclaims
Across all persistent volume claims associated with the storage-class-name, the total number of
persistent volume claims
For example, if an operator wants to quota storage with
gold
storage class separate from
bronze
storage class, the operator can
define a quota as follows:
gold.storageclass.storage.k8s.io/requests.storage: 500Gi
bronze.storageclass.storage.k8s.io/requests.storage: 100Gi
In release 1.8, quota support for local ephemeral storage is added as an alpha feature:
requests.ephemeral-storage
Across all pods in the namespace, the sum of local ephemeral storage requests cannot exceed this value.
limits.ephemeral-storage
Across all pods in the namespace, the sum of local ephemeral storage limits cannot exceed this value.
Same as
.
Note:
When using a CRI container runtime, container logs will count against the ephemeral storage quota.
This can result in the unexpected eviction of pods that have exhausted their storage quotas.
Refer to
Object Count Quota
You can set quota for the total number of certain resources of all standard,
namespaced resource types using the following syntax:
count/&lt;resource&gt;.&lt;group&gt;
for resources from non-core groups
count/&lt;resource&gt;
for resources from the core group
Here is an example set of resources users may want to put under object count quota:
count/persistentvolumeclaims
count/services
count/secrets
count/configmaps
count/replicationcontrollers
count/deployments.apps
count/replicasets.apps
count/statefulsets.apps
count/jobs.batch
count/cronjobs.batch
The same syntax can be used for custom resources.
For example, to create a quota on a
widgets
custom resource in the
API group, use
count/widgets.example.com
.
When using
count/*
resource quota, an object is charged against the quota if it exists in server storage.
These types of quotas are useful to protect against exhaustion of storage resources. For example, you may
want to limit the number of Secrets in a server given their large size. Too many Secrets in a cluster can
actually prevent servers and controllers from starting. You can set a quota for Jobs to protect against
a poorly configured CronJob. CronJobs that create too many Jobs in a namespace can lead to a denial of service.
It is also possible to do generic object count quota on a limited set of resources.
The following types are supported:
Th

e total number of ConfigMaps that can exist in the namespace.
pods
The total number of Pods in a non-terminal state that can exist in the namespace. A pod is in a terminal state if
.status.phase in (Failed, Succeeded)
is true.
The total number of ReplicationControllers that can exist in the namespace.
The total number of ResourceQuotas that can exist in the namespace.
services
The total number of Services that can exist in the namespace.
The total number of Services of type
NodePort
secrets
The total number of Secrets that can exist in the namespace.
pods
quota counts and enforces a maximum on the number of
pods
created in a single namespace that are not terminal. You might want to set a
pods
quota on a namespace to avoid the case where a user creates many small pods and
exhausts the cluster's supply of Pod IPs.
Quota Scopes
Each quota can have an associated set of
scopes
. A quota will only measure usage for a resource if it matches
the intersection of enumerated scopes.
When a scope is added to the quota, it limits the number of resources it supports to those that pertain to the scope.
Resources specified on the quota outside of the allowed set results in a validation error.
Scope
Match pods where
.spec.activeDeadlineSeconds &gt;= 0
NotTerminating
.spec.activeDeadlineSeconds is nil
Match pods that have best effort quality of service.
NotBestEffort
Match pods that do not have best effort quality of service.
Match pods that references the specified
.
Match pods that have cross-namespace pod
(anti)affinity terms
.
The
scope restricts a quota to tracking the following resource:
pods
The
,
,
and
scopes restrict a quota to tracking the following resources:
pods
cpu
memory
Note that you cannot specify both the
and the
scopes in the same quota, and you cannot specify both the
and
scopes in the same quota either.
The
supports the following values in the
operator
field:
In
NotIn
Exists
DoesNotExist
When using one of the following values as the
scopeName
when defining the
, the
operator
must be
Exists
.
If the
operator
is
In
or
NotIn
, the
values
field must have at least
one value. For example:
:
:
-
scopeName
:
operator
:
In
values
:
- middle
If the
operator
is
Exists
or
, the
values
field must
NOT
be
Resource Quota Per PriorityClass
Pods can be created at a specific
priority
.
You can control a pod's consumption of system resources based on a pod's priority, by using the
field in the quota spec.
A quota is matched and consumed only if
in the quota spec selects the pod.
When quota is scoped for priority class using
field, quota object
is restricted to track only following resources:
pods
cpu
memory
This example creates a quota object and matches it with pods at specific priorities. The example
works as follows:
Pods in the cluster have one of the three priority classes, "low", "medium", "high".
One quota object is created for each priority.
Save the following YAML to a file
quota.yml
.
:
v1
kind
:
List
items
:
-
:
v1
kind
:
metadata
:
name
:
pods-high
spec
:
hard
:
cpu
:
"10

00"
memory
:
200Gi
pods
:
"10"
:
:
-
operator
:
In
scopeName
:
values
:
[
"high"
]
-
:
v1
kind
:
metadata
:
name
:
pods-medium
spec
:
hard
:
cpu
:
"10"
memory
:
20Gi
pods
:
"10"
:
:
-
operator
:
In
scopeName
:
values
:
[
"medium"
]
-
:
v1
kind
:
metadata
:
name
:
pods-low
spec
:
hard
:
cpu
:
"5"
memory
:
10Gi
pods
:
"10"
:
:
-
operator
:
In
scopeName
:
values
:
[
"low"
]
Apply the YAML using
.
kubectl create -f ./quota.yml
resourcequota/pods-high created
resourcequota/pods-medium created
resourcequota/pods-low created
Verify that
Used
quota is
0
using
kubectl describe quota
.
Name:       pods-high
Resource    Used  Hard
--------    ----  ----
cpu         0     1k
memory      0     200Gi
pods        0     10

Name:       pods-low
cpu         0     5
memory      0     10Gi
pods        0     10

Name:       pods-medium
cpu         0     10
memory      0     20Gi
pods        0     10
Create a pod with priority "high". Save the following YAML to a
file
high-priority-pod.yml
.
:
v1
kind
:
Pod
metadata
:
name
:
high-priority
spec
:
:
-
name
:
image
:
ubuntu
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
]
resources
:
requests
:
memory
:
"10Gi"
cpu
:
"500m"
limits
:
memory
:
"10Gi"
cpu
:
"500m"
:
high
Apply it with
.
kubectl create -f ./high-priority-pod.yml
Verify that "Used" stats for "high" priority quota,
pods-high
, has changed and that
the other two quotas are unchanged.
cpu         500m  1k
memory      10Gi  200Gi
pods        1     10

cpu         0     5
pods        0     10

cpu         0     10
pods        0     10
Cross-namespace Pod Affinity Quota
Operators can use
quota scope to limit which namespaces are allowed to
have pods with affinity terms that cross namespaces. Specifically, it controls which pods are allowed
to set
or
fields in pod affinity terms.
Preventing users from using cross-namespace affinity terms might be desired since a pod
with anti-affinity constraints can block pods from all other namespaces
from getting scheduled in a failure domain.
Using this scope operators can prevent certain namespaces (
foo-ns
in the example below)
from having pods that use cross-namespace pod affinity by creating a resource quota object in
that namespace with
CrossNamespaceAffinity
scope and hard limit of 0:
:
v1
kind
:
metadata
:
name
:
disable-cross-namespace-affinity
namespace
:
foo-ns
spec
:
hard
:
pods
:
"0"
:
:
-
scopeName
:
If operators want to disallow using
and
by default, and
only allow it for specific namespaces, they could configure
as a limited resource by setting the kube-apiserver flag --admission-control-config-file
to the path of the following configuration file:
:
kind
:
plugins
:
-
name
:
"ResourceQuota"
:
:
kind
:
ResourceQuotaConfiguration
limitedResources
:
-
resource
:
pods
matchScopes
:
-
scopeName
:
With the above configuration, pods can use
na

mespaces
and
in pod affinity only
if the namespace where they are created have a resource quota object with
scope and a hard limit greater than or equal to the number of pods using those fields.
Requests compared to Limits
When allocating compute resources, each container may specify a request and a limit value for either CPU or memory.
The quota can be configured to quota either value.
If the quota has a value specified for
or
, then it requires that every incoming
container makes an explicit request for those resources. If the quota has a value specified for
or
,
then it requires that every incoming container specifies an explicit limit for those resources.
Kubectl supports creating, updating, and viewing quotas:
kubectl create namespace myspace
cat
&lt;&lt;EOF &gt; compute-resources.yaml
kind: ResourceQuota
metadata:
name: compute-resources
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
EOF
kubectl create -f ./compute-resources.yaml --namespace
=
myspace
cat
&lt;&lt;EOF &gt; object-counts.yaml
metadata:
name: object-counts
spec:
hard:
configmaps: "10"
persistentvolumeclaims: "4"
pods: "4"
replicationcontrollers: "20"
secrets: "10"
services: "10"
services.loadbalancers: "2"
EOF
kubectl create -f ./object-counts.yaml --namespace
=
myspace
kubectl get quota --namespace
=
myspace
NAME                    AGE
compute-resources       30s
object-counts           32s
kubectl describe quota compute-resources --namespace
=
myspace
Name:                    compute-resources
Namespace:               myspace
Resource                 Used  Hard
--------                 ----  ----
limits.cpu               0     2
limits.memory            0     2Gi
requests.cpu             0     1
requests.memory          0     1Gi
requests.nvidia.com/gpu  0     4
kubectl describe quota object-counts --namespace
=
myspace
Name:                   object-counts
Namespace:              myspace
Resource                Used    Hard
--------                ----    ----
configmaps              0       10
persistentvolumeclaims  0       4
pods                    0       4
replicationcontrollers  0       20
secrets                 1       10
services                0       10
services.loadbalancers  0       2
Kubectl also supports object count quota for all standard namespaced resources
using the syntax
:
kubectl create quota
test
--hard
=
=
2,count/replicasets.apps
=
4,count/pods
=
3,count/secrets
=
4
=
myspace
=
=
myspace --replicas
=
2
kubectl describe quota --namespace
=
myspace
Name:                         test
Namespace:                    myspace
Resource                      Used  Hard
--------                      ----  ----
count/deployments.apps        1     2
count/pods                    2     3
count/replicasets.apps        1     4
count/secrets                 1     4
Quota and Cluster Capacity
ResourceQuotas are independent of the cluster capacity. They are
expressed in absolute units. So, if you add nodes to your cluster, this does
not
automatically give each namespace the ability to consume more resources.
Sometimes more complex policies may be desired, such as:
Proportionally divide total cluster resources among several teams.
Allow each tenant to grow resource usage as needed, but have a generous
limit to prevent accidental resource exhaustion.
Detect demand from one namespace, add nodes, and increase quota.
Such policies could be implemented using
as building blocks, by
writing a "controller" that watches the quota usage and adjusts the quota
hard limits of each namespace according to other signals.
Note that resource quota divides up a

ggregate cluster resources, but it creates no
restrictions around nodes: pods from several namespaces may run on the same node.
Limit Priority Class consumption by default
It may be desired that pods at a particular priority, eg. "cluster-services",
should be allowed in a namespace, if and only if, a matching quota object exists.
With this mechanism, operators are able to restrict usage of certain high
priority classes to a limited number of namespaces and not every namespace
will be able to consume these priority classes by default.
To enforce this,
flag
should be
used to pass path to the following configuration file:
:
kind
:
plugins
:
-
name
:
:
:
kind
:
:
-
resource
:
pods
:
-
scopeName
:
operator
:
In
values
:
[
"cluster-services"
]
Then, create a resource quota object in the
policy/priority-class-resourcequota.yaml
:
v1
kind
:
metadata
:
name
:
pods-cluster-services
spec
:
:
:
-
operator
:
In
scopeName
:
values
:
[
]
kubectl apply -f https://k8s.io/examples/policy/priority-class-resourcequota.yaml -n kube-system
resourcequota/pods-cluster-services created
In this case, a pod creation will be allowed if:
the Pod's
is not specified.
the Pod's
is specified to a value other than
cluster-services
.
the Pod's
is set to
, it is to be created
in the
namespace, and it has passed the resource quota check.
A Pod creation request is rejected if its
is set to
and it is to be created in a namespace other than
.
See
ResourceQuota design doc
See a
detailed example for how to use resource quota
.
Read
Quota support for priority class design doc
.
See
LimitedResources
10.3 - Process ID Limits And Reservations
Kubernetes allow you to limit the number of process IDs (PIDs) that a
Pod
can use.
You can also reserve a number of allocatable PIDs for each
node
for use by the operating system and daemons (rather than by Pods).
Process IDs (PIDs) are a fundamental resource on nodes. It is trivial to hit the
task limit without hitting any other resource limits, which can then cause
instability to a host machine.
Cluster administrators require mechanisms to ensure that Pods running in the
cluster cannot induce PID exhaustion that prevents host daemons (such as the
kubelet
or
,
and potentially also the container runtime) from running.
In addition, it is important to ensure that PIDs are limited among Pods in order
to ensure they have limited impact on other workloads on the same node.
Note:
On certain Linux installations, the operating system sets the PIDs limit to a low default,
such as
32768
. Consider raising the value of
.
You can configure a kubelet to limit the number of PIDs a given Pod can consume.
For example, if your node's host OS is set to use a maximum of
262144
PIDs and
expect to host less than
250
Pods, one can give each Pod a budget of
1000
PIDs to prevent using up that node's overall number of available PIDs. If the
admin wants to overcommit PIDs similar to CPU or memory, they may do so as well
with some additional risks. Either way, a single Pod will not be able to bring
the whole machine down. This kind of resource limiting helps to prevent simple
fork bombs from affecting operation of an entire cluster.
Per-Pod PID limiting allows administrators to protect one Pod from another, but
does not ensure that all Pods scheduled onto that host are unable to impact the node overall.
Per-Pod limiting also does not protect the node agents themselves from

 PID exhaustion.
You can also reserve an amount of PIDs for node overhead, separate from the
allocation to Pods. This is similar to how you can reserve CPU, memory, or other
resources for use by the operating system and other facilities outside of Pods
PID limiting is a an important sibling to
compute
resource
requests
and limits. However, you specify it in a different way: rather than defining a
Pod's resource limit in the
.spec
for a Pod, you configure the limit as a
setting on the kubelet. Pod-defined PID limits are not currently supported.
Caution:
This means that the limit that applies to a Pod may be different depending on
where the Pod is scheduled. To make things simple, it's easiest if all Nodes use
the same PID resource limits and reservations.
Node PID limits
Kubernetes allows you to reserve a number of process IDs for the system use. To
configure the reservation, use the parameter
in the
and
command line options to the kubelet.
The value you specified declares that the specified number of process IDs will
be reserved for the system as a whole and for Kubernetes system daemons
Pod PID limits
Kubernetes allows you to limit the number of processes running in a Pod. You
specify this limit at the node level, rather than configuring it as a resource
limit for a particular Pod. Each Node can have a different PID limit.
To configure the limit, you can specify the command line parameter
--pod-max-pids
to the kubelet, or set
PodPidsLimit
in the kubelet
.
PID based eviction
You can configure kubelet to start terminating a Pod when it is misbehaving and consuming abnormal amount of resources.
This feature is called eviction. You can
Configure Out of Resource Handling
for various eviction signals.
Use
eviction signal to configure the threshold for number of PIDs used by Pod.
You can set soft and hard eviction policies.
However, even with the hard eviction policy, if the number of PIDs growing very fast,
node can still get into unstable state by hitting the node PIDs limit.
Eviction signal value is calculated periodically and does NOT enforce the limit.
PID limiting - per Pod and per Node sets the hard limit.
Once the limit is hit, workload will start experiencing failures when trying to get a new PID.
It may or may not lead to rescheduling of a Pod,
depending on how workload reacts on these failures and how liveleness and readiness
probes are configured for the Pod. However, if limits were set correctly,
you can guarantee that other Pods workload and system processes will not run out of PIDs
when one Pod is misbehaving.
PID Limiting enhancement document
For historical context, read
Process ID Limiting for Stability Improvements in Kubernetes 1.14
.
Read
Managing Resources for Containers
.
.
10.4 - Node Resource Managers
In order to support latency-critical and high-throughput workloads, Kubernetes offers a suite of Resource Managers. The managers aim to co-ordinate and optimise node's resources alignment for pods configured with a specific requirement for CPUs, devices, and memory (hugepages) resources.
The main manager, the Topology Manager, is a Kubelet component that co-ordinates the overall resource management process through its
policy
.
The configuration of individual managers is elaborated in dedicated documents:
CPU Manager Policies
Device Manager
Memory Manager Policies
11 - Scheduling, Preemption and Eviction
In Kubernetes, scheduling refers to making sure that
Pods
are matched to

Nodes
so that the
kubelet
can run them. Preemption
is the process of terminating Pods with lower
Priority
so that Pods with higher Priority can schedule on Nodes. Eviction is the process
of terminating one or more Pods on Nodes.
Resource Bin Packing for Extended Resources
11.1 - Kubernetes Scheduler
refers to making sure that
Pods
Nodes
so that
Kubelet
can run them.
Scheduling overview
A scheduler watches for newly created Pods that have no Node assigned. For
every Pod that the scheduler discovers, the scheduler becomes responsible
for finding the best Node for that Pod to run on. The scheduler reaches
this placement decision taking into account the scheduling principles
described below.
If you want to understand why Pods are placed onto a particular Node,
or if you're planning to implement a custom scheduler yourself, this
page will help you learn about scheduling.
is the default scheduler for Kubernetes and runs as part of the
.
kube-scheduler is designed so that, if you want and need to, you can
write your own scheduling component and use that instead.
Kube-scheduler selects an optimal node to run newly created or not yet
scheduled (unscheduled) pods. Since containers in pods - and pods themselves -
can have different requirements, the scheduler filters out any nodes that
don't meet a Pod's specific scheduling needs. Alternatively, the API lets
you specify a node for a Pod when you create it, but this is unusual
and is only done in special cases.
In a cluster, Nodes that meet the scheduling requirements for a Pod
are called
feasible
nodes. If none of the nodes are suitable, the pod
remains unscheduled until the scheduler is able to place it.
The scheduler finds feasible Nodes for a Pod and then runs a set of
functions to score the feasible Nodes and picks a Node with the highest
score among the feasible ones to run the Pod. The scheduler then notifies
the API server about this decision in a process called
binding
.
Factors that need to be taken into account for scheduling decisions include
individual and collective resource requirements, hardware / software /
policy constraints, affinity and anti-affinity specifications, data
locality, inter-workload interference, and so on.
Node selection in kube-scheduler
kube-scheduler selects a node for the pod in a 2-step operation:
Filtering
Scoring
The
filtering
step finds the set of Nodes where it's feasible to
schedule the Pod. For example, the PodFitsResources filter checks whether a
candidate Node has enough available resource to meet a Pod's specific
resource requests. After this step, the node list contains any suitable
Nodes; often, there will be more than one. If the list is empty, that
Pod isn't (yet) schedulable.
In the
scoring
step, the scheduler ranks the remaining nodes to choose
the most suitable Pod placement. The scheduler assigns a score to each Node
that survived filtering, basing this score on the active scoring rules.
Finally, kube-scheduler assigns the Pod to the Node with the highest ranking.
If there is more than one node with equal scores, kube-scheduler selects
one of these at random.
There are two supported ways to configure the filtering and scoring behavior
of the scheduler:
allow you to configure

Predicates
for filtering and
Priorities
for scoring.
Scheduling Profiles
allow you to configure Plugins that implement different scheduling stages, including:
QueueSort
,
Filter
,
Score
,
Bind
,
Reserve
,
Permit
, and others. You can also configure the kube-scheduler to run different profiles.
Read about
scheduler performance tuning
Read about
Read the
for kube-scheduler
Read the
kube-scheduler config (v1beta3)
reference
configuring multiple schedulers
topology management policies
Learn about scheduling of Pods that use volumes in:
Volume Topology Support
Storage Capacity Tracking
11.2 - Assigning Pods to Nodes
You can constrain a
Pod
so that it is
to run on particular
node(s)
,
or to
prefer
to run on particular nodes.
There are several ways to do this and the recommended approaches all use
to facilitate the selection.
Often, you do not need to set any such constraints; the
scheduler
will automatically do a reasonable placement
(for example, spreading your Pods across nodes so as not place Pods on a node with insufficient free resources).
However, there are some circumstances where you may want to control which node
the Pod deploys to, for example, to ensure that a Pod ends up on a node with an SSD attached to it,
or to co-locate Pods from two different services that communicate a lot into the same availability zone.
You can use any of the following methods to choose where Kubernetes schedules
specific Pods:
field matching against
node labels
Affinity and anti-affinity
nodeName
field
Node labels
Like many other Kubernetes objects, nodes have
labels
. You can
attach labels manually
.
Kubernetes also populates a standard set of labels on all nodes in a cluster. See
for a list of common node labels.
Note:
The value of these labels is cloud provider specific and is not guaranteed to be reliable.
For example, the value of
may be the same as the node name in some environments
and a different value in other environments.
Node isolation/restriction
Adding labels to nodes allows you to target Pods for scheduling on specific
nodes or groups of nodes. You can use this functionality to ensure that specific
Pods only run on nodes with certain isolation, security, or regulatory
properties.
If you use labels for node isolation, choose label keys that the
kubelet
cannot modify. This prevents a compromised node from setting those labels on
itself so that the scheduler schedules workloads onto the compromised node.
The
prevents the kubelet from setting or modifying labels with a
prefix.
To make use of that label prefix for node isolation:
Ensure you are using the
and have
enabled
the
Add labels with the
prefix to your nodes, and use those labels in your
.
example.com.node-restriction.kubernetes.io/fips=true
or
example.com.node-restriction.kubernetes.io/pci-dss=true
.
is the simplest recommended form of node selection constraint.
field to your Pod specification and specify the
you want the target node to have.
Kubernetes only schedules the Pod onto nodes that have each of the labels you
specify.
See
for more
is the simplest way to constrain Pods to nodes with specific
labels. Affinity and anti-affinity expands the types of constraints you can
define. Some of the benefits of affinity and anti-affinity include:
The affinity/anti-affinity language is more expressive.
only
selects nodes with all the specified labels. Affinity/anti-affinity gives you

more control over the selection logic.
You can indicate that a rule is
soft
or
preferred
, so that the scheduler
still schedules the Pod even if it can't find a matching node.
You can constrain a Pod using labels on other Pods running on the node (or other topological domain),
instead of just node labels, which allows you to define rules for which Pods
can be co-located on a node.
The affinity feature consists of two types of affinity:
Node affinity
functions like the
field but is more expressive and
allows you to specify soft rules.
Inter-pod affinity/anti-affinity
allows you to constrain Pods against labels
on other Pods.
Node affinity is conceptually similar to
, allowing you to constrain which nodes your
Pod can be scheduled on based on node labels. There are two types of node
affinity:
: The scheduler can't
schedule the Pod unless the rule is met. This functions like
,
but with a more expressive syntax.
: The scheduler tries to
find a node that meets the rule. If a matching node is not available, the
scheduler still schedules the Pod.
Note:
In the preceding types,
IgnoredDuringExecution
means that if the node labels
change after Kubernetes schedules the Pod, the Pod continues to run.
You can specify node affinities using the
.spec.affinity.nodeAffinity
field in
your Pod spec.
For example, consider the following Pod spec:
pods/pod-with-node-affinity.yaml
:
v1
kind
:
Pod
metadata
:
name
:
with-node-affinity
spec
:
affinity
:
:
:
:
-
:
-
key
:
operator
:
In
values
:
- antarctica-east1
- antarctica-west1
:
-
weight
:
1
:
:
-
key
:
another-node-label-key
operator
:
In
values
:
- another-node-label-value
:
-
name
:
image
:
In this example, the following rules apply:
The node
must
have a label with the key
and
the value of that label
must
be either
antarctica-east1
or
antarctica-west1
.
The node
preferably
has a label with the key
and
the value
another-node-label-value
.
operator
field to specify a logical operator for Kubernetes to use when
interpreting the rules. You can use
In
,
NotIn
,
Exists
,
,
Gt
and
Lt
.
NotIn
and
allow you to define node anti-affinity behavior.
to repel Pods from specific nodes.
Note:
If you specify both
and
,
both
must be satisfied
for the Pod to be scheduled onto a node.
If you specify multiple terms in
associated with
types, then the Pod can be scheduled onto a node if one of the specified terms
can be satisfied (terms are ORed).
If you specify multiple expressions in a single
field associated with a
term in
, then the Pod can be scheduled onto a node only
if all the expressions are satisfied (expressions are ANDed).
See
Node affinity weight
weight
between 1 and 100 for each instance of the
affinity type. When the
scheduler finds nodes that meet all the other scheduling requirements of the Pod, the
scheduler iterates through every preferred rule that the node satisfies and adds the
weight
for that expression to a sum.
The final sum is added to the score of other priority functions for the node.
Nodes with the highest total score are prioritized when the scheduler makes a
scheduling decision for the Pod.
pods/pod-with-affinity-anti-affinity.yaml
:
v1
kind
:
Pod
metadata
:
name
:

with-affinity-anti-affinity
spec
:
affinity
:
:
:
:
-
:
-
key
:
operator
:
In
values
:
- linux
:
-
weight
:
1
:
:
-
key
:
label-1
operator
:
In
values
:
- key-1
-
weight
:
50
:
:
-
key
:
label-2
operator
:
In
values
:
- key-2
:
-
name
:
image
:
If there are two possible nodes that match the
rule, one with the
label-1:key-1
label and another with the
label-2:key-2
label, the scheduler
considers the
weight
of each node and adds the weight to the other scores for
that node, and schedules the Pod onto the node with the highest final score.
Note:
If you want Kubernetes to successfully schedule the Pods in this example, you
must have existing nodes with the
kubernetes.io/os=linux
label.
Node affinity per scheduling profile
When configuring multiple
scheduling profiles
, you can associate
a profile with a node affinity, which is useful if a profile only applies to a specific set of nodes.
To do so, add an
to the
args
plugin
in the
:
kind
:
profiles
:
-
:
-
:
foo-scheduler
:
-
name
:
args
:
:
:
:
-
:
-
key
:
scheduler-profile
operator
:
In
values
:
- foo
The
is applied to all Pods that set
to
, in addition to the
NodeAffinity specified in the PodSpec.
That is, in order to match the Pod, nodes need to satisfy
and
the Pod's
.spec.NodeAffinity
.
Since the
is not visible to end users, its behavior might be
unexpected to them. Use node labels that have a clear correlation to the
scheduler profile name.
Note:
The DaemonSet controller, which
creates Pods for DaemonSets
,
does not support scheduling profiles. When the DaemonSet controller creates
Pods, the default Kubernetes scheduler places those Pods and honors any
rules in the DaemonSet controller.
Inter-pod affinity and anti-affinity
Inter-pod affinity and anti-affinity allow you to constrain which nodes your
Pods can be scheduled on based on the labels of
Pods
already running on that
node, instead of the node labels.
Inter-pod affinity and anti-affinity rules take the form "this
Pod should (or, in the case of anti-affinity, should not) run in an X if that X
is already running one or more Pods that meet rule Y", where X is a topology
domain like node, rack, cloud provider zone or region, or similar and Y is the
rule Kubernetes tries to satisfy.
You express these rules (Y) as
with an optional associated list of namespaces. Pods are namespaced objects in
Kubernetes, so Pod labels also implicitly have namespaces. Any label selectors
for Pod labels should specify the namespaces in which Kubernetes should look for those
labels.
You express the topology domain (X) using a
, which is the key for
the node label that the system uses to denote the domain. For examples, see
.
Note:
Inter-pod affinity and anti-affinity require substantial amount of
processing which can slow down scheduling in large clusters significantly. We do
not recommend using them in clusters larger than several hundred nodes.
Note:
Pod anti-affinity requires nodes to be consistently labelled, in other words,
every node in the cluster must have an appropriate label matching
.
If some or all nodes are missing the specified
label, it can lead
to unintended behavior.
Types of inter-pod

 affinity and anti-affinity
Similar to
are two types of Pod affinity and
anti-affinity as follows:
For example, you could use
affinity to tell the scheduler to
co-locate Pods of two services in the same cloud provider zone because they
communicate with each other a lot. Similarly, you could use
anti-affinity to spread Pods
from a service across multiple cloud provider zones.
To use inter-pod affinity, use the
field in the Pod spec.
For inter-pod anti-affinity, use the
field in the Pod
spec.
Pod affinity example
Consider the following Pod spec:
pods/pod-with-pod-affinity.yaml
:
v1
kind
:
Pod
metadata
:
name
:
with-pod-affinity
spec
:
affinity
:
:
:
-
:
:
-
key
:
security
operator
:
In
values
:
- S1
:
:
:
-
weight
:
100
:
:
:
-
key
:
security
operator
:
In
values
:
- S2
:
:
-
name
:
image
:
This example defines one Pod affinity rule and one Pod anti-affinity rule. The
Pod affinity rule uses the "hard"
, while the anti-affinity rule
uses the "soft"
.
The affinity rule says that the scheduler can only schedule a Pod onto a node if
the node is in the same zone as one or more existing Pods with the label
security=S1
. More precisely, the scheduler must place the Pod on a node that has the
topology.kubernetes.io/zone=V
label, as long as there is at least one node in
that zone that currently has one or more Pods with the Pod label
.
The anti-affinity rule says that the scheduler should try to avoid scheduling
the Pod onto a node that is in the same zone as one or more Pods with the label
security=S2
. More precisely, the scheduler should try to avoid placing the Pod on a node that has the
topology.kubernetes.io/zone=R
label if there are other nodes in the
same zone currently running Pods with the
Security=S2
Pod label.
To get yourself more familiar with the examples of Pod affinity and anti-affinity,
.
In
,
NotIn
,
Exists
and
values in the
operator
field for Pod affinity and anti-affinity.
In principle, the
can be any allowed label key with the following
exceptions for performance and security reasons:
For Pod affinity and anti-affinity, an empty
field is not allowed in both
and
.
For
Pod anti-affinity rules,
the admission controller
limits
to
. You can modify or disable the
admission controller if you want to allow custom topologies.
and
, you can optionally specify a list
of namespaces which the
should match against using the
field at the same level as
and
.
If omitted or empty,
defaults to the namespace of the Pod where the
affinity/anti-affinity definition appears.
Namespace selector
You can also select matching namespaces using
, which is a label query over the set of namespaces.
The affinity term is applied to namespaces selected by both
and the
field.
Note that an empty
({}) matches all namespaces, while a null or empty
list and
null
n

amespaceSelector
matches the namespace of the Pod where the rule is defined.
More practical use-cases
Inter-pod affinity and anti-affinity can be even more useful when they are used with higher
level collections such as ReplicaSets, StatefulSets, Deployments, etc. These
rules allow you to configure that a set of workloads should
be co-located in the same defined topology; for example, preferring to place two related
Pods onto the same node.
For example: imagine a three-node cluster. You use the cluster to run a web application
and also an in-memory cache (such as Redis). For this example, also assume that latency between
the web application and the memory cache should be as low as is practical. You could use inter-pod
affinity and anti-affinity to co-locate the web servers with the cache as much as possible.
In the following example Deployment for the Redis cache, the replicas get the label
app=store
. The
rule tells the scheduler to avoid placing multiple replicas
with the
app=store
label on a single node. This creates each cache in a
separate node.
:
apps/v1
kind
:
metadata
:
name
:
redis-cache
spec
:
selector
:
:
app
:
store
replicas
:
3
template
:
metadata
:
labels
:
app
:
store
spec
:
affinity
:
:
:
-
:
:
-
key
:
app
operator
:
In
values
:
- store
:
:
-
name
:
redis-server
image
:
redis:3.2-alpine
The following example Deployment for the web servers creates replicas with the label
app=web-store
.
The Pod affinity rule tells the scheduler to place each replica on a node that has a Pod
with the label
app=store
. The Pod anti-affinity rule tells the scheduler never to place
multiple
servers on a single node.
:
apps/v1
kind
:
metadata
:
name
:
web-server
spec
:
selector
:
:
app
:
web-store
replicas
:
3
template
:
metadata
:
labels
:
app
:
web-store
spec
:
affinity
:
:
:
-
:
:
-
key
:
app
operator
:
In
values
:
- web-store
:
:
:
-
:
:
-
key
:
app
operator
:
In
values
:
- store
:
:
-
name
:
web-app
image
:
nginx:1.16-alpine
Creating the two preceding Deployments results in the following cluster layout,
where each web server is co-located with a cache, on three separate nodes.
node-1
node-2
node-3
webserver-1
webserver-2
webserver-3
cache-1
cache-2
cache-3
The overall effect is that each cache instance is likely to be accessed by a single client, that
is running on the same node. This approach aims to minimize both skew (imbalanced load) and latency.
You might have other reasons to use Pod anti-affinity.
See the
ZooKeeper tutorial
for an example of a StatefulSet configured with anti-affinity for high
availability, using the same technique as this example.
nodeName
nodeName
is a more direct form of node selection than affinity or
.
nodeName
is a field in the Pod spec. If the
nodeName
field
is not empty, the scheduler ignores the Pod and the kubelet on the named node
tries to place the Pod on that node. Using
nodeName
overrules using
or affinity and anti-affinity rules.
Some of the limitations of using
nodeName
to select nodes are:
If the named node does not exist, the Pod will not run, and in
some cases may be automatically deleted.
If the named node does not have the resources to accommodate the
Pod, the Pod will fail and its reason will indicate why,
for example OutOfmemory or OutOfcpu.
Node names in cloud environments are not always predictable or
stable.
Here is an example of a Pod spec using the
nodeName
field:
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
-
name
:
nginx
imag

e
:
nginx
nodeName
:
kube-01
The above Pod will only run on the node
kube-01
.
You can use
topology spread constraints
to control how
Pods
are spread across your cluster among failure-domains such as regions, zones, nodes, or among any other
topology domains that you define. You might do this to improve performance, expected availability, or
overall utilization.
Read
to learn more about how these work.
.
Read the design docs for
and for
inter-pod affinity/anti-affinity
.
Learn about how the
topology manager
takes part in node-level
Learn how to use
.
affinity and anti-affinity
.
11.3 - Pod Overhead
When you run a Pod on a Node, the Pod itself takes an amount of system resources. These
resources are additional to the resources needed to run the container(s) inside the Pod.
is a way to account for the resources consumed by the Pod
infrastructure on top of the container requests &amp; limits.
In Kubernetes, the Pod's overhead is set at
admission
time according to the overhead associated with the Pod's
.
A pod's overhead is considered in addition to the sum of container resource requests when
scheduling a Pod. Similarly, the kubelet will include the Pod overhead when sizing the Pod cgroup,
and when carrying out Pod eviction ranking.
Configuring Pod overhead
You need to make sure a
is utilized which defines the
overhead
field.
Usage example
To work with Pod overhead, you need a RuntimeClass that defines the
overhead
field. As
an example, you could use the following RuntimeClass definition with a virtualization container
runtime that uses around 120MiB per Pod for the virtual machine and the guest OS:
:
kind
:
metadata
:
name
:
kata-fc
handler
:
kata-fc
overhead
:
podFixed
:
memory
:
"120Mi"
cpu
:
"250m"
Workloads which are created which specify the
kata-fc
RuntimeClass handler will take the memory and
cpu overheads into account for resource quota calculations, node scheduling, as well as Pod cgroup sizing.
Consider running the given example workload, test-pod:
:
v1
kind
:
Pod
metadata
:
name
:
test-pod
spec
:
:
kata-fc
:
-
name
:
busybox-ctr
image
:
stdin
:
true
tty
:
true
resources
:
limits
:
cpu
:
500m
memory
:
100Mi
-
name
:
nginx-ctr
image
:
nginx
resources
:
limits
:
cpu
:
1500m
memory
:
100Mi
At admission time the RuntimeClass
updates the workload's PodSpec to include the
overhead
as described in the RuntimeClass. If the PodSpec already has this field defined,
the Pod will be rejected. In the given example, since only the RuntimeClass name is specified, the admission controller mutates the Pod
to include an
overhead
.
After the RuntimeClass admission controller has made modifications, you can check the updated
Pod overhead value:
kubectl get pod test-pod -o
jsonpath
=
'{.spec.overhead}'
map[cpu:250m memory:120Mi]
If a
is defined, the sum of container requests as well as the
overhead
field are counted.
When the kube-scheduler is deciding which node should run a new Pod, the scheduler considers that Pod's
overhead
as well as the sum of container requests for that Pod. For this example, the scheduler adds the
requests and the overhead, then looks for a node that has 2.25 CPU and 320 MiB of memory available.
Once a Pod is scheduled to a node, the kubelet on that node creates a new
cgroup
for the Pod. It is within this pod that the underlying
container runtime will create containers.
If the resource has a limit defined for each container (Guaranteed QoS or Burstable QoS with limits defined),
the kubelet will set an upper limit for the pod cgroup associated with that resource (cpu.cfs_quota_us for CPU
and memory.l

imit_in_bytes memory). This upper limit is based on the sum of the container limits plus the
overhead
defined in the PodSpec.
For CPU, if the Pod is Guaranteed or Burstable QoS, the kubelet will set
cpu.shares
sum of container requests plus the
overhead
Looking at our example, verify the container requests for the workload:
jsonpath
=
'{.spec.containers[*].resources.limits}'
The total container requests are 2000m CPU and 200MiB of memory:
map[cpu: 500m memory:100Mi] map[cpu:1500m memory:100Mi]
Check this against what is observed by the node:
kubectl describe node | grep test-pod -B2
The output shows requests for 2250m CPU, and for 320MiB of memory. The requests include Pod overhead:
Namespace    Name       CPU Requests  CPU Limits   Memory Requests  Memory Limits  AGE
  ---------    ----       ------------  ----------   ---------------  -------------  ---
  default      test-pod   2250m (56%)   2250m (56%)  320Mi (1%)       320Mi (1%)     36m
Verify Pod cgroup limits
Check the Pod's memory cgroups on the node where the workload is running. In the following example,
crictl
is used on the node, which provides a CLI for CRI-compatible container runtimes. This is an
advanced example to show Pod overhead behavior, and it is not expected that users should need to check
cgroups directly on the node.
First, on the particular node, determine the Pod identifier:
# Run this on the node where the Pod is scheduled
POD_ID
=
"
$(
sudo crictl pods --name test-pod -q
)
"
From this, you can determine the cgroup path for the Pod:
sudo crictl inspectp -o
=
json
$POD_ID
| grep cgroupsPath
The resulting cgroup path includes the Pod's
pause
container. The Pod level cgroup is one directory above.
"cgroupsPath": "/kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2/7ccf55aee35dd16aca4189c952d83487297f3cd760f1bbf09620e206e7d0c27a"
In this specific case, the pod cgroup path is
kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2
.
Verify the Pod level cgroup setting for memory:
# Run this on the node where the Pod is scheduled.
# Also, change the name of the cgroup to match the cgroup allocated for your pod.
cat /sys/fs/cgroup/memory/kubepods/podd7f4b509-cf94-4951-9417-d1087c92a5b2/memory.limit_in_bytes
This is 320 MiB, as expected:
335544320
Some
kube_pod_overhead_*
metrics are available in
kube-state-metrics
to help identify when Pod overhead is being utilized and to help observe stability of workloads
running with a defined overhead.
Read the
PodOverhead Design
enhancement proposal for extra context
11.4 - Pod Scheduling Readiness
Pods were considered ready for scheduling once created. Kubernetes scheduler
does its due diligence to find nodes to place all pending Pods. However, in a
real-world case, some Pods may stay in a "miss-essential-resources" state for a long period.
These Pods actually churn the scheduler (and downstream integrators like Cluster AutoScaler)
in an unnecessary manner.
By specifying/removing a Pod's
.spec.schedulingGates
, you can control when a Pod is ready
to be considered for scheduling.
Configuring Pod schedulingGates
The
field contains a list of strings, and each string literal is perceived as a
criteria that Pod should be satisfied before considered schedulable. This field can be initialized
only when a Pod is created (either by the client, or mutated during admission). After creation,
each schedulingGate can be removed in arbitrary order, but addition of a new scheduling gate is disallowed.
stateDiagram-v2
s1: pod created
s2: pod scheduling gated
s3: pod scheduling ready
s4: pod running
if: empty scheduling gates?
[*] --&gt; s1
s1 --&gt; if
s2 --&gt; if: scheduling gate removed
if --&gt; s2: no
if --&gt; s3: yes
s3 --&gt; s4
s4 --&gt; [*]
enabled
To mar

k a Pod not-ready for scheduling, you can create it with one or more scheduling gates like this:
pods/pod-with-scheduling-gates.yaml
:
v1
kind
:
Pod
metadata
:
name
:
test-pod
spec
:
:
-
name
:
foo
-
name
:
bar
:
-
name
:
pause
image
:
After the Pod's creation, you can check its state using:
kubectl get pod test-pod
The output reveals it's in
SchedulingGated
state:
NAME       READY   STATUS            RESTARTS   AGE
test-pod   0/1     SchedulingGated   0          7s
You can also check its
field by running:
jsonpath
=
'{.spec.schedulingGates}'
[{"name":"foo"},{"name":"bar"}]
To inform scheduler this Pod is ready for scheduling, you can remove its
entirely
by re-applying a modified manifest:
pods/pod-without-scheduling-gates.yaml
:
v1
kind
:
Pod
metadata
:
name
:
test-pod
spec
:
:
-
name
:
pause
image
:
You can check if the
is cleared by running:
jsonpath
=
The output is expected to be empty. And you can check its latest status by running:
kubectl get pod test-pod -o wide
Given the test-pod doesn't request any CPU/memory resources, it's expected that this Pod's state get
transited from previous
to
Running
:
NAME       READY   STATUS    RESTARTS   AGE   IP         NODE  
test-pod   1/1     Running   0          15s   10.0.0.4   node-2
The metric
comes with a new label
"gated"
to distinguish whether a Pod
has been tried scheduling but claimed as unschedulable, or explicitly marked as not ready for
scheduling. You can use
scheduler_pending_pods{queue="gated"}
to check the metric result.
Read the
PodSchedulingReadiness KEP
11.5 - Pod Topology Spread Constraints
You can use
Pods
are spread across your cluster
among failure-domains such as regions, zones, nodes, and other user-defined topology
domains. This can help to achieve high availability as well as efficient resource
utilization.
You can set
cluster-level constraints
as a default,
or configure topology spread constraints for individual workloads.
Imagine that you have a cluster of up to twenty nodes, and you want to run a
workload
that automatically scales how many replicas it uses. There could be as few as
two Pods or as many as fifteen.
When there are only two Pods, you'd prefer not to have both of those Pods run on the
same node: you would run the risk that a single node failure takes your workload
offline.
In addition to this basic usage, there are some advanced usage examples that
enable your workloads to benefit on high availability and cluster utilization.
As you scale up and run more Pods, a different concern becomes important. Imagine
that you have three nodes running five Pods each. The nodes have enough capacity
to run that many replicas; however, the clients that interact with this workload
are split across three different datacenters (or infrastructure zones). Now you
have less concern about a single node failure, but you notice that latency is
higher than you'd like, and you are paying for network costs associated with
sending network traffic between the different zones.
You decide that under normal operation you'd prefer to have a similar number of replicas
scheduled
into each infrastructure zone,
and you'd like the cluster to self-heal in the case that there is a problem.
Pod topology spread constraints offer you a declarative way to configure that.
field
The Pod API includes a field,
spec.topologySpreadConstraints
. The usage of this field looks like
the following:
---
:
v1
kind
:
Pod
metadata
:
name
:
example-pod
spec
:
# Configure a topology spread constraint
:
-
maxSkew
:
&lt;integer&gt;
:
&lt;integer

&gt;
# optional; beta since v1.25
:
&lt;string&gt;
:
:
&lt;object&gt;
:
&lt;list&gt;
# optional; alpha since v1.25
:
[Honor|Ignore]
# optional; beta since v1.26
:
### other Pod fields go here
You can read more about this field by running
kubectl explain Pod.spec.topologySpreadConstraints
or
refer to
section of the API reference for Pod.
Spread constraint definition
You can define one or multiple
entries to instruct the
kube-scheduler how to place each incoming Pod in relation to the existing Pods across
your cluster. Those fields are:
maxSkew
describes the degree to which Pods may be unevenly distributed. You must
specify this field and the number must be greater than zero. Its semantics differ
according to the value of
:
if you select
whenUnsatisfiable: DoNotSchedule
, then
maxSkew
defines the
maximum permitted difference between the number of matching pods in the target
topology and the
global minimum
(the minimum number of matching pods in an eligible domain or zero if the number of eligible domains is less than MinDomains).
For example, if you have 3 zones with 2, 2 and 1 matching pods respectively,
MaxSkew
is set to 1 then the global minimum is 1.
whenUnsatisfiable: ScheduleAnyway
, the scheduler gives higher
precedence to topologies that would help reduce the skew.
indicates a minimum number of eligible domains. This field is optional.
A domain is a particular instance of a topology. An eligible domain is a domain whose
nodes match the node selector.
Note:
The
field is a beta field and disabled by default in 1.25. You can enable it by enabling the
.
must be greater than 0, when specified.
You can only specify
.
When the number of eligible domains with match topology keys is less than
,
Pod topology spread treats global minimum as 0, and then the calculation of
skew
is performed.
The global minimum is the minimum number of matching Pods in an eligible domain,
or zero if the number of eligible domains is less than
.
When the number of eligible domains with matching topology keys equals or is greater than
, this value has no effect on scheduling.
, the constraint behaves as if
is 1.
is the key of
. Nodes that have a label with this key
and identical values are considered to be in the same topology.
We call each instance of a topology (in other words, a &lt;key, value&gt; pair) a domain. The scheduler
will try to put a balanced number of pods into each domain.
Also, we define an eligible domain as a domain whose nodes meet the requirements of
nodeAffinityPolicy and nodeTaintsPolicy.
indicates how to deal with a Pod if it doesn't satisfy the spread constraint:
DoNotSchedule
(default) tells the scheduler not to schedule it.
ScheduleAnyway
tells the scheduler to still schedule it while prioritizing nodes that minimize the skew.
is used to find matching Pods. Pods
that match this label selector are counted to determine the
number of Pods in their corresponding topology domain.
See
Label Selectors
is a list of pod label keys to select the pods over which
spreading will be calculated. The keys are used to lookup values from the pod labels, those key-value labels are ANDed with
to select the group of existing pods over which spreading will be calculated for the incoming pod. Keys that don't exist in the pod labels will be ignored. A null or empty list means only match against the
.
With
, users don't need to update the
pod.spec
between different revisions. T

he controller/operator just needs to set different values to the same
label
key for different revisions. The scheduler will assume the values automatically based on
. For example, if users use Deployment, they can use the label keyed with
, which is added automatically by the Deployment controller, to distinguish between different revisions in a single Deployment.
:
-
maxSkew
:
1
:
:
:
- app
- pod-template-hash
Note:
The
field is an alpha field added in 1.25. You have to enable the
in order to use it.
indicates how we will treat Pod's nodeAffinity/nodeSelector
when calculating pod topology spread skew. Options are:
Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations.
Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations.
If this value is null, the behavior is equivalent to the Honor policy.
Note:
The
is a beta-level field and enabled by default in 1.26. You can disable it by disabling the
.
indicates how we will treat node taints when calculating
pod topology spread skew. Options are:
Honor: nodes without taints, along with tainted nodes for which the incoming pod
has a toleration, are included.
Ignore: node taints are ignored. All nodes are included.
If this value is null, the behavior is equivalent to the Ignore policy.
Note:
The
.
When a Pod defines more than one
topologySpreadConstraint
, those constraints are
combined using a logical AND operation: the kube-scheduler looks for a node for the incoming Pod
that satisfies all the configured constraints.
Topology spread constraints rely on node labels to identify the topology
domain(s) that each
node
is in.
For example, a node might have labels:
region
:
us-east-1
zone
:
us-east-1a
Note:
For brevity, this example doesn't use the
well-known
label keys
and
. However,
those registered label keys are nonetheless recommended rather than the private
(unqualified) label keys
region
and
zone
that are used here.
You can't make a reliable assumption about the meaning of a private label key
between different contexts.
Suppose you have a 4-node cluster with the following labels:
NAME    STATUS   ROLES    AGE     VERSION   LABELS
node1   Ready    &lt;none&gt;   4m26s   v1.16.0   node=node1,zone=zoneA
node2   Ready    &lt;none&gt;   3m58s   v1.16.0   node=node2,zone=zoneA
node3   Ready    &lt;none&gt;   3m17s   v1.16.0   node=node3,zone=zoneB
node4   Ready    &lt;none&gt;   2m43s   v1.16.0   node=node4,zone=zoneB
Then the cluster is logically viewed as below:
graph TB
n3(Node3)
n4(Node4)
end
n1(Node1)
n2(Node2)
end
enabled
Consistency
You should set the same Pod topology spread constraints on all pods in a group.
Usually, if you are using a workload controller such as a Deployment, the pod template
takes care of this for you. If you mix different spread constraints then Kubernetes
follows the API definition of the field; however, the behavior is more likely to become
confusing and troubleshooting is less straightforward.
You need a mechanism to ensure that all the nodes in a topology domain (such as a
cloud provider region) are labelled consistently.
To avoid you needing

 to manually label nodes, most clusters automatically
populate well-known labels such as
topology.kubernetes.io/hostname
. Check whether
your cluster supports this.
Topology spread constraint examples
Example: one topology spread constraint
Suppose you have a 4-node cluster where 3 Pods labelled
foo: bar
are located in
node1, node2 and node3 respectively:
graph BT
p3(Pod) --&gt; n3(Node3)
n4(Node4)
end
p1(Pod) --&gt; n1(Node1)
p2(Pod) --&gt; n2(Node2)
end
class n1,n2,n3,n4,p1,p2,p3 k8s;
enabled
If you want an incoming Pod to be evenly spread with existing Pods across zones, you
can use a manifest similar to:
pods/topology-spread-constraints/one-constraint.yaml
kind
:
Pod
:
v1
metadata
:
name
:
mypod
labels
:
foo
:
bar
spec
:
:
-
maxSkew
:
1
:
zone
:
:
:
foo
:
bar
:
-
name
:
pause
image
:
registry.k8s.io/pause:3.1
From that manifest,
topologyKey: zone
implies the even distribution will only be applied
to nodes that are labelled
zone: &lt;any value&gt;
(nodes that don't have a
zone
label
are skipped). The field
tells the scheduler to let the
incoming Pod stay pending if the scheduler can't find a way to satisfy the constraint.
If the scheduler placed this incoming Pod into zone
A
, the distribution of Pods would
become
[3, 1]
. That means the actual skew is then 2 (calculated as
3 - 1
), which
violates
maxSkew: 1
. To satisfy the constraints and context for this example, the
incoming Pod can only be placed onto a node in zone
B
:
graph BT
p4(mypod) --&gt; n4(Node4)
end
end
class p4 plain;
enabled
OR
graph BT
p4(mypod) --&gt; n3
n4(Node4)
end
end
enabled
You can tweak the Pod spec to meet various kinds of requirements:
Change
maxSkew
to a bigger value - such as
2
- so that the incoming Pod can
be placed into zone
A
as well.
Change
to
node
so as to distribute the Pods evenly across nodes
instead of zones. In the above example, if
maxSkew
remains
1
, the incoming
Pod can only be placed onto the node
node4
.
Change
to
to ensure the incoming Pod to be always schedulable (suppose other scheduling APIs
are satisfied). However, it's preferred to be placed into the topology domain which
has fewer matching Pods. (Be aware that this preference is jointly normalized
with other internal scheduling priorities such as resource usage ratio).
Example: multiple topology spread constraints
This builds upon the previous example. Suppose you have a 4-node cluster where 3
existing Pods labeled
foo: bar
are located on node1, node2 and node3 respectively:
graph BT
n4(Node4)
end
end
classDef plain 

fill:#ddd,stroke:#fff,stroke-width:4px,color:#000;
enabled
You can combine two topology spread constraints to control the spread of Pods both
by node and by zone:
pods/topology-spread-constraints/two-constraints.yaml
kind
:
Pod
:
v1
metadata
:
name
:
mypod
labels
:
foo
:
bar
spec
:
:
-
maxSkew
:
1
:
zone
:
:
:
foo
:
bar
-
maxSkew
:
1
:
node
:
:
:
foo
:
bar
:
-
name
:
pause
image
:
In this case, to match the first constraint, the incoming Pod can only be placed onto
B
; while in terms of the second constraint, the incoming Pod can only be
scheduled to the node
node4
. The scheduler only considers options that satisfy all
defined constraints, so the only valid placement is onto node
node4
.
Example: conflicting topology spread constraints
Multiple constraints can lead to conflicts. Suppose you have a 3-node cluster across 2 zones:
graph BT
p4(Pod) --&gt; n3(Node3)
p5(Pod) --&gt; n3
end
p2(Pod) --&gt; n1
p3(Pod) --&gt; n2(Node2)
end
class n1,n2,n3,n4,p1,p2,p3,p4,p5 k8s;
enabled
If you were to apply
two-constraints.yaml
(the manifest from the previous example)
to
this
cluster, you would see that the Pod
mypod
stays in the
Pending
state.
This happens because: to satisfy the first constraint, the Pod
mypod
can only
B
; while in terms of the second constraint, the Pod
mypod
can only schedule to node
node2
. The intersection of the two constraints returns
an empty set, and the scheduler cannot place the Pod.
To overcome this situation, you can either increase the value of
maxSkew
or modify
one of the constraints to use
. Depending on
circumstances, you might also decide to delete an existing Pod manually - for example,
if you are troubleshooting why a bug-fix rollout is not making progress.
Interaction with node affinity and node selectors
The scheduler will skip the non-matching nodes from the skew calculations if the
incoming Pod has
spec.nodeSelector
or
spec.affinity.nodeAffinity
defined.
Example: topology spread constraints with node affinity
Suppose you have a 5-node cluster ranging across zones A to C:
graph BT
n4(Node4)
end
end
enabled
graph BT
subgraph "zoneC"
n5(Node5)
end
class n5 k8s;
class zoneC cluster;
enabled
and you know that zone
C
must be excluded. In this case, you can compose a manifest
as below, so that Pod
mypod
will be placed into zone
B
instead of zone
C
.
Similarly, Kubernetes also respects
.
pods/topology-spread-constraints/one-constraint-with-nodeaffinity.yaml
kind
:
Pod
:

v1
metadata
:
name
:
mypod
labels
:
foo
:
bar
spec
:
:
-
maxSkew
:
1
:
zone
:
:
:
foo
:
bar
affinity
:
:
:
:
-
:
-
key
:
zone
operator
:
NotIn
values
:
- zoneC
:
-
name
:
pause
image
:
Implicit conventions
There are some implicit conventions worth noting here:
Only the Pods holding the same namespace as the incoming Pod can be matching candidates.
The scheduler bypasses any nodes that don't have any
topologySpreadConstraints[*].topologyKey
present. This implies that:
any Pods located on those bypassed nodes do not impact
maxSkew
calculation - in the
above example, suppose the node
node1
does not have a label "zone", then the 2 Pods will
be disregarded, hence the incoming Pod will be scheduled into zone
A
.
the incoming Pod has no chances to be scheduled onto this kind of nodes -
in the above example, suppose a node
node5
has the
mistyped
label
zone-typo: zoneC
(and no
zone
label set). After node
node5
joins the cluster, it will be bypassed and
Pods for this workload aren't scheduled there.
Be aware of what will happen if the incoming Pod's
topologySpreadConstraints[*].labelSelector
doesn't match its own labels. In the
above example, if you remove the incoming Pod's labels, it can still be placed onto
B
, since the constraints are still satisfied. However, after that
placement, the degree of imbalance of the cluster remains unchanged - it's still zone
A
having 2 Pods labelled as
foo: bar
, and zone
B
having 1 Pod labelled as
foo: bar
. If this is not what you expect, update the workload's
to match the labels in the pod template.
Cluster-level default constraints
It is possible to set default topology spread constraints for a cluster. Default
topology spread constraints are applied to a Pod if, and only if:
It doesn't define any constraints in its
.spec.topologySpreadConstraints
.
It belongs to a Service, ReplicaSet, StatefulSet or ReplicationController.
Default constraints can be set as part of the
plugin
arguments in a
scheduling profile
.
The constraints are specified with the same
API above
must be empty. The selectors are calculated from the Services,
ReplicaSets, StatefulSets or ReplicationControllers that the Pod belongs to.
An example configuration might look like follows:
:
kind
:
profiles
:
-
:
:
-
name
:
args
:
:
-
maxSkew
:
1
:
:
:
List
Note:
The
plugin
is disabled by default. The Kubernetes project recommends using
Built-in default constraints
If you don't configure any cluster-level default constraints for pod topology spreading,
then kube-scheduler acts as if you specified the following default topology constraints:
:
-
maxSkew
:
3
:
:
-
maxSkew
:
5
:
:
Also, the legacy
plugin, which provides an equivalent behavior,
is disabled by default.
Note:
The
plugin does not score the nodes that don't have
the topology keys specified in the spreading constraints. This might result
in a different default behavior compared to the legacy
plugin when
using the default topology constraints.
If your nodes are not expected to have
both
and
labels set, define your own constraints
instead of using the Kuberne

tes defaults.
If you don't want to use the default Pod spreading constraints for your cluster,
you can disable those defaults by setting
to
List
and leaving
empty
in the
plugin configuration:
:
kind
:
profiles
:
-
:
:
-
name
:
args
:
:
[]
:
List
Comparison with podAffinity and podAntiAffinity
control how Pods are scheduled in relation to one another - either more packed
or more scattered.
attracts Pods; you can try to pack any number of Pods into qualifying
topology domain(s).
repels Pods. If you set this to
mode then
only a single Pod can be scheduled into a single topology domain; if you choose
then you lose the ability to enforce the
constraint.
For finer control, you can specify topology spread constraints to distribute
Pods across different topology domains - to achieve either high availability or
cost-saving. This can also help on rolling update workloads and scaling out
replicas smoothly.
For more context, see the
section of the enhancement proposal about Pod topology spread constraints.
Known limitations
There's no guarantee that the constraints remain satisfied when Pods are removed. For
example, scaling down a Deployment may result in imbalanced Pods distribution.
You can use a tool such as the
to rebalance the Pods distribution.
Pods matched on tainted nodes are respected.
See
Issue 80921
.
The scheduler doesn't have prior knowledge of all the zones or other topology
domains that a cluster has. They are determined from the existing nodes in the
cluster. This could lead to a problem in autoscaled clusters, when a node pool (or
node group) is scaled to zero nodes, and you're expecting the cluster to scale up,
because, in this case, those topology domains won't be considered until there is
at least one node in them.
You can work around this by using an cluster autoscaling tool that is aware of
Pod topology spread constraints and is also aware of the overall set of topology
domains.
Introducing PodTopologySpread
explains
maxSkew
in some detail, as well as covering some advanced usage examples.
Read the
section of
the API reference for Pod.
11.6 - Taints and Tolerations
is a property of
Pods
that
attracts
them to
a set of
nodes
(either as a preference or a
hard requirement).
Taints
are the opposite -- they allow a node to repel a set of pods.
Tolerations
are applied to pods. Tolerations allow the scheduler to schedule pods with matching
taints. Tolerations allow scheduling but don't guarantee scheduling: the scheduler also
evaluates other parameters
as part of its function.
Taints and tolerations work together to ensure that pods are not scheduled
onto inappropriate nodes. One or more taints are applied to a node; this
marks that the node should not accept any pods that do not tolerate the taints.
Concepts
You add a taint to a node using
.
kubectl taint nodes node1
key1
=
value1:NoSchedule
places a taint on node
node1
. The taint has key
key1
, value
value1
, and taint effect
.
This means that no pod will be able to schedule onto
node1
unless it has a matching toleration.
To remove the taint added by the command above, you can run:
key1
=
value1:NoSchedule-
You specify a toleration for a pod in the PodSpec. Both of the following tolerations "match" the
taint created by the
line above, and thus a pod with either toleration would be able
to schedule onto
node1
:
:
-
key
:
"key1"
operator
:
"Equal"
value
:
"value1"
effect
:
:
-
key
:
"key1"
oper

ator
:
"Exists"
effect
:
Here's an example of a pod that uses tolerations:
pods/pod-with-toleration.yaml
:
v1
kind
:
Pod
metadata
:
name
:
nginx
labels
:
env
:
test
spec
:
:
-
name
:
nginx
image
:
nginx
:
:
-
key
:
"example-key"
operator
:
"Exists"
effect
:
operator
is
Equal
.
A toleration "matches" a taint if the keys are the same and the effects are the same, and:
the
operator
is
Exists
(in which case no
value
should be specified), or
the
operator
is
Equal
and the
value
s are equal.
Note:
There are two special cases:
An empty
key
with operator
Exists
matches all keys, values and effects which means this
will tolerate everything.
An empty
effect
matches all effects with key
key1
.
The above example used
effect
of
. Alternatively, you can use
effect
of
PreferNoSchedule
.
This is a "preference" or "soft" version of
-- the system will
try
to avoid placing a
pod that does not tolerate the taint on the node, but it is not required. The third kind of
effect
is
NoExecute
, described later.
You can put multiple taints on the same node and multiple tolerations on the same pod.
The way Kubernetes processes multiple taints and tolerations is like a filter: start
with all of a node's taints, then ignore the ones for which the pod has a matching toleration; the
remaining un-ignored taints have the indicated effects on the pod. In particular,
if there is at least one un-ignored taint with effect
then Kubernetes will not schedule
the pod onto that node
if there is no un-ignored taint with effect
but there is at least one un-ignored taint with
effect
then Kubernetes will
try
to not schedule the pod onto the node
NoExecute
then the pod will be evicted from
the node (if it is already running on the node), and will not be
scheduled onto the node (if it is not yet running on the node).
For example, imagine you taint a node like this
key1
=
key1
=
value1:NoExecute
key2
=
value2:NoSchedule
And a pod has two tolerations:
:
-
key
:
"key1"
operator
:
"Equal"
value
:
"value1"
effect
:
-
key
:
"key1"
operator
:
"Equal"
value
:
"value1"
effect
:
"NoExecute"
In this case, the pod will not be able to schedule onto the node, because there is no
toleration matching the third taint. But it will be able to continue running if it is
already running on the node when the taint is added, because the third taint is the only
one of the three that is not tolerated by the pod.
Normally, if a taint with effect
NoExecute
is added to a node, then any pods that do
not tolerate the taint will be evicted immediately, and pods that do tolerate the
taint will never be evicted. However, a toleration with
NoExecute
effect can specify
an optional
field that dictates how long the pod will stay bound
to the node after the taint is added. For example,
:
-
key
:
"key1"
operator
:
"Equal"
value
:
"value1"
effect
:
:
3600
means that if this pod is running and a matching taint is added to the node, then
the pod will stay bound to the node for 3600 seconds, and then be evicted. If the
taint is removed before that time, the pod will not be evicted.
Example Use Cases
Taints and tolerations are a flexible way to steer pods
away
from nodes or evict
pods that shouldn't be running. A few of the use cases are
Dedicated Nodes
: If you want to dedicate a set of nodes for exclusive use by
a particular set of users, you can add a taint to those nodes (say,
kubectl taint nodes nodename dedicated=groupName:NoSchedule
) and then add a corresponding
toleration to their pods (this would be done most easily by writing a custom
).
The pods with the tolerations will then be allowed to

 use the tainted (dedicated) nodes as
well as any other nodes in the cluster. If you want to dedicate the nodes to them
and
ensure they
only
use the dedicated nodes, then you should additionally add a label similar
to the taint to the same set of nodes (e.g.
dedicated=groupName
), and the admission
controller should additionally add a node affinity to require that the pods can only schedule
onto nodes labeled with
.
Nodes with Special Hardware
: In a cluster where a small subset of nodes have specialized
hardware (for example GPUs), it is desirable to keep pods that don't need the specialized
hardware off of those nodes, thus leaving room for later-arriving pods that do need the
specialized hardware. This can be done by tainting the nodes that have the specialized
hardware (e.g.
kubectl taint nodes nodename special=true:NoSchedule
or
kubectl taint nodes nodename special=true:PreferNoSchedule
) and adding a corresponding
toleration to pods that use the special hardware. As in the dedicated nodes use case,
it is probably easiest to apply the tolerations using a custom
.
For example, it is recommended to use
Extended
Resources
to represent the special hardware, taint your special hardware nodes with the
extended resource name and run the
admission controller. Now, because the nodes are tainted, no pods without the
toleration will schedule on them. But when you submit a pod that requests the
extended resource, the
admission controller will
automatically add the correct toleration to the pod and that pod will schedule
on the special hardware nodes. This will make sure that these special hardware
nodes are dedicated for pods requesting such hardware and you don't have to
manually add tolerations to your pods.
Taint based Evictions
: A per-pod-configurable eviction behavior
when there are node problems, which is described in the next section.
The
NoExecute
taint effect, mentioned above, affects pods that are already
running on the node as follows
pods that do not tolerate the taint are evicted immediately
pods that tolerate the taint without specifying
in
their toleration specification remain bound forever
pods that tolerate the taint with a specified
remain
bound for the specified amount of time
The node controller automatically taints a Node when certain conditions
are true. The following taints are built in:
: Node is not ready. This corresponds to
the NodeCondition
Ready
being "
False
".
: Node is unreachable from the node
controller. This corresponds to the NodeCondition
Ready
being "
Unknown
".
: Node has memory pressure.
: Node has disk pressure.
: Node has PID pressure.
: Node's network is unavailable.
: Node is unschedulable.
: When the kubelet is started
with "external" cloud provider, this taint is set on a node to mark it
as unusable. After a controller from the cloud-controller-manager initializes
this node, the kubelet removes this taint.
In case a node is to be evicted, the node controller or the kubelet adds relevant taints
with
NoExecute
effect. If the fault condition returns to normal the kubelet or node
controller can remove the relevant taint(s).
Note:
The control plane limits the rate of adding node new taints to nodes. This rate limiting
manages the number of evictions that are triggered when many nodes become unreachable at
once (for example: if there is a network disruption).
for a Pod to define how long that Pod stays bound
to a failing or unresponsive Node.
For example, you might want to keep an application wit

h a lot of local state
bound to node for a long time in the event of network partition, hoping
that the partition will recover and thus the pod eviction can be avoided.
The toleration you set for that Pod might look like:
:
-
key
:
"node.kubernetes.io/unreachable"
operator
:
"Exists"
effect
:
:
6000
Note:
Kubernetes automatically adds a toleration for
and
with
tolerationSeconds=300
,
unless you, or a controller, set those tolerations explicitly.
These automatically-added tolerations mean that Pods remain bound to
Nodes for 5 minutes after one of these problems is detected.
DaemonSet
pods are created with
NoExecute
tolerations for the following taints with no
:
This ensures that DaemonSet pods are never evicted due to these problems.
The control plane, using the node
,
automatically creates taints with a
effect for
.
The scheduler checks taints, not node conditions, when it makes scheduling
decisions. This ensures that node conditions don't directly affect scheduling.
For example, if the
node condition is active, the control plane
adds the
taint and does not schedule new pods
onto the affected node. If the
node condition is active, the
control plane adds the
taint.
You can ignore node conditions for newly created pods by adding the corresponding
Pod tolerations. The control plane also adds the
toleration on pods that have a
QoS class
other than
. This is because Kubernetes treats pods in the
or
Burstable
QoS classes (even pods with no memory request set) as if they are
able to cope with memory pressure, while new
pods are not scheduled
onto the affected node.
The DaemonSet controller automatically adds the following
tolerations to all daemons, to prevent DaemonSets from breaking.
(1.14 or later)
(1.10 or later)
(
host network only
)
Adding these tolerations ensures backward compatibility. You can also add
arbitrary tolerations to DaemonSets.
Read about
and how you can configure it
Read about
11.7 - Scheduling Framework
The scheduling framework is a pluggable architecture for the Kubernetes scheduler.
It adds a new set of "plugin" APIs to the existing scheduler. Plugins are compiled into the scheduler. The APIs allow most scheduling features to be implemented as plugins, while keeping the
scheduling "core" lightweight and maintainable. Refer to the
design proposal of the
scheduling framework
for more technical information on the design of the
framework.
Framework workflow
The Scheduling Framework defines a few extension points. Scheduler plugins
register to be invoked at one or more extension points. Some of these plugins
can change the scheduling decisions and some are informational only.
Each attempt to schedule one Pod is split into two phases, the
cycle
and the
binding cycle
.
Scheduling Cycle &amp; Binding Cycle
The scheduling cycle selects a node for the Pod, and the binding cycle applies
that decision to the cluster. Together, a scheduling cycle and binding cycle are
referred to as a "scheduling context".
Scheduling cycles are run serially, while binding cycles may run concurrently.
A scheduling or binding cycle can be aborted if the Pod is determined to
be unschedulable or if there is an internal error. The Pod will be returned to
the queue and retried.
The following picture shows the scheduling context of a Pod and the extensi

on
points that the scheduling framework exposes. In this picture "Filter" is
equivalent to "Predicate" and "Scoring" is equivalent to "Priority function".
One plugin may register at multiple extension points to perform more complex or
stateful tasks.
scheduling framework extension points
QueueSort
These plugins are used to sort Pods in the scheduling queue. A queue sort plugin
essentially provides a
Less(Pod1, Pod2)
function. Only one queue sort
plugin may be enabled at a time.
PreFilter
These plugins are used to pre-process info about the Pod, or to check certain
conditions that the cluster or the Pod must meet. If a PreFilter plugin returns
an error, the scheduling cycle is aborted.
Filter
These plugins are used to filter out nodes that cannot run the Pod. For each
node, the scheduler will call filter plugins in their configured order. If any
filter plugin marks the node as infeasible, the remaining plugins will not be
called for that node. Nodes may be evaluated concurrently.
PostFilter
These plugins are called after Filter phase, but only when no feasible nodes
were found for the pod. Plugins are called in their configured order. If
any postFilter plugin marks the node as
Schedulable
, the remaining plugins
will not be called. A typical PostFilter implementation is preemption, which
tries to make the pod schedulable by preempting other Pods.
PreScore
These plugins are used to perform "pre-scoring" work, which generates a sharable
state for Score plugins to use. If a PreScore plugin returns an error, the
scheduling cycle is aborted.
Score
These plugins are used to rank nodes that have passed the filtering phase. The
scheduler will call each scoring plugin for each node. There will be a well
defined range of integers representing the minimum and maximum scores. After the
NormalizeScore
phase, the scheduler will combine node
scores from all plugins according to the configured plugin weights.
These plugins are used to modify scores before the scheduler computes a final
ranking of Nodes. A plugin that registers for this extension point will be
called with the
Score
results from the same plugin. This is called
once per plugin per scheduling cycle.
For example, suppose a plugin
BlinkingLightScorer
ranks Nodes based on how
many blinking lights they have.
func
ScoreNode
(_
*
v1.pod, n
*
v1.Node) (
int
,
error
) {
return
getBlinkingLightCount
(n)
}
However, the maximum count of blinking lights may be small compared to
NodeScoreMax
. To fix this,
should also register for this
extension point.
func
NormalizeScores
(scores
map
[
string
]
int
) {
highest
:=
0
for
_, score
:=
range
scores {
highest =
max
(highest, score)
}
for
node, score
:=
range
scores {
scores[node] = score
*
/
highest
}
}
If any NormalizeScore plugin returns an error, the scheduling cycle is
aborted.
Note:
Plugins wishing to perform "pre-reserve" work should use the
NormalizeScore extension point.
Reserve
A plugin that implements the Reserve extension has two methods, namely
Reserve
and
Unreserve
, that back two informational scheduling phases called Reserve
and Unreserve, respectively. Plugins which maintain runtime state (aka "stateful
plugins") should use these phases to be notified by the scheduler when resources
on a node are being reserved and unreserved for a given Pod.
The Reserve phase happens before the scheduler actually binds a Pod to its
designated node. It exists to prevent race conditions while the scheduler waits
for the bind to succeed. The
Reserve
method of each Reserve plugin may succeed
or fail; if one
Reserve
method call fails, subsequent plugins are not executed
and the Reserve phase is considered to have failed. If the
Reserve
method of
all plugins succeed, the Reserve phase is considered to be successful and the
rest of the scheduling cycle and the binding cycle are executed.
The Unreserve phase is triggered if the Reserve phase or a later phase fails.
When this happens, the
Unreserve
method of
a

ll
Reserve plugins will be
executed in the reverse order of
Reserve
method calls. This phase exists to
clean up the state associated with the reserved Pod.
Caution:
The implementation of the
Unreserve
method in Reserve plugins must be
idempotent and may not fail.
Permit
Permit
plugins are invoked at the end of the scheduling cycle for each Pod, to
prevent or delay the binding to the candidate node. A permit plugin can do one of
the three things:
approve
Once all Permit plugins approve a Pod, it is sent for binding.
deny
If any Permit plugin denies a Pod, it is returned to the scheduling queue.
This will trigger the Unreserve phase in
Reserve plugins
.
wait
(with a timeout)
If a Permit plugin returns "wait", then the Pod is kept in an internal "waiting"
Pods list, and the binding cycle of this Pod starts but directly blocks until it
gets approved. If a timeout occurs,
wait
becomes
deny
and the Pod is returned to the scheduling queue, triggering the
Unreserve phase in
.
Note:
While any plugin can access the list of "waiting" Pods and approve them
(see
FrameworkHandle
), we expect only the permit
plugins to approve binding of reserved Pods that are in "waiting" state. Once a Pod
is approved, it is sent to the
PreBind
phase.
PreBind
These plugins are used to perform any work required before a Pod is bound. For
example, a pre-bind plugin may provision a network volume and mount it on the
target node before allowing the Pod to run there.
If any PreBind plugin returns an error, the Pod is
rejected
and
returned to the scheduling queue.
Bind
These plugins are used to bind a Pod to a Node. Bind plugins will not be called
until all PreBind plugins have completed. Each bind plugin is called in the
configured order. A bind plugin may choose whether or not to handle the given
Pod. If a bind plugin chooses to handle a Pod,
the remaining bind plugins are
skipped
.
PostBind
This is an informational extension point. Post-bind plugins are called after a
Pod is successfully bound. This is the end of a binding cycle, and can be used
to clean up associated resources.
Plugin API
There are two steps to the plugin API. First, plugins must register and get
configured, then they use the extension point interfaces. Extension point
interfaces have the following form.
type
Plugin
interface
{
Name
()
string
}
type
QueueSortPlugin
interface
{
Plugin
Less
(
*
v1.pod,
*
v1.pod)
bool
}
type
PreFilterPlugin
interface
{
Plugin
PreFilter
(context.Context,
*
framework.CycleState,
*
v1.pod)
error
}
// ...
Plugin configuration
You can enable or disable plugins in the scheduler configuration. If you are using
Kubernetes v1.18 or later, most scheduling
plugins
are in use and
enabled by default.
In addition to default plugins, you can also implement your own scheduling
plugins and get them configured along with default plugins. You can visit
scheduler-plugins
If you are using Kubernetes v1.18 or later, you can configure a set of plugins as
a scheduler profile and then define multiple profiles to fit various kinds of workload.
Learn more at
.
11.8 - Dynamic Resource Allocation
Dynamic resource allocation is a new API for requesting and sharing resources
between pods and containers inside a pod. It is a generalization of the
persistent volumes API for generic resources. Third-party resource drivers are
responsible for tracking and allocating resources. Different kinds of
resources support arbitrary parameters for defining requirements and
initialization.
dynamic resource allocation, but it
needs to be
enabled
explicitly. You also must
install a resource driver for specific resources that are meant to be managed
using this API. If you are not running Kubernetes v1.26,
check the documentation for that version of Kubernetes.
API
The new
resource.k8s.io/v1alpha1
API group
provides four new types:
Def

ines which resource driver handles a certain kind of
resource and provides common parameters for it. ResourceClasses
are created by a cluster administrator when installing a resource
driver.
Defines a particular resource instances that is required by a
workload. Created by a user (lifecycle managed manually, can be shared
between different Pods) or for individual Pods by the control plane based on
a ResourceClaimTemplate (automatic lifecycle, typically used by just one
Pod).
Defines the spec and some meta data for creating
ResourceClaims. Created by a user when deploying a workload.
Used internally by the control plane and resource drivers
to coordinate pod scheduling when ResourceClaims need to be allocated
for a Pod.
Parameters for ResourceClass and ResourceClaim are stored in separate objects,
typically using the type defined by a
CRD
that was created when
installing a resource driver.
The
core/v1
PodSpec
defines ResourceClaims that are needed for a Pod in a new
field. Entries in that list reference either a ResourceClaim
or a ResourceClaimTemplate. When referencing a ResourceClaim, all Pods using
this PodSpec (for example, inside a Deployment or StatefulSet) share the same
ResourceClaim instance. When referencing a ResourceClaimTemplate, each Pod gets
its own instance.
The
list for container resources defines whether a container gets
access to these resource instances, which makes it possible to share resources
between one or more containers.
Here is an example for a fictional resource driver. Two ResourceClaim objects
will get created for this Pod and each container gets access to one of them.
:
kind
:
name
:
resource.example.com
:
resource-driver.example.com
---
:
cats.resource.example.com/v1
kind
:
ClaimParameters
name
:
large-black-cat-claim-parameters
spec
:
color
:
black
size
:
large
---
:
kind
:
metadata
:
name
:
large-black-cat-claim-template
spec
:
spec
:
:
:
apiGroup
:
cats.resource.example.com
kind
:
name
:
–--
:
v1
kind
:
Pod
metadata
:
name
:
pod-with-cats
spec
:
:
-
name
:
container0
image
:
ubuntu:20.04
command
:
[
"sleep"
,
"9999"
]
resources
:
claims
:
-
name
:
cat-0
-
name
:
image
:
command
:
[
"sleep"
,
"9999"
]
resources
:
claims
:
-
name
:
cat-1
:
-
name
:
cat-0
source
:
resourceClaimTemplateName
:
-
name
:
cat-1
source
:
:
In contrast to native resources (CPU, RAM) and extended resources (managed by a
device plugin, advertised by kubelet), the scheduler has no knowledge of what
dynamic resources are available in a cluster or how they could be split up to
satisfy the requirements of a specific ResourceClaim. Resource drivers are
responsible for that. They mark ResourceClaims as "allocated" once resources
for it are reserved. This also then tells the scheduler where in the cluster a
ResourceClaim is available.
ResourceClaims can get allocated as soon as they are created ("immediate
allocation"), without considering which Pods will use them. The default is to
delay allocation until a Pod gets scheduled which needs the ResourceClaim
(i.e. "wait for first consumer").
In that mode, the scheduler checks all ResourceClaims needed by a Pod and
creates a PodScheduling object where it informs the resource drivers
responsible for those ResourceClaims about nodes that the scheduler considers
suitable for the Pod. The resource drivers respond by excluding nodes that
don't have enough of the driver's resources left. Once the scheduler has that
information, it selects one node and stores that choice in the PodScheduling
object. The resource drivers the

n allocate their ResourceClaims so that the
resources will be available on that node. Once that is complete, the Pod
gets scheduled.
As part of this process, ResourceClaims also get reserved for the
Pod. Currently ResourceClaims can either be used exclusively by a single Pod or
an unlimited number of Pods.
One key feature is that Pods do not get scheduled to a node unless all of
their resources are allocated and reserved. This avoids the scenario where a Pod
gets scheduled onto one node and then cannot run there, which is bad because
such a pending Pod also blocks all other resources like RAM or CPU that were
set aside for it.
The scheduler plugin must be involved in scheduling Pods which use
ResourceClaims. Bypassing the scheduler by setting the
nodeName
field leads
to Pods that the kubelet refuses to start because the ResourceClaims are not
reserved or not even allocated. It may be possible to
remove this
limitation
in the
future.
Enabling dynamic resource allocation
Dynamic resource allocation is an
alpha feature
and only enabled when the
feature
gate
and the
API group
are enabled. For details on that, see the
and
.
kube-scheduler, kube-controller-manager and kubelet also need the feature gate.
A quick check whether a Kubernetes cluster supports the feature is to list
ResourceClass objects with:
kubectl get resourceclasses
If your cluster supports dynamic resource allocation, the response is either a
list of ResourceClass objects or:
No resources found
If not supported, this error is printed instead:
error: the server doesn't have a resource type "resourceclasses"
The default configuration of kube-scheduler enables the "DynamicResources"
plugin if and only if the feature gate is enabled. Custom configurations may
have to be modified to include it.
In addition to enabling the feature in the cluster, a resource driver also has to
be installed. Please refer to the driver's documentation for details.
Dynamic Resource Allocation KEP
.
11.9 - Scheduler Performance Tuning
Kubernetes v1.14 [beta]
is the Kubernetes default scheduler. It is responsible for placement of Pods
on Nodes in a cluster.
Nodes in a cluster that meet the scheduling requirements of a Pod are
called
feasible
Nodes for the Pod. The scheduler finds feasible Nodes
for a Pod and then runs a set of functions to score the feasible Nodes,
picking a Node with the highest score among the feasible ones to run
the Pod. The scheduler then notifies the API server about this decision
in a process called
Binding
.
This page explains performance tuning optimizations that are relevant for
large Kubernetes clusters.
In large clusters, you can tune the scheduler's behaviour balancing
scheduling outcomes between latency (new Pods are placed quickly) and
accuracy (the scheduler rarely makes poor placement decisions).
You configure this tuning setting via kube-scheduler setting
. This KubeSchedulerConfiguration setting determines
a threshold for scheduling nodes in your cluster.
Setting the threshold
The
option accepts whole numeric values between 0
and 100. The value 0 is a special number which indicates that the kube-scheduler
should use its compiled-in default.
If you set
above 100, kube-scheduler acts as if you
had set a value of 100.
To change the value, edit the
kube-scheduler configuration file
and then restart the scheduler.
In many cases, the configuration file can be found at
/etc/kubernetes/config/kube-scheduler.yaml
.
After you have made this change, you can run
kubectl get pods -n kube-system | grep kube-scheduler
to verify that the kube-scheduler component is healthy.
Node scoring threshold
To improve scheduling performance, the kube-scheduler can stop looking for
feasible nodes once it has 

found enough of them. In large clusters, this saves
time compared to a naive approach that would consider every node.
You specify a threshold for how many nodes are enough, as a whole number percentage
of all the nodes in your cluster. The kube-scheduler converts this into an
integer number of nodes. During scheduling, if the kube-scheduler has identified
enough feasible nodes to exceed the configured percentage, the kube-scheduler
stops searching for more feasible nodes and moves on to the
scoring phase
.
How the scheduler iterates over Nodes
describes the process in detail.
Default threshold
If you don't specify a threshold, Kubernetes calculates a figure using a
linear formula that yields 50% for a 100-node cluster and yields 10%
for a 5000-node cluster. The lower bound for the automatic value is 5%.
This means that, the kube-scheduler always scores at least 5% of your cluster no
matter how large the cluster is, unless you have explicitly set
to be smaller than 5.
If you want the scheduler to score all nodes in your cluster, set
to 100.
Example
Below is an example configuration that sets
to 50%.
:
kubescheduler.config.k8s.io/v1alpha1
kind
:
algorithmSource
:
provider
:
DefaultProvider
...
:
50
Tuning percentageOfNodesToScore
must be a value between 1 and 100 with the default
value being calculated based on the cluster size. There is also a hardcoded
minimum value of 50 nodes.
Note:
In clusters with less than 50 feasible nodes, the scheduler still
checks all the nodes because there are not enough feasible nodes to stop
the scheduler's search early.
In a small cluster, if you set a low value for
, your
change will have no or little effect, for a similar reason.
If your cluster has several hundred Nodes or fewer, leave this configuration option
at its default value. Making changes is unlikely to improve the
scheduler's performance significantly.
An important detail to consider when setting this value is that when a smaller
number of nodes in a cluster are checked for feasibility, some nodes are not
sent to be scored for a given Pod. As a result, a Node which could possibly
score a higher value for running the given Pod might not even be passed to the
scoring phase. This would result in a less than ideal placement of the Pod.
You should avoid setting
very low so that kube-scheduler
does not make frequent, poor Pod placement decisions. Avoid setting the
percentage to anything below 10%, unless the scheduler's throughput is critical
for your application and the score of nodes is not important. In other words, you
prefer to run the Pod on any Node as long as it is feasible.
This section is intended for those who want to understand the internal details
of this feature.
In order to give all the Nodes in a cluster a fair chance of being considered
for running Pods, the scheduler iterates over the nodes in a round robin
fashion. You can imagine that Nodes are in an array. The scheduler starts from
the start of the array and checks feasibility of the nodes until it finds enough
Nodes as specified by
. For the next Pod, the
scheduler continues from the point in the Node array that it stopped at when
checking feasibility of Nodes for the previous Pod.
If Nodes are in multiple zones, the scheduler iterates over Nodes in various
zones to ensure that Nodes from different zones are considered in the
feasibility checks. As an example, consider six nodes in two zones:
Zone 1: Node 1, Node 2, Node 3, Node 4
Zone 2: Node 5, Node 6
The Scheduler evaluates feasibility of the nodes in this order:
Node 1, Node 5, Node 2, Node 6, Node 3, Node 4
After going over all the Nodes, it goes back to Node 1.
Check the
11.10 - Re

source Bin Packing
In the
scheduling-plugin
of kube-scheduler, there are two
scoring strategies that support the bin packing of resources:
and
.
Enabling bin packing using MostAllocated strategy
The
strategy scores the nodes based on the utilization of resources, favoring the ones with higher allocation.
For each resource type, you can set a weight to modify its influence in the node score.
To set the
strategy for the
plugin, use a
similar to the following:
:
kind
:
profiles
:
-
:
-
args
:
:
resources
:
-
name
:
cpu
weight
:
1
-
name
:
memory
weight
:
1
-
name
:
intel.com/foo
weight
:
3
-
name
:
intel.com/bar
weight
:
3
type
:
name
:
To learn more about other parameters and their default configuration, see the API documentation for
.
Enabling bin packing using RequestedToCapacityRatio
The
strategy allows the users to specify the resources along with weights for
each resource to score nodes based on the request to capacity ratio. This
allows users to bin pack extended resources by using appropriate parameters
to improve the utilization of scarce resources in large clusters. It favors nodes according to a
configured function of the allocated resources. The behavior of the
in
the
score function can be controlled by the
field.
Within the
field, you can configure two parameters:
and
resources
. The
shape
in the
parameter allows the user to tune the function as least requested or most
requested based on
and
score
values. The
resources
parameter
consists of
name
of the resource to be considered during scoring and
weight
specify the weight of each resource.
the bin packing behavior for extended resources
and
using the
field.
:
kind
:
profiles
:
-
:
-
args
:
:
resources
:
-
name
:
weight
:
3
-
name
:
weight
:
3
:
shape
:
-
:
0
score
:
0
-
:
100
score
:
10
type
:
name
:
Referencing the
file with the kube-scheduler
flag
--config=/path/to/config/file
will pass the configuration to the
scheduler.
.
Tuning the score function
shape
is used to specify the behavior of the
function.
shape
:
-
:
0
score
:
0
-
:
100
score
:
10
The above arguments give the node a
score
of 0 if
is 0% and 10 for
100%, thus enabling bin packing behavior. To enable least
requested the score value must be reversed as follows.
shape
:
-
:
0
score
:
10
-
:
100
score
:
0
resources
is an optional parameter which defaults to:
resources
:
-
name
:
cpu
weight
:
1
-
name
:
memory
weight
:
1
It can be used to add extended resources as follows:
resources
:
-
name
:
weight
:
5
-
name
:
cpu
weight
:
3
-
name
:
memory
weight
:
1
The
weight
parameter is optional and is set to 1 if not specified. Also, the
weight
cannot be set to a negative value.
Node scoring for capacity allocation
Below is an example of how the node score is calculated for a given set of values.
Requested resources:
intel.com/foo : 2
memory: 256MB
cpu: 2
Resource weights:
intel.com/foo : 5
memory: 1
cpu: 3
FunctionShapeP

oint {{0, 0}, {100, 10}}
Node 1 spec:
Available:
  intel.com/foo: 4
  memory: 1 GB
  cpu: 8

Used:
  intel.com/foo: 1
  memory: 256MB
  cpu: 1
Node score:
intel.com/foo  = resourceScoringFunction((2+1),4)
               = (100 - ((4-3)*100/4)
               = (100 - 25)
               = 75                       # requested + used = 75% * available
               = rawScoringFunction(75) 
               = 7                        # floor(75/10) 

memory         = resourceScoringFunction((256+256),1024)
               = (100 -((1024-512)*100/1024))
               = 50                       # requested + used = 50% * available
               = rawScoringFunction(50)
               = 5                        # floor(50/10)

cpu            = resourceScoringFunction((2+1),8)
               = (100 -((8-3)*100/8))
               = 37.5                     # requested + used = 37.5% * available
               = rawScoringFunction(37.5)
               = 3                        # floor(37.5/10)

NodeScore   =  (7 * 5) + (5 * 1) + (3 * 3) / (5 + 1 + 3)
            =  5
Node 2 spec:
  intel.com/foo: 8
  memory: 1GB
  cpu: 8
Used:
  intel.com/foo: 2
  memory: 512MB
  cpu: 6
intel.com/foo  = resourceScoringFunction((2+2),8)
               =  (100 - ((8-4)*100/8)
               =  (100 - 50)
               =  50
               =  rawScoringFunction(50)
               = 5

memory         = resourceScoringFunction((256+512),1024)
               = (100 -((1024-768)*100/1024))
               = 75
               = rawScoringFunction(75)
               = 7

cpu            = resourceScoringFunction((2+6),8)
               = (100 -((8-8)*100/8))
               = 100
               = rawScoringFunction(100)
               = 10

NodeScore   =  (5 * 5) + (7 * 1) + (10 * 3) / (5 + 1 + 3)
            =  7
11.11 - Pod Priority and Preemption
Pods
can have
priority
. Priority indicates the
importance of a Pod relative to other Pods. If a Pod cannot be scheduled, the
scheduler tries to preempt (evict) lower priority Pods to make scheduling of the
pending Pod possible.
Warning:
In a cluster where not all users are trusted, a malicious user could create Pods
at the highest possible priorities, causing other Pods to be evicted/not get
scheduled.
An administrator can use ResourceQuota to prevent users from creating pods at
high priorities.
See
limit Priority Class consumption by default
How to use priority and preemption
To use priority and preemption:
Add one or more
PriorityClasses
.
Create Pods with
set to one of the added
PriorityClasses. Of course you do not need to create the Pods directly;
normally you would add
to the Pod template of a
collection object like a Deployment.
Keep reading for more information about these steps.
Note:
Kubernetes already ships with two PriorityClasses:
and
.
These are common classes and are used to
ensure that critical components are always scheduled first
.
A PriorityClass is a non-namespaced object that defines a mapping from a
priority class name to the integer value of the priority. The name is specified
in the
name
field of the PriorityClass object's metadata. The value is
specified in the required
value
field. The higher the value, the higher the
priority.
The name of a PriorityClass object must be a valid
,
and it cannot be prefixed with
system-
.
A PriorityClass object can have any 32-bit integer value smaller than or equal
to 1 billion. Larger numbers are reserved for critical system Pods that should
not normally be preempted or evicted. A cluster admin should create one
PriorityClass object for each such mapping that they want.
PriorityClass also has two optional fields:
and
.
The
field indicates t

hat the value of this PriorityClass should
be used for Pods without a
. Only one PriorityClass with
set to true can exist in the system. If there is no
PriorityClass with
set, the priority of Pods with no
is zero.
The
field is an arbitrary string. It is meant to tell users of the
cluster when they should use this PriorityClass.
Notes about PodPriority and existing clusters
If you upgrade an existing cluster without this feature, the priority
of your existing Pods is effectively zero.
Addition of a PriorityClass with
set to
true
does not
change the priorities of existing Pods. The value of such a PriorityClass is
used only for Pods created after the PriorityClass is added.
If you delete a PriorityClass, existing Pods that use the name of the
deleted PriorityClass remain unchanged, but you cannot create more Pods that
use the name of the deleted PriorityClass.
Example PriorityClass
:
kind
:
metadata
:
name
:
value
:
1000000
:
false
:
"This priority class should be used for XYZ service pods only."
Non-preempting PriorityClass
Pods with
preemptionPolicy: Never
will be placed in the scheduling queue
ahead of lower-priority pods,
but they cannot preempt other pods.
A non-preempting pod waiting to be scheduled will stay in the scheduling queue,
until sufficient resources are free,
and it can be scheduled.
Non-preempting pods,
like other pods,
are subject to scheduler back-off.
This means that if the scheduler tries these pods and they cannot be scheduled,
they will be retried with lower frequency,
allowing other pods with lower priority to be scheduled before them.
Non-preempting pods may still be preempted by other,
high-priority pods.
defaults to
,
which will allow pods of that PriorityClass to preempt lower-priority pods
(as is existing default behavior).
If
is set to
Never
,
pods in that PriorityClass will be non-preempting.
An example use case is for data science workloads.
A user may submit a job that they want to be prioritized above other workloads,
but do not wish to discard existing work by preempting running pods.
The high priority job with
will be scheduled
ahead of other queued pods,
as soon as sufficient cluster resources "naturally" become free.
Example Non-preempting PriorityClass
:
kind
:
metadata
:
name
:
high-priority-nonpreempting
value
:
1000000
:
Never
:
false
:
"This priority class will not cause other pods to be preempted."
Pod priority
After you have one or more PriorityClasses, you can create Pods that specify one
of those PriorityClass names in their specifications. The priority admission
controller uses the
field and populates the integer value of
the priority. If the priority class is not found, the Pod is rejected.
The following YAML is an example of a Pod configuration that uses the
PriorityClass created in the preceding example. The priority admission
controller checks the specification and resolves the priority of the Pod to
1000000.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
labels
:
env
:
test
spec
:
:
-
name
:
nginx
image
:
nginx
:
:
Effect of Pod priority on scheduling order
When Pod priority is enabled, the scheduler orders pending Pods by
their priority and a pending Pod is placed ahead of other pending Pods
with lower priority in the scheduling queue. As a result, the higher
priority Pod may be scheduled sooner than Pods with lower priority if
its scheduling requirements are met. If such Pod cannot be scheduled,
scheduler will continue and tries to schedule other lower priority Pods.
When Pods are created, the

y go to a queue and wait to be scheduled. The
scheduler picks a Pod from the queue and tries to schedule it on a Node. If no
Node is found that satisfies all the specified requirements of the Pod,
preemption logic is triggered for the pending Pod. Let's call the pending Pod P.
Preemption logic tries to find a Node where removal of one or more Pods with
lower priority than P would enable P to be scheduled on that Node. If such a
Node is found, one or more lower priority Pods get evicted from the Node. After
the Pods are gone, P can be scheduled on the Node.
User exposed information
When Pod P preempts one or more Pods on Node N,
field of Pod
P's status is set to the name of Node N. This field helps scheduler track
resources reserved for Pod P and also gives users information about preemptions
in their clusters.
Please note that Pod P is not necessarily scheduled to the "nominated Node".
The scheduler always tries the "nominated Node" before iterating over any other nodes.
After victim Pods are preempted, they get their graceful termination period. If
another node becomes available while scheduler is waiting for the victim Pods to
terminate, scheduler may use the other node to schedule Pod P. As a result
and
nodeName
of Pod spec are not always the same. Also, if
scheduler preempts Pods on Node N, but then a higher priority Pod than Pod P
arrives, scheduler may give Node N to the new higher priority Pod. In such a
case, scheduler clears
of Pod P. By doing this, scheduler
makes Pod P eligible to preempt Pods on another Node.
Limitations of preemption
Graceful termination of preemption victims
When Pods are preempted, the victims get their
graceful termination period
.
They have that much time to finish their work and exit. If they don't, they are
killed. This graceful termination period creates a time gap between the point
that the scheduler preempts Pods and the time when the pending Pod (P) can be
scheduled on the Node (N). In the meantime, the scheduler keeps scheduling other
pending Pods. As victims exit or get terminated, the scheduler tries to schedule
Pods in the pending queue. Therefore, there is usually a time gap between the
point that scheduler preempts victims and the time that Pod P is scheduled. In
order to minimize this gap, one can set graceful termination period of lower
priority Pods to zero or a small number.
PodDisruptionBudget is supported, but not guaranteed
A
(PDB)
allows application owners to limit the number of Pods of a replicated application
that are down simultaneously from voluntary disruptions. Kubernetes supports
PDB when preempting Pods, but respecting PDB is best effort. The scheduler tries
to find victims whose PDB are not violated by preemption, but if no such victims
are found, preemption will still happen, and lower priority Pods will be removed
despite their PDBs being violated.
Inter-Pod affinity on lower-priority Pods
A Node is considered for preemption only when the answer to this question is
yes: "If all the Pods with lower priority than the pending Pod are removed from
the Node, can the pending Pod be scheduled on the Node?"
Note:
Preemption does not necessarily remove all lower-priority
Pods. If the pending Pod can be scheduled by removing fewer than all
lower-priority Pods, then only a portion of the lower-priority Pods are removed.
Even so, the answer to the preceding question must be yes. If the answer is no,
the Node is not considered for preemption.
If a pending Pod has inter-pod
affinity
to one or more of the lower-priority Pods on the Node, the inter-Pod affinity
rule cannot be satisfied in the absence of those lower-priority Pods. In this case,
the scheduler does not preempt any Pods on the Node. Instead, it looks for another
Node. The scheduler might find a suitable Node or it might not. There is no
guarantee that the pending Pod can be scheduled.
Our recommended solution for this problem is to create inte

r-Pod affinity only
towards equal or higher priority Pods.
Cross node preemption
Suppose a Node N is being considered for preemption so that a pending Pod P can
be scheduled on N. P might become feasible on N only if a Pod on another Node is
preempted. Here's an example:
Pod P is being considered for Node N.
Pod Q is running on another Node in the same Zone as Node N.
Pod P has Zone-wide anti-affinity with Pod Q (
topologyKey: topology.kubernetes.io/zone
).
There are no other cases of anti-affinity between Pod P and other Pods in
the Zone.
In order to schedule Pod P on Node N, Pod Q can be preempted, but scheduler
does not perform cross-node preemption. So, Pod P will be deemed
unschedulable on Node N.
If Pod Q were removed from its Node, the Pod anti-affinity violation would be
gone, and Pod P could possibly be scheduled on Node N.
We may consider adding cross Node preemption in future versions if there is
enough demand and if we find an algorithm with reasonable performance.
Pod priority and pre-emption can have unwanted side effects. Here are some
examples of potential problems and ways to deal with them.
Pods are preempted unnecessarily
Preemption removes existing Pods from a cluster under resource pressure to make
room for higher priority pending Pods. If you give high priorities to
certain Pods by mistake, these unintentionally high priority Pods may cause
preemption in your cluster. Pod priority is specified by setting the
field in the Pod's specification. The integer value for
priority is then resolved and populated to the
priority
field of
podSpec
.
To address the problem, you can change the
for those Pods
to use lower priority classes, or leave that field empty. An empty
is resolved to zero by default.
When a Pod is preempted, there will be events recorded for the preempted Pod.
Preemption should happen only when a cluster does not have enough resources for
a Pod. In such cases, preemption happens only when the priority of the pending
Pod (preemptor) is higher than the victim Pods. Preemption must not happen when
there is no pending Pod, or when the pending Pods have equal or lower priority
than the victims. If preemption happens in such scenarios, please file an issue.
Pods are preempted, but the preemptor is not scheduled
When pods are preempted, they receive their requested graceful termination
period, which is by default 30 seconds. If the victim Pods do not terminate within
this period, they are forcibly terminated. Once all the victims go away, the
preemptor Pod can be scheduled.
While the preemptor Pod is waiting for the victims to go away, a higher priority
Pod may be created that fits on the same Node. In this case, the scheduler will
schedule the higher priority Pod instead of the preemptor.
This is expected behavior: the Pod with the higher priority should take the place
of a Pod with a lower priority.
Higher priority Pods are preempted before lower priority pods
The scheduler tries to find nodes that can run a pending Pod. If no node is
found, the scheduler tries to remove Pods with lower priority from an arbitrary
node in order to make room for the pending pod.
If a node with low priority Pods is not feasible to run the pending Pod, the scheduler
may choose another node with higher priority Pods (compared to the Pods on the
other node) for preemption. The victims must still have lower priority than the
preemptor Pod.
When there are multiple nodes available for preemption, the scheduler tries to
choose the node with a set of Pods with lowest priority. However, if such Pods
have PodDisruptionBudget that would be violated if they are preempted then the
scheduler may choose another node with higher priority Pods.
When multiple nodes exist for preemption and none of the above scenarios apply,
the scheduler chooses a node with the lowest priority.
Interactions between Pod priority and quality of service
Pod priority and
QoS class
are two ort

hogonal features with few interactions and no default restrictions on
setting the priority of a Pod based on its QoS classes. The scheduler's
preemption logic does not consider QoS when choosing preemption targets.
Preemption considers Pod priority and attempts to choose a set of targets with
the lowest priority. Higher-priority Pods are considered for preemption only if
the removal of the lowest priority Pods is not sufficient to allow the scheduler
to schedule the preemptor Pod, or if the lowest priority Pods are protected by
.
The kubelet uses Priority to determine pod order for
.
You can use the QoS class to estimate the order in which pods are most likely
to get evicted. The kubelet ranks pods for eviction based on the following factors:
Whether the starved resource usage exceeds requests
Amount of resource usage relative to requests
See
Pod selection for kubelet eviction
kubelet node-pressure eviction does not evict Pods when their
usage does not exceed their requests. If a Pod with lower priority is not
exceeding its requests, it won't be evicted. Another Pod with higher priority
that exceeds its requests may be evicted.
Read about using ResourceQuotas in connection with PriorityClasses:
11.12 - Node-pressure Eviction
kubelet
The
kubelet
monitors resources
like memory, disk space, and filesystem inodes on your cluster's nodes.
When one or more of these resources reach specific consumption levels, the
kubelet can proactively fail one or more pods on the node to reclaim resources
and prevent starvation.
During a node-pressure eviction, the kubelet sets the
PodPhase
for the
selected pods to
Failed
. This terminates the pods.
.
The kubelet does not respect your configured
or the pod's
. If you use
soft eviction thresholds
,
the kubelet respects your configured
eviction-max-pod-grace-period
. If you use
hard eviction thresholds
, it uses a
0s
grace period for termination.
If the pods are managed by a
workload
resource (such as
or
) that
replaces failed pods, the control plane or
creates new
pods in place of the evicted pods.
Note:
The kubelet attempts to
reclaim node-level resources
before it terminates end-user pods. For example, it removes unused container
images when disk resources are starved.
The kubelet uses various parameters to make eviction decisions, like the following:
Eviction signals
Eviction thresholds
Monitoring intervals
Eviction signals are the current state of a particular resource at a specific
point in time. Kubelet uses eviction signals to make eviction decisions by
comparing the signals to eviction thresholds, which are the minimum amount of
the resource that should be available on the node.
Kubelet uses the following eviction signals:
Eviction Signal
:=
node.status.capacity[memory]
-
node.stats.memory.workingSet
:=
node.stats.fs.available
:=
node.stats.fs.inodesFree
:=
node.stats.runtime.imagefs.available
:=
node.stats.runtime.imagefs.inodesFree
:=
node.stats.rlimit.maxpid
-
node.stats.rlimit.curproc
In this table, the
column shows how kubelet gets the value of the
signal. Each signal supports either a percentage or a literal value. Kubelet
calculates the percentage value relative to the total capacity associated with
the signal.
The value for
is derived

 from the cgroupfs instead of tools
like
free -m
. This is important because
free -m
does not work in a
container, and if users use the
feature, out of resource decisions
are made local to the end user Pod part of the cgroup hierarchy as well as the
root node. This
script
reproduces the same set of steps that the kubelet performs to calculate
. The kubelet excludes inactive_file (i.e. # of bytes of
file-backed memory on inactive LRU list) from its calculation as it assumes that
memory is reclaimable under pressure.
The kubelet supports the following filesystem partitions:
nodefs
: The node's main filesystem, used for local disk volumes, emptyDir,
log storage, and more. For example,
nodefs
contains
/var/lib/kubelet/
.
imagefs
: An optional filesystem that container runtimes use to store container
images and container writable layers.
Kubelet auto-discovers these filesystems and ignores other filesystems. Kubelet
does not support other configurations.
Some kubelet garbage collection features are deprecated in favor of eviction:
Existing Flag
New Flag
Rationale
or
existing eviction signals can trigger image garbage collection
--image-gc-low-threshold
--eviction-minimum-reclaim
eviction reclaims achieve the same behavior
--maximum-dead-containers
-
deprecated once old logs are stored outside of container's context
--maximum-dead-containers-per-container
-
--minimum-container-ttl-duration
-
You can specify custom eviction thresholds for the kubelet to use when it makes
eviction decisions.
Eviction thresholds have the form
[eviction-signal][operator][quantity]
, where:
eviction-signal
is the
eviction signal
to use.
operator
is the
relational operator
you want, such as
&lt;
(less than).
quantity
is the eviction threshold amount, such as
1Gi
. The value of
quantity
must match the quantity representation used by Kubernetes. You can use either
literal values or percentages (
%
).
For example, if a node has
10Gi
of total memory and you want trigger eviction if
the available memory falls below
1Gi
, you can define the eviction threshold as
either
memory.available&lt;10%
or
. You cannot use both.
You can configure soft and hard eviction thresholds.
Soft eviction thresholds
A soft eviction threshold pairs an eviction threshold with a required
administrator-specified grace period. The kubelet does not evict pods until the
grace period is exceeded. The kubelet returns an error on startup if there is no
specified grace period.
You can specify both a soft eviction threshold grace period and a maximum
allowed pod termination grace period for kubelet to use during evictions. If you
specify a maximum allowed grace period and the soft eviction threshold is met,
the kubelet uses the lesser of the two grace periods. If you do not specify a
maximum allowed grace period, the kubelet kills evicted pods immediately without
graceful termination.
You can use the following flags to configure soft eviction thresholds:
eviction-soft
: A set of eviction thresholds like
that can trigger pod eviction if held over the specified grace period.
eviction-soft-grace-period
: A set of eviction grace periods like
that define how long a soft eviction threshold must hold before triggering a Pod eviction.
: The maximum allowed grace period (in seconds)
to use when terminating pods in response to a soft eviction threshold being met.
Hard eviction thresholds
A hard eviction threshold has no grace period. When a hard eviction threshold is
met, the kubelet kills pods immediately without graceful termination to reclaim
the starved resource.
flag to configure a set of hard eviction
thresholds like
memory.av

ailable&lt;1Gi
.
The kubelet has the following default hard eviction thresholds:
memory.available&lt;100Mi
nodefs.available&lt;10%
imagefs.available&lt;15%
(Linux nodes)
These default values of hard eviction thresholds will only be set if none
of the parameters is changed. If you changed the value of any parameter,
then the values of other parameters will not be inherited as the default
values and will be set to zero. In order to provide custom values, you
should provide all the thresholds respectively.
Eviction monitoring interval
The kubelet evaluates eviction thresholds based on its configured
housekeeping-interval
10s
.
The kubelet reports node conditions to reflect that the node is under pressure
because hard or soft eviction threshold is met, independent of configured grace
periods.
The kubelet maps eviction signals to node conditions as follows:
Available memory on the node has satisfied an eviction threshold
,
,
, or
Available disk space and inodes on either the node's root filesystem or image filesystem has satisfied an eviction threshold
Available processes identifiers on the (Linux) node has fallen below an eviction threshold
The kubelet updates the node conditions based on the configured
10s
.
Node condition oscillation
In some cases, nodes oscillate above and below soft eviction thresholds without
holding for the defined grace periods. This causes the reported node condition
to constantly switch between
true
and
false
, leading to bad eviction decisions.
To protect against oscillation, you can use the
eviction-pressure-transition-period
flag, which controls how long the kubelet must wait before transitioning a node
condition to a different state. The transition period has a default value of
5m
.
Reclaiming node level resources
The kubelet tries to reclaim node-level resources before it evicts end-user pods.
When a
node condition is reported, the kubelet reclaims node-level
resources based on the filesystems on the node.
With
imagefs
If the node has a dedicated
imagefs
filesystem for container runtimes to use,
the kubelet does the following:
If the
nodefs
filesystem meets the eviction thresholds, the kubelet garbage collects
dead pods and containers.
If the
imagefs
filesystem meets the eviction thresholds, the kubelet
deletes all unused images.
Without
imagefs
If the node only has a
nodefs
filesystem that meets eviction thresholds,
the kubelet frees up disk space in the following order:
Garbage collect dead pods and containers
Delete unused images
If the kubelet's attempts to reclaim node-level resources don't bring the eviction
signal below the threshold, the kubelet begins to evict end-user pods.
The kubelet uses the following parameters to determine the pod eviction order:
Whether the pod's resource usage exceeds requests
The pod's resource usage relative to requests
As a result, kubelet ranks and evicts pods in the following order:
or
Burstable
pods where the usage exceeds requests. These pods
are evicted based on their Priority and then by how much their usage level
exceeds the request.
pods and
Burstable
pods where the usage is less than requests
are evicted last, based on their Priority.
Note:
The kubelet does not use the pod's QoS class to determine the eviction order.
You can use the QoS class to estimate the most likely pod eviction order when
reclaiming resources like memory. QoS does not apply to EphemeralStorage requests,
so the above scenario will not apply if the node is, for example, under
.
pods are guaranteed only when requests and limits are specified for
all the containers and they are equal. These po

ds will never be evicted because
of another pod's resource consumption. If a system daemon (such as
kubelet
and
journald
) is consuming more resources than were reserved via
or
allocations, and the node only has
or
Burstable
pods using less resources than requests left on it,
then the kubelet must choose to evict one of these pods to preserve node stability
and to limit the impact of resource starvation on other pods. In this case, it
will choose to evict pods of lowest Priority first.
When the kubelet evicts pods in response to
inode
or
PID
starvation, it uses
the Priority to determine the eviction order, because
inodes
and
PIDs
have no
requests.
The kubelet sorts pods differently based on whether the node has a dedicated
imagefs
filesystem:
With
imagefs
If
nodefs
is triggering evictions, the kubelet sorts pods based on
nodefs
usage (
local volumes + logs of all containers
).
If
imagefs
is triggering evictions, the kubelet sorts pods based on the
writable layer usage of all containers.
Without
imagefs
If
nodefs
is triggering evictions, the kubelet sorts pods based on their total
disk usage (
local volumes + logs &amp; writable layer of all containers
)
Minimum eviction reclaim
In some cases, pod eviction only reclaims a small amount of the starved resource.
This can lead to the kubelet repeatedly hitting the configured eviction thresholds
and triggering multiple evictions.
flag or a
kubelet config file
to configure a minimum reclaim amount for each resource. When the kubelet notices
that a resource is starved, it continues to reclaim that resource until it
reclaims the quantity you specify.
For example, the following configuration sets minimum reclaim amounts:
:
kind
:
:
:
"500Mi"
:
"1Gi"
:
"100Gi"
:
:
"0Mi"
:
"500Mi"
:
"2Gi"
In this example, if the
signal meets the eviction threshold,
the kubelet reclaims the resource until the signal reaches the threshold of
1Gi
,
and then continues to reclaim the minimum amount of
500Mi
it until the signal
reaches
1.5Gi
.
Similarly, the kubelet reclaims the
imagefs
resource until the
signal reaches
102Gi
.
eviction-minimum-reclaim
is
0
for all resources.
Node out of memory behavior
If the node experiences an out of memory (OOM) event prior to the kubelet
being able to reclaim memory, the node depends on the
oom_killer
to respond.
The kubelet sets an
oom_score_adj
value for each container based on the QoS for the pod.
Quality of Service
-997
1000
Burstable
min(max(2, 1000 - (1000 * memoryRequestBytes) / machineMemoryCapacityBytes), 999)
Note:
The kubelet also sets an
value of
-997
for containers in Pods that have
Priority
.
If the kubelet can't reclaim memory before a node experiences OOM, the
calculates an
oom_score
based on the percentage of memory it's
using on the node, and then adds the
to get an effective
oom_score
for each container. It then kills the container with the highest score.
This means that containers in low QoS pods that consume a large amount of memory
relative to their scheduling requests are killed first.
Unlike pod eviction, if a container is OOM killed, the
kubelet
can restart it
based on its
.
The following sections describe best practices for eviction configuration.
Schedulable resources and eviction policies
When you configure the kubelet with an eviction policy, you should make sure that
the scheduler will not schedule pods if they will trigger eviction because they
immediately induce memory pressure.
Consider the following scenario:
Node memory capacity:
10Gi
Operator wants to reserve 10% of memory capacity for sy

stem daemons (kernel,
kubelet
, etc.)
Operator wants to evict Pods at 95% memory utilization to reduce incidence of system OOM.
For this to work, the kubelet is launched as follows:
--system-reserved=memory=1.5Gi
In this configuration, the
flag reserves
1.5Gi
of memory
for the system, which is
10% of the total memory + the eviction threshold amount
.
The node can reach the eviction threshold if a pod is using more than its request,
or if the system is using more than
1Gi
of memory, which makes the
signal fall below
500Mi
and triggers the threshold.
DaemonSet
Pod Priority is a major factor in making eviction decisions. If you do not want
the kubelet to evict pods that belong to a
DaemonSet
, give those pods a high
enough
priorityClass
in the pod spec. You can also use a lower
or the default to only allow
DaemonSet
pods to run when there are enough
The following sections describe known issues related to out of resource handling.
kubelet may not observe memory pressure right away
By default, the kubelet polls
cAdvisor
to collect memory usage stats at a
regular interval. If memory usage increases within that window rapidly, the
kubelet may not observe
fast enough, and the
OOMKiller
will still be invoked.
flag to enable the
memcg
notification API on the kubelet to get notified immediately when a threshold
is crossed.
If you are not trying to achieve extreme utilization, but a sensible measure of
overcommit, a viable workaround for this issue is to use the
and
flags to allocate memory for the system.
active_file memory is not considered as available memory
On Linux, the kernel tracks the number of bytes of file-backed memory on active
LRU list as the
active_file
statistic. The kubelet treats
memory
areas as not reclaimable. For workloads that make intensive use of block-backed
local storage, including ephemeral local storage, kernel-level caches of file
and block data means that many recently accessed cache pages are likely to be
counted as
. If enough of these kernel block buffers are on the
active LRU list, the kubelet is liable to observe this as high resource use and
taint the node as experiencing memory pressure - triggering pod eviction.
https://github.com/kubernetes/kubernetes/issues/43916
You can work around that behavior by setting the memory limit and memory request
the same for containers likely to perform intensive I/O activity. You will need
to estimate or measure an optimal memory limit value for that container.
(QoS)
11.13 - API-initiated Eviction
Eviction
You can request eviction by calling the Eviction API directly, or programmatically
using a client of the
API server
, like the
command. This
creates an
Eviction
object, which causes the API server to terminate the Pod.
and
.
Using the API to create an Eviction object for a Pod is like performing a
policy-controlled
DELETE
operation
on the Pod.
Calling the Eviction API
Kubernetes language client
to access the Kubernetes API and create an
Eviction
object. To do this, you
POST the attempted operation, similar to the following example:
policy/v1
Note:
policy/v1
Eviction is available in v1.22+. Use
with prior releases.
{
:
"policy/v1"
,
"kind"
:
"Eviction"
,
: {
"name"
:
"quux"
,
:
"default"
}
}
Note:
Deprecated in v1.22 in favo

r of
policy/v1
{
:
"policy/v1beta1"
,
"kind"
:
,
: {
"name"
:
"quux"
,
:
"default"
}
}
Alternatively, you can attempt an eviction operation by accessing the API using
curl
or
wget
, similar to the following example:
curl -v -H
'Content-type: application/json'
https://your-cluster-api-endpoint.example/api/v1/namespaces/default/pods/quux/eviction -d @eviction.json
How API-initiated eviction works
When you request an eviction using the API, the API server performs admission
checks and responds in one of the following ways:
200 OK
: the eviction is allowed, the
Eviction
subresource is created, and
the Pod is deleted, similar to sending a
DELETE
request to the Pod URL.
429 Too Many Requests
: the eviction is not currently allowed because of the
configured
.
You may be able to attempt the eviction again later. You might also see this
response because of API rate limiting.
500 Internal Server Error
: the eviction is not allowed because there is a
misconfiguration, like if multiple PodDisruptionBudgets reference the same Pod.
If the Pod you want to evict isn't part of a workload that has a
PodDisruptionBudget, the API server always returns
200 OK
and allows the
eviction.
If the API server allows the eviction, the Pod is deleted as follows:
The
Pod
resource in the API server is updated with a deletion timestamp,
after which the API server considers the
Pod
resource to be terminated. The
Pod
resource is also marked with the configured grace period.
The
kubelet
on the node where the local Pod is running notices that the
Pod
resource is marked for termination and starts to gracefully shut down the
local Pod.
While the kubelet is shutting the Pod down, the control plane removes the Pod
from
Endpoint
and
objects. As a result, controllers no longer consider the Pod as a valid object.
After the grace period for the Pod expires, the kubelet forcefully terminates
the local Pod.
The kubelet tells the API server to remove the
Pod
resource.
The API server deletes the
Pod
resource.
Troubleshooting stuck evictions
In some cases, your applications may enter a broken state, where the Eviction
API will only return
429
or
500
responses until you intervene. This can
happen if, for example, a ReplicaSet creates pods for your application but new
pods do not enter a
Ready
state. You may also notice this behavior in cases
where the last evicted Pod had a long termination grace period.
If you notice stuck evictions, try one of the following solutions:
Abort or pause the automated operation causing the issue. Investigate the stuck
application before you restart the operation.
Wait a while, then directly delete the Pod from your cluster control plane
instead of using the Eviction API.
Learn how to protect your applications with a
.
.
.
12 - Cluster Administration
concepts
.
Setup
distros
.
Note:
, such as
, or
?
, or
If you are con

figuring Kubernetes on-premises
fits best.
or on
?
Do you
?
.
describes
describes
sysctl
.
Auditing
12.1 - Certificates
.
12.2 - Managing Resources
and
labels
.
---
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
nginx
spec
:
type
:
ports
:
-
port
:
80
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
labels
:
app
:
nginx
spec
:
replicas
:
3
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
-f
A URL can also be specified as a configuration source, which is handy for deploying directly from configuration files 

checked into GitHub:
kubectl
deleted
service
deleted
-l
or
app
=
nginx
deleted
service
deleted
Because
kubectl
$()
or
xargs
:
$(
)
{}
(
S
)
AGE
-o name
grep
.
or
-R
flag.
manifests
└── pvc
(
)
or
-R
The
etc.
The
-f
kubectl
.
app
labels
:
app
:
guestbook
tier
:
frontend
while t

he Redis master and slave would have different
tier
role
label:
labels
:
app
:
guestbook
tier
:
backend
role
:
master
and
labels
:
app
:
guestbook
tier
:
backend
role
:
slave
0
0
0
0
0
0
0
0
=
=
slave
0
3m
0
3m
canary
track
track
stable
:
name
:
frontend
replicas
:
3
...
labels
:
app
:
guestbook
tier
:
frontend
track
:
stable
...
image
:
and then you can create a new release of the guestbook frontend that carries the
track
canary
name
:
replicas
:
1
...
labels
:
app
:
guestbook
tier
:
frontend
track
:
canary
...
image
:
track
selector
:
app
:
guestbook
tier
:
frontend
.
.
app
=
nginx
tier
=
fe
app
=
0
23m       fe
0
23m       fe
0
23m       fe
-L
or
--label-column

s
).
labels
and
.
=
kind: pod
metadata:
...
and
document.
kubectl
=
1
app
=
nginx
0
30m
=
1
--max
=
3
,
and
document.
(see
),
Note that
and
:
get
apply
EDITOR
or
document.
You can use
JSON merge patch, and strategic merge p

atch. See
and
.
kubectl
=
=
1
=
3
from
to
.
how to use
kubectl
.
See
.
12.3 - Cluster Networking
Highly-coupled container-to-container communications: this is solved by
Pods
and
localhost
Services
.
here
.
See
this page

.
12.4 - Logging Architecture
.
Pod
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
args
:
]
You can use
-c
kubectl logs counter -c count
See the
stdout
and
stderr
streams.
as the
.
by running
.
,
and
,
using the
.
as in
Note:
.
returns at mo

st 10MiB of data.
pods
)
(usually
).
DaemonSet
.
Linux
Windows
.
.log
/var/log
argument
--log-dir
.
, read
.
).
Although
instead.
directory
.
, read
.
the
/var/log
directory, bypassing the default logging mechanism (the components
.
Note:
.
/var/log
Because the logging agent must run on every no

de, it is recommended to run the agent
as a
DaemonSet
.
stdout
.
stdout
and
stderr
stdout
or
stderr
stream.
stdout
or
stderr
stdout
and
stderr
like
.
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
args
:
- /bin/sh
- -c
- &gt;
i=0;
do
sleep 1;
done
:
-
name
:
varlog
mountPath
:
/var/log
volumes
:
-
name
:
varlog
emptyDir
:
{}
stdout
stream of
stdout
stream.
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
args
:
- /bin/sh
- -c
- &gt;
i=0;
do
sleep 1;
done
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
image
:
args
:
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
image
:
args
:
:
-
name
:
varlog
mountPath
:
/var/log
volumes
:
-
name
:
varlog
emptyDir
:
{}
...
...
stdout
If you have an application that writes to a single file, it's recommended t

o set
logrotate
stdout
and
stderr
Note:
ConfigMap
:
v1
kind
:
ConfigMap
metadata
:
name
:
data
:
:
|
type tail
type tail
Note:
:
v1
kind
:
Pod
metadata
:
name
:
counter
spec
:
:
-
name
:
count
image
:
args
:
- /bin/sh
- -c
- &gt;
i=0;
do
sleep 1;
done
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
image
:
env
:
-
name
:
value
:
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
mountPath
:
volumes
:
-
name
:
varlog
emptyDir
:
{}
-
name
:
configMap
:
name
:
Read about
12.5 - Metrics For Kubernetes System Components
.
/metrics
flag.
kubelet
Note that
kubelet
,
and
RBAC
/metrics
apiV

ersion
:
kind
:
metadata
:
name
:
rules
:
-
:
-
verbs
:
- get
section.
The flag
. The too
A
A
In release
1.n
, the metric is deprecated, and it can be emitted by default.
In release
1.n+1
.
In release
1.n+2
1.12
to
1.13
A
1.12
1.14
The
.
for GCE, AWS, Vsphere and OpenSt

ack.
namespace
pod name
priority
cpu
)
cores
)
of
Never
or
OnFailure
Succeeded
or
Failed
and
.
/metrics
desired if, for example, a metric is causing a performance problem. The input is a list of
).
where
12.6 - System Logs
Klog
klog
.
Kubernetes is in 

the process of simplifying logging in its components.
--log-dir
and as
Usage
Warning:
Log formatting and value serialization are subject to change.
=
Example:
=
%+v
.
KEP
.
or
WithName
in
behaves

 that supports contextual logging.
$
cd
$GOPATH
$
...
$
=
true
...
The
example
prefix and
foo="bar"
runtime
and
WithName
$
=
false
...
Warning:
.
The
{
"ts"
:
,
"v"
:
4
,
"msg"
:
,
"pod"
:{
"name"
:
"nginx-1"
,
:
"default"
},
"status"
:
"ready"
}
ts
v
err
msg
kubelet
The
-v
.log
/var/log
.log
/var/log
/var/log
logrotate
tool.
The
logrotate
Read about
Read about
Read about
12.7 - Traces For Kubernetes System Components
.
For a complete guide t

o collecting traces and using the collector, see
.
, 4317.
receivers
:
otlp
:
protocols
:
grpc
:
exporters
:
logging
:
logLevel
:
debug
service
:
pipelines
:
traces
:
receivers
:
[otlp]
exporters
:
[logging]
with
:
kind
:
:
100
.
.
:
kind
:
:
:
true
tracing
:
:
100
Stability
Read about
12.8 - Proxies in Kubernetes
Proxies
The
:
The
:
The
kube proxy
:
A Proxy/Load-ba

lancer in front of apiserver(s):
12.9 - API Priority and Fairness
(i.e. the
and
need not
Caution:
command execution or log tailing—are not subject to the API
does
apply to
watch
watch
requests
limit.
Feature
Gates
API Group
with: (a) a
v1alpha1
v1beta1
(b)
v1beta2
and
v1beta3
\
=
=
false
\
=
=
=
false
\
with
.
Concepts
isolation by maintaining separate concurrency lim

its, so that requests assigned
flows
flags
and
list
on the server, among requests that take a similar amount of time to
watch
how long a
watch
watch
begin with
create
watch
is over.
relevant
watch
Queuing
the same priority level. Each request is assigned to

 a
flow
— which
server.
Resources
v1alpha1
version
distributed among the existing PriorityLevelConfigurations in
(or
), and all
Caution:
v1beta3
Caution:
and
between mutating and 

non-mutating requests; if you want to treat them
type
A type of
Reject
Queue
queues
Changing
handSize
Note:
A larger
handSize
handSize
also
.
HandSize
Queues
1 elephant
12
32
10
32
10
64
9
64
8
64
8
128
7
128
7
256
6
256
6
512
6
1024
and
Caution:
smaller
name
.
rules
subjects
and
at least
one of its
or
incoming r

equest is for a resource or non-resource URL) matches the request.
For the
name
verbs
,
apiGroups
,
resources
,
, and
*
either
ByUser
may be
Defaults
exempt
exempt
catch-all
catch-all
mandatory
catch-all
priority level has a very small concurrency
The
node-high
The
system
them.
The
endpoints
,
,
or
leases
or
The
The
Pods.
The
kubectl
Each
kube

-apiserver
Each
alone.
. If
true
then the
false
of the
introduced in release 1.22 and their consideration of
to
false
.
.
Caution:
:
kind
:
metadata
:
name
:
spec
:
:
1000
:
name
:
exempt
rules
:
-
:
-
no

nResourceURLs
:
-
-
"/livez"
-
"/readyz"
verbs
:
-
"*"
subjects
:
-
kind
:
Group
group
:
name
:
and
=
=
Metrics
Note:
and
and
,
reason
. The
reason
queued,
time-out
vector (cumulative since server start) of requests that began
mutating
or
readOnly
.
served.
is a
phase
waiting
and
executing
) and
mutating
and
readOnly
and
.
and
.
the labels
and
.
apiserver_flowcontrol_priority_level_request

_utilization
is a
phase
waiting
and
executing
) and
is a
with
phase).
is a
the labels
and
Note:
is a gauge hold

ing the fair
execute
Note:
the labels
and
and
.
and
.
is a
and
. The
a request.
PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests,
...
syste

m,            system-nodes,   12,         0,                   system:node:127.0.0.1, 2020-07-23T15:26:57.179170694Z,
the
.
.
12.10 - Installing Addons
Note:
ACI
Antrea
Calico
is a networking and network policy provider. Calico supports a flexible set of networking options so you can choose the most efficient option for your situation, including non-overlay and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts, pods, and (if using Istio &amp; Envoy) applications at the service mesh layer.
Canal
Cilium
CNI-Genie
Contiv
. The
installer
Contrail
, based on
Flannel
Knitter
Multus
is a Multi plugin for multiple network support in Kubernetes to support all C

NI plugins (e.g. Calico, Cilium, Contiv, Flannel), in addition to SRIOV, DPDK, OVS-DPDK and VPP based workloads in Kubernetes.
Nodus
NSX-T
Nuage
Romana
API.
Weave Net
CoreDNS
installed
Dashboard
KubeVirt
The
Events
or
.
13 - Extending Kubernetes
Kubernetes is highly configurable and extensible. As a result, there is rarely a need to fork or
submit patches to the Kubernetes project code.
This guide describes the options for customizing a Kubernetes cluster. It is aimed at
who want to understand
how to adapt their Kubernetes cluster to the needs of their work environment. Developers who are
prospective
Platform Developers
or
Kubernetes Project
Contributors
will also
find it useful as an introduction to what extension points and patterns exist, and their
trade-offs and limitations.
Customization approaches can be broadly divided into
, which only
involves changing command line arguments, local configuration files, or API resources; and
,
which involve running additional programs, additional network services, or both.
This document is primarily about
.
Configuration files
and
command arguments
are documented in the
Reference
section of the online
documentation, with a page for each binary:
kubelet
Command arguments and configuration files may not always be changeable in a hosted Kubernetes service or a
distribution with managed installation. When they are changeable, they are usually only changeable
by the cluster operator. Also, they are subject to change in future Kubernetes versions, and
setting them may require restarting processes. For those reasons, they should be used only when
there are no other options.
Built-in
policy APIs
, such as
,
and Role-based Access Control
(
RBAC
), are built-in Kubernetes APIs that provide declaratively configured policy settings.
APIs are typically usable even with hosted Kubernetes services and with managed Kubernetes installations.
The built-in policy APIs follow the same conventions as other Kubernetes resources such as Pods.
When you use a po

licy APIs that is
stable
, you benefit from a
defined support policy
like other Kubernetes APIs.
For these reasons, policy APIs are recommended over
and
where suitable.
Extensions are software components that extend and deeply integrate with Kubernetes.
They adapt it to support new types and new kinds of hardware.
Many cluster administrators use a hosted or distribution instance of Kubernetes.
These clusters come with extensions pre-installed. As a result, most Kubernetes
users will not need to install extensions and even fewer users will need to author new ones.
Extension patterns
Kubernetes is designed to be automated by writing client programs. Any
program that reads and/or writes to the Kubernetes API can provide useful
automation.
Automation
can run on the cluster or off it. By following
the guidance in this doc you can write highly available and robust automation.
Automation generally works with any Kubernetes cluster, including hosted
clusters and managed installations.
There is a specific pattern for writing client programs that work well with
Kubernetes called the
pattern. Controllers typically read an object's
.spec
, possibly do things, and then
update the object's
.status
.
A controller is a client of the Kubernetes API. When Kubernetes is the client and calls
out to a remote service, Kubernetes calls this a
webhook
. The remote service is called
a
webhook backend
. As with custom controllers, webhooks do add a point of failure.
Note:
Outside of Kubernetes, the term “webhook” typically refers to a mechanism for asynchronous
notifications, where the webhook call serves as a one-way notification to another system or
component. In the Kubernetes ecosystem, even synchronous HTTP callouts are often
described as “webhooks”.
In the webhook model, Kubernetes makes a network request to a remote service.
With the alternative
binary Plugin
model, Kubernetes executes a binary (program).
Binary plugins are used by the kubelet (for example,
CSI storage plugins
and
CNI network plugins
),
and by kubectl (see
).
This diagram shows the extension points in a Kubernetes cluster and the
clients that access it.
Kubernetes extension points
Key to the figure
Users often interact with the Kubernetes API using
kubectl
.
Plugins
customise the behaviour of clients. There are generic extensions that can apply to different clients,
as well as specific ways to extend
kubectl
.
The API server handles all requests. Several types of extension points in the API server allow
authenticating requests, or blocking them based on their content, editing content, and handling
deletion. These are described in the
API Access Extensions
section.
The API server serves various kinds of
resources
.
Built-in resource kinds
, such as
pods
, are defined by the Kubernetes project and can't be changed.
Read
to learn about extending the Kubernetes API.
The Kubernetes scheduler
decides
which nodes to place pods on. There are several ways to extend scheduling, which are
described in the
Scheduling extensions
section.
Much of the behavior of Kubernetes is implemented by programs called
, that are
clients of the API server. Controllers are often used in conjunction with custom resources.
Read
combining new APIs with automation
and
Changing built-in resources
The kubelet runs on servers (nodes), and helps pods appear like virtual servers with their own IPs on
the cluster network.
allow for different implementations of
pod networking.
You can use
to integrate custom hardware or other special
node-local facilities, and make these available to Pods running in your cluster. The kubelet
includes support for working with device plugins.
The kubelet also mounts and unmounts
volume
for pods and their containers.
You can use
Storage Plugins
to add support for new kinds
of storage and other volume types.
Exte

nsion point choice flowchart
If you are unsure where to start, this flowchart can help. Note that some solutions may involve
several types of extensions.
Flowchart guide to select an extension approach
Client extensions
Plugins for kubectl are separate binaries that add or replace the behavior of specific subcommands.
The
kubectl
tool can also integrate with
credential plugins
These extensions only affect a individual user's local environment, and so cannot enforce site-wide policies.
If you want to extend the
kubectl
tool, read
.
Custom resource definitions
Consider adding a
Custom Resource
to Kubernetes if you want to define new controllers, application
configuration objects or other declarative APIs, and to manage them using Kubernetes tools, such
as
kubectl
.
For more about Custom Resources, see the
concept guide.
You can use Kubernetes'
API Aggregation Layer
to integrate the Kubernetes API with additional services such as for
metrics
.
Combining new APIs with automation
A combination of a custom resource API and a control loop is called the
pattern. If your controller takes
the place of a human operator deploying infrastructure based on a desired state, then the controller
may also be following the
.
The Operator pattern is used to manage specific applications; usually, these are applications that
maintain state and require care in how they are managed.
You can also make your own custom APIs and control loops that manage other resources, such as storage,
or to define policies (such as an access control restriction).
When you extend the Kubernetes API by adding custom resources, the added resources always fall
into a new API Groups. You cannot replace or change existing API groups.
Adding an API does not directly let you affect the behavior of existing APIs (such as Pods), whereas
do.
API access extensions
When a request reaches the Kubernetes API Server, it is first
, then
,
and is then subject to various types of
admission control
(some requests are in fact not
authenticated, and get special treatment). See
for more on this flow.
Each of the steps in the Kubernetes authentication / authorization flow offers extension points.
maps headers or certificates
in all requests to a username for the client making the request.
Kubernetes has several built-in authentication methods that it supports. It can also sit behind an
authenticating proxy, and it can send a token from an
Authorization:
header to a remote service for
verification (an
)
if those don't meet your needs.
determines whether specific
users can read, write, and do other operations on API resources. It works at the level of whole
resources -- it doesn't discriminate based on arbitrary object fields.
If the built-in authorization options don't meet your needs, an
authorization webhook
allows calling out to custom code that makes an authorization decision.
Dynamic admission control
After a request is authorized, if it is a write operation, it also goes through
Admission Control
steps.
In addition to the built-in steps, there are several extensions:
The
Image Policy webhook
restricts what images can be run in containers.
To make arbitrary admission control decisions, a general
Admission webhook
can be used. Admission webhooks can reject creations or updates.
Some admission webhooks modify the incoming request data before it is handled further by Kubernetes.
Infrastructure extensions
Device plugins
allow a node to discover new Node resources (in addition to the
builtin ones like cpu and memory) via a
.
Storage plugins
(CSI) plugins provide
a way to extend Kubernetes with supports for new kinds of volumes. The vol

umes can be backed by
durable external storage, or provide ephemeral storage, or they might offer a read-only interface
to information using a filesystem paradigm.
Kubernetes also includes support for
plugins,
which are deprecated since Kubernetes v1.23 (in favour of CSI).
FlexVolume plugins allow users to mount volume types that aren't natively supported by Kubernetes. When
you run a Pod that relies on FlexVolume storage, the kubelet calls a binary plugin to mount the volume.
The archived
design proposal has more detail on this approach.
The
Kubernetes Volume Plugin FAQ for Storage Vendors
includes general information on storage plugins.
Network plugins
Your Kubernetes cluster needs a
in order to have a working Pod network
and to support other aspects of the Kubernetes network model.
allow Kubernetes to work with different networking topologies and technologies.
The scheduler is a special type of controller that watches pods, and assigns
pods to nodes. The default scheduler can be replaced entirely, while
continuing to use other Kubernetes components, or
multiple schedulers
can run at the same time.
This is a significant undertaking, and almost all Kubernetes users find they
do not need to modify the scheduler.
You can control which
scheduling plugins
are active, or associate sets of plugins with different named
scheduler profiles
.
You can also write your own plugin that integrates with one or more of the kube-scheduler's
.
Finally, the built in
component supports a
webhook
that permits a remote HTTP backend (scheduler extension) to filter and / or prioritize
the nodes that the kube-scheduler chooses for a pod.
Note:
You can only affect node filtering
and node prioritization with a scheduler extender webhook; other extension points are
not available through the webhook integration.
Learn more about infrastructure extensions
CSI
storage plugins
kubectl plugins
Extension API Servers
13.1 - Compute, Storage, and Networking Extensions
This section covers extensions to your cluster that do not come as part as Kubernetes itself.
You can use these extensions to enhance the nodes in your cluster, or to provide the network
fabric that links Pods together.
CSI
and
(CSI) plugins
provide a way to extend Kubernetes with supports for new kinds of volumes. The volumes can
be backed by durable external storage, or provide ephemeral storage, or they might offer a
read-only interface to information using a filesystem paradigm.
plugins, which are deprecated since Kubernetes v1.23 (in favour of CSI).
FlexVolume plugins allow users to mount volume types that aren't natively
supported by Kubernetes. When you run a Pod that relies on FlexVolume
storage, the kubelet calls a binary plugin to mount the volume. The archived
The
Device plugins allow a node to discover new Node facilities (in addition to the
built-in node resources such as
cpu
and
memory
), and provide these custom node-local
facilities to Pods that request them.
A network plugin allow Kubernetes to work with different networking topologies and technologies.
Kubernetes 1.26 is compatible with
CNI
network plugins.
13.1.1 - Network Plugins
Kubernetes 1.26 supports
(CNI) plugins for cluster networking. You must use a CNI plugin that is compatible with

 your
cluster and that suits your needs. Different plugins are available (both open- and closed- source)
in the wider Kubernetes ecosystem.
A CNI plugin is required to implement the
Kubernetes network model
.
You must use a CNI plugin that is compatible with the
v0.4.0
or later
releases of the CNI specification. The Kubernetes project recommends using a plugin that is
compatible with the
v1.0.0
CNI specification (plugins can be compatible with multiple spec versions).
A Container Runtime, in the networking context, is a daemon on a node configured to provide CRI
Services for kubelet. In particular, the Container Runtime must be configured to load the CNI
plugins required to implement the Kubernetes network model.
Note:
Prior to Kubernetes 1.24, the CNI plugins could also be managed by the kubelet using the
cni-bin-dir
and
network-plugin
command-line parameters.
These command-line parameters were removed in Kubernetes 1.24, with management of the CNI no
longer in scope for kubelet.
See
if you are facing issues following the removal of dockershim.
For specific information about how a Container Runtime manages the CNI plugins, see the
documentation for that Container Runtime, for example:
CRI-O
For specific information about how to install and manage a CNI plugin, see the documentation for
that plugin or
networking provider
.
Network Plugin Requirements
For plugin developers and users who regularly build or deploy Kubernetes, the plugin may also need
specific configuration to support kube-proxy. The iptables proxy depends on iptables, and the
plugin may need to ensure that container traffic is made available to iptables. For example, if
the plugin connects containers to a Linux bridge, the plugin must set the
net/bridge/bridge-nf-call-iptables
sysctl to
1
to ensure that the iptables proxy functions
correctly. If the plugin does not use a Linux bridge, but uses something like Open vSwitch or
some other mechanism instead, it should ensure container traffic is appropriately routed for the
proxy.
By default, if no kubelet network plugin is specified, the
noop
plugin is used, which sets
net/bridge/bridge-nf-call-iptables=1
to ensure simple configurations (like Docker with a bridge)
work correctly with the iptables proxy.
Loopback CNI
In addition to the CNI plugin installed on the nodes for implementing the Kubernetes network
model, Kubernetes also requires the container runtimes to provide a loopback interface
lo
, which
is used for each sandbox (pod sandboxes, vm sandboxes, ...).
Implementing the loopback interface can be accomplished by re-using the
CNI loopback plugin.
or by developing your own code to achieve this (see
this example from CRI-O
).
Support hostPort
The CNI networking plugin supports
hostPort
. You can use the official
portmap
plugin offered by the CNI plugin team or use your own plugin with portMapping functionality.
If you want to enable
hostPort
support, you must specify
portMappings capability
in your
cni-conf-dir
{
"name"
:
"k8s-pod-network"
,
"cniVersion"
:
"0.4.0"
,
"plugins"
: [
{
"type"
:
"calico"
,
"log_level"
:
"info"
,
"datastore_type"
:
,
"nodename"
:
,
"ipam"
: {
"type"
:
"host-local"
,
"subnet"
:
"usePodCidr"
},
"policy"
: {
"type"
:
"k8s"
},
: {
"kubeconfig"
:
"/etc/cni/net.d/calico-kubeconfig"
}
},
{
"type"
:
"portmap"
,
"capabilities"
: {
"portMappings"
:
true
},
"externalSetMarkChain"
:
"KUBE-MARK-MASQ"
}
]
}
Support traffic shaping
Experimental Feature
The CNI networking plugin also supports pod ingress and egress traffic shaping. You can use the
official
bandwidth
plugin offered by the CNI plugin team or use your own plugin with bandwidth control functionality.
bandwidth
plugin to your CNI
configuration file (default
) and ensure that the binary is included in your CNI
bin dir (default
).
{

"name"
:
,
:
"0.4.0"
,
"plugins"
: [
{
"type"
:
"calico"
,
:
"info"
,
:
,
:
,
"ipam"
: {
"type"
:
,
"subnet"
:
},
"policy"
: {
"type"
:
"k8s"
},
: {
:
}
},
{
"type"
:
"bandwidth"
,
: {
:
true
}
}
]
}
Now you can add the
and
annotations to your Pod. For example:
:
v1
kind
:
Pod
metadata
:
:
:
1M
:
1M
...
13.1.2 - Device Plugins
Device plugins let you configure your cluster with support for devices or resources that require vendor-specific setup, such as GPUs, NICs, FPGAs, or non-volatile main memory.
device plugin framework
that you can use to advertise system hardware resources to the
Kubelet
.
Instead of customizing the code for Kubernetes itself, vendors can implement a
device plugin that you deploy either manually or as a
DaemonSet
.
The targeted devices include GPUs, high-performance NICs, FPGAs, InfiniBand adapters,
and other similar computing resources that may require vendor specific initialization
and setup.
Device plugin registration
The kubelet exports a
Registration
gRPC service:
service Registration {
	rpc Register(RegisterRequest) returns (Empty) {}
}
A device plugin can register itself with the kubelet through this gRPC service.
During the registration, the device plugin needs to send:
The name of its Unix socket.
The Device Plugin API version against which it was built.
The
ResourceName
it wants to advertise. Here
needs to follow the
extended resource naming scheme
as
vendor-domain/resourcetype
.
(For example, an NVIDIA GPU is advertised as
.)
Following a successful registration, the device plugin sends the kubelet the
list of devices it manages, and the kubelet is then in charge of advertising those
resources to the API server as part of the kubelet node status update.
For example, after a device plugin registers
hardware-vendor.example/foo
with the kubelet
and reports two healthy devices on a node, the node status is updated
to advertise that the node has 2 "Foo" devices installed and available.
Then, users can request devices as part of a Pod specification
(see
container
).
Requesting extended resources is similar to how you manage requests and limits for
other resources, with the following differences:
Extended resources are only supported as integer resources and cannot be overcommitted.
Devices cannot be shared between containers.
Example
Suppose a Kubernetes cluster is running a device plugin that advertises resource
on certain nodes. Here is an example of a pod requesting this resource to run a demo workload:
---
:
v1
kind
:
Pod
metadata
:
name
:
demo-pod
spec
:
:
-
name
:
demo-container-1
image
:
resources
:
limits
:
:
2
#
# This Pod needs 2 of the hardware-vendor.example/foo devices
# and can only schedule onto a Node that's able to satisfy
# that need.
#
# If the Node has more than 2 of those devices available, the
# remainder would be available for other Pods to use.
Device plugin implementation
The general workflow of a device plugin includes the following steps:
Initialization. During this phase, the device plugin performs vendor specific
initialization and setup to make sure the devices are in a ready state.
The plugin starts a gRPC service, with a Unix socket under host path
/var/lib/kubelet/device-plugins/
, that implements the following interfaces:
service DevicePlugin {
      // GetDevicePluginOptions returns options to be communicated with Device Manager.
      rpc GetDevicePluginOptions(Empty) returns (DevicePluginOptions) {}

      //

 ListAndWatch returns a stream of List of Devices
      // Whenever a Device state change or a Device disappears, ListAndWatch
      // returns the new list
      rpc ListAndWatch(Empty) returns (stream ListAndWatchResponse) {}

      // Allocate is called during container creation so that the Device
      // Plugin can run device specific operations and instruct Kubelet
      // of the steps to make the Device available in the container
      rpc Allocate(AllocateRequest) returns (AllocateResponse) {}

      // GetPreferredAllocation returns a preferred set of devices to allocate
      // from a list of available ones. The resulting preferred allocation is not
      // guaranteed to be the allocation ultimately performed by the
      // devicemanager. It is only designed to help the devicemanager make a more
      // informed allocation decision when possible.
      rpc GetPreferredAllocation(PreferredAllocationRequest) returns (PreferredAllocationResponse) {}

      // PreStartContainer is called, if indicated by Device Plugin during registeration phase,
      // before each container start. Device plugin can run device specific operations
      // such as resetting the device before making devices available to the container.
      rpc PreStartContainer(PreStartContainerRequest) returns (PreStartContainerResponse) {}
}
Note:
Plugins are not required to provide useful implementations for
GetPreferredAllocation()
or
PreStartContainer()
. Flags indicating which
(if any) of these calls are available should be set in the
DevicePluginOptions
message sent back by a call to
GetDevicePluginOptions()
. The
kubelet
will
always call
to see which optional functions are
available, before calling any of them directly.
The plugin registers itself with the kubelet through the Unix socket at host
path
/var/lib/kubelet/device-plugins/kubelet.sock
.
After successfully registering itself, the device plugin runs in serving mode, during which it keeps
monitoring device health and reports back to the kubelet upon any device state changes.
It is also responsible for serving
Allocate
gRPC requests. During
Allocate
, the device plugin may
do device-specific preparation; for example, GPU cleanup or QRNG initialization.
If the operations succeed, the device plugin returns an
AllocateResponse
that contains container
runtime configurations for accessing the allocated devices. The kubelet passes this information
to the container runtime.
Handling kubelet restarts
A device plugin is expected to detect kubelet restarts and re-register itself with the new
kubelet instance. A new kubelet instance deletes all the existing Unix sockets under
/var/lib/kubelet/device-plugins
when it starts. A device plugin can monitor the deletion
of its Unix socket and re-register itself upon such an event.
Device plugin deployment
You can deploy a device plugin as a DaemonSet, as a package for your node's operating system,
or manually.
The canonical directory
requires privileged access,
so a device plugin must run in a privileged security context.
If you're deploying a device plugin as a DaemonSet,
must be mounted as a
Volume
in the plugin's
PodSpec
.
If you choose the DaemonSet approach you can rely on Kubernetes to: place the device plugin's
Pod onto Nodes, to restart the daemon Pod after failure, and to help automate upgrades.
Previously, the versioning scheme required the Device Plugin's API version to match
exactly the Kubelet's version. Since the graduation of this feature to Beta in v1.12
this is no longer a hard requirement. The API is versioned and has been stable since
Beta graduation of this feature. Because of this, kubelet upgrades should be seamless
but there still may be changes in the API before stabilization making upgrades not
guaranteed to be non-breaking.
Note:
Although the Device Manager component of Kubernetes is a generally available feature,
the
devi

ce plugin API
is not stable. For information on the device plugin API and
version compatibility, read
Device Plugin API versions
.
As a project, Kubernetes recommends that device plugin developers:
Watch for Device Plugin API changes in the future releases.
Support multiple versions of the device plugin API for backward/forward compatibility.
To run device plugins on nodes that need to be upgraded to a Kubernetes release with
a newer device plugin API version, upgrade your device plugins to support both versions
before upgrading these nodes. Taking that approach will ensure the continuous functioning
of the device allocations during the upgrade.
Monitoring device plugin resources
In order to monitor resources provided by device plugins, monitoring agents need to be able to
discover the set of devices that are in-use on the node and obtain metadata to describe which
container the metric should be associated with.
metrics
exposed by device monitoring agents should follow the
Kubernetes Instrumentation Guidelines
,
identifying containers using
pod
,
namespace
, and
container
prometheus labels.
The kubelet provides a gRPC service to enable discovery of in-use devices, and to provide metadata
for these devices:
// PodResourcesLister is a service provided by the kubelet that provides information about the
// node resources consumed by pods and containers on the node
service PodResourcesLister {
    rpc List(ListPodResourcesRequest) returns (ListPodResourcesResponse) {}
    rpc GetAllocatableResources(AllocatableResourcesRequest) returns (AllocatableResourcesResponse) {}
}
List
gRPC endpoint
The
List
endpoint provides information on resources of running pods, with details such as the
id of exclusively allocated CPUs, device id as it was reported by device plugins and id of
the NUMA node where these devices are allocated. Also, for NUMA-based machines, it contains the
information about memory and hugepages reserved for a container.
// ListPodResourcesResponse is the response returned by List function
message ListPodResourcesResponse {
    repeated PodResources pod_resources = 1;
}

// PodResources contains information about the node resources assigned to a pod
message PodResources {
    string name = 1;
    string namespace = 2;
    repeated ContainerResources containers = 3;
}

// ContainerResources contains information about the resources assigned to a container
message ContainerResources {
    repeated ContainerDevices devices = 2;
    repeated int64 cpu_ids = 3;
    repeated ContainerMemory memory = 4;
}

// ContainerMemory contains information about memory and hugepages assigned to a container
message ContainerMemory {
    string memory_type = 1;
    uint64 size = 2;
    TopologyInfo topology = 3;
}

// Topology describes hardware topology of the resource
message TopologyInfo {
        repeated NUMANode nodes = 1;
}

// NUMA representation of NUMA node
message NUMANode {
        int64 ID = 1;
}

// ContainerDevices contains information about the devices assigned to a container
message ContainerDevices {
    string resource_name = 1;
    repeated string device_ids = 2;
}
Note:
cpu_ids in the
ContainerResources
in the
List
endpoint correspond to exclusive CPUs allocated
to a particular container. If the goal is to evaluate CPUs that belong to the shared pool, the
List
endpoint needs to be used in conjunction with the
endpoint as explained
below:
Call
to get a list of all the allocatable CPUs
Call
GetCpuIds
on all
in the system
Subtract out all of the CPUs from the
GetCpuIds
calls from the
call
GetAllocatableResources provides information on resources initially available on the worker node.
It provides more information than kubelet exports to APIServer.
Note:
GetAlloc

atableResources
should only be used to evaluate
resources on a node. If the goal is to evaluate free/unallocated resources it should be used in
conjunction with the List() endpoint. The result obtained by
would remain
the same unless the underlying resources exposed to kubelet change. This happens rarely but when
it does (for example: hotplug/hotunplug, device health changes), client is expected to call
GetAlloctableResources
endpoint.
However, calling
endpoint is not sufficient in case of cpu and/or memory
update and Kubelet needs to be restarted to reflect the correct resource capacity and allocatable.
// AllocatableResourcesResponses contains informations about all the devices known by the kubelet
message AllocatableResourcesResponse {
    repeated ContainerDevices devices = 1;
    repeated int64 cpu_ids = 2;
    repeated ContainerMemory memory = 3;
}
Starting from Kubernetes v1.23, the
is enabled by default.
You can disable it by turning off the
.
Preceding Kubernetes v1.23, to enable this feature
kubelet
--feature-gates=KubeletPodResourcesGetAllocatable=true
ContainerDevices
do expose the topology information declaring to which NUMA cells the device is
affine. The NUMA cells are identified using a opaque integer ID, which value is consistent to
what device plugins report
when they register themselves to the kubelet
.
The gRPC service is served over a unix socket at
/var/lib/kubelet/pod-resources/kubelet.sock
.
Monitoring agents for device plugin resources can be deployed as a daemon, or as a DaemonSet.
/var/lib/kubelet/pod-resources
requires privileged access, so monitoring
agents must run in a privileged security context. If a device monitoring agent is running as a
DaemonSet,
Volume
in the device monitoring agent's
PodSpec
.
Support for the
PodResourcesLister service
requires
It is enabled by default starting with Kubernetes 1.15 and is v1 since Kubernetes 1.20.
Device plugin integration with the Topology Manager
The Topology Manager is a Kubelet component that allows resources to be co-ordinated in a Topology
aligned manner. In order to do this, the Device Plugin API was extended to include a
TopologyInfo
struct.
    repeated NUMANode nodes = 1;
}

    int64 ID = 1;
}
Device Plugins that wish to leverage the Topology Manager can send back a populated TopologyInfo
struct as part of the device registration, along with the device IDs and the health of the device.
The device manager will then use this information to consult with the Topology Manager and make
resource assignment decisions.
supports setting a
nodes
nil
or a list of NUMA nodes. This
allows the Device Plugin to advertise a device that spans multiple NUMA nodes.
Setting
to
nil
or providing an empty list of NUMA nodes for a given device
indicates that the Device Plugin does not have a NUMA affinity preference for that device.
An example
struct populated for a device by a Device Plugin:
pluginapi.Device{ID: "25102017", Health: pluginapi.Healthy, Topology:&amp;pluginapi.TopologyInfo{Nodes: []*pluginapi.NUMANode{&amp;pluginapi.NUMANode{ID: 0,},}}}
Device plugin examples
Note:
Here are some examples of device plugin implementations:
The
AMD GPU device plugin
The
Intel device plugins
for
Intel GPU, FPGA, QAT, VPU, SGX, DSA, DLB and IAA devices
The
Kube

Virt device plugins
for
hardware-assisted virtualization
The
NVIDIA GPU device plugin for Container-Optimized OS
The
RDMA device plugin
The
SocketCAN device plugin
The
Solarflare device plugin
The
SR-IOV Network device plugin
The
Xilinx FPGA device plugins
for Xilinx FPGA devices
scheduling GPU resources
using device
plugins
advertising extended resources
on a node
Read about using
hardware acceleration for TLS ingress
with Kubernetes
13.2 - Extending the Kubernetes API
13.2.1 - Custom Resources
are extensions of the Kubernetes API. This page discusses when to add a custom
resource to your Kubernetes cluster and when to use a standalone service. It describes the two
methods for adding custom resources and how to choose between them.
A
resource
is an endpoint in the
that
stores a collection of
of a certain kind; for example, the built-in
pods
resource contains a collection of Pod objects.
A
is an extension of the Kubernetes API that is not necessarily available in a default
Kubernetes installation. It represents a customization of a particular Kubernetes installation. However,
many core Kubernetes functions are now built using custom resources, making Kubernetes more modular.
Custom resources can appear and disappear in a running cluster through dynamic registration,
and cluster admins can update custom resources independently of the cluster itself.
Once a custom resource is installed, users can create and access its objects using
kubectl
, just as they do for built-in resources
like
Pods
.
Custom controllers
On their own, custom resources let you store and retrieve structured data.
When you combine a custom resource with a
custom controller
, custom resources
provide a true
declarative API
.
enforces a separation of responsibilities. You declare the desired state of
your resource. The Kubernetes controller keeps the current state of Kubernetes
objects in sync with your declared desired state. This is in contrast to an
imperative API, where you
instruct
a server what to do.
You can deploy and update a custom controller on a running cluster, independently
of the cluster's lifecycle. Custom controllers can work with any kind of resource,
but they are especially effective when combined with custom resources. The
combines custom
resources and custom controllers. You can use custom controllers to encode domain knowledge
for specific applications into an extension of the Kubernetes API.
Should I add a custom resource to my Kubernetes cluster?
When creating a new API, consider whether to
aggregate your API with the Kubernetes cluster APIs
or let your API stand alone.
Consider API aggregation if:
Prefer a stand-alone API if:
Your API is
Declarative
.
Your API does not fit the
model.
You want your new types to be readable and writable using
kubectl
.
kubectl
support is not required
You want to view your new types in a Kubernetes UI, such as dashboard, alongside built-in types.
Kubernetes UI support is not required.
You are developing a new API.
You already have a program that serves your API and works well.
You are willing to accept the format restriction that Kubernetes puts on REST resource paths, such as API Groups and Namespaces. (See the
.)
You need to have specific REST paths to be compatible with an already defined REST API.
Your resources are naturally scoped to a cluster or namespaces of a cluster.
Cluster or namespace scoped resources are a poor fit; you need control over the specifics of resource paths.
You want to reuse
Kubernetes API support features
.
You don't need those features.
Declarative APIs
In a Declarative API, typically:
Your API consists of a relatively small number of relatively small objects (resources).
The objects define configuration of applications or infrastructure.
The objects are updated 

relatively infrequently.
Humans often need to read and write the objects.
The main operations on the objects are CRUD-y (creating, reading, updating and deleting).
Transactions across objects are not required: the API represents a desired state, not an exact state.
Imperative APIs are not declarative.
Signs that your API might not be declarative include:
The client says "do this", and then gets a synchronous response back when it is done.
The client says "do this", and then gets an operation ID back, and has to check a separate
Operation object to determine completion of the request.
You talk about Remote Procedure Calls (RPCs).
Directly storing large amounts of data; for example, &gt; a few kB per object, or &gt; 1000s of objects.
High bandwidth access (10s of requests per second sustained) needed.
Store end-user data (such as images, PII, etc.) or other large-scale data processed by applications.
The natural operations on the objects are not CRUD-y.
The API is not easily modeled as objects.
You chose to represent pending operations with an operation ID or an operation object.
Should I use a ConfigMap or a custom resource?
Use a ConfigMap if any of the following apply:
There is an existing, well-documented configuration file format, such as a
mysql.cnf
or
pom.xml
.
You want to put the entire configuration into one key of a ConfigMap.
The main use of the configuration file is for a program running in a Pod on your cluster to
consume the file to configure itself.
Consumers of the file prefer to consume via file in a Pod or environment variable in a pod,
rather than the Kubernetes API.
You want to perform rolling updates via Deployment, etc., when the file is updated.
Note:
Use a
Secret
for sensitive data, which is similar
to a ConfigMap but more secure.
Use a custom resource (CRD or Aggregated API) if most of the following apply:
You want to use Kubernetes client libraries and CLIs to create and update the new resource.
You want top-level support from
kubectl
; for example,
kubectl get my-object object-name
.
You want to build new automation that watches for updates on the new object, and then CRUD other
objects, or vice versa.
You want to write automation that handles updates to the object.
You want to use Kubernetes API conventions like
.spec
,
.status
, and
.metadata
.
You want the object to be an abstraction over a collection of controlled resources, or a
summarization of other resources.
Adding custom resources
Kubernetes provides two ways to add custom resources to your cluster:
CRDs are simple and can be created without any programming.
requires programming, but allows more control over API behaviors like how data is stored and
conversion between API versions.
Kubernetes provides these two options to meet the needs of different users, so that neither ease
of use nor flexibility is compromised.
Aggregated APIs are subordinate API servers that sit behind the primary API server, which acts as
a proxy. This arrangement is called
(AA).
To users, the Kubernetes API appears extended.
CRDs allow users to create new types of resources without adding another API server. You do not
need to understand API Aggregation to use CRDs.
Regardless of how they are installed, the new resources are referred to as Custom Resources to
distinguish them from built-in Kubernetes resources (like pods).
Note:
Avoid using a Custom Resource as data storage for application, end user, or monitoring data:
architecture designs that store application data within the Kubernetes API typically represent
a design that is too closely coupled.
Architecturally,
cloud native
application architectures
favor loose coupling between components. If part of your workload requires a backing service for
its routine operation, run that backing service as a component or consume it as an external service.
This way, your workload does not rely on the Kubernetes API for its normal operation.
The

API resource allows you to define custom resources.
Defining a CRD object creates a new custom resource with a name and schema that you specify.
The Kubernetes API serves and handles the storage of your custom resource.
The name of a CRD object must be a valid
.
This frees you from writing your own API server to handle the custom resource,
but the generic nature of the implementation means you have less flexibility than with
API server aggregation
.
custom controller example
for an example of how to register a new custom resource, work with instances of your new resource type,
and use a controller to handle events.
Usually, each resource in the Kubernetes API requires code that handles REST requests and manages
persistent storage of objects. The main Kubernetes API server handles built-in resources like
pods
and
services
, and can also generically handle custom resources through
CRDs
.
The
allows you to provide specialized implementations for your custom resources by writing and
deploying your own API server.
The main API server delegates requests to your API server for the custom resources that you handle,
making them available to all of its clients.
Choosing a method for adding custom resources
CRDs are easier to use. Aggregated APIs are more flexible. Choose the method that best meets your needs.
Typically, CRDs are a good fit if:
You have a handful of fields
You are using the resource within your company, or as part of a small open-source project (as
opposed to a commercial product)
Comparing ease of use
CRDs are easier to create than Aggregated APIs.
CRDs
Aggregated API
Do not require programming. Users can choose any language for a CRD controller.
Requires programming and building binary and image.
No additional service to run; CRDs are handled by API server.
An additional service to create and that could fail.
No ongoing support once the CRD is created. Any bug fixes are picked up as part of normal Kubernetes Master upgrades.
May need to periodically pickup bug fixes from upstream and rebuild and update the Aggregated API server.
No need to handle multiple versions of your API; for example, when you control the client for this resource, you can upgrade it in sync with the API.
You need to handle multiple versions of your API; for example, when developing an extension to share with the world.
Advanced features and flexibility
Aggregated APIs offer more advanced API features and customization of other features; for example, the storage layer.
Feature
CRDs
Help users prevent errors and allow you to evolve your API independently of your clients. These features are most useful when there are many clients who can't all update at the same time.
Yes. Most validation can be specified in the CRD using
. Any other validations supported by addition of a
Validating Webhook
.
Yes, arbitrary validation checks
See above
Yes, either via
default
keyword (GA in 1.17), or via a
Mutating Webhook
(though this will not be run when reading from etcd for old objects).
Yes
Multi-versioning
Allows serving the same object through two API versions. Can help ease API changes like renaming fields. Less important if you control your client versions.
Yes
Yes
Custom Storage
If you need storage with a different performance mode (for example, a time-series database instead of key-value store) or isolation for security (for example, encryption of sensitive information, etc.)
No
Yes
Custom Business Logic
Perform arbitrary checks or actions when creating, reading, updating or deleting an object
Yes, using
Webhooks
.
Yes
Scale Subresource
Allows systems like HorizontalPodAutoscaler and PodDisruptionBudget interact with your new resource
Yes
Yes
Status Subresource
Allows fine-grained access control where user writes the spec section and the controller writes the status section. Allows increme

nting object Generation on custom resource data mutation (requires separate spec and status sections in the resource)
Yes
Yes
Other Subresources
Add operations other than CRUD, such as "logs" or "exec".
No
Yes
strategic-merge-patch
The new endpoints support PATCH with
Content-Type: application/strategic-merge-patch+json
. Useful for updating objects that may be modified both locally, and by the server. For more information, see
"Update API Objects in Place Using kubectl patch"
No
Yes
Protocol Buffers
The new resource supports clients that want to use Protocol Buffers
No
Yes
OpenAPI Schema
Is there an OpenAPI (swagger) schema for the types that can be dynamically fetched from the server? Is the user protected from misspelling field names by ensuring only allowed fields are set? Are types enforced (in other words, don't put an
int
in a
string
field?)
Yes, based on the
schema (GA in 1.16).
Yes
Common Features
When you create a custom resource, either via a CRD or an AA, you get many features for your API,
compared to implementing it outside the Kubernetes platform:
Feature
What it does
CRUD
The new endpoints support CRUD basic operations via HTTP and
kubectl
Watch
The new endpoints support Kubernetes Watch operations via HTTP
Discovery
Clients like
kubectl
and dashboard automatically offer list, display, and field edit operations on your resources
json-patch
merge-patch
HTTPS
The new endpoints uses HTTPS
Built-in Authentication
Access to the extension uses the core API server (aggregation layer) for authentication
Built-in Authorization
Access to the extension can reuse the authorization used by the core API server; for example, RBAC.
Block deletion of extension resources until external cleanup happens.
Admission Webhooks
Set default values and validate extension resources during any create/update/delete operation.
UI/CLI Display
Kubectl, dashboard can display extension resources.
Unset versus Empty
Clients can distinguish unset fields from zero-valued fields.
Client Libraries Generation
Kubernetes provides generic client libraries, as well as tools to generate type-specific client libraries.
Labels and annotations
Common metadata across objects that tools know how to edit for core and custom resources.
Preparing to install a custom resource
There are several points to be aware of before adding a custom resource to your cluster.
Third party code and new points of failure
While creating a CRD does not automatically add any new points of failure (for example, by causing
third party code to run on your API server), packages (for example, Charts) or other installation
bundles often include CRDs as well as a Deployment of third-party code that implements the
business logic for a new custom resource.
Installing an Aggregated API server always involves running a new Deployment.
Storage
Custom resources consume storage space in the same way that ConfigMaps do. Creating too many
custom resources may overload your API server's storage space.
Aggregated API servers may use the same storage as the main API server, in which case the same
warning applies.
Authentication, authorization, and auditing
CRDs always use the same authentication, authorization, and audit logging as the built-in
resources of your API server.
If you use RBAC for authorization, most RBAC roles will not grant access to the new resources
(except the cluster-admin role or any role created with wildcard rules). You'll need to explicitly
grant access to the new resources. CRDs and Aggregated APIs often come bundled with new role
definitions for the types they add.
Aggregated API servers may or may not use the same authentication, authorization, and auditing as
the primary API server.
Accessing a custom resource
can be used to access
custom resources. Not all

 client libraries support custom resources. The
Go
and
Python
client
libraries do.
When you add a custom resource, you can access it using:
kubectl
The Kubernetes dynamic client.
A REST client that you write.
A client generated using
Kubernetes client generation tools
(generating one is an advanced undertaking, but some projects may provide a client along with
the CRD or AA).
.
Extend the Kubernetes API with CustomResourceDefinition
.
13.2.2 - Kubernetes API Aggregation Layer
The aggregation layer allows Kubernetes to be extended with additional APIs, beyond what is
offered by the core Kubernetes APIs.
The additional APIs can either be ready-made solutions such as a
metrics server
, or APIs that you develop yourself.
The aggregation layer is different from
,
which are a way to make the
recognise new kinds of object.
Aggregation layer
The aggregation layer runs in-process with the kube-apiserver. Until an extension resource is
registered, the aggregation layer will do nothing. To register an API, you add an
object, which "claims" the URL path in the Kubernetes API. At that point, the aggregation layer
will proxy anything sent to that API path (e.g.
/apis/myextension.mycompany.io/v1/…
) to the
registered APIService.
The most common way to implement the APIService is to run an
extension API server
in Pod(s) that
run in your cluster. If you're using the extension API server to manage resources in your cluster,
the extension API server (also written as "extension-apiserver") is typically paired with one or
more
. The apiserver-builder
library provides a skeleton for both extension API servers and the associated controller(s).
Response latency
Extension API servers should have low latency networking to and from the kube-apiserver.
Discovery requests are required to round-trip from the kube-apiserver in five seconds or less.
If your extension API server cannot achieve that latency requirement, consider making changes that
let you meet it.
To get the aggregator working in your environment,
.
Then,
setup an extension api-server
Read about
in the API reference
Alternatively: learn how to
extend the Kubernetes API using Custom Resource Definitions
.
13.3 - Operator pattern
Operators are software extensions to Kubernetes that make use of
to manage applications and their components. Operators follow
Kubernetes principles, notably the
.
The
aims to capture the key aim of a human operator who
is managing a service or set of services. Human operators who look after
specific applications and services have deep knowledge of how the system
ought to behave, how to deploy it, and how to react if there are problems.
People who run workloads on Kubernetes often like to use automation to take
care of repeatable tasks. The operator pattern captures how you can write
code to automate a task beyond what Kubernetes itself provides.
Operators in Kubernetes
Kubernetes is designed for automation. Out of the box, you get lots of
built-in automation from the core of Kubernetes. You can use Kubernetes
to automate deploying and running workloads,
and
you can automate how
Kubernetes does that.
concept lets you extend the cluster's behaviour without modifying the code of Kubernetes
itself by linking
to
one or more custom resources. Operators are clients of the Kubernetes API that act as
controllers for a
.
An example operator
Some of the things that you can use an operator to automate include:
deploying an application on demand
taking and restoring backups of that application's state
handling upgrades of the application code alongside related changes such
as database schemas or extra configuration settings
publishing 

a Service to applications that don't support Kubernetes APIs to
discover them
simulating failure in all or part of your cluster to test its resilience
choosing a leader for a distributed application without an internal
member election process
What might an operator look like in more detail? Here's an example:
A custom resource named SampleDB, that you can configure into the cluster.
A Deployment that makes sure a Pod is running that contains the
controller part of the operator.
A container image of the operator code.
Controller code that queries the control plane to find out what SampleDB
resources are configured.
The core of the operator is code to tell the API server how to make
reality match the configured resources.
If you add a new SampleDB, the operator sets up PersistentVolumeClaims
to provide durable database storage, a StatefulSet to run SampleDB and
a Job to handle initial configuration.
If you delete it, the operator takes a snapshot, then makes sure that
the StatefulSet and Volumes are also removed.
The operator also manages regular database backups. For each SampleDB
resource, the operator determines when to create a Pod that can connect
to the database and take backups. These Pods would rely on a ConfigMap
and / or a Secret that has database connection details and credentials.
Because the operator aims to provide robust automation for the resource
it manages, there would be additional supporting code. For this example,
code checks to see if the database is running an old version and, if so,
creates Job objects that upgrade it for you.
Deploying operators
The most common way to deploy an operator is to add the
Custom Resource Definition and its associated Controller to your cluster.
The Controller will normally run outside of the
,
much as you would run any containerized application.
For example, you can run the controller in your cluster as a Deployment.
Using an operator
Once you have an operator deployed, you'd use it by adding, modifying or
deleting the kind of resource that the operator uses. Following the above
example, you would set up a Deployment for the operator itself, and then:
kubectl get SampleDB
# find configured databases
kubectl edit SampleDB/example-database
# manually change some settings
…and that's it! The operator will take care of applying the changes
as well as keeping the existing service in good shape.
Writing your own operator
If there isn't an operator in the ecosystem that implements the behavior you
want, you can code your own.
You also implement an operator (that is, a Controller) using any language / runtime
that can act as a
client for the Kubernetes API
.
Following are a few libraries and tools you can use to write your own cloud native
operator.
Note:
Charmed Operator Framework
Java Operator SDK
Kopf
(Kubernetes Operator Pythonic Framework)
kube-rs
(Rust)
kubebuilder
KubeOps
(.NET operator SDK)
KUDO
(Kubernetes Universal Declarative Operator)
Metacontroller
along with WebHooks that
you implement yourself
Operator Framework
shell-operator
Read the
CNCF
Operator White Paper
.
Find ready-made operators on
OperatorHub.io
to suit your use case
Publish
your operator for other people to use
Read
CoreOS' original article
that introduced the operator pattern (this is an archived version of the original article).
Read an
article
from Google Cloud about best practices for building operators
Home
Blog
Training
Partners
Community
CC BY 4.0

ObjectMeta | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Reference
name
(string)
(string)
namespace
(string)
labels
System
Must be empty before the object is deleted from the registry. Each entry is an identifier for the responsible component that will remove the entry from the list. If the deletionTimestamp of the object is non-nil, entries in this list can only be removed.

 Finalizers may be processed and removed in any order. Order is NOT enforced because it introduces significant risk of stuck finalizers. finalizers is a shared field, any actor with permission can reorder it. If the finalizer list is processed in order, then this can lead to a situation in which the component responsible for the first finalizer in the list is waiting for a signal (field value, external system, or other) produced by a component responsible for a finalizer later in the list, resulting in a deadlock. Without enforced ordering finalizers are free to order amongst themselves and are not vulnerable to ordering changes in the list.
(string)
(string)
', where
', where
', where
', where
(string)
(string)
(string)
(Time)
uid
List of objects depended by this object. If ALL objects in the list have been deleted, this object will be garbage collected. If this object is managed by a controller, then an entry in this list will po

int to this controller, with the controller field set to true. There cannot be more than one managing controller.
(boolean)
(boolean)
Read-only
(Time)
(int64)
(Time)
(int64)
(string)
An opaque 

value that represents the internal version of this object that can be used by clients to determine when objects have changed. May be used for optimistic concurrency, change detection, and the watch operation on a resource or set of resources. Clients must treat these values as opaque and passed unmodified back to the server. They may only be valid for a particular resource or set of resources.
selfLink
(string)
uid
(string)
Feedback
Yes
No
.
or
.
System
Read-only
Home
Blog
Training
Partners
Community
CC BY 4.0

Job with Pod-to-Pod Communication | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Run Jobs
to learn the IPs of the other Pods, but it's much simpler to rely on Kubernetes' built-in DNS resolution.
without
Job
.
minikube
.
Note:
Set up a
job-name
subdomain
:
Example
The Job is completed only after all pods successfully p

ing each other using hostnames.
Note:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
clusterIP
:
None
selector
:
job-name
:
---
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
3
:
3
:
Indexed
template
:
spec
:
subdomain
:
:
Never
:
-
name
:
image
:
command
:
- bash
- -c
- |
for i in 0 1 2
do
do
sleep 1
fi
done
done
using:
Note:
None
or
Default
.
here
.
Feedback
Yes
No
.
or
.
Last modified November 30, 2022 at 9:06 AM PST:
Fix format by removing erroneous ticks (446e5aed53)
Example
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Interactive Tutorial - Updating Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Ex

tending the Kubernetes API
Tasks

ing Applications
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Co

nfiguring Redis using a ConfigMap
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Adm

ission (v1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Expose Pod Information to Containers Through Files | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

nstall Tools
Debug Pods
D

ebug Init Containers
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Po

d Security Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
volume
,
A
.
minikube
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
zone
:
cluster
:
rack
:
rack-22
:
build
:
two
builder
:
john-doe
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
sleep 5;
done;
:
-
name
:
podinfo
mountPath
:
volumes
:
-
name
:
podinfo
:
items
:
-
path
:
"labels"
fieldRef
:
fieldPath
:
-
path
:
fieldRef
:
fieldPath
:
Volume,
.
Look at the
items
defines a
volume.
labels
.
.
Note:
kubect

l apply -f https://k8s.io/examples/pods/inject/dapi-volume.yaml
labels
file:

kubectl
exec
labels
file:
The output shows that all of the Pod's labels have been written
to the
labels
file:
cluster
=
rack
=
"rack-22"
zone
=
file:
labels
and
files
. In the
..data
is
labels
and

total 8
..data
rename(2)
.
Note:
subPath
/#
exit
container
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
sleep 5;
done;
resources
:
requests
:
memory
:
"32Mi"
cpu
:
"125m"
limits
:
memory
:
"64Mi"
cpu
:
"250m"
:
-
name
:
podinfo
mountPath
:
volumes
:
-
name
:
podinfo
:
items
:
-
path
:
:
:
resource
:
divisor
:
1m
-
path
:
:
:
resource
:
divisor
:
1m
-
path
:
:
:
resource
:
divisor
:
1Mi
-
path
:
:
:
resource
:
divisor
:
1Mi
volume
,
.
Look at the
items
,
1m
should be
cpu_limit
. The
divisor
1
. A divisor of 1 me

ans cores for
cpu
bytes for
memory
kubectl
exec
cpu_limit
file:
,
mem_limit
and
files.
Secrets
.
Read the
spec
that you
Check the
Volume
Check the
Check the
Check the
Feedback
Yes
No
.
or
.
Last modified August 10, 2022 at 10:43 AM PST:
Fix broken link -- downward-api (9b7af96880)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
ListMeta
Patch
Quantity
Status
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Delete a StatefulSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
This task shows you how to delete a
.
This task assumes you have an application running on your cluster represented by a StatefulSet.
Deleting a StatefulSet
You can delete a StatefulSet in the same way you delete other resources in Kubernetes: use the
command, and specify the StatefulSet either by file or by name.
kubectl delete -f &lt;file.yaml&gt;
kubectl delete statefulsets &lt;statefulset-name&gt;
You may need to delete the associated headless service separately after the StatefulSet itself is deleted.
kubectl delete service &lt;service-name&gt;
When deleting a StatefulSet through
kubectl
, the StatefulSet scales down to 0. All Pods that are part of this workload are also deleted. If you want to delete only the StatefulSet and not the Pods, use
.
kubectl delete -f &lt;file.yaml&gt; --cascade
=
orphan
By passing
to
, the Pods managed by the StatefulSet are left behind even after the StatefulSet object itself is deleted. If the pods have a label
, you can then delete them as follows:
kubectl delete pods -l app.kubernetes.io/name
=
MyApp
Deleting the Pods in a StatefulSet will not delete the associated volumes. This is to ensure that you have the chance to copy data off the volume before deleting it. Deleting the PVC after the pods have terminated might trigger deletion of the backing Persistent Volumes depending on the storage class and reclaim policy. You should never assume ability to access a volume after claim deletion.
Note:
Use caution when deleting a PVC, as it may lead to data loss.
Complete deletion of a StatefulSet
To delete everything in a StatefulSet, including the associated pods, you can run a series of commands similar to the following:
grace
=
$(
kubectl get pods &lt;stateful-set-pod&gt; --template
'{{.spec.terminationGracePeriodSeconds}}'
)
kubectl delete statefulset -l app.kubernetes.io/name
=
MyApp
sleep
$grace
kubectl delete pvc -l app.kubernetes.io/name
=
MyApp
In the example above, the Pods have the label
; substitute your own label as appropriate.
Force deletion of StatefulSet pods
If you find that some pods in your StatefulSet are stuck in the 'Terminating' or 'Unknown' states for an extended period of time, you may need to manually intervene to forcefully delete the pods from the apiserver. This is a potentially dangerous task. Refer to
for detai

ls.
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure a kubelet image credential provider | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

)
Tasks
.
ECR

:
kind
:
providers
:
-
name
:
ecr
#
# Globs can be used in the domain, but not in the port or the path. Globs are supported
#
#
# - gcr.io
:
-
-
-
-
-
:
"12h"
:
args
:
env
:
-
name
:
value
:
The
providers
name
.
The
field for each credential provider is used by the kubelet to determine whe

ther a plugin should be invoked
*.k8s.io
or
k8s.*.io
,
k8s.*
*.io
*.k8s.io
.
gcr.io
in the
.
Read the
.
Feedback
Yes
No
.
or
.
Last modified December 05, 2022 at 9:30 PM PST:
Updating the content and references in the docs for Credential Provider (#38139) (2df2cb5612)
Home
Blog
Training
Partners
Community
CC BY 4.0

ResourceClaimTemplate v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extending Kubern

etes
Tasks
Migrate from PodSecurityPolicy to the Built-I

n PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfile

, ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event 

Rate Limit Configuration (v1alpha1)
Reference
kind
metadata
(
)
spec
(
spec
(
metadata
(
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
resour

ceVersion
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string

(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The 

Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Kubelet Configuration (v1alpha1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Debug Pods
Get a She

ll to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restr

ict a Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryptio

n Configuration (v1)
Reference
Field
string
kind
string
providers
Field
name
string
[]string
*.k8s.io
or
k8s.*.io
k8s.*
.
*.io
*.k8s.io
.
123456789.dkr.ecr.us-east-1.amazonaws.co

m
gcr.io
string
args
[]string
env
Field
name
string
value
string
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Running in multiple zones | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Insta

lpha1)
This page describes running Kubernetes across multiple zones.
Kubernetes is designed so that a single Kubernetes cluster can run
across multiple failure zones, typically where these zones fit within
a logical grouping called a
region
. Major cloud providers define a region
as a set of failure zones (also called
availability zones
) that provide
a consistent set of features: within a region, each zone offers the same
APIs and services.
Typical cloud architectures aim to minimize the chance that a failure in
one zone also impairs services in another zone.
Control plane behavior
All
support running as a pool of interchangeable resources, replicated per
When you deploy a cluster control plane, place replicas of
control plane components across multiple failure zones. If availability is
an important concern, select at least three failure zones and replicate
each individual control plane component (API server, scheduler, etcd,
cluster controller manager) across at least three failure zones.
If you are running a cloud controller manager then you should
also replicate this across all the failure zones you selected.
Note:
Kubernetes does not provide cross-zone resilience for the API server
endpoints. You can use various techniques to improve availability for
the cluster API server, including DNS round-robin, SRV records, or
a third-party load balancing solution with health checking.
Node behavior
Kubernetes automatically spreads the Pods for
workload resources (such as
or
)
across different nodes in a cluster. This spreading helps
reduce the impact of failures.
When nodes start up, the kubelet on each node automatically adds
labels
to the Node object
that represents that specific kubelet in the Kubernetes API.
These labels can include
zone information
.
If your cluster spans multiple zones or regions, you can use node labels
to control how Pods are spread across your cluster among fault domains:
regions, zones, and even specific nodes.
These hints enable the
scheduler
to place
Pods for better expected availability, reducing the risk that a correlated
failure affects your whole workload.
For example, you can set a constraint to make sure that the
3 replicas of a StatefulSet are all running in different zones to each
other, whenever that is feasible. You can define this declaratively
without explicitly defining which availability

 zones are in use for
each workload.
Distributing nodes across zones
Kubernetes' core does not create nodes for you; you need to do that yourself,
or use a tool such as the
Cluster API
to
manage nodes on your behalf.
Using tools such as the Cluster API you can define sets of machines to run as
worker nodes for your cluster across multiple failure domains, and rules to
automatically heal the cluster in case of whole-zone service disruption.
Manual zone assignment for Pods
You can apply
node selector constraints
to Pods that you create, as well as to Pod templates in workload resources
such as Deployment, StatefulSet, or Job.
Storage access for zones
When persistent volumes are created, the
automatically adds zone labels to any PersistentVolumes that are linked to a specific
zone. The
scheduler
then ensures,
through its
NoVolumeZoneConflict
predicate, that pods which claim a given PersistentVolume
are only placed into the same zone as that volume.
for PersistentVolumeClaims that specifies the failure domains (zones) that the
storage in that class may use.
To learn about configuring a StorageClass that is aware of failure domains or zones,
see
Allowed topologies
.
By itself, Kubernetes does not include zone-aware networking. You can use a
to configure cluster networking, and that network solution might have zone-specific
elements. For example, if your cloud provider supports Services with
, the load balancer might only send traffic to Pods running in the
same zone as the load balancer element processing a given connection.
Check your cloud provider's documentation for details.
For custom or on-premises deployments, similar considerations apply.
Service
and
Ingress
behavior, including handling
of different failure zones, does vary depending on exactly how your cluster is set up.
Fault recovery
When you set up your cluster, you might also need to consider whether and how
your setup can restore service if all the failure zones in a region go
off-line at the same time. For example, do you rely on there being at least
one node able to run Pods in a zone?
Make sure that any cluster-critical repair work does not rely
on there being at least one healthy node in your cluster. For example: if all nodes
are unhealthy, you might need to run a repair Job with a special
toleration
so that the repair
can complete enough to bring at least one node into service.
Kubernetes doesn't come with an answer for this challenge; however, it's
something to consider.
To learn how the scheduler places Pods in a cluster, honoring the configured constraints,
visit
Scheduling and Eviction
.
Feedback
Yes
No
.
or
.
Last modified May 10, 2022 at 11:01 AM PST:
Move Pod Topology Spread Constraints into scheduling (6700656cb4)
Home
Blog
Training
Partners
Community
CC BY 4.0

Reconfigure a Node's Kubelet in a Live Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

a1)
Tasks
Caution:
The
kubelet
ConfigMap
each
Node
to use it.
.
kubeadm
, refer to
.
See the
.
Feedback
Yes
No
.
or
.
Last modified May 20, 2022 at 12:04 AM PST:
fix typo (45a7eb57f9)
Home
Blog
Training
Partners
Community
CC BY 4.0

{"version":"https://jsonfeed.org/version/1.1","title":"Auto-refreshing Official CVE Feed","home_page_url":"https://kubernetes.io","feed_url":"https://kubernetes.io/docs/reference/issues-security/official-cve-feed/index.json","description":"Auto-refreshing official CVE feed for Kubernetes repository","authors":[{"name":"Kubernetes Community","url":"https://www.kubernetes.dev"}],"items":[{"id":"CVE-2022-3294","url":"https://github.com/kubernetes/kubernetes/issues/113757","external_url":"https://www.cve.org/cverecord?id=CVE-2022-3294","summary":"Node address isn't always verified when proxying"},{"id":"CVE-2022-3162","url":"https://github.com/kubernetes/kubernetes/issues/113756","external_url":"https://www.cve.org/cverecord?id=CVE-2022-3162","summary":"Unauthorized read of Custom Resources"},{"id":"CVE-2022-3172","url":"https://github.com/kubernetes/kubernetes/issues/112513","external_url":"https://www.cve.org/cverecord?id=CVE-2022-3172","summary":"Aggregated API server can cause clients to be redirected (SSRF)"},{"id":"CVE-2021-25749","url":"https://github.com/kubernetes/kubernetes/issues/112192","external_url":"https://www.cve.org/cverecord?id=CVE-2021-25749","summary":"`runAsNonRoot` logic bypass for Windows containers"},{"id":"CVE-2021-25741","url":"https://github.com/kubernetes/kubernetes/issues/104980","external_url":"https://www.cve.org/cverecord?id=CVE-2021-25741","summary":"Symlink Exchange Can Allow Host Filesystem Access"},{"id":"CVE-2021-25737","url":"https://github.com/kubernetes/kubernetes/issues/102106","external_url":"https://www.cve.org/cverecord?id=CVE-2021-25737","summary":"Holes in EndpointSlice Validation Enable Host Network Hijack"},{"id":"CVE-2021-3121","url":"https://github.com/kubernetes/kubernetes/issues/101435","external_url":"https://www.cve.org/cverecord?id=CVE-2021-3121","summary":"Processes may panic upon receipt of malicious protobuf messages"},{"id":"CVE-2021-25735","url":"https://github.com/kubernetes/kubernetes/issues/100096","external_url":"https://www.cve.org/cverecord?id=CVE-2021-25735","summary":"Validating Admission Webhook does not observe some previous fields"},{"id":"CVE-2020-8554","url":"https://github.com/kubernetes/kubernetes/issues/97076","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8554","summary":"Man in the middle using LoadBalancer or ExternalIPs"},{"id":"CVE-2020-8566","url":"https://github.com/kubernetes/kubernetes/issues/95624","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8566","summary":"Ceph RBD adminSecrets exposed in logs when loglevel &gt;= 4"},{"id":"CVE-2020-8565","url":"https://github.com/kubernetes/kubernetes/issues/95623","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8565","summary":"Incomplete fix for CVE-2019-11250 allows for token leak in logs when logLevel &gt;= 9"},{"id":"CVE-2020-8564","url":"https://github.com/kubernetes/kubernetes/issues/95622","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8564","summary":"Docker config secrets leaked when file is malformed and log level &gt;= 4"},{"id":"CVE-2020-8563","url":"https://github.com/kubernetes/kubernetes/issues/95621","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8563","summary":"Secret leaks in kube-controller-manager when using vSphere provider"},{"id":"CVE-2020-8557","url":"https://github.com/kubernetes/kubernetes/issues/93032","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8557","summary":"Node disk DOS by writing to container /etc/hosts"},{"id":"CVE-2020-8559","url":"https://github.com/kubernetes/kubernetes/issues/92914","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8559","summary":"Privilege escalation from compromised node to cluster"},{"id":"CVE-2020-8558","url":"https://github.com/kubernetes/kubernetes/issues/92315","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8558","summary":"Node setting allows for neighboring hosts to bypass localhost boundary"},{"id":"CVE-2020-8555","url":"https://github.com/kubernetes/ku

bernetes/issues/91542","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8555","summary":"Half-Blind SSRF in kube-controller-manager"},{"id":"CVE-2020-10749","url":"https://github.com/kubernetes/kubernetes/issues/91507","external_url":"https://www.cve.org/cverecord?id=CVE-2020-10749","summary":"IPv4 only clusters susceptible to MitM attacks via IPv6 rogue router advertisements"},{"id":"CVE-2019-11254","url":"https://github.com/kubernetes/kubernetes/issues/89535","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11254","summary":"kube-apiserver Denial of Service vulnerability from malicious YAML payloads"},{"id":"CVE-2020-8552","url":"https://github.com/kubernetes/kubernetes/issues/89378","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8552","summary":"apiserver DoS (oom)"},{"id":"CVE-2020-8551","url":"https://github.com/kubernetes/kubernetes/issues/89377","external_url":"https://www.cve.org/cverecord?id=CVE-2020-8551","summary":"Kubelet DoS via API"},{"id":"CVE-2019-11251","url":"https://github.com/kubernetes/kubernetes/issues/87773","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11251","summary":"kubectl cp symlink vulnerability"},{"id":"CVE-2018-1002102","url":"https://github.com/kubernetes/kubernetes/issues/85867","external_url":"https://www.cve.org/cverecord?id=CVE-2018-1002102","summary":"Unvalidated redirect"},{"id":"CVE-2019-11255","url":"https://github.com/kubernetes/kubernetes/issues/85233","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11255","summary":"CSI volume snapshot, cloning and resizing features can result in unauthorized volume data access or mutation"},{"id":"CVE-2019-11253","url":"https://github.com/kubernetes/kubernetes/issues/83253","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11253","summary":"Kubernetes API Server JSON/YAML parsing vulnerable to resource exhaustion attack"},{"id":"CVE-2019-11250","url":"https://github.com/kubernetes/kubernetes/issues/81114","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11250","summary":"Bearer tokens are revealed in logs"},{"id":"CVE-2019-11248","url":"https://github.com/kubernetes/kubernetes/issues/81023","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11248","summary":"/debug/pprof exposed on kubelet's healthz port"},{"id":"CVE-2019-11249","url":"https://github.com/kubernetes/kubernetes/issues/80984","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11249","summary":"Incomplete fixes for CVE-2019-1002101 and CVE-2019-11246, kubectl cp potential directory traversal"},{"id":"CVE-2019-11247","url":"https://github.com/kubernetes/kubernetes/issues/80983","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11247","summary":"API server allows access to custom resources via wrong scope"},{"id":"CVE-2019-11245","url":"https://github.com/kubernetes/kubernetes/issues/78308","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11245","summary":"container uid changes to root after first restart or if image is already pulled to the node"},{"id":"CVE-2019-11243","url":"https://github.com/kubernetes/kubernetes/issues/76797","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11243","summary":"rest.AnonymousClientConfig() does not remove the serviceaccount credentials from config created by rest.InClusterConfig()"},{"id":"CVE-2019-11244","url":"https://github.com/kubernetes/kubernetes/issues/76676","external_url":"https://www.cve.org/cverecord?id=CVE-2019-11244","summary":"`kubectl:-http-cache=\u003cworld-accessible dir&gt;` creates world-writeable cached schema files"},{"id":"CVE-2019-1002100","url":"https://github.com/kubernetes/kubernetes/issues/74534","external_url":"https://www.cve.org/cverecord?id=CVE-2019-1002100","summary":"json-patch requests can exhaust apiserver resources"},{"id":"CVE-2018-1002105","url":"https://github.com/kubernetes/kubernetes/issues/71411","external_url":"https://www.cve.org/cverecord?id=CVE-2018-1002105","summary":"proxy request handling in kube-

apiserver can leave vulnerable TCP connections"},{"id":"CVE-2018-1002101","url":"https://github.com/kubernetes/kubernetes/issues/65750","external_url":"https://www.cve.org/cverecord?id=CVE-2018-1002101","summary":"smb mount security issue"},{"id":"CVE-2018-1002100","url":"https://github.com/kubernetes/kubernetes/issues/61297","external_url":"https://www.cve.org/cverecord?id=CVE-2018-1002100","summary":"Kubectl copy doesn't check for paths outside of it's destination directory."},{"id":"CVE-2017-1002102","url":"https://github.com/kubernetes/kubernetes/issues/60814","external_url":"https://www.cve.org/cverecord?id=CVE-2017-1002102","summary":"atomic writer volume handling allows arbitrary file deletion in host filesystem"},{"id":"CVE-2017-1002101","url":"https://github.com/kubernetes/kubernetes/issues/60813","external_url":"https://www.cve.org/cverecord?id=CVE-2017-1002101","summary":"subpath volume mount handling allows arbitrary file access in host filesystem"},{"id":"CVE-2017-1002100","url":"https://github.com/kubernetes/kubernetes/issues/47611","external_url":"https://www.cve.org/cverecord?id=CVE-2017-1002100","summary":"Azure PV should be Private scope not Container scope"},{"id":"CVE-2017-1000056","url":"https://github.com/kubernetes/kubernetes/issues/43459","external_url":"https://www.cve.org/cverecord?id=CVE-2017-1000056","summary":"PodSecurityPolicy admission plugin authorizes incorrectly"}]}

Client Authentication (v1beta1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
Field
string
kind
string
spec
status
Cluster
Field
server
string
string
bool
[]byte
proxy-url
string
cluster.
bool
config
clusters:
cluster:
...
audience: 06e3fbd18de8 # arbitr

ary config
Field
cluster
Cluster
bool
Field
token
string
string
string
Feedback
Yes
No
.
or
.
Cluster
Home
Blog
Training
Partners
Community
CC BY 4.0

Virtual IPs and Service Proxies | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
Every
node
).
The
virtual IP
Services
of
type
other than
.
cleanup
For exam

ple, if your operating system doesn't allow you to run iptables commands,
iptables
proxy mode
clusterIP
and
port
,
userspace
Example
earlier
Service.
iptables
section
of the
):
...
iptables
:
:
1s
:
30s
...
The
0s
, multiple

1s
may
The
to avoid
1h
enable the
feature
gate
for
.
1s
or
metrics
In
ipvs
calls
netlink
state.
these are:
rr
lc
dh
sh
sed
nq
Note:

to
ClientIP
None
).
Note:
virtual
IP
etcd
ClusterIP
.
lower
Services.
and
fields
Cluster
and
Local
Cluster
Local
Local
Cluster
and
Local
to
Cluster
Local
t

o only
Local
If the
Local
all
NodePort
and
.
read
.
Read about
Services
Read about
Ingresses
Read the
Feedback
Yes
No
.
or
.
iptables
proxy mode
Home
Blog
Training
Partners
Community
CC BY 4.0

Create a Windows HostProcess Pod | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Consolidation of administrative tasks and security policies. This reduces the degree of
How t

o install containerd.
and
See
.
not
Control
Policy
true
true
false
spec
:
:
:
:
true
:
:
true
:
-
name
:
test
image
:
command
:
- ping
- -t
-
127.0.0.1
:
windows
upon the
Applications running inside the container can access volume mounts directly via the volumeMount's
specified
mountPath
$CONTAINER_SANDBOX_MOUNT

_POINT\var\run\secrets\kubernetes.io\serviceaccount\
ephemeral
Note:
Example:
icacls
.
Set
:
:
:
true
:
.
.
or
.
Feedback
Yes
No
.
or
.
Last modified December 22, 2022 at 11:01 PM PST:
Update create-hostprocess-pod.md (b3bfa1eab0)

Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm join | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Reference
Kubeadm
Synopsis
--token flag can be used instead of specifying

 each token individually.
Options
--dry-run
-h, --help
authentic

ity of that data. The root CA can also be discovered directly via a
process.
.
--help
command,
flag.
=
=
field in
.
RFC7469
's/^.* //'
Example
commands:
join
if the
.
command.
the control-plane

. To use the mode the joining nodes must skip the hash validation of the
Example
command:
file.
Example
commands:
Using
ConfigMap
ConfigMap with some data needed for validation of the control-plane node'

s identity is exposed publicly by
flag of the
Fetch the
jsonpath
=
:
v1
kind
:
Config
clusters
:
-
cluster
:
:
server
:
name
:
""
contexts
:
[]
:
""
:
{}
users
:
[]
Use the
.
but before
.
Caution:
--config
contain a
--config
command.
the
command.
.
.
or
.
Feedback
Yes
No
.
or
.
Last modified December 11, 2022 at 12:01 PM PST:
[en] Replace grep with kubectl get -o jsonpath. (ac35be2122)
Home
Blog
Training
Partners
Community
CC BY 4.0

Extend the Kubernetes API with CustomResourceDefinitions | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Tasks
.
minikube
.
CustomResourceDefinitions themselves are non-namespaced and are available to
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
# kind is normally th

e CamelCased singular type. Your resource manifests use this.
kind
:
CronTab
:
- ct
The
kind
CronTab
cronSpec
and
image
CronTab
. The kind
CronTab
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
NAME                 AGE
cronSpec
and
image
fields
:
v1
items
:
-
:
kind
:
CronTab
metadata
:
:
:
|
{
:
,
"kind"
:
"CronTab"
,
:{
:{},
"name"
:
,
:
"default"
},
"spec"
:{
:
"* * * * */5"
,
"image"
:
}}
:
:
1
name
:
namespace
:
default
:
"1326"
uid
:
spec
:
cronSpec
:
'* * * * */5'
image
:
kind
:
List
metadata
:
:
""
selfLink
:
""
find the requested resource (get crontabs.stable.example.com)
fields
,
kind
and
metadata
With
which:
type
(via
or
(via
items
a node with
a node with
x-kubernetes-preserve-unknown-fields: t

rue
allOf
,
anyOf
,
oneOf
or
not
,
type
,
default
,
,
nullable
within an
allOf
,
anyOf
,
oneOf
or
not
if
metadata
and
allOf:
    foo:
      ...
  foo:
    ...
allOf:
    foo:
      ...
allOf:
- items:
      foo:
        ...
items:
    foo:
      ...
allOf:
- items:
      foo:
        ...
  foo:
  metadata:
      name:
        items:
anyOf:
    bar:
the type of
foo
bar
inside of
anyOf
bar
's
type
is within
anyOf
(rule 3).
anyOf
(rule 3).
type
:
object
:
:
foo
:
type
:
string
pattern
:
"abc"
bar
:
type
:
integer
metadata
:
type
:
object
:
name
:
type
:
string
pattern
:
"^a"
anyOf
:
-
:
bar
:
minimum
:
42
required
:
[
"bar"
]
etcd
.
ConfigMap
,
pruned
to
might lack
might be
true
.
with
set to
true
Set
to
false
.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
:
42
=
false
:
kind
:
CronTab
metadata
:
:
:
1
name
:
namespace
:
default
:
"285"
uid
:
spec
:
cronSpec
:
'* * * * */5'
image
:
the
kubectl
By default, all unspecified fields for 

a custom resource, across all versions, are pruned. It is possible though to
in the
.
type
:
object
:
json
:
:
true
The field
json
type
:
object
:
json
:
:
true
type
:
object
:
object
):
type
:
object
:
json
:
:
true
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
bar
:
type
:
string
json
:
spec
:
foo
:
abc
bar
:
def
something
:
x
status
:
something
:
x
json
:
spec
:
foo
:
abc
bar
:
def
status
:
something
:
x
something
spec
type
:
object
:
foo
:
:
true
anyOf:
...
and
allOf:
  - anyOf:
...
In
,
)
and
kind
fields.
by setting
type
:
object
:
foo
:
:
true
:
true
foo
foo:
  kind: Pod
  spec:
    ...
Because
The use of
With
, the
,
kind
and
metadata
See
CustomResourceDefinition and migrating your objects from one version to another.
:
kind
:
CronTab
metadata
:
:
When the
responsibility of each controller to remove its finalizer fr

om the list.
,
.
,
,
,
,
id
,
,
readOnly
,
writeOnly
,
xml
,
$ref
.
The field
true
.
The field
false
.
The field
.
The
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * *"
image
:
replicas
:
15
The CronTab "my-new-cron-object" is invalid: []: Invalid value: map[string]interface {}{"apiVersion":"stable.example.com/v1", "kind":"CronTab", "metadata":map[string]interface {}{"name":"my-new-cron-object", "namespace":"default", "deletionTimestamp":interface {}(nil), "deletionGracePeriodSeconds":(*int64)(nil), "creationTimestamp":"2017-09-05T05:20:07Z", "uid":"e14d79e7-91f9-11e7-a598-f0761cb232d1", "clusterName":""}, "spec":map[string]interface {}{"cronSpec":"* * * *", "image":"my-awesome-cron-image", "replicas":15}}:
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
5
crontab
created
CustomResourceValidationExpress

ions
false
, for the
.
And
self
...
      spec:
          ...
          replicas:
        required:
          - replicas
:
kind
:
CronTab
metadata
:
name
:
spec
:
:
0
replicas
:
20
:
10
The
rule
under
The
message
Invalid value: apiextensions.ValidationRule{Rule:"self == true", Message:""}: compilation failed: ERROR: \&lt;input&gt;:1:6: found no matching overload for '_==_' applied to '(int, bool)'
has(self)
Rule
Purpose

Xref:
,
kind
,
and
spec
and
status
in the
...
        spec:
            ...
        status:
via
...
      spec:
          ...
          foo:
and all
.
...
      spec:
          ...
            foo:
self[i]
and
...
      ...
      foo:
        items:
self
...
      spec:
          ...
          foo:
Examples:
string
The
,
kind
,
and

is not
.
__
__dot__
.
__dash__
-
__slash__
/
namespace
x-prop
redact__d
string
of
set
or
map
i.e.,
set
:
X + Y
X
Y
map
:
X + Y
X
Y
X
and
Y
in
Y
CEL type
map
'array
list
'boolean'
boolean
double
int (64)
'null'
null_type
'string'
string
'string' with format=byte (base64 encoded)
bytes
xref:
CEL types
,
,
.
macros
CEL
oldSelf
type
:
string
enum
:
[
"low"
,
"medium"
,
"high"
]
:
-
rule
:
message
:
Unlike other rules, transition rules apply only to operations mee

ting the following criteria:
path
rule
array
;
any
set
or
atomic
self
with
oldSelf
.
Use Case
Rule
or
But if
foo
self.foo.contains("someString")
foo
is.
foo
.
foo
is not
maxItems
,
, and
maxLength
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
string
:
-
rule
:
CEL rule exceeded budget by more than 100x (try simplifying the rule, or adding maxI

tems, maxProperties, and 
self.all
foo
,
'a string'
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
maxLength
:
10
:
-
rule
:
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
:
-
rule
:
maxLength
:
10
self.all
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
integer
:
-
rule
:
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
array
items
:
type
:
integer
:
-
rule
:
foo
self.all
.
Note:
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
default
:
"5 0 * * *"
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
default
:
1
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
cronSpec
and
replicas
:
kind
:
CronTab
metadata
:
name
:
spec
:
image
:
leads to
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"5 0 * * *"
image
:
replicas
:
1
metadata
metadata
fields of
metadata
) are not pruned during CustomResourc

eDefinition creation, but
false
true
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
nullable
:
false
default
:
"default"
bar
:
type
:
string
nullable
:
true
baz
:
type
:
string
foo
and
bar
and
baz
spec
:
foo
:
null
bar
:
null
baz
:
null
leads to
spec
:
foo
:
"default"
bar
:
null
with
foo
bar
, and
baz
default.
which are
and
The
kubectl
and
)
show up in
and
paths
.
The fields
allOf
,
anyOf
,
oneOf
and
not
If
type
,
nullable
,
items
and
Spec
,
Replicas
, and
Age
columns.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
:
-
name
:
Spec
type
:
string
:
jsonPath
:
-
name
:
Replicas
type
:
integer
:
jsonPath
:
-
name
:
Age
type
:
date
jsonPath
:
Notice the
NAME
,
SPEC
,
REPLICAS
, and
AGE
Note:
The
NAME
Priority
priority
field. 

Currently, the priority
-o wide
flag).
0
0
Type
A column's
type
):
integer
number
string
– strings
boolean
–
true
or
false
date
Format
A column's
format
int32
int64
float
double
byte
date
date-time
password
format
kubectl
/status
and
/scale
/status
.status
and
.spec
JSONPaths
PUT
/status
PUT
/status
resource.
PUT
/
POST
/
PATCH
The
.metadata
or
.status
.
example
format
items
maximum
maxItems
maxLength
minimum
minItems
minLength
pattern
required
title
type
/scale
The
/scale
.
.
.spec
the
/scale
.
.status
/scale
.
.status
or
.spec
/scale
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
in

teger
status
:
type
:
object
:
replicas
:
type
:
integer
:
type
:
string
:
status
:
{}
scale
:
:
:
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
3
and
command.
of the
=
5
crontabs
scaled
jsonpath
=
5
all
).
You can use
all
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
- all
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
:
CronTab
:
NAME                          AGE
Read about
.
See
.
Serve
of a
Feedback
Yes
No
.
or
.
Last modified August 25, 2022 at 3:58 PM PST:
Style tweak the crd task file (ef511e7e65)
Create a CustomResourceDefin

ition
Home
Blog
Training
Partners
Community
CC BY 4.0

Overview
API Groups
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Scale
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Pod v1 core
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Log
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Scale
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Scale
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read Scale
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ClusterCIDR v1alpha1 networking.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Write Oper

ations
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
CSIStorageCapacity v1 storage.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
HorizontalPodAutoscaler v2 autoscaling
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PodScheduling v1alpha1 resource.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ResourceClaimTemplate v1alpha1 resource.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ResourceClass v1alpha1 resource.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ValidatingAdmissionPolicy v1alpha1 admissionregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Wat

ch List
ValidatingAdmissionPolicyBinding v1alpha1 admissionregistration.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Read
List
FlowSchema v1beta3 flowcontrol.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PriorityLevelConfiguration v1beta3 flowcontrol.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
SelfSubjectReview v1alpha1 authentication.k8s.io
Create
Create
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Create
NetworkPolic

y v1 networking.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
AllocationResult v1alpha1 resource.k8s.io
ClaimSource v1 core
ContainerResourceMetricSource v2 autoscaling
ContainerResourceMetricStatus v2 autoscaling
CrossVersionObjectReference v2 autoscaling
Eviction v1 policy
ExternalMetricSource v2 autoscaling
ExternalMetricStatus v2 autoscaling
FlowDistinguisherMethod v1beta3 flowcontrol.apiserver.k8s.io
FlowSchemaCondition v1beta3 flowcontrol.apiserver.k8s.io
GRPCAction v1 core
GroupSubject v1beta3 flowcontrol.apiserver.k8s.io
HPAScalingPolicy v2 autoscaling
HPAScalingRules v2 autoscaling
HorizontalPodAutoscalerBehavior v2 autoscaling
HorizontalPodAutoscalerCondition v2 autoscaling
IngressLoadBalancerIngress v1 networking.k8s.io
IngressLoadBalancerStatus v1 networking.k8s.io
IngressPortStatus v1 networking.k8s.io
JSON v1 apiextensions.k8s.

io
LifecycleHandler v1 core
LimitResponse v1beta3 flowcontrol.apiserver.k8s.io
LimitedPriorityLevelConfiguration v1beta3 flowcontrol.apiserver.k8s.io
MatchResources v1alpha1 admissionregistration.k8s.io
MetricIdentifier v2 autoscaling
MetricSpec v2 autoscaling
MetricStatus v2 autoscaling
MetricTarget v2 autoscaling
MetricValueStatus v2 autoscaling
NamedRuleWithOperations v1alpha1 admissionregistration.k8s.io
NonResourcePolicyRule v1beta3 flowcontrol.apiserver.k8s.io
ObjectMetricSource v2 autoscaling
ObjectMetricStatus v2 autoscaling
ParamKind v1alpha1 admissionregistration.k8s.io
ParamRef v1alpha1 admissionregistration.k8s.io
PodFailurePolicy v1 batch
PodFailurePolicyOnExitCodesRequirement v1 batch
PodFailurePolicyOnPodConditionsPattern v1 batch
PodFailurePolicyRule v1 batch
PodOS v1 core
PodResourceClaim v1 core
PodSchedulingGate v1 core
PodsMetricSource v2 autoscaling
PodsMetricStatus v2 autoscaling
PolicyRulesWithSubjects v1beta3 flowcontrol.apiserver.k8s.io
PriorityLevelConfigurationCondition v1beta3 flowcontrol.apiserver.k8s.io
PriorityLevelConfigurationReference v1beta3 flowcontrol.apiserver.k8s.io
QueuingConfiguration v1beta3 flowcontrol.apiserver.k8s.io
ResourceClaim v1 core
ResourceClaimConsumerReference v1alpha1 resource.k8s.io
ResourceClaimParametersReference v1alpha1 resource.k8s.io
ResourceClaimSchedulingStatus v1alpha1 resource.k8s.io
ResourceClassParametersReference v1alpha1 resource.k8s.io
ResourceMetricSource v2 autoscaling
ResourceMetricStatus v2 autoscaling
ResourcePolicyRule v1beta3 flowcontrol.apiserver.k8s.io
SecretKey

Selector v1 core
ServiceAccountSubject v1beta3 flowcontrol.apiserver.k8s.io
StatefulSetOrdinals v1 apps
StatefulSetPersistentVolumeClaimRetentionPolicy v1 apps
Subject v1beta3 flowcontrol.apiserver.k8s.io
TypedObjectReference v1 core
UncountedTerminatedPods v1 batch
UserSubject v1beta3 flowcontrol.apiserver.k8s.io
Validation v1alpha1 admissionregistration.k8s.io
ValidationRule v1 apiextensions.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
FlowDistinguisherMethod v1beta2 flowcontrol.apiserver.k8s.io
FlowSchema v1beta2 flowcontrol.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
FlowSchemaCondition v1beta2 flowcontrol.apiserver.k8s.io
GroupSubject v1beta2 flowcontrol.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
LimitResponse v1beta2 flowcontrol.apiserver.k8s.io
LimitedPriorityLevelConfiguration v1beta2 flowcontrol.apiserver.k8s.io
NonResourcePolicyRule v1beta2 flowcontrol.apiserver.k8s.io
PolicyRulesWithSubjects v1beta2 flowcontrol.apiserver.k8s.io
PriorityLevelConfiguration v1beta2 flowcontrol.apiserver.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
PriorityLevelConfigurationCondition v1beta2 flowcontrol.apiserver.k8s.io
PriorityLevelConfigurationReference v1beta2 flowcontrol.apiserver.k8s.io
QueuingConfiguration v1beta2 flowcontrol.apiserver.k8s.io
ResourceClaim v1alpha1 resource.k8s.io
Create
Patch
Replace
Delete
Read
List
Watch
Watch List
ResourcePolicyRule v1beta2 flowcontrol.apiserver.k8s.io
ServiceAccountSubject v1beta2 flowcontrol.apiserver.k8s.io
ServiceReference v1 

Subject v1beta2 flowcontrol.apiserver.k8s.io
UserSubject v1beta2 flowcontrol.apiserver.k8s.io
Copyright 2016-2022 The Kubernetes Authors.
Generated at: 2022-12-09 18:19:41 (-03)
v1.26.0
Workloads
Cluster
Metadata
: This is defined by the user and describes the desired state of system.  Fill this in when creating or updating an object.
Create
Update
Replace
and
Patch
:
Replace
:
Note
: The
ConfigMap
or
Secret
Pod
Pod
s are
Patch
:
Read
Get
,
List
and
Watch
:
Get
List
: Like
List
Watch
Delete
Delete will delete a resource.  Depending on the specific resource, child objects may or may not be garbage col

lected by the server.  See
Rollback
API Groups
Group
Version
v1, v1alpha1
v1
v1
apps
v1
v1
v2, v1
batch
v1
v1
v1
core
v1
v1
v1
v1beta3, v1beta2
v1alpha1
v1
policy
v1
v1
resource.k8s.io
v1alpha1
v1
WORKLOADS
Pods
Volumes
and
or
Secret
The most common Controllers are:
Jobs
example
Group
Version
Kind
core
v1
Container
Warning:
Pod
,
Job
, or
Field
args
Arguments to the entrypoint. The container image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell
command
Entrypoint array. Not executed within a shell. The container image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell
env
EnvVar
array
:
merge
:
name
envFrom
array
List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot be update

d.
image
string
Container image name. More info: https://kubernetes.io/docs/concepts/containers/images This field is optional to allow higher level config management to default or override container images in workload controllers like Deployments and StatefulSets.
string
lifecycle
Lifecycle
Probe
name
string
ports
array
:
merge
:
List of ports to expose from the container. Not specifying a port here DOES NOT prevent that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Modifying this array with strategic merge patch may corrupt the data. For more information See https://github.com/kubernetes/kubernetes/issues/108255. Cannot be updated.
Probe
Periodic probe of container service readiness. Container will be removed from service endpoints if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
resources
SecurityContext defines the security options the container should be run with. If set, the fields of SecurityContext override the equivalent fields of PodSecurityContext. More info: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
Probe
stdin
boolean
stdinOnce
boolean
string

string
tty
boolean
array
:
merge
:
array
:
merge
:
mountPath
string
Field
string
image
string
The image the container is running. More info: https://kubernetes.io/docs/concepts/containers/images.
imageID
string
lastState
name
string
ready
boolean
integer
started
boolean
state
Group
Version
Kind
batch
v1
CronJob
Field
string
kind
string
metadata
spec
status
Field
string
integer
schedule
string
The schedule 

in Cron format, see https://en.wikipedia.org/wiki/Cron.
integer
integer
suspend
boolean
timeZone
string
The time zone name for the given schedule, see https://en.wikipedia.org/wiki/List_of_tz_database_time_zones. If not specified, this will default to the time zone of the kube-controller-manager process. The set of valid time zone names and the time zone offset is loaded from the system-wide time zone database by the API server during CronJob validation and the controller manager during execution. If no system-wide time zone database can be found a bundled version of the database is used instead. If the time zone name becomes invalid during the lifetime of a CronJob or due to a change in host configuration, the controller will stop creating new new Jobs and will create a system event with the reason UnknownTimeZone. More information can be found in https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/#time-zones This is beta field and must be enabled via the `CronJobTimeZone` feature gate.
Field
active
array
Time
Time
Field
string
items
CronJob
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: Thi

s will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
202
CronJob
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
CronJob
OK
201
CronJob
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as def

ined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue

dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recomm

ended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
CronJob
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometime

s referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be ident

ical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer tha

n the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects

 that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields,

 provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
CronJob
OK
201
CronJob
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
CronJob
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
CronJob
Response
Code
200
CronJob
OK
201
CronJob
Created
example
kind: DaemonSe

t
metadata:
spec:
  selector:
  template:
    metadata:
      labels:
    spec:
        command:
        - /bin/sh
        args:
        - -c
        - &gt;-
          sleep 10 ;
          done
Group
Version
Kind
apps
v1
DaemonSet
Field
string
kind
string
metadata
spec
status
Field
integer
integer
selector
template
Field
integer
array
:
merge
:
type
integer
integer
The total number of nodes that should be running the daemon pod (including nodes correctly running the daemon pod). More info: https://kubernetes.io/docs/concepts/workloads/controllers/

integer
integer
integer
integer
integer
integer
Field
string
items
DaemonSet
array
kind
string
metadata
ListMeta
Field
maxSurge
Create
curl 

request example
kubectl
command
metadata:
spec:
  selector:
  template:
    metadata:
      labels:
    spec:
        command:
        - /bin/sh
        args:
        - -c
        - &gt;-
          sleep 10 ;
          done
curl
command (
requires
)
metadata:
spec:
  selector:
  template:
    metadata:
      labels:
    spec:
        command:
        - /bin/sh
        args:
        - -c
        - &gt;-
          sleep 10 ;
          done
Output
{
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
              "/bin/sh"
            ],
            "args": [
              "-c",
            ],
          }
        ],
      }
    }
  },
  }
}
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder enc

ounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
DaemonSet
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
202
DaemonSet
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldMana

ger is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
DaemonSet
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
Delete
kubectl
command
curl
command (
requires
)
Output
{
}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/apps/v

1/namespaces/{namespace}/daemonsets
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Whether and how garbage collection will be

 performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
kubectl
command
curl
command (
requires
)
Parameter
name
namespace
Parameter
pretty
Response
Code
200
DaemonSet
OK
List
Parameter
namespace
Parameter
pretty
continue

limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any mod

ifications.
limit
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field

. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result 

with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are r

eturned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
PATCH /apis/apps/v1/

namespaces/{namespace}/daemonsets/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
DaemonSet
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and wi

ll ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
DaemonSet
Response
Code
200
DaemonSet
OK
201
DaemonSet
Created
example
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
Group
Version
Kind
apps
v1
Field
string
kind
string
metadata
spec
status
Field
integer
paused
boolean
integer
replicas
integer
integer
selector
strategy
:
template
Template describes the pods that wi

ll be created.
Field
integer
integer
array
:
merge
:
type
integer
integer
replicas
integer
integer
integer
Field
string
items
array
kind
string
metadata
ListMeta
Field
type
string
Field
maxSurge
Create
kubectl
command
metadata:
spec:
  sele

ctor:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        ports:
curl
command (
requires
)
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        ports:
Output
{
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    },
  },
}
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFi

eldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
kubectl
command
curl
command (
requires
)
Output
{
    },
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    },
  },
  }
}
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropp

ed from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
kubectl
command
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        ports:
curl
command (
requires
)
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
        ports:
Output
{
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    },
  },
  }
}
Parameter
name
namespace
object name

 and auth scope, such as for teams and projects
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
kubectl
command
curl
command (
requires
)
Output
{
}
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default p

olicy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was pres

ent at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
kubectl
command
curl
command (
requires
)
Output
{
    },
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    }
  },
  }
}
{
    },
    "anno

tations": {
    }
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
            "ports": [
              {
              }
            ],
          }
        ],
      }
    },
      }
    }
  },
  }
}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
kubectl
command
curl
command (
requires
)
Output
{
  "items": [
    {
        },
        }
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
              {
              }
            ],
          }
        },
          }
        }
      },
      }
    },
    {
        "uid": "1b33145a-9c63-11

e6-9c54-42010a800148",
        },
        }
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
              {
                "ports": [
                  {
                  }
                ],
              }
            ],
          }
        },
          }
        }
      },
      }
    }
  ]
}
{
  "items": [
    {
        },
        }
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
              {
              }
            ],
          }
        },
          }
        }
      },
      }
    },
    {
        },
          "depl

oyment.kubernetes.io/revision": "1"
        }
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
              {
                "ports": [
                  {
                  }
                ],
              }
            ],
          }
        },
          }
        }
      },
      }
    }
  ]
}
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are availab

le. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewe

r than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
kubectl
command
curl
command (
requires
)
Output
{
			},
			}
		},
		"spec": {
				}
			},
					}
				},
				"spec": {
						{
							"ports": [
								{
								}
							],
							},
						}
					],
					}
				}
			},
				}
			}
		},
		}
	}
}
{
			"namespace

": "default",
			},
			}
		},
		"spec": {
				}
			},
					}
				},
				"spec": {
						{
							"ports": [
								{
								}
							],
							},
						}
					],
					}
				}
			},
				}
			}
		},
		}
	}
}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and cli

ents should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
A selector to res

trict the list of returned objects by their labels. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the respons

e, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values ar

e: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Read Scale
Parameter
name

namespace
Parameter
pretty
Response
Code
200
Scale
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unkn

own field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
Scale
OK
201
Scale
Created
example
kind: Job
metadata:
spec:
  template:
    metadata:
    spec:
      - name: pi
Group
Version
Kind
batch
v1
Job
Field
string
kind
string
metadata
spec
JobSpec
status
JobStatus
Field
integer
integer
string
CompletionMode specifies how Pod completions are tracked. It can be `NonIndexed` (default) or `Indexed`.  `NonIndexed` means that the Job is considered complete when there have been .spec.completions successfully completed Pods. Each Pod completion is homologous to each other.  `Indexed` means that the Pods of a Job get an associated completion index from 0 to (.spec.completions - 1), available in the annotation batch.kubernetes.io/job-completion-index. The Job is considered complete when there is one successfully completed Pod for each index. When value is `Indexed`, .spec.completions must be specified and `.spec.parallelism` must be less than or equal to 10^5. In addition, The Pod name takes the form `$(job-name)-$(index)-$(random-string)`, the Pod hostname takes the form `$(job-name)-$(index)`.  More completion modes can be added in the future. If the Job controller observes a mode that it doesn't recognize, which is possible during upgrades due to version skew, the controller skips up

dates for the Job.
integer
boolean
integer
Specifies the maximum desired number of pods the job should run at any given time. The actual number of pods running in steady state will be less than this number when ((.spec.completions - .status.successful) &lt; .spec.parallelism), i.e. when the work left to do is less than max parallelism. More info: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/
Specifies the policy of handling failed pods. In particular, it allows to specify the set of actions and conditions which need to be satisfied to take the associated action. If empty, the default behaviour applies - the counter of failed pods, represented by the jobs's .status.failed field, is incremented and it is checked against the backoffLimit. This field cannot be used in combination with restartPolicy=OnFailure.  This field is alpha-level. To use this field, you must enable the `JobPodFailurePolicy` feature gate (disabled by default).
selector
suspend
boolean
template
integer
Field
active
integer
string
CompletedIndexes holds the completed indexes when .spec.completionMode = "Indexed" in a text format. The indexes are represented as decimal integers separated by commas. The numbers are listed in increasing order. Three or more consecutive numbers are compressed and repre

sented by the first and last element of the series, separated by a hyphen. For example, if the completed indexes are 1, 3, 4, 5 and 7, they are represented as "1,3-5,7".
Time
array
:
merge
:
type
failed
integer
ready
integer
The number of pods which have a Ready condition.  This field is beta-level. The job controller populates the field when the feature gate JobReadyPods is enabled (enabled by default).
startTime
Time
succeeded
integer
UncountedTerminatedPods
UncountedTerminatedPods holds the UIDs of Pods that have terminated but the job controller hasn't yet accounted for in the status counters.  The job controller creates pods with a finalizer. When a pod terminates (succeeded or failed), the controller does three steps to account for it in the job status: (1) Add the pod UID to the arrays in this field. (2) Remove the pod finalizer. (3) Remove the pod UID from the arrays while increasing the corresponding     counter.  Old jobs might not be tracked using this field, in which case the field remains null.
Field
string
items
Job
array
kind
string
metadata
ListMeta
Create
kubectl
command
kind: Job
metadata:
spec:
  template:
    metadata:
    spec:
      - name: pi
curl
command (
requires
)
kind: Job
metadata:
spec:
  template:
    metadata:
    spec:
      - name: pi
Output
job "examp

le-job" created
{
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
              "perl",
              "-wle",
            ],
          }
        ],
      }
    }
  },
}
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Job
Response
Code
200
Job
OK
201
Job
Created
202
Job
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run sta

ges will be processed
force
Parameter
body
Patch
Response
Code
200
Job
OK
201
Job
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Job
Response
Code
200
Job
OK
201
Job
Created
Delete
kubectl request 

example
kubectl
command
curl
command (
requires
)
Output
{
}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an e

rror response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
limit
Parameter
body
Response
Code
200
Status
OK
Read
kubectl
command
curl
command (
requires
)
Output
{
    "name": "example-

job",
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
              "perl",
              "-wle",
            ],
          }
        ],
      }
    }
  },
  }
}
{
  },
  "spec": {
      }
    },
        }
      },
      "spec": {
          {
              "perl",
              "-wle",
            ],
          }
        ],
      }
    }
  },
  }
}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Job
OK
List
kubectl
command
curl
command (
requires
)
Output
{
  },
  "items": [
    {
      },
      "spec": {
            "controller-uid": "d93a3569-a2be-11e6-a008-fa

043d458cc7"
          }
        },
            }
          },
          "spec": {
              {
                  "perl",
                  "-wle",
                ],
              }
            ],
          }
        }
      },
      }
    }
  ]
}
{
  },
  "items": [
    {
      },
      "spec": {
          }
        },
            }
          },
          "spec": {
              {
                  "perl",
                  "-wle",
                ],
              }
            ],
          }
        }
      },
      }
    }
  ]
}
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired 

error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
JobList
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the sp

ecified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
JobList
OK
Watch
kubectl
command
curl
command (
requires
)
Output
{
			"resourceVers

ion": "7482",
		},
		"spec": {
				}
			},
					}
				},
				"spec": {
						{
								"perl",
								"-wle",
							],
							},
						}
					],
					}
				}
			}
		},
		}
	}
}
{
		},
		"spec": {
				}
			},
					}
				},
				"spec": {
						{
								"perl",
								"-wle",
							],
							},
						}
					],
					}
				}
			}
		},
		}
	}
}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the late

st snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whethe

r due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set

 when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
Descripti

on
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Job
OK
201
Job
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Job
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning respo

nse header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Job
Response
Code
200
Job
OK
201
Job
Created
Pod v1 core
example
kind: Pod
metadata:
spec:
Group
Version
Kind
core
v1
Pod
Warning:
,
Job
, or
.
Field
string
kind
string
metadata
spec
PodSpec
status
PodStatus
Field
integer
affinity
Affinity
boolean
Container
array
:
merge
:
name
dnsConfig
dnsPolicy
string
boolean
array
:
merge
:
name
List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated actions 

such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod's ephemeralcontainers subresource.
HostAlias
array
:
merge
:
ip
hostIPC
boolean
boolean
hostPID
boolean
hostUsers
boolean
hostname
string
array
:
merge
:
name
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod
Container
array
:
merge
:
name
nodeName
string
object
os
PodOS
Specifies the OS of the containers in the pod. Some pod and container fields are restricted if this is set.  If the OS field is set to linux, the following fields must be unset: -securityContext.windowsOptions  If the OS field is set to windows, following fields must be unset: - spec.hostPID - spec.hostIPC - spec.hostUsers - spec.securityContext.seLinuxOptions - spec.securityContext.seccompProfile - spec.securityContext.fsGroup - spec.securityContext.fsGroupChangePolicy - spec.securityContext.sysctls - spec.shareProcessNamespace - spec.securityContext.runAsUser - spec.securityContext.runAsGroup - spec.se

curityContext.supplementalGroups - spec.containers[*].securityContext.seLinuxOptions - spec.containers[*].securityContext.seccompProfile - spec.containers[*].securityContext.capabilities - spec.containers[*].securityContext.readOnlyRootFilesystem - spec.containers[*].securityContext.privileged - spec.containers[*].securityContext.allowPrivilegeEscalation - spec.containers[*].securityContext.procMount - spec.containers[*].securityContext.runAsUser - spec.containers[*].securityContext.runAsGroup
overhead
object
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md
string
priority
integer
string
array
If specified, all readiness gates will be evaluated for pod readiness. A pod is ready when all its containers are ready AND all conditions specified in the readiness gates have status equal to "True" More info: https://git.k8s.io/enhancements/keps/sig-network/580-pod-readiness-gates
PodResourceClaim
array
:
:
name
ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume them by name.  This is an alpha field and requires enabling the DynamicResourceAllocation feature gate.  This field is immutable.
string
string
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod.  If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the "legacy" RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class
string
PodSchedulingGate
array
:
merge
:
name
SchedulingGates is an opaque list of values that if specified will block scheduling the pod. More info:  https://git.k8s.io/enhancements/keps/sig-scheduling/3521-pod-scheduling-readiness.  This is an alpha-level feature enabled by PodSchedulingReadiness feature gate.

string
string
boolean
boolean
subdomain
string
integer
array
array
:
merge
:
volumes
Volume
array
:
:
name
Field
array
:
merge
:
type
array
The list has one entry per container in the manifest. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#pod-and-container-status
array
hostIP
string
array
message
string
string
nominatedNodeName is set only when this pod preempts other pods on the node, but i

t cannot be scheduled right away as preemption victims receive their graceful termination periods. This field does not guarantee that the pod will be scheduled on this node. Scheduler may decide to place the pod elsewhere if other nodes become available sooner. Scheduler may also decide to give the resources on this node to a higher priority pod that is created after preemption. As a result, this field may be different than PodSpec.nodeName when the pod is scheduled.
phase
string
podIP
string
podIPs
PodIP
array
:
merge
:
ip
qosClass
string
reason
string
startTime
Time
Field
string
items
Pod
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun

Parameter
body
Pod
Response
Code
200
Pod
OK
201
Pod
Created
202
Pod
Accepted
Parameter
name
namespace
Parameter
dryRun
pretty
Parameter
body
Eviction
Response
Code
200
Eviction
OK
201
Eviction
Created
202
Eviction
Accepted
Patch
Parameter
name
namespace
object name and auth scope, such as for teams and

 projects
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Pod
OK
201
Pod
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a B

adRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Pod
Response
Code
200
Pod
OK
201
Pod
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Pod
OK
202
Pod
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
gracePeri

odSeconds
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Pod
OK
List
Parameter
namespace
Parameter
pretty
If 'true', then the output is pretty print

ed.
continue
limit
watch
Watch for changes to the described resources and return them as a stream of add, update, and remove notificatio

ns. Specify resourceVersion.
Response
Code
200
PodList
OK
Parameter
continue
limit
pretty
tim

watch
Response
Code
200
PodList
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVer

sion
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be incl

uded in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empt

y, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for non-apply patch requests

.
Parameter
body
Patch
Response
Code
200
Pod
OK
201
Pod
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Pod
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Pod
Response
Code
200
Pod
OK
201
Pod
Created
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, an

d will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
Pod
OK
201
Pod
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Pod
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Pod
Response
Code
200
Pod
OK
201
Pod
Created
Parameter
name
namespace
Query Para

meters
Parameter
ports
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
ports
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
GET /api/v1/namespaces/{namespace}/pods/{name}/proxy/{path}
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response

Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Read Log
Parameter
name
namespace
Parameter
container
follow
pretty
previous
tailLines
Response
Code
200
string
OK
example
metadata:
spec:
  selector:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
Group
Version
Kind
apps
v1
Warning:
Field
string
kind
string
Kind is a string value representing the REST resource this object represents. Serv

ers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
spec
status
Field
integer
replicas
integer
selector
template
Field
integer
array
:
merge
:
type
integer
integer
integer
replicas
integer
Replicas is the most recently observed number of replicas. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller/#what-is-a-replicationcontroller
Field
string
items
array
kind
string
metadata
ListMeta
Create
H

Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` fea

ture gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Eithe

r this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using 

limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not support

ed when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent f

rom the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does no

t recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are retu

rned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
watch individual changes to a list of ReplicaSet. deprecated: use the 'watch' parameter with

 a list operation instead.
Parameter
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the c

all, regardless of any activity or inactivity.
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.o

rg/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
Read Scale
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Scale
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
PATCH /apis/apps/v1/namespaces/{namespace}/replicaset

s/{name}/scale
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Scale
OK
201
Scale
Created
example
metadata:
spec:
  template:
    metadata:
      labels:
        app: nginx
    spec:
Group
Version
Kind
core
v1
Warning:
Field
string
kind
string
metadata
spec
Spec defines the specification of the desir

ed behavior of the replication controller. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Field
integer
replicas
integer
selector
object
template
Field
integer
array
:
merge
:
type
integer
integer
integer
replicas
integer
Replicas is the most recently observed number of replicas. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller
ReplicationControllerList v1 core
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
If 'true', then the output is pr

etty printed.
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. T

he error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default 

policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result 

can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supporte

d when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of

 continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks a

re sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
watch indivi

dual changes to a list of ReplicationController. deprecated: use the 'watch' parameter with a list operation instead.
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun

Parameter
body
Response
Code
200
OK
201
Created
Read Scale
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Scale
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all un

known and duplicate fields encountered.
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Scale
OK
201
Scale
Created
Group
Version
Kind
apps
v1
Field
string
kind
string
metadata
spec
status
Field
integer
Minimum number of seconds for which a newly created pod should be ready without any of its container crashing for it to be considered available. Defaults to 0 (pod wi

ll be considered available as soon as it is ready)
ordinals
StatefulSetOrdinals
StatefulSetPersistentVolumeClaimRetentionPolicy
persistentVolumeClaimRetentionPolicy describes the lifecycle of persistent volume claims created from volumeClaimTemplates. By default, all persistent volume claims are created as needed and retained until manually deleted. This policy allows the lifecycle to be altered, for example by deleting persistent volume claims when their stateful set is deleted, or when their pod is scaled down. This requires the StatefulSetAutoDeletePVC feature gate to be enabled, which is alpha.  +optional
string
replicas
integer
integer
selector
string
template
array
Field
integer
integer
collisionCount 

is the count of hash collisions for the StatefulSet. The StatefulSet controller uses this field as a collision avoidance mechanism when it needs to create the name for the newest ControllerRevision.
array
:
merge
:
type
integer
string
integer
integer
replicas
integer
string
integer
Field
string
items
array
kind
string
metadata
ListMeta
Standard list's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enable

d. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore

 any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key,

 but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at

 the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensu

res that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to 

support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch f

rom the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last

 duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Read Scale
Parameter
name
namespace
Parameter
p

retty
Response
Code
200
Scale
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Scale
Response
Code
200
Scale
OK
201
Scale
Created
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request w

ill still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
Scale
OK
201
Scale
Created
Workloads
Service
command.
Services
Ingress
Group
Version
Kind
core
v1
Endpoints
Field
string
kind
string
metadata
subsets
array
Field
string
items
Endpoints
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun

Parameter
body
Endpoints
Response
Code
200
Endpoints
OK
201
Endpoints
Created
202
Endpoints
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned by other people. Force flag must be unset for no

n-apply patch requests.
Parameter
body
Patch
Response
Code
200
Endpoints
OK
201
Endpoints
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Endpoints
Response
Code
200
Endpoints
OK
201
Endpoints
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter

body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from th

e object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Endpoints
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue

` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
A selector to

 restrict the list of returned objects by their labels. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue v

alue is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
Field
string
addressType specifies the type of address carried by this EndpointSlice. All addresses in this slice must be the 

same type. This field is immutable after creation. The following address types are currently supported: * IPv4: Represents an IPv4 Address. * IPv6: Represents an IPv6 Address. * FQDN: Represents a Fully Qualified Domain Name.
string
endpoints
Endpoint
array
kind
string
metadata
ports
array
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields a

re present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last dupl

icate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from t

he previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parame

ter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first lis

t result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures t

hat a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may

 choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maxim

um number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported 

when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
ClusterCIDRList [networking/v1alpha1]
Field
string
kind
string
metadata
spec
Spec is the desired state of the ClusterCIDR. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
ClusterCIDRSpec v1alpha1 networking
ClusterCIDR [networking/v1alpha1]
Field
ipv4
string
ipv6
string

integer
ClusterCIDRList v1alpha1 networking
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications

 should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and du

plicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specif

ied. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
cont

inue
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ClusterCIDR. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/networking.k8s.io/v1alpha1/watch/clustercidrs/{name}
Parameter
name
name of the ClusterCID

R
Parameter
continue
limit
pretty
watch
Watch for changes to 

the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
Watch List
watch individual changes to a list of ClusterCIDR. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/networking.k8s.io/v1alpha1/watch/clustercidrs
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highl

y recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1
Ingress
Field
string
kind
string
metadata
spec
status
Field
string
IngressClassName is the name of an IngressClass cluster resource. Ingress controller implementations use this field to know whether they should be serving this Ingress resource, by a transitive connection (controller -&gt; IngressClass -&gt; Ingress resource). Although the `kubernetes.io/ingress.class` annotation (simple constant name) was never formally defined, it was widely supported by Ingress controllers to create a direct binding between Ingress controller and Ingress resources. Newly created Ingress resources should prefer using the field. However, even though the annotation is officially deprecated, for backwards compatibility reasons, ingress controllers should still honor that annotation if present.
rules
array
tls
array
Field
IngressLoadBalancerStatus
Field
string
items
Ingress
array
kind
string
Kind is a string val

ue representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
202
Ingress
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default beha

vior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
201
Ingress
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used.

 Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it

 must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due 

to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume

 bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of Ingress. deprecated: use the 'watch' parameter wit

h a list operation instead.
Parameter
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api

-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See htt

watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Ingress
OK
201
Ingress
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Ingress
OK
Parameter
name
namespace
Parameter
pretty
If 'true', the

n the output is pretty printed.
dryRun
Parameter
body
Ingress
Response
Code
200
Ingress
OK
201
Ingress
Created
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
Field
string
Field
string
items
Ingress

Class
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown fi

eld that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this

 field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a cl

ient that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
A selecto

r to restrict the list of returned objects by their fields. Defaults to everything.
limit
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the ne

xt key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration

 change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
example
metadata:
spec:
  ports:
      port: 80
  selector:
    app: nginx
Group
Version
Kind
core
v1
Service
Field
apiVersio

n
string
kind
string
metadata
spec
status
Most recently observed status of the service. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
Field
boolean
allocateLoadBalancerNodePorts defines if NodePorts will be automatically allocated for services with type LoadBalancer.  Default is "true". It may be set to "false" if the cluster load-balancer does not rely on NodePorts.  If the caller requests specific NodePorts (by specifying a value), those requests will be respected, regardless of this field. This field may only be set for services with type LoadBalancer and will be cleared if the type is changed to any other type.
clusterIP
string
ClusterIPs is a list of IP addresses assigned to this service, and are usually assigned randomly.  If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated to the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be empty) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above).  Valid values are "None", empty string (""), or a valid IP address.  Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required.  Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName.  If this field is not specified, it will be initialized from the clusterIP field.  If this field is specified, clients must ensure that clusterIPs[0] 

and clusterIP have the same value.  This field may hold a maximum of two entries (dual-stack IPs, in either order). These IPs must correspond to the values of the ipFamilies field. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
string
externalName is the external reference that discovery mechanisms will return as an alias for this service (e.g. a DNS CNAME record). No proxying will be involved.  Must be a lowercase RFC-1123 hostname (https://tools.ietf.org/html/rfc1123) and requires `type` to be "ExternalName".
string
integer
healthCheckNodePort specifies the healthcheck nodePort for the service. This only applies when type is set to LoadBalancer and externalTrafficPolicy is set to Local. If a value is specified, is in-range, and is not in use, it will be used.  If not specified, a value will be automatically allocated.  External systems (e.g. load-balancers) can use this port to determine if a given node holds endpoints for this service or not.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type). This field cannot be updated once set.
string
IPFamilies is a list of IP families (e.g. IPv4, IPv6) assigned to this service. This field is usually assigned automatically based on cluster configuration and the ipFamilyPolicy field. If this field is specified manually, the requested family is available in the cluster, and ipFamilyPolicy allows it, it will be used; otherwise creation of the service will fail. This field is conditionally mutable: it allows for adding or removing a secondary IP family, but it does not allow changing the primary IP family of the Service. Valid values are "IPv4" and "IPv6".  This field only applies to Services of types ClusterIP, NodePort, and LoadBalancer, and does apply to "headless" services. This field will be wiped when updating a Service to type ExternalName.  This field may hold a maximum of two entries (dual-stack families, in either order).  These f

amilies must correspond to the values of the clusterIPs field, if specified. Both clusterIPs and ipFamilies are governed by the ipFamilyPolicy field.
string
string
string
If specified and supported by the platform, this will restrict traffic through the cloud-provider load-balancer will be restricted to the specified client IPs. This field will be ignored if the cloud-provider does not support the feature." More info: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/
ports
array
:
merge
:
port
boolean
selector
object
string
Supports "ClientIP" and "

None". Used to maintain session affinity. Enable client IP based session affinity. Must be ClientIP or None. Defaults to None. More info: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
type
string
type determines how the Service is exposed. Defaults to ClusterIP. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. "ClusterIP" allocates a cluster-internal IP address for load-balancing to endpoints. Endpoints are determined by the selector or if that is not specified, by manual construction of an Endpoints object or EndpointSlice objects. If clusterIP is "None", no virtual IP is allocated and the endpoints are published as a set of endpoints rather than a virtual IP. "NodePort" builds on ClusterIP and allocates a port on every node which routes to the same endpoints as the clusterIP. "LoadBalancer" builds on NodePort and creates an external load-balancer (if supported in the current cloud) which routes to the same endpoints as the clusterIP. "ExternalName" aliases this service to the specified externalName. Several other fields do not apply to ExternalName services. More info: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
Field
Condition
array
:
merge
:
type
Field
string
items
Service
array
kind
string
metadata
ListMeta
Create
kubectl
command
metadata:
spec:
  ports:
      port: 80
  selector:
      app: nginx
curl
command (
requires
)
metadata:
spec:
  ports:
      port: 80
  selector:
      app: nginx
Output
{
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
  }
}
POST /api/v1/namespace

s/{namespace}/services
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Service
Response
Code
200
Service
OK
201
Service
Created
202
Service
Accepted
Patch
kubectl
command
curl
command (
requires
)
Output
"" patched
{
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
        {
        }
      ]
    }
  }
}
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity

 that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
Service
OK
201
Service
Created
Replace
kubectl
command
metadata:
spec:
  ports:
    port: 80
  selector:
    app: nginx
curl
command (
requires
)
metadata:
spec:
  ports:
    port: 80
  selector:
    app: nginx
Output
{
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
        {
        }
      ]
    }
  }
}
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecogniz

ed dryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Service
Response
Code
200
Service
OK
201
Service
Created
Delete
kubectl
command
curl
command (
requires
)
Output
{
}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Del

eteOptions
Response
Code
200
Service
OK
202
Service
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object'

s finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
kubectl
command
curl
command (
requires
)
Output
{
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
        {
        }
      ]
    }
  }
}
{
  },
  "spec": {
    "ports": [
      {
      }
    ],
    },
  },
        {
        }
      ]
    }
  }
}
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Service
OK
List
kubectl
command
curl
command (
requires
)
Parameter
namespace
o

Parameter
pretty
continue
limit
Timeout for the list/watch call. This limits the duration of the call, regardless of

 any activity or inactivity.
watch
Response
Code
200
OK
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
Watch
kubectl
command
curl
command (
requires
)
Output
{
		},
		"spec": {
			"ports": [
				{
				}
			],
			},
		},
					{
					}
				]
			}
		}
	}
}
{
		},
		"spec": {
			"ports": [
				{
				}
			],
			},
		},
					{
					}
				]
			}
		}
	}
}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next k

ey, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is 

no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The contin

ue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Query Pa

rameters
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Service
OK
201
Service
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Service
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This 

will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Service
Response
Code
200
Service
OK
201
Service
Created
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Path is the part of URLs that include service endpoints, suffixes, and parameters to use for the current proxy request to service. For example, the whole request URL is http://localhost/api/v1/namespac

es/kube-system/services/elasticsearch-logging/_search?q=user:kimchy. Path is _search?q=user:kimchy.
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
Parameter
path
Response
Code
200
string
OK
Parameter
name
namespace
path
Parameter
path
Response
Code
200
string
OK
Secrets
Volumes
for providing a filesystem external

 to the Container.  Maybe shared across Containers within the same Pod and have a lifetime persisting beyond a Container or Pod.
Group
Version
Kind
core
v1
ConfigMap
Field
string
object
data
object
immutable
boolean
kind
string
metadata
Field
string
items
ConfigMap
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and

 is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
ConfigMap
Response
Code
200
ConfigMap
OK
201
ConfigMap
Created
202
ConfigMap
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
ConfigMap
OK
201
ConfigMap
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making the

se changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
ConfigMap
Response
Code
200
ConfigMap
OK
201
ConfigMap
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to 

fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourc

eVersionMatch
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
ConfigMap
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or dele

ted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not su

pported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can 

be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last reso

urceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the

 next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
CSIDriver
Field
string
kind
string
metadata
Standard object metadata. metadata.Name indicates the name of the CSI driver that this object refers to; it MUST be the same name returned by the CSI GetPluginName() call for that driver. The driver name must be 63 characters or less, beginning and ending with an alphanumeric character ([a-z0-9A-Z]) with dashes (-), dots (

.), and alphanumerics between. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
Field
boolean
string
Defines if the underlying volume supports changing ownership and permission of the volume before being mounted. Refer to the specific FSGroupPolicy values for additional details.  This field is immutable.  Defaults to ReadWriteOnceWithFSType, which will examine each volume to determine if Kubernetes should modify ownership and permissions of the volume. With the default policy the defined fsGroup will only be applied if a fstype is defined and the volume's access mode contains ReadWriteOnce.
boolean
boolean
RequiresRepublish indicates the CSI driver wants `NodePublishVolume` being periodically called to reflect any possible change in the mounted volume. This field defaults to false.  Note: After a successful initial NodePublishVolume call, subsequent calls to NodePublishVolume should only update the contents of the volume. New mount points will not be seen by a running container.
boolean
SELinuxMount specifies if the CSI driver supports "-o context" mount option.  When "true", the CSI driver must ensure that all volumes provided by this CSI driver can be mounted separately with different `-o context` options. This is typical for storage backends that provide volumes as filesystems on block devices or as independent shared volumes. Kubernetes will call NodeStage / NodePublish with "-o context=xyz" mount option when mounting a ReadWriteOncePod volume used in Pod that has explicitly set SELinux context. In the future, it may be expanded to other volume AccessModes. In any case, Kubernetes will ensure that the volume is mounted

 only with a single SELinux context.  When "false", Kubernetes won't pass any special SELinux mount options to the driver. This is typical for volumes that represent subdirectories of a bigger shared filesystem.  Default is "false".
boolean
If set to true, storageCapacity indicates that the CSI volume driver wants pod scheduling to consider the storage capacity that the driver deployment will report by creating CSIStorageCapacity objects with capacity information.  The check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver deployment has published some suitable CSIStorageCapacity object.  Alternatively, the driver can be deployed with the field unset or false and it can be flipped later when storage capacity information has been published.  This field was immutable in Kubernetes &lt;= 1.22 and now is mutable.
array
TokenRequests indicates the CSI driver needs pods' service account tokens it is mounting volume for to do necessary authentication. Kubelet will pass the tokens in VolumeContext in the CSI NodePublishVolume calls. The CSI driver should parse and validate the following VolumeContext: "csi.storage.k8s.io/serviceAccount.tokens": {   "&lt;audience&gt;": {     "token": &lt;token&gt;,     "expirationTimestamp": &lt;expiration timestamp in RFC3339&gt;,   },   ... }  Note: Audience in each TokenRequest should be different and at most one token is empty string. To receive a new token after expiry, RequiresRepublish can be used to trigger NodePublishVolume periodically.
Field
string
items
CSIDriver
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldVali

dation
Parameter
body
CSIDriver
Response
Code
200
CSIDriver
OK
201
CSIDriver
Created
202
CSIDriver
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
CSIDriver
OK
201
CSIDriver
Created
Replace
Parameter
name
Parameter
pretty
dryRun

Parameter
body
CSIDriver
Response
Code
200
CSIDriver
OK
201
CSIDriver
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
CSIDriver
OK
202
CSIDriver
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it doe

s not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https

://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
CSIDriver
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent cont

inued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of r

esults. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
CSINode
Field
string
kind
string
metadata
spec
Field
drivers
array
:
merge
:
name
Field
string
items
CSINode
array
kind
string
metadata
ListMeta
Standard list metadata More info: https://git.k8s.io/community/contributors/devel/sig-architectu

re/api-conventions.md#metadata
Create
Parameter
pretty
dryRun
Parameter
body
CSINode
Response
Code
200
CSINode
OK
201
CSINode
Created
202
CSINode
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any dupl

icate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
CSINode
OK
201
CSINode
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
CSINode
Response
Code
200
CSINode
OK
201
CSINode
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a c

ascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
CSINode
OK
202
CSINode
Accepted
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's fin

alizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
CSINode
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in th

e event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long 

as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Secret
Field
string
data
object
immutable
boolean
kind
string
metadata
Standard object's metadata. More info: https://gi

t.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
object
type
string
Used to facilitate programmatic handling of secret data. More info: https://kubernetes.io/docs/concepts/configuration/secret/#secret-types
Field
string
items
Secret
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Secret
Response
Code
200
Secret
OK
201
Secret
Created
202
Secret
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryR

un directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
Secret
OK
201
Secret
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields 

encountered.
Parameter
body
Secret
Response
Code
200
Secret
OK
201
Secret
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this val

ue is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Secret
OK
List
Parameter
namespace
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this f

lag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
OK
GE

T /api/v1/secrets
Parameter
continue
limit
pretty
watch
Watch

 for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/doc

watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a 

client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The ser

ver guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
A
Field
string
kind
string
metadata
spec
spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
status
status represents the current information/status of a persistent volume claim. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
Field
accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
dataSource field can be used to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.
TypedObjectReference
dataSourceRef specifies the object fr

om which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef: * While dataSource only allows two specific types of objects, dataSourceRef   allows any non-core object, as well as PersistentVolumeClaim objects. * While dataSource ignores disallowed values (dropping them), dataSourceRef   preserves all values, and generates an error if a disallowed value is   specified. * While dataSource only allows local objects, dataSourceRef allows objects   in any namespaces. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled.
resources
resources represents the minimum resources the volume should have. If RecoverVolumeExpansionFailure feature is enabled users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources
selector
string
storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1
string
string
Field
accessModes contains the actual access modes the volume backing the PVC has. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1
object
capacity
object
array
:
merge
:
type
conditions is the current Condition of persistent volume claim. If underlying persistent volume is being resized then the Condition will be set to 'ResizeStarted'.
phase
string
string
resizeStatus stores status of resize operation. ResizeStatus is not set by default but when expansion is complete resizeStatus is set to empty string by resize contr

oller or kubelet. This is an alpha field and requires enabling RecoverVolumeExpansionFailure feature.
Field
string
items
array
items is a list of persistent volume claims. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is ma

king these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
PersistentVolumeClai

m
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is n

il, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
allowWatchBookmarks requests watch eve

nts with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
Per

sistentVolumeClaimList
OK
Parameter
continue
limit
pretty
timeoutSeco

nds
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limi

t - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may cho

ose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fie

lds encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Field
boolean
array
string
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer

 this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
string
string
string
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created

202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only 

persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
pretty
continue
dryRun
A selector to restrict the list

 of returned objects by their fields. Defaults to everything.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
allowWatchBookmarks requests watch even

ts with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
Stor

ageClassList
OK
Watch
Parameter
name
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resou

rceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
Group
Version
Kind
v1
v1beta1
CSIStorageCapacityList [storage/v1]
Field
string
capacity
Quantity
Capacity is the value reported by the CSI driver in its GetCapacityResponse for a GetCapacityRequest with topology and parameters that match the previous fields.  The semantic is currently (CSI spec 1.2) defined as: The available capacity, in bytes, of the storage that can be used to provision volumes. If not set, that information is currently unavailable.
kind
string
Quantity
metadata
string
CSIStorageCapacityList v1 storage
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#res

ources
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Val

id values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of

 the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of ite

ms (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical que

ry parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this 

is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/storage.

k8s.io/v1/watch/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
namespace
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
Watch List
GET /apis/storage.k8s.io/v1/watch/namespaces/{namespace}/csistoragecapacities
Parameter
namespace
Parameter
continue
limit
pretty
If 'true', then the output is pre

tty printed.
watch
Response
Code
200
OK
GET /apis/storage.k8s.io/v1/watch/csistoragecapacities
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. T

his is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Volume
Field
awsElasticBlockStore represents an AWS Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
azureDisk
azureFile
cephfs
cinder
cinder represents a cinder volume attached and mounted on kubelets host machine. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
configMap
csi
emptyDir
emptyDir represents a temporary directory that shares a pod's lifetime. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
ephemeral
ephemeral represents a volume that is handled by a cluster storage driver. The volume's lifecycle is tied to the pod that defines it - it will be created before the pod starts, and deleted when the pod is removed.  Use this if: a) the volume is only needed while the pod runs, b) features of normal volumes like restoring from snapshot or capacity    tracking are needed, c) the storage driver is specified through a storage class, and d) the storage driver supports dynamic volume provisioning through    a PersistentVolumeClaim (see EphemeralVolumeSource for more    information on the connection between this volume type    and PersistentVolumeClaim).  Use PersistentVolumeClaim or one of the vendor-specific APIs for volumes that persist for longer than the lifecycle of an individual pod.  Use CSI for light-weight local ephemeral volumes if the CSI driver is meant to be used that way - see the documentation of the driver for more information.  A pod can use both types of ephemeral volumes and persistent volumes at the same time.
fc
flocker
F

lockerVolumeSource
gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
gitRepo
glusterfs
glusterfs represents a Glusterfs mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/glusterfs/README.md
hostPath
hostPath represents a pre-existing file or directory on the host machine that is directly exposed to the container. This is generally used for system agents or other privileged things that are allowed to see the host machine. Most containers will NOT need this. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
iscsi
iscsi represents an ISCSI Disk resource that is attached to a kubelet's host machine and then exposed to the pod. More info: https://examples.k8s.io/volumes/iscsi/README.md
name
string
name of the volume. Must be a DNS_LABEL and unique within the pod. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
nfs
nfs represents an NFS mount on the host that shares a pod's lifetime More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
persistentVolumeClaimVolumeSource represents a reference to a PersistentVolumeClaim in the same namespace. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
projected
quobyte
rbd
rbd represents a Rados Block Device mount on the host that shares a pod's lifetime. More info: https://examples.k8s.io/volumes/rbd/README.md
scaleIO
secret
secret represents a secret that should populate this volume. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret
storageos
Group
Version
Kind
v1
Field
string
kind
string
metadata
Standard obj

ect metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
status
Field
attacher
string
nodeName
string
source
Field
attached
boolean
object
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate fi

eld that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, pro

vided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
pretty
continue

dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/

reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred t

o as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results a

re available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200

OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
METADATA
Event
Group
Version
Kind
apps
v1
Field
string
data
kind
string
metadata
revision
integer
Revision indicates the revision of the state represented b

y Data.
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optio

nal for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
object name and auth scope, such as for teams an

d projects
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of respon

ses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this i

s not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement book

marks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200

RevisionList
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
pretty

nMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they

 see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue 

will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
Field
string
kind
string
metadata
Standard object's metadata More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
status
Field
group
string
names
boolean
preserveUnknownFields indicates that object fields which are not specified in the OpenAPI schema should be preserved when persisting to storage. apiVersion, kind, metadata and known fields inside metadata are always preserved. This field is deprecated in favor of setting `x-preserve-unknown-fields` to true in `spec.versions[*].schema.openAPIV3Schema`. See https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#field-pruning for details.
scope
string
versions
array
versions is the list of all API versions of the defined custom resource. Version names are used to compute the order in which served versio

ns are listed in API discovery. If the version string is "kube-like", it will sort above non "kube-like" version strings, which are ordered lexicographically. "Kube-like" versions start with a "v", then are followed by a number (the major version), then optionally the string "alpha" or "beta" and another number (the minor version). These are sorted first by GA &gt; beta &gt; alpha (where GA is a version with no suffix such as beta or alpha), and then by comparing major version, then minor version. An example sorted list of versions: v10, v2, v1, v11beta2, v10beta3, v3beta1, v12alpha1, v11alpha2, foo1, foo10.
Field
array
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default w

hen the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate

 is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue

dryRun
limit
Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or

 inactivity.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
resourceVersion sets a constraint on what resource versions 

a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first re

quest is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all o

f the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Force is going to "force" Apply requests. It means user will re-acquire conflicting fields owned 

by other people. Force flag must be unset for non-apply patch requests.
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Event
Field
action
string
string
integer
Time
Time
eventTime
MicroTime
eventTime is the time when this Event was first observed. It is requir

ed.
kind
string
metadata
note
string
reason
string
regarding
related
string
string
series
type
string
Field
string
items
Event
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the 

default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
202
Event
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Event
OK
201
Event
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must

 be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a confi

guration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit

onMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Event
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first r

equest is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
EventList
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch i

s true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
EventList
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the sa

me initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion 

value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next

 key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Field
string
kind
string
metadata
spec
LimitRange [core/

v1]
Field
limits
array
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined 

by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
object name and auth scope,

 such as for teams and projects
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of respons

es to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue o

ption should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients shoul

d not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind LimitRange. deprecated: use the 'watch' pa

rameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for lis

t calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
GET /api/v1/watch/namespaces/{namespace}/limitranges
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result wa

s calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any

 subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v2
v1
HorizontalPodAutoscalerList [autoscaling/v2]
Field
string
kind
string
metadata
spec
status
HorizontalPodAutoscalerSpec v2 autoscaling
HorizontalPodAutoscaler [autoscaling/v2]
Field
behavior
integer
metrics
array
integer
minReplicas is the lower limit for the number of replicas to which the autoscaler can scale down.  It defaults to 1 pod.  minReplicas is allowed to be 0 if the alpha feature gate HPAScaleToZero is enabled and at least one Object or External metric is configured.  Scaling is active as long as at least one m

etric value is available.
HorizontalPodAutoscalerStatus v2 autoscaling
Field
array
:
merge
:
type
array
integer
integer
Time
integer
HorizontalPodAutoscalerList v2 autoscaling
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplica

te fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ign

ore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may se

nd another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-v

ersions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensure

s that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch
GET /apis/autoscaling/v2/watch/namespaces/{namespace}/horizontalpodautoscalers/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query

 to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/autoscaling/v2/watch/namespaces/{namespace}/horizontalpodautoscalers
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the las

t resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/autoscaling/v2/watch/horizontalpodautoscalers
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the se

rver will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply pat

ch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are pr

esent. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Field
string
kind
string
metadata
webhooks
array
:
merge
:
name
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: T

his will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also

 enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting fr

om the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Timeout for the list/watch call. This limits the duration of

 the call, regardless of any activity or inactivity.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit

ed.
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a

 single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to 

determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
Field
string
kind
string
metadata
webhooks
array
:
merge
:
name
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/admissionregi

stration.k8s.io/v1/validatingwebhookconfigurations
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the

 request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Whether and how garbage c

ollection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks 

of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value r

eturned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue 

field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
PodSchedulingList [resource/v1alpha1]
Field
string
kind
string
Kind is a string value representing the RE

ST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
spec
status
PodSchedulingSpec v1alpha1 resource
PodScheduling [resource/v1alpha1]
Field
PotentialNodes lists nodes where the Pod might be able to run.  The size of this field is limited to 128. This is large enough for many clusters. Larger clusters may need more attempts to find a node that suits all pending resources. This may get increased in the future, but not reduced.
string
PodSchedulingStatus v1alpha1 resource
Field
ResourceClaimSchedulingStatus
array
PodSchedulingList v1alpha1 resource
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fiel

ds would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that 

are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If ob

jects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be inc

luded in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PodScheduling. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/resource.k8s.io/v1alpha1/watch/namespaces/{namespace}/podschedulings/{name}
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and c

lients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of PodScheduling. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/resource.k8s.io/v1alpha1/watch/namespaces/{namespace}/podschedulings
Parameter
namespace
Parameter
continue
labelS

elector
limit
pretty
watch
Response
Code
200
OK
GET /apis/resource.k8s.io/v1alpha1/watch/podschedulings
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the fi

rst list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun

 `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
core
v1
Ap

pears In:
Field
string
kind
string
metadata
template
Field
metadata
spec
PodSpec
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no

 other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or 

duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client m

ay send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resou

rce-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensu

re they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued w

ill be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PodTemplate. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only us

e the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
A selector to restrict the list of retur

ned objects by their labels. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys ar

e after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
policy
v1
Field
string
kind
string
metadata
spec
status
Field
An eviction is allowed if at most "maxUnavailable" pods selected by "selector" are unavailable after the evictio

n, i.e. even in absence of the evicted pod. For example, one can prevent all voluntary evictions by specifying 0. This is a mutually exclusive setting with "minAvailable".
selector
:
replace
string
UnhealthyPodEvictionPolicy defines the criteria for when unhealthy pods should be considered for eviction. Current implementation considers healthy pods, as pods that have status.conditions item with type="Ready",status="True".  Valid policies are IfHealthyBudget and AlwaysAllow. If no policy is specified, the default behavior will be used, which corresponds to the IfHealthyBudget policy.  IfHealthyBudget policy means that running pods (status.phase="Running"), but not yet healthy can be evicted only if the guarded application is not disrupted (status.currentHealthy is at least equal to status.desiredHealthy). Healthy pods will be subject to the PDB for eviction.  AlwaysAllow policy means that all running pods (status.phase="Running"), but not yet healthy are considered disrupted and can be evicted regardless of whether the criteria in a PDB is met. This means perspective running pods of a disrupted application might not get a chance to become healthy. Healthy pods will be subject to the PDB for eviction.  Additional policies may be added in the future. Clients making eviction decisions should disallow eviction of unhealthy pods if they encounter an unrecognized policy in this field.  This field is alpha-level. The eviction API uses this field when the feature gate PDBUnhealthyPodEvictionPolicy is enabled (disabled by default).
Field
Condition
array
:
merge
:
type
Conditions contain conditions for PDB. The disruption controller sets the DisruptionAllowed condition. The following are known values for the reason field (additional reasons could be added in the future): - SyncFailed: The controller encountered an error and wasn't able to compute               the number of allowed disruptions. Therefore no disruptions are               allowed and the status of the condition will be False. - InsufficientPods: The number of pods are either at or below the number                     required by the PodDisruptionBudget. No disruptions are                     allowed and the status of the condition will be False. - SufficientPods: There are more pods than required by the PodDisruptionBudget.                   The condition will be True, and the number of allowed                   disruptions are provided by the disruptionsAllowed property.
integer
integer
object
DisruptedPods contains information about pods whose eviction was processed by the API server eviction subresource handler but has not yet been observed by the PodDisruptionBudget controller. A pod will be in this map from the time when the API server processed the eviction request to the time when the pod is seen by PDB controller as having been marked for deletion (or after a timeout). The key in the map is the name of the pod and the value is the time when the API server processed the eviction request. If the deletion didn't occur and a pod is still there it will be removed from the list automatically by PodDisruptionBudget controller after some time. If everything goes smooth this map should be empty for the most of the time. Large number of entrie

s in the map may indicate problems with pod deletions.
integer
integer
integer
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will 

result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from th

e server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
A selector to restrict the list of returned objects by their fields. Defaults to everythin

g.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
object name and auth sc

ope, such as for teams and projects
Parameter
pretty
continue
limit
Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inact

ivity.
watch
Response
Code
200
OK
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVe

rsionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can 

ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results 

are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (u

p to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request

 with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Field
string
string
boolean
globalDefault specifies whether this PriorityClass should be considered as the default priority for pods that do not have any priority class. Only one PriorityClass can be marked as `globalDefault`. 

However, if more than one PriorityClasses exists with their `globalDefault` field set to true, the smallest value of such global default PriorityClasses will be used as the default priority.
kind
string
metadata
string
value
integer
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Pa

th Parameters
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: 

This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be

 deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the serv

er's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind PriorityClass. deprecated: use the 'watch' parameter with a li

st operation instead, filtered to a single item with the 'fieldSelector' parameter.
Parameter
name
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. Se

e https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
ResourceClaimTemplateList [resource/v1alpha1]
Field
string
kind
string
metadata
spec
Describes the ResourceClaim that is to be generated.  This field is immutable. A ResourceClaim will get created by the control plane for a Pod when needed and then not get updated anymore.
ResourceClaimTemplateSpec v1alpha1 resource
ResourceClaimTemplate [resource/v1alpha1]
Field
metadata
spec
ResourceClaimTemplateList v1alpha1 resource
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidati

on instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
PUT /apis/resource.k8s.io/v1alpha1/namespaces/{namesp

ace}/resourceclaimtemplates/{name}
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
ResourceClaimTemplat

e
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizer

s list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum n

umber of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and no

t miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ResourceClaimTemplate. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/resource.k8s.io/v1alpha1/watch/namespaces/{namespace}/resourceclaimtemplates/{name}
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client

 needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ResourceClaimTemplate. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/resource.k8s.io/v1alpha1/watch/namespaces/{namespace}/resourceclaimtemplates
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results fr

om the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
GET /apis/resource.k8s.io/v1alpha1/watch/resourceclaimtemplates
Parameter
allowWatchBookmarks requests watch events with type "BOO

KMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion

.
Response
Code
200
OK
Group
Version
Kind
v1alpha1
ResourceClassList [resource/v1alpha1]
Field
string
string
DriverName defines the name of the dynamic resource driver that is used for allocation of a ResourceClaim that uses this class.  Resource drivers have a unique name in forward domain order (acme.example.com).
kind
string
metadata
ResourceClassParametersReference
Only nodes matching the selector will be considered by the scheduler when trying to find a Node that fits a Pod when that Pod uses a ResourceClaim that has not been allocated yet.  Setting this field is optional. If null, all nodes are candidates.
ResourceClassList v1alpha1 resource
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any dupl

icate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are si

lently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the r

esponse, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Parameter
body
Response
Code
200
Status
OK
Read

Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
resourceVersionMatch determines how resourceVersion is appl

ied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ResourceClass. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/resource.k8s.io/v1alpha1/watch/resourceclasses/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If 

objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ResourceClass. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/resource.k8s.io/v1alpha1/watch/resourceclasses
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single

 list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
ValidatingAdmissionPolicyList [admissionregistration/v1alpha1]
Field
string
kind
string
metadata
spec
ValidatingAdmissionPolicySpec
ValidatingAdmissionPolicySpec v1alpha1 admissionregistration
ValidatingAdmissionPolicy [admissionregistration/v1alpha1]
Field
string
MatchResources
MatchConstraints specifies what resources this policy is designed to validate. The AdmissionPolicy cares about a request if it matches _all_ Constraints. However, in order to prevent clusters from being put into an unstable state that cannot be recovered from via the API ValidatingAdmissionPolicy cannot match ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding. Required.
paramKind
ParamKind
array
Validations contain CEL expressions which 

is used to apply the validation. A minimum of one validation is required for a policy definition. Required.
ValidatingAdmissionPolicyList v1alpha1 admissionregistration
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is re

quired for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
name of the

 ValidatingAdmissionPolicy
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will s

et the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the

 continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ValidatingAdmissionPolicy. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/admissionregistration.k8s.io/v1alpha1/watch/validatingadmissionpolicies/{name}
Parameter
name
Parameter

rks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Watch for changes to the described resources and return them as a stream of add, update, and remov

e notifications. Specify resourceVersion.
Response
Code
200
OK
Watch List
watch individual changes to a list of ValidatingAdmissionPolicy. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/admissionregistration.k8s.io/v1alpha1/watch/validatingadmissionpolicies
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMat

ch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
ValidatingAdmissionPolicyBindingList [admissionregistration/v1alpha1]
Field
string
kind
string
metadata
spec
ValidatingAdmissionPolicyBindingSpec
ValidatingAdmissionPolicyBindingSpec v1alpha1 admissionregistration
ValidatingAdmissionPolicyBinding [admissionregistration/v1alpha1]
Field
paramRef
ParamRef
string
ValidatingAdmissionPolicyBindingList v1alpha1 admissionregistration
Field
string
items
array
List of PolicyBinding.
kind
string
metadata
ListMeta
Create
create a Validating

AdmissionPolicyBinding
POST /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicybindings
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
partially update the specified ValidatingAdmissionPolicyBinding
PATCH /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicybindings/{name}
Parameter
name
name of the ValidatingAdmissionPolicyBinding
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default wh

en the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
replace the specified ValidatingAdmissionPolicyBinding
PUT /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicybindings/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
delete a ValidatingAdmissionPolicyBinding
DELETE /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicybindings/{name}
Parameter
name
Parameter
pretty
dryRun

Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
delete collection of ValidatingAdmissionPolicyBinding
DELETE /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicybindings
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued reques

ts. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
read the specified ValidatingAdmissionPolicyBinding
GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicybindings/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
list or watch objects of kind ValidatingAdmissionPolicyBinding
GET /apis/admissionregistration.k8s.io/v1alpha1/validatingadmissionpolicybindings
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as l

ong as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
ValidatingAdmissionPolicyBindingList
OK
Watch
watch changes to an object of kind ValidatingAdmissionPolicyBinding. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/admissionregistration.k8s.io/v1alpha1/watch/validatingadmissionpolicybindings/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the se

rver will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
watch individual changes to a list of ValidatingAdmissionPolicyBinding. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/admissionregistration.k8s.io/v1alpha1/watch/validatingadmissionpolicybindings
Parameter
continue
The continue option should be set when retrieving more results f

rom the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
CLUSTER
Group
Version
Kind
v1
Field
string
APIVersion defines the versi

oned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
metadata
spec
status
Field
caBundle
string
group
string
integer
boolean
service
version
string
integer
Field
array
:
merge
:
type
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-archite

cture/api-conventions.md#resources
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFi

eldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be

 orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watc

h from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another l

ist request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified co

ntinue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldManager is a name associated

 with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any du

plicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
core
v1
Binding
Field
string
kind
string
metadata
target
Create
Parameter
namespace
Parameter
dryRun
pretty
Parameter
body
Binding
Response
Code
200
Binding
OK
201
Binding
Created
202
Binding
Accepted
Group
Version
Kind
v1
Field
string
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.i

o/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
spec
status
Field
integer
expirationSeconds is the requested duration of validity of the issued certificate. The certificate signer may issue a certificate with a different validity duration so a client must check the delta between the notBefore and and notAfter fields in the issued certificate to determine the actual duration.  The v1.22+ in-tree implementations of the well-known Kubernetes signers will honor this field as long as the requested duration is not greater than the maximum duration they will honor per the --cluster-signing-duration CLI flag to the Kubernetes controller manager.  Certificate signers may not honor this field for various reasons:    1. Old signer that is unaware of the field (such as the in-tree      implementations prior to v1.22)   2. Signer whose configured maximum is shorter than the requested duration   3. Signer whose configured minimum is longer than the requested duration  The minimum valid value for expirationSeconds is 600, i.e. 10 minutes.
extra
object
groups
request
string
string
signerName indicates the requested signer, and is a qualified name.  List/watch requests for CertificateSigningRequests can filter on this field using a "spec.signerName=NAME" fieldSelector.  Well-known Kubernetes signers are:  1. "kubernetes.io/kube-apiserver-client": issues client certificates that can be used to authenticate to kube-apiserver.   Requests for this signer are never auto-approved by kube-controller-manager, can be issued by the "csrsigning" controller in kube-controller-manager.  2. "kubernetes.io/kube-apiserver-client-kubelet": issues client certificates that kubelets use to authenticate to kube-apiserver.   Requests for this signer can be auto-approved by the "csrapproving" controller in kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager.  3. "kubernetes.io/kubelet-serving" issues serving certificates that kubelets use to serve TLS endpoints, which kube-apiserver can connect to securely.   Requests for this signer are never auto-approved by kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager.  More details are available at https://k8s.io/docs/reference/access-authn-authz/certificate-signing-requests/#kubernetes-signers  Custom signerNames can also be specified. The signer defines:  1. Trust distribution: how trust (CA bundles) are distributed.  2. Permitted subjects: and behavior when a disallowed subject is requested.  3. Required, permitted, or forbidden x509 extensions in the request (including whether subjectAltNames are allowed, which types, restrictions on allowed values) and behavior when a disallowed extension is requested.  4. Required, permitted, or forbidden key usages / extended key usages.  5. Expiration/certificate lifetime

: whether it is fixed by the signer, configurable by the admin.  6. Whether or not requests for CA certificates are allowed.
uid
string
usages
username
string
Field
string
array
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processing of the request. Valid values are: - All:

 all dry run stages will be processed
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Body Pa

rameters
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
Descr

iption
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy ma

y be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit

watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications

.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest sn

apshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/

PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created

Group
Version
Kind
v1
Field
string
kind
string
metadata
rules
array
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/rbac.authorization.k8s.io/v1/cl

usterroles/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enab

led. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
The duration in seconds before the obj

ect should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are s

ent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ClusterRole. deprecated: use the 'watch' paramete

r with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
Parameter
name
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/r

eference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may b

e served from. See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Group
Version
Kind
v1
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from

 the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle

 objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with t

he 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
r

watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to

 support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Field
string
array
:
merge
:
type
kind
string
metadata
Field
string
items
array
kind
string
metadata
ListMeta
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
HTTP Req

uest
Parameter
continue
limit
pretty
Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inacti

vity.
watch
Response
Code
200
OK
Group
Version
Kind
v1beta3
v1beta2
FlowSchemaList [flowcontrol/v1beta3]
Field
string
kind
string
metadata
spec
status
FlowSchemaSpec v1beta3 flowcontrol
FlowSchema [flowcontrol/v1beta3]
Field
integer
rules
array
FlowSchemaStatus v1beta3 flowcontrol
Field
array
:
merge
:
type
FlowSchemaList v1beta3 flowcontrol
Field
string
items
array
kind
string
metadata
ListMeta
Write Op

erations
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from 

the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Backgro

und' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should

 the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added to/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a valu

e that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/watch/flowschemas/{name}
Parameter
name
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resource

Version value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/watch/flowschemas
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list

 starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in th

e request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Lease
Field
string
APIVersion defines t

he versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
metadata
spec
LeaseSpec
Field
MicroTime
string
integer
integer
renewTime
MicroTime
Field
string
items
Lease
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fi

elds. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Lease
Response
Code
200
Lease
OK
201
Lease
Created
202
Lease
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Lease
OK
201
Lease
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. V

alid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Lease
Response
Code
200
Lease
OK
201
Lease
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list st

arting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Timeout for the list/watch call. This limits the du

ration of the call, regardless of any activity or inactivity.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Lease
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object 

that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
LeaseList
OK
Parameter
continue
limit

pretty
watch
Response
Code
200
LeaseList
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose 

not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of response

s to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may st

art a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
status
Create
Parameter
namesp

ace
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
core
v1
Namespace
Field
string
kind
string
metadata
spec
status
Field
Field
array
:
merge
:
type
phase
string
Field
string
APIVersion defines the ver

sioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
items
Namespace
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Namespace
Response
Code
200
Namespace
OK
201
Namespace
Created
202
Namespace
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabl

ed. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
Namespace
OK
201
Namespace
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Namespace
Response
Code
200
Namespace
OK
201
Namespace
Created
Delete
Parameter
name
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type wil

l be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Read
Parameter
name
Parameter
pretty
Response
Code
200
Namespace
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more re

sults are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue

limit
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encount

ered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
Namespace
OK
201
Namespace
Created
Parameter
name
Parameter
pretty
Response
Code
200
Namespace
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Namespace
Response
Code
200
Namespace
OK
201
Namespace
Created
Group
Version
Kind
core
v1
Node
Field
string
kind
string
metadata
spec
NodeSpec
Spec defines the beha

vior of a node. https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Field
string
podCIDR
string
podCIDRs
:
merge
string
taints
Taint
array
boolean
Field
addresses
array
:
merge
:
type
List of addresses reachable to the node. Queried from cloud provider, if available. More info: https://kubernetes.io/docs/concepts/nodes/node/#addresses Note: This field is declared as mergeable, but the merge key is not sufficiently unique, which can cause data corruption when it is merged. Callers should instead use a full-replacement patch. See https://pr.k8s.io/79391 for an example.
object
capacity
object
array
:
merge
:
type
config
images
array
nodeInfo
phase
string
array
Field
string
items
Node
array
kind
string
metadata
ListMeta
Cr

eate
Parameter
pretty
dryRun
Parameter
body
Node
Response
Code
200
Node
OK
201
Node
Created
202
Node
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields en

countered.
force
Parameter
body
Patch
Response
Code
200
Node
OK
201
Node
Created
Replace
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Node
Response
Code
200
Node
OK
201
Node
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Acc

epted
Parameter
pretty
continue
dryRun
limit
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be se

t, but not both. The default policy is decided by the existing finalizer set in the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
Node
OK
List
Parameter
pretty
continue
limit

watch
Response
Code
200
NodeList
OK
Watch
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same ini

tial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
A selector to restrict the list of returned objects by their fields. Defaults 

to everything.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from th

e object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
Node
OK
201
Node
Created
Parameter
name
Parameter
pretty
Response
Code
200
Node
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Node
Response
Code
200
Node
OK
201
Node
Created
Parameter
name
Parameter
path
Response
Code
200
string
OK
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
Query 

Parameter
path
Response
Code
200
string
OK
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
Parameter
path
Response
Code
200
string
OK
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
Parameter
path
Response
Code
200
string
OK
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
Parameter
name
Parameter
path
Response
Code
200
string
OK
Parameter
name
path
Parameter
path
Response
Code
200
string
OK
Group
Version
Kind
core
v1
Pods
using
.
Field
string
kind
string
metadata
spec
spec defines a specification of a persistent volume owned by the cluster. Provisioned by an administrator. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
status
status represents the current information/status for the persistent volume. Populated by the system. Read-only. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistent-volumes
Persist

entVolumeSpec v1 core
Field
accessModes contains all ways the volume can be mounted. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes
azureDisk
azureFile
capacity
object
capacity is the description of the persistent volume's resources and capacity. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#capacity
cephfs
cinder
claimRef
claimRef is part of a bi-directional binding between PersistentVolume and PersistentVolumeClaim. Expected to be non-nil when bound. claim.VolumeName is the authoritative bind between PV and PVC. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#binding
csi
fc
flocker
gcePersistentDisk represents a GCE Disk resource that is attached to a kubelet's host machine and then exposed to the pod. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
glusterfs
glusterfs represents a Glusterfs volume that is attached to a host and exposed to the pod. Provisioned by an admin. More info: https://examples.k8s.io/volumes/glusterfs/README.md
hostPath
hostPath represents a directory on the host. Provisioned by a developer or tester. This is useful for single-node development and testing only! On-host storage is not supported in any way and WILL NOT WORK in a multi-node cluster. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
iscsi
local
mountOptions is the list of mount options, e.g. ["ro", "soft"]. Not validated - mount will simply fail if one is invalid. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#mount-options
nfs
nfs represents an NFS mount on the host. Provisioned by an admin. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
string
persistentVolumeReclaimPolicy defines what happens to a persistent volume when released from its claim. Valid options are Retain (default for manual

ly created PersistentVolumes), Delete (default for dynamically provisioned PersistentVolumes), and Recycle (deprecated). Recycle must be supported by the volume plugin underlying this PersistentVolume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#reclaiming
quobyte
rbd
scaleIO
string
storageos
storageOS represents a StorageOS volume that is attached to the kubelet's host machine and mounted into the pod More info: https://examples.k8s.io/volumes/storageos/README.md
string
Field
message
string
phase
string
phase indicates if a volume is available, bound to a claim, or released by a claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#phase
reason
string
Field
string
items
array
items is a list of persistent volumes. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fi

elds, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result i

n an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue va

lue is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes r

eferred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be 

dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta3
v1beta2
PriorityLevelConfigurationList [flowcontrol/v1beta3]
Field
string
kind
string
metadata
spec
`spec` is

 the specification of the desired behavior of a "request-priority". More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
PriorityLevelConfigurationSpec v1beta3 flowcontrol
PriorityLevelConfiguration [flowcontrol/v1beta3]
Field
limited
type
string
PriorityLevelConfigurationStatus v1beta3 flowcontrol
Field
array
:
merge
:
type
PriorityLevelConfigurationList v1beta3 flowcontrol
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplic

ate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unkn

own or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with t

he token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults

 to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the

 object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Watch
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/watch/prioritylevelconfigurations/{name}
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarant

ees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
GET /apis/flowcontrol.apiserver.k8s.io/v1beta3/watch/prioritylevelconfigurations
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects ar

e filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun

force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
core
v1
Field
string
kind
string
metadata
spec
Spec defines the desired quota. https://git.k8s.io

/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
status
Field
hard
object
scopes
Field
hard
object
used
object
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadReque

st error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun

Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list 

result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is

 using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argum

ent and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If mor

e items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersio

n value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the d

ecoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Role
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized v

alues. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
metadata
rules
array
Field
string
items
Role
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
202
Role
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
field

Manager
force
Parameter
body
Patch
Response
Code
200
Role
OK
201
Role
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Role
Response
Code
200
Role
OK
201
Role
Created
Delete
H

Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object va

lue if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Role
OK
List
Parameter
namespace
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Cli

ents should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
RoleList
OK
Pa

rameter
continue
limit
pretty
watch
Watch for changes to the described resourc

es and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
RoleList
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-

api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures th

at a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supporte

d if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
Field
string
kind
string
metadata
roleRef
RoleRef
subjects
Subject
array
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldManager 

is a name associated with the actor or entity that is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
Descript

ion
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
DeleteOpti

ons
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
Deprecated: please use the PropagationPolicy, this field will be deprecated in 1.7. Should the dependent objects be orphaned. If true/false, the "orphan" finalizer will be added t

o/removed from the object's finalizers list. Either this field or PropagationPolicy may be set, but not both.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit

watch
Response
Code
200
OK
Parameter
continue
A

limit
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue fie

ld. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value fr

om a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's d

iscretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
R

untimeClass
Field
string
handler
string
kind
string
metadata
overhead
Overhead
Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. For more details, see  https://kubernetes.io/docs/concepts/scheduling-eviction/pod-overhead/
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning res

ponse header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) c

ontaining unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is

 inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
Bo

dy Parameters
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
Parameter
pretty
continue
limit

watch
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that i

s using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The serve

r guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
status
Field
Create
Parameter
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped f

rom the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1alpha1
Field
string
kind
string
metadata
status
SelfSubjectReviewStatus v1alpha1 authentication
SelfSubjectReview [authentication/v1alpha1]
Field
userInfo
UserInfo
Create
Parameter
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or 

if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
status
Field
namespace
string
Create
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
core
v1
Field
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest 

internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
boolean
array
kind
string
metadata
secrets
array
:
merge
:
name
Secrets is a list of the secrets in the same namespace that pods running using this ServiceAccount are allowed to use. Pods are only limited to this list if this service account has a "kubernetes.io/enforce-mountable-secrets" annotation set to "true". This field should not be used to find auto-generated service account token secrets for use outside of pods. Instead, tokens can be requested directly using the TokenRequest API, or service account token secrets can be manually created. More info: https://kubernetes.io/docs/concepts/configuration/secret
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignor

e all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in an error response and no further processi

ng of the request. Valid values are: - All: all dry run stages will be processed
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result wi

th identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
r

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that n

o more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to det

ermine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
A selector to restrict the list of returned objects by their fields. Defaults to ev

erything.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, whic

h is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will resp

ond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1alpha1
Field
string
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architectur

e/api-conventions.md#types-kinds
metadata
spec
status
API server instances report the version they can decode and the version they encode objects to when persisting objects in the backend.
Field
Field
string
array
array
Field
string
items
array
Items holds a list of StorageVersion
kind
string
metadata
ListMeta
Create
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
StorageVer

sion
OK
201
Created
202
Accepted
Patch
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is enc

ountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
pretty
continue
dryRun

limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/internal.apiserver.k8s.io/v1alpha1/st

orageversions
Parameter
pretty
continue
limit
watch
Watch for

 changes to the described resources and return them as a stream of add, update, and remove notifications. Specify resourceVersion.
Response
Code
200
OK
Watch
Parameter
name
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/reference/using-api/api-conce

pts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible obj

ects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
Parameter
pretty
Response
Code
200
OK
PUT /apis/internal.apiserver.k8s.io/

v1alpha1/storageversions/{name}/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
status
Field
extra
object
groups
uid
string
user
string
LocalSubjec

tAccessReview [authorization/v1]
Field
allowed
boolean
denied
boolean
string
reason
string
Create
Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
status
Status is filled in by the server and indicates

 whether the token can be authenticated.
Field
audiences
integer
Field
Time
token
string
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
status
Field
audiences
token
string
Field
audiences
boolean
error
string
user
UserInfo
Create

Parameter
dryRun
pretty
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Group
Version
Kind
v1
Field
string
kind
string
metadata
spec
status
Status is the current state of the NetworkPolicy. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status
Field
egress
array
ingress
array
List of ingress rules to be applied to the selected pods. Traffic is allowed to a pod if there are no NetworkPolicies selecting the pod (and cluster policy otherwise allows the traffic), OR if the traffic source is the pod's local node, OR 

if the traffic matches at least one ingress rule across all of the NetworkPolicy objects whose podSelector matches the pod. If this field is empty then this NetworkPolicy does not allow any traffic (and serves solely to ensure that the pods it selects are isolated by default)
NetworkPolicyStatus v1 networking
Field
Condition
array
:
merge
:
type
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature g

ate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity t

hat is making these changes. The value must be less than or 128 characters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint.
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the 

specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
dryRun
limit
resourceVersion sets a constraint on what resource versions a request may be served from. See https://kubernetes.io/docs/re

ference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that 

the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
A selector to restrict the list of returned obj

ects by their labels. Defaults to everything.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - ob

jects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired erro

r together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldManager is a name associated with the actor or entity that is making these changes. The value must be less than or 128 ch

aracters long, and only contain printable characters, as defined by https://golang.org/pkg/unicode/#IsPrint. This field is required for apply requests (application/apply-patch) but optional for non-apply patch types (JsonPatch, MergePatch, StrategicMergePatch).
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will

 fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
meta
v1
APIGroup
Field
string
kind
string
name
string
array
versions
array
Group
Version
Kind
meta
v1
Field
group
string
kind
string
name
string
boolean
string
string
verbs
version
string
version is the preferred version of the resource.  Empty implies the version of the containing resource

 list For subresources, this may have a different value, for example: v1 (while inside a v1beta1 version of the core resource's group)".
Group
Version
Kind
v1
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
meta
v1
Field
string
kind
string
array
versions
Group
Version
Kind
core
v1

Field
fsType
string
fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
partition
integer
readOnly
boolean
readOnly value true will force the readOnly setting in VolumeMounts. More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
volumeID
string
volumeID is unique ID of the persistent disk resource in AWS (Amazon EBS volume). More info: https://kubernetes.io/docs/concepts/storage/volumes#awselasticblockstore
Group
Version
Kind
core
v1
Affinity
Field
Describes node affinity 

scheduling rules for the pod.
Group
Version
Kind
v1
Field
array
Group
Version
Kind
v1alpha1
AllocationResult
ResourceClaimStatus [resource/v1alpha1]
Field
availableOnNodes
This field will get set by the resource driver after it has allocated the resource driver to inform the scheduler where it can schedule Pods using the ResourceClaim.  Setting this field is optional. If null, the resource is available everywhere.
resourceHandle
string
ResourceHandle contains arbitrary data returned by the driver after a successful allocation. This is opaque for Kubernetes. Driver documentation may explain to users how to interpret this data if needed.  The maximum size of this field is 16KiB. This may get increased in the future, but not reduced.
shareable
boolean
Group
Version
Kind
core
v1
Field
string
name
string
Group
Version
Kind
core
v1
Field
string
diskName
string
diskURI
string
fsType
string
kind
string
kind expected values are Shared: multiple blob disks per storage account  Dedicated: single blob disk per storage account  Managed: azure managed data disk (only in managed availability set). defaults to shared
readOnly
boolean
Group
Version
Kind
core
v1
Field
readOnly
boolean
string
string
shareName
string
Group
Version
Kind
core
v1
Field
readOnly
boolean
readOnly defaults to false (read/write). ReadOn

ly here will force the ReadOnly setting in VolumeMounts.
string
secretName is the  name of secret that contains Azure Storage Account Name and Key
shareName
string
Group
Version
Kind
v1
Field
string
kind
string
name
string
uid
string
Group
Version
Kind
v1
Field
name
string
nodeID
string
Group
Version
Kind
core
v1
Field
driver
string
fsType
string
nodeExpandSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodeExpandVolume call. This is an alpha field and requires enabling CSINodeExpandSecret feature gate. This field is optional, may be omitted if no secr

et is required. If the secret object contains more than one secret, all secrets are passed.
readOnly
boolean
object
string
Group
Version
Kind
core
v1
Field
driver
string
fsType
string
nodePublishSecretRef is a reference to the secret object containing sensitive information to pass to the CSI driver to complete the CSI NodePublishVolume and NodeUnpublishVolume calls. This field is optional, and  may be empty if no secret is required. If the secret object contains more than one secret, all secret references are passed.
readOnly
boolean
object
Group
Version
Kind
core
v1
Field
add
drop
Group
Version
Kind
core
v1
Field
monitors
monitors is Required: Monitors is a collection of Ceph monitors More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
path
string
readOnly
boolean
readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
string
secretFile is Optional: SecretFile is the path to key ring for User, default is /etc/ceph/user.secret More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
secretRef
secretRef is Optional: SecretRef is reference to the authentication secret for User, default is empty. More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
user
string
user is Optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
Group
Version
Kind
core
v1
CephFSVolumeS

ource
Field
monitors
path
string
readOnly
boolean
string
secretRef
user
string
user is optional: User is the rados user name, default is admin More info: https://examples.k8s.io/volumes/cephfs/README.md#how-to-use-it
Group
Version
Kind
v1
Field
Time
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
Field
fsType
string
fsType Filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
readOnly
boolean
readOnly is Optional: Defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
secretRef
volumeID
string
volumeID used to identify the volume in cinder. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
Group
Version
Kind
core
v1
Represents a ci

nder volume resource in Openstack. A Cinder volume must exist before mounting to a container. The volume must also be in the same region as the kubelet. Cinder volumes support ownership management and SELinux relabeling.
Field
fsType
string
fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
readOnly
boolean
readOnly defaults to false (read/write). ReadOnly here will force the ReadOnly setting in VolumeMounts. More info: https://examples.k8s.io/mysql-cinder-pd/README.md
secretRef
volumeID
string
Group
Version
Kind
core
v1
ClaimSource
ClaimSource describes a reference to a ResourceClaim.

Exactly one of these fields should be set.  Consumers of this type must treat an empty object as if it has an unknown value.
PodResourceClaim [core/v1]
Field
resourceClaimName
string
string
ResourceClaimTemplateName is the name of a ResourceClaimTemplate object in the same namespace as this pod.  The template will be used to create a new ResourceClaim, which will be bound to this pod. When this pod is deleted, the ResourceClaim will also be deleted. The name of the ResourceClaim will be &lt;pod name&gt;-&lt;resource name&gt;, where &lt;resource name&gt; is the PodResourceClaim.Name. Pod validation will reject the pod if the concatenated name is not valid for a ResourceClaim (e.g. too long).  An existing ResourceClaim with that name that is not owned by the pod will not be used for the pod to avoid using an unrelated resource by mistake. Scheduling and pod startup are then blocked until the unrelated ResourceClaim is removed.  This field is immutable and no changes will be made to the corresponding ResourceClaim by the control plane after creating the ResourceClaim.
Group
Version
Kind
core
v1
Field
integer
Group
Version
Kind
core
v1
Field
error
string
message
string
status
string
type
string
Group
Version
Kind
meta
v1
Condition
NetworkPolicyStatus [networking/v1]
Field
Time
message
string
integer
observedGeneration represents

 the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
reason
string
status
string
type
string
Group
Version
Kind
core
v1

Field
name
string
optional
boolean
Group
Version
Kind
core
v1
Field
key
string
name
string
optional
boolean
Group
Version
Kind
core
v1
ConfigMapNodeConfigSource contains the information to reference a ConfigMap as a config source for the Node. This API is deprecated since 1.22: https://git.k8s.io/enhancements/keps/sig-node/281-dynamic-kubelet-configuration
Field
string
name
string
namespace
string
string
uid
string
Group
Version
Kind
core
v1

Field
items
KeyToPath
array
name
string
optional
boolean
Group
Version
Kind
core
v1
Adapts a ConfigM

ap into a volume.

Field
integer
items
KeyToPath
array
name
string
optional
boolean
Group
Version
Kind
core
v1
Field
names
sizeBytes
integer
Group
Version
Kind
core
v1
Field
integer
hostIP
string
hostPort
integer
name
string
protocol
string
Group
Version
Kind
v2
MetricSpec [autoscaling/v2]
Field
container
string
name
string
target
Group
Version
Kind
v2
ContainerResourceMetricStatus indicates the current value of a resource metric known to Kubernetes, as specified in requests and limits, describing a single container in each pod in the current scale target (e.g. CPU or memory).  Such me

trics are built in to Kubernetes, and have special scaling options on top of those available to normal per-pod metrics using the "pods" source.
MetricStatus [autoscaling/v2]
Field
container
string
current
name
string
Group
Version
Kind
core
v1
Field
running
waiting
Group
Version
Kind
core
v1
Field
startedAt
Time
Group
Version
Kind
core
v1
Field
string
exitCode
integer
Time
message
string
reason
string
signal
integer
startedAt
Time
Group
Version
Kind
core
v1
Field
message
string
reason
string
Group
Version
Kind
v2
v1
HorizontalPodAutoscalerSpec [autoscaling/v2]
ObjectMetricSource [autoscaling/v2]
ObjectMetricStatus [autoscaling/v2]
Field
string
kind
string
Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
name
string
Group
Version
Kind
v1
Field
string
format
string
jsonPath
string
name
string
priority
integer
priority is an integer defining the relative importance of this column compared to others. Lower numbers are considered higher priority. Columns that may be omitted in limited space scenarios should be given a p

riority greater than 0.
type
string
Group
Version
Kind
v1
Field
strategy
string
webhook
Group
Version
Kind
v1
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1
Field
kind
string
listKind
string
plural
string
singular
string
Group
Version
Kind
v1
Field
array
boolean
deprecated indicates this version of the custom resource API is depr

ecated. When set to true, API requests to this version receive a warning header in the server response. Defaults to false.
string
name
string
schema
served
boolean
storage
boolean
Group
Version
Kind
v1
Field
string
string
string
Group
Version
Kind
v1
CustomResourceSubresourceStatus defines how to serve the status subresource for CustomResources. Status is represented by the `.status` JSON path inside of a CustomResource. When set, * exposes a /status subresource for the custom resource * PUT requests to the /status subresource take a custom resource object, and ignore changes to anything except the status stanza * PUT/POST/PATCH requests to the custom resource ignore changes to the status stanza
Field
Group
Version
Kind
v1
Field
scale
status
CustomRes

ourceSubresourceStatus
Group
Version
Kind
v1
Field
Group
Version
Kind
core
v1
Field
Port
integer
Group
Version
Kind
apps
v1
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
apps
v1
Field
type
string
Group
Version
Kind
meta
v1
Eviction [policy/v1]
Field
string
dryRun
integer
kind
string
boolean
string
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in the m

etadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Group
Version
Kind
apps
v1
Field
Time
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
Field
items
array
Group
Version
Kind
core
v1
Field
fieldRef
mode
integer
path
string
Group
Version
Kind
core
v1
Field
integer
items
array
Group
Version
Kind
core
v1
Field
medium
string
medium represents what type of storage medium should back this directory. The default is "" which means to use the node's default medium. Must be an empty string (default) or Memory. More info: https://kubernetes.io/docs/concepts/storage/volumes#emptydir
sizeLimit
Quantity
sizeLimit is the total amount of local storage required for this EmptyDir volume. Th

e size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info: http://kubernetes.io/docs/user-guide/volumes#emptydir
Group
Version
Kind
v1
Endpoint
Field
addresses
addresses of this endpoint. The contents of this field are interpreted according to the corresponding EndpointSlice addressType field. Consumers must handle different types of addresses in the context of their own capabilities. This must contain at least one address but no more than 100. These are all assumed to be fungible and clients may choose to only use the first element. Refer to: https://issue.k8s.io/106267
object
hints
hostname
string
nodeName
string
targetRef
zone
string
Group
Version
Kind
core
v1
Field
hostname
string
ip
string
nodeName
string
targetRef
Group
Version
Kind
v1
Field
ready
boolean
serving
boolean
boolean
Group
Version
Kind
discov

ery.k8s.io
v1
Field
forZones
ForZone
array
Group
Version
Kind
core
v1
Field
string
The application protocol for this port. This field follows standard Kubernetes label syntax. Un-prefixed names are reserved for IANA standard service names (as per RFC-6335 and https://www.iana.org/assignments/service-names). Non-standard protocols should use prefixed names such as mycompany.com/my-custom-protocol.
name
string
port
integer
protocol
string
Group
Version
Kind
core
v1

	{
	  Addresses: [{"ip": "10.10.1.1"}, {"ip": "10.10.2.2"}],
	  Ports:     [{"name": "a", "port": 8675}, {"name": "b", "port": 309}]
	}

	a: [ 10.10.1.1:8675, 10.10.2.2:8675 ],
	b: [ 10.10.1.1:309, 10.10.2.2:309 ]
Field
addresses
array
array
ports
array
Group
Version
Kind
core
v1
Field
prefix
string
secretRef
Group
Version
Kind
core
v1
EnvVar
Field
name
string
value
string
valueFrom
Group
Version
Kind
core
v1
Field
fieldRef
Selects a resource of the

 container: only resources limits and requests (limits.cpu, limits.memory, limits.ephemeral-storage, requests.cpu, requests.memory and requests.ephemeral-storage) are currently supported.
Group
Version
Kind
core
v1

Field
args
Arguments to the entrypoint. The image's CMD is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell
command
Entrypoint array. Not executed within a shell. The image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for escaping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info: https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/#running-a-command-in-a-shell
env
EnvVar
array
:
merge
:
name
envFrom
array
image
string
Container image name. More info: https://kubernetes.io/docs/concepts/containers/images
string
lifecycle
Lifecycle
Probe
name
string
ports
array
:
merge
:
Probe
resources
Optional: SecurityContext defines the security options the ephemeral container should be run with. If set, the fields of SecurityContex

t override the equivalent fields of PodSecurityContext.
Probe
stdin
boolean
stdinOnce
boolean
string
If set, the name of the container from PodSpec that this ephemeral container targets. The ephemeral container will be run in the namespaces (IPC, PID, etc) of this container. If not set then the ephemeral container uses the namespaces configured in the Pod spec.  The container runtime must implement support for this feature. If the runtime does not support namespace targeting then the result of setting this field is undefined.
string
string
tty
boolean
array
:
merge
:
array
:
merge
:
mountPath
string
Group
Version
Kind
core
v1
Field
Will be used to create a stand-alone PVC to provision the volume. The pod in which this EphemeralVolumeSource is embedded will be the owner of the PVC, i.e. the PVC will be deleted together with the pod.  The name of the PVC will be `&lt;pod name&gt;-&lt;volume name&gt;` where `&lt;volume name&gt;` is the name from the `PodSpec.Volumes` array entry. Pod validation will reject the pod if the concatenated name is not valid for a PVC (for example, too long).  An existing PVC with that name that is not owned by the pod will *not* be used for the pod to avoid using an unrelated volume by mistake. Starting the pod is then blocked until the unrelated PVC is removed. If such a pre-created PVC is meant to be used by the pod, the PVC has to updated with an owner reference to the pod once t

he pod exists. Normally this should not be necessary, but it may be useful when manually reconstructing a broken cluster.  This field is read-only and no changes will be made by Kubernetes to the PVC after it has been created.  Required, must not be nil.
Group
Version
Kind
v1
Field
count
integer
MicroTime
Group
Version
Kind
core
v1
Field
component
string
host
string
Group
Version
Kind
policy
v1
Eviction
Field
string
kind
string
metadata
Group
Version
Kind
core
v1
LifecycleHandler [core/v1]
Field
command
Group
Version
Kind
v1
Field
string
url
string
Group
Version
Kind
v2
Field
metric
target
Group
Version
Kind
v2
Field
current
current contains the current 

value for the given metric
metric
Group
Version
Kind
core
v1
Field
fsType
string
lun
integer
readOnly
boolean
wwids
Group
Version
Kind
meta
v1
FieldsV1
FieldsV1 stores a set of fields in a data structure like a Trie, in JSON format.

Field
Group
Version
Kind
core
v1
Field
driver
string
fsType
string
options
object
readOnly
boolean
secretRef
Group
Version
Kind
core
v1
Field
driver
string
fsType
string
options
object
readOnly
boolean
secretRef
secretRef is Optional: secretRef is reference to the secret object containing sensitive information to pass to the plugin scripts. This may be empty if no secret object is specified. If the secret object contains more than one secret, all secrets

 are passed to the plugin scripts.
Group
Version
Kind
core
v1
Field
string
string
Group
Version
Kind
v1beta3
v1beta2
FlowSchemaSpec [flowcontrol/v1beta3]
Field
type
string
Group
Version
Kind
v1beta3
v1beta2
FlowSchemaStatus [flowcontrol/v1beta3]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1
ForZone
Field
name
string
Group
Version
Kind
core
v1

Field
fsType
string
fsType is filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
partition
integer
partition is the partition in the volume that you want to mount. If omitted, the default is to mount by volume name. Examples: For volume /dev/sda1, you specify the partition as "1". Similarly, the volume partition for /dev/sda is "0" (or you can leave the property empty). More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
pdName
string
pdName is unique name of the PD resource in GCE. Used to identify the disk in GCE. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
readOnly
boolean
readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#gcepersistentdisk
Group
Version
Kind
core
v1
GRPCAction
Field
port
integer
service
string
Service is the name of the service to place in the gRPC HealthCheckRequest (see https://github.com/grpc/grpc/blob/master/doc/hea

lth-checking.md).  If this is not specified, the default behavior is defined by gRPC.
Group
Version
Kind
core
v1

Field
directory
string
directory is the target directory name. Must not contain or start with '..'.  If '.' is supplied, the volume directory will be the git repository.  Otherwise, if specified, the volume will contain the git repository in the subdirectory with the given name.
string
revision
string
Group
Version
Kind
core
v1
Field
endpoints
string
endpoints is the endpoint name that details Glusterfs topology. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
string
endpointsNamespace is the namespace that contains Glusterfs endpoint. If this field is empty, the EndpointNamespace defaults to the same namespace as the bound PVC. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
path
string
path is the Glusterfs volume path. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
readOnly
boolean
readOnly here will force the Glusterfs volume to be mounted with read-only permissions. Defaults to false. More info: https://examples.k8s.io/volumes/glusterfs/README.md#create-a-pod
Group
Version
Kind
core
v1
Field
endpoints
string
path
string
readOnly
boolean
Group
Version
Kind
v1beta3
v1beta2
Subject [flowcontrol/v1beta3]
Field
name
string
Group
Version
Kind
meta
v1
Field
string
version
string
Group
Version
Kind
v2
HPAScalingRules [autoscaling/v2]
Field
perio

dSeconds
integer
type
string
value
integer
Group
Version
Kind
v2
HorizontalPodAutoscalerBehavior [autoscaling/v2]
Field
policies
array
string
integer
Group
Version
Kind
core
v1
Field
host
string
array
path
string
port
scheme
string
Group
Version
Kind
core
v1
Field
name
string
value
string
Group
Version
Kind
v1
Field
backend
path
string
pathType
string
PathType determines the interpretation of the Path matching. PathType can be one of the following values: * Exact: Matches the URL path exactly. * Prefix: Matches based on a URL path prefix split by '/'. Matching is   done on a path element by element basis. A path element refers is the   list of labels in the path split by the '/' separator. A request is a   match for path p if every p is an element-wise prefix of p of the   request path. Note that if the last element of the path is a substring   of the last element in request path, it is not a match (e.g. /foo/bar   matches /foo/bar/baz, but does not match /foo/barbaz). * ImplementationSpecific: Interpretation of the Path matching is up to   the Ingress

Class. Implementations can treat this as a separate PathType   or treat it identically to Prefix or Exact path types. Implementations are required to support all path types.
Group
Version
Kind
v1
Field
paths
array
Group
Version
Kind
v2
Field
scaleDown
scaleUp
Group
Version
Kind
v2
HorizontalPodAutoscalerStatus [autoscaling/v2]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
HostAlias
Field
hostnames
ip
string
Group
Version
Kind
core
v1
Field
path
string
path of the directory on the host. If the path is a symlink, it will follow the link to the real path. More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
type
string
type for HostPath Volume Defaults to "" More info: https://kubernetes.io/docs/concepts/storage/volumes#hostpath
Group
Version
Kind
v1
IPBlock
Field
cidr
string
except
Group
Version
Kind
core
v1
ISCSIPersistentVolumeSource represents an ISCSI disk. IS

CSI volumes can only be mounted as read/write once. ISCSI volumes support ownership management and SELinux relabeling.
Field
boolean
boolean
fsType
string
fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#iscsi
string
iqn
string
string
lun
integer
portals
readOnly
boolean
secretRef
string
Group
Version
Kind
core
v1
Field
boolean
boolean
fsType
string
string
iqn
string
string
lun
integer
portals
readOnly
boolean
secretRef
string
Group
Version
Kind
v1
Field
resource
service
Ingress

ServiceBackend
Group
Version
Kind
v1
Field
apiGroup
string
kind
string
name
string
namespace
string
scope
string
Group
Version
Kind
v1
IngressLoadBalancerIngress
IngressLoadBalancerStatus [networking/v1]
Field
hostname
string
ip
string
ports
IngressPortStatus
array
Group
Version
Kind
v1
Field
ingress
array
Group
Version
Kind
v1
IngressLoadBalancerIngress [networking/v1]
Field
error
string
port
integer
protocol
string
Group
Version
Kind
v1
Field
host
string
Host is the fully qualified domain name of a network host, as defined by RFC 3986. Note the following deviations from the "host" part of the URI as defined in RFC 3986: 1. IPs are not allowed. Currently an IngressRuleValue can only apply to    the IP in the Spec of the parent Ingress. 2. The `:` delimiter is not respected because ports are not allowed. 	  Currently the port of an Ingress is implicitly :80 for http and 	  :443 for https. Both these may change in the future. Incoming requests are matched against the host before the IngressRuleValue. If the host is unspecified, the Ingress routes all traffic based on the specified IngressRuleValue.  Host can be "precise" which is a domain name without the terminating dot of a network host (e.g. "foo.bar.com") or "wildcard", which is a domain name prefixed with a single wildcard label (e.g. "*.foo.com"). The wildcard character '\*' must appear by itsel

f as the first DNS label and matches only a single label. You cannot have a wildcard label by itself (e.g. Host == "*"). Requests will be matched against the Host field in the following way: 1. If Host is precise, the request matches this rule if the http host header is equal to Host. 2. If Host is a wildcard, then the request matches this rule if the http host header is to equal to the suffix (removing the first label) of the wildcard rule.
http
Group
Version
Kind
v1
Field
name
string
port
Group
Version
Kind
v1
Field
hosts
string
Group
Version
Kind
v1
JSON
Field
Group
Version
Kind
v1
Field
$ref
string
$schema
string
allOf
array
anyOf
array
default
JSON
object
object
string
enum
JSON
array
example
JSON
boolean
boolean
format
string
format is an OpenAPI v3 format string. Unknown formats are ignored. The following formats are validated:  - bsonobjectid: a bson object ID, i.e. a 24 characters hex string - uri: an URI as parsed by Golang net/url.ParseRequestURI - email: an email address as parsed by Golang net/mail.ParseAddress - hostname: a valid representation for an Internet host name, as defined by RFC 1034, section 3.1 [RFC1034]. - ipv4: an IPv4 IP as parsed by Golang net.ParseIP - ipv6: an IPv6 IP as parsed by Golang net.ParseIP - cidr: a CIDR as parsed by Golang net.ParseCIDR - mac: a MAC address as parsed by Golang net.ParseMAC - uuid: an UUID that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?[0-9a-f]{4}-?[0-9a-f]{4}-?[0-9a-f]{12}$ - uuid3: an UUID3 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?3[0-9a-f]{3}-?[0-9a-f]{4}-?[0-9a-f]{12}$ - uuid4: an UUID4 that allows uppercase defined by the regex (?i)^[0-9a-f]{8}-?[0-9a-f]{4}-?4[0-9a-f]{3}-?[89ab][0-9a-f]{3}-?[0-9a-f]{12}$ - uuid5: an UUID5 that allows uppercase defined by the regex (?i)^[0-9

a-f]{8}-?[0-9a-f]{4}-?5[0-9a-f]{3}-?[89ab][0-9a-f]{3}-?[0-9a-f]{12}$ - isbn: an ISBN10 or ISBN13 number string like "0321751043" or "978-0321751041" - isbn10: an ISBN10 number string like "0321751043" - isbn13: an ISBN13 number string like "978-0321751041" - creditcard: a credit card number defined by the regex ^(?:4[0-9]{12}(?:[0-9]{3})?|5[1-5][0-9]{14}|6(?:011|5[0-9][0-9])[0-9]{12}|3[47][0-9]{13}|3(?:0[0-5]|[68][0-9])[0-9]{11}|(?:2131|1800|35\d{3})\d{11})$ with any non digit characters mixed in - ssn: a U.S. social security number following the regex ^\d{3}[- ]?\d{2}[- ]?\d{4}$ - hexcolor: an hexadecimal color code like "#FFFFFF: following the regex ^#?([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$ - rgbcolor: an RGB color code like rgb like "rgb(255,255,2559" - byte: base64 encoded binary data - password: any kind of string - date: a date string like "2006-01-02" as defined by full-date in RFC3339 - duration: a duration string like "22 ns" as parsed by Golang time.ParseDuration or compatible with Scala duration format - datetime: a date time string like "2014-12-15T19:30:20.000Z" as defined by date-time in RFC3339.
id
string
items
maxItems
integer
maxLength
integer
integer
maximum
number
minItems
integer
minLength
integer
integer
minimum
number
number
not
nullable
boolean
oneOf
array
pattern
string
object
object
required
title
string
type
string
boolean
boolean
boolean
string
string
x-kubernetes-map-type annotates an object to further describe its topology. This extension must only be used when type is object and may have 2 possible values:  1) `granular`:      These maps are actual maps (key-value pairs) and e

ach fields are independent      from each other (they can each be manipulated by separate actors). This is      the default behaviour for all maps. 2) `atomic`: the list is treated as a single entity, like a scalar.      Atomic maps will be entirely replaced when updated.
boolean
ValidationRule
array
:
merge
:
rule
x-kubernetes-validations describes a list of validation rules written in the CEL expression language. This field is an alpha-level. Using this field requires the feature gate `CustomResourceValidationExpressions` to be enabled.
Group
Version
Kind
v1
Field
Group
Version
Kind
v1
Field
Group
Version
Kind
batch
v1
Field
Time
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
batch
v1
Field
metadata
spec
JobSpec
Group
Version
Kind
core
v1
KeyToPath
Field
key
string
mode
integer
path
string
Group
Version
Kind
meta
v1
CSIStorageCapacity [storage/v1]
CSIStorageCapacity [sto

rage/v1beta1]
MatchResources [admissionregistration/v1alpha1]
MetricIdentifier [autoscaling/v2]
Field
array
object
Group
Version
Kind
meta
v1
Field
key
string
:
merge
:
key
operator
string
values
Group
Version
Kind
core
v1
Lifecycle
Field
postStart
preStop
PreStop is called immediately before a container is terminated due to an API request or management event such as liveness/startup probe failure, preemption, resource contention, etc. The handler is not called if the container crashes or exits. The Pod's termination grace period countdown begins before the PreStop hook is executed. Regardless of the outcome of the handler, the container will eventually terminate within the Pod's termination grace period (unless delayed by finalizers). Other management of the container blocks until the hook completes or until the termination grace period is reached. More info: https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks
Group
Version
Kind
core
v1
Field
exec
httpGet
tcpSocket
Group
Version
Kind
core
v1
Limi

tRangeSpec [core/v1]
Field
default
object
object
max
object
object
min
object
type
string
Group
Version
Kind
v1beta3
v1beta2
LimitedPriorityLevelConfiguration [flowcontrol/v1beta3]
Field
queuing
type
string
Group
Version
Kind
v1beta3
  - How are requests for this priority level limited?
  - What should be done with requests that exceed the limit?
v1beta2
PriorityLevelConfigurationSpec [flowcontrol/v1beta3]
Field
integer
`borrowingLimitPercent`, if present, configures a limit on how many seats this priority level can borrow from other priority levels. The limit is known as this level's BorrowingConcurrencyLimit (BorrowingCL) and is a limit on the total number of seats that this level may borrow at any one time. This field holds the ratio of that limit to the level's nominal concurrency limit. When this field is non-nil, it must hold a non-negative integer and the limit is calculated as follows.  BorrowingCL(i) = round( NominalCL(i) * borrowingLimitPercent(i)/100.0 )  The value of this field can be more than 100, implying that this priority level can borrow a number of seats that is greater than its own nominal concurrency limit (NominalCL). When this field is left `nil`, the limit is effectively infinite.
integer
`lendablePercent` prescribes the fraction of the level's NominalCL that can be borrowed by other priority levels. The value of this field must be between 0 and 100, inclusive, and it defaults to 0. The number of seats that other levels can borrow from this level, known as this level's LendableConcurrencyLimit (LendableCL), is defined as follows.  LendableCL(i) = round( NominalCL(i) * lendablePercent(i)/100.0 )
integer
`nominalConcurrencyShares` (NCS) contributes to the computation of the NominalConcurrencyLimit (NominalCL) of this level. This is the number of execution seats available at this priority level. This is used both for requests dispatched from this priority level as well as requests dispatched from other priority levels borrowing seats from this level. The server's concurrency limit (ServerCL) is divided among the Limited priority levels in proportion to their NCS values:  NominalCL(i)  = ceil( ServerCL * NCS(i) / sum_ncs ) sum_ncs = sum[limited priority level k] N

CS(k)  Bigger numbers mean a larger nominal concurrency limit, at the expense of every other Limited priority level. This field has a default value of 30.
Group
Version
Kind
meta
v1
ListMeta
FlowSchemaList [flowcontrol/v1beta2]
PriorityLevelConfigurationList [flowcontrol/v1beta2]
ResourceClaimList [resource/v1alpha1]
Field
continue
string
integer
string
String that identifies the server's internal version of this object that can be used by clients to determine when objects have changed. Value must be treated as opaque by clie

nts and passed unmodified back to the server. Populated by the system. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
selfLink
string
Group
Version
Kind
core
v1
Field
hostname
string
ip
string
ports
array
Group
Version
Kind
core
v1
Field
ingress
array
Group
Version
Kind
core
v1
Field
name
string
Group
Version
Kind
core
v1
Field
fsType
string
path
string
Group
Version
Kind
meta
v1
Field
string
string
fieldsV1
FieldsV1
manager
string
operation
string
string
Subresource is the name of the subresource used to update that object, or empty string if the object was updated through the main resource. The value of this field is used to distinguish between managers, even if they share the same name. For example, a status update will be distinct from a regular update using the same manager name. Note that the APIVersion field is not related to the Subresource field and it always correspon

ds to the version of the main resource.
time
Time
Group
Version
Kind
v1alpha1
ValidatingAdmissionPolicyBindingSpec [admissionregistration/v1alpha1]
ValidatingAdmissionPolicySpec [admissionregistration/v1alpha1]
Field
excludeResourceRules
NamedRuleWithOperations
array
string
matchPolicy defines how the "MatchResources" list is used to match incoming requests. Allowed values are "Exact" or "Equivalent".  - Exact: match a request only if it exactly matches a specified rule. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, but "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/v1beta1 would not be sent to the ValidatingAdmissionPolicy.  - Equivalent: match a request if modifies a resource listed in rules, even via another API group or version. For example, if deployments can be modified via apps/v1, apps/v1beta1, and extensions/v1beta1, and "rules" only included `apiGroups:["apps"], apiVersions:["v1"], resources: ["deployments"]`, a request to apps/v1beta1 or extensions/v1beta1 would be converted to apps/v1 and sent to the ValidatingAdmissionPolicy.  Defaults to "Equivalent"
NamespaceSelector decides whether to run the admission control policy on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the policy.  For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "runlevel",       "operator": "NotIn",       "values": [         "0",         "1"       ]     }   ] }  If instead you want to only run the policy on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "environment",       "operator": "In",       "values": [         "prod",         "staging"       ]     }   ] }  See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ for more examples of label selectors.  Default to the empty LabelSelector, which matches everything.
re

sourceRules
array
ResourceRules describes what operations on what resources/subresources the ValidatingAdmissionPolicy matches. The policy cares about an operation if it matches _any_ Rule.
Group
Version
Kind
v2
ExternalMetricSource [autoscaling/v2]
ExternalMetricStatus [autoscaling/v2]
PodsMetricSource [autoscaling/v2]
PodsMetricStatus [autoscaling/v2]
Field
name
string
selector
Group
Version
Kind
v2
Field
external
object
pods
resource
type
string
Group
Version
Kind
v2
Field
external
object
ObjectMet

ricStatus
pods
resource
type
string
Group
Version
Kind
v2
ContainerResourceMetricSource [autoscaling/v2]
ResourceMetricSource [autoscaling/v2]
Field
integer
Quantity
type
string
value
Quantity
Group
Version
Kind
v2
ContainerResourceMetricStatus [autoscaling/v2]
ResourceMetricStatus [autoscaling/v2]
Field
integer
Quantity
value
Quantity
Group
Version
Kind
meta
v1
MicroTime
Field
Group
Version
Kind
v1
Field
string
FailurePolicy defines how unrecognized errors from the admission endpoint are handled - allowe

d values are Ignore or Fail. Defaults to Fail.
string
name
string
string
reinvocationPolicy indicates whether this webhook should be called multiple times as part of a single admission evaluation. Allowed values are "Never" and "IfNeeded".  Never: the webhook will not be called more than once in a single admission evaluation.  IfNeeded: the webhook will be called at least one additional time as part of the admission evaluation if the object being admitted is modified by other admission plugins after the initial webhook call. Webhooks that specify this option *must* be idempotent, able to process objects they previously admitted. Note: * the number of additional invocations is not guaranteed to be exactly one. * if additional invocations result in further modifications to the object, webhooks are not guaranteed to be invoked again. * webhooks that use this option may be reordered to minimize the number of additional invocations. * to validate an object after all mutations are g

uaranteed complete, use a validating admission webhook instead.  Defaults to "Never".
rules
array
string
integer
Group
Version
Kind
core
v1
Field
path
string
path that is exported by the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
readOnly
boolean
readOnly here will force the NFS export to be mounted with read-only permissions. Defaults to false. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
server
string
server is the hostname or IP address of the NFS server. More info: https://kubernetes.io/docs/concepts/storage/volumes#nfs
Group
Version
Kind
v1alpha1
Field
apiGroups
resources
scope
string
scope specifies the scope of this rule. Valid values are "Cluster", "Namespaced", and "*" "Cluster" means that only cluster-scoped resources will match this rule. Namespace API objects are cluster-scoped. "Namespaced" means that only namespaced resources will match this rule. "*" means that there 

are no scope restrictions. Subresources match the scope of their parent resource. Default is "*".
Group
Version
Kind
core
v1
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1
Field
ports
array
to
array
Group
Version
Kind
v1
Field
from
array
ports
array
Group
Version
Kind
v1
Field
ipBlock
IPBlock
This is a label selector which selects Pods. This field follows standard label selector semantics; if present but empty, it selects all pods.  If NamespaceSelector is also set, then the NetworkPolicyPeer as a whole selects the Pods matching PodSelector in the Namespaces selected by NamespaceS

elector. Otherwise it selects the Pods matching PodSelector in the policy's own Namespace.
Group
Version
Kind
v1
Field
endPort
integer
port
protocol
string
Group
Version
Kind
core
v1
Field
address
string
type
string
Group
Version
Kind
core
v1
Field
array
Group
Version
Kind
core
v1
Field
Time
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
Field
configMap
Group
Version
Kind
core
v1
Field
active
assigned

error
string
Group
Version
Kind
core
v1
Field
Group
Version
Kind
core
v1
AllocationResult [resource/v1alpha1]
ClusterCIDRSpec [networking/v1alpha1]
ResourceClass [resource/v1alpha1]
Field
array
Group
Version
Kind
core
v1
Field
key
string
operator
string
values
An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be int

erpreted as an integer. This array is replaced during a strategic merge patch.
Group
Version
Kind
core
v1
Field
array
array
Group
Version
Kind
core
v1
Field
string
bootID
string
string
string
string
string
machineID
string
string
osImage
string
string
Group
Version
Kind
v1
Field
path
string
verb
string
Group
Version
Kind
v1beta3
v1beta2
PolicyRulesWithSubjects [flowcontrol/v1beta3]
Field
verbs
Group
Version
Kind
v1
Field
verbs
Verb is a list of kubernetes non-resource API verbs, like: get, post, put, delete, patch, head,

 options.  "*" means all.
Group
Version
Kind
core
v1
Field
string
fieldPath
string
Group
Version
Kind
meta
v1
FlowSchema [flowcontrol/v1beta2]
PriorityLevelConfiguration [flowcontrol/v1beta2]
ResourceClaim [resource/v1alpha1]
ResourceClaimTemplateSpec [resource/v1alpha1]
Field
object
Time
integer
Time
DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the s

erver when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as the finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.  Populated by the system when a graceful deletion is requested. Read-only. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
:
merge
string
GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.  If this field is specified and the generated name exists, the server will return a 409.  Applied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#idempotency
integer
labels
object
array
name
string
Name must be unique within a namespace. Is required when creating resources, although some resources may allow a client to request the generation of an appropriate name automatically. Name is primarily intended for creation idempotence and configuration definition. Cannot 

be updated. More info: http://kubernetes.io/docs/user-guide/identifiers#names
namespace
string
array
:
merge
:
uid
string
selfLink
string
uid
string
Group
Version
Kind
v2
Field
metric
target
Group
Version
Kind
v2
Field
current
metric
Group
Version
Kind
core
v1
Field
string
fieldPath
string
If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2

]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
kind
string
name
string
namespace
string
string
uid
string
Group
Version
Kind
v1
Overhead
Field
podFixed
object
Group
Version
Kind
meta
v1
Field
string
boolean
If true, AND if the owner has the "foregroundDeletion" finalizer, then the owner cannot be deleted from the key-value store until this reference is removed. See https://kubernetes.io/docs/concepts/architecture/garbage-collection/#foreground-deletion for how the garbage collector interacts with this field and enforces the foreground deletion. Defaults to false. To set this field, a user needs "delete" permission of the owner, otherwise 422 (Unprocessable Entity) will be returned.
boolean
kind
string
name
string
uid
string
Group
Version
Kind
v1alpha1
ParamKind
Field
string
kind
string
Group
Version
Kind
v1alpha1
ParamRef
Field
name
string
namespace
string
Group
Version
Kind
meta
v1
Patch
Field
Group
Version
Kind
core
v1
Field
Time
Time
message
string
reason
string
reason is a 

unique, this should be a short, machine understandable string that gives the reason for condition's last transition. If it reports "ResizeStarted" that means the underlying persistent volume is being resized.
status
string
type
string
Group
Version
Kind
core
v1
Field
metadata
spec
Group
Version
Kind
core
v1
Field
claimName
string
claimName is the name of a PersistentVolumeClaim in the same namespace as the pod using this volume. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims
readOnly
boolean
Group
Version
Kind
core
v1
Field
fsType
string
pdID
string
Group
Version
Kind
core
v1
Field
array
array
Group
Version
Kind
core
v1
Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key &lt;topologyKey&gt; matches that of any node on which a pod of the set o

f pods is running
Field
string
Group
Version
Kind
core
v1
Field
array
array
Group
Version
Kind
core
v1
Field
Time
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
Field
options
array
A list of DNS resolver options. This will be merged with the base options generated from DNSPo

licy. Duplicated entries will be removed. Resolution options given in Options will override those that appear in the base DNSPolicy.
searches
Group
Version
Kind
core
v1
Field
name
string
Required.
value
string
Group
Version
Kind
batch
v1
Field
rules
PodFailurePolicyRule
array
Group
Version
Kind
batch
v1
PodFailurePolicyOnExitCodesRequirement
PodFailurePolicyRule [batch/v1]
Field
string
operator
string
Represents the relationship between the container exit code(s) and the specified values. Containers completed with success (exit code 0) are excluded from the requirement check. Possible values are: - In: the requirement is satisfied if at least one container exit code   (might be multiple if there are multiple containers not restricted   by the 'containerName' field) is in the set of specified values. - NotIn: the requirement is satisfied if at least one container exit code   (might be multiple if there are multiple containers not restricted   by the 'containerName' field) is not in the set of specified values. Additional values are considered to be added in the future. Clients should react to an unknown operator by assuming the requirement is not satisfied.
values
Group
Version
Kind
batch
v1
PodFailurePolicyOnPodConditionsPattern
Field
status
string
type
string
Group
Version
Kind
batch
v1
PodFailurePo

licy [batch/v1]
Field
action
string
Specifies the action taken on a pod failure when the requirements are satisfied. Possible values are: - FailJob: indicates that the pod's job is marked as Failed and all   running pods are terminated. - Ignore: indicates that the counter towards the .backoffLimit is not   incremented and a replacement pod is created. - Count: indicates that the pod is handled in the default way - the   counter towards the .backoffLimit is incremented. Additional values are considered to be added in the future. Clients should react to an unknown action by skipping the rule.
array
Group
Version
Kind
core
v1
PodIP

	IP: An IP address allocated to the pod. Routable at least within the cluster.
Field
ip
string
Group
Version
Kind
core
v1
PodOS
Field
name
string
Name is the name of the operating system. The currently supported values are linux and windows. Additional value may be defined in future and can be one of: https://github.com/opencontainers/runtime-spec/blob/master/config.md#platform-specific-configuration Clients should expect to handle additional values and treat unrecognized values in this field as os: null
Group
Version
Kind
core
v1
Field
string
Group
Version
Kind
core
v1
Field
name
string
source
Group
Version
Kind
core
v1
Field
name
string
Group
Version
Kind
core
v1
Field
fsGroup
integer
A special supplemental group that applies to all containers in a pod. Some volume types allow the Kubelet to change the ownership of that volume to be owned by the pod:  1. The owning GID will be the FSGroup 2. The setgid bit is set (new files created in the volume will be owned by FSGroup) 3. The permission bits are OR'd with rw-rw----  If unset, the Kubelet will not modify the ownership and permissions of any volume. Note that this field cannot be set when spec.os.name is windows.
string
fsGroupChangePolicy defines behavior of changing ownership and permission of the volume before being exposed inside Pod. This field will only apply to volume types whi

ch support fsGroup based ownership(and permissions). It will have no effect on ephemeral volume types such as: secret, configmaps and emptydir. Valid values are "OnRootMismatch" and "Always". If not specified, "Always" is used. Note that this field cannot be set when spec.os.name is windows.
integer
The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in SecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
boolean
runAsUser
integer
The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in SecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
The SELinux context to be applied to all containers. If unspecified, the container runtime will allocate a random SELinux context for each container.  May also be set in SecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
sysctls
Sysctl
array
Group
Version
Kind
v2
Field
metric
target
Group
Version
Kind
v2
Field
current
current conta

ins the current value for the given metric
metric
Group
Version
Kind
v1
Field
apiGroups
APIGroups is the name of the APIGroup that contains the resources.  If multiple API groups are specified, any action requested against one of the enumerated resources in any API group will be allowed. "" represents the core API group and "*" represents all API groups.
resources
Resources is a list of resources this rule applies to. '\*' represents all resources.
verbs
Verbs is a list of Verbs that apply to ALL the ResourceKinds contained in this rule. '\*' represents all verbs.
Group
Version
Kind
v1beta3
v1beta2
Field
array
array
subjects
Subject
array
Group
Version
Kind
core
v1
Field
error
string
port
integer
protocol
string
Group
Version
Kind
core
v1
Field
fsType
string
fSType represents the filesystem type to mount Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs". Implicitly in

ferred to be "ext4" if unspecified.
readOnly
boolean
volumeID
string
Group
Version
Kind
meta
v1
Field
string
uid
string
Group
Version
Kind
core
v1
Field
weight
integer
Group
Version
Kind
v1beta3
v1beta2
PriorityLevelConfigurationStatus [flowcontrol/v1beta3]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1beta3
v1beta2
Field
name
string
Group
Version
Kind
core
v1
Probe
Field
exec
integer
grpc
httpGet
integer
Number of seconds after the container has started before liveness probes are initiated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes
integer
integer
tcpSocket
integer
Optional duration in seconds the pod needs to terminate gracefully upon probe failure. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. If this value

 is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this value overrides the value provided by the pod spec. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate. Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.
integer
Group
Version
Kind
core
v1
Field
integer
sources
array
Group
Version
Kind
core
resource
Quantity

``` &lt;quantity&gt;        ::= &lt;signedNumber&gt;&lt;suffix&gt;

	(Note that &lt;suffix&gt; may be empty, from the "" case in &lt;decimalSI&gt;.)

&lt;decimalExponent&gt; ::= "e" &lt;signedNumber&gt; | "E" &lt;signedNumber&gt; ```

Examples:

CSIStorageCapaci

ty [storage/v1]
MetricTarget [autoscaling/v2]
MetricValueStatus [autoscaling/v2]
Field
Group
Version
Kind
v1beta3
v1beta2
LimitResponse [flowcontrol/v1beta3]
Field
handSize
integer
integer
queues
integer
Group
Version
Kind
core
v1
Field
group
string
readOnly
boolean
registry
string
tenant
string
user
string
volume
string
Group
Version
Kind
core
v1
Field
fsType
string
fsType is the filesystem type of the volume that you want to mount. Tip: Ensure that the filesystem type is supported by the host operating system. Examples: "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if unspecified. More info: https://kubernetes.io/docs/concepts/storage/volumes#rbd
image
string
image is the rados image name. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
keyring
string
keyring is the path to key ring for RBDUser. Default is /etc/ceph/keyring. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
monitors
monitors is a collection of Ceph monitors. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
pool
string
pool is the rados pool name. Default is rbd. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
readOnly
boolean
readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info

: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
secretRef
secretRef is name of the authentication secret for RBDUser. If provided overrides keyring. Default is nil. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
user
string
user is the rados user name. Default is admin. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
Group
Version
Kind
core
v1
Field
fsType
string
image
string
keyring
string
monitors
pool
string
readOnly
boolean
readOnly here will force the ReadOnly setting in VolumeMounts. Defaults to false. More info: https://examples.k8s.io/volumes/rbd/README.md#how-to-use-it
secretRef
user
string
Group
Version
Kind
apps
v1
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
core
v1
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1
Field
group
string
name
string
namespace
string
Namespace is the namespace of the action being requested.  Currently, there is no distinction between no namespace and all namespaces "" (empty) is defaulted for LocalSubjectAccessReviews "" (empty) is empty for cluster-scoped resources "" (empty) means "all" for n

amespace scoped resources from a SubjectAccessReview or SelfSubjectAccessReview
resource
string
string
verb
string
version
string
Group
Version
Kind
core
v1
v1alpha1
ResourceRequirements [core/v1]
Field
name
string
Group
Version
Kind
v1alpha1
ResourceClaimConsumerReference
Field
apiGroup
string
name
string
resource
string
uid
string
Group
Version
Kind
v1alpha1
ResourceClaimParametersReference
ResourceClaimSpec [resource/v1alpha1]
Field
apiGroup
string
kind
string
name
string
Group
Version
Kind
v1alpha1
PodSchedulingStatus [resource/v1alpha1]
Field
name
string
unsuitableNodes
UnsuitableNodes lists nodes that the ResourceClaim cannot be allocated for.  The size of this field is limited to 128, the same as for PodSchedulingSpec.PotentialNodes. This may get increased in the future, but not reduced.
Group
Version
Kind
v1alpha1
Field
apiGroup
string
kind
string
name
string
namespace
string
Group
Version
Kind
core
v1
ResourceFieldSelector represents contain

er resources (cpu, memory) and their output format
Field
string
divisor
Quantity
resource
string
Group
Version
Kind
v2
Field
name
string
target
Group
Version
Kind
v2
Field
current
name
string
Group
Version
Kind
v1beta3
ResourcePolicyRule is a predicate that matches some resource requests, testing the request's verb and the target resource. A ResourcePolicyRule matches a resource request if and only if: (a) at least one member of verbs matches the request, (b) at least one member of apiGroups matches the request, (c) at least one member of resources matches the request, and (d) either (d1) the request does not specify a namespace (i.e., `Namespace==""`) and clusterScope is true or (d2) the request specifies a namespace and least one member of namespaces matches the request's namespace.
v1beta2
Field
apiGroups
boolean
resources
verbs
Group
Version
Kind
core
v1
ResourceRequire

ments describes the compute resource requirements.
Field
claims
array
Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.  This is an alpha field and requires enabling the DynamicResourceAllocation feature gate.  This field is immutable.
limits
object
requests
object
Group
Version
Kind
v1
Field
apiGroups
resources
verbs
Group
Version
Kind
v1
RoleRef
Field
apiGroup
string
kind
string
name
string
Group
Version
Kind
apps
v1
Field
partition
integer
Group
Version
Kind
v1
Field
apiGroups
APIGroups is the API groups t

he resources belong to. '\*' is all groups. If '\*' is present, the length of the slice must be one. Required.
resources
scope
string
Group
Version
Kind
core
v1
Field
level
string
role
string
type
string
user
string
Group
Version
Kind
v1
Scale
Field
string
kind
string
metadata
spec
ScaleSpec
status
Group
Version
Kind
core
v1
Field
fsType
string
gateway
string
string
readOnly
boolean
secretRef
secretRef references to the secret for ScaleIO user and other sensitive i

nformation. If this is not provided, Login operation will fail.
boolean
string
string
system
string
string
Group
Version
Kind
core
v1
Field
fsType
string
gateway
string
string
readOnly
boolean
secretRef
boolean
string
string
system
string
string
Group
Version
Kind
v1
Field
object
array
Group
Version
Kind
core
v1
Field
array
Group
Version
Kind
core
v1
Field
operator
string
scopeName
string
values
Group
Version
Kind
core
v1
SeccompProfile defines a pod/container's seccomp profile settings. Only o

ne profile source may be set.
Field
string
type
string
Group
Version
Kind
core
v1

The contents of the target Secret's Data field will represent the key-value pairs as environment variables.
Field
name
string
optional
boolean
Group
Version
Kind
core
v1
Field
key
string
name
string
optional
boolean
Group
Version
Kind
core
v1

Field
items
KeyToPath
array
name
string
optional
boolean
Group
Version
Kind
core
v1
Field
name
string
namespace
string
Group
Version
Kind
core
v1

Field
integer
defaultMode is Optional: mode bits used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511. YAML accepts both octal and decimal values, JSON requires decimal values for mode bi

ts. Defaults to 0644. Directories within the path are not affected by this setting. This might be in conflict with other options that affect the file mode, like fsGroup, and the result can be other mode bits set.
items
KeyToPath
array
optional
boolean
string
secretName is the name of the secret in the pod's namespace to use. More info: https://kubernetes.io/docs/concepts/storage/volumes#secret
Group
Version
Kind
core
v1
Field
boolean
boolean
procMount
string
boolean
integer
The GID to run the entrypoint of the container process. Uses runtime default if unset. May also be set in PodSecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
boolean
runAsUser
integer
The UID to run the entrypoint of the container process. Defaults to user specified in image metadata if unspecified. May also be set in PodSecurityContext.  If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
The SELinux context to be applied to the container. If unspecified, the container runtime will allocate a random SELinux context for each container.  May also be set in PodSecurityContext.  If set in bo

th SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence. Note that this field cannot be set when spec.os.name is windows.
Group
Version
Kind
meta
v1
Field
string
string
Group
Version
Kind
v1alpha1
Field
string
string
Group
Version
Kind
v1beta3
v1beta2
Field
name
string
namespace
string
Group
Version
Kind
core
v1
Field
audience
string
integer
path
string
Group
Version
Kind
v1
Field
name
string
Name is the name of the port on the Se

rvice. This is a mutually exclusive setting with "Number".
number
integer
Group
Version
Kind
core
v1
Field
string
name
string
nodePort
integer
The port on each node on which this service is exposed when type is NodePort or LoadBalancer.  Usually assigned by the system. If a value is specified, in-range, and not in use it will be used, otherwise the operation will fail.  If not specified, a port will be allocated if this Service requires one.  If this field is specified when creating a Service which does not need it, creation will fail. This field will be wiped when updating a Service to no longer need it (e.g. changing type from NodePort to ClusterIP). More info: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
port
integer
protocol
string
Group
Version
Kind
v1
Field
name
string
namespace
string
path
string
port
integer
Group
Version
Kind
core
v1
Field
clientIP
Group
Version
Kind
apps
v1
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
apps
v1
St

atefulSetSpec [apps/v1]
Field
start
integer
start is the number representing the first replica's index. It may be used to number replicas from an alternate index (eg: 1-indexed) over the default 0-indexed names, or to orchestrate progressive movement of replicas from one StatefulSet to another. If set, replica indices will be in the range:   [.spec.ordinals.start, .spec.ordinals.start + .spec.replicas). If unset, defaults to 0. Replica indices will be in the range:   [0, .spec.replicas).
Group
Version
Kind
apps
v1
Field
string
WhenDeleted specifies what happens to PVCs created from StatefulSet VolumeClaimTemplates when the StatefulSet is deleted. The default policy of `Retain` causes PVCs to not be affected by StatefulSet deletion. The `Delete` policy causes those PVCs to be deleted.
string
WhenScaled specifies what happens to PVCs created from StatefulSet VolumeClaimTemplates when the StatefulSet is scaled down. The default policy of `Retain` causes PVCs to not be affected by a scaledown. The `Delete` policy causes the associated PVCs for any excess pods above the replica count to be deleted.
Group
Version
Kind
apps
v1
Field
type
string
Group
Version
Kind
meta
v1
Status
Field
string
code
integer
details
kind
string
message
string
metadata
ListMeta
reason
string
status
string
Group
Version
Kind
meta
v1
Field
field
string
The field of the resource that has caused

 this error, as named by its JSON serialization. May include dot and postfix notation for nested attributes. Arrays are zero-indexed.  Fields may appear more than once in an array of causes due to fields having multiple errors. Optional.  Examples:   "name" - the field "name" on the current resource   "items[0].name" - the field "name" on the first array entry in "items"
message
string
reason
string
Group
Version
Kind
meta
v1
Field
causes
array
group
string
kind
string
name
string
integer
uid
string
Group
Version
Kind
core
v1
Field
fsType
string
readOnly
boolean
secretRef
secretRef specifies the secret to use for obtaining the StorageOS API credentials.  If not specified, default values will be attempted.
string
volumeName is the human-readable name of the StorageOS volume.  Volume names are only unique within a namespace.
string
volumeNamespace specifies the scope of the volume within StorageOS.  If no namespace is specified then the Pod's namespace will be used.  This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.
Group
Version
Kind
core
v1
Field
fsType
string
readOnly
boolean
secretRef
secretRef specifies the secret to use for

 obtaining the StorageOS API credentials.  If not specified, default values will be attempted.
string
string
Group
Version
Kind
v1alpha1
Field
Time
message
string
integer
reason
string
status
string
type
string
Group
Version
Kind
v1beta3
Subject
v1beta2
v1
Field
group
`group` matches based on user group name.
kind
string
`kind` indicates which one of the other fields is non-empty. Required
`serviceAccount` matches ServiceAccounts.
user
`user` matches based on username.
Group
Version
Kind
v1
Field
string
boolean
array
array
Group
Version
Kind
core
v1
Sysctl
Field
name
string
value
string
Group
Version
Kind
core
v1
Field
host
string
Optional: H

ost name to connect to, defaults to the pod IP.
port
Group
Version
Kind
core
v1
Taint
Field
effect
string
key
string
timeAdded
Time
value
string
Group
Version
Kind
meta
v1
Time
FlowSchemaCondition [flowcontrol/v1beta3]
FlowSchemaCondition [flowcontrol/v1beta2]
HorizontalPodAutoscalerCondition [autoscaling/v2]
PriorityLevelConfigurationCondition [flowcontrol/v1beta3]
PriorityLevelConfigurationCondition [flowcontrol/v1beta2]
Field
Group
Version
Kind
core
v1
Field
effect
string
key
string
operator
string
integer
value
string
Group
Version
Kind
core
v1
Field
key
string
values
An array of string values. One value must match the label to be selected. Each ent

ry in Values is ORed.
Group
Version
Kind
core
v1
Field
array
Group
Version
Kind
core
v1
Field
maxSkew
integer
MaxSkew describes the degree to which pods may be unevenly distributed. When `whenUnsatisfiable=DoNotSchedule`, it is the maximum permitted difference between the number of matching pods in the target topology and the global minimum. The global minimum is the minimum number of matching pods in an eligible domain or zero if the number of eligible domains is less than MinDomains. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 2/2/1: In this case, the global minimum is 1. | zone1 | zone2 | zone3 | |  P P  |  P P  |   P   | - if MaxSkew is 1, incoming pod can only be scheduled to zone3 to become 2/2/2; scheduling it onto zone1(zone2) would make the ActualSkew(3-1) on zone1(zone2) violate MaxSkew(1). - if MaxSkew is 2, incoming pod can be scheduled onto any zone. When `whenUnsatisfiable=ScheduleAnyway`, it is used to give higher precedence to topologies that satisfy it. It's a required field. Default value is 1 and 0 is not allowed.
integer
MinDomains indicates a minimum number of eligible domains. When the number of eligible domains with matching topology keys is less than minDomains, Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. And when the number of eligible domains with matching topology keys equals or greater than minDomains, this value has no effect on scheduling. As a result, when the number of eligible domains is less than minDomains, scheduler won't schedule more than maxSkew Pods to those domains. If value is nil, the constraint behaves as if MinDomains is equal to 1. Valid values are integers greater than 0. When value is not nil, WhenUnsatisfiable must be DoNotSchedule.  For example, in a 3-zone cluster, MaxSkew is set to 2, MinDomains is set to 5 and pods with the same labelSelector spread as 2/2/2: | zone1 | zone2 | zone3 | |  P P  |  P P  |  P P  | The number of domains is less than 5(MinDomains), so "global minimum" is treated as 0. In this situation, new pod with the same labelSelector cannot be scheduled, because computed skew will be 3(3 - 0) if new Pod is scheduled to any of the three zones, it will violate MaxSkew.  This is a beta field and requires the MinDomainsInPodTopologySpread feature gate to be enabled (enabled by default).
string
NodeAffinityPolicy indicates how we will treat Pod's nodeAffinity/nodeSelector when calculating pod topology spread skew. Options are: - Honor: only nodes matching nodeAffinity/nodeSelector are included in the calculations. -

 Ignore: nodeAffinity/nodeSelector are ignored. All nodes are included in the calculations.  If this value is nil, the behavior is equivalent to the Honor policy. This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag.
string
NodeTaintsPolicy indicates how we will treat node taints when calculating pod topology spread skew. Options are: - Honor: nodes without taints, along with tainted nodes for which the incoming pod has a toleration, are included. - Ignore: node taints are ignored. All nodes are included.  If this value is nil, the behavior is equivalent to the Ignore policy. This is a beta-level feature default enabled by the NodeInclusionPolicyInPodTopologySpread feature flag.
string
string
WhenUnsatisfiable indicates how to deal with a pod if it doesn't satisfy the spread constraint. - DoNotSchedule (default) tells the scheduler not to schedule it. - ScheduleAnyway tells the scheduler to schedule the pod in any location,   but giving higher precedence to topologies that would help reduce the   skew. A constraint is considered "Unsatisfiable" for an incoming pod if and only if every possible node assignment for that pod would violate "MaxSkew" on some topology. For example, in a 3-zone cluster, MaxSkew is set to 1, and pods with the same labelSelector spread as 3/1/1: | zone1 | zone2 | zone3 | | P P P |   P   |   P   | If WhenUnsatisfiable is set to DoNotSchedule, incoming pod can only be scheduled to zone2(zone3) to become 3/2/1(3/1/2) as ActualSkew(2-1) on zone2(zone3) satisfies MaxSkew(1). In other words, the cluster can still be imbalanced, but scheduler won't make it *more* imbalanced. It's a required field.
Group
Version
Kind
core
v1
Field
apiGroup
string
kind
string
name
string
Group
Version
Kind
core
v1
Field
apiGroup
string
kind
string
name
string
namespace
string
Group
Version
Kind
batch
v1
UncountedTerminatedPods holds UID

s of Pods that have terminated but haven't been accounted in Job status counters.
Field
failed
succeeded
Group
Version
Kind
v1
UserInfo
SelfSubjectReviewStatus [authentication/v1alpha1]
Field
extra
object
groups
uid
string
username
string
Group
Version
Kind
v1beta3
v1beta2
Field
name
string
Group
Version
Kind
v1
Field
string
string
name
string
NamespaceSelector decides whether to run the webhook on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the webhook.  For example, to run the webhook on any objects whose namespace is not associated with "runlevel" of "0" or "1";  you will set the selector as fo

llows: "namespaceSelector": {   "matchExpressions": [     {       "key": "runlevel",       "operator": "NotIn",       "values": [         "0",         "1"       ]     }   ] }  If instead you want to only run the webhook on any objects whose namespace is associated with the "environment" of "prod" or "staging"; you will set the selector as follows: "namespaceSelector": {   "matchExpressions": [     {       "key": "environment",       "operator": "In",       "values": [         "prod",         "staging"       ]     }   ] }  See https://kubernetes.io/docs/concepts/overview/working-with-objects/labels for more examples of label selectors.  Default to the empty LabelSelector, which matches everything.
rules
array
string
integer
Group
Version
Kind
v1alpha1
Field
string
Expression represents the expression which will be evaluated by CEL. ref: https://github.com/google/cel-spec CEL expressions have access to the contents of the Admission request/response, organized into CEL variables as well as some other useful variables:  'object' - The object from the incoming request. The value is null for DELETE requests. 'oldObject' - The existing object. The value is null for CREATE requests. 'request' - Attributes of the admission request([ref](/pkg/apis/admission/types.go#AdmissionRequest)). 'params' - Parameter resource referred to by the policy binding being evaluated. Only populated if the policy has a ParamKind.  The `apiVersion`, `kind`, `metadata.name` and `metadata.generateName` are always accessible from the root of the object. No other metadata properties are accessible.  Only property names of the form `[a-zA-Z_.-/][a-zA-Z0-9_.-/]*` are accessible. Accessible property names are escaped according to the following rules when accessed in the ex

pression: - '__' escapes to '__underscores__' - '.' escapes to '__dot__' - '-' escapes to '__dash__' - '/' escapes to '__slash__' - Property names that exactly match a CEL RESERVED keyword escape to '__{keyword}__'. The keywords are: 	  "true", "false", "null", "in", "as", "break", "const", "continue", "else", "for", "function", "if", 	  "import", "let", "loop", "package", "namespace", "return". Examples:   - Expression accessing a property named "namespace": {"Expression": "object.__namespace__ &gt; 0"}   - Expression accessing a property named "x-prop": {"Expression": "object.x__dash__prop &gt; 0"}   - Expression accessing a property named "redact__d": {"Expression": "object.redact__underscores__d &gt; 0"}  Equality on arrays with list type of 'set' or 'map' ignores element order, i.e. [1, 2] == [2, 1]. Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:   - 'set': `X + Y` performs a union where the array positions of all elements in `X` are preserved and     non-intersecting elements in `Y` are appended, retaining their partial order.   - 'map': `X + Y` performs a merge where the array positions of all keys in `X` are preserved but the values     are overwritten by values in `Y` when the key sets of `X` and `Y` intersect. Elements in `Y` with     non-intersecting keys are appended, retaining their partial order. Required.
message
string
reason
string
Group
Version
Kind
v1
Field
message
string
rule
string
Rule represents the expression which will be evaluated by CEL. ref: https://github.com/google/cel-spec The Rule is scoped to the location of the x-kubernetes-validations extension in the schema. The `self` variable in the CEL expression is bound to the scoped value. Example: - Rule scoped to the root of a resource with a status subresource: {"rule": "self.status.actual &lt;= self.spec.maxDesired"}  If the Rule is scoped to an object with properties, the accessible properties of the object are field selectable via `self.field` and field presence can be checked via `has(self.field)`. Null valued fields are treated as absent fields in CEL expressions. If the Rule is scoped to an object with additionalProperties (i.e. a map) the value of the map are accessible via `self[mapKey]`, map containment can be checked via `mapKey in self` and all entries of the map are accessible via CEL macros and functions such as `self.all(...)`. If the Rule is scoped to an array, the elements of the array are accessible via `self[i]` and also by macros and functions. If the Rule is scoped to a scalar, `self` is bound to the scalar value. Examples: - Rule scoped to a map of objects: {"rule": "self.components[

'Widget'].priority &lt; 10"} - Rule scoped to a list of integers: {"rule": "self.values.all(value, value &gt;= 0 &amp;&amp; value &lt; 100)"} - Rule scoped to a string value: {"rule": "self.startsWith('kube')"}  The `apiVersion`, `kind`, `metadata.name` and `metadata.generateName` are always accessible from the root of the object and from any x-kubernetes-embedded-resource annotated objects. No other metadata properties are accessible.  Unknown data preserved in custom resources via x-kubernetes-preserve-unknown-fields is not accessible in CEL expressions. This includes: - Unknown field values that are preserved by object schemas with x-kubernetes-preserve-unknown-fields. - Object properties where the property schema is of an "unknown type". An "unknown type" is recursively defined as:   - A schema with no type and x-kubernetes-preserve-unknown-fields set to true   - An array where the items schema is of an "unknown type"   - An object where the additionalProperties schema is of an "unknown type"  Only property names of the form `[a-zA-Z_.-/][a-zA-Z0-9_.-/]*` are accessible. Accessible property names are escaped according to the following rules when accessed in the expression: - '__' escapes to '__underscores__' - '.' escapes to '__dot__' - '-' escapes to '__dash__' - '/' escapes to '__slash__' - Property names that exactly match a CEL RESERVED keyword escape to '__{keyword}__'. The keywords are: 	  "true", "false", "null", "in", "as", "break", "const", "continue", "else", "for", "function", "if", 	  "import", "let", "loop", "package", "namespace", "return". Examples:   - Rule accessing a property named "namespace": {"rule": "self.__namespace__ &gt; 0"}   - Rule accessing a property named "x-prop": {"rule": "self.x__dash__prop &gt; 0"}   - Rule accessing a property named "redact__d": {"rule": "self.redact__underscores__d &gt; 0"}  Equality on arrays with x-kubernetes-list-type of 'set' or 'map' ignores element order, i.e. [1, 2] == [2, 1]. Concatenation on arrays with x-kubernetes-list-type use the semantics of the list type:   - 'set': `X + Y` performs a union where the array positions of all elements in `X` are preserved and     non-intersecting elements in `Y` are appended, retaining their partial order.   - 'map': `X + Y` performs a merge where the array positions of all keys in `X` are preserved but the values     are overwritten by values in `Y` when the key sets of `X` and `Y` intersect. Elements in `Y` with     non-intersecting keys are appended, retaining their partial order.
Group
Version
Kind
v1
Field
string
Group
Version
Kind
core
v1
Field
string
name
string
Group
Version
Kind
v1
Field
me

ssage
string
time
Time
Group
Version
Kind
core
v1
Field
mountPath
string
string
name
string
readOnly
boolean
subPath
string
string
Group
Version
Kind
core
v1
Field
required
Group
Version
Kind
v1
Field
count
integer
Group
Version
Kind
core
v1
Field
configMap
secret
Group
Version
Kind
core
v1
Field
fsType
string
string
string
string
Group
Version
Kind
meta
v1
Field
object
type
string
Group
Ver

sion
Kind
v1
Field
caBundle
string
service
url
string
Group
Version
Kind
v1
Field
conversionReviewVersions is an ordered list of preferred `ConversionReview` versions the Webhook expects. The API server will use the first version in the list which it supports. If none of the versions specified in this list are supported by API server, conversion will fail for the custom resource. If a persisted Webhook configuration specifies allowed versions and does not include any versions known to the API Server, calls to the webhook will fail.
Group
Version
Kind
core
v1
Field
weight
integer
Group
Version
Kind
core
v1
Field
string
string
boolean
HostProcess determines if a container should be run as a 'Host Process' container. This field is alpha-level and will only be honored by components that enable the WindowsHostProcessContainers feature flag. Setting this field without the feature flag will result in errors 

when validating the Pod. All of a Pod's containers must have the same effective HostProcess value (it is not allowed to have a mix of HostProcess containers and non-HostProcess containers).  In addition, if HostProcess is true then HostNetwork must also be set to true.
string
Group
Version
Kind
v1beta1
v1
Field
string
capacity
Quantity
kind
string
Quantity
metadata
string
Field
string
items
array
kind
string

metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder e

ncounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value

 must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whethe

r more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value

 is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this va

lue is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
GET /apis/storage.k8s.io/v1beta1/watch/namespaces/{namespace}/csistoragecapacities/{name}
Parameter
name
name of the CSIStorageCapa

city
namespace
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
pretty
resourceVersion sets a constraint on what resource versions a request may be served from. See https://

kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks

 of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
v1
v2
Field
string
kind
string
name
string
Group
Version
Kind
v1
Field
string
name
string
port
integer
protocol
string
Group
Version
Kind
core
v1
Event
Field
action
string
string
count
integer
eventTime
MicroTime
Time
kind
string
Time
message
string

metadata
reason
string
related
string
string
series
source
type
string
Field
string
items
Event
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
202
Event
Accepted
Patch
Parameter
name
name of the

 Event
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
Event
OK
201
Event
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValida

tion` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Event
Response
Code
200
Event
OK
201
Event
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
A s

elector to restrict the list of returned objects by their fields. Defaults to everything.
limit
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
Event
OK
List
Parameter
namespace

Parameter
pretty
continue
limit
watch

Response
Code
200
EventList
OK
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set S

ee https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
EventList
OK
Watch
Parameter
name
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present a

t the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a si

ngle list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose

 not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
Group
Version
Kind
core
v1
Field
count
integer
MicroTime
Group
Version
Kind
v1beta2
v1beta3
FlowSchemaSpec [flowcontrol/v1beta2]
Field
type
string
Group
Version
Kind
v1beta2
v1beta3
Field
string
kind
string
metadata
spec
status
FlowSchemaSpec v1beta2 flowcontrol
Field
`distinguisherMethod` defines how to compute the flow distinguisher for 

requests that match this schema. `nil` specifies that the distinguisher is disabled and thus will always be the empty string.
integer
rules
array
FlowSchemaStatus v1beta2 flowcontrol
Field
array
FlowSchemaList v1beta2 flowcontrol
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate

 fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown fiel

d that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas
Parameter
pretty
continue
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized dryRun directive will result in

 an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/flowcont

rol.apiserver.k8s.io/v1beta2/flowschemas
Parameter
pretty
continue
limit

watch
Response
Code
200
OK
Watch
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/flowschemas/{name}
Parameter
name
Parameter
continue
limit
pretty

watch
Response
Code
200
OK
Watch List
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/flowschemas
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possi

ble objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
pretty
watch
Response
Code
200
OK
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK

/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta2
v1beta3
FlowSchemaStatus [flowcontrol/v1beta2]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1beta2
v1beta3
Subject [flowcontrol/v1beta2]
Field
name
string
Group
Version
Kind
v1
v2
Field
string
kind
string
metadata
Standard object metadata. Mor

e info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
spec
status
Field
integer
integer
integer
Field
integer
integer
integer
Time
integer
Field
string
items
array
kind
string
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun

Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
If 'true',

dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
Parameter
namespace

ects
Parameter
pretty
continue
dryRun
limit
Whether and how garbage collection will be performed. Either this field or OrphanDependents may be set, but not both. The default policy is decided by the existing finalizer set in

 the metadata.finalizers and the resource-specific default policy. Acceptable values are: 'Orphan' - orphan the dependents; 'Background' - allow the garbage collector to delete the dependents in the background; 'Foreground' - a cascading policy that deletes all dependents in the foreground.
Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amoun

t of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is returned.
watch
Response
Code
200
OK
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Watch
Parameter
name
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is incon

sistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch List
Parameter
namespace
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no 

longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, 

this field is ignored.
continue
limit
pretty
watch
Response
Code
200
OK
Parameter
name
name of th

e HorizontalPodAutoscaler
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fie

lds that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta2
v1beta3
LimitedPriorityLevelConfiguration [flowcontrol/v1beta2]
Field
queuing
type
string
Group
Version
Kind
v1beta2
v1beta3
PriorityLevelConfigurationSpec [flowcontrol/v1beta2]
Field
integer
`assuredConcurrencyShares` (ACS) configures the execution limit, which is a limit on the number of requests of this priority level that may be exeucting at a given time.  ACS must be a positive number. The server's concurrency limit (SCL) is divided among the concurrency-controlled priority levels in proportion to their assured concurrency shares. This produces the assured concurrency value (ACV) --- the number of requests that may be executing at a time --- for each such priority level:              ACV(l) = ceil( SCL * ACS(l) / ( sum[priority levels k] ACS(k) ) )  bigger numbers of ACS mean more reserved concurrent requests (at the expense of every other PL). This field has a default value of 30.
integer
integer
`lendablePercent` prescribes the fraction of the level's NominalCL that can be borrowed by other priority levels. The value of this field must be between 0 and 100, inclusive, and it defaults to 0. The number of seats that other levels can borrow from this level, known as this level's LendableConcurrencyLimit (LendableCL), is defined as follows.  LendableCL(i) = round( NominalCL(i)

 * lendablePercent(i)/100.0 )
Group
Version
Kind
v1beta2
v1beta3
PolicyRulesWithSubjects [flowcontrol/v1beta2]
Field
verbs
Group
Version
Kind
v1beta2
v1beta3
Field
array
array
subjects
Subject
array
Group
Version
Kind
v1beta2
v1beta3
Field
string
kind
string
metadata
spec
status
PriorityLevelConfigurationSpec v1beta2 flowcontrol
Field
limited
type
string
`type` indicates whether this priority level is subject to limitation on request execution.  A value of `"Exempt"` means that requests of this priority level are not subject to a limit (and thus are never queued) and do not detract fr

om the capacity made available to other priority levels.  A value of `"Limited"` means that (a) requests of this priority level _are_ subject to limits and (b) some of the server's limited capacity is made available exclusively to this priority level. Required.
PriorityLevelConfigurationStatus v1beta2 flowcontrol
Field
array
PriorityLevelConfigurationList v1beta2 flowcontrol
Field
string
items
array
kind
string
metadata
ListMeta
Create
POST /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
When present, indicates that modifications should not be persisted. An invalid or unrecognized d

ryRun directive will result in an error response and no further processing of the request. Valid values are: - All: all dry run stages will be processed
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server 

will contain all unknown and duplicate fields encountered.
Parameter
body
Response
Code
200
OK
201
Created
Delete
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
Status
OK
202
Status
Accepted
DELETE /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations
Parameter
pretty
continue
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. Th

e value zero indicates delete immediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
Parameter
body
Response
Code
200
Status
OK
Read
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}
Parameter
name
Parameter
pretty
Response
Code
200
OK
List
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations
Parameter
pretty
allowWatchBookmarks requests watch events with type "BOOKMARK". Servers that do not implement bookm

arks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
watch
Response
Code
200
OK
Watch
watch changes to an obje

ct of kind PriorityLevelConfiguration. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/prioritylevelconfigurations/{name}
Parameter
name
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It is highly recommen

ded that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
Watch List
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/watch/prioritylevelconfigurations
Parameter
continue
limit
limit is a maximum number of responses to return for a list call. If more items exist, the server will set the `continue` field on the list metadata to a value that can be used with the same initial query to retrieve the next set of results. Setting a limit may return fewer than the requested amount of items (up to zero items) in the event all requested objects are filtered out and clients should only use the presence of the continue field to determine whether more results are available. Servers may choose not to support the limit argument and will return all of the available results. If limit is specified and the continue field is empty, clients may assume that no more results are available. This field is not supported if watch is true.  The server guarantees that the objects returned when using continue will be identical to issuing a single list call without a limit - that is, no objects created, modified, or deleted after the first request is issued will be included in any subsequent continued requests. This is sometimes referred to as a consistent snapshot, and ensures that a client that is using limit to receive smaller chunks of a very large result can ensure they see all possible objects. If objects are updated during a chunked list the version of the object that was present at the time the first list result was calculated is ret

urned.
pretty
watch
Response
Code
200
OK
PATCH /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}/status
Parameter
name
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
GET /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations/{name}/status
Parameter
name
Parameter
pretty
Response
Code
200
OK
PUT /apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelc

onfigurations/{name}/status
Parameter
name
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta2
v1beta3
PriorityLevelConfigurationStatus [flowcontrol/v1beta2]
Field
Time
message
string
reason
string
status
string
type
string
Group
Version
Kind
v1beta2
v1beta3
Field
name
string
Group
Version
Kind
v1beta2
v1beta3
LimitResponse [flowcontrol/v1beta2]
Field
handSize
integer
`handSize` is a small positive number that configures the shuffle sharding of requests into queues.  When enqueuing a request at this priority level the request's flow identifier (a string pair) is hashed and the hash value is used to shuffle the list of queues and deal a hand of the size specified here.  The request is put into one of the shortest queues in that hand. `handSize` must be no larger than `queues`, and should be significantly smaller (so that a few heavy flows do not saturate most of the queues).  See the user-facing docume

ntation for more extensive guidance on setting this field.  This field has a default value of 8.
integer
queues
integer
Group
Version
Kind
v1alpha1
v1
Field
string
kind
string
metadata
spec
status
ResourceClaimSpec v1alpha1 resource
Field
string
ParametersRef references a separate object with arbitrary parameters that will be used by the driver when allocating a resource for the claim.  The object must be in the same namespace as the ResourceClaim.
string
ResourceClaimStatus v1alpha1 resource
Field
boolean
DeallocationRequested indicates that a ResourceClaim is to be deallocated.  The driver then must deallocate this claim and reset the field together with clearing the Allocation field.  While DeallocationRequested is set, no new consumers may be added to ReservedFor.
string
array
ReservedFor indicates which entities are currently allowed to use the claim. A Pod which references a ResourceClaim which is not reserved for that Pod will not be started.  There can be at most 32 such reservations. This may get increased in the future, but not reduced.
ResourceClaimList v1alpha1 resource
Field
string
items
array
kind
string
Kind is a string value 

representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
ListMeta
Create
Parameter
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
202
Accepted
Patch
Parameter
name
namespace
Parameter
pretty
dryRun
fieldValidation instructs the server on how to handle objects in the request (POST/PUT/PATCH) containing unknown or duplicate fields, provided that the `ServerSideFieldValidation` feature gate is also enabled. Valid values are: - Ignore: This will ignore any unknown fields that are silently dropped from the object, and will ignore all but the last duplicate field that the decoder encounters. This is the default behavior prior to v1.23 and is the default behavior when 

the `ServerSideFieldValidation` feature gate is disabled. - Warn: This will send a warning via the standard warning response header for each unknown field that is dropped from the object, and for each duplicate field that is encountered. The request will still succeed if there are no other errors, and will only persist the last of any duplicate fields. This is the default when the `ServerSideFieldValidation` feature gate is enabled. - Strict: This will fail the request with a BadRequest error if any unknown fields would be dropped from the object, or if any duplicate fields are present. The error returned from the server will contain all unknown and duplicate fields encountered.
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Replace
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Delete
Parameter
name
namespace
Parameter
pretty
dryRun
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period for the specified 

type will be used. Defaults to a per object value if not specified. zero means delete immediately.
Parameter
body
Response
Code
200
OK
202
Accepted
Parameter
namespace
Parameter
pretty
continue
dryRun
limit

Parameter
body
Response
Code
200
Status
OK
Read
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
List
Parameter
namespace
Parameter
pretty
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 Resour

ceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
watch
Response
Code
200
OK
Parameter
continue
The continue option should be set when retrieving more results from the server. Since this value is server defined, clients may only use the continue value from a previous query result with identical query parameters (except for the value of continue) and the server may 

reject a continue value it does not recognize. If the specified continue value is no longer valid whether due to expiration (generally five to fifteen minutes) or a configuration change on the server, the server will respond with a 410 ResourceExpired error together with a continue token. If the client needs a consistent list, it must restart their list without the continue field. Otherwise, the client may send another list request with the token received with the 410 error, the server will respond with a list starting from the next key, but from the latest snapshot, which is inconsistent from the previous list results - objects that are created, modified, or deleted after the first list request will be included in the response, as long as their keys are after the "next key".  This field is not supported when watch is true. Clients may start a watch from the last resourceVersion value returned by the server and not miss any modifications.
limit
pretty
watch
Response
Code
200
OK
Watch
watch changes to an object of kind ResourceClaim. deprecated: use the 'watch' parameter with a list operation instead, filtered to a single item with the 'fieldSelector' parameter.
GET /apis/resource.k8s.io/v1alpha1/watch/namespaces/{namespace}/resourceclaims/{name}
Parameter
name
namespace
Parameter
allowWatchBookmarks requests watch events with type "BOOKMARK". 

Servers that do not implement bookmarks may ignore this flag and bookmarks are sent at the server's discretion. Clients should not assume bookmarks are returned at any specific interval, nor may they assume the server will send any BOOKMARK event during a session. If this is not a watch, this field is ignored.
continue
limit
pretty
watch
Respon

se
Code
200
OK
Watch List
watch individual changes to a list of ResourceClaim. deprecated: use the 'watch' parameter with a list operation instead.
GET /apis/resource.k8s.io/v1alpha1/watch/namespaces/{namespace}/resourceclaims
Parameter
namespace
Parameter
continue
limit
pretty
resourceVersionMatch determines how resourceVersion is applied to list calls. It i

s highly recommended that resourceVersionMatch be set for list calls where resourceVersion is set See https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions for details.  Defaults to unset
watch
Response
Code
200
OK
GET /apis/resource.k8s.io/v1alpha1/watch/resourceclaims
Parameter
continue
limit

pretty
watch
Response
Code
200
OK
Parameter
name
namespace
Parameter
pretty
dryRun
force
Parameter
body
Patch
Response
Code
200
OK
201
Created
Parameter
name
namespace
Parameter
pretty
Response
Code
200
OK
PUT /apis/resource.k8s.io/v1alpha1/na

mespaces/{namespace}/resourceclaims/{name}/status
Parameter
name
namespace
Parameter
pretty
dryRun
Parameter
body
Response
Code
200
OK
201
Created
Group
Version
Kind
v1beta2
v1beta3
Field
apiGroups
boolean
resources
verbs
Group
Version
Kind
v1beta2
v1beta3
Field
name
string
`name` is the name of matching ServiceAccount object

s, or "*" to match regardless of name. Required.
namespace
string
Group
Version
Kind
v1
Field
name
string
namespace
string
path
string
port
integer
Group
Version
Kind
v1
Field
name
string
namespace
string
port
integer
Group
Version
Kind
v1
Subject
v1beta3
v1beta2
Field
apiGroup
string
kind
string
name
string
namespace
string
Group
Version
Kind
v1beta2
Subject
v1beta3
v1
Field
group
kind
string
user
Group
Version
Kind
v1
Field
audience
string
integer
Group
Version
Kind
v1beta2
v1beta3
Field
name
string
Group
Version
Kind
v1
Field
caBundle
string
service
url
string
url gives the location of the webhook, in standard URL form (`scheme://host:port/path`). Exactly one of `url` or `service` must be sp

ecified.  The `host` should not refer to a service running in the cluster; use the `service` field instead. The host might be resolved via external DNS in some apiservers (e.g., `kube-apiserver` cannot resolve in-cluster DNS as that would be a layering violation). `host` may also be an IP address.  Please note that using `localhost` or `127.0.0.1` as a `host` is risky unless you take great care to run this webhook on all hosts which run an apiserver which might need to make calls to this webhook. Such installs are likely to be non-portable, i.e., not easy to turn up in a new cluster.  The scheme must be "https"; the URL must begin with "https://".  A path is optional, and if present may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier.  Attempting to use a user or basic auth e.g. "user:password@" is not allowed. Fragments ("#...") and query parameters ("?...") are not allowed, either.

Example: Deploying Cassandra with a StatefulSet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tutorials
state
).
.
Note:
node
tutorial, the Pods that belong to the StatefulSet are Cassandra nodes and are members
ring
Nodes
.
seed list
Service
.
Use a
Pods
.
minikube
Pods
,
Services
, and
.
Caution:
Minikube
5120
--cpus
=
4
Service
Pods
:
v1
kind
:

Service
metadata
:
labels
:
app
:
cassandra
name
:
cassandra
spec
:
clusterIP
:
None
ports
:
-
port
:
9042
selector
:
app
:
cassandra
file:
cassandra
Note:
:
apps/v1
kind
:
metadata
:
name
:
cassandra
labels
:
app
:
cassandra
spec
:
:
cassandra
replicas
:
3
selector
:
:
app
:
cassandra
template
:
metadata
:
labels
:
app
:
cassandra
spec
:
:
1800
:
-
name
:
cassandra
image
:
:
Always
ports
:
-
:
7000
name
:
-
:
7001
name
:
-
:
7199
name
:
jmx
-
:
9042
name
:
cql
resources
:
limits
:
cpu
:
"500m"
memory
:
1Gi
requests
:
cpu
:
"500m"
memory
:
1Gi
:
:
add
:
- IPC_LOCK
lifecycle
:
preStop
:
exec
:
command
:
- /bin/sh
- -c
env
:
-
name
:
value
:
512M
-
name
:
value
:
100M
-
name
:
value
:
-
name
:
value
:
"K8Demo"
-
name
:
value
:
-
name
:
value
:
-
name
:
POD_IP
valueFrom
:
fieldRef
:
fieldPath
:
:
exec
:
command
:
- -c
- /ready-probe.sh
:
15
:
5
:
-
name
:
mountPath
:
:
-
metadata
:
name
:
spec
:
:
[
]
:
fast
resources
:
requests
:
storage
:
1Gi
---
kind
:
:
metadata
:
name
:
fast
:
:
type
:
pd-ssd
file:
The
=
0
1m

0
8s
nodetool
kubectl
exec
Use
replicas
field.
#
:
apps/v1
kind
:
metadata
:
:
:
1
labels
:
app
:
cassandra
name
:
cassandra
namespace
:
default
:
"323"
uid
:
spec
:
replicas
:
3
Deleting or scaling a StatefulSet down does not delete the volumes associated with the StatefulSet.
Warning:
grace
=
$(
=
jsonpath
=
)
\
app
=
cassandra
\
echo
"Sleeping
${
grace
}
seconds"
2
\
sleep
$grace
\
app
=
cassandra
app
=
cassandra
.
.
32
0.0.0.0
.
Feedback
Yes
No
.

or
.
Last modified April 30, 2022 at 9:21 AM PST:
[en] modify link about debug (e6276724bb)
Home
Blog
Training
Partners
Community
CC BY 4.0

Interactive Tutorial - Scaling Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Network Plug

ins
Tasks
Monitoring, Logging, and D

ebugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configuring a

 Java Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Image Policy API (v1alpha

1)
Tutorials
Home
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Secret | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

ha1)
Reference
Secret
Secret
Secret
: v1
kind
: Secret
metadata
(
)
data
immutable
(boolean)
type
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Secret
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string

(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
202 (
Secret
update
name
(
in path
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Reference docs for ku

bernetes 1.23 (02445930e5)
Secret
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Run Applications | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operato

r pattern
Tasks
Debug Pods
Determine the Reason 

for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Clust

er Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

)
Tasks
Run and manage both stateless and stateful applications.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Policy Resources | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
1 - LimitRange
: v1
kind
metadata
(
)
spec
(
)
limits
Quantity
)
Quantity
)
Quantity
)
Quantity
)
Quantity
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
t

imeoutSeconds
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - ResourceQuota
: v1
kind
metadata
(
)
spec
(
)
status
(
)
hard
Quantity
)
scopeSelector is also a collection of filters like 

scopes that must match each object tracked by a quota but expressed using ScopeSelectorOperator in combination with possible values. For a resource to match, both scopes AND scopeSelector (if specified in spec), must be matched.
scopes
hard
Quantity
)
used
Quantity
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
):

 boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status

): OK
3 - NetworkPolicy
kind
metadata
(
)
spec
(
)
status
(
)
(
ingress
(IPBlock)
Except is a slice of CIDRs that should not be included within an IP Block Valid examples are "192.168.1.0/24" o

r "2001:db8::/64" Except values will be rejected if they are outside the CIDR range
(
)
(
)
(int32)
(string)
egress
egress.to
(IPBlock)
IPBlock describes a particular CIDR (Ex. "192.168.1.0/24","2001:db8::/64") that is allowed to the pods matched by a NetworkPolicySpec's podSelector. The except entry describes CIDRs that should not be included within this 

rule.
(
)
(
)
(int32)
(string)
type
(string), requ

ired
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update

name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
4 - PodDisruptionBudget
kind
metadata
(
)
spec
(
)
status
(
)
An eviction is allowed if at most "maxUnavailable" pods selected by "selector" are unavailable after the eviction, i.e. even in absence of the evicted pod. For example, one can prevent all volu

ntary evictions by specifying 0. This is a mutually exclusive setting with "minAvailable".
selector
(
)
(string)
type
SufficientPods: There are more pods than required by the PodDisruptionBudget.
lastTransitionTime is the last time the condition transitione

d from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
(int64)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in qu

ery
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
DELETE /apis/pol

icy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

Debug Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

(v1)
Tasks
Debug Pods
Debug Pods
This is
not
this guide
.
${
POD_NAME
}
Running
Pending
hostPort
hostPort
hostPort
hostPort
Waiting
Waiting
.
If your pod is not behaving as 

you expected, it may be that there was an error in your
command
as
commnd
option.
.
command
as
commnd
46757
]
46757
]
false
and then
endpoints
${
}
...
spec
:
-
selector
:
name
:
nginx
type
:
frontend
=
name
=
=
frontend
Please see
Service
Endpoints
, and your
Pods
are
Feedback
Yes
No
.
or
.
Last modified June 16, 2022 at 12:22 PM PST:
Update debug-pods.md (aa302cac0e)
Home
Blog
Training
Partners
Community

CC BY 4.0

Debug Running Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Your
Pod
Debugging
Pods
.
kubectl
.
Using
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"128Mi"
cpu
:
"500m"
ports
:
-
:
80
IPs:
  nginx:
    Limits:
      cpu:     500m
    Requests:
      cpu:        500m
    Enviro

nment:  &lt;none&gt;
    Mounts:
  Ready             True 
Volumes:
QoS Class:                   Guaranteed
Events:
To find out w

hy the nginx-deployment-1370807587-fz9sd pod is not running, we can use
  Node:		/
  IP:
    nginx:
      QoS Tier:
      Limits:
        cpu:	1
      Requests:
        cpu:	1
  Volumes:
  Events:
=
argument.
-o yaml
:
v1
kind
:
Pod
metadata
:
:
:
labels
:
app
:
nginx
:
name
:
namespace
:
default
:
-
:
apps/v1
:
true
:
true
kind
:
name
:
uid
:
:
"1364"
uid
:
spec
:
:
-
image
:
nginx
:
Always
name
:
nginx
ports
:
-
:
80
protocol
:
TCP
resources
:
limits
:
cpu
:
500m
memory
:
128Mi
requests
:
cpu
:
500m
memory
:
128Mi
:
:
File
:
-
mountPath
:
name
:
readOnly
:
true
dnsPolicy
:
:
true
nodeName
:
:
priority
:
0
:
Alw

ays
:
:
{}
:
default
:
default
:
30
:
-
effect
:
NoExecute
key
:
operator
:
Exists
:
300
-
effect
:
NoExecute
key
:
operator
:
Exists
:
300
volumes
:
-
name
:
projected
:
:
420
sources
:
-
:
:
3607
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
status
:
:
-
:
null
:
status
:
"True"
type
:
-
:
null
:
status
:
"True"
type
:
Ready
-
:
null
:
status
:
"True"
type
:
-
:
null
:
status
:
"True"
type
:
:
-
:
image
:
imageID
:
lastState
:
{}
name
:
nginx
ready
:
true
:
0
started
:
true
state
:
running
:
startedAt
:
hostIP
:
phase
:
Running
podIP
:
10.88.0.3
podIPs
:
-
ip
:
10.88.0.3
-
ip
:
2001
:db8::1
qosClass
:
startTime
:
${
POD_NAME
}
${
}
${
POD_NAME
}
${
}
If the
includes
:
kubectl
exec
${
POD_NAME
}
-c
${
}
--
${
CMD
}
${
ARG1
}
${
ARG2
}
...
${
ARGN
}
Note:
kubectl
exec
-i
and
-t
kubectl
exec
.
.
=
=
Never
pause
images.
kubectl
exec
. If you
-i
/
argument,
kubectl
=
=
Defa

ulting debug container name to debugger-8xzrl.
/ #
--target
creates.
Note:
The
--target
ps
does not reveal processes in other
:
...
...
Use
to create a
busybox
busybox
:
=
=
myapp
named
that adds a
=
=
Note:
flag.
The
-i
.
The
works, see
.
  myapp:
    ...
    Args:
      false
You can use
kubectl debug myapp -it --copy-to=myapp-debug --container=m

yapp -- sh
/ #
Note:
or
The
-i
.
:
Now use
to
ubuntu
:
syntax as
.
*=ubuntu
to
ubuntu
.
, run:
=
ubuntu
/host
.
and
will fail.
Feedback
Yes
No
.
or
.
Using
Home
Blog
Training
Partners
Community
CC BY 4.0

Accessing the Kubernetes API from a Pod | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install a

1)
Tasks
This guide demonstrates how to access the Kubernetes API from within a pod.
minikube
When accessing the API from within a Pod, locating and authenticating
to the API server are slightly different to the external client case.
The easiest way to use the Kubernetes API from a Pod is to use
one of the official
. These
libraries can automatically discover the API server and authenticate.
Using Official Client Libraries
From within a Pod, the recommended ways to connect to the Kubernetes API are:
For a Go client, use the official
Go client library
.
The
rest.InClusterConfig()
function handles API host discovery and authentication automatically.
See
an example here
.
For a Python client, use the official
Python client library
.
The
config.load_incluster_config()
See
.
There are a number of other libraries available, please refer to the
page.
In each case, the service account credentials of the Pod are used to communicate
securely with the API server.
While running in a Pod, the Kubernetes apiserver is accessible via a Service named
in the
default
namespace. Therefore, Pods can use the
hostname to query the API server. Official client libraries
do this automatically.
The recommended way to authenticate to the API server is with a
credential. By default, a Pod
is associated with a service account, and a credential (token) for that
service account is placed into the filesystem tree of each container in that Pod,
at
.
If available, a certificate bundle is placed into the filesystem tree of each
container at
/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
, and should be
used to verify the serving certificate of the API server.
Finally, the default namespace to be used for namespaced API operations is placed in a file
at
/var/run/secrets/k

ubernetes.io/serviceaccount/namespace
in each container.
If you would like to query the API without an official client library, you can run
as the
command
of a new sidecar container in the Pod. This way,
will authenticate
to the API and expose it on the
localhost
interface of the Pod, so that other containers
in the Pod can use it directly.
Without using a proxy
It is possible to avoid using the kubectl proxy by passing the authentication token
directly to the API server. The internal certificate secures the connection.
# Point to the internal API server hostname
APISERVER
=
https://kubernetes.default.svc
# Path to ServiceAccount token
SERVICEACCOUNT
=
# Read this Pod's namespace
NAMESPACE
=
$(
cat
${
}
/namespace
)
# Read the ServiceAccount bearer token
TOKEN
=
$(
cat
${
}
/token
)
# Reference the internal certificate authority (CA)
CACERT
=
${
}
/ca.crt
curl --cacert
${
CACERT
}
--header
${
TOKEN
}
"
-X GET
${
APISERVER
}
/api
The output will be similar to this:
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Feedback
Yes
No
.
or
.
Last modified March 25, 2022 at 6:48 PM PST:
Remove duplicated contents for accessing the API server (58a00cfb36)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

member
,
reviewer
, or
approver
.
teams
on GitHub:
@name
OWNERS
files.
uses
two
:
approve
OWNERS
and
can act as a reviewer or approver of files in that subdirector

y and its
OWNERS
.
lgtm
and
approve
hold
/hold
comment
/lgtm
comment).
lgtm
/lgtm
comment.
/approve
or
.
Feedback
Yes
No
.
or
.
Last modified August 01, 2022 at 2:39 AM PST:
Change sig-docs-xx-reviewers to -reviews in docs (35587abca4)
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubelet CredentialProvider (v1beta1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

n Linux
Debug Pods
G

et a Shell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespace Lev

el
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver E

ncryption Configuration (v1)
Reference
Field
string
kind
string
image
string
Field
string
kind
string
auth
'.
.k8s.io' is also suppor

ted. Each glob can only match
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
1 - Declarative Management of Kubernetes Objects Using Configuration Files
to
apply
will make.
Install
kubectl
.
minikube
.
The
kubectl
See
Overview
files to
Use
to create all objects, except those that already exist,
Note:
Add the
-R
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Run

Note:
diff
uses
,
.
Since
diff
PATCH
,
CREATE
, and
UPDATE
See
:
:
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
Note:
Add the
-R
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
Note:
:
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
replicas
.
:
=
2
:
replic

as
replicas
field:
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
replicas
:
2
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
Update the
to
field:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
The
replicas
.
The
image
from
.
The
The
The
field.
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
replicas
:
2
# ...
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
Warning:
Mixing
create
and
replace
create
and
replace
that
.
Warning:
War

ning:
--prune
Warning:
and
You can use
with
-o yaml
Caution:
A
patch
When
The
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
replicas
:
2
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
:
Clear fields explicitly set to null in the local obje

ct configuration file
.
Action:
Clear
image
Action:
image
Set the
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
selector
:
:
# ...
app
:
nginx
replicas
:
2
# ...
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
primitive
image
and
replicas
Action:
Replace.
map
object
labels
,
,
spec
and
metadata
Action:
list
,
ports
, and
args
are lists.
Action:
Varies.
When
spec
spec
is
spec
, such as
replicas
Note:
-
Action
Yes
Yes
-
Yes
No
-
No
-
Yes
No
-
No
Fields that represent maps are merged by comparing each of the subfields or elements of the map:
Note:
-
Action
Yes
Yes
-
Yes
No
-
No
-
Yes
No
-
No
Treat the list the same as a primitive field.

 Replace or delete the
Example:
Use
args
args
Any
args
args
args
:
[
"a"
,
"b"
]
args
:
[
"a"
,
"c"
]
args
:
[
"a"
,
"b"
,
"d"
]
args
:
[
"a"
,
"c"
]
. The
types.go
Example:
Use
by
name
.
:
-
name
:
nginx
image
:
-
:
image
:
-
:
image
:
:
-
name
:
nginx
image
:
-
name
:
image
:
-
:
image
:
:
-
name
:
nginx
image
:
-
name
:
image
:
-
name
:
image
:
args
:
[
"run"
]
-
:
image
:
:
-
name
:
nginx
image
:
-
name
:
image
:
args
:
[
"run"
]
-
name
:
image
:
-
name
:
image
:
args
args
no element with that name appeared in the last-applied-configuration.
Note:
the
tag in
types.go
If no
strategy
:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
la

bels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
:
:
apps/v1
kind
:
# ...
spec
:
selector
:
:
app
:
nginx
:
5
replicas
:
1
strategy
:
:
maxSurge
:
1
:
1
type
:
template
:
metadata
:
:
null
labels
:
app
:
nginx
spec
:
:
-
image
:
:
name
:
nginx
ports
:
-
:
80
protocol
:
TCP
resources
:
{}
:
dnsPolicy
:
:
Always
:
{}
:
30
# ...
Example:
spec
:
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
spec
:
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
spec
:
strategy
:
type
:
:
maxSurge
:
1
:
1
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
spec
:
strategy
:
:
:
maxSurge
:
1
:
1
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.
to
values.
The user changes
to
Recreate
. The
If the
of
Recreate
.
Fields that do not app

ear in the configuration file can be cleared by
null
Use
.
.
.
Note:
status
Note:
Set the
Set the
Warning:
Example:
selector
:
:
:
template
:
metadata
:
labels
:
:
2 - Declarative Management of Kubernetes Objects Using Kustomize
Kustomize
through a
.
with
or
-k
flag:
Install
kubectl
.
minikube

.
and
files
list in
file:
cat
FOO=Bar
EOF
cat
files:
EOF
:
v1
data
:
:
|
FOO=Bar
kind
:
ConfigMap
metadata
:
name
:
envs
list in
.env
file:
cat
FOO=Bar
EOF
cat
envs:
- .env
EOF
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
name
:
Note:
.env
literals
cat
literals:
- FOO=Bar
EOF
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
name
:
FOO=Bar
EOF
:
apps/v1
kind
:
metadata
:
name
:
my-app
labels
:
app
:
my-app
spec
:
selector
:
:
app
:
my-app
template
:
metadata
:
labels
:
app
:
my-app
spec
:
:
-
name
:
app
image
:
my-app
:
-
name
:
config
mountPath
:
/config
volumes
:
-
name
:
config
configMap
:
name
:
EOF
resources
:
:
-
name
:
files
:
EOF
:
v1
data
:
:
|
FOO=Bar
kind
:
ConfigMap
metadata
:
name
:
---
:
apps/v1
kind
:
metadata
:
labels
:
app
:
my-app
name
:
my-app
spec
:
selector
:
:
app
:
my-app
template
:
metadata
:
labels
:
app
:
my-

app
spec
:
:
-
image
:
my-app
name
:
app
:
-
mountPath
:
/config
name
:
config
volumes
:
-
configMap
:
name
:
name
:
config
files
list in
cat
EOF
cat
files:
EOF
:
v1
data
:
:
kind
:
Secret
metadata
:
name
:
type
:
Opaque
literals
list in
cat
literals:
EOF
:
v1
data
:
password
:
c2VjcmV0
username
:
YWRtaW4=
kind
:
Secret
metadata
:
name
:
type
:
Opaque
Like ConfigMaps, generated Secrets can be used in Deployments by referring to the name of the secretGenerator:
cat
EOF
cat
metadata:
labels:
spec:
selector:
template:
metadata:
labels:
spec:
- name: app
volumes:
secret:
EOF
cat
files:
EOF
cat
literals:
- FOO=Bar
labels:
EOF
Run
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
:
note
:
generated
labels
:
type
:
generated
name
:
cat
metadata:
labels:
app: nginx
spec:
selector:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
EOF
cat
app: bingo
EOF
Run
:
apps/v1
kind
:
metadata
:
:
:
labels
:
app
:
bingo
name
:
namespace
:
spec
:
selector
:
:
app
:
bingo
template
:
metadata
:
:
:
labels
:
app
:
bingo
spe

c
:
:
-
image
:
nginx
name
:
nginx
Composing
resources
resources
list.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
cat
EOF
Patches can be used to apply different customizations to Resources. Kustomize supports different patching
and
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
metadata:
spec:
EOF
cat
metadata:
spec:
template:
spec:
limits:
EOF
cat
EOF
Run
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
3
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
nginx
name
:
my-nginx
ports
:
-
:
80
resources
:
limits
:
memory
:
512Mi
JSON patch
through
.
through
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
value: 3
EOF
cat
&lt;&lt;EOF &gt;./kustomizati

on.yaml
- target:
EOF
Run
to see the
replicas
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
3
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
nginx
name
:
my-nginx
ports
:
-
:
80
images
field in
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
images:
EOF
Run
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
2
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
name
:
my-nginx
ports
:
-
:
80
or
file. It is
vars
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
EOF
cat
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
cat
vars:
objref:
EOF
Run
:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
2
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
command
:
- start
- --host
image
:
nginx
name
:
my-nginx
bases
and
overlays
. A
base
as long as a
overlay
bases
. A
base
mkdir base
cat
metadata:
s

pec:
selector:
template:
metadata:
labels:
spec:
EOF
cat
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
cat
EOF
mkdir dev
cat
bases:
- ../base
EOF
mkdir prod
cat
bases:
- ../base
EOF
Use
or
-k
in
kubectl
.
Note that
-k
,
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
EOF
:
:
:
deleted
Field
Type
namespace
string
string
string
resources
[]string
[]
[]
bases
[]string
[]string
[]
Patch
vars
[]
Var
images
[]
Image
[]string
crds
[]string
Kustomize
3 - Managing Kubernetes Objects Using Imperative Commands
kubectl
explains how those commands are organized and how to use the

m to manage live objects.
Install
kubectl
.
minikube
.
The
kubectl
See
The
kubectl
run
expose
autoscale
The
kubectl
to create.
create
command.
command.
-h
The
kubectl
These commands are named to enable users unfamiliar with Kubernetes
scale
annotate
label
The
kubectl
set
Note:
The
kubectl
edit
patch
.
delete
Note:
You can use
get
get -h
describe
logs
: Prints the s

tdout and stderr for a container running in a Pod.
Using
set
in a
create
set
and
create
create
set
create
=
"None"
=
set
The
The
The
Using
--edit
You can use
=
"None"
=
The
.
The
4 - Imperative Management of Kubernetes Objects Using Configuration Files
kubectl
Install
kubectl
.
minikube
.
The
kubectl
See
You can use
Warning:
replace
, where
the
replace
You can use
You can use
Note:
metadata
name
.
You will have to use other

 flags for deleting the object. For example:
You can use
The
-o yaml
Use
The
create
,
replace
, and
delete
replace
replace
Warning:
selector
:
:
:
template
:
metadata
:
labels
:
:
5 - Update API Objects in Place Using kubectl patch
minikube
.
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
image
:
nginx
:
-
effect
:
key
:
dedicated
value
:
test-team
Create the Dep

loyment:
1/1
spec
:
template
:
spec
:
:
-
name
:
image
:
redis
:
-
image
:
redis
:
Always
name
:
...
-
image
:
nginx
:
Always
name
:
...
2/2
  ...
  ...
.
key
field of
PodSpec
of
merge
:
type
PodSpec
struct
{
...
...
}
:
{
...,
{
},
"name"
,
"merge"
}
.
spec
:
template
:
spec
:
:
-
effect
:
key
:
disktype
value
:
ssd
:
-
effect
:
key
:
disktype
value
:
ssd
replace
.
type
PodSpec
struct
{
...
`json:"tole

rations,omitempty" protobuf:"bytes,22,opt,name=tolerations"`
...
}
.
The
type
Merge type
json
merge
strategic
.
type
strategic
spec
:
template
:
spec
:
:
-
name
:
image
:
type
to
merge
:
The
list that you specified in the patch has only one Container.
list.
spec:
...
1/1
0
1m
0
1m
strategy:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
strategy
:
:
maxSurge
:
30
%
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
image
:
nginx
strategy.
spec
:
strategy
:
type
:
Recreate
type
as
Recreate
:
type
spec
:
strategy
:
:
- type
type
:
Recreate
type
key of the
strategy
spec
:
strategy
:
type
:
Recreate
template
:
$

list.
The
Strategy
of
:
type
struct
{
...
...
}
:
{
...,
{
"$ref":
,
,
},
....
}
.
The
{
"spec"
: {
: {
"spec"
: {
: [
{
"name"
:
,
"image"
:
"redis"
}
]
}
}
}
}
with
The flag
status
and
scale
status
or
scale
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
app
=
nginx
flag:
=
'scale'
--type
=
'merge'
-p
app
=
nginx
nginx-deployment-7fb96c846b-mlgns   1

/1     Running   0          107s
...
spec
:
replicas
:
3
...
status
:
...
:
3
:
3
replicas
:
3
Note:
If you run
Summary
,
,
,
,
and
.
Note:
Home
Blog
Training
Partners
Community
CC BY 4.0

Mapping from dockercli to crictl | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Note:
Note:
This page is deprecated and will be removed in Kubernetes 1.27.
crictl
CRI
crictl
cri-tools
docker
crictl
.
docker
crictl
docker
Note:
crictl
docker
docker cli
crictl
attach
attach
,
exec
exec
,
--user
,
images
images
info
info
inspect
inspect
,
inspecti
logs
logs
--details
ps
ps
stats
stats
version
version
docker cli
crictl
create
create
kill
stop
--signal
pull
pull
,
rm
rm
rmi
rmi
run
run
start
start
stop
stop
update
update
--restart
,
--blkio-weig

ht
crictl
inspectp
pods
List pods
runp
rmp
stopp
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
1 - Run a Stateless Application Using a Deployment
This page shows how to run an application using a Kubernetes Deployment object.
Create an nginx deployment.
Use kubectl to list information about the deployment.
Update the deployment.
minikube
.
Creating and exploring an nginx deployment
You can run an application by creating a Kubernetes Deployment object, and you
can describe a Deployment in a YAML file. For example, this YAML file describes
a Deployment that runs the nginx:1.14.2 Docker image:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Create a Deployment based on the YAML file:
Name:     nginx-deployment
 CreationTimestamp:  Tue, 30 Aug 2016 18:11:37 -0700
 Labels:     app=nginx
 Annotations:    deployment.kubernetes.io/revision=1
 Selector:   app=nginx
 Replicas:   2 desired | 2 updated | 2 total | 2 available | 0 unavailable
 StrategyType:   RollingUpdate
 MinReadySeconds:  0
 RollingUpdateStrategy:  1 max unavailable, 1 max surge
   Labels:       app=nginx
    nginx:
     Image:              nginx:1.14.2
     Port:               80/TCP
     Environment:        &lt;none&gt;
     Mounts:             &lt;none&gt;
   Volumes:              &lt;none&gt;
   Available     True    MinimumReplicasAvailable
   Progressing   True    NewReplicaSetAvailable
 OldReplicaSets:   &lt;none&gt;
 NewReplicaSet:    nginx-deployment-1771418926 (2/2 replicas created)
 No events.
List the Pods created by the deployment:
kubectl get pods -l app=nginx
 nginx-deployment-1771418926-7o5ns   1/1       Running   0          16h
 nginx-deployment-1771418926-r18az   1/1       Running   0          16h
Display information about a Pod:
where
is the name of one of your Pods.
Updating the deployment
You can update the deployment by applying a new YAML file. This YAML file
specifies that the deployment should be updated to us

e nginx 1.16.1.
application/deployment-update.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
# Update the version of nginx from 1.14.2 to 1.16.1
ports
:
-
:
80
Apply the new YAML file:
kubectl apply -f https://k8s.io/examples/application/deployment-update.yaml
Watch the deployment create pods with new names and delete the old pods:
Scaling the application by increasing the replica count
You can increase the number of Pods in your Deployment by applying a new YAML
file. This YAML file sets
replicas
to 4, which specifies that the Deployment
should have four Pods:
application/deployment-scale.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
4
# Update the replicas from 2 to 4
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
kubectl apply -f https://k8s.io/examples/application/deployment-scale.yaml
Verify that the Deployment has four Pods:
 nginx-deployment-148880595-4zdqq   1/1       Running   0          25s
 nginx-deployment-148880595-6zgi1   1/1       Running   0          25s
 nginx-deployment-148880595-fxcez   1/1       Running   0          2m
 nginx-deployment-148880595-rwovn   1/1       Running   0          2m
Delete the deployment by name:
kubectl delete deployment nginx-deployment
ReplicationControllers -- the Old Way
The preferred way to create a replicated application is to use a Deployment,
which in turn uses a ReplicaSet. Before the Deployment and ReplicaSet were
added to Kubernetes, replicated applications were configured using a
.
.
2 - Run a Single-Instance Stateful Application
minikube
.
,
or
used here.
:
v1
kind
:
Service
metadata
:
name
:
mysql
spec
:
ports
:
-
port
:
3306
selector
:
app
:
mysql
clusterIP
:
None
---
:
apps/v1
kind
:
metadata
:
name
:
mysql
spec
:
sele

ctor
:
:
app
:
mysql
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
mysql
spec
:
:
-
image
:
mysql:5.6
name
:
mysql
env
:
-
name
:
value
:
password
ports
:
-
:
3306
name
:
mysql
:
-
name
:
mountPath
:
volumes
:
-
name
:
:
claimName
:
:
v1
kind
:
metadata
:
name
:
labels
:
type
:
local
spec
:
:
manual
capacity
:
storage
:
20Gi
:
hostPath
:
path
:
---
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
20Gi
    mysql:
     Mounts:
   Volumes:
 Events:
know your stateful MySQL d

atabase is up and running.

mysql&gt;
Updating
with the
.
Use
strategy:
in the Deployment configuration
not
Recreate
.
Volumes
and
3 - Run a Replicated Stateful Application
.
Note:
minikube
,
or
used here.
and
,
Pods
,
Services
, and
.
:
v1
kind
:
ConfigMap
metadata
:
name
:
mysql
labels
:
app
:
mysql
:
mysql
data
:
:
|
[mysqld]
log-bin
:
|
[mysqld]
my.cnf
In this case, you want the primary server to be able to serve replication logs t

o replicas
There's nothing special about the ConfigMap itself that causes different
:
v1
kind
:
Service
metadata
:
name
:
mysql
labels
:
app
:
mysql
:
mysql
spec
:
ports
:
-
name
:
mysql
port
:
3306
clusterIP
:
None
selector
:
app
:
mysql
---
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
mysql
:
mysql
readonly
:
"true"
spec
:
ports
:
-
name
:
mysql
port
:
3306
selector
:
app
:
mysql
mysql
resolving
replicas.
writes.
:
apps/v1
kind
:
metadata
:
name
:
mysql
spec
:
selector
:
:
app
:
mysql
:
mysql
:
mysql
replicas
:
3
template
:
metadata
:
labels
:
app
:
mysql
:
mysql
spec
:
:
-
name
:
image
:
mysql:5.7
command
:
- bash
-
"-c"
- |
set -ex
else
fi
:
-
name
:
conf
mountPath
:
-
name
:
mountPath
:
-
name
:
image
:
command
:
- bash
-
"-c"
- |
set -ex
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
:
-
name
:
mysql
image
:
mysql:5.7
env
:
-
name
:
value
:
"1"
ports
:
-
name
:
mysql
:
3306
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
resources
:
requests
:
cpu
:
500m
memory
:
1Gi
:
exec
:
command
:
[
,
"ping"
]

:
30
:
10
:
5
:
exec
:
# Check we can execute queries over TCP (skip-networking is off).
command
:
[
"mysql"
,
"-h"
,
,
"-e"
,
"SELECT 1"
]
:
5
:
2
:
1
-
name
:
image
:
ports
:
-
name
:
:
3307
command
:
- bash
-
"-c"
- |
set -ex
fi
fi
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
volumes
:
-
name
:
conf
emptyDir
:
{}
-
name
:
configMap
:
name
:
mysql
:
-
metadata
:
name
:
data
spec
:
:
[
]
resources
:
requests
:
storage
:
10Gi
app
=
Running
:
Press
Ctrl+C
Note:
.
mysql-0
,
mysql-1
, and
mysql-2
.
The scrip

t determines its own ordinal index by extracting it from the end of
the Pod name, which is returned by the
hostname
command.
conf.d
or
conf.d
.
0
,
N
is
N+1
.
mysql
mysqld
sidecar
.
The
mysqld
and
(
)
mysql:5.7
mysql
=
=
Never --
\
EOF
=
=
Never --
\
+---------

+
| message |
| hello   |
in a loop:
=
=
Never --
\
bash -ic
Ctrl+C
The
for the
mysql
kubectl
exec
mysql-2
and
mysql
Look for
1/2
in the
READY
column:
102
anymore.
server-id
as
,
102
mysql-2
.
kubectl
exec
mysql-2
102
drain
.
Replace
Caution:
cluster.

 -o wide --watch
NAME      READY   STATUS          RESTARTS   AGE       IP            NODE
[...]
102
=
5
app
=
103
and
104
the
existed:
=
=
Never --
\
| message |
| hello   |
=
3
Note:
app
=
mysql
Cancel the
Ctrl+C
app
=
mysql
app
=
mysql
What's

 next
.
.
.
.
Look in the
4 - Scale a StatefulSet
.
or
=
,
update
:
:
Or use
:
perform scaling only when you are sure that your stateful application cluster is
.
5 - Delete a StatefulSet
.
kubectl
.
=
orphan
By passing
to

=
MyApp
Note:
grace
=
$(
)
=
MyApp
sleep
$grace
=
MyApp
.
6 - Force Delete StatefulSet Pods
This page shows how to delete Pods which are part of a
stateful set
, and explains the considerations to keep in mind when doing so.
This is a fairly advanced task and has the potential to violate some of the properties inherent to StatefulSet.
Before proceeding, make yourself familiar with the considerations enumerated below.
StatefulSet considerations
In normal operation of a StatefulSet, there is
never
a need to force delete a StatefulSet Pod. The
StatefulSet controller
is responsible for creating, scaling and deleting members of the StatefulSet. It tries to ensure that the specified number of Pods from ordinal 0 through N-1 are alive and ready. StatefulSet ensures that, at any time, there is at most one Pod with a given identity running in a cluster. This is referred to as
at most one
semantics provided by a StatefulSet.
Manual force deletion should be undertaken with caution, as it has the potential to violate the at most one semantics inherent to StatefulSet. StatefulSets may be used to run distributed and clustered applications which have a need for a stable network identity and stable storage. These applications often have configuration which relies on an ensemble of a fixed number of members with fixed identities. Having multiple members with the same identity can be disastrous and may lead to data loss (e.g. split brain scenario in quorum-based systems).
You can perform a graceful pod deletion with the following command:
kubectl delete pods &lt;pod&gt;
For the above to lead to graceful termination, the Pod
must not
specify a
of 0. The practice of setting a
of 0 seconds is unsafe and strongly discouraged
for StatefulSet Pods. Graceful deletion is safe and will ensure that the Pod
shuts down gracefully
before the kubelet deletes the name from the apiserver.
A Pod is not deleted automatically when a node is unreachable.
The Pods running on an unreachable Node enter the 'Terminating' or 'Unknown' state after a
timeout
.
Pods may also enter these states when the user attempts graceful deletion of a Pod
on an unreachable Node.
The only ways in which a Pod in such a state can be removed from the apiserver are as follows:
The Node object is deleted (either by you, or by the
Node Controller
).
The kubelet on the unresponsive Node starts responding, kills the Pod and removes the entry from th

e apiserver.
Force deletion of the Pod by the user.
The recommended best practice is to use the first or second approach. If a Node is confirmed to be dead (e.g. permanently disconnected from the network, powered down, etc), then delete the Node object. If the Node is suffering from a network partition, then try to resolve this or wait for it to resolve. When the partition heals, the kubelet will complete the deletion of the Pod and free up its name in the apiserver.
Normally, the system completes the deletion once the Pod is no longer running on a Node, or the Node is deleted by an administrator. You may override this by force deleting the Pod.
Force Deletion
Force deletions
do not
wait for confirmation from the kubelet that the Pod has been terminated. Irrespective of whether a force deletion is successful in killing a Pod, it will immediately free up the name from the apiserver. This would let the StatefulSet controller create a replacement Pod with that same identity; this can lead to the duplication of a still-running Pod, and if said Pod can still communicate with the other members of the StatefulSet, will violate the at most one semantics that StatefulSet is designed to guarantee.
When you force delete a StatefulSet pod, you are asserting that the Pod in question will never again make contact with other Pods in the StatefulSet and its name can be safely freed up for a replacement to be created.
If you want to delete a Pod forcibly using kubectl version &gt;= 1.5, do the following:
kubectl delete pods &lt;pod&gt; --grace-period
=
0
--force
If you're using any version of kubectl &lt;= 1.4, you should omit the
--force
option and use:
=
0
If even after these commands the pod is stuck on
Unknown
state, use the following command to remove the pod from the cluster:
kubectl patch pod &lt;pod&gt; -p
'{"metadata":{"finalizers":null}}'
Always perform force deletion of StatefulSet Pods carefully and with complete knowledge of the risks involved.
.
7 - Horizontal Pod Autoscaling
a
or
), with the
Pods
.
vertical
a
DaemonSet
.)
.
There is
of using
graph BT
scale
end
en

abled
finds the target resource defined by the
,
API
(
,
, or
). The
API is
.
scale
.
value:
200m
is
100m
50m
When a
or
the
Before checking the tolerance and deciding on 

the final values, the control
are
Ready
.
or
the pod was before it became ready, that pod is set aside as well.
The
original
used.
API Object
the
.
When you create a HorizontalPodAutoscaler API object, ma

ke sure the name specified is a valid
.
.
thrashing
,
or
flapping
Kubernetes lets you perform a rolling update on a Deployment. In that
replicas
replicas
cpu
and
memory
should
like this:
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
60
Algorithm
Note:
Algorithm
type
:
:
name
:
cpu
container
:
target
:
type
:
:
60
in the
Note:
(such as a Deplo

yment), you should update the associated HPA to track both the new and
(the
to scale based on a custom metric (that is not built in to Kubernetes or any Kubernetes component).
API.
See
(the
The
.
the HPA V2
,
and
.
and
.
(the
v2
behavior
field
(see the
)
scaleUp
and / or
scaleDown
under the
behavior
field.
flapping
behavior
behavior
:
scaleDown
:
policies
:
-
type
:
Pods
value
:
4
:
60
-
type
:
Percent
value
:
10
:
60
(Pods)
(Percent)
only be used when the number of pod replicas is more than 40. With 40 or less repl

icas, the first policy will be applied.
(Pods)
is applied
The policy selection can be changed by specifying the
Min
Disabled
flapping
of
scale.
scaleDown
.
behavior
:
scaleDown
:
:
300
behavior
:
scaleDown
:
:
300
policies
:
-
type
:
Percent
value
:
100
:
15
scaleUp
:
:
0
policies
:
-
type
:
Percent
value
:
100
:
15
-
type
:
Pods
value
:
4
:
15
:
Max
300
behavior
:
scaleDown
:
:
60
behavior
:
scaleDown
:
policies
:
-
type
:
Percent
value
:
10
:
60
to
Min
means
behavior
:
scaleDown
:
policies
:
-
type
:
Percent
value
:
10
:
60
-
type
:
Pods
value
:
5
:
60
:
Min
The
value of
Disabled
behavior
:
scaleDown
:
:
Disabled
H

orizontalPodAutoscaler, like every API resource, is supported in a standard way by
kubectl
.
command.
.
.
foo
80%
to
false
of
).
kubectl
.
Read a
.
Read the
8 - HorizontalPodAutoscaler Walkthrough
A
a
or
), with the
Pods
.
vertical
This document walks you through an example of enabling HorizontalPodAutoscal

er to
minikube
.
).
the
kubelets
,
using an
to add
.
Service
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
-
name
:
image
:
registry.k8s.io/hpa-example
ports
:
-
:
80
resources
:
limits
:
cpu
:
500m
requests
:
cpu
:
200m
---
:
v1
kind
:
Service
metadata
:
name
:
labels
:
run
:
spec
:
ports
:
-
port
:
80
selector
:
run
:
kubectl
. There is
part of
kubectl
.spec
.
See
=
50
--min
=
1
--max
=
10
(the
TARGET

=
=
Now run:
Note:
busybox
the load generation by typing
.
form:
Open the
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
metrics
:
-
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
50
status
:
:
1
:
:
1
:
1
:
-
type
:
Resource
resource
:
name
:
cpu
current
:
:
0
:
0
metrics
.
of

instead of
, and
.
only
support a
target
type of
.
type
:
Pods
pods
:
metric
:
name
:
target
:
type
:
:
1k
target
types of
both
Value
and
. With
Value
metric.
type
:
Object
object
:
metric
:
name
:
:
:
kind
:
Ingress
name
:
target
:
type
:
Value
value
:
2k
If you provide multiple such metric blocks, the HorizontalPodAutoscaler will consider each metric in turn.
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
metrics
:
-
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
50
-
type
:
Pods
pods
:
metric
:
name
:
target
:
type
:
:
1k
-
type
:
Object
object
:
metric
:
name
:
:
:
kind
:
Ingress
name
:
target
:
type
:
Value
value
:
10k
status
:
:
1
:
:
1
:
1
:
-
type
:
Resource
resource
:
name
:
cpu
current
:
:
0
:
0
-
type
:
Object
object
:
metric
:
name
:
:
:
kind
:
Ingress
name
:
current
:
value
:
10k
labels
with the
verb
type
:
Object
object
:
metric
:
name
:
selector
:
{
:
{
verb
:
GET}}
match multiple ser

ies. The selector is additive, and cannot select metrics
not
Pods
Object
type).
with
.
metric
name
and
selector
External
Object
.
,
Value
and
Object
type.
-
type
:
External
external
:
metric
:
name
:
selector
:
:
queue
:
target
:
type
:
:
30
When possible, it's preferable to use the custom metric target types instead of external metrics, since it's
in any way.
we can use
:
Events:
,
False
fetching metrics. Finally, the la

st condition,
quantity
10500m
10.5
between
1
and
1500m
, or
1
and
1.5
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
:
50
9 - Specifying a Disruption Budget for your Application
nodes.
.
and/or
.
.
; the same
.
.
Multiple-instance Stateful application such as Consul, ZooKeeper, or etcd:
Concern: Do not 

reduce number of instances below quorum, otherwise writes fail.
Values for
or
"50%"
to
"50%"
you set
to
"50%"
code
A
Note:
and
in a single
.
is the
scale
.
5 or more
healthy
selector
.
Note:
If you set
you are requiring zero voluntary evictions. When you set zero voluntary evictions for a workload
object such as 

ReplicaSet, then you cannot successfully drain a Node running one of those Pods.
.
.
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
:
2
selector
:
:
app
:
zookeeper
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
:
1
selector
:
:
app
:
zookeeper
zk-pdb
:
policy/v1
kind
:
metadata
:
:
…
:
:
1
name
:
zk-pdb
…
status
:
:
3
:
2
:
1
:
3
:
1
item with
and
.
Note:
on the
API server
.
By using
to the
policy.
Policies:
).
Ready
), but not yet healthy are considered disrupted and can be evicted
This means pr

ospective running pods of a disrupted application might not get a chance to become healthy.
Ready
Note:
Pods in
Pending
,
Succeeded
or
Failed
only
.
10 - Accessing the Kubernetes API from a Pod
minikube
. These
.
The
See
.
.
The
See
.
page.
in the
default
at
.
at
/var/run/secrets/kubernetes.io/serviceaccount/namespace
as th

e
command
localhost
APISERVER
=
=
NAMESPACE
=
$(
cat
${
}
)
TOKEN
=
$(
cat
${
}
/token
)
CACERT
=
${
}
/ca.crt
${
CACERT
}
--header
${
TOKEN
}
"
-X GET
${
APISERVER
}
/api
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
content.
S[ ] -.-
end
T[ ] -.-
end
enabled
using
Hugo
.
GitHub
the
You can use
or
in our
/content/
. Each
.
see
.
must
read
the
and
.
your
git config
CNCF CLA.
Scenario
Branch
main
v1.27
dev-1.27
branch.
#sig-docs
on Slack.
Note:
change to the same code sample in mu

ltiple languages, open a separate PR for
The
1 - Opening a pull request
Note:
.
section.
GitHub)]
0[ ] -.-
end
end
enabled
form.
Note:
Select
.
Select
.
The
The
Subject
The
Body
Leave the
Note:
.
Select
.
Go to the
tab.
#sig-docs
git
installed
reposi

tory] --&gt; 2[Create local clone
S[ ] -.-
end
T[ ] -.-
end
enabled
Select
Fork
.
:
cd
website
1
website
upstream
remote:
cd
website
origin
and
upstream
and
's
:
Note:
.
main
with
.
.
.
.
#sig-docs
branch is
:
git status
git status

Note:
in your commit message. You can 

add those to the pull request
, which can
Note:
Ctrl+C
,
hugo
Hugo
.
1
cd
Ctrl+C
,
end
to your PR]
end
enabled
Select
.
Select
.
From the
From the
compare
Select
.
Title
or
in the
Select the
button.
.
Netlify
.
Details
Details
website with your changes applied. This 

is how reviewers check your changes.
.
-a
--amend
Use
Note:
If you use
Note:
,
#sig-docs
's
git status
,
, and
===
.
Note:
.
git status
Note:
,
#sig-docs
Commits
git log
Note:
vim
-i

...

pick
squash
and
pick
.
Note:
.
To:
and
into
as a part of the ti

meline.
The
comments.
README.md
,
, and
Read
Reviewing
2 - Documenting a feature for a release
#sig-docs
.
the
community
in the
, which is
release.
Beta

 and Stable features are generally a higher documentation priority than
Open a
draft
dev-1.27
,
then click
.
PR(s) and
issue(s).
.
#sig-docs
table
to
dev-1.27
3 - Submitting blog posts and case studies
broadly to the Kubernetes community. Appropriate

 blog content includes:
.
Check the
for
blog team
The
license
Consider concentrating the long technical content as a

 call to action of the blog post, and
Hugo
generator
.
.
Do
not
causes
---
layout
:
blog
title
:
date
:
slug
:
---
.
draft post
evergreen
:
true
Tutorials
CNCF
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggreg

ation Layer
Tasks
Debug Pods
De

termine the Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

ndards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

r Configuration (v1)
Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Access Applications in a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Configure load balancing, port forwarding, or setup firewall or DNS configurations to access applications in a cluster.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
1 - Deploy and Access the Kubernetes Dashboard
.
Warning:
kubectl
.
The UI can
only
Note:
not
namespace
Dashboard lets you create and deploy a containerized application as a Deployment and optional Service with a simple wizard.
manifest
Click the
CREATE
App name
A
label
added to the Deployment and S

ervice, if any, that will be deployed.
namespace
.
A
Service
Service
Note:
Labels
: Default
labels
to be used
Example:
Namespace
.
:
.
file.
and
:
for the container. By default, Pods run with unbounded CPU and memory limits.
and
:
.
Privileged containers can make use of capabilities like man

ipulating the network stack and accessing devices.
.
syntax.
API
default
Workloads
Services
Storage
.
2 - Accessing Clusters
kubectl
.
a
,
Many of the
examples
kubectl
.
curl or wget, or a browser, there are several ways to locate and a

uthenticate:
=
8080
See
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Use
and
metadata:
EOF
while
'^token'
do
echo
&gt;&amp;
2
sleep
1
done
APISERVER
=
$(
":"
| tr -d
" "
)
TOKEN
=
$(
'^token'
| cut -f2 -d
':'
| tr -d
" "
)
curl
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Using
jsonpath
:
APISERVER
=
$(
jsonpath
=
)
TOKEN
=
$(
jsonpath
=
)
curl
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
~/.kube
for this.
Go
and
Python
Go client
, see
example
.
If the applica

tion is deployed as a Pod in the cluster, please refer to the
.
To use
. See
example
.
There are
.
The
:
The
:
The
kube proxy
:
3 - Configure Access to Multiple Clusters
This page shows how to configure access to multiple clusters by using
configuration files. After your clusters, users, and contexts are defined in
one or more configuration files, you can quickly switch between clusters by using the
command.
Note:
A file that is used to configure access to a cluster is sometimes called
a
.
Warning:
Only use kubeconfig files from trusted sources. Using a specially-crafted kubeconfig
file could result in malicious code execution or file exposure.
minikube
To check that
kubectl
is installed,
run
kubectl version --client
. The kubectl version should be
within one minor version
of your
cluster's API server.
Define clusters, us

ers, and contexts
Suppose you have two clusters, one for development work and one for scratch work.
In the
cluster, your frontend developers work in a namespace called
frontend
,
and your storage developers work in a namespace called
storage
. In your
scratch
cluster,
developers work in the default namespace, or they create auxiliary namespaces as they
see fit. Access to the development cluster requires authentication by certificate. Access
to the scratch cluster requires authentication by username and password.
Create a directory named
config-exercise
. In your
directory, create a file named
config-demo
:
v1
kind
:
Config
:
{}
clusters
:
-
cluster
:
name
:
-
cluster
:
name
:
scratch
users
:
-
name
:
developer
-
name
:
experimenter
contexts
:
-
context
:
name
:
dev-frontend
-
context
:
name
:
dev-storage
-
context
:
name
:
exp-scratch
A configuration file describes clusters, users, and contexts. Your
file
has the framework to describe two clusters, two users, and three contexts.
Go to your
directory. Enter these commands to add cluster details to
your configuration file:
kubectl config --kubeconfig
=
config-demo set-cluster development --server
=
https://1.2.3.4 --certificate-authority
=
fake-ca-file
=
config-demo set-cluster scratch --server
=
https://5.6.7.8 --insecure-skip-tls-verify
Add user details to your configuration file:
Caution:
Storing passwords in Kubernetes client config is risky. A better alternative would be to use a credential plugin and store them separately. See:
=
config-demo set-credentials developer --client-certificate
=
fake-cert-file --client-key
=
fake-key-seefile
=
config-demo set-credentials experimenter --username
=
exp --password
=
some-password
Note:
To delete a user you can run
kubectl --kubeconfig=config-demo config unset users.&lt;name&gt;
To remove a cluster, you can run
kubectl --kubeconfig=config-demo config unset clusters.&lt;name&gt;
To remove a context, you can run
kubectl --kubeconfig=config-demo config unset contexts.&lt;name&gt;
Add context details to your configuration file:
=
config-demo set-context dev-frontend --cluster
=
development --namespace
=
frontend --user
=
developer
=
config-demo set-context dev-storage --cluster
=
=
storage --user
=
developer
=
config-demo set-context exp-scratch --cluster
=
scratch --namespace
=
default --user
=
Open your
file to see the added details. As an alternative to opening the
file, you can use the
config view
command.
=
config-demo view
The output shows the two clusters, two users, and three contexts:
:
v1
clusters
:
-
cluster
:
:
server
:
name
:
-
cluster
:
:
true
server
:
https://5.6.7.8
name
:
scratch
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
storage
user
:
developer
name
:
-
context
:
cluster
:
scratch
namespace
:
default
user
:
name
:
:
""
kind
:
Config
:
{}
users
:
-
name
:
developer
user
:
:
fake-cert-file
:
fake-key-file
-
name
:
user
:
# Documentation note (this comment is NOT part of the command output).
# Storing passwords in Kubernetes client config is risky.
# A better alternative would be to use a credential plugin
# and store the credentials separately.
# See https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
password
:
username
:
exp
The
,
an

d
above are the placeholders
for the pathnames of the certificate files. You need to change these to the actual pathnames
of certificate files in your environment.
Sometimes you may want to use Base64-encoded data embedded here instead of separate
certificate files; in that case you need to add the suffix
-data
to the keys, for example,
,
,
.
Each context is a triple (cluster, user, namespace). For example, the
context says, "Use the credentials of the
developer
user to access the
frontend
namespace of the
cluster".
Set the current context:
=
config-demo use-context dev-frontend
Now whenever you enter a
kubectl
command, the action will apply to the cluster,
and namespace listed in the
context. And the command will use
the credentials of the user listed in the
context.
To see only the configuration information associated with
the current context, use the
--minify
flag.
=
config-demo view --minify
The output shows configuration information associated with the
context:
:
v1
clusters
:
-
cluster
:
:
server
:
name
:
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
:
kind
:
Config
:
{}
users
:
-
name
:
developer
user
:
:
:
Now suppose you want to work for a while in the scratch cluster.
Change the current context to
:
=
config-demo use-context exp-scratch
Now any
kubectl
command you give will apply to the default namespace of
the
scratch
cluster. And the command will use the credentials of the user
context.
View configuration associated with the new current context,
.
=
Finally, suppose you want to work for a while in the
storage
cluster.
:
=
config-demo use-context dev-storage
.
=
Create a second configuration file
In your
config-demo-2
:
v1
kind
:
Config
:
{}
contexts
:
-
context
:
cluster
:
namespace
:
ramp
user
:
developer
name
:
dev-ramp-up
The preceding configuration file defines a new context named
.
Set the KUBECONFIG environment variable
See whether you have an environment variable named
. If so, save the
current value of your
environment variable, so you can restore it later.
Linux
export
KUBECONFIG_SAVED
=
"
"
Windows PowerShell
$Env:KUBECONFIG_SAVED
=
$ENV:KUBECONFIG
The
environment variable is a list of paths to configuration files. The list is
colon-delimited for Linux and Mac, and semicolon-delimited for Windows. If you have
a
environment variable, familiarize yourself with the configuration files
in the list.
Temporarily append two paths to your
environment variable. For example:
Linux
export
=
"
${
}
:config-demo:config-demo-2"
$Env:KUBECONFIG
=(
"config-demo;config-demo-2"
)
In your
directory, enter this command:
The output shows merged information from all the files listed in your
environment variable. In particular, notice that the merged information has the
context from the
file and the three contexts from
the
file:
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
-
context
:

cluster
:
namespace
:
ramp
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
storage
user
:
developer
name
:
-
context
:
cluster
:
scratch
namespace
:
default
user
:
name
:
For more information about how kubeconfig files are merged, see
Explore the $HOME/.kube directory
If you already have a cluster, and you can use
kubectl
to interact with
the cluster, then you probably have a file named
config
in the
Go to
, and see what files are there. Typically, there is a file named
config
. There might also be other configuration files in this directory. Briefly
familiarize yourself with the contents of these files.
Append $HOME/.kube/config to your KUBECONFIG environment variable
If you have a
file, and it's not already listed in your
environment variable, append it to your
environment variable now.
Linux
export
=
"
${
}
:
${
HOME
}
/.kube/config"
Windows Powershell
=
"$Env:KUBECONFIG;$HOME\.kube\config"
View configuration information merged from all the files that are now listed
in your
environment variable. In your config-exercise directory, enter:
Clean up
Return your
environment variable to its original value. For example:
Linux
export
=
"
$KUBECONFIG_SAVED
"
=
$ENV:KUBECONFIG_SAVED
Check the subject represented by the kubeconfig
It is not always obvious what attributes (username, groups) you will get after authenticating to the cluster.
It can be even more challenging if you are managing more than one cluster at the same time.
There is a
kubectl
alpha subcommand command to check subject attributes, such as username,
for your selected Kubernetes client context:
.
Read
to learn about this in more detail.
4 - Use Port Forwarding to Access Applications in a Cluster
to connect to a MongoDB
server running in a Kubernetes cluster. This type of connection can be useful
for database debugging.
minikube
.
Install
MongoDB Shell
.
Creating MongoDB deployment and service
Create a Deployment that runs MongoDB:
kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-deployment.yaml
The output of a successful command verifies that the deployment was created:
deployment.apps/mongo created
View the pod status to check that it is ready:
The output displays the pod created:
NAME                     READY   STATUS    RESTARTS   AGE
mongo-75f59d57f4-4nd6q   1/1     Running   0          2m4s
View the Deployment's status:
The output displays that the Deployment was created:
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
mongo   1/1     1            1           2m21s
The Deployment automatically manages a ReplicaSet.
View the ReplicaSet status using:
kubectl get replicaset
The output displays that the ReplicaSet was created:
NAME               DESIRED   CURRENT   READY   AGE
mongo-75f59d57f4   1         1         1       3m12s
Create a Service to expose MongoDB on the network:
kubectl apply -f https://k8s.io/examples/application/mongodb/mongo-service.yaml
Th

e output of a successful command verifies that the Service was created:
service/mongo created
Check the Service created:
kubectl get service mongo
The output displays the service created:
NAME    TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
mongo   ClusterIP   10.96.41.183   &lt;none&gt;        27017/TCP   11s
Verify that the MongoDB server is running in the Pod, and listening on port 27017:
# Change mongo-75f59d57f4-4nd6q to the name of the Pod
kubectl get pod mongo-75f59d57f4-4nd6q --template
=
'{{(index (index .spec.containers 0).ports 0).containerPort}}{{"\n"}}'
The output displays the port for MongoDB in that Pod:
27017
27017 is the TCP port allocated to MongoDB on the internet.
Forward a local port to a port on the Pod
allows using resource name, such as a pod name, to select a matching pod to port forward to.
kubectl port-forward mongo-75f59d57f4-4nd6q 28015:27017
which is the same as
kubectl port-forward pods/mongo-75f59d57f4-4nd6q 28015:27017
or
kubectl port-forward deployment/mongo 28015:27017
or
kubectl port-forward replicaset/mongo-75f59d57f4 28015:27017
or
kubectl port-forward service/mongo 28015:27017
Any of the above commands works. The output is similar to this:
Forwarding from 127.0.0.1:28015 -&gt; 27017
Forwarding from [::1]:28015 -&gt; 27017
Note:
does not return. To continue with the exercises, you will need to open another terminal.
Start the MongoDB command line interface:
mongosh --port
28015
At the MongoDB command line prompt, enter the
ping
command:
db.runCommand( { ping: 1 } )
A successful ping request returns:
{ ok: 1 }
Optionally let
kubectl
choose the local port
If you don't need a specific local port, you can let
kubectl
choose and allocate
the local port and thus relieve you from having to manage local port conflicts, with
the slightly simpler syntax:
kubectl port-forward deployment/mongo :27017
The
kubectl
tool finds a local port number that is not in use (avoiding low ports numbers,
because these might be used by other applications). The output is similar to:
Forwarding from 127.0.0.1:63753 -&gt; 27017
Forwarding from [::1]:63753 -&gt; 27017
Connections made to local port 28015 are forwarded to port 27017 of the Pod that
is running the MongoDB server. With this connection in place, you can use your
local workstation to debug the database that is running in the Pod.
Note:
is implemented for TCP ports only.
The support for UDP protocol is tracked in
issue 47862
.
.
5 - Use a Service to Access an Application in a Cluster
This page shows how to create a Kubernetes Service object that external
clients can use to access an application running in a cluster. The Service
provides load balancing for an application that has two running instances.
minikube
Run two instances of a Hello World application.
Create a Service object that exposes a node port.
Creating a service for an application running in two pods
Here is the configuration file for the application Deployment:
service/access/hello-application.yaml
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
run
:
replicas
:
2
template
:
metadata
:
labels
:
run
:
spec
:
:
-
name
:
image
:
ports
:
-
:
8080
protocol
:
TCP
Run a Hello

 World application in your cluster:
Create the application Deployment using the file above:
kubectl apply -f https://k8s.io/examples/service/access/hello-application.yaml
.
The ReplicaSet has two
Pods
=
NodePort --name
=
Name:                   example-service
Labels:
run
=
Selector:
run
=
Type:                   NodePort
IP:                     10.32.0.16
Port:                   &lt;unset&gt; 8080/TCP
TargetPort:             8080/TCP
NodePort:               &lt;unset&gt; 31496/TCP
Endpoints:              10.200.1.4:8080,10.200.2.5:8080
Make a note of the NodePort value for the service. For example,
in the preceding output, the NodePort value is 31496.
List the pods that are running the Hello World application:
=
"run=load-balancer-example"
--output
=
wide
NAME                           READY   STATUS    ...  IP           NODE
hello-world-2895499144-bsbk5   1/1     Running   ...  10.200.1.4   worker1
hello-world-2895499144-m1pwt   1/1     Running   ...  10.200.2.5   worker2
Get the public IP address of one of your nodes that is running
a Hello World pod. How you get this address depends on how you set
up your cluster. For example, if you are using Minikube, you can
see the node address by running
. If you are
using Google Compute Engine instances, you can use the
gcloud compute instances list
command to see the public addresses of your
nodes.
On your chosen node, create a firewall rule that allows TCP traffic
on your node port. For example, if your Service has a NodePort value of
31568, create a firewall rule that allows TCP traffic on port 31568. Different
cloud providers offer different ways of configuring firewall rules.
Use the node address and node port to access the Hello World application:
curl http://&lt;public-node-ip&gt;:&lt;node-port&gt;
where
&lt;public-node-ip&gt;
is the public IP address of your node,
and
&lt;node-port&gt;
is the NodePort value for your service. The
response to a successful request is a hello message:
Using a service configuration file
As an alternative to using
, you can use a
service configuration file
to create a Service.
kubectl delete services example-service
Follow the
tutorial.
6 - Connect a Frontend to a Backend Using Services
frontend
and a
backend
Service
object.
hello
object.
nginx
You need to have a Kubernetes

 cluster, and the kubectl command-line tool must
minikube
.
, which
NodePort
instead.
---
:
apps/v1
kind
:
metadata
:
name
:
backend
spec
:
selector
:
:
app
:
hello
tier
:
backend
track
:
stable
replicas
:
3
template
:
metadata
:
labels
:
app
:
hello
tier
:
backend
track
:
stable
spec
:
:
-
name
:
hello
image
:
ports
:
-
name
:
http
:
80
...
Selector:                       app=hello,tier=backend,track=stable
   hello:
Events:
...
hello
selectors
to find
---
:
v1
kind
:
Service
metadata
:
name
:
hello
spec
:
selector
:
app
:
hello
tier
:
backend
ports
:
-
protocol
:
TCP
port
:
80
:
http
...
hello
routes
app: hello
and
.
backend
hello
hello
, which

 is the value
of the
name
hello
}

server {
    listen 80;

    }
}
---
:
v1
kind
:
Service
metadata
:
name
:
frontend
spec
:
selector
:
app
:
hello
tier
:
frontend
ports
:
-
protocol
:
"TCP"
port
:
80
:
80
type
:
...
---
:
apps/v1
kind
:
metadata
:
name
:
frontend
spec
:
selector
:
:
app
:
hello
tier
:
frontend
track
:
stable
replicas
:
1
template
:
metadata
:
labels
:
app
:
hello
tier
:
frontend
track
:
stable
spec
:
:
-
name
:
nginx
image
:
lifecycle
:
preStop
:
exec
:
command
:
[
,
"-s"
,
"quit"
]
...
Create the frontend Deployment and Service:
Note:
be to use a
ConfigMap
,
frontend
:
heading:
frontend
cluster.
${
}
{
"message"
:
"Hello"
}
Services
7 - Create an External Load Balancer
Service
, you have
externally-accessible IP address that sends traffic to the correct port on

 your cluster
nodes,
.
Ingress
Ingress
minikube
type
:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
example
ports
:
-
port
:
8765
:
9376
type
:
its
flag:
=
8765
=
9376
\
--name
=
=
named
example
).
reference
.
kubectl
:
LoadBalancer Ingress:     192.0.2.89
.
Note:
source IP
.spec
options:
Cluster
Local
.
Cluster
Local
Service
type
is set
to
Local
.
Setting
to Local

 in the Service manifest
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
example
ports
:
-
port
:
8765
:
9376
:
Local
type
:
Where
or
type
.
type
type
within the cluster and extends it by programming the (external to Kubernetes) load balancer with entries for the nodes
Follow the
tutorial
Read about
Service
Read about
Ingress
8 - List All Container Images Running in a Cluster
This page shows how to use kubectl to list all of the Container images
for Pods running in a cluster.
minikube
.
In this exercise you will use kubectl to fetch all of the Pods
running in a cluster, and format the output to pull out the list
of Containers for each.
List all Container images in all namespaces
Fetch all Pods in all namespaces using
Format the output to include only the list of Container image names
using
-o jsonpath={.items[*].spec.containers[*].image}
. This will recursively parse out the
image
field from the returned json.
See the
jsonpath reference
for further information on how to use jsonpath.
Format the output using standard tools:
tr
,
sort
,
uniq
Use
tr
to replace spaces with newlines
Use
sort
to sort the results
Use
uniq
to aggregat

e image counts
jsonpath
=
"{.items[*].spec.containers[*].image}"
|
\
tr -s
'[[:space:]]'
'\n'
|
\
sort |
\
uniq -c
The jsonpath is interpreted as follows:
.items[*]
: for each returned value
.spec
: get the spec
.containers[*]
: for each container
.image
: get the image
Note:
When fetching a single Pod by name, for example
kubectl get pod nginx
,
the
.items[*]
portion of the path should be omitted because a single
Pod is returned instead of a list of items.
List Container images by Pod
The formatting can be controlled further by using the
range
operation to
iterate over elements individually.
jsonpath
=
'{range .items[*]}{"\n"}{.metadata.name}{":\t"}{range .spec.containers[*]}{.image}{", "}{end}{end}'
|
\
sort
List Container images filtering by Pod label
To target only Pods matching a specific label, use the -l flag. The
following matches only Pods with labels matching
app=nginx
.
jsonpath
=
-l
app
=
nginx
List Container images filtering by Pod namespace
To target only pods in a specific namespace, use the namespace flag. The
following matches only Pods in the
kubectl get pods --namespace kube-system -o
jsonpath
=
List Container images using a go-template instead of jsonpath
As an alternative to jsonpath, Kubectl supports using
go-templates
for formatting the output:
kubectl get pods --all-namespaces -o go-template --template
=
"{{range .items}}{{range .spec.containers}}{{.image}} {{end}}{{end}}"
Reference
Jsonpath
reference guide
Go template
9 - Set up Ingress on Minikube with the NGINX Ingress Controller
An
Ingress
is an API object that defines rules which allow external access
to services in a cluster. An
fulfills the rules set in the Ingress.
This page shows you how to set up a simple Ingress which routes requests to Service web or web2 depending on the HTTP URI.
minikube
Your Kubernetes server must be at or later than version 1.19.
.
If you are using an older Kubernetes version, switch to the documentation
for that version.
Create a Minikube cluster
Using Katacoda
Locally
If you already
installed Minikube
locally, run
to create a cluster.
Enable the Ingress controller
To enable the NGINX Ingress controller, run the following command:
enable
ingress
Verify that the NGINX Ingress controller is running
minikube v1.19 or later
minikube v1.18.1 or earlier
kubectl get pods -n ingress-nginx
Note:
It can take up to a minute before you see these pods running OK.
NAME                                        READY   STATUS      RESTARTS    AGE
ingress-nginx-admission-create-g9g49        0/1     Completed   0          11m
ingress-nginx-admission-patch-rqp78         0/1     Completed   1          11m
ingress-nginx-controller-59b45fb494-26npt   1/1     Running     0          11m
kubectl get pods -n kube-system
Note:
default-http-backend-59868b7dd6-xb8tq       1/1       Running   0          1m
kube-addon-manager-minikube                 1/1       Running   0          3m
kube-dns-6dcb57bcc8-n4xd4                   3/3       Running   0          2m
kubernetes-dashboard-5498ccf677-b8p5h       1/1    

   Running   0          2m
nginx-ingress-controller-5984b97644-rnkrg   1/1       Running   0          1m
storage-provisioner                         1/1       Running   0          2m
Make sure that you see a Pod with a name that starts with
nginx-ingress-controller-
.
Deploy a hello, world app
Create a Deployment using the following command:
kubectl create deployment web --image
=
deployment.apps/web created
Expose the Deployment:
kubectl expose deployment web --type
=
NodePort --port
=
8080
service/web exposed
Verify the Service is created and is available on a node port:
kubectl get service web
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
web       NodePort   10.104.133.249   &lt;none&gt;        8080:31637/TCP   12m
Visit the Service via NodePort:
minikube service web --url
http://172.17.0.15:31637
Note:
Katacoda environment only: at the top of the terminal panel, click the plus sign, and then click
. Enter the NodePort, in this case
31637
.
Hello, world!
Version: 1.0.0
Hostname: web-55b8c6998d-8k564
You can now access the sample app via the Minikube IP address and NodePort. The next step lets you access
the app using the Ingress resource.
Create an Ingress
The following manifest defines an Ingress that sends traffic to your Service via hello-world.info.
Create
example-ingress.yaml
from the following file:
service/networking/example-ingress.yaml
:
kind
:
Ingress
metadata
:
name
:
example-ingress
:
:
/$1
spec
:
rules
:
-
host
:
hello-world.info
http
:
paths
:
-
path
:
/
pathType
:
Prefix
backend
:
service
:
name
:
web
port
:
number
:
8080
Create the Ingress object by running the following command:
kubectl apply -f https://k8s.io/examples/service/networking/example-ingress.yaml
ingress.networking.k8s.io/example-ingress created
Verify the IP address is set:
kubectl get ingress
Note:
This can take a couple of minutes.
You should see an IPv4 address in the ADDRESS column; for example:
NAME              CLASS    HOSTS              ADDRESS        PORTS   AGE
example-ingress   &lt;none&gt;   hello-world.info   172.17.0.15    80      38s
Add the following line to the bottom of the
/etc/hosts
file on
your computer (you will need administrator access):
172.17.0.15 hello-world.info
Note:
If you are running Minikube locally, use
minikube ip
to get the external IP. The IP address displayed within the ingress list will be the internal IP.
After you make this change, your web browser sends requests for
hello-world.info URLs to Minikube.
Verify that the Ingress controller is directing traffic:
curl hello-world.info
You should see:
Note:
If you are running Minikube locally, you can visit hello-world.info from your browser.
Create a second Deployment
Create another Deployment using the following command:
kubectl create deployment web2 --image
=
deployment.apps/web2 created
Expose the second Deployment:
kubectl expose deployment web2 --port
=
8080
--type
=
NodePort
service/web2 exposed
Edit the existing Ingress
Edit the existing
manifest, and add the
following lines at the end:
-
path
:
/v2
pathType
:
Prefix
backend
:
service
:
name
:
web2
port
:
number
:
8080
Apply the changes:
kubectl apply -f example-ingress.yaml
ingress.networking/example-ingress configured
Test your Ingress
Access the 1st version of the Hello World app.
Access the 2nd version of the Hello World app.
curl hello-world.info/v2
The output is similar

 to:
Version: 2.0.0
Hostname: web2-75cd47646f-t8cjk
Note:
If you are running Minikube locally, you can visit hello-world.info and hello-world.info/v2 from your browser.
Ingress
Services
10 - Communicate Between Containers in the Same Pod Using a Shared Volume
minikube
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
Never
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
image
:
nginx
:
-
name
:
mountPath
:
-
name
:
image
:
debian
:
-
name
:
mountPath
:
/pod-data
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
]
.
.
/pod-data
kind: Pod
metadata:
  ...
  ...
spec:
  ...

    image: debian
    ...
        ...
    ...

    ...
    ...
    state:
      running:
    ...
curl
Helper and primary applications often need to communicate wit

h each other.
.
.
See
.
See
See
Volume
.
See
Pod
.
11 - Configure DNS for a Cluster
.
12 - Access Services Running on Clusters
minikube
.
nodes
,
pods
and
services
all have
NodePort
or
services
and
Described
here
.
.
not others. Browsers and other tools may or may not be installed. Cluster DNS may not work.
with the
command:
elasticsearch-logging is running

 at https://192.0.2.1/api/v1/namespaces/kube-system/services/elasticsearch-logging/proxy
at
.
Note:
See
http://
/proxy
port_name
port_name
https:
:
Examples
{
:
,
"status"
:
"yellow"
,
:
false
,
:
1
,
:
1
,
:
5
,
:
5
,
:
0
,
:
0
,
:
5
}
https
Some web apps may not work, particularly those with client side javascript that construct URLs in a
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a l

ist of trademarks of The Linux Foundation, please see our

Tutorials | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Tutorials
1:
2:
2.1:
2.1.1:
2.1.2:
2.2:
2.2.1:
2.2.2:
2.3:
2.3.1:
2.3.2:
2.4:
2.4.1:
2.4.2:
2.5:
2.5.1:
2.5.2:
2.6:
2.6.1:
2.6.2:
3:
3.1:
3.1.1:
3.1.2:
3.2:
4:
Security
4.1:
4.2:
4.3:
4.4:
5:
5.1:
5.2:
6:
6.1:
6.2:
6.3:
6.4:
7:
Services
7.1:
7.2:
This section of the Kubernetes documentation contains tutorials.
A tutorial shows how to accomplish a goal that is larger than a single
task
. Typically a tutorial has several sections,
each of which has a sequence of steps.
Before walking through each tutorial, you may want to bookmark the
Standardized Glossary
page for later references.
Basics
is an in-depth interactive tutorial that helps you understand the Kubernetes system and try out some basic Kubernetes features.
Introduction to Kubernetes (edX)
Configuring Redis Using a ConfigMap
Example: WordPress and MySQL with Persistent Volumes
Example: Deploying Cassandra with Stateful Sets
Running ZooKeeper, A CP Distributed System
Services
Security
Apply Pod Security Standards at Cluster level
Apply Pod Security Standards at Namespace level
AppArmor
seccomp
If you would like to write a tutorial, see
Content Page Types
for information about the tutorial page type.
1 - Hello Minikube
Note:
See
Click

Note:
.
30000
.
Note:
The
dashboard
.
The
dashboard
Ctrl+C
dashboard
--url
Pod
Use the
kubectl create deployment hello-node --image
=
=
8080
View the
kubectl
Note:
kubectl
.
Service
.
command:
=
=
8080
The
the
command.
.
8080
in services output. This po

rt number is randomly generated and it can be different for you. Type your number in the port number text box, then click Display Port. Using the example from earlier, you would type
30369
.
addons
:
enable
pod/kube-controller-manager-minikube        1/1       Running   0          34m

Disable
:
Clean up
.
.
.
2 - Learn Kubernetes Basics
The tutorials use Katacoda to run a virtual terminal in your web browser that runs Minikube, a small-scale local deployment of Kubernetes that can run anywhere. There's no need to install any software or configure anything; each int

eractive tutorial runs directly out of your web browser itself.
2.1 - Create a Cluster
cluster
2.1.1 - Using Minikube to Create a Cluster
The
Nodes
Summary:
Minikube
etcd
A Kubernetes cluster can be deployed on either physical or virtual machines. To get started with Kubernetes development, you can use Minikube. Minikube is a lightweight Kubernetes implementation that creates a VM on your local machine and deploys a simple cluster containing only one node. Minikube is available for Linux, macOS, and Windows 

systems. The Minikube CLI provides basic bootstrapping operations for working with your cluster, including start, stop, status, and delete. For this tutorial, however, you'll use a provided online terminal with Minikube pre-installed.
›
2.1.2 - Interactive Tutorial - Creating a Cluster
Home
2.2 - Deploy an App
2.2.1 - Using kubectl to Create a Deployment
cluster.
Summary:
Kubectl
Kubectl
5
and
6
).
›
2.2.2 - Interactive Tutorial - Deploying an App
.
Home
2.3 - Explore Your App
2.3.1 - Viewing Pods and Nodes
2
Pod
to host your application instance. A Pod is a Kubernetes abstraction that represents a group of one or more application containers (such as Docker), and some shared resources for those containers. Those resources includ

e:
Summary:
Pods
Nodes
Nodes
Node
In Module
2
›
2.3.2 - Interactive Tutorial - Exploring Your App
Home
2.4 - Expose Your App Publicly
2.4.1 - Using a Service to Expose Your App
Pods
lifecycle
might then dynamically drive the cluster back to the desired state via the creation of new Pods to keep your application running. As another example, consider an image-processing backend with 3 replicas. Those replicas are exchangeable; the front-end system should n

ot care about backend replicas or even if a Pod is lost and recreated. That said, each Pod in a Kubernetes cluster has a unique IP address, even Pods on the same Node, so there needs to be a way of automatically reconciling changes among Pods so that your applications continue to function.
selector
type
ClusterIP
NodePort
CNAME
kube-dns
.
selector
selector
.
Summary
›
2.4.2 - Interactive Tutorial - Exposing Your App
Home
2.5 - Scale Your App
2.5.1 - Running Multiple Instances of Your App
Service
Scaling
is accomplished by changing the

 number of replicas in a Deployment
Summary:
Previous
Next
›
2.5.2 - Interactive Tutorial - Scaling Your App
Home
2.6 - Update Your App
2.6.1 - Performing a Rolling Update
Summary:
Previous
Next
›
2.6.2 - Interactive Tutorial - Updating Your App
3 - Configuration
3.1 - Example: Configuring a Java Microservice
3.1.1 - Externalizing config using MicroProfile, ConfigMaps and Secrets
Before yo

u begin
here
.
here
.
3.1.2 - Interactive Tutorial - Configuring a Java Microservice
3.2 - Configuring Redis using a ConfigMap
task.
minikube
.
kubectl
.
Real World Example: Configuring Redis using a ConfigM

ap
cat
metadata:
data:
EOF
config
The
key
and
path
under
on the
config
volume.
The
config
by
.
from the
:
v1
kind
:
Pod
metadata
:
name
:
redis
spec
:
:
-
name
:
redis
image
:
command
:
-
env
:
-
name
:
MASTER
value
:
"true"
ports
:
-
:
6379
resources
:
limits
:
cpu
:
"0.1"
:
-
mountPath
:
name
:
data
-
mountPath
:
name
:
config
volumes
:
-
name
:
data
emptyDir
:
{}
-
name
:
config
configMap
:
name
:
items
:
-
key
:
path
:

key in the
key:
Data
====
Use
redis-cli
kubectl
exec
Check
maxmemory
:
1
)
2
)
"0"
:
:
1
)
2
)
:
v1
kind
:
ConfigMap
metadata
:
name
:
data
:
:
|
Data
====
----
redis-cli
via
kubectl
exec
Check
maxmemory
:
1
)
2
)
"0"
Returns:
1
)
2
)
kubectl apply -f https://raw.githubusercontent.com/kubernetes/website/main/content/en/examples/pods/con

fig/redis-pod.yaml
kubectl
exec
Check
maxmemory
:
1
)
2
)
"2097152"
:
1
)
2
)
.
4 - Security
4.1 - Apply Pod Security Standards at the Cluster Level
Note
.
baseline
.
KinD
kubectl
enforce
,
audit
, and
warn
.

Use
=
\
=
Baseline
=
\
=
baseline
namespace/local-path-sto

rage labeled
=
\
=
baseline
and
latest
version:
baseline
enforce
mode.
warn
and
audit
mode.
The
baseline
plugins:
    defaults:
EOF
Note:
v1beta1
.
v1alpha1
.
nodes:
  - |
            mountPath: /etc/confi

g
EOF
Note:
add
/tmp
.

kind: Pod
metadata:
spec:
      ports:
EOF
Clean up
Run
and
created.
Run a
4.2 - Apply Pod Security Standards at the Namespace Level
Note
baseline
.
KinD
kubectl
Create a
KinD
cluster as follo

ws:

example
:
\
=
baseline
\
=
latest
enforce
the
baseline
warn
and
audit
for
\
=
baseline
\
=
latest
\
=
\
=
latest
\
=
\
=
latest
example
cat
kind: Pod
metadata:
spec:
ports:
EOF
example
default
example
default
namespace
Clean up
Run
Run a
Apply
baseline
enforce
warn
and
audit
mode.
4.3 - Restrict a Container's Access to Resources with AppArmor
application to reduce

 its potential attack surface and provide greater in-depth defense. It is
enforcing
complain
Make sure:
will
=
jsonpath
=
file:
Y
Note:
Docker
,
CRI-O
or
.
.
=
jsonpath
=
gke-test-default-pool-239f5d02-gyn2: kubelet is posting ready status. AppArmor 

enabled
Note:
).
:
Where
See the
kubectl
exec
Example
flags
=(
)
{
file,
deny /** w,
}
.
NODES
=(
)
for
NODE in
${
NODES
[*]
}
;
do
ssh
$NODE
file,
deny /** w,
}
EOF'
done
:
v1
kind
:
Pod
metadata
:
name
:
:
:
spec
:
:
-
name
:
hello
image
:
command
:
[
"sh"
,
"-c"
,
]
14s        14s         1         hello-a

pparmor   Pod                                Normal    Scheduled   {default-scheduler }                           Successfully assigned hello-apparmor to gke-test-default-pool-239f5d02-gyn2
kubectl
exec
kubectl
exec
:
v1
kind
:
Pod
metadata
:
name
:
:
:
spec
:
:
-
name
:
hello
image
:
command
:
[
"sh"
,
"-c"
,
]
EOF
IP:
  hello:
    Image ID:
    Port:
    Command:
      sh
      -c
    Mounts:
Volumes:
Events:
Pod Cannot enforce AppArmor: profile "k8s-apparmor-example-allow-wr

ite" is not loaded
Through a
DaemonSet
here
.
image.
Example
.
error.
Note:
and
.
bane
dmesg
.
key
:
Where
value
.
4.4 - Restrict a Container's Syscalls with seccomp
node
Learn how to apply a s

eccomp profile to a container
Observe auditing of syscalls made by a container process
kind
and
kubectl
.
curl
Note:
run as
.
profiles/
{
:
}
{
:
}
{
:
,
: [
,
,
],
: [
{
"names"
: [
"accept4"
,
,
,
"futex"
,
"madvise"
,
,
,
,
"vfork"
,
"mmap"
,
"read"
,
"write"
,
"close"
,
,
,
"munmap"
,
"brk"
,
,
,
,
"gettid"
,
"clone"
,
"bind"
,
"socket"
,
"openat"
,
,
,
,
"listen"
,
,
,
,
"connect"
,
"dup2"
,
,
"execve"
,
"exit"
,
"fcntl"
,
"getpid"
,
"getuid"
,
"ioctl"
,
,
,
"open"
,
"poll"
,
,
"sendto"
,
,
,
"writev"
],
"action"
:
}
]
}
kind
:
kind
:
Cluster
nodes
:
-
role
:
:
-
hostPath
:
:
kind.yaml
:
See
Nodes
within the
.
that configuration

:
=
kind.yaml
After the new Kubernetes cluster is ready, identify the Docker container running
docker ps
profiles/
docker
exec
enabled
.
Note:
.
.
feature
gate
command
file
kind
kind
provides
feature
:
kind
:
Cluster
:
:
:
true
nodes
:
-
role
:

image
:
:
- |
-
role
:
worker
image
:
:
- |
=
=
using
to run
worker:
docker
exec
\
{
:
,
: [
,
,
],
: [
{
"names"
: [
"..."
]
}
]
}
:
v1
kind
:
Pod
metadata
:
name
:
audit-pod
labels
:
app
:
audit-pod
spec
:
:
:
type
:
Localhost
:
:
-
name
:
image
:
args
:
-
:
:
false
Note:
Services
5678
curl
to run the
curl
docker
exec
in
tail
calls from
http-echo
:
http-echo
, and if you
curl
Jul  6 15:37:40 my-machi

ne kernel: [369128.669452] audit: type=1326 audit(1594067860.484:14536): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=51 compat=0 ip=0x46fe1f code=0x7ffc0000
http-echo
process by
syscall=
syscalls.
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
app
:
spec
:
:
:
type
:
Localhost
:
:
-
name
:
image
:
args
:
-
:
:
false
http-echo
seen in syslog of the first example where th

e profile set
. Now the profile is setting
,
sent to
syslog
.
:
v1
kind
:
Pod
metadata
:
name
:
fine-pod
labels
:
app
:
fine-pod
spec
:
:
:
type
:
Localhost
:
:
-
name
:
image
:
args
:
-
:
:
false
tail
http-echo
:
5678
Use
curl
docker
exec
syslog
.
Note:
.
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
app
:
spec
:
:
:
type
:
:
-
name
:
image
:
args
:
-
:
:
false
5 - Stateless Applications
5.1 - Exposing an External IP Address to Access an Application in a Cluster

Install
kubectl
.
,
Configure
kubectl
:
apps/v1
kind
:
metadata
:
labels
:
:
name
:
spec
:
replicas
:
5
selector
:
:
:
template
:
metadata
:
labels
:
:
spec
:
:
-
image
:
name
:
ports
:
-
:
8080
.
Pods
=
=
Note:
The
this page
Note:
Port
and
NodePort
Port
NodePort
is 32377.
=
wide
where
&lt;extern

al-ip&gt;
)
Port
will
.
5.2 - Example: Deploying PHP Guestbook application with Redis
ready)
Docker
Redis
Clean up.
minikube
.
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
replicas
:
1
selector
:
:
app
:
redis
template
:
metadata
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
:
-
name
:
leader
image
:
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
6379
file:
Service
to
Pods.
application/guestbook/redis-leader-service.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
ports
:
-
port
:
6379
:
6379
selector
:
app
:
redis
role
:
leader
tier
:
backend
file:
kubernetes     ClusterIP   10.0.0.1     &lt;n

one&gt;        443/TCP    1m
Note:
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
replicas
:
2
selector
:
:
app
:
redis
template
:
metadata
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
:
-
name
:
follower
image
:
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
6379
file:
Service
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
ports
:
-
port
:
6379
selector
:
app
:
redis
role
:
follower
tier
:
backend
file:
Note:
The guestbook app uses a PHP frontend. It is configured to communicate with
:
apps/v1
kind
:
metadata
:
name
:
frontend
spec
:
replicas
:
3
selector
:
:
app
:
guestbook
tier
:
frontend
template
:
metadata
:
labels
:
app
:
guestbook
tier
:
frontend
spec
:
:
-
name
:
php-redis
image
:
env
:
-
name
:
value
:
"dns"
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:

-
:
80
file:
app
=
tier
=
frontend
The
Redis
ClusterIP
.
ClusterIP
ClusterIP
.
Note:
.
:
v1
kind
:
Service
metadata
:
name
:
frontend
labels
:
app
:
guestbook
tier
:
frontend
spec
:
ports
:
-
port
:
80
selector
:
app
:
guestbook
tier
:
frontend
file:
8080
80
Note:
Run the following command to scale up the number

 of frontend Pods:
=
5
=
2
app
=
redis
app
=
redis
6 - Stateful Applications
6.1 - StatefulSet Basics
.
Pods
The
kubectl
Note:
tutorial.
concept.
,
nginx
web
.
:
v1
kind
:
Service
metadata
:

name
:
nginx
labels
:
app
:
nginx
spec
:
ports
:
-
port
:
80
name
:
web
clusterIP
:
None
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
web
spec
:
:
"nginx"
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
name
:
web
:
-
name
:
www
mountPath
:
:
-
metadata
:
name
:
www
spec
:
:
[
]
resources
:
requests
:
storage
:
1Gi
web.yaml
app
=
nginx
web.yaml
.
NGINX
nginx
web
web       2         1         20s
n
{0..n-1}
app
=
nginx
web-1
web-0
Pod is
Running
(see
Pod Phase
)
and
Ready
(see
type
in
).
Note:
.
app
=
nginx
.
.
Since the
web
web-0
and
web-1
.
hostname
for
i in
0
1;
do
kubectl
exec
"web-
$i
"
-- sh -c
;
done
web-0
web-1
Use
to execute
nslookup
dnsutils
package.
Using
nslookup
=
Never --rm

Address 1: 10.24

4.2.6
exit
)
app
=
nginx
app
=
nginx
app
=
nginx
Use
and
for
i in
0
1;
do
kubectl
exec
web-
$i
-- sh -c
;
done
web-0
web-1
then, run:

exit
)
(
Ready.
,
web-0
and
web-1
:
app
=
nginx
.
. The
spec
for
i in
0
1;
do
kubectl
exec
"web-
$i
"
-- sh -c
;
done
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
Note:
you will need to fix the permissions of the 

directory mounted by the
(due to a
),
curl
app
=
nginx
app
=
nginx
app
=
nginx
web-0
web-1
web-0
and
web-1
node
web-0
and
web-1
replicas
or
Scaling Up
app
=
nginx
to 5:
=
5
app
=
nginx
app
=
nginx
Wait for
web-4
and
web-3
app
=
nginx
web-3

     1/1       Terminating   0         42s
app
=
nginx
and
OnDelete
.
The
Patch the
web
web
=
'json'
-p
=
app
=
nginx -w
The Pods in the StatefulSet are updated in rever

se ordinal order. The
for
p in
0
1
2;
do
"web-
$p
"
; echo;
done
Note:
to view
partition
the
.
Patch the
web
field:
=
'json'
-p
=
app
=
nginx -w
partition
.
partition
above
.
Wait for
web-2
app
=
nginx -w
partition
web-2
the
partition
.
Delete the
web-1
Pod:
web-1
app
=
nginx -w
web-2    

 1/1       Running       0          2m
Get the
web-1
web-1
canary
the
partition
update.
2
0
:
app
=
nginx -w
for
p in
0
1
2;
do
"web-
$p
"
; echo;
done
partition
to
0
On Delete
The
OnDelete
to
OnDelete
.
deleted.
Use
=
orphan
app
=
nginx
web
has been deleted, all of the Pod

s are still Running and Ready.
Delete
web-0
:
app
=
nginx
As the
web
web-0
app
=
nginx
nginx
nginx
app
=
nginx
When the
web
web-0
.
Since
web-1
web-0
with
replicas
web-0
web-1
web-2
was
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
web-0
web-0
app
=
nginx
app
=
nginx
Note:
nginx
kubectl apply -f web.yam

l
files:
for
i in
0
1;
do
kubectl
exec
-i -t
"web-
$i
"
done
web-0
web-1
web-0
and
web-1
nginx
...and the
web
above.
Parallel
:
v1
kind
:
Service
metadata
:
name
:
nginx
labels
:
app
:
nginx
spec
:
ports
:
-
port
:
80
name
:
web
clusterIP
:
None
selector
:
app
:
nginx
---
:
apps/v1
kind
:
metadata
:
name
:
web
spec
:
:
"nginx"
:
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
name
:
web
:
-
name
:
www
mountPath
:
:
-
metadata
:
name
:
www
spec
:
:
[
]
resources
:
requests
:
storage
:
1Gi
of the
web
Parallel
.
app
=
nginx -w
app
=
nginx -w
web-0
and
web-1
=
4
kubectl
# sts is an abbreviation

 for statefulset
app
=
nginx -w
nginx
Service:
Note:
6.2 - Example: Deploying WordPress and MySQL with Persistent Volumes
A
(PV) is a piece of storage in the cluster that has been

 manually provisioned by an administrator, or dynamically provisioned by Kubernetes using a
. A
Warning:
Note:
Create a
with
Clean up
minikube
.
kubectl
MySQL and Wordpress each require a PersistentVolume to store data. Their PersistentVolumeClaims will be created at the deployment step.
Warning:
hostPath
hostPath
hostPath
/tmp
Note:
hostPath
Note:
this guide
.
A
Secret
kubectl
.
cat
literals:
EOF
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
ports
:
-
port
:
3306
selector
:
app
:
wordpress
tier
:
mysql
clusterIP
:
None
---
:
v1
kind
:
metadata
:
name
:
labels
:
app
:
wordpress
spe

c
:
:
resources
:
requests
:
storage
:
20Gi
---
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
selector
:
:
app
:
wordpress
tier
:
mysql
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
wordpress
tier
:
mysql
spec
:
:
-
image
:
mysql:5.6
name
:
mysql
env
:
-
name
:
valueFrom
:
:
name
:
key
:
password
ports
:
-
:
3306
name
:
mysql
:
-
name
:
mountPath
:
volumes
:
-
name
:
:
claimName
:
:
v1
kind
:
Service
metadata
:
name
:
wordpress
labels
:
app
:
wordpress
spec
:
ports
:
-
port
:
80
selector
:
app
:
wordpress
tier
:
frontend
type
:
---
:
v1
kind
:
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
:
resources
:
requests
:
storage
:
20Gi
---
:
apps/v1
kind
:
metadata
:
name
:
wordpress
labels
:
app
:
wordpress
spec
:
selector
:
:
app
:
wordpress
tier
:
frontend
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
wordpress
tier
:
frontend
spec
:
:
-
image
:
name
:
wordpress
env
:
-
name
:
value
:
-
name
:
valueFrom
:
:
name
:
key
:
password
ports
:
-
:
80
name
:
wordpress
:
-
name
:
mountPath
:
volumes
:
-
name
:
:
claimName
:
Add them to
file.
cat
EOF
The
1
9s
Note:
Note:
RUNNING
.

s wordpress
Note:
NodePort
Warning:
Jobs
6.3 - Example: Deploying Cassandra with a StatefulSet
state
).
.
Note:
node
ring
Nodes
.
seed list
Service
.
Use a
Pods
.
minikube
Pods
,
Services
, and
.
Caution:
Minikube
5120
--cpus
=
4
Service
Pods
:
v1
kind
:
Service
metadata
:
labels
:
app
:
cassandra
name
:
cassandra
spec
:
clusterIP
:
None
ports
:
-
port
:
9042
selector
:
app
:
cassandra
file:
kubectl get svc cas

sandra
cassandra
Note:
:
apps/v1
kind
:
metadata
:
name
:
cassandra
labels
:
app
:
cassandra
spec
:
:
cassandra
replicas
:
3
selector
:
:
app
:
cassandra
template
:
metadata
:
labels
:
app
:
cassandra
spec
:
:
1800
:
-
name
:
cassandra
image
:
:
Always
ports
:
-
:
7000
name
:
-
:
7001
name
:
-
:
7199
name
:
jmx
-
:
9042
name
:
cql
resources
:
limits
:
cpu
:
"500m"
memory
:
1Gi
requests
:
cpu
:
"500m"
memory
:
1Gi
:
:
add
:
- IPC_LOCK
lifecycle
:
preStop
:
exec
:
command
:
- /bin/sh
- -c
env
:
-
name
:
value
:
512M
-
name
:
value
:
100M
-
name
:
value
:
-
name
:
value
:
"K8Demo"
-
name
:
value
:
-
name
:
value
:
-
name
:
POD_IP
valueFrom
:
fieldRef
:
fieldPath
:
:
exec
:
command
:
- -c
:
15
:
5
:
-
name
:
mountPath
:
:
-
metadata
:
name
:
spec
:
:
[
]
:
fast
resources
:
requests
:
storage
:
1Gi
---
kind
:
:
metadata
:
name
:
fast
:
:
type
:
pd-ssd
file:
The
=
0
1m
0
8s
n

odetool
kubectl
exec
Use
replicas
field.
#
:
apps/v1
kind
:
metadata
:
:
:
1
labels
:
app
:
cassandra
name
:
cassandra
namespace
:
default
:
"323"
uid
:
spec
:
replicas
:
3
Warning:
grace
=
$(
=
jsonpath
=
)
\
app
=
cassandra
\
echo
"Sleeping
${
grace
}
seconds"
2
\
sleep
$grace
\
app
=
cassandra
app
=
cassandra
.
.
32
0.0.0.0
.
6.4 - Running ZooKeeper, A Distributed System Coordinator
on
,
,
and
.
Pods
You must have a cluster with at least four nodes, and each node requires at least 2 CPUs and 4 GiB of mem

ory. In this tutorial you will cordon and drain the cluster's nodes.
tutorial.
ZooKeeper
is a
Zab
The ensemble uses the Zab protocol to elect a leader, and the ensemble cannot write data until that election is complete. Once complete, the ensemble uses Zab to ensure that it replicates all writes to a quorum before it acknowledges and makes them visible to clients. Without respect to weighted quorums, a quorum is a majority component of the ensemble containing the current leader. For instance, if the ensemble has three servers, a component that contains the leader and one other server constitutes a quorum. If the ensemble can not achieve a quorum, the ensemble cannot write data.
,
a
Service
,
a
,
and a
.
:
v1
kind
:
Service
metadata
:
name
:
zk-hs
labels
:
app
:
zk
spec
:
ports
:
-
port
:
2888
name
:
server
-
port
:
3888
name
:
clusterIP
:
None
selector
:
app
:
zk
---
:
v1
kind
:
Service
metadata
:
name
:
zk-cs
labels
:
app
:
zk
spec
:
ports
:
-
port
:
2181
name
:
client
selector
:
app
:
zk
---
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
selector
:
:
app
:
zk
:
1
---
:
apps/v1
kind
:
metadata
:
name
:
zk
spec
:
selector
:
:
app
:
zk
:
zk-hs
replicas
:
3
:
type
:
:
template
:
metadata
:
labels
:
app
:
zk
spec
:
affinity
:
:
:
-
:
:
-
key
:
"app"
operator
:
In
values
:
- zk
:
:
-
name
:
:
Always
image
:
resources
:
requests
:
memory
:
"1Gi"
cpu
:
"0.5"
ports
:
-
:
2181
name
:
client
-
:
2888
name
:
server
-
:
3888
name
:
command
:
- sh
- -c
-
--conf_dir=/opt

/zookeeper/conf \
:
exec
:
command
:
- sh
- -c
-
:
10
:
5
:
exec
:
command
:
- sh
- -c
-
:
10
:
5
:
-
name
:
datadir
mountPath
:
:
runAsUser
:
1000
fsGroup
:
1000
:
-
metadata
:
name
:
datadir
spec
:
:
[
]
resources
:
requests
:
storage
:
10Gi
manifest.
zk-hs
zk-cs
Service,
the
zk-pdb
zk
Use
app
=
zk
Once the
zk-2
CTRL-C
a
ZooKeeper
server.
Use
zk
for
i in
0
1
2;
do
kubectl
exec
zk-
$i
done
replicas
zk
3
zk-0
,
zk-1
, and
zk-2
.
zk-0
zk-1
zk-2
myid
myid
for
i in
0
1
2;
do
echo
"myid zk-
$i
"
;kubectl
exec
zk-
$i
done
myid zk-0
1
myid zk-1
2
myid zk-2
3
zk
for
i in
0
1
2;
do
kubectl
exec
zk-
$i
done
The
zk-hs
.
resolve the FQDNs to the Pods' IP a

ddresses. If Kubernetes reschedules the Pods, it will update the A records with the Pods' new IP addresses, but the A records names will not change.
zoo.cfg
. Use
zoo.cfg
file in the
zk-0
Pod.
kubectl
exec
In the
server.1
,
server.2
, and
server.3
1
,
2
, and
3
myid
the
zk
app
=
zk
in its
myid
file.
servers
zoo.cfg
files
zkCli.sh
world
to the path
/hello
on the
zk-0
kubectl
exec
WATCHER::

zk-1
kubectl
exec
zk-0
ensemble.
WATCHER::

world

section,
Use the
zk
app
=
zk
When
zk-0
CTRL-C
.
zk
app
=
zk
Once the
zk-2
CTRL-C
,
from the
zk-2
Pod.
kubectl
exec
zk
WATCHER::

world
The
zk
spec
:
-
metadata
:
name
:
datadir
:
:
anything
spec
:
:
[
]
resources
:
requests
:
storage
:
20Gi
The
the
.
's
.
app
=
zk
When the
The
template
mounts the PersistentVolumes in the ZooKeeper ser

vers' data directories.
:
-
name
:
datadir
mountPath
:
zk
same
and
Get the
zk
…
command:
      - sh
      - -c
…
Log4j
, and, by default,
zk
.
kubectl
exec
Use
20
2016-12-06 19:34:26,222 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFacto

ry@192] - Accepted socket connection from /127.0.0.1:52750
2016-12-06 19:34:46,149 [myid:1] - INFO  [Thread-1141:NIOServerCnxn@1008] - Closed socket connection for client /127.0.0.1:52767 (no session established for client)
The
zk
's Pod
template
contains a
.
:
runAsUser
:
1000
fsGroup
:
1000
zk-0
Pod.
kubectl
exec
As the
runAsUser
0 S zookeep+    27     1  0  80   0 - 1155556 -    20:46 ?        00:00:19 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.

2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg
zk-0
Pod.
kubectl
exec
fsGroup
The
The
zk
You can use
cpus
=
'json'
-p
=
Use
Use the
history
sts/zk
REVISION
1
2
Use the
never
override
zk-0
Pod.
kubectl
exec
zookeep+    27     1  0 15:03 ?        00:00:03 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Dzookeeper.log.dir=/var/log/zookeeper -Dzookeeper.root.logger=INFO,CONSOLE -cp /usr/bin/../build/classes:/usr/bin/../build/lib/*.jar:/usr/bin/../share/zookeeper/zookeeper-3.4.9.jar:/usr/bin/../share/zookeeper/slf4j-log4j12-1.6.1.jar:/usr/bin/

../share/zookeeper/slf4j-api-1.6.1.jar:/usr/bin/../share/zookeeper/netty-3.10.5.Final.jar:/usr/bin/../share/zookeeper/log4j-1.2.16.jar:/usr/bin/../share/zookeeper/jline-0.9.94.jar:/usr/bin/../src/java/lib/*.jar:/usr/bin/../etc/zookeeper: -Xmx2G -Xms2G -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.local.only=false org.apache.zookeeper.server.quorum.QuorumPeerMain /usr/bin/../etc/zookeeper/zoo.cfg
zk
app
=
zk
zk-0
kubectl
exec
The Pod
template
for the
zk
:
exec
:
command
:
- sh
- -c
-
:
15
:
5
ruok
    exit 0
else
    exit 1
fi
zk
app
=
zk
zk-0
.
kubectl
exec
app
=
zk
:
exec
:
command
:
- sh
- -c
-
:
15
:
5
to specify both. This ensures that only heal

thy servers in the ZooKeeper
.
zk
.
for
i in
0
1
2;
do
$i
{{
}}
;
echo
""
;
done
zk
zk
have a
affinity
:
:
:
-
:
:
-
key
:
"app"
operator
:
In
values
:
- zk
:
The
field tells the
app
label
as
zk
. The
zk-pdb
.
The
zk
zk
.
app
=
zk
for
i in
0
1
2;
do
$i
{{
}}
;
echo
""
;
done
Use
zk-0
$(
{{
}}
)

WARNING: Deleting pods not managed by Replic

ationController, ReplicaSet, Job, or DaemonSet: fluentd-cloud-logging-kubernetes-node-pb41, kube-proxy-kubernetes-node-pb41; Ignoring DaemonSet-managed pods: node-problem-detector-v0.1-o5elz
zk-0
zk-1
$(
{{
}}
)
The
zk-1
zk
contains a
app
=
zk
zk-2
$(
{{
}}
)

pod/zk-2
Use
CTRL-C
zk-2
zk-budget
Use
zkCli.sh
zk-0
.
kubectl
exec
Watche

dEvent state:SyncConnected type:None path:null
world
Use
zk-1
zk-1
app
=
zk
zk-2
$(
{{
}}
)
This time
succeeds.
zk-2
You can use
Use
7 - Services
7.1 - Connecting Applications with Services
Exposing pod

s to the cluster
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
selector
:
:
run
:
my-nginx
replicas
:
2
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
name
:
my-nginx
image
:
nginx
ports
:
-
:
80
run
=
run
=
=
POD_IP
[
map
[
]]
[
map
[
]]
curl
not
You can read more about the
:
:
v1
kind
:
Service
metadata
:
name
:
my-nginx
labels
:
run
:
my-nginx
spec
:
ports
:
-
port
:
80
protocol
:
TCP
selector
:
run
:
my-nginx
with the
(
port
: is the
Service).
View
Service
.
labels
.
as an endpoint. New Pods that match the Service's selector will automatically get

 added
=
my-nginx
from
about the
.
.
Note:
flag to
false
on the
pod spec
.
kubectl
exec
my-nginx-3800858182-jr4a2 -- printenv | grep SERVICE
before
the
=
=
2;
run
=
kubectl
exec
DNS
=
kube-dns
or
. Let's run another cur

l application to test this:
=
:
[
]
A
secret
make keys
KEY
=
CERT
=
=
365
-newkey rsa:2048 -keyout /d/tmp/nginx.key -out /d/tmp/nginx.crt -subj
"/CN=my-nginx/O=my-nginx"
# Convert the keys to base64 encoding
cat /d/tmp/nginx.crt | base64
cat /d/tmp/nginx.key | base64
Use the output from the previous commands to create a yaml file as follows. The base64 encoded value should all be on a single line.
:
"v1"
kind
:
"Secret"
metadata
:
name
:
"nginxsecret"
namespace
:
"default"
type
:
data
:
tls.crt
:
"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIekNDQWdlZ0F3SUJBZ0lKQUp5M3lQK0pzMlpJTUEwR0NTcUdTSWIzRFFFQkJRVUFNQ1l4RVRBUEJnTlYKQkFNVENHNW5hVzU0YzNaak1SRXdEd1lEVlFRS0V3aHVaMmx1ZUhOMll6QWVGdzB4TnpFd01qWXdOekEzTVRKYQpGdzB4T0RFd01qWXdOekEzTVRKYU1DWXhFVEFQQmdOVkJBTVRDRzVuYVc1NGMzWmpNUkV3RHdZRFZRUUtFd2h1CloybHVlSE4yWXpDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSjFxSU1SOVdWM0IKMlZIQlRMRmtobDRONXljMEJxYUhIQktMSnJMcy8vdzZhU3hRS29GbHlJSU94NGUrMlN5ajBFcndCLzlYTnBwbQppeW1CL3JkRldkOXg5UWhBQUxCZkVaTmNiV3NsTVFVcnhBZW50VWt1dk1vLzgvMHRpbGhjc3paenJEYVJ4NEo5Ci82UVRtVVI3a0ZTWUpOWTVQZkR3cGc3dlVvaDZmZ1Voam92VG42eHNVR0M2QURVODBpNXFlZWhNeVI1N2lmU2YKNHZpaXdIY3hnL3lZR1JBRS9mRTRqakxCdmdONjc2SU90S01rZXV3R0ljNDFhd05tNnNTSzRqYUNGeGpYSnZaZQp2by9kTlEybHhHWCtKT2l3SEhXbXNhdGp4WTRaNVk3R1ZoK0QrWnYvcW1mMFgvbVY0Rmo1NzV3ajFMWVBocWtsCmdhSXZYRyt4U1FVQ0F3RUFBYU5RTUU0d0hRWURWUjBPQkJZRUZPNG9OWkI3YXc1OUlsYkROMzhIYkduYnhFVjcKTUI4R0ExVWRJd1FZTUJhQUZPNG9OWkI3YXc1OUlsYkROMzhIYkduYnhFVjdNQXdHQTFVZEV3UUZNQU1CQWY4dwpEUVlKS29aSWh2Y05BUUVGQlFBRGdnRUJBRVhTMW9FU0lFaXdyMDhWcVA0K2NwTHI3TW5FMTducDBvMm14alFvCjRGb0RvRjdRZnZqeE04Tzd2TjB0clcxb2pGSW0vWDE4ZnZaL3k4ZzVaWG40Vm8zc3hKVmRBcStNZC9jTStzUGEKNmJjTkNUekZqeFpUV0UrKzE5NS9zb2dmOUZ3VDVDK3U2Q3B5N0M3MTZvUXRUakViV05VdEt4cXI0Nk1OZWNCMApwRFhWZmdWQTRadkR4NFo3S2RiZDY5eXM3OVFHYmg5ZW1PZ05NZFlsSUswSGt0ejF5WU4vbVpmK3FqTkJqbWZjCkNnMnlwbGQ0Wi8rUUNQZjl3SkoybFIrY2FnT0R4elBWcGxNSEcybzgvTHFDdnh6elZPUDUxeXdLZEtxaUMwSVEKQ0I5T2wwWW5scE9UNEh1b2hSUzBPOStlMm9KdFZsNUIyczRpbDlhZ3RTVXFxUlU9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
tls.key
:
"LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQ2RhaURFZlZsZHdkbFIKd1V5eFpJWmVEZWNuTkFhbWh4d1NpeWF5N1AvOE9ta3NVQ3FCWmNpQ0RzZUh2dGtzbzlCSzhBZi9WemFhWm9zcApnZjYzUlZuZmNmVUlRQUN3WHhHVFhHMXJKVEVGSzhRSHA3VkpMcnpLUC9QOUxZcFlYTE0yYzZ3MmtjZUNmZi

trCkU1bEVlNUJVbUNUV09UM3c4S1lPNzFLSWVuNEZJWTZMMDUrc2JGQmd1Z0ExUE5JdWFubm9UTWtlZTRuMG4rTDQKb3NCM01ZUDhtQmtRQlAzeE9JNHl3YjREZXUraURyU2pKSHJzQmlIT05Xc0RadXJFaXVJMmdoY1kxeWIyWHI2UAozVFVOcGNSbC9pVG9zQngxcHJHclk4V09HZVdPeGxZZmcvbWIvNnBuOUYvNWxlQlkrZStjSTlTMkQ0YXBKWUdpCkwxeHZzVWtGQWdNQkFBRUNnZ0VBZFhCK0xkbk8ySElOTGo5bWRsb25IUGlHWWVzZ294RGQwci9hQ1Zkank4dlEKTjIwL3FQWkUxek1yall6Ry9kVGhTMmMwc0QxaTBXSjdwR1lGb0xtdXlWTjltY0FXUTM5SjM0VHZaU2FFSWZWNgo5TE1jUHhNTmFsNjRLMFRVbUFQZytGam9QSFlhUUxLOERLOUtnNXNrSE5pOWNzMlY5ckd6VWlVZWtBL0RBUlBTClI3L2ZjUFBacDRuRWVBZmI3WTk1R1llb1p5V21SU3VKdlNyblBESGtUdW1vVlVWdkxMRHRzaG9reUxiTWVtN3oKMmJzVmpwSW1GTHJqbGtmQXlpNHg0WjJrV3YyMFRrdWtsZU1jaVlMbjk4QWxiRi9DSmRLM3QraTRoMTVlR2ZQegpoTnh3bk9QdlVTaDR2Q0o3c2Q5TmtEUGJvS2JneVVHOXBYamZhRGR2UVFLQmdRRFFLM01nUkhkQ1pKNVFqZWFKClFGdXF4cHdnNzhZTjQyL1NwenlUYmtGcVFoQWtyczJxWGx1MDZBRzhrZzIzQkswaHkzaE9zSGgxcXRVK3NHZVAKOWRERHBsUWV0ODZsY2FlR3hoc0V0L1R6cEdtNGFKSm5oNzVVaTVGZk9QTDhPTm1FZ3MxMVRhUldhNzZxelRyMgphRlpjQ2pWV1g0YnRSTHVwSkgrMjZnY0FhUUtCZ1FEQmxVSUUzTnNVOFBBZEYvL25sQVB5VWs1T3lDdWc3dmVyClUycXlrdXFzYnBkSi9hODViT1JhM05IVmpVM25uRGpHVHBWaE9JeXg5TEFrc2RwZEFjVmxvcG9HODhXYk9lMTAKMUdqbnkySmdDK3JVWUZiRGtpUGx1K09IYnRnOXFYcGJMSHBzUVpsMGhucDBYSFNYVm9CMUliQndnMGEyOFVadApCbFBtWmc2d1BRS0JnRHVIUVV2SDZHYTNDVUsxNFdmOFhIcFFnMU16M2VvWTBPQm5iSDRvZUZKZmcraEppSXlnCm9RN3hqWldVR3BIc3AyblRtcHErQWlSNzdyRVhsdlhtOElVU2FsbkNiRGlKY01Pc29RdFBZNS9NczJMRm5LQTQKaENmL0pWb2FtZm1nZEN0ZGtFMXNINE9MR2lJVHdEbTRpb0dWZGIwMllnbzFyb2htNUpLMUI3MkpBb0dBUW01UQpHNDhXOTVhL0w1eSt5dCsyZ3YvUHM2VnBvMjZlTzRNQ3lJazJVem9ZWE9IYnNkODJkaC8xT2sybGdHZlI2K3VuCnc1YytZUXRSTHlhQmd3MUtpbGhFZDBKTWU3cGpUSVpnQWJ0LzVPbnlDak9OVXN2aDJjS2lrQ1Z2dTZsZlBjNkQKckliT2ZIaHhxV0RZK2Q1TGN1YSt2NzJ0RkxhenJsSlBsRzlOZHhrQ2dZRUF5elIzT3UyMDNRVVV6bUlCRkwzZAp4Wm5XZ0JLSEo3TnNxcGFWb2RjL0d5aGVycjFDZzE2MmJaSjJDV2RsZkI0VEdtUjZZdmxTZEFOOFRwUWhFbUtKCnFBLzVzdHdxNWd0WGVLOVJmMWxXK29xNThRNTBxMmk1NVdUTThoSDZhTjlaMTltZ0FGdE5VdGNqQUx2dFYxdEYKWSs4WFJkSHJaRnBIWll2NWkwVW1VbGc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K"
:
v1
kind
:
Service
metadata
:
name
:
my-nginx
labels
:
run
:
my-nginx
spec
:
type
:
NodePort
ports
:
-
port
:
8080
:
80
protocol
:
TCP
name
:
http
-
port
:
443
protocol
:
TCP
name
:
https
selector
:
run
:
my-nginx
---
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
selector
:
:
run
:
my-nginx
replicas
:
1
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
volumes
:
-
name
:
secret
:
:
-
name
:
configMap
:
name
:
:
-
name
:
image
:
ports
:
-
:
443
-
:
80
:
-
mountPath
:
name
:
-
mountPath
:
name
:
The
.
before
run
=
=
POD_IP
[
map
[
]]
...
-k
parameter to curl in the last step, this is because we don't know anything ab

out the pods running nginx at certificate generation time,
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
curlpod
replicas
:
1
template
:
metadata
:
labels
:
app
:
curlpod
spec
:
volumes
:
-
name
:
secret
:
:
:
-
name
:
curlpod
command
:
- sh
- -c
image
:
:
-
mountPath
:
name
:
app
=
curlpod
kubectl
exec
...
...
NodePort
,
5
spec:
ports:
nodePort:
31704
port:
8080
80
- name: https
nodePort:
32453
port:
443
443
selector:
1
--
...
...
Type
of
my-nginx
NodePort
to
:
...
to
...
...
7.2 - Using Source IP
NAT
Source NAT
replacing the destination IP on a packet; in this page, that usually means replacing with the IP address of 

a
Pod
VIP
Service
minikube
=
,
iptables
You can test source IP preservation by creating a Service over the source IP app:
=
=
80
=
8080
ClusterIP
=
=
Never --rm
ip addr
…then use
wget
...
The
NodePort
Service:
=
=
80
=
8080
--type
=
NodePort
NODEPORT
=
$(
jsonpath
=
)
N

ODES
=
$(
jsonpath
=
)
for the
for
node in
$NODES
;
do
curl -s
$node
:
$NODEPORT
done
node2
node2
Visually:
.
If you set
Local
,
endpoint.
Set the
for
node in
$NODES
;
do
1
-s
$node
:
$NODEPORT
done
right
, which
does
Visually:
Ready
with
an
=
=
80
=
8080
--type
=
...
field to
Local
without
Visually:
:
32122
The
service.spec.he

althCheckNodePort
/healthz
app
=
Use
curl
/healthz
is
then use
curl
...
.
Forwarded
or
.
the
app
=
Read how to
Home
Blog
Training
Partners
Community
CC BY 4.0

Debug Init Containers | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
and
.
minikube
.
.
.
Init:1/2
See
    ...
    ...
    ...
    ...
This command will return the same information as above in raw JSON.
Init Containers that run a she

ll script print
set -x
Init:
Status
Meaning
Init:N/M
The Pod has
M
N
Pending
or
Running
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm join phase | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Kubeadm
,
phase
Synopsis
Options
-h, --help
preflight
Synopsis
Examples
Options
--dry-run
-h, --help

all
certs
Synopsis
Examples
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--discovery-to

ken string
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Generate the manifests for the new control pla

ne components
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
all
etcd
Synopsis
Examples
# Joins a machin

e as a control plane instance
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
Don't apply any changes; just output what woul

d be done.
-h, --help
or
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-apiserver Configuration (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Field
string
kind
string
plugins
Field
name
string
path
string
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Deprecated API Migration Guide | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Reference
v1.29
The
v1.29
The
:
v1.27
The
v1.27
The
v1.26
The
v1.26
The
The
v1.25
The
v1.25
CronJob
The
batch/v1
The
API version, available 

since v1.21.
:
nodeName
field
zone
field
topology
Event
The
:
type
Normal
and
Warning
regarding
action
,
reason
,
, and
Events
use
eventTime
Events)
use
Events)
use
count
Events)
use
Events)
use
Events)
The
The
policy/v1
policy/v1
:
an empty
(
{}
policy/v1
an empty
Migrate to
or a
.
.
.
v1.22
The
v1.22
The
Ignore
to
Fail
for v1
Exact
to
for v1
30s
to
10s
for v1
None
and
webhooks[*].admissionReviewVer

sions
v1
and
v1beta1
)
The
instead
instead
instead
instead
is moved to
in v1
is moved to
in v1
In
items, the
JSONPath
jsonPath
#66531
)
The
The
The
#32709
)
The
:
API
blocks
Lease
The
Ingress
The
and
fi

eld is renamed to
pathType
Prefix
,
Exact
, and
v1beta1
.
The
The
The
The
since v1.6
v1.13
v1.16
The
v1.16
The
DaemonSet
The
and
apps/v1
is removed
was
OnDelete
)
The
,
, and
apps/v1
is removed
600
10
was
2
maxSurge
and
25%
was
1
)
The
and
apps/v1
was
OnDelete
)
The
ext

ensions/v1beta1
,
, and
apps/v1
The
What to do
Use
command (
.
apps/v1
.
Feedback
Yes
No
.
or
.
Last modified December 08, 2022 at 9:38 AM PST:
Adjust deprecation language for releases that have already happened (8a2bcf65c6)
v1.29
v1.27
v1.26
v1.25
v1.22
v1.16
What to do
Home
Blog
Training
Partners
Community
CC BY 4.0

Scheduling | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Weave Net for NetworkPolicy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Follow the
guide.
iptables
Running
and
2/2 READY
. (
2/2
weave
and
weave-npc
.)
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

Reference
kubelet
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Use a Service to Access an Application in a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operato

)
Tasks
minikube
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
run
:
replicas
:
2
template
:
metadata
:
labels
:
run
:
spec
:
:
-
name
:
image
:
ports
:
-
:
8080
protocol
:
TCP
.
Pods
=
=
Labels:
run
=
Selector:
run
=
Port:                   &lt;uns

et&gt; 8080/TCP
=
--output
=
wide
. If you are
nodes.
where
and
Follow the
tutorial.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Use Kube-router for NetworkPolicy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

bectl on Linux
Debug Pods
Debug Running

 Pods
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namesp

ace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apis

erver Encryption Configuration (v1)
Tasks
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Networking | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Learn how to configure networking for your cluster.
1:
2:
1 - Adding entries to Pod /etc/hosts with HostAliases
Adding entries to a Pod's
file provides Pod-level override of hostname resolution when DNS and other options are not applicable. You can add these custom entries with the HostAliases field in PodSpec.
Modification not using HostAliases is not suggested because the file is managed by the kubelet and can be overwritten on during Pod creation/restart.
Default hosts file content
Start an Nginx Pod which is assigned a Pod IP:
kubectl run nginx --image nginx
Examine a Pod IP:
=
wide
The hosts file content would look like this:
kubectl
exec
nginx -- cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1	localhost
::1	localhost ip6-localhost ip6-loopback
fe00::0	ip6-localnet
fe00::0	ip6-mcastprefix
fe00::1	ip6-allnodes
fe00::2	ip6-allrouters
10.200.0.4	nginx
hosts
file only includes IPv4 and IPv6 boilerplates like
localhost
and its own hostname.
Adding additional entries with hostAliases
In addition to the default boilerplate, you can add additional entries to the
hosts
file.
For example: to resolve
foo.local
,
bar.local
to
127.0.0.1
and
foo.remote
,
bar.remote
to
10.1.2.3
, you can configure HostAliases for a Pod under
.spec.hostAliases
:
service/networking/hostaliases-pod.yaml
:
v1
kind
:
Pod
metadata
:
name
:
hostaliases-pod
spec
:
:
Never
:
-
ip
:
hostnames
:
-
"foo.local"
-
"bar.local"
-
ip
:
hostnames
:
-
"foo.remote"
-
"bar.remote"
:
-
name
:
cat-hosts
image
:
command
:
- cat
args
:
-
"/etc/hosts"
You can start a Pod with that configuration by running:
kubectl apply -f https://k8s.io/examples/service/networking/hostaliases-pod.yaml
pod/hostaliases-pod created
Examine a Pod's details to see its IPv4 address and its status:
kubectl get pod --output
=
wide
NAME                           READY     STATUS      RESTARTS   AGE       IP              NODE
hostaliases-pod                0/1       Completed   0          6s        10.200.0.5      worker0
The
hosts
file content looks like this:
kubectl logs hostaliases-pod
10.200.0.5	hostaliases-pod

# Entries added by HostAliases.
127.0.0.1	foo.local	bar.local
10.1.2.3	foo.remote	bar.remote
with the additional entries specified at the bottom.
Why does the kubelet manage the hosts file?
The kubelet manages the
hosts
file for each container of the Pod to prevent the container runtime from
modifying the file after the containers have already been started.
Historically, Kubernetes always used Docker Engine as its container runtime, and Docker Engine would
then modify the
file after each container had started.
Current Kubernetes can use a variety of container runtimes; even so, the kubelet manages the
hosts file within each container so that the outcome is as intended regardless of which
container runtime you use.
Caution:
Avoid making manual changes to the hosts file inside a container.
If you make manual changes to the hosts file,
those changes are lost when the container exits.
2 - Validate IPv4/IPv6 dual-stack
This document shares how to validate IPv4/IPv6 dual-stack enabled Kubernete

s clusters.
Provider support for dual-stack networking (Cloud provider or otherwise must be able to provide Kubernetes nodes with routable IPv4/IPv6 network interfaces)
A
that supports dual-stack networking.
Dual-stack enabled
cluster
Your Kubernetes server must be at or later than version v1.23.
.
Note:
While you can validate with an earlier version, the feature is only GA and officially supported since v1.23.
Validate addressing
Validate node addressing
Each dual-stack Node should have a single IPv4 block and a single IPv6 block allocated. Validate that IPv4/IPv6 Pod address ranges are configured by running the following command. Replace the sample node name with a valid dual-stack Node from your cluster. In this example, the Node's name is
k8s-linuxpool1-34450317-0
:
kubectl get nodes k8s-linuxpool1-34450317-0 -o go-template --template
=
'{{range .spec.podCIDRs}}{{printf "%s\n" .}}{{end}}'
10.244.1.0/24
2001:db8::/64
There should be one IPv4 block and one IPv6 block allocated.
Validate that the node has an IPv4 and IPv6 interface detected. Replace node name with a valid node from the cluster. In this example the node name is
:
=
'{{range .status.addresses}}{{printf "%s: %s\n" .type .address}}{{end}}'
Hostname: k8s-linuxpool1-34450317-0
InternalIP: 10.0.0.5
InternalIP: 2001:db8:10::5
Validate Pod addressing
Validate that a Pod has an IPv4 and IPv6 address assigned. Replace the Pod name with a valid Pod in your cluster. In this example the Pod name is
pod01
:
kubectl get pods pod01 -o go-template --template
=
'{{range .status.podIPs}}{{printf "%s\n" .ip}}{{end}}'
10.244.1.4
2001:db8::4
You can also validate Pod IPs using the Downward API via the
status.podIPs
fieldPath. The following snippet demonstrates how you can expose the Pod IPs via an environment variable called
MY_POD_IPS
within a container.
env:
        - name: MY_POD_IPS
          valueFrom:
            fieldRef:
              fieldPath: status.podIPs
The following command prints the value of the
environment variable from within a container. The value is a comma separated list that corresponds to the Pod's IPv4 and IPv6 addresses.
kubectl
exec
-it pod01 --
set
| grep MY_POD_IPS
MY_POD_IPS=10.244.1.4,2001:db8::4
The Pod's IP addresses will also be written to
within a container. The following command executes a cat on
on a dual stack Pod. From the output you can verify both the IPv4 and IPv6 IP address for the Pod.
kubectl
exec
-it pod01 -- cat /etc/hosts
127.0.0.1    localhost
::1    localhost ip6-localhost ip6-loopback
fe00::0    ip6-localnet
fe00::0    ip6-mcastprefix
fe00::1    ip6-allnodes
fe00::2    ip6-allrouters
10.244.1.4    pod01
2001:db8::4    pod01
Validate Services
Create the following Service that does not explicitly define
. Kubernetes will assign a cluster IP for the Service from the first configured
and set the
to
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Use
kubectl
to view the YAML for the Service.
The Service has
set to
and
set to an IPv4 address from the first configured range set via
flag on kube-controller-manager.
:
v1
kind
:
Service
metadata
:
name
:
namespace
:
default
spec
:
clusterIP
:
10.0.217.164
:
-
:
- IPv4
:
ports
:
-
port
:
80
protocol
:
TCP
:
9376
selector
:
:
MyApp
:
None
type
:
ClusterIP
statu

s
:
:
{}
Create the following Service that explicitly defines
IPv6
as the first array element in
. Kubernetes will assign a cluster IP for the Service from the IPv6 range configured
and set the
to
.
service/networking/dual-stack-ipfamilies-ipv6.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
- IPv6
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Use
kubectl
set to
and
set to an IPv6 address from the IPv6 range set via
:
v1
kind
:
Service
metadata
:
labels
:
:
MyApp
name
:
spec
:
clusterIP
:
2001
:db8:fd00::5118
:
-
2001
:
- IPv6
:
ports
:
-
port
:
80
protocol
:
TCP
:
80
selector
:
:
MyApp
:
None
type
:
ClusterIP
status
:
:
{}
in
. Kubernetes will assign both IPv4 and IPv6 addresses (as this cluster has dual-stack enabled) and select the
based on the address family of the first element in the
array.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Note:
The
command will only show the primary IP in the
field.
kubectl get svc -l app.kubernetes.io/name
=
MyApp
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT
(
S
)
AGE
my-service   ClusterIP   10.0.216.242   &lt;none&gt;        80/TCP    5s
Validate that the Service gets cluster IPs from the IPv4 and IPv6 address blocks using
. You may then validate access to the service via the IPs and ports.
kubectl describe svc -l app.kubernetes.io/name
=
MyApp
Name:              my-service
Labels:            app.kubernetes.io/name=MyApp
Selector:          app.kubernetes.io/name=MyApp
Type:              ClusterIP
IP Family Policy:  PreferDualStack
IP Families:       IPv4,IPv6
IP:                10.0.216.242
IPs:               10.0.216.242,2001:db8:fd00::af55
Port:              &lt;unset&gt;  80/TCP
TargetPort:        9376/TCP
Endpoints:         &lt;none&gt;
Session Affinity:  None
Create a dual-stack load balanced Service
If the cloud provider supports the provisioning of IPv6 enabled external load balancers, create the following Service with
in
,
IPv6
as the first element of the
array and the
type
field set to
.
service/networking/dual-stack-prefer-ipv6-lb-svc.yaml
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
:
- IPv6
type
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Check the Service:
=
MyApp
Validate that the Service receives a
address from the IPv6 address block along with an
. You may then validate access to the service via the IP and port.
NAME         TYPE           CLUSTER-IP            EXTERNAL-IP        PORT
(
S
)
AGE
my-service   LoadBalancer   2001:db8:fd00::7ebc   2603:1030:805::5   80:30790/TCP   35s
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The

 Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
1 - ServiceAccount
: v1
kind
metadata
(
)
(boolean)
([]
)
secrets
([]
)
name
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue

(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - TokenRequest
kind
metadata
(
)
spec
(
status
(
)
TokenRequestSpec contains client provided p

arameters of a token request.
audiences
(string)
(string)
(string)
(string)
(int64)
token
create
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
3 - TokenReview
kind
metadata
(
)
spec
(
status
(
)
audiences
token
(string)
audiences
Audiences are audience identifiers chosen by the authenticator that are compatible with both the TokenReview and token. An identifier is any identifier in the intersection of the TokenReviewSpec audiences and the token's audiences. A client of the TokenReview API that sets the spec.audiences field should validate that a compatible audience ide

ntifier is returned in the status.audiences field to ensure that the TokenReview server is audience aware. If a TokenReview returns an empty status.audience field where status.authenticated is "true", the token is valid against the audience of the Kubernetes API server.
(boolean)
error
(string)
user
user.uid
(string)
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
4 - CertificateSigningRequest
kind
metadata
(
)
spec
(
status
(
)
request
Reques

ts for this signer can be auto-approved by the "csrapproving" controller in kube-controller-manager, and can be issued by the "csrsigning" controller in kube-controller-manager.
(int32)
extra
groups
uid
(string)
usages
username
(string)
([]byte)
certificate is populated with an issued certificate by the signer after an Approved condition is present. This field is set via the /status sub

resource. Once populated, this field is immutable.
base64(
...
)
(Time)
(Time)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
)

pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/certificat

es.k8s.io/v1/certificatesigningrequests/{name}/approval
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Home
Blog
Training
Partners
Community
CC BY 4.0

PR wranglers | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

ation (v1)
SIG Docs
approvers
.
Duties
Review
Style
and
Content
guides.
size/XS
size/XXL
CLA
.
Use
this
sig/
label(s).
SIG
on the PR.
Use the
/approve
/lgtm
issues as
.
:
Needs LGTM
:
:
/approve
Quick Wins
: Lists PRs against the main branch with no clear blockers. (change "XS" in the

 size label as you work through the PRs [XS, S, M, L, XL, XXL]).
dev-
using:

/close
Note:
The
.
.
Feedback
Yes
No
.
or
.
Last modified May 31, 2022 at 10:22 AM PST:
rename fejta bot to k8s-triage-robot (ab3910b337)
Duties
Home
Blog
Training
Partners
Community
CC BY 4.0

Pull an Image from a Private Registry | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Secret
Docker Hub
minikube
docker
Docker ID
Use the
docker
log in
section of
.
View the
file:
{
"auths"
: {
: {
"auth"
:
}
}
}
Note:
auth
entry but a
.
\
=
=
\
--type
=
If you need more control (for example, to set a namespace or a label o

n the new
Be sure to:
set
type
to
Example:
:
v1
kind
:
Secret
metadata
:
name
:
namespace
:
data
:
:
UmVhbGx5IHJlYWxseSByZWVlZWVlZWVlZWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGxsbGx5eXl5eXl5eXl5eXl5eXl5eXl5eSBsbGxsbGxsbGxsbGxsbG9vb29vb29vb29vb29vb29vb29vb29vb29vb25ubm5ubm5ubm5ubm5ubm5ubm5ubm5ubmdnZ2dnZ2dnZ2dnZ2dnZ2dnZ2cgYXV0aCBrZXlzCg==
type
:
, it means
file.
regcred
:
=
=
=
=
where:
Use
regcred
.
Note:
kubectl
regcred
regcred
=
yaml
:
v1
kind
:
Secret
metadata
:
...
name
:
regcred
...
data
:
:
type
:
=
{
"auths"
:{
:{
:
"janedoe"
,
:
,
"email"
:
,
"auth"
:
}}}
auth
echo
:
file.
regcred
regcred
:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
:
-
name
:
regcred
In file
, replace
The
field in the config

uration file specifies that
regcred
.
Secrets
Secret
.
.
See
.
See the
container definitions
of a Pod
Feedback
Yes
No
.
or
.
Last modified October 09, 2022 at 5:11 PM PST:
Fix curl command in pull-image-private-registry.md (bb4dfd6fcd)
regcred
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Kubernetes is an open source container orchestration engine for automating deployment, scaling, and management of containerized applications. The open source project is hosted by the Cloud Native Computing Foundation.
1:
1 - Available Documentation Versions
This website contains documentation for the current version of Kubernetes
and the four previous versions of Kubernetes.
The availability of documentation for a Kubernetes version is separate from whether
that release is currently supported.
Read
Support period
to learn about
which versions of Kubernetes are officially supported, and for how long.
Home
Blog
Training
Partners
Community
CC BY 4.0

Validate IPv4/IPv6 dual-stack | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

t Up kubectl on Linux
Debug Pods
Debug 

Running Pods
Auditing
Run Jobs
TLS
Tutorials
Security

 Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
ku

be-apiserver Encryption Configuration (v1)
Tasks
This document shares how to validate IPv4/IPv6 dual-stack enabled Kubernetes clusters.
A
cluster
.
Note:
:
=
:
=
pod01
:
=
env:
            fieldRef:
kubectl
exec
set
The Pod's IP addresses will

 also be written to
kubectl
exec
and set the
to
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Use
kubectl
set to
and
:
v1
kind
:
Service
metadata
:
name
:
namespace
:
default
spec
:
clusterIP
:
:
-
:
- IPv4
:
ports
:
-
port
:
80
protocol
:
TCP
:
9376
selector
:
:
MyApp
:
None
type
:
ClusterIP
status
:
:
{}
IPv6
and set the
to
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
- IPv6
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Use
kubectl
set to
and
:
v1
kind
:
Service
metadata
:
labels
:
:
MyApp
name
:
spec
:
clusterIP
:
2001
:
-
2001
:
- IPv6
:
ports
:
-
port
:
80
protocol
:
TCP
:
80
selector
:
:
MyApp
:
None
type
:
ClusterIP
status
:
:
{}
in
array.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Note:
The
field.
=
MyApp
(
S
)
AGE
=
MyApp
Annotations: 

      &lt;none&gt;
in
,
IPv6
type
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
:
- IPv6
type
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
=
MyApp
(
S
)
AGE
Feedback
Yes
No
.
or
.
Last modified September 08, 2022 at 9:37 AM PST:
use use IPv6 Address Prefix Reserved for Documentation (86264eae3b)
Home
Blog
Training
Partners
Community
CC BY 4.0

Ingress | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

lpha1)
Reference
Ingress
Ingress
Ingress
kind
: Ingress
metadata
(
)
spec
(
)
status
(
)
(
)
(string)
rules
(string)
Host is the

 fully qualified domain name of a network host, as defined by RFC 3986. Note the following deviations from the "host" part of the URI as defined in RFC 3986: 1. IPs are not allowed. Currently an IngressRuleValue can only apply to
2. The
:
/
?
(
(string)
tls
tls.hosts
Hosts are a list of hosts included in the TLS certificate. The values in this list must match the name/s used in the tlsSecret. Defaults to the wildcard host setting for the loadbalancer controller fulfilling this Ingress, if left unspecif

ied.
(string)
resource
(
)
service
(string)
(int32)
(string)
(string)
(string)
items
([]
Ingress
(string)
kind
(string)
metadata
(
ListMeta
)
get
name

(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
202 (
Ingress
update
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
PATCH /apis/networking.k8s.io/v1/namespaces/{namespace}/i

ngresses/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Ingress
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Reviewing pull requests
Read the
and
Read the
S[ ] -.-
end
T[ ] -.-
end
by label]
enabled
Go to
.
tab.
Details
link of the
line in the
Go to the
+
Review 

changes
move
?
Content
Website
Docs Home
listing?
Other
;
nit:
.
;
2 - Reviewing for approvers and reviewers
SIG Docs
Revi

ewers
and
Approvers
Reviewing a PR
.
Using the
/assign
Note:
Look at the
reviewers
Content
and
Style
/approve
or
/lgtm
true:
Note:
Prow
is
format.
/lgtm
/approve
Approvers
/assign
Anyone
/close
/hold
Anyone
Adds the
Anyone
label.
.
filter
Add the
and
tr

iage/needs-information
labels.
Label
help
or
Help Wanted and Good First Issue Labels
#sig-docs
on Slack or
the
.
)
or
)
)
)
Label
command.
label.
kubectl
We expect
your qu

estion to the `#kubernetes-users` channel in

Home
Blog
Training
Partners
Community
CC BY 4.0

Use Cilium for NetworkPolicy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on L

inux
Debug Pods
Get 

a Shell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security

Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encr

yption Configuration (v1)
Tasks
.
minikube
.
=
cni
cilium-ca
command.
status
command
here
.
The remainder of the Getting Started Guide explains how to enforce both L3/L4
Understanding 

Cilium components
=
=
cilium
...
A
cilium
.
Feedback
Yes
No
.
or
.
Last modified November 17, 2022 at 1:06 PM PST:
Update Cilium installation steps (a794cc6469)
Home
Blog
Training
Partners
Community
CC BY 4.0

Event | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
Event
Event
Event
kind
: Event
metadata
(
)
eventTime
action
(string)
(int32)
(Time)
(Time)
(string)
(string)
note
(string)
reason
(string)
regarding
(
)
regarding contains the object

 this Event is about. In most cases it's an Object reporting controller implements, e.g. ReplicaSetController implements ReplicaSets and this event is emitted because it acts on some changes in a ReplicaSet object.
related
(
)
(string)
(string)
series
type
(string)
EventList
kind
metadata
(
ListMeta
)
items
([]
Event
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
create
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 

(
Event
): OK
201 (
Event
): Created
202 (
Event
update
name
(
in path
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Event
EventList
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Authenticating | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
accounts.
stored as
Secrets
as
kubectl
kubelets
or
.
or
.
Extra fields: a map of strings to l

ist of strings which holds additional information authorizers may find useful.
.
The
or the
.
openssl
See
Note:
.
kubeadm
. It is
intentionally limited to discourage

 users from using these tokens past
kubeadm
)
cluster.
Please see
kubeadm
.
field of a
PodSpec
.
Note:
:
apps/v1
kind
:
metadata
:
name
:
namespace
:
default
spec
:
replicas
:
3
template
:
metadata
:
# ...
spec
:
:
:
-
name
:
nginx
image
:
above
,
and
.
Warning:
ID Token
.
id_token
(not the
)
above
kube -&gt;&gt; api:

 4. Authorization: Bearer...
enabled
,
id_token
and a
When using
kubectl
, use your
id_token
with the
--token
kubectl
sends your
id_token
id_token
kubectl
kubectl
id_token
The
id_token
id_token
.
Parameter
Example
Required
https://
Yes
Yes
sub
email
or
name
email
sub
No
system:
oidc:
email
where
. The value
-
oidc:
No
groups
No
system:
oidc:
and
.
oidc:
No
No
/etc/kuberne

tes/ssl/kc-ca.pem
No
RS512
No
azp
others
).
dex
,
Keycloak
,
UAA
, or
.
Support
CA
flag set to
TRUE
UAA
Dex
oidc
id_token
id_token
,
,
client_id
, and
id_token
\
=
oidc
\
=
=(
issuer url
)
\
=
client-id
=(
)
\
=
=(
)
\
=
=(
)
\
=
=(
)
\
=
id-token
=(
)
\
=
oidc
\
=
=
\
=
client-id
=
\
=
=
\
=
=
=
\
=
=
\
=
id-token
=
eyJraWQiOiJDTj1vaWRjaWRwLnRyZW1vbG8ubGFuLCBPVT1EZW1vLCBPPVRybWVvbG8gU2VjdXJpdHksIEw9QXJsaW5ndG9uLCBTVD1WaXJnaW5pYSwgQz1VUy1DTj1rdWJlLWNhLTEyMDIxNDc5MjEwMzYwNzMyMTUyIiwiYWxnIjoiUlMyNTYifQ.eyJpc3MiOiJodHRwczovL29pZGNpZHAudHJlbW9sby5sYW46ODQ0My9hdXRoL2lkcC9PaWRjSWRQIiwiYXVkIjoia3ViZXJuZXRlcyIsImV4cCI6MTQ4MzU0OTUxMSwianRpIjoiMm96US15TXdFcHV4WDlHZUhQdy1hZyIsImlhdCI6MTQ4MzU0OTQ1MSwibmJmIjoxNDgzNTQ5MzMxLCJzdWIiOiI0YWViMzdiYS1iNjQ1LTQ4ZmQtYWIzMC0xYTAxZWU0MWUyMTgifQ.w6p4J_6qQ1HzTG9nrEOrubxIMb9K5hzcMPxc9IxPx2K4xO9l-oFiUw93daH3m5pluP6K7eOE6txBuRVfEcpJSwlelsOsW8gb8VJcnzMS9EnZpeA0tW_p-mnkFc3VcfyXuhe5R3G7aa5d8uHv70yJ9Y3-UhjiN9EhpMd

fPAoEB9fYKKkJRzF7utTTIPGrSaSU6d2pcpfYKaxIwePzEkT4DfcQthoZdy9ucNvvLoi1DIC-UocFD8HLs8LYKEqSxQvOcvnThbObJ9af71EwmuE21fO5KzMW20KtAeget1gnldOosPtz1G5EwvaQ401-RPQzPGMVBld0_zMCAwZttJ4knw
users
:
-
name
:
mmosley
user
:
:
config
:
client-id
:
:
id-token
:
:
:
:
name
:
oidc
Once your
id_token
expires,
kubectl
id_token
using your
and
and
id_token
in your
.
--token
Option
The
kubectl
--token
id_token
=
or
v1beta1
.
clusters
users
:
v1
kind
:
Config
clusters
:
-
name
:
cluster
:
:
server
:
users
:
-
name
:
user
:
:
:
:
webhook
contexts
:
-
context
:
cluster
:
user
:
name-of-api-

server
name
:
webhook
above
,
and
must
Note:
.
{
,
"kind":
,
"spec":
{
"token":
,
[
,
]
}
}
{
,
"kind":
,
"spec":
{
"token":
,
[
,
]
}
}
status
spec
{
,
"kind":
,
"status":
{
true
,
"user":
{
# Required
,
# Optional
"uid":
"42"
,
"groups":
[
,
"qa"
],
"extra":
{
[
,
]
}
},
[
]
}
}
{
,
"kind":
,
"status":
{
true
,
"user":
{
# Required
,
# Optional
"uid":
"42"
,
"groups":
[
,
"qa"
],
# Thi

s should not contain confidential data, as it can be recorded in logs
"extra":
{
[
,
]
}
},
[
]
}
}
{
,
"kind":
,
"status":
{
false
,
"error":
}
}
{
,
"kind":
,
"status":
{
false
,
"error":
}
}
.
Note:
.
GET
/
HTTP
/
1.1
:
fido
:
dogs
:
:
:
openid
:
profile
name
:
fido
groups
:
- dogs
extra
:
:
scopes
:
- openid
- profile
not
treated as anonym

ous requests, and given a username of
.
error.
user or the
*
user or
*
and
.
Note:
.
Note:
When using
kubectl
set the
--as
header.
Set the
--as
and
flag:
=
=
Note:
kubectl
set user and group impersonation he

aders:
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"users"
,
"groups"
,
]
verbs
:
[
]
apiGroup
.
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
,
"uids"
]
verbs
:
[
]
of
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"users"
]
verbs
:
[
]
:
[
]
-
apiGroups
:
[
""
]
resources
:
[
"groups"
]
verbs
:
[
]
:
[
,
"admins"
]
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
]
:
[
"view"
,
]
-
apiGroups
:
[
]
resources
:
[
"uids"
]
verbs
:
[
]
:
[
]
Note:
and a
,
not a
Role
and
.
kubectl
and
kubelet
kubectl
command.
to submit a
:
v1
kind
:
Config
users
:
-
name
:
my-user
user
:
exec
:
command
:
# API version to use when decoding the ExecCredentials resource. Requ

ired.
#
#
:
env
:
-
name
:
"FOO"
value
:
"bar"
args
:
-
"arg1"
-
"arg2"
:
|
...
:
true
:
Never
clusters
:
-
name
:
cluster
:
server
:
:
:
-
name
:
extension
:
arbitrary
:
config
this
:
you
:
[
"can"
,
"put"
,
,
"here"
]
contexts
:
-
name
:
context
:
cluster
:
user
:
my-user
:
:
v1
kind
:
Config
users
:
-
name
:
my-user
user
:
exec
:
command
:
#
#
:
env
:
-
name
:
"FOO"
value
:
"bar"
args
:
-
"arg1"
-
"arg2"
:
|
...
:
true
# or "Always" (this exec plugin requires standard input to 

function). Optional.
:
Never
clusters
:
-
name
:
cluster
:
server
:
:
:
-
name
:
extension
:
arbitrary
:
config
this
:
you
:
[
"can"
,
"put"
,
,
"here"
]
contexts
:
-
name
:
context
:
cluster
:
user
:
my-user
:
,
the binary
-
name
:
my-user
user
:
exec
:
command
:
:
:
Never
object to
stdout
.
status
.
stdin
stdin
stdin
stdin
stdin
(see table
.
Value
Meaning
Never
Always
{
:
,
"kind"
:
,
"status"
: {
"token"
:
}
}
{
:
,
"kind"
:
,
"status"
: {
"token"
:
}
}
and
client.authentication.k8s.i

o/v1beta1
{
:
,
"kind"
:
,
"status"
: {
:
,
:
}
}
{
:
,
"kind"
:
,
"status"
: {
:
,
:
}
}
RFC 3339
{
:
,
"kind"
:
,
"status"
: {
"token"
:
,
:
}
}
{
:
,
"kind"
:
,
"status"
: {
"token"
:
,
:
}
}
on the
user.exec
.
{
:
,
"kind"
:
,
"spec"
: {
"cluster"
: {
"server"
:
,
:
"LS0t..."
,
"config"
: {
:
"config"
,
"this"
:
,
"you"
: [
"can"
,
"put"
,
,
"here"
]
}
},
:
true
}
}
{
:
,
"kind"
:
,
"spec"
: {
"cluster"
: {
"server"
:
,
:
"LS0t..."
,
"config"
: {
:
"config"
,
"this"
:
,
"you"
: [
"can"
,
"put"
,
,
"here"
]
}
},
:
true
}
}
):
{
:
,
"kind"
:
}
{
:
,
"kind"
:
,
"status"
: {
: {
"name"
:
,
"uid"
:
,
"groups"
: [
"viewers"
,
"editors"
,
],
"extra"
: {
: [
]
}
}
}
}
kubectl alpha auth whoa

mi
JSON
YAML
{
:
,
"kind"
:
,
"status"
: {
: {
:
,
"uid"
:
,
"groups"
: [
,
,
],
"extra"
: {
"skills"
: [
"reading"
,
],
: [
"math"
,
"sports"
]
}
}
}
}
:
kind
:
status
:
userInfo
:
username
:
jane.doe
uid
:
groups
:
- students
- teachers
extra
:
skills
:
- reading
- learning
subjects
:
- math
- sports
or
.
Note:
userInfo
including
.
Note:
the
API group
enabled.
Read the
Read the
Feedback
Yes
No
.
or
.
Last modified November 28, 2022 at 1:20 AM PST:
Update content/en/docs/reference/access-authn-authz/authentication.md (cb9dc5a4ac)
Home
Blog
Training
Partners
Community
CC BY 4.0

Interactive Tutorial - Configuring a Java Microservice | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tutorials
Feedback
Yes
No
.
or
.
Last modified May 29, 2022 at 2:17 PM PST:
Extend Katacoda removal warning (a7c5ec8eda)
Home
Blog
Training
Partners
Community
CC BY 4.0

Restrict a Container's Syscalls with seccomp | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tutorials
Security
node
kind
and
kubectl
.
curl
Note:
run as
.
profiles/
{
:
}
{
:
}
{
:
,
: [
,
,
],
: [
{
"names"
: [
"accept4"
,
,
,
"futex"
,
"madvise"
,
,
,
,
"vfork"
,
"mmap"
,

"read"
,
"write"
,
"close"
,
,
,
"munmap"
,
"brk"
,
,
,
,
"gettid"
,
"clone"
,
"bind"
,
"socket"
,
"openat"
,
,
,
,
"listen"
,
,
,
,
"connect"
,
"dup2"
,
,
"execve"
,
"exit"
,
"fcntl"
,
"getpid"
,
"getuid"
,
"ioctl"
,
,
,
"open"
,
"poll"
,
,
"sendto"
,
,
,
"writev"
],
"action"
:
}
]
}
kind
:
kind
:
Cluster
nodes
:
-
role
:
:
-
hostPath
:
:
kind.yaml
:
See
Nodes
within the
.
=
kind.yaml
docker ps
profiles/
docker
exec
enabled
.
(seccom

p disabled) mode.
Note:
.
.
feature
gate
command
file
kind
kind
provides
feature
:
kind
:
Cluster
:
:
:
true
nodes
:
-
role
:
image
:
:
- |
-
role
:
worker
image
:
:
- |
=
=
using
to run
worker:
docker
exec
\
{
:
,
: [
,
,
],
: [
{
"names"
: [
"..."
]
}
]
}
:
v1
kind
:
Pod
metadata
:
name
:
audit-pod
labels
:
app
:
audit-pod
spec
:
:
:
type
:
Localhost
:
:
-
name
:
image
:
args
:
-
:
:
false
Note:
secc

omp.security.alpha.kubernetes.io/pod
Services
5678
curl
to run the
curl
docker
exec
in
tail
calls from
http-echo
:
http-echo
, and if you
curl
Jul  6 15:37:40 my-machine kernel: [369128.669452] audit: type=1326 audit(1594067860.484:14536): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e syscall=51 compat=0 ip=0x46fe1f code=0x7ffc0000
Jul  6 15:38:40 my-machine kernel: [369188.671648] audit: type=1326 audit(1594067920.488:14559): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=29064 comm="http-echo" exe="/http-echo" sig=0 arch=c000003e 

syscall=270 compat=0 ip=0x4559b1 code=0x7ffc0000
http-echo
process by
syscall=
syscalls.
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
app
:
spec
:
:
:
type
:
Localhost
:
:
-
name
:
image
:
args
:
-
:
:
false
http-echo
,
sent to
syslog
.
:
v1
kind
:
Pod
metadata
:
name
:
fine-pod
labels
:
app
:
fine-pod
spec
:
:
:
type
:
Localhost
:
:
-
name
:
image
:
args
:
-
:
:
false
tail
http-echo
:
5678
Use
curl
# Change 6a96207fed4b to the cont

rol plane container ID you saw from "docker ps"
docker
exec
syslog
.
Note:
.
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
app
:
spec
:
:
:
type
:
:
-
name
:
image
:
args
:
-
:
:
false
Feedback
Yes
No
.
or
.
Last modified August 14, 2022 at 3:36 PM PST:
Fix the extra '{' or '}' (8183d5a62d)
Home
Blog
Training
Partners
Community
CC BY 4.0

Tasks | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Tasks
1:
1.1:
1.2:
1.3:
1.4:
Tools Included
1.4.1:
bash auto-completion on Linux
1.4.2:
bash auto-completion on macOS
1.4.3:
fish auto-completion
1.4.4:
kubectl-convert overview
1.4.5:
PowerShell auto-completion
1.4.6:
verify kubectl install
1.4.7:
What's next?
1.4.8:
zsh auto-completion
2:
2.1:
2.1.1:
2.1.2:
2.1.3:
2.1.4:
2.1.5:
2.2:
2.2.1:
2.2.2:
2.2.3:
2.2.4:
2.2.5:
2.2.6:
2.3:
2.4:
2.4.1:
2.4.2:
2.4.3:
2.4.4:
2.4.5:
2.4.6:
2.5:
2.5.1:
2.5.2:
2.5.3:
2.5.4:
2.5.5:
2.5.6:
2.6:
2.7:
2.8:
2.9:
2.10:
2.11:
2.12:
2.13:
2.14:
2.15:
2.16:
2.17:
2.18:
2.19:
2.20:
2.21:
2.22:
2.23:
2.24:
2.25:
2.26:
2.27:
2.28:
2.29:
2.30:
2.31:
2.32:
2.33:
2.34:
2.35:
2.36:
2.37:
2.38:
2.39:
2.40:
2.41:
2.42:
3:
3.1:
3.2:
3.3:
3.4:
Configure 

RunAsUserName for Windows pods and containers
3.5:
3.6:
3.7:
3.8:
3.9:
3.10:
3.11:
3.12:
3.13:
3.14:
3.15:
3.16:
3.17:
3.18:
3.19:
3.20:
3.21:
3.22:
3.23:
3.24:
3.25:
3.26:
4:
4.1:
4.1.1:
Debug Pods
4.1.2:
4.1.3:
4.1.4:
4.1.5:
4.1.6:
4.1.7:
4.2:
4.2.1:
4.2.2:
4.2.3:
4.2.4:
4.2.5:
Auditing
4.2.6:
4.2.7:
4.2.8:
5:
5.1:
5.2:
5.3:
5.4:
5.5:
6:
6.1:
6.2:
6.3:
7:
7.1:
7.2:
7.3:
7.4:
7.5:
7.6:
8:
8.1:
8.2:
8.3:
8.4:
8.5:
8.6:
8.7:
8.8:
8.9:
8.10:
9:
Run Jobs
9.1:
9.2:
9.3:
9.4:
9.5:
9.6:
9.7:
10:
10.1:
10.2:
10.3:
10.4:
10.5:
10.6:
10.7:
10.8:
10.9:
10.10:

10.11:
10.12:
11:
11.1:
11.2:
11.2.1:
11.2.2:
11.3:
11.4:
11.5:
11.6:
11.7:
12:
TLS
12.1:
12.2:
12.3:
13:
13.1:
13.2:
14:
14.1:
14.2:
15:
16:
17:
This section of the Kubernetes documentation contains pages that
show how to do individual tasks. A task page shows how to do a
single thing, typically by giving a short sequence of steps.
If you would like to write a task page, see
Creating a Documentation Pull Request
.
1 - Install Tools
Set up Kubernetes tools on your computer.
kubectl
The Kubernetes command-line tool,
kubectl
, allows
you to run commands against Kubernetes clusters.
You can use kubectl to deploy applications, inspect and manage cluster resources,
and view logs. For more information including a complete list of kubectl operations, see the
kubectl
.
kubectl is installable on a variety of Linux platforms, macOS and Windows.
Find your preferred operating system below.
kind
kind
lets you run Kubernetes on
your local computer. This tool requires that you have
Docker
installed and configured.
The kind
Quick Start
page
shows you what you need to do to get up and running with kind.
View kind Quick Start Guide
minikube
Like
kind
,
minikube
is a tool that lets you run Kubernetes
locally.
minikube
runs an all-in-one or a multi-node local Kubernetes cluster on your personal
computer (including Windows, macOS and Linux PCs) so that you can try out
Kubernetes, or for daily development work.
You can follow the official
Get Started!
guide if your focus is
on getting the tool installed.
View minikube Get Started! Guide
minikube
working, you can use it to
run a sample application
.
kubeadm
kubeadm
tool to create and manage Kubernetes clusters.
It performs the actions necessary to get a minimum viable, secure cluster up and running in a user friendly way.
shows you how to install kubeadm.
Once installed, you can use it to
create a cluster
.
View kubeadm Install Guide
1.1 - Install and Set Up kubectl on Linux
You must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.26 client can communicate with v1.25, v1.26, and v1.27 control planes.
Using the latest compatible version of kubectl helps avoid unforeseen issues.
The following methods exist for installing kubectl on Linux:
Install kubectl binary with curl on Linux
Install using native package management
Install using other package management
Download the latest release with the command:
curl -LO
"https://dl.k8s.io/release/
$(
curl -L -s https://dl.k8s.io/release/stable.txt
)
/bin/linux/amd64/kubectl"
Note:
To download a specific version, replace the
$(curl -L -s https://dl.k8s.io/release/stable.txt)
portion of the command with the specific version.
For example, to download version v1.26.0 on Linux, type:
curl -LO https://dl.k8s.io/release/v1.26.0/bin/linux/amd64/kubectl
Validate the binary (optional)
Download the kubectl checksum file:
curl -LO
"https://dl.k8s.io/
$(
)
/bin/linux/amd64/kubec

tl.sha256"
Validate the kubectl binary against the checksum file:
echo
"
$(
cat kubectl.sha256
)
kubectl"
| sha256sum --check
If valid, the output is:
kubectl: OK
If the check fails,
sha256
exits with nonzero status and prints output similar to:
kubectl: FAILED
sha256sum: WARNING:
1
computed checksum did NOT match
Note:
Download the same version of the binary and checksum.
Install kubectl
sudo install -o root -g root -m
0755
kubectl /usr/local/bin/kubectl
Note:
If you do not have root access on the target system, you can still install kubectl to the
~/.local/bin
chmod +x kubectl
mkdir -p ~/.local/bin
mv ./kubectl ~/.local/bin/kubectl
# and then append (or prepend) ~/.local/bin to $PATH
Test to ensure the version you installed is up-to-date:
Or use this for detailed view of version:
kubectl version --client --output=yaml
Debian-based distributions
Red Hat-based distributions
Update the
apt
package index and install packages needed to use the Kubernetes
apt
repository:
sudo apt-get update
sudo apt-get install -y ca-certificates curl
If you use Debian 9 (stretch) or earlier you would also need to install
apt-transport-https
:
sudo apt-get install -y apt-transport-https
Download the Google Cloud public signing key:
sudo curl -fsSLo /etc/apt/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
Add the Kubernetes
apt
echo
"deb [signed-by=/etc/apt/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main"
| sudo tee /etc/apt/sources.list.d/kubernetes.list
Update
apt
package index with the new repository and install kubectl:
sudo apt-get install -y kubectl
Note:
In releases older than Debian 12 and Ubuntu 22.04,
/etc/apt/keyrings
does not exist by default.
You can create this directory if you need to, making it world-readable but writeable only by admins.
cat
&lt;&lt;EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
enabled=1
gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
sudo yum install -y kubectl
Snap
Homebrew
If you are on Ubuntu or another Linux distribution that supports the
snap
package manager, kubectl is available as a
snap
snap install kubectl --classic
If you are on Linux and using
Homebrew
package manager, kubectl is available for
installation
.
brew install kubectl
Verify kubectl configuration
In order for kubectl to find and access a Kubernetes cluster, it needs a
,
which is created automatically when you create a cluster using
or successfully deploy a Minikube cluster.
By default, kubectl configuration is located at
.
Check that kubectl is properly configured by getting the cluster state:
If you see a URL response, kubectl is correctly configured to access your cluster.
If you see a message similar to the following, kubectl is not configured correctly or is not able to connect to a Kubernetes cluster.
The connection to the server &lt;server-name:port&gt; was refused - did you specify the right host or port?
For example, if you are intending to run a Kubernetes cluster on your laptop (locally), you will need a tool like Minikube to be installed first and then re-run the commands stated above.
If kubectl cluster-info returns the url response but you can't access your cluster, to check whether it is configured properly, use:
Optional kubectl configurations and plugins
Enable shell autocompletion
kubectl provides autocompletion support for Bash, Zsh, Fish, and PowerShell, which can save you a lot of typing.
Below are the procedures to set up autocompletion for Bash, Fish, and Zsh.
Bash
Fish
Zsh
The kubectl

 completion script for Bash can be generated with the command
. Sourcing the completion script in your shell enables kubectl autocompletion.
However, the completion script depends on
, which means that you have to install this software first (you can test if you have bash-completion already installed by running
type _init_completion
).
Install bash-completion
bash-completion is provided by many package managers (see
here
). You can install it with
apt-get install bash-completion
or
yum install bash-completion
, etc.
The above commands create
/usr/share/bash-completion/bash_completion
, which is the main script of bash-completion. Depending on your package manager, you have to manually source this file in your
~/.bashrc
file.
To find out, reload your shell and run
. If the command succeeds, you're already set, otherwise add the following to your
~/.bashrc
file:
source
Reload your shell and verify that bash-completion is correctly installed by typing
.
Enable kubectl autocompletion
Bash
You now need to ensure that the kubectl completion script gets sourced in all your shell sessions. There are two ways in which you can do this:
User
System
echo
'source &lt;(kubectl completion bash)'
&gt;&gt;~/.bashrc
kubectl completion bash | sudo tee /etc/bash_completion.d/kubectl &gt; /dev/null
If you have an alias for kubectl, you can extend shell completion to work with that alias:
echo
'alias k=kubectl'
echo
'complete -o default -F __start_kubectl k'
Note:
bash-completion sources all completion scripts in
/etc/bash_completion.d
.
Both approaches are equivalent. After reloading your shell, kubectl autocompletion should be working.
To enable bash autocompletion in current session of shell, run
exec bash
:
exec
bash
Note:
Autocomplete for Fish requires kubectl 1.23 or later.
The kubectl completion script for Fish can be generated with the command
To do so in all your shell sessions, add the following line to your
~/.config/fish/config.fish
file:
kubectl completion fish |
source
After reloading your shell, kubectl autocompletion should be working.
The kubectl completion script for Zsh can be generated with the command
To do so in all your shell sessions, add the following to your
~/.zshrc
file:
source
&lt;
(
)
If you have an alias for kubectl, kubectl autocompletion will automatically work with it.
If you get an error like
2: command not found: compdef
, then add the following to the beginning of your
~/.zshrc
file:
autoload -Uz compinit
compinit
Install
plugin
A plugin for Kubernetes command-line tool
kubectl
, which allows you to convert manifests between different API
versions. This can be particularly helpful to migrate manifests to a non-deprecated api version with newer Kubernetes release.
For more info, visit
migrate to non deprecated apis
curl -LO
$(
)
/bin/linux/amd64/kubectl-convert"
Download the kubectl-convert checksum file:
curl -LO
$(
)
/bin/linux/amd64/kubectl-convert.sha256"
Validate the kubectl-convert binary against the checksum file:
echo
"
$(
cat kubectl-convert.sha256
)
kubectl-convert"
kubectl-convert: OK
sha256
kubectl-convert: FAILED
1
Note:
Download the same version

 of the binary and checksum.
Install kubectl-convert
0755
kubectl-convert /usr/local/bin/kubectl-convert
Verify plugin is successfully installed
kubectl convert --help
If you do not see an error, it means the plugin is successfully installed.
Install Minikube
See the
getting started guides
for more about creating clusters.
Learn how to launch and expose your application.
If you need access to a cluster you didn't create, see the
Sharing Cluster Access document
.
Read the
kubectl reference docs
1.2 - Install and Set Up kubectl on macOS
The following methods exist for installing kubectl on macOS:
Install kubectl binary with curl on macOS
Install with Homebrew on macOS
Install with Macports on macOS
Download the latest release:
Intel
Apple Silicon
curl -LO
$(
)
/bin/darwin/amd64/kubectl"
curl -LO
$(
)
/bin/darwin/arm64/kubectl"
Note:
For example, to download version v1.26.0 on Intel macOS, type:
curl -LO
"https://dl.k8s.io/release/v1.26.0/bin/darwin/amd64/kubectl"
And for macOS on Apple Silicon, type:
curl -LO
"https://dl.k8s.io/release/v1.26.0/bin/darwin/arm64/kubectl"
Intel
curl -LO
$(
)
/bin/darwin/amd64/kubectl.sha256"
curl -LO
$(
)
/bin/darwin/arm64/kubectl.sha256"
echo
"
$(
)
kubectl"
| shasum -a
256
--check
shasum
shasum: WARNING:
1
Note:
Make the kubectl binary executable.
chmod +x ./kubectl
Move the kubectl binary to a file location on your system
PATH
.
sudo mv ./kubectl /usr/local/bin/kubectl
sudo chown root: /usr/local/bin/kubectl
Note:
Make sure
is in your PATH environment variable.
If you are on macOS and using
Homebrew
package manager, you can install kubectl with Homebrew.
Run the installation command:
or
brew install kubernetes-cli
Macports
package manager, you can install kubectl with Macports.
sudo port selfupdate
sudo port install kubectl
,
.
If you see a message similar to the following, k

ubectl is not configured correctly or is not able to connect to a Kubernetes cluster.
kubectl provides autocompletion support for Bash, Zsh, Fish, and PowerShell which can save you a lot of typing.
Bash
Fish
Zsh
The kubectl completion script for Bash can be generated with
. Sourcing this script in your shell enables kubectl completion.
However, the kubectl completion script depends on
which you thus have to previously install.
Warning:
There are two versions of bash-completion, v1 and v2. V1 is for Bash 3.2 (which is the default on macOS), and v2 is for Bash 4.1+. The kubectl completion script
doesn't work
correctly with bash-completion v1 and Bash 3.2. It requires
bash-completion v2
and
Bash 4.1+
. Thus, to be able to correctly use kubectl completion on macOS, you have to install and use Bash 4.1+ (
instructions
). The following instructions assume that you use Bash 4.1+ (that is, any Bash version of 4.1 or newer).
Upgrade Bash
The instructions here assume you use Bash 4.1+. You can check your Bash's version by running:
echo
$BASH_VERSION
If it is too old, you can install/upgrade it using Homebrew:
brew install bash
Reload your shell and verify that the desired version is being used:
echo
$SHELL
Homebrew usually installs it at
/usr/local/bin/bash
.
Note:
As mentioned, these instructions assume you use Bash 4.1+, which means you will install bash-completion v2 (in contrast to Bash 3.2 and bash-completion v1, in which case kubectl completion won't work).
You can test if you have bash-completion v2 already installed with
. If not, you can install it with Homebrew:
brew install bash-completion@2
As stated in the output of this command, add the following to your
~/.bash_profile
file:
brew_etc
=
"
$(
brew --prefix
)
/etc"
[[
-r
"
${
brew_etc
}
/profile.d/bash_completion.sh"
]]
.
"
${
brew_etc
}
Reload your shell and verify that bash-completion v2 is correctly installed with
.
You now have to ensure that the kubectl completion script gets sourced in all your shell sessions. There are multiple ways to achieve this:
Source the completion script in your
file:
echo
&gt;&gt;~/.bash_profile
Add the completion script to the
/usr/local/etc/bash_completion.d
kubectl completion bash &gt;/usr/local/etc/bash_completion.d/kubectl
echo
echo
If you installed kubectl with Homebrew (as explained
here
), then the kubectl completion script should already be in
/usr/local/etc/bash_completion.d/kubectl
. In that case, you don't need to do anything.
Note:
The Homebrew installation of bash-completion v2 sources all the files in the
BASH_COMPLETION_COMPAT_DIR
directory, that's why the latter two methods work.
In any case, after reloading your shell, kubectl completion should be working.
Note:
. Sourcing the completion script in your shell enab

les kubectl autocompletion.
file:
source
~/.zshrc
file:
source
&lt;
(
)
~/.zshrc
file:
compinit
Install
plugin
kubectl
Intel
curl -LO
$(
)
/bin/darwin/amd64/kubectl-convert"
curl -LO
$(
)
/bin/darwin/arm64/kubectl-convert"
Intel
curl -LO
$(
)
/bin/darwin/amd64/kubectl-convert.sha256"
curl -LO
$(
)
/bin/darwin/arm64/kubectl-convert.sha256"
echo
"
$(
)
| shasum -a
256
--check
shasum
1
Note:
Make kubectl-convert binary executable
chmod +x ./kubectl-convert
Move the kubectl-convert binary to a file location on your system
PATH
.
sudo mv ./kubectl-convert /usr/local/bin/kubectl-convert
sudo chown root: /usr/local/bin/kubectl-convert
Note:
Make sure
See the
.
Read the
1.3 - Install and Set Up kubectl on Windows
The following methods exist for installing kubectl on Windows:
Install kubectl binary with curl on Windows
Install on Windows using Chocolatey, Scoop, or winget
Download the
latest release v1.26.0
.
Or if you have
curl
installed, use this command:
curl.exe -LO
"https://dl.k8s.io/release/v1.26.0/bin/windows/amd64/kubectl.exe"
Note:
To find out the latest stable version (for example, for scripting), take a look at
https://dl.k8s.io/release/stable.txt
.
kubectl
checksum file:
"https://dl.k8s.io/v1.26.0/bin/windows/amd64/kubectl.exe.sha256"
Validate the
kubectl
binary against the checksum file

:
Using Command Prompt to manually compare
CertUtil
's output to the checksum file downloaded:
CertUtil -hashfile kubectl.exe SHA256
type
kubectl.exe.sha256
Using PowerShell to automate the verification using the
-eq
operator to get a
True
or
False
result:
$(
Get-FileHash
-Algorithm SHA256 .\kubectl.exe).Hash
-eq
$(
.\kubectl.exe.sha256)
Append or prepend the
kubectl
binary folder to your
PATH
Test to ensure the version of
kubectl
is the same as downloaded:
Note:
Docker Desktop for Windows
adds its own version of
kubectl
to
PATH
.
If you have installed Docker Desktop before, you may need to place your
PATH
entry before the one added by the Docker Desktop installer or remove the Docker Desktop's
kubectl
.
To install kubectl on Windows you can use either
Chocolatey
package manager,
Scoop
command-line installer, or
winget
package manager.
choco
scoop
winget
choco install
kubernetes-cli
scoop install kubectl
winget install -e --id Kubernetes.kubectl
Navigate to your home directory:
# If you're using cmd.exe, run: cd %USERPROFILE%
cd
~
Create the
.kube
mkdir .kube
Change to the
.kube
directory you just created:
cd
.kube
Configure kubectl to use a remote Kubernetes cluster:
New-Item
config -type file
Note:
Edit the config file with a text editor of your choice, such as Notepad.
,
.
Below are the procedures to set up autocompletion for PowerShell.
The kubectl completion script for PowerShell can be generated with the command
.
$PROFILE
file:
Out-String
|
Invoke-Expression
This command will regenerate the auto-completion script on every PowerShell start up. You can also add the generated script directly to your
$PROFILE
file.
To add the generated script to your
$PROFILE
file, run the following line in your powershell prompt:
kubectl completion powershell &gt;&gt;
$PROFILE
Install
plugin
kubectl
"https://dl.k8s.io/release/v1.26.0/bin/windows/amd64/kubectl-convert.exe"
Validate the bin

ary (optional).
"https://dl.k8s.io/v1.26.0/bin/windows/amd64/kubectl-convert.exe.sha256"
binary against the checksum file:
CertUtil
CertUtil -hashfile kubectl-convert.exe SHA256
type
kubectl-convert.exe.sha256
-eq
True
or
False
result:
$($(CertUtil -hashfile .\
.exe SHA256)[1]
-replace
" "
,
""
)
-eq
$(
type
.\
.exe.sha256)
PATH
Verify the plugin is successfully installed.
See the
.
Read the
1.4 - Tools Included
Snippets to be included in the main kubectl-installs-*.md pages.
1.4.1 - bash auto-completion on Linux
Some optional configuration for bash auto-completion on Linux.
The kubectl completion script for Bash can be generated with the command
).
here
or
, etc.
~/.bashrc
file.
~/.bashrc
file:
source
.
Bash
User
System
echo
echo
echo
Note:
.
exec bash
:
exec
bash
1.4.2 - bash auto-completion on macOS
Some optional configuration for bash auto-completion on macOS.
Warning:
and
Bash 4.1+
). The following instructions 

assume that you use Bash 4.1+ (that is, any Bash version of 4.1 or newer).
echo
echo
$SHELL
.
Note:
file:
brew_etc
=
"
$(
)
/etc"
[[
-r
"
${
brew_etc
}
]]
.
"
${
brew_etc
}
.
file:
echo
echo
echo
here
Note:
1.4.3 - fish auto-completion
Optional configuration to enable fish shell auto-completion.
Note:
file:
source
1.4.4 - kubectl-convert overview
A kubectl plugin that allows you to convert manifests from one version of a Kubernetes API to a different version.
kubectl
1.4.5 - PowerShell auto-completion
Some optional configuration for powershell auto-completion.
.
$PROFILE
file:
|
$PROFILE
file.
$PROFILE
$PROFILE
1.4.6 - verify kubectl

 install
How to verify kubectl.
,
.
1.4.7 - What's next?
What's next after installing kubectl.
See the
.
Read the
1.4.8 - zsh auto-completion
Some optional configuration for zsh auto-completion.
~/.zshrc
file:
source
&lt;
(
)
~/.zshrc
file:
compinit
2 - Administer a Cluster
2.1 - Administration with kubeadm
2.1.1 - Certificate Management with kubeadm
kubeadm
.
flag or the
.
By default this is
.
,
CA into
and
,
ca.crt
ca.key
and
CERTIFICATE                EXPIRES           

       RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED

,
and
).
Warning:
kubeadm
Note:
for
.
.
Warning:
bug
. After
to
and
with:
:
:
upgrade
.
upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping
Note:
to
or to
.
Warning:
bug
is
false
for the
.
command.
.
To restart 

a static Pod you can temporarily remove its manifest file from
value in
.
Warning:
Note:
Caution:
,
.
and
flags.
:
:
kind
:
:
extraArgs
:
:
:
See
You can create certificate signing requests with
.
--csr-dir
If
--csr-dir
) is used.
.
As with
--csr-dir
flag.
In
openssl
openssl ca
command
.
In
cfssl
.
).
For more information about manual rotation or replacement of CA

, see
.
to a
:
:
kind
:
---
:
kind
:
:
true
kubelet
key has a
.
field in
The field
.
field
to
true
.
Note:
Such a controller is not a secure mechanism unless it not only verifies the CommonName
to have
.
!
options.
.
--config
:
:
kind
:
# Will be used as the target "cluster" 

in the kubeconfig
:
:
:
=
jsonpath
=
johndoe
appdevs
group:
2.1.2 - Configuring a cgroup driver
.
The
page
systemd
of the
cgroupfs
systemd
.
This
Note:
,
kubeadm
systemd
.
kind
:
:
:
v1.21.0
---
kind
:
:
:
systemd
Note:
The
ConfigMap
init
,
join
and
upgrade
Using the
cgroupfs
driver
To use
cgroupfs
about its value. This applies to a case where you do not wish future versions
systemd
cgroupfs
driver,
systemd
driver
systemd
in-place,
Note:
systemd
Call
.
:
systemd
kubelet:
Updat

e the cgroup driver on all nodes
using
systemd
Set
in
using
2.1.3 - Reconfiguring a kubeadm cluster
operator
.
The
and
Note:
Warning:
in the
key.
Note:
The
.
under the
apiServer
,
,
scheduler
or
etcd
extraArgs
*SANs
.
kubeadm init phase certs &lt;compo

nent-name&gt; --config &lt;config-file&gt;
The
.
The
Note:
in the
key.
Run
flags
Note:
Note:
During
from the
.
after
.
key.
Once the
Note:
kubeadm deploys CoreDNS as a Deployment called
coredns
kube-dns
,
both in the
Note:
kubeadm writes Labels, Taints, CRI socket and other information on the Node object for a particul

ar
During
patches
.
stored in
or
.
2.1.4 - Upgrading kubeadm clusters
y &gt; x
.
.
minor
must
.
.
--config
flag of
with
instead.
apt update

file.
\
kubeadm
=
1.26.x-00
\
=
Note:
.
Note:
If
via the
--config

Check the
addons
page to
\
kubelet
=
1.26.x-00
kubectl
=
1.26.x-00
\
=
Ubu

ntu, Debian or HypriotOS
\
kubeadm
=
1.26.x-00
\
=
\
kubelet
=
1.26.x-00
kubectl
=
1.26.x-00
\
=
The
STATUS
Ready
If
again.
:
manually restored in
Ready
state
CoreDNS
and
Optionally backups the kube-apis

erver certificate.
2.1.5 - Upgrading Windows nodes
minikube
.
cluster
path-to
kubelet
path-to
kubelet
path-to
Note:
2.2 - Migrating from dockershim
dockershim to other container runtimes.
Check out
to unde

rstand your options for an alternative.
There is a
you can
2.2.1 - Changing the Container Runtime on a Node from Docker Engine to containerd
page
.
Note:
.
Replace
Follow the
guide
Linux
.
$Version
), and
:
$Version
$Version
Copy-Item
-Path
".\bin\"
cd
default
|
Out-File
and
.
kubeadm
file.
from
).
unix://
ideally.
Run

Note:
CentOS
Debian
Fedora
Ubuntu
.
Caution:
2.2.2 - Migrate Docker Engine nodes from dockershim to cri-dockerd
Note:
FAQ page
.
.
adapter
.
Note:
.
node:
Install
.
.
to
.
installed and started on each node.
A
.
Replace
\
Open
Modify the
flag to
.
Node
object
Node
object:
=
.
Change
kubeadm.alpha.ku

bernetes.io/cri-socket
from
to
.
Node
.
The
.
Read the
.
.
2.2.3 - Find Out What Container Runtime is Used on a Node
kubectl
kubectl
. See
Use
kubectl
outputs
endpoint
on
page.
CRI
protocol
Note:
You can check which socket you use by checking the kubelet configuration on your
nodes.
tr
or
pgrep
flag.
flag is not
remote
,
If the
,
.
2.2.4 - Troubleshooting CNI plugin-related errors
To avoid CNI plugin-rel

ated errors, verify that you are using or upgrading to a
to:
.
to add
node
.
"An
file"
section.
For
Upgrade to or install the most current version of the container runtime.
).
lo
loopback
. The
loopback
cni
loopback
i

s started.
cat
{
{
"ipam": {
[{
}],
[{
}]
],
]
}
},
{
}
]
}
EOF
2.2.5 - Check whether dockershim removal affects you
The
.
removal.
docker ps
),
),
.
If you have such tooling configured, test the behavior on test
A
kubelet
blog post.
Switching to Containerd as a container runtime eliminates the mi

ddleman. All the
docker ps
or
.
Note:
Read
Read the
2.2.6 - Migrating telemetry and security agents from dockershim
Note:
runtime
and
gained the ability to support multiple runtimes, and now supports any runtime
.
docker ps
or
docker top
to list
dockerd
volume
; or
hostPath
.
remove the
grep '/va

r/run/docker.sock'
\
-o
=
jsonpath
=
\
| sort
\
| grep
Note:
directory
/var/run
this
example
).
in
Google doc
.
Aqua
Datadog
datadog
dd-agent
Dynatrace
Falco
falco
Check
,
,
, and
.
The
.
The
Remove
.
being reported when docker is installed on node and no metrics when docker is not installed.
monitor.
Note:
2.3 - Generate Certificates Manually
easyrsa
,
openssl
or
cfssl
.
easyrsa
easyrsa
easyrsa3
.
cd
--batch
--req-cn
${
MASTER_IP
}

--days
=
"IP:
${
MASTER_IP
}
,"
\
"IP:
${
}
,"
\
\
\
\
\
\
--days
=
10000
\
Copy
,
, and
=
=
=
openssl
openssl
2048
-days
"/CN=
${
MASTER_IP
}
"
-days
10000
2048
)
csr.conf
).
[ req ]
=
2048
prompt
=
no
=
sha256
=
req_ext
=
dn
[ dn ]
C
=
ST
=
L
=
O
=
OU
=
CN
=
[ req_ext ]
=
DNS.1
=
DNS.2
=
DNS.3
=
DNS.4
=
DNS.5
=
IP.1
=
IP.2
=
[ v3_ext ]
=
=
CA:FALSE
keyUsage
=
=
=
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key
\
10000
\
cfssl
cfssl
mkdir cert
cd
cert
../cfssl print-defaults con

fig &gt; config.json
:
{
"signing"
: {
"default"
: {
"expiry"
:
"8760h"
},
: {
: {
"usages"
: [
"signing"
,
,
,
],
"expiry"
:
"8760h"
}
}
}
}
{
"CN"
:
,
"key"
: {
"algo"
:
"rsa"
,
"size"
:
2048
},
"names"
:[{
"C"
:
,
"ST"
:
,
"L"
:
,
"O"
:
,
"OU"
:
}]
}
ca.pem
):
{
"CN"
:
,
"hosts"
: [
,
,
,
,
,
,
,
],
"key"
: {
"algo"
:
"rsa"
,
"size"
:
2048
},
"names"
: [{
"C"
:
,
"ST"
:
,
"L"
:
,
"O"
:
,
"OU"
:
}]
}
and
=
=
\
--config
=
=
\
done.
in the
task page.
2.4 - Manage Memory, CPU, and API Resources
2.4.1 - Configure Default Memory Requests and Limits for a Namespace
Define a default memory resource limit for a namespace, so that every new Pod in that namespace has a memory resource limit configured.
namespace
.
limit
,
minikube
Each node in your cluster must have

 at least 2 GiB of memory.
.
:
v1
kind
:
metadata
:
name
:
spec
:
limits
:
-
default
:
memory
:
512Mi
:
memory
:
256Mi
type
:
Container
=
then the
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
=
=
=
limits:
requests:
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"1Gi"
=
=
=
  limits:
  requests:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
memory
:
"128Mi"
=
=
=
  limits:
  requests:
Note:
A
does
not
limit
request
value specified for the container in the spec that a client submits to the API serve

r. If that happens, the final Pod will not be scheduleable.
See
Clean up
2.4.2 - Configure Default CPU Requests and Limits for a Namespace
namespace
.
limit
request
,
minikube
read
.
.
:
v1
kind
:
metadata
:
name
:
spec
:
limits
:
-
default
:
cpu
:
1
:
cpu
:
0.5
type
:
Container
=
Here's a manifest for a Pod that has one contain

er. The container
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
=
=
=
cpu
cpu
.
limits:
cpu:
"1"
requests:
cpu: 500m
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
cpu
:
"1"
=
View the
cpu
:
  limits:
    cpu: "1"
  requests:
    cpu: "1"
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
cpu
:
"0.75"
=
View the
cpu
  limits:
    cpu: "1"
  requests:
    cpu: 750m
For every Pod that runs in the namespace, each of its containers must have a CPU limit.
Clean up
Configure Quality of Ser

vice for Pods
2.4.3 - Configure Minimum and Maximum Memory Constraints for a Namespace
namespace
.
minikube
:
v1
kind
:
metadata
:
name
:
spec
:
limits
:
-
max
:
memory
:
1Gi
min
:
memory
:
500Mi
type
:
Container
=
=
=
yaml
limits:
  - default:
    max:
    min:
of memory.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"800Mi"
requests
:
memory
:
"600Mi"
=
=
=
=
resources
:
limits
:
memory
:
800Mi
requests
:
memory
:
600Mi
=

:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"1.5Gi"
requests
:
memory
:
"800Mi"
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"800Mi"
requests
:
memory
:
"100Mi"
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
=
=
=
yaml
  limits:
  requests:
applied a
:
# Look for the "Requests:" section of the output
=
=

Clean up
2.4.4 - Configure Minimum and Maximum CPU Constraints for a Namespace
namespace
minikube
See
:
:
v1
kind
:
metadata
:
name
:
spec
:
limits
:
-
max
:
cpu
:
"800m"
min
:
cpu
:
"200m"
type
:
Container
=
=
=
The output shows the minimum and maximum CPU constraints as expected. But
limits
:
-
default
:
cpu
:
800m
:
cpu
:
800m
max
:
cpu
:
800m
min
:
cpu
:
200m
type
:
Container
Note:
default
and
Here's a manifest for

 a Pod that has one container. The container manifest
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
cpu
:
"800m"
requests
:
cpu
:
"500m"
=
=
=
=
resources
:
limits
:
cpu
:
800m
requests
:
cpu
:
500m
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
cpu
:
"1.5"
requests
:
cpu
:
"500m"
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
cpu
:
"800m"
requests
:
cpu
:
"100m"
=
That container is not acceptable because it specifies a CPU request that is lower than the
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
=
resources
:
limits
:
cpu
:
800

m
requests
:
cpu
:
800m
Clean up
2.4.5 - Configure Memory and CPU Quotas for a Namespace
namespace
.
object.
minikube
:
v1
kind
:
metadata
:
name
:
spec
:
hard
:
:
"1"
:
1Gi
:
"2"
:
2Gi
=
=
=
yaml
The memory request total for all Pods in that namespace must not exceed 1 GiB

.
See
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"800Mi"
cpu
:
"800m"
requests
:
memory
:
"600Mi"
cpu
:
"400m"
=
=
=
=
yaml
status:
  hard:
  used:
jq
JSONPath
)
used
values,
and
=
jsonpath
=
| jq .
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
redis
resources
:
limits
:
memory
:
"1Gi"
cpu
:
"800m"
requests
:
memory
:
"700Mi"
cpu
:
"400m"
=
the memory request total for all Pods running in a namespace.
.
Clean up
2.4.6 - Configure a Pod Quota for a Namespace
in a
Namespace
. You specify quotas in 

a
object.
minikube
:
v1
kind
:
metadata
:
name
:
pod-demo
spec
:
hard
:
pods
:
"2"
=
=
=
yaml
spec
:
hard
:
pods
:
"2"
status
:
hard
:
pods
:
"2"
used
:
pods
:
"0"
:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
purpose
:
replicas
:
3
template
:
metadata
:
labels
:
purpose
:
spec
:
:
-
name
:
image
:
nginx
=
=
=
yaml
spec
:
...
replicas
:
3
...
status
:
:
2
...
:
:
:
pods=2'
CronJobs
Clean up
2.5 - Install a Network Policy Provider
2.5.1 - Use Antrea for NetworkPolicy
.
Follow
2.5.2 - Use Calico for NetworkPolicy

cloud
or
local
cluster.
:
gcloud
.
flag.
Syntax
[
]
Example
=
calico
Running
.
.
2.5.3 - Use Cilium for NetworkPolicy
.
minikube
.
=
cni
cilium-ca
command.
status
command
here
.
=
=
cilium
...
A
cilium
Have fun, and if you have questions, contact us 

using the
.
2.5.4 - Use Kube-router for NetworkPolicy
2.5.5 - Romana for NetworkPolicy
.
Follow the
.
.
2.5.6 - Weave Net for NetworkPolicy
Follow the
guide.
iptables
Running
and
2/2 READY
. (
2/2
weave
and
weave-npc
.)
.
2.6 - Access Clusters Using the Kubernetes API
minikube
.
kubectl
.
a
,
Check the location and credentials that 

kubectl knows about with this command:
Many of the
examples
.
curl
or
wget
=
8080
&amp;
See
{
: [
"v1"
],
: [
{
:
,
:
}
]
}
Using
grep/cut
approach:
jsonpath
=
export
=
APISERVER
=
$(
jsonpath
=
)
metadata:
EOF
while
'^token'
do
echo
&gt;&amp;
2
sleep
1
done
TOKEN
=
$(
jsonpath
=
)
curl -X GET
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
~/.kube
for this.
Go
,
Python
,
Java
,
dotnet
,
, and
Haskell
for accessing the API from other l

anguages and how they authenticate.
Go client
See
Note:
example
:
package
main
import
(
"context"
"fmt"
)
func
main
() {
config, _
:=
(
""
,
)
:=
(config)
pods, _
:=
CoreV1
().
Pods
(
""
).
List
(context.
TODO
fmt.
Printf
(
,
len
}
.
To use
. See
example
:
from
import
config
.
v1
=
client
.
print
(
)
ret
=
v1
.
=
False
)
for
i
in
ret
.
items:
print
(
"
%s
\t
%s
\t
%s
"
%
(i
.
status
.
pod_ip, i
.
metadata
.
.
metadata
.
name))
, run:
cd
java
See
example
:
package
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
import
;
/**
*
*
*/
public
class
{
public
static
void
main
(
String
[]
args
)
throws
,
{
=
;
=
.
(
.
(
new
(
))).
build
();
.
(
client
);
=
new
CoreV1Api
();
=
api
.
(
null
,
null
,
null
,
null
,
null
,
null
,
null
,
null
,
null
);
System
.
out
.
println
(
);
for
(
V1Pod item
:
list
.
getItems
())
{
System
.
out
.
println
(
item
.
().
getName
());
}
}
}
To use
dotnet clien

t
See
example
:
using
System
;
using
k8s
;
namespace
simple
{
internal
class
PodList
{
private
static
void
Main(
string
[] args)
{
var
new
);
var
"default"
);
foreach
(
var
item
in
{
}
if
0
)
{
"Empty!"
);
}
}
}
}
To install
. See
example
:
const
k8s
=
require(
);
const
kc
=
new
const
k8sApi
=
'default'
});
See
The
example
:
::
IO
()
=
do
oidcCache
&lt;-
$
newTVar
$
Map
.
fromList
[]
&lt;-
$
mgr
kcfg
(
CoreV1
.
Accept
MimeJSON
))
&gt;&gt;=
print
2.7 - Advertise Extended Resources for a Node
minikube
.
[
{
"op"
:
"add"
,
"path"
:
,
"value"
:
"4"
}
]
Replace
\
\
--data
\
http://localhost:8001/api/v1/nodes/&lt;your-node-nam

e&gt;/status
Note:
~1
Capacity
:
cpu
:
2
memory
:
2049008Ki
:
4
.
Capacity
:
...
:
8
Capacity
:
...
:
800Gi
Clean up

[
  {
  }
]
Replace
\
\
--data
\
2.8 - Autoscale the DNS Service in a Cluster
minikube
.
Make sure
Deter

mine whether DNS horizontal autoscaling is already enabled
List the
namespace
:
=
...
...
.
=
=
...
...
=
coredns
or
kube-dns
.
where
Note:
kube-dns.
image.
kind
:
:
v1
metadata
:
name
:
namespace
:
---
kind
:
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
""
]
resources
:
[
"nodes"
]
verbs
:
[
"list"
,
"watch"
]
-
apiGroups
:
[
""
]
resources
:
[
]
verbs
:
[
"get"
,
"update"
]
-
apiGroups
:
[
"apps"
]
resources
:
[
,
]
verbs
:
[
"get"
,
"update"
]
-
apiGroups
:
[
""
]
resources
:
[
]
verbs
:
[
"get"
,
"create"
]
---
kind
:
:
metadata
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
---
:
apps/v1
kind
:
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
:
"true"
spec
:
selector
:
:
k8s-app
:
template
:
metadata
:
labels
:
k8s-app
:
spec
:
:
:
:
type
:
:
[
65534
]
fsGroup
:
65534
:
:
linux
:
-
name
:
image
:
resources
:
requests
:
cpu
:
"20m"
memory
:
"10Mi"
command
:
- --v=2
:
-
key
:
operator
:
"Exists"
:
with your s

cale target.
ConfigMap
exists:
=
...
...
=
linear
:
and
are
floats.
cores,
.
=
0
=
=
...
...
=
,
An autoscaler Pod runs a client that polls the Kubernetes API server for the
patterns:
linear
and
ladder
.
Read about
.
.
2.9 - Change the default StorageClass
You need to have a Kubernetes cluster, and the k

ubectl command-line tool must
minikube
.
standard
(
default
)
(default)
.
set to
true
false
.
false
:
where
standard
.
without
gold
(
default
)
.
2.10 - Change the Reclaim Policy of a PersistentVolume
This page shows how to change the reclaim policy of a Kubernetes
minikube
.

where
Note:
double
"{\"
spec\
":{\"
":\"
Retain\
"}}"
Retain
.
.
.
field
2.11 - Cloud Controller Manager Administration
The
kubernetes authentication/authorization: cloud-controller-manager may need RBAC rules set to speak to the kubernetes 

apiserver
and
kubelet
Examples
.
---
:
v1
kind
:
metadata
:
name
:
namespace
:
---
:
kind
:
metadata
:
name
:
roleRef
:
apiGroup
:
kind
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
---
:
apps/v1
kind
:
DaemonSet
metadata
:
labels
:
k8s-app
:
name
:
namespace
:
spec
:
selector
:
:
k8s-app
:
template
:
metadata
:
labels
:
k8s-app
:
spec
:
:
:
-
name
:
# for in-tree providers we use 

registry.k8s.io/cloud-controller-manager
image
:
command
:
:
-
key
:
value
:
"true"
effect
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
:
""
here
.
.
2.12 - Configure a kubelet image credential provider
API calls to a cloud 

provider service are required to retrieve authentication information for a registry.
.
ECR
:
kind
:
providers
:
-
name
:
ecr
#
#
#
# - gcr.io
:
-
-
-
-
-

:
"12h"
:
args
:
env
:
-
name
:
value
:
The
providers
name
.
The
*.k8s.io
or
k8s.*.io
,
k8s.*
*.io
*.k8s.io
.
gcr.io
in the
.
Read the
.
2.13 - Configure Quotas for API Objects
PersistentVolumeClaims and Services. A quota restricts the number of
object.
minikube
.
:
v1
kind
:
metadata
:
name
:
spec
:
hard
:
:
"1

"
:
"2"
:
"0"
=
=
=
yaml
status
:
hard
:
:
"1"
:
"2"
:
"0"
used
:
:
"0"
:
"0"
:
"0"
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
3Gi
=
=
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
4Gi
=
Notes
by quotas:
String
API Object
"pods"
Pod
Service
"secrets"
Secret
ConfigMap
Clean up
2.14 - Control CPU Management Policies on the Node
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you d

o not already have a
minikube
.
CFS quota
kubelet
flag or the
field in
.
none
static
.
flag.
key=value
and
Drain
the node.
CPUManager so that the cpu-sets set up by the new policy won’t conflict with it.
The
none
and
The
static
requests
using the
.
Note:
System services such as the container runtime and the kubelet itself 

can continue to run on these exclusive CPUs.  The exclusivity only extends to other pods.
Note:
state file
or
and
and
Burstable
CPU
requests
requests
Note:
and/or
or
when
As
CPU
limit
workload.
spec
:
:
-
name
:
nginx
image
:
nginx
requests
or
limits
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
Burstable
requests
do not
equal
limits
and the
cpu
pool.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
requests
:
memory
:
"100Mi"
cpu
:
"1"
Burstable
requests
do not
equal
limits
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
requests
:
memory
:
"200Mi"
cpu
:
"2"
requests
limits
.
nginx
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"1.5"
requests
:
memory
:
"200Mi"
cpu
:
"1.5"
requests
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
limits
and
requests
limits
nginx
You can toggle groups of opti

ons on and off based upon their maturity level
policy:
If the
to the
.
.
If the
If the
minimum
policy and does not apply to hardware where the number of sockets is greater
The
to
cores.
The
to the
and
2.15 - Control Topology Management Policies on a node
workloads in fields such as telecommunications, scientific comput

ing, machine learning, financial
minikube
.
scope
and
policy
.
The
scope
pod
or
container
policy
,
,
, etc.).
scopes
and
policies
Note:
See
.
Note:

container
(default)
pod
flag.
The
container
pod
scope.
pod scope
pod
.
Using the
pod
none
(default)
Note:
pod
the same
The
preferred, Topology

 Manager will reject this pod from the node. This will result in a pod in a
not
attempt to
have the
error.
not
attempt to
error.
and
If the
and
the
spec
:
:
-
name
:
nginx
image
:
nginx
requests
or
limits
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
Burstable
none
static
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
me

mory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
are equal
to
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
are equal
to
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
:
"1"
:
"1"
requests
:
:
"1"
:
"1"
static
static
pod, the
none
pod, the
static
2.16 - Customizing DNS Service
Pod(s)
minikube
.
using the
.
Note:
kube-dns
in the
field.
the legacy
kube-dns
kube-dns
flag.
.
reverse IP address lookups (P

TR records), and more. For more information, see
.
If a Pod's
dnsPolicy
is set to
default
But see
.
CoreDNS
.
Corefile
,
ConfigMap
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
errors
health {
}
ready
ttl 30
}
cache 30
loop
reload
}
plugins
errors
health
lameduck
ready
ttl
The
kube-dns
.
The
in the
format
forward
cache
loop
reload
using the
.
Example
Consul
    errors
    cache 30
}
point the
forward
Core

file
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
errors
health
}
cache 30
loop
reload
}
errors
cache 30
}
Note:
Read
2.17 - Debugging DNS Resolution
minikube
addon
kube-dns.
.
:
v1
kind
:
Pod
metadata
:
name
:
dnsutils
namespace
:
default
spec
:
:
-
name
:
dnsutils
image
:
command
:
- sleep
-
:
:
Always
Note:
default
.
nslookup
kubectl
exec

If the
nslookup
(See
and
kubectl
exec
kubectl
exec

or
kubectl
exec

Use the
=
=
kube-dns
...
...
Note:
k8s-app
is
kube-dns
will have to deplo

y it manually.
Use the
=
=
kube-dns
.:53
command.
=
...
...
Note:
kube-dns
for
command.
=
log
ConfigMap
named
coredns
Then add
log
:
v1
kind
:
ConfigMap
metadata
:
name
:
coredns
namespace
:
data
:
Corefile
:
|
.:53 {
log
errors
health
upstream
}
cache 30
loop
reload
}
.:53
service
and
endpoint
:
  services                         []   

              []              [list watch]
...
  verbs:
  - list
  - watch
...
kubectl
exec
kubectl
exec
.
flag
(With
, this is
).
dnsmasq
flag.
See
.
Read
2.18 - Declare Network Policy
Note:
minikube
.
Antrea
Calico
Cilium
Romana
Weave Net
Create an
nginx
nginx
nginx
.
nginx
. The
nginx
Pod 

and Deployment are found in the
default

nginx
nginx
default
=
1
nginx
nginx
service
nginx
:
kind
:
metadata
:
name
:
spec
:
:
:
app
:
nginx
ingress
:
-
from
:
-
:
:
access
:
"true"
.
Note:
app=nginx
nginx
file:
nginx
=
1
nginx
=
1
nginx
2.19 - Developing Cloud Controller Manager
component
Out of tree
Create a go p

ackage with an implementation that satisfies
.
Use
main.go
main.go
main.go
init
.
In tree
DaemonSet
2.20 - Enable Or Disable A Kubernetes API
.
as a
The
api/all
.
to the
.
Read the
for the
2.21 - Encrypting Secret Data at Rest
minikube
.
The
.
Caution:
For high-availability configurations (with two or more control plane nodes), the
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
aesgcm
:
keys
:
-
name
:
key1
secret
:
-
name
:
key2
secret
:
-
aescbc
:
keys
:
-
name
:
key1
secret
:
-
name
:
key2
secret
:
-
secretbox
:
keys
:
-
name
:
key1
secret
:
Each
resources
resource
or
and

 the cluster version is 1.26 or newer,
See the
section.
The
providers
identity
or
aescbc
.
Caution:
Providers
Name
Strength
Speed
Key Length
identity
None
N/A
N/A
N/A
secretbox
Strong
Faster
32-byte
aesgcm
Fastest
aescbc
PKCS#7
padding
Weak
Fast
32-byte
kms
PKCS#7
Strongest
Fast
32-bytes
.
Caution:
kms
provider for additional security.
identity
keys.
:
kind
:
resources
:
-
resources
:
- secrets
- pandas.awesome.bears.examp

le
providers
:
-
aescbc
:
keys
:
-
name
:
key1
secret
:
-
identity
:
{}
head -c
32
secret
struct.
Set the
flag on the
to point to
:
v1
kind
:
Pod
metadata
:
:
:
:
6443
:
null
labels
:
component
:
tier
:
name
:
namespace
:
spec
:
:
-
command
:
...
:
...
-
name
:
enc
mountPath
:
readonly
:
true
...
volumes
:
...
-
name
:
enc
hostPath
:
path
:
type
:
...
Caution:
etcdctl
secret1
in the
default
=
mykey
=
mydata
Using the
etcdctl
where
[...]
=
3
etcdctl
\
--cacert
=
\
--cert
=
\
--key
=
\
00000000
2f
72
65
67
69
73
74
72
79
2f
73
65
63
72
65
74
|
|
00000010
73
2f
64
65
66
61
75
6c
74
2f
73
65
63
72
65
74
|
|
00000020
31
0a
6b
38
73
3a
65
6e
63
3a
61
65
73
63
62
63
|
|
00000030
3a
76
31
3a
6b
65
79
31
3a
c7
6c
e7
d3
09
bc
06
|
|
00000040
25
51
91
e4
e0
6c
e5
b1
4d
7a
8b
3d
b9
c2
7c
6e
|
|
00000050
b4
79
df
05
28
ae
0d
8e
5f
35
13
2c
c0
18
99
3e
|
|
[...]
00000110
23
3a
0d
fc
28
ca
48
2d
6b
2d
46
cc
72
0b
70
4c
|
|
00000120
a5
fc
35
43
12
4e
60
ef
bf
6f
fe
cf
df
0b
ad
1f
|
|
00000130
82
c4
88
53
02
da
3e
66
ff
0a
|
|
0000013a
the
aescbc
etcd
key1
is used in
etcd
and in
.
, wi

th contents of
mydata
Note:
keys
Run
identity
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
aescbc
:
keys
:
-
name
:
key1
secret
:
.
2.22 - Guaranteed Scheduling For Critical Add-On Pods
A static pod marked as critical, can't be evicted. However, a non-static pods marked as critical are always rescheduled.
or
.
.
2.23 - IP Masquerade Agent User Guide
.
minikube
.
The
configures iptables rules to hide a pod's IP address behind the cluster node's IP address. This is typically done when sending traffi

c to destinations outside the cluster's pod
CIDR
range.
Key Terms
Link Local
RFC 1918
CIDR
CIDR
MASQUERADE  all  --  anywhere             anywhere             /* ip-masq-agent:

 outbound traffic should be subject to MASQUERADE (this match must come after cluster-local CIDR matches) */ ADDRTYPE match dst-type !LOCAL
DaemonSet
=
true
here
ConfigMap
ConfigMap
Note:
:
:
-
10.0.0.0
/8
:
60s
=
=
:
-
10.0.0.0
/8
:
60s
:
true
2.24 - Limit Storage Consumption
,
,
and
.
minikube
.
Adding a
via
. The admission

 controller that enforces limit ranges will reject any PVC that is above or below
:
v1
kind
:
metadata
:
name
:
spec
:
limits
:
-
type
:
max
:
storage
:
2Gi
min
:
storage
:
1Gi
:
v1
kind
:
metadata
:
name
:
spec
:
hard
:
:
"5"
:
"5Gi"
Summary
2.25 - Migrate Replicated Control Plane To Use Cloud Controller Manager
component
.
and the
on
or
and
clo

ud-controller-manager
flag and
and
.
and
and
.
role.
runs
route
,
service
, and
kind
:
:
:
:
-
name
:
route
component
:
-
name
:
service
component
:
-
name
:
component
:
component
to
*
kind
:
:
:
:
-
name
:
route
component
:
*
-
name
:
service
component
:
*
-
name
:
component
:
*
Restart

component
to
kind
:
:
:
:
-
name
:
route
component
:
-
name
:
service
component
:
-
name
:
component
:
and
.
flag set to
external
for
.
.
and
and
and
.
.
For
and
Special case: migrating

 the Node IPAM controller
nodeipam
kind
:
:
:
:
-
name
:
route
component
:
*
-
name
:
service
component
:
*
-
name
:
component
:
*
-
name
:
nodeipam
-
component
:
*
Read the
2.26 - Namespaces Walkthrough
A scope for
Names
.
minikube
.
You have an
.
Pods
,
Services
, and
.
and
.
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
name
:
Create the
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
name
:
developmen

t   Active    29s       name=development
:
v1
clusters
:
-
cluster
:
:
REDACTED
server
:
name
:
contexts
:
-
context
:
cluster
:
user
:
name
:
:
kind
:
Config
:
{}
users
:
-
name
:
user
:
:
REDACTED
:
REDACTED
token
:
-
name
:
user
:
password
:
username
:
admin
=
\
--cluster
=
\
--user
=
=
\
--cluster
=
\
--user
=
:
v1
clusters
:
-
cluster
:
:
REDACTED
server
:
name
:
contexts
:
-
context
:
cluster
:
user
:
name
:
-
context
:
cluster
:
namespace
:
user
:
name
:
dev
-
context
:
cluster
:
namespace
:
user
:
name
:
prod
:
kind
:
Config
:
{}
users
:
-
name
:
user
:
:
REDACTED
:
REDACTED
token
:
-
name
:
user
:
password
:
username
:
admin
dev
:
apps/v1
kind
:
metadata
:
labels
:
app
:
snowflake
name
:
snowflake
spec
:
replicas
:
2
selector
:
:
app
:
snowflake
template
:
metadata
:
labels
:
app
:
snowflake
spec
:
:
-
image
:
:
Always
name
:
snowflake
snowflake
app
=
snowflake
And this is great, de

velopers are able to do what they want, and they do not have to worry about affecting content in the
The
=
=
5
app
=
cattle
2.27 - Operating etcd clusters for Kubernetes
back up
plan
.
minikube
.
stable.
.
3.2.10+
.
.
=
http://
:2379
\
=
http://
:2379
.
Make sure
.
Configure an etcd cluster either by static member informati

on or by dynamic
.
,
,
,
, and
=
http://
$IP1
$IP2
$IP3
$IP4
$IP5
=
http://
$IP1
$IP2
$IP3
$IP4
$IP5
:2379
.
$LB
.
.
peer.key
and
peer.cert
and
and
and
and
that are
etcd.ca
and
.
flags
,
and
.
Note:
.
member.
unique name for each member to avoid human errors. For example

, consider a
,
, and
. When
member1
fails,
.
member1
:
=
flag, then
etcd.
dir.
=
10.0.0.4
:
export
ETCD_NAME
=
"member4"
export
=
export
=
existing
etcd
[
flags
]
flag, then
.
$ENDPOINT
to the file
:
=
3
$ENDPOINT
=
3
=
volume.
=
3
etcdctl -h
will list various opti

ons available from etcdctl. For example, you can take a snapshot by specifying
=
3
=
\
--cacert
=
=
=
\
where
,
cert-file
and
key-file
the
.
=
3
=
3
export
=
3
.
. Replace
and
instead.
Note:
stop
all
,
,
kubelet
Note:
2.28 - Reconfigure a Node's Kubelet in a Live Cluster
Caution:
The
kubelet
in a runni

ng Kubernetes cluster,
ConfigMap
each
Node
to use it.
.
kubeadm
, refer to
.
See the
.
2.29 - Reserve Compute Resources for System Daemons
Capacity
node.
The
kubelet
minikube
.
to set an
.
v1.Node
of
in the CLI.
kubelet
.
kubelet
kubelet
.
The
kubelet
flag.
cgroupfs
systemd
proper system behavior. For

 example, if operators use the
systemd
kubelet
must
systemd
:
:
kubelet
,
,
, etc.
cpu
,
memory
, and
,
pid
may be
kubelet
flag.
does not
create
systemd
,
with
.slice
appended.
:
:
like
sshd
,
udev
, etc.
memory
for the
kernel
too since
kernel
in
cpu
,
memory
, and
,
pid
may be
daemons.
Note that
kubelet
does not
create
exist.
kubelet
systemd
,
with
.slice
appended.
:
:
does not
have
and
,
defined by
and
options.
defined by this option, other mechanism outside Kubernet

es should be used.
:
provides
memory
and
flag, the
kubelet
:
capacity
for pods.
kubelet
on the
pods
.
kubelet
and
&amp;
values in
or
,
or
.
kubelet
should
pods
.
over time.
Node has
32Gi
of
memory
,
16 CPUs
and
100Gi
of
Storage
is set to
is set to
is set to
88Gi
requests
If
and/or
kubelet
evicts pods whenever the 

overall node memory
storage
2.30 - Running Kubernetes Node Components as a Non-root User
.
.
Note:
.
.
and
Enable the
kind
kind
See
.
minikube
minikube
Note:
sysbox
Sysbox
See
/proc
and
/sys
Note:
K3s
K3s
See
Usernetes
Usernetes
$HOME
See
Note:
.
with
.
You do
not
chroot()
nor
in
in
the namespace (

not
outside
/etc
/run
/var/logs
Note:
cgroup v2
Note:
,
VPNKit
, or
.
NodePort
socat(1)
.
See
the
package
of k3s.
CRI-O
version =
2
[plugins.
]
true
true
true
[plugins.
[plugins.
false
.
.
to be set.
[crio]
"overlay"
]
.
.
:
kind
:
:
:
true
# W

e use cgroupfs that is delegated by systemd, so we do not use "systemd" driver
:
When the
.
/dev/kmsg
.
.
The
Sysbox
:
kind
:
mode
:
conntrack
:
:
0
:
0s
:
0s
Caveats
nfs
and
iscsi
local
,
hostPath
,
emptyDir
,
configMap
,
secret
, and
page
See Also
Usernetes
2.31 - Safely Drain a Node
node
,
.
concept,
and have
for
.
Use
You can use
to
Note:
By default
the
When
(or equivalentl

y, if on a cloud platform, delete the virtual machine backing the node).
The
,
healthy
;
healthy
(such as
.
.
2.32 - Securing a Cluster
minikube
.
resources (pods, services, no

des) and can be namespace-scoped or cluster-scoped. A set of out-of-the-box
Node
and
RBAC
policies
in a
namespace
Baseline
or
To prevent specific modules from 

being automatically loaded, you can uninstall them from

The
and the
Caution:
The
and archive the audit file on 

a secure server.
do not use.
like the
.
If you use
and allow
kube-
Secret
or
ConfigMap
Join the
2.33 - Set Kubelet parameters via a config file
struct.
apiVe

rsion
:
kind
:
address
:
port
:
20250
:
false
:
:
"200Mi"
Note:
Note:
.
See
--config
If
--config
.
2.34 - Share a Cluster with Namespaces
Have an
.
Pods
,
Services
, and
.
default

Namespace
to define
Hard
Namespace
a
Namespace
.

See
Active
Namespace
Note:
kube-
:
v1
kind
:
Namespace
metadata
:
name
:
Then run:
DNS label
.
design doc
.
Warning:
state.
and
.
Create the
=
=
=
2
snowflake
=
kubectl

 get pods -l
app
=
=
The
=
=
=
=
=
5
-n
=
=
app
=
cattle -n
=
A scope for
Names
.
Service
DNS entry
.
.
See
.
2.35 - Upgrade A Cluster
This page provides an overview of the steps you 

should follow to upgrade a
kubectl
kubeadm
kubeadm
kubectl
.
Caution:
plugins.
kubectl
.
drain
You can use
The
kubectl
pod.yaml
kind
to
.
Refer to
and
2.36 - Use Cascading Deletion in a Cluster
.
minikube
app
=
=
yaml
:
v1
...
:

-
:
apps/v1
:
true
:
true
kind
:
name
:
uid
:
...
kubectl
.
kubectl
or the
=
=
8080
Use
curl
\
-d
\
-H
finalizer
like this:
    ]
    ...
.
Use either
kubectl
.
kubectl
--cascade
flag or the
argument.
=
=
8080
Use
curl
\
-d
\
-H
...
}
orphan
kubectl
.
=
orphan
=
8080
Use
curl
\
-d
\
-H
orphan
in the
  "orphan"
],
...
app
=
nginx
.
.
2.37 - Using a KMS provider for data encryption
This p

age shows how to configure a Key Management Service (KMS) provider and plugin to enable secret data encryption. Currently there are two KMS API versions. KMS v1 will continue to work while v2 develops in maturity. If you are not sure which KMS API version to pick, choose v1.
minikube
.
KMS v1
KMS v2
kms
in the
providers
KMS v1
name
endpoint
cachesize
timeout
KMS v2
name
endpoint
cachesize
timeout
See
.
you use a proto file to create 

a stub file that you can use to develop the gRPC server code.
KMS v1
api.pb.go
api.proto
KMS v2
api.pb.go
api.proto
Notes
KMS v1
v1beta1
v1beta1
as
.
v1beta1
unix
)
/@
, i.e.
KMS v2
v2alpha1
v2alpha1
as
, "ok" as
unix
)
/@
, i.e.
kms
provider
Set the
KMS v1
:
kind
:
resources
:
-
resources
:

- secrets
providers
:
-
kms
:
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
-
kms
:
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
KMS v2
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
kms
:
:
v2
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
-
kms
:
name
:
endpoint
:
cachesize
:
100
timeout
:
3s
Setting
to
true
KMS v1 only
KMS v2 only
No KMS
None
.
,
etc.
providers
identity
.
,
etcdctl
secret1
in the
default
=
mykey
=
mydata
Using the
etcdctl
=
3
[
...
]
where
[...]
kms
kms
Add the
kms
:
kind
:
resources
:
-

resources
:
- secrets
providers
:
-
kms
:
name
:
endpoint
:
cachesize
:
100
-
aescbc
:
keys
:
-
name
:
key1
secret
:
kms
provider.
Place the
identity
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
kms
:
name
:
endpoint
:
cachesize
:
100
2.38 - Using CoreDNS for Service Discovery
minikube
.
CoreDNS
CNCF
.
kube-dns
For
kubeadm
kubeadm
using
kube-dns
kubeadm
kube-dns
.
kubeadm
tool,
kubeadm
.
CoreDNS
for the
.
2.39 - Using NodeLocal DNSCache in Kubernetes Clusters
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acti

ng as control plane hosts. If you do not already have a
minikube
.
serviceIP
(
default
Note:
"
kubedns
=
`
jsonpath
={
}
`
domain
=
localdns
=
is "
is the
sed -i
$localdns
$domain
$kubedns
/g"
and
the
pods.
as well as
sed -i
$localdns
$domain
$kubedns
/g"
&lt;node-local

-address&gt;
.
The
Run
to use
flag,
.
namespace
CoreDNS
.
kube-dns
namespace
shown in
.
The
kube-dns
The
From
,
using the
If a
The
in
2.40 - Using sysctls in a Kubernetes Cluster
sysctl
Note:
/
or
.
using a
the page
from
You need to have a Kubernetes cluster, and the kubectl command-line to

ol must
minikube
virtual
kernel.
)
net.
)
vm.
)
dev.
)
.
safe
and
unsafe
safe
isolated
safe
of a pod.
safe
.
safe
set:
,
,
,
Note:
All
safe
All
unsafe
unsafe
Unsafe
\
...
For
Minikube
flag:
=
...
Only
in future versions of the Linux kernel.
,
,
,
,
net.*
and
and
safe
and
unsafe
sysctls in
Warning:
destabilizing your oper

ating system.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
sysctls
:
-
name
:
value
:
"0"
-
name
:
value
:
"1024"
-
name
:
value
:
"65536"
...
Warning:
unsafe
, the use of
unsafe
sysctls
tainted
feature
unsafe
unsafe
sysctls it
feature
or
2.41 - Utilizing the NUMA-aware Memory Manager
QoS class
.
minikube
.
See
;
See
.
.
kubelet
To immediately put the Memory Manager into operation follow the guidelines in the section
.
cgroups
(i.e.
) for pods.
and below:
Startup
(section
).
Memory Mana

ger KEP: Memory Maps at start-up (with examples)
.
flag when
Static
Runtime
Reference
.
Static
).
).
Policies
kubelet
flag
:
None
(default)
Static
The
None
pod, the
Static
NodeMap
object.
or
Burstable
pod, the
Static
NodeMap
object.
The
mechanism
,
and
.
A new
flag was added to Memory Manager to allow for this total reserved memory
10Gi
the
1Gi
9Gi
.
If at least one node allocatable parameter is non-zero, you will need

 to specify
In fact,
100Mi
if
Static
memory
or
Syntax:
N
0
memory
or
value
1Gi
or
or
,
where
i
),
we reserve
3Gi
Note:
not
zero.
=
=
true
=
cpu
=
4,memory
=
4Gi
=
cpu
=
1,memory
=
1Gi
=
Static
None
QoS class.
pod.
The following excerpts from pod manifests assign a pod to the
QoS class.
requests
limits
:
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
system logs - include valuable information for debugging, e

.g., about generated hints
(includes
)
can be used
Use
or
cgroups
and
updates.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
consumer
:
Never
resources
:
limits
:
cpu
:
"2"
memory
:
150Gi
requests
:
cpu
:
"2"
memory
:
150Gi
command
:
[
"sleep"
,
]
:
{
:
"Static"
,
:{
"0"
:{
:
1
,
:{
:{
"total"
:
0
,
:
0
,
:
0
,
:
0
,
"free"
:
0
},
"memory"
:{
"total"
:
,
:
,
:
,
:
,
"free"
:
0
}
},
"nodes"
:[
0
,
1
]
},
"1"
:{
:
1
,
:{
:{
"total"
:
0
,
:
0
,
:
0
,
:
0
,
"free"
:
0
},
"memory"
:{
"total"
:
,
:
,
:
,
:
,
"free"
:
}
},
"nodes"
:[
0
,
1
]
}
},
"entries"
:{
:{
:[
{
:[
0
,
1
],
"type"
:
"memory"
,
"size"
:
}
]
}
},
:
}
It can be deduced from the state file that the pod was pinned to both NUMA nodes, i.e.:
:
[
0
,
1
]
,
cgroups
0
and
1
this
and
this
sections.
"memory"
0
(
"free":0
1
(
).
bytes.
The lin

e
3Gi
0
,
by using
flag.
,
message.
2.42 - Verify Signed Kubernetes Artifacts
cosign
(
)
curl
URL
=
BINARY
=
kubectl
FILES
=(
"
$BINARY
"
"
$BINARY
.sig"
"
$BINARY
.cert"
)
for
FILE in
"
${
FILES
[@]
}
"
;
do
3
3
"
$URL
/
$FILE
"
-o
"
$FILE
"
done
cosign
:
"
$BINARY
"
"
$BINARY
"
"
$BINARY
"
.cert
Note:
Keyless
.
to
Releases
.
the
command:
=
1
Note:
in
KEYLESS
.
$(
)
| awk
input
=
while
IFS
=
read
-r image
do
=
1
"
$image
"
done
&lt;
"
$input
"
example:
.
to
section.
e.g.
)
3 - Configure Pods and Containers
3.1 - Assign Memory Resources to Containers and Pods
request
limit
to a
be configured to communicate with your clus

ter. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have a
minikube
.
enable
API (
.
NAME
field
.
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
requests
:
memory
:
"100Mi"
limits
:
memory
:
"200Mi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"150M"
,
,
"1"
]
The
args
The
=
=
=
=
...
resources
:
requests
:
memory
:
100Mi
limits
:
memory
:
200Mi
...
Run
=
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
requests
:
memory
:
"50Mi"
limits
:
me

mory
:
"100Mi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"250M"
,
,
"1"
]
In the
args
=
=
1
24s
=
=
lastState
:
:
:
exitCode
:
137
:
reason
:
OOMKilled
startedAt
:
null
=
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
requests
:
memory
:
"1000Gi"
limits
:
memory
:
"1000Gi"
command
:
[
"stress"
]
args
:
[
"--vm"
,
"1"
,
,
"150M"
,
,
"1"
]
=
=

d memory-demo-3 --namespace=mem-example
=
Events:
=
Clean up
3.2 - Assign CPU Resources to Containers and Pods
request
and a CPU
limit
to
minikube
.
Minikube
, run the
enable
A

PI,
.
NAME
Create a
Namespace
field
.
:
v1
kind
:
Pod
metadata
:
name
:
cpu-demo
namespace
:
spec
:
:
-
name
:
image
:
resources
:
limits
:
cpu
:
"1"
requests
:
cpu
:
"0.5"
args
:
- -cpus
-
"2"
The
args
The
-cpus "2"
=
=
=
=
resources
:
limits
:
cpu
:
"1"
requests
:
cpu
:
500m
Use
=
-cpu "2"
Note:
CPU units
CPU
1 AWS vCPU
1 GCP Core
=
the capacity of any Node

 in your cluster. Here is the configuration file for a Pod
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
resources
:
limits
:
cpu
:
"100"
requests
:
cpu
:
"100"
args
:
- -cpus
-
"2"
=
=
=
Events:
=
the limit.
Clean up
3.3 - Configure GMSA for Windows Pods and containers
In Kubernetes, GMSA credential specs are configured at a Kubernetes cluster-wid

e scope as Custom Resources. Windows Pods, as well as individual containers within a Pod, can be configured to use a GMSA for domain based functions (e.g. Kerberos authentication) when interacting with other Windows services.
kubectl
A
YAML
A
script
The
.
module
:
Use
,
kind
,
metadata
and
credspec
:
:
kind
:
metadata
:
name
:
credspec
:
:
:
-
Name
:
WebApp1
Scope
:
CONTOSO
-
Name
:
WebApp1

Scope
:
:
:
DnsName
:
:
Guid
:
#GUID
:
WebApp1
:
CONTOSO
Sid
:
use
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
[
]
resources
:
[
]
verbs
:
[
"use"
]
:
[
]
to use
:
kind
:
metadata
:
name
:
namespace
:
default
subjects
:
-
kind
:
name
:
default
namespace
:
default
roleRef
:
kind
:
name
:
apiGroup
:
:
:
apps/v1
kind
:
metadata
:
labels
:
run
:
name
:
namespace
:
default
spec
:
replicas
:
1
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
:
:
:
-
image
:
:
Always
name
:
iis
:
:
windows
:
apps/v1
kind
:
metadata
:
labels
:
run
:
name
:
namespace
:
default
spec
:
replicas
:
1
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
-
image
:
:
Always
name
:
iis
:
:
:
:
:
windows
use
The container runtime configures each Windows container with the specified GMSA credential spec so that the container c

an assume the identity of the GMSA in Active Directory and access services in the domain using that identity.
reg add
here
exec
command.
iis-auth
exec
command.
:
image
:
lifecycle
:
postStart
:
exec
:
command
:
[
,
,
]
:
lifecycle
netlogon
3.4 - Configure RunAsUserName for Windows pods and containers
runAsUser
field (
(
field.
The Windows security

 context options that you specify for a Pod apply to all Containers and init Containers in the Pod.
field set:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
:
:
:
-
name
:
image
:
command
:
[
"ping"
,
"-t"
,
]
:
:
windows
kubectl
exec
echo
field (
(
field.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
:
:
:
-
name
:
image
:
command
:
[
"ping"
,
"-t"
,
]
:
:
:
:
:
windows
kubectl
exec
echo
, where
DOMAIN\
DOMAIN
and
USER
:
The
0x00-0x1F
,
0x7F
)
The
DOMAIN
.
.
(dot) or
-
(dash).
The
USER
only
.
field:
,
,
,
.
here
and
here
.
3.5 - Create a Windows HostProcess Pod
workloads on a Windows host. These containers operat

e as
and
See
.
not
Control
Policy
securityContext.win

dowsOptions.hostProcess
true
true
false
spec
:
:
:
:
true
:
:
true
:
-
name
:
test
image
:
command
:
- ping
- -t
-
127.0.0.1
:
windows
upon the
specified
mountPath
ephemeral
local use

r account to be created and joined to the specified usergroup, from which the container is run.
Note:
Example:
icacls
.
Set
:
:
:
true
:
.
.
or
.
3.6 - Configure Quality of Service for Pods
minikube
.
Burstable
:
v1
kind
:
Pod
metadata
:
name
:
qos-demo
namespace
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"700m"
requests
:
memory
:
"200Mi"
cpu
:
"700m"
=
=
=
yaml
spec
:
:
...
resources
:
limits
:
cpu
:
700m
memory

:
200Mi
requests
:
cpu
:
700m
memory
:
200Mi
...
status
:
qosClass
:
Note:
the limit.
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
=
=
=
yaml
spec
:
:
-
image
:
nginx
:
Always
name
:
resources
:
limits
:
memory
:
200Mi
requests
:
memory
:
100Mi
...
status
:
qosClass
:
Burstable
=
Here is the configuration file for a Pod that has one Container. The Container has no memory or CPU
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
nginx
=
=
=
yaml
spec
:
:
...
resources
:
{}
...
status
:
qosClass
:
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
memory
:
"200Mi"
-
name
:
image
:
redis
=
=
=
yaml
spec
:
:
...
name
:
resources
:
requests
:
memory
:
200Mi
...
name
:
resources
:
{}
...
status
:
qosClass
:
Burstable
=
Clean up
Configure Default CPU Request

s and Limits for a Namespace
3.7 - Assign Extended Resources to a Container
minikube
.
.
where
foo
is a
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
:
3
limits
:
:
3
In the configuration file, you can see that the Container requests 3 dongles.
Limits
:
:
3
Requests
:
:
3
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
:
2
limits
:
:
2
...
Events:
  ...
Clean up
3.8 - Configure a Pod to Use a Volume for Storage

Volume
minikube
.
emptyDir
:
v1
kind
:
Pod
metadata
:
name
:
redis
spec
:
:
-
name
:
redis
image
:
redis
:
-
name
:
mountPath
:
volumes
:
-
name
:
emptyDir
:
{}
the Pod:
kubectl
exec
cd
echo
kill
where
of
Always
.
kubectl
exec
test-file
cd
test-file
See
Volume
.
See
Pod
.
emptyDir
Volumes
3.9 - Configure a Pod to Use a PersistentVolume for Storage
PersistentVolu

me.
kubectl
Minikube
.
.
.
/mnt/data
In the
/mnt/data
file:
sudo sh -c
Note:
sudo
, you can
sudo
hostPath
an NFS share, or an Amazon Elastic Block Store volume. Cluster administrators can also
use
to set up
.
:
v1
kind
:
metadata
:
name
:
labels
:
type
:
local
spec
:
:
manual
capacity
:
storage
:
10Gi
:
hostPath
:
path
:
/mnt/data
on the
manual
STATUS
of
Available
. This
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
3Gi
plane finds a suitable Persistent

Volume with the same StorageClass, it binds the
STATUS
of
Bound
.
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
volumes
:
-
name
:
:
claimName
:
:
-
name
:
image
:
nginx
ports
:
-
:
80
name
:
:
-
mountPath
:
name
:
kubectl
exec
# Be sure to run these 3 commands inside the root shell that comes from
apt update
file on the
Clean up
:
v1
kind
:
Pod
metadata
:
name
:
test
spec
:
:
-
name
:
test
image
:
nginx
:
-
name
:
config
mountPath
:
subPath
:
html
-
name
:
config
mountPath
:
subPath
:
volumes
:
-
name
:
config
:
claimName
:
Use the
:
v1
kind
:
metadata
:
name
:
pv1
:
:
"1234"
Pod's 

security context are. Every GID, whether it originates from a PersistentVolume
Note:
.
Read the
.
Reference
3.10 - Configure a Pod to Use a Projected Volume for Storage
projected
secret
,
configMap
,
,
and
Note:
minikube
.
Secrets
projected
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
args
:
- sleep
-
"86400"
:
-
name
:
mountPath
:
readOnly
:
true
volumes
:
-
name
:
projected
:
sources
:
-
secret
:
name
:
user
-
secret
:
name
:
pass
echo
-n
"admin"
echo
-n
=
=
the Pod:
kubectl
exec
Clean up
projected
volumes.
Read the
3.11 - Configure a Security Context for a Pod or Container
.
:
:
AppArmor
:
Seccomp
has

minikube
.
field
field is a
object.
and an
emptyDir
volume:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
runAsUser
:
1000
:
3000
fsGroup
:
2000
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
,
"sleep 1h"
]
:
-
name
:
mountPath
:
:
:
false
runAsUser
Since
fsGroup
kubectl
exec
ps
runAsUser
:
...
/data
cd
/data
ls -l
fsGroup
.
cd
demo
echo
ls -l
testfile
fsGroup
.
id
gid
field.
If the
gid
exit
fsGroup
mounted.
to control th

e way that Kubernetes checks and manages ownership and permissions
-
fsGroup
Always
:
runAsUser
:
1000
:
3000
fsGroup
:
2000
:
Note:
secret
,
configMap
,
and
emptydir
.
, the
fsGroup
provided
fsGroup
fsGroup
.
field
field is a
object.
field:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
runAsUser
:
1000
:
-
name
:
image
:
:
runAsUser
:
2000
:
false
kubectl
exec
ps aux
of
runAsUser
...
exit
With
,
field.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
sec-ctx-3
image
:
kubec

tl get pod security-context-demo-3
kubectl
exec
ps aux
cd
/proc/1
cat status
...
...
exit
adds the
and
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
sec-ctx-4
image
:
:
:
add
:
[
,
]
kubectl
exec
cd
/proc/1
cat status
...
...
.
See
Note:
CAP_XXX
.
omit the
CAP_
, include
SYS_TIME
field
in the
field is a
type
and
.
type
include
,
, and
Localhost
.
. It
flag).
...
:
:
type
:
:
...
:
:
type
:
Localhost
:
field in
the
field is an
...
:
:
level
:
Note:
.
ReadWriteOnceP

od
and
must
.
have
set.
CSI
legacy
iscsi
-o context
by setting
in
Note:
nodes are
fsGroup
and
are
fsGroup
fsGroup
. See the
level
Warning:
Clean up
document
3.12 - Configure Service Accounts for Pods
API server
.
A
user
not
have a User
API.
minikube
default
namespace
.
default
.
If you do not spe

cify a ServiceAccount when you create a Pod, Kubernetes
default
.
See
in use.
kubelet
:
v1
kind
:
metadata
:
name
:
:
false
...
:
v1
kind
:
Pod
metadata
:
name
:
my-pod
spec
:
:
:
false
...
.spec
default
with:
metadata:
EOF
.
:
v1
kind
:
metadata
:
:
name
:
namespace
:
default
:
"272500"
uid
:
.
field
Note:
The
.
on the
.
Cleanup
kubectl
:
Note:
Kubernetes v1.26, API credentials are obtained directly by us

ing the
API,
.
.
metadata:
EOF

Data
====
token:          ...
Note:
token
First,
.
.
=
\
=
=
\
=
sa.yaml
:
v1
kind
:
metadata
:
:
name
:
default
namespace
:
default
:
"243024"
uid
:
uid
:
v1
kind
:
metadata
:
:
name
:
default
namespace
:
default
uid
:
:
-
name
:
kubec

tl run nginx --image
=
=
Never
=
jsonpath
=
Note:
:
If
not
spec
.
vault
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
-
image
:
nginx
name
:
nginx
:
-
mountPath
:
name
:
:
volumes
:
-
name
:
projected
:
sources
:
-
:
path
:
:
7200
audience
:
vault
,

.
Note:
. In
https
.
.
.
.
Note:
and
RBAC
include a
.
group,
or
jwks_uri
jwks_uri
https
scheme.
See also:
Read the
Read about
Read about
Secrets
or learn to
.
Read about
.
Read the
3.13 - Pull an Image from a Private Registry
Secret
Docker Hub
minikube
docker
Docker ID
On your laptop, you must authenticate with a registry in order to pull a private imag

e.
Use the
docker
log in
section of
.
View the
file:
{
"auths"
: {
: {
"auth"
:
}
}
}
Note:
auth
entry but a
.
\
=
=
\
--type
=
Be sure to:
set
type
to
Example:
:
v1
kind
:
Secret
metadata
:
name
:
namespace
:
data
:
:
type
:
, it means
file.
regcred
:
=
=
=
=
where:
Use
regcred
.
Note:
kubectl
regcred
regcred
=
yaml
:
v1
kind
:
Secret
metadata
:
...
name
:
regcred
...
data
:
:
type
:
=
{
"auths"
:{

:{
:
"janedoe"
,
:
,
"email"
:
,
"auth"
:
}}}
auth
echo
:
file.
regcred
regcred
:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
:
-
name
:
regcred
In file
, replace
The
regcred
.
Secrets
Secret
.
.
See
.
See the
of a Pod
3.14 - Configure Liveness, Readiness and Startup Probes
The
kubelet
minikube
:
v1
kind
:
Pod
metadata
:
labels
:
test
:
liveness
name
:
spec
:
:
-
name
:
liveness
image
:
args
:
- /bin/sh
- -c
- touch /t

mp/healthy; sleep 30; rm -f /tmp/healthy; sleep 600
:
exec
:
command
:
- cat
:
5
:
5
Container
.
The
/bin/sh -c
file.
  Normal  Started    7s    kubelet, node01    Started container liveness
RESTARTS
RESTARTS
image.
:
v1
kind
:
Pod
metadata
:
labels
:
test
:
liveness
name
:
spec
:
:
-
name
:
liveness
image
:
args
:
- /server
:
httpGet
:
path
:
/healthz
port
:
8080
:
-
name
:
value
:
Awesome
:
3
:
3
The
field specifies that the kub

elet should perform a liveness
/healthz
path
server.go
.
/healthz
handler
http.
(
,
func
*
duration
:=
time.
Now
().
Sub
(started)
if
duration.
Seconds
() &gt;
10
{
w.
(
500
)
w.
Write
([]
byte
(fmt.
Sprintf
(
Seconds
())))
}
else
{
w.
(
200
)
w.
Write
([]
byte
(
"ok"
))
}
})
(or
the HTTP liveness probe uses that proxy.
:
v1
kind
:
Pod
metadata
:
name
:
goproxy
labels
:
app
:
goproxy
spec
:
:
-
name
:
goproxy
image
:
ports
:
-
:
8080
:
tcpSocket
:
port
:
8080
:
5
:
10
:
tcpSocket
:
port
:
8080
:
15
:
20
goproxy
seconds.
goproxy
,
:
v1
kind
:
Pod
metadata
:
name
:
etcd-

with-grpc
spec
:
:
-
name
:
etcd
image
:
command
:
[
,
,
,
,
,
,
,
,
"debug"
]
ports
:
-
:
2379
:
grpc
:
port
:
2379
:
10
port
service
.
Note:
,
.
127.0.0.1
.
-tls
).
If
false
not
port
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
:
1
:
10
:
httpGet
:
path
:
/healthz
port
:
:
30
:
10
.
Services.
Note:
Readiness probes runs on the container during its

 whole lifecycle.
Caution:
do not
field.
:
exec
:
command
:
- cat
:
5
:
5
Probes
checks:
try
Note:
was not respected for exec probes:
(set it to
false
)
value set.
1.20+
with the
Caution:
httpGet
:
host
scheme
path
port
host
field in
httpGet
. If
scheme
HTTPS
host
field.
host
, under
httpGet
, should be 

set
host
Host
header in
.
Host
header:
, and
Accept
(where
1.26
*/*
:
httpGet
:
:
-
name
:
Accept
value
:
:
httpGet
:
:
-
name
:
value
:
:
httpGet
:
:
-
name
:
Accept
value
:
""
:
httpGet
:
:
-
name
:
value
:
""
TCP probes
host
was used
was set.
Note:
by default. For users choosing to disable this feature, please note the following:
The
field if
spec
:
:
3600
:
-
name
:
test
image
:
...
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
:
1
:
60
:
60
.
Pod
probe(s)
3.15 - Assign Pods to Nodes
minikube
.
List the
nodes
worker0   Ready     &lt;none&gt;   1d      v1.13.0        ...

,kubernetes.io/hostname
=
worker0
=
worker1
=
worker2
disktype
=
ssd
where
label:
=
=
worker0
=
worker1
=
worker2
worker0
node has a
label.
a
label.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
labels
:
env
:
test
spec
:
:
-
name
:
nginx
image
:
nginx
:
:
disktype
:
ssd
=
wide
0
nodeName
.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
nodeName
:
foo-node
:
-
name
:
nginx
image
:
nginx
:
foo-node
only.
.
nodes
.
3.16 - Assign Pods to Nodes using Node Affinity
minikube
.
=
worker0
=
worker1
=
worker2
disktype
=
ssd
where
label:
worker2   

Ready     &lt;none&gt;   1d      v1.13.0        ...,kubernetes.io/hostname=worker2
worker0
node has a
label.
.
label.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
affinity
:
:
:
:
-
:
-
key
:
disktype
operator
:
In
values
:
- ssd
:
-
name
:
nginx
image
:
nginx
:
=
wide
.
label.
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
affinity
:
:
:
-
weight
:
1
:
:
-
key
:
disktype
operator
:
In
values
:
- ssd
:
-
name
:
nginx
image
:
nginx
:
=
wide
.
3.17 - Configure Pod Initialization
minikube
.
:
v1
kind
:
Pod
metadata
:
name
:
init-demo
spec
:
:
-
name
:
nginx
image
:
nginx
ports
:
-
:
80
:
-
name
:
workdir
mountPath
:
:
-
name
:
install
image
:
command
:
- wget
-
"-O"
-
:
-
name
:
workdir
mountPath
:
dnsPolicy
:
Default
volumes
:
-
name
:
workdir
emptyDir
:
{}
/work-dir
Volume at

  ...
  ...
.
.
Volumes
.
3.18 - Attach Handlers to Container Lifecycle Events
minikube
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
lifecycle
:
postStart
:
exec
:
command
:
[
"/bin/sh"
,
"-c"
,
]
preStop
:
exec
:
command
:
[
"/bin/sh"
,
"-c"
,
]
message
postStart
message
file:
Kubernetes' management of the Container blocks until the preStop

 handler completes,
.
Note:
.
completed
.
.
.
Reference
Lifecycle
Container
See
in
PodSpec
3.19 - Configure a Pod to Use a ConfigMap
minikube
.
kubectl
Use the
,
files
, or
:
.
You can use
or
You can use
=
and
in the

Data
====
----
lives=3
secret.code.l

ives=30
----
The
and
data
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"516"
uid
:
data
:
:
|
lives=3
:
|
You can use
=

Data
====
----
lives=3
=
=

Data
====
----
lives=3
----
When
kubectl
data
.
enemies
=
aliens
lives
=
3
allowed
=
"true"
\
=
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
gam

e-config-env-file
namespace
:
default
:
"809965"
uid
:
data
:
allowed
:
'"true"'
enemies
:
aliens
lives
:
"3"
kubectl
\
=
\
=
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"810136"
uid
:
data
:
allowed
:
'"true"'
color
:
purple
enemies
:
aliens
how
:
lives
:
"3"
textmode
:
"true"
data
argument:
=
=
where
=
=
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"530"
uid
:
data
:
:
|
lives=3
You can use
with the
=
=
=
=
charm
data
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
:
"651"
uid
:
data
:
:
very
:
charm
kubectl
supports
cat
files:
EOF
1
37s
{
:
"v1"
,
"data"
:
{
:
"enemies=aliens\nlives=3\nenemies.cheat=true\nenemies.cheat.level=noGood

Rotten\nsecret.code.p...
Data
====
----
lives=3
cat
files:
EOF
and
,
as
cat
literals:
EOF
=
=
very
Assign the
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
:
Never
.
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
---
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
log_level
:
INFO
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
LOG_LEVEL
valueFrom
:
:
name
:
key
:
log_level
:
Never
and
.
Note:
c

onfigmap/configmap-multikeys.yaml
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
:
charm
Use
envFrom
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
envFrom
:
-
:
name
:
:
Never
and
.
command
and
args
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
,
]
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
valueFrom
:
:
name
:
key
:
:
Never
very charm
data
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
default
data
:
:
very
:
charm
volumes
).
The
command
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
:
Never
Caution:
Note:
Use the
path
volume at
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:

command
:
[
"/bin/sh"
,
"-c"
,
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
items
:
-
key
:
path
:
keys
:
Never
very
Caution:
basis. The
Secrets
Secrets
Note:
/etc
data
.
:
v1
kind
:
ConfigMap
metadata
:
:
name
:
namespace
:
default
data
:
:
hello
:
world
:
|-
ConfigMap
optional
optional
optional
.
If you use
envFrom
Namespace
optional
:
v1
kind
:
Pod
metadata
:
name
:
d

api-test-pod
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
"env"
]
env
:
-
name
:
valueFrom
:
:
name
:
a-config
key
:
akey
optional
:
true
:
Never
a-config
a-config
a key named
akey
akey
in the
a-config
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"/bin/sh"
,
"-c"
,
]
:
-
name
:
mountPath
:
volumes
:
-
name
:
configMap
:
name
:
no
-config
optional
:
true
:
Never
Note:
subPath
.
3.20 - Share Process Namespace between Containers in a Pod
minikube
field of
.spec
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
true
:
-
name
:
nginx
image
:
nginx
-
name
:
shell
image
:
:
:
add
:
stdin
:
true
tty
:
true
nginx
shell
ps
:
ps ax
You can signal processes 

in other containers. For example, send
SIGHUP
to
nginx
kill
-HUP
8
ps ax
link.

events {
systemd
) or run
kill -HUP 1
kill -HUP 1
(
/pause
/proc
link.
This makes debugging easier, but it also means
3.21 - Use a User Namespace With a Pod
host.
HIGH
or
CRITICAL
minikube
.
if your contai

ner runtime doesn't support user namespaces, the
new
pod.spec
hostUsers
field of
.spec
to
false
:
v1
kind
:
Pod
metadata
:
name
:
userns
spec
:
hostUsers
:
false
:
-
name
:
shell
command
:
[
"sleep"
,
]
image
:
debian
:
replacing
$pid
3.22 - Create static Pods
API server
);
Kubelet
mirror Pod
Note:
DaemonSet
instead.
Note:
The
spec
(e.g.,
,
ConfigMap
,
Secret
, etc).
Note:
.
minikube
.
CRI-O
or a
.
,
my-node1
.
:
cat

kind: Pod
metadata:
labels:
spec:
- name: web
ports:
- name: web
EOF
or add the
.
argument
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
role
:
myrole
spec
:
:
-
name
:
web
image
:
nginx
ports
:
-
name
:
web
:
80
protocol
:
TCP
crictl ps
Note:
crictl
NAME
.
Note:
Labels
selectors
, etc.
kubectl
doesn't
sleep
20
crictl ps
The running kubelet periodically scans the configured dire

ctory (
#
sleep
20
crictl ps
sleep
20
crictl ps
3.23 - Translate a Docker Compose File to Kubernetes Resources
.
minikube
.
.
# Linux
# macOS
# Windows
tarball
.
go get
EPEL
EPEL
.
If you have
EPEL
Homebrew
:
file.
version
:
"2"
services
:
:
image
:
ports
:
-
"6379"
:
image
:
ports
:
-
"6379"
:
frontend
:
image
:
ports
:
-
"80:80"
:
labels
:
:
kubectl
, run
and then
.
   INFO Kube

rnetes file "redis-slave-service.yaml" created
minikube
No events.
.
User Guide
CLI
Labels
Restart
example
ls
ls
mlbparks-deployment.yaml  m

ongodb-service.yaml                       redis-slave-service.jsonmlbparks-service.yaml  
OpenShift
example
and
Note:
.
kompose
and
Services
-j
objects,
, or
Helm
charts.
created
created
created
created
The
created
created
created
created
The
flag:
.
created
created
created
created
The
Helm
run:
INFO Kubernetes file "redis-deployment

.yaml" created
Labels
kompose
version
:
"2"
services
:
nginx
:
image
:
nginx
:
foobar
build
:
./foobar
cap_add
:
- ALL
:
foobar
labels
:
:
nodeport
version
:
"2"
services
:
web
:
image
:
ports
:
-
links
:
- redis
labels
:
:
redis
:
image
:
redis:3.0
ports
:
-
"6379"
Key
Value
Note:
The
ports
kompose
will fail.
Restart
restart
restart
value.
restart
Pod
""
Always
always
Always
Pod
OnFailure
no
Pod
Never
Note:
or
.
pival
pi
.
version
:
'2'
services
:
pival
:
image
:
perl
command
:
[
"perl"
,
,
"-wle"
,
]
restart
:
_
-
_
files.
3.24 - Enforce Pod Security Standards by Configuring the Built-in Admission Controller
.
.
.
Ensure the
:
kind
:
plugins
:
-
name
:
:
:
kind
:
PodSecurityConfigu

ration
#
#
defaults
:
enforce
:
:
"latest"
audit
:
:
"latest"
warn
:
:
"latest"
:
usernames
:
[]
:
[]
:
[]
Note:
v1beta1
.
v1alpha1
.
3.25 - Enforce Pod Security Standards with Namespace Labels
,
baseline
and
.
.
Ensure the
baseline
that:
Blocks
baseline
meet the
baseline
and
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
:
baseline
:
v1.26
:
:
v1.26
:
:
v1.26
Note:
When an
enforce
--dry-run
dry run
=
\
=
baseline
baseline
:
\
=
baseline
\
=
baseline
=
policy to
\
=
\
=
v1.26
3.26 - Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller
audit
and
warn
Your Kub

ernetes server must be at or later than version v1.22.
.
concepts.
or use a
.
.
labels on
.
0
policies
If a PodSecurityPo

licy is mutating pods, then you could end up with pods that don't meet the Pod
Caution:
,
Caution:
MustRunAs
ALL
MustRunAs
RunAsAny
for
false
profile.
PSP_NAME
=
jsonpath
=
$PSP_NAME
.spec.s

ecurityContext.fsGroup
and
:
use
on all PSPs
use
them.
and
- Using the
-o
jsonpath
=
| tr
" "
"\n"
| sort -u
,
Caution:
existing
=
=
$LEVEL
existing
level.
mode works similarly, but returns the warning to the user immediat

ely. You can set the audit level
=
$LEVEL
=
$LEVEL
.
contain
.
4 - Monitoring, Logging, and D

ebugging
- Useful
- Useful
release
Questions
Concepts
Setup
provides
Tasks
show how to
Tutorials
are more
Reference
kubectl
.
.
please
,
before
!
Slack
channel.
,
channel.
Country
Channels
China
#cn-users
,
Finland
#fi-users
France
#fr-users
,
Germany
#de-users
,
India
#in-users
,
Italy
#it-users
,
Japan
#jp-users
,
Korea
#kr-users
#nl-users
Norway
Poland
#pl-users
Russia
#ru-users
Spain
#es-users
Sweden
#se-users
Turkey
#tr-users
,
Forum
.
.
4.1 - Troubleshooting Applications
4.1.1 - Debug Pods
This is
not
this guid

e
.
${
POD_NAME
}
Running
Pending
hostPort
hostPort
hostPort
hostPort
Waiting
Waiting
.
command
as
commnd
option.
.
command
as
commnd
46757
]
46757
]
false
and then
to debug 

your pods.
endpoints
${
}
...
spec
:
-
selector
:
name
:
nginx
type
:
frontend
=
name
=
=
frontend
Please see
Service
Endpoints
, and your
Pods
are
4.1.2 - Debug Services
Note:
kubectl
exec
Setup
=
kubectl
=
3
YAML:
:
apps/v1
kind
:
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
:
app
:
hostnames
replicas
:
3
template
:
metadata
:
labels
:
app
:
hostnames
spec
:
:
-
name
:
hostnames
image
:
app
=
hostnames
You can also confirm that your Pods are se

rving. You can get the list of
app
=
hostnames
\
=
for
do
wget -qO-
$ep
done
there.
=
80
=
9376
:
v1
kind
:
Service
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
app
:
hostnames
ports
:
-
name
:
default
protocol
:
TCP
port
:
80
:
9376
traffic to
name.

The "cluster.loca

l" is your cluster domain, which COULD be different in your
Note:

The
kubelet
with the
flag.
The
search
kubelet
with the
commands.
The
options
ndots

section
above).
for
i in
$(
seq
1
3
)
;
do
done
{
"kind"
:
"Service"
,
:
"v1"
,
: {
"name"
:
,
:
"default"
,
"uid"
:
,
:
"347189"
,
:
,
"labels"
: {
"app"
:
}
},
"spec"
: {
"ports"
: [
{
"name"
:
"default"
,
:
"TCP"
,
"port"
:
80
,
:
9376
,
:
0
}
],
: {
"app"
:
},
:
,
"type"
:
,
:
"None"
},
"status"
: {
: {}
}
}
?
Is the
protocol
Does th

e Service have any Endpoints?
app
=
hostnames
The
debug pods
.
ENDPOINTS
column is
the
, but the
the
above.
Note:
for
do
wget -qO-
$ep
done
I10

27 22:14:54.038140    5063 proxier.go:352] Setting endpoints for "kube-system/kube-dns:dns-tcp" to [10.244.1.3:53]
required
conntrack
conntrack
on Ubuntu)
line
and
one
and one
chain with
IPVS mode
...
...
endpoint, it will create corresponding real servers. In this example, ser

vice
,
,
).
with the
-v
iptables
Kubelet
exposes a
flag
or
.
Confirm
is set to
or
.
is set to
match
hairpin
Kubelet
has
/sys
for
do
cat
$intf
done
1
1
1
1
, ensure
Kubelet
cbr0
bridge is
Seek help
Slack
or
Forum
or
GitHub
.
Visit the
4.1.3 - Debug a StatefulSet
=
MyApp
If you find that any Pods lis

ted are in
Unknown
or
task for
guide.
.
4.1.4 - Determine the Reason for Pod Failure
.
minikube
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
debian
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
]
command
and
args
the
=
yaml
:
v1
kind
:
Pod
...
lastState
:
:
:
...
exitCode
:
0
:
...
message
:
|
...
=
=
value of
or
.
:
v1

kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
debian
:
File
to "
See the
field in
Container
.
.
.
4.1.5 - Debug Init Containers
and
.
minikube
.
.
.
Init:1/2
See
    ...
    ...
    ...
    ...
set -x
Init:
Status
Meaning
Init:N/M
The Pod has
M
N
Pending
or
Running
4.1.6 - Debu

Your
Pod
Debugging
Pods
.
kubectl
.
Using
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"128Mi"
cpu
:
"500m"
ports
:
-
:
80
IPs:
  nginx:
    Limits:
      cpu:     500m
    Requests:
      cpu:        500m
    Mounts:
  Ready             True 
Volumes:
Events:
Here you can see configuration information about the container(s) and Pod (labels, resource requirements, etc.), as well as status information a

bout the container(s) and Pod (state, readiness, restart count, events, etc.).
  Node:		/
  IP:
    nginx:
      QoS Tier:
      Limits:
        cpu:	1
      Requests:
        cpu:	1
  Volumes:
  Events:
(and

 possibly others). The message tells us that there were not enough resources for the Pod on any of the nodes.
=
argument.
-o yaml
:
v1
kind
:
Pod
metadata
:
:
:
labels
:
app
:
nginx
:
name
:
namespace
:
default
:
-
:
apps/v1
:
true
:
true
kind
:
name
:
uid
:
:
"1364"
uid
:
spec
:
:
-
image
:
nginx
:
Always
name
:
nginx
ports
:
-
:
80
protocol
:
TCP
resources
:
limits
:
cpu
:
500m
memory
:
128Mi
requests
:
cpu
:
500m
memory
:
128Mi
:
:
File
:
-
mountPath
:
name
:
readOnly
:
true
dnsPolicy
:
:
true
nodeName
:
:
priority
:
0
:
Always
:
:
{}
:
default
:
default
:
30
:
-
effect
:
NoExecute
key
:
operator
:
Exists
:
300
-
effect
:
NoExecute
key
:
operator
:
Exists
:
300
volumes
:
-
name
:
projected
:
:
420
sources
:
-
:
:
3607
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
status
:
:
-
:
null
:
status
:
"True"
type
:
-
:
null
:
status
:
"True"
type
:
Ready
-
:
null
:
status
:
"True"
type
:
-
:
null
:
status
:
"True"
type
:
:
-
:
image
:
imageID
:
lastState
:
{}
name
:
nginx
ready
:
true
:
0
started
:
true
state
:
running
:
startedAt
:
hostI

P
:
phase
:
Running
podIP
:
10.88.0.3
podIPs
:
-
ip
:
10.88.0.3
-
ip
:
2001
:db8::1
qosClass
:
startTime
:
${
POD_NAME
}
${
}
${
POD_NAME
}
${
}
If the
includes
:
kubectl
exec
${
POD_NAME
}
-c
${
}
--
${
CMD
}
${
ARG1
}
${
ARG2
}
...
${
ARGN
}
Note:
kubectl
exec
-i
and
-t
kubectl
exec
.
.
=
=
Never
pause
images.
kubectl
exec
. If you
-i
/
argument,
kubectl
=
=
/ #
--target
creates.
Note:
The
--target
ps
:
...
...
Use
situations. For e

xample, you can't run
to create a
busybox
busybox
:
=
=
myapp
named
that adds a
=
=
Note:
flag.
The
-i
.
The
works, see
.
  myapp:
    ...
    Args:
      false
You can use
/ #
Note:
or
The
-i
.
:
Now use
to
ubuntu
:
syntax as
.
*=ubuntu
to
ubuntu
.
If none of these appr

oaches work, you can find the Node on which the Pod is
, run:
=
ubuntu
/host
.
and
will fail.
4.1.7 - Get a Shell to a Running Container
minikube
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
nginx
image
:
nginx
:
-
name
:
mountPath
:
:
true
dnsPolicy
:
Default
kubectl
exec
Note:
--
ls /
ls /
tcpdump
lsof
ps aux
has an
emptyDir
at
.
file in the
echo
file:
exit
.
exit
kubectl
exec
kubectl
exec
kubectl
exec
kubectl
exec
or
-c
to
named
main-app
and
. The following com

mand would open a
main-app
kubectl
exec
Note:
-i
and
-t
--stdin
and
--tty
Read about
4.2 - Troubleshooting Clusters
the
Ready
state.
and
Lease:
  Hostna

me:    kube-worker-1
Capacity:
  cpu:                2
  pods:               110
  cpu:                2
  pods:               110
Events:
...
:
v1
kind
:
Node
metadata
:
:
:
:
"0"
:
"true"
:
labels
:
:
amd64
:
linux
:
amd64
:
:
linux
name
:
:
"4026"
uid
:
spec
:
{}
status
:
addresses
:
-
address
:
type
:
-
address
:
type
:
Hostname
:
cpu
:
"2"
:
:
"0"
memory
:
1922788Ki
pods
:
"110"
capacity
:
cpu
:
"2"
:
:
"0"
memory
:
2025188Ki
pods
:
"110"
:
-
:
:
message
:
reason
:
WeaveIsUp
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"True"
type
:
Ready
:
:
Port
:
10250
nodeInfo
:
:
amd64
bootID
:
22333234
-7a6b-44d4-

9ce1-67e31dc7e369
:
:
5.13.0-28
-generic
:
v1.23.3
:
v1.23.3
machineID
:
:
linux
osImage
:
:
- logs from
Results
Results
Results
Results
Results
Results
etc.
Mitigates: Node shutdo

wn
Use
to
Use
crictl
to
Use
to
4.2.1 - Resource metrics pipeline
The
(HPA) and
(VPA)
command.
Note:
.
S[
]
Server]
end
L[API
server]
W[HPA]
end
top]
enabled
cAdvisor
kubelet
and
/stats
kubelet
/stats
endpoint.
the
Note:
minikube
jq
for easier
reading:
"/apis/me

trics.k8s.io/v1beta1/nodes/minikube"
| jq
'.'
curl
:
{
"kind"
:
,
:
,
: {
"name"
:
,
:
,
:
},
:
,
"window"
:
"30s"
,
"usage"
: {
"cpu"
:
,
"memory"
:
}
}
jq
| jq
'.'
curl
:
{
"kind"
:
,
:
,
: {
"name"
:
,
:
,
:
,
:
},
:
,
"window"
:
"30s"
,
: [
{
"name"
:
,
"usage"
: {
"cpu"
:
,
"memory"
:
"22244Ki"
}
}
]
}
for the
API.
,
the
and the
.
Note:
CPU
.
Memory
.
using the
command.
kubelet
API
/stat

s/summary
.
.
4.2.2 - Tools for Monitoring Resources
pods
,
services
, and
or
utility.
and
API.
kubelet
for CPU and
cAdvisor
).
or
API.
Logging
exec
4.2.3 - Monitor Node Health
DaemonSet
Condition
s
or as
Event
s.
.
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting 

as control plane hosts. If you do not already have a
minikube
.
Addon
.
kubectl
kubectl
:
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
Note:
kubectl
:
Create
directory
The
ConfigMap
config/
Create the
ConfigMap
:
=
config/
Change the
to use the
ConfigMap
:
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
-
name
:
config
mountPath
:
/config
readOnly
:
true
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
-
name
:
config
configMap
:
name
:

 https://k8s.io/examples/debug/node-problem-detector-configmap.yaml
Note:
kubectl
.
ConfigMap
.
A
such as
filelog
,
kmsg
,
kernel
,
abrt
,
and
systemd
.
A
.
A
.
A
.
.
Exporter
.
.
4.2.4 - Debugging Kubernetes nodes with crictl
crictl
crictl
cri-tools
crictl
crictl
.
The
crictl
or
crictl
Set the
and
flags.
Set the
and
Set the endpoint in the configur

ation file
crictl
.
Note:
crictl
timeout
or
debug
--timeout
and
--debug
crictl
crictl
.
crictl
Warning:
If you use
crictl
crictl
is not a
List pods
run
=
nginx
1f73f2d81bf98       busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47                                   7 minutes ago       Running             sh         

                1
crictl ps
crictl
exec
N
=
1
Using
crictl
{
: {
"name"
:
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
},
:
"/tmp"
,
"linux"
: {
}
}
Use the
Using
crictl
Pod config
:
{
: {
"name"
:
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
},
:
"/tmp"
,
"linux"
: {
}
}
:
{
: {
"name"
:
"busybox"
},
"image"
:{
"image"
:
"busybox"
},
"command"
: [
"top"
],
:
,
"linux"
: {
}
}
returned.
Created
.
CONTAINER ID        IMAGE               CREATED             STATE    

           NAME                ATTEMPT
:
Running
.
crictl ps
crictl
.
Map
docker
crictl
.
4.2.5 - Auditing
auditing
stage
chain.
Panic
Note:
Event
API group
.
None
Metadata
Request
using the
rules
field
must
:
kind
:
Policy
:
-
rules
:
-
level
:
resources
:
-
group
:
""
resources
:
[
"pods"
]
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
,
"pods/st

atus"
]
-
level
:
None
resources
:
-
group
:
""
resources
:
[
]
:
[
]
-
level
:
None
users
:
[
]
verbs
:
[
"watch"
]
resources
:
-
group
:
""
resources
:
[
,
]
-
level
:
None
:
[
]
:
-
"/api*"
-
-
level
:
Request
resources
:
-
group
:
""
resources
:
[
]
:
[
]
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
"secrets"
,
]
-
level
:
Request
resources
:
-
group
:
""
-
group
:
-
level
:
Metadata
:
-
Metadata
level:
:
kind
:
Policy
rules
:
-
level
:
Metadata
Policy
API group
.
Note:
request to
:
[
{
"op"
:
"replace"
,
"path"
:
,
"value"
:
0
},
{
"op"
:
"remove"
,
"path"
:
}
]
JSONlines
format.
flags:
-
hostPath
=
\
=
...
:
-
mountPath
:
name
:
audit
readOnly
:
true
-
mountPath
:
name
:
audit-log
readOnly
:
false
and finally c

onfigure the
hostPath
:
...
volumes
:
-
name
:
audit
hostPath
:
path
:
type
:
File
-
name
:
audit-log
hostPath
:
path
:
type
:
.
webhook
with
log
in the flag
webhook
log
webhook
log
.
batch
blocking
batch
mode:
on
and
webhook
and
log
or
au

dit-webhook-truncate-enabled
.
Event
and the
Policy
4.2.6 - Debugging Kubernetes Nodes With Kubectl
node
command.
minikube
.
Use the
=
ubuntu
ip
,
ifconfig
,
nc
,
ping
, and
ps
mtr
,
tcpdump
, and
curl
Note:
/host
in the Pod.
kubelet
Logs from
/host
.
will fail.
Note:
The
Check
4.2.7 - Developing and debugging services locally using telepresence
Note:
This section links to third party projects that provide functionality required by Kubernetes. The Kubern

etes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the
kubectl
, run
 
...
Use the
Where:
And
.
4.2.8 - Windows debugging tips
See
Note:
or
this guide
.
My Windows Pods do not have network connecti

vity
enabled
on all
commands.
cni.conf
(see
) is
from the
in
cni.conf
            ]
NodePort
C:\k\
| ? Name
-ieq
"cbr0"
| ? Name
-Like
start.ps1
section
.
.
wincat
.
,
,
,
,
Flanneld is stuck in "Waiting fo

r the Network to be created"
issue
;
start.ps1
,
)
-file
to restart
=
=
=
1500
=
true
topic
5 - Manage Kubernetes Objects
5.1 - Declarative Management of Kubernetes Objects Using Configuration Files
to
apply
will make.
Install
kubectl
.
minikube
.
The
kubectl
See
Overview
files to
Use
annotation on each object. The annotation contains t

he contents of the object
Note:
Add the
-R
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Run
Note:
diff
uses
,
.
Since
diff
PATCH
,
CREATE
, and
UPDATE
See
:
:
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
Note:
Add the
-R
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
Note:
:
kind
:
metadata
:
:
# ...
:
|
# .

..
spec
:
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
replicas
.
:
=
2
:
replicas
replicas
field:
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
replicas
:
2
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
Update the
to
field:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
The
replicas
.
The
image
from
.
The
The
The
field.
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
replicas
:
2
# ...
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
Warning:
Mixing
create
and
replace
create
and
replace
that
kub

ectl apply
.
Warning:
Warning:
--prune
Warning:
and
You can use
with
-o yaml
Caution:
A
patch
When
The
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
"ports":[{"co

ntainerPort":80}]}]}}}}
# ...
spec
:
replicas
:
2
# ...
:
5
selector
:
:
# ...
app
:
nginx
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
:
.
Action:
Clear
image
Action:
image
Set the
:
apps/v1
kind
:
metadata
:
:
# ...
:
|
# ...
spec
:
selector
:
:
# ...
app
:
nginx
replicas
:
2
# ...
template
:
metadata
:
# ...
labels
:
app
:
nginx
spec
:
:
-
image
:
# ...
name
:
nginx
ports
:
-
:
80
# ...
# ...
# ...
# ...
primitive
image
and
replicas
Action:
Replace.
map
object
labels
,
,
spec
and
metadata
Action:
list
,
ports
, and
args
are lists.
Action:
Varies.
When
spec
spec
is
spec
, such as
replicas
Note:
-
Action
Yes
Yes
-
Yes
No
-
No
-
Yes
No
-
No
Note:
-
Action

Yes
Yes
-
Yes
No
-
No
-
Yes
No
-
No
Example:
Use
args
args
Any
args
args
args
:
[
"a"
,
"b"
]
args
:
[
"a"
,
"c"
]
args
:
[
"a"
,
"b"
,
"d"
]
args
:
[
"a"
,
"c"
]
. The
types.go
Example:
Use
by
name
.
:
-
name
:
nginx
image
:
-
:
image
:
-
:
image
:
:
-
name
:
nginx
image
:
-
name
:
image
:
-
:
image
:
:
-
name
:
nginx
image
:
-
name
:
image
:
-
name
:
image
:
args
:
[
"run"
]
-
:
image
:
:
-
name
:
nginx
image
:
-
name
:
image
:
args
:
[
"run"
]
-
name
:
image
:
-
name
:
image
:
args
args
Note:
the
p

atchStrategy
tag in
types.go
If no
strategy
:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
:
5
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
:
:
:
apps/v1
kind
:
# ...
spec
:
selector
:
:
app
:
nginx
:
5
replicas
:
1
strategy
:
:
maxSurge
:
1
:
1
type
:
template
:
metadata
:
:
null
labels
:
app
:
nginx
spec
:
:
-
image
:
:
name
:
nginx
ports
:
-
:
80
protocol
:
TCP
resources
:
{}
:
dnsPolicy
:
:
Always
:
{}
:
30
# ...
Example:
spec
:
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
spec
:
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
spec
:
strategy
:
type
:
:
maxSurge
:
1
:
1
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
spec
:
strategy
:
:
:
maxSurge
:
1
:
1
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.
to
values.
to
Recreate
. The
If the
it

 would have been more clear that they needed to be deleted.
of
Recreate
.
null
Use
.
.
.
Note:
status
Note:
Set the
Set the
Warning:
Example:
selector
:
:
:
template
:
metadata
:
labels
:
:
5.2 - Declarative Management of Kubernetes Objects Using Kustomize
Kustomize
through a
.
with
--kus

tomize
or
-k
flag:
Install
kubectl
.
minikube
.
and
files
list in
file:
cat
FOO=Bar
EOF
cat
files:
EOF
:
v1
data
:
:
|
FOO=Bar
kind
:
ConfigMap
metadata
:
name
:
envs
list in
.env
file:
cat
FOO=Bar
EOF
cat
envs:
- .env
EOF
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
name
:
Note:
.env
literals
cat
literals:
- FOO=Bar
EOF
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
name
:
FOO=Bar
EOF
:
apps/v1
kind
:
metadata
:
name
:
my-app
labels
:
app
:
my-app
spec
:
selector
:
:
app
:
my-app
template
:
metadata
:
labels
:
app
:
my-app
spec
:
:
-
name
:
app
image
:
my-app
:
-
name
:
config
mountPath
:
/config
volumes
:
-
name
:
config
configMap
:
name
:
EOF
cat &lt;&lt;EOF &gt;./kustomization.ya

ml
resources
:
:
-
name
:
files
:
EOF
:
v1
data
:
:
|
FOO=Bar
kind
:
ConfigMap
metadata
:
name
:
---
:
apps/v1
kind
:
metadata
:
labels
:
app
:
my-app
name
:
my-app
spec
:
selector
:
:
app
:
my-app
template
:
metadata
:
labels
:
app
:
my-app
spec
:
:
-
image
:
my-app
name
:
app
:
-
mountPath
:
/config
name
:
config
volumes
:
-
configMap
:
name
:
name
:
config
files
list in
cat
EOF
cat
files:
EOF
:
v1
data
:
:
kind
:
Secret
metadata
:
name
:
type
:
Opaque
literals
list in
cat
literals:
EOF
:
v1
data
:
password
:
c2VjcmV0
username
:
YWRtaW4=
kind
:
Secret
metadata
:
name
:
type
:
Opaque
cat
EOF
cat
metadata:
labels:
spec:
selector:
template:
metadata:
labels:
spec:
- name: app
volumes:
secret:
EOF
cat
files:
EOF
cat
literals:
- FOO=Bar
labels:
EOF
Run
:
v1
data
:
FOO
:
Bar
kind
:
ConfigMap
metadata
:
:
note
:
generated
labels
:
type
:
generated
name
:
cat
metadata:
labels:
app: nginx
spec:
selector:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
EOF
cat
&lt;&lt;EOF &gt;./kustomization.yam

l
app: bingo
EOF
Run
:
apps/v1
kind
:
metadata
:
:
:
labels
:
app
:
bingo
name
:
namespace
:
spec
:
selector
:
:
app
:
bingo
template
:
metadata
:
:
:
labels
:
app
:
bingo
spec
:
:
-
image
:
nginx
name
:
nginx
Composing
resources
resources
list.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
cat
EOF
and
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
metadata:
spec:
EOF
cat
metadata:
spec:
template:
spec:
limits:
EOF
cat
EOF
Run
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
3
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
nginx
name
:
my-nginx
ports
:
-
:
80
resources
:
limits
:
memory
:
512Mi
JSON patch
through
.
. For example, increasing the re

plica number of a Deployment object can also be done
through
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
value: 3
EOF
cat
- target:
EOF
Run
to see the
replicas
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
3
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
nginx
name
:
my-nginx
ports
:
-
:
80
images
field in
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
images:
EOF
Run
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
replicas
:
2
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
image
:
name
:
my-nginx
ports
:
-
:
80
or
file. It is
vars
.
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
EOF
cat
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
cat
vars:
objref:
EOF
Run
:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
2
selector
:
:
run
:
my-nginx
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
command
:
- start
- --host
image
:
nginx
name
:
my-nginx
bases
and
overlays
. A
base
as long as a
kustomization.yam

l
overlay
bases
. A
base
mkdir base
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
EOF
cat
metadata:
labels:
spec:
ports:
- port: 80
selector:
EOF
cat
EOF
mkdir dev
cat
bases:
- ../base
EOF
mkdir prod
cat
bases:
- ../base
EOF
Use
or
-k
in
kubectl
.
Note that
-k
,
cat
metadata:
spec:
selector:
template:
metadata:
labels:
spec:
ports:
EOF
cat
EOF
:
:
:
deleted
Field
Type
namespace
string
string
string
resources
[]string
[]
[]
bases
[]string
[]string
[]
Patch
vars
[]
Var
images
[]
Image
[]string
Each entr

y in this list should resolve to a file containing
crds
[]string
Kustomize
5.3 - Managing Kubernetes Objects Using Imperative Commands
kubectl
Install
kubectl
.
minikube
.
The
kubectl
See
The
kubectl
run
expose
autoscale
The
kubectl
to create.
create
command.
command.
-h
The
kubectl
scale
annotate
label
The
kubectl
set
Note:
The
kubectl
edit
patch
.
delete
Note:
You can use
for both imperative commands and imp

erative object
get
get -h
describe
logs
Using
set
in a
create
set
and
create
create
set
create
=
"None"
=
set
The
The
The
Using
--edit
You can use
=
"None"
=
The
.
The
5.4 - Imperative Management of Kubernetes Objects Using Configuration Files
kubectl
Install
kubectl
.
minikube
.
The
kubectl
See
You can use
Warning:
replace
, where
the
file. Independently managed fields must be copied to the confi

guration
replace
You can use
You can use
Note:
metadata
name
.
You can use
The
-o yaml
Use
The
create
,
replace
, and
delete
replace
replace
Warning:
selector
:
:
:
template
:
metadata
:
labels
:
:
5.5 - Update API Objects in Place Using kubectl patch
minikube

.
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
2
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
image
:
nginx
:
-
effect
:
key
:
dedicated
value
:
test-team
1/1
spec
:
template
:
spec
:
:
-
name
:
image
:
redis
:
-
image
:
redis
:
Always
name
:
...
-
image
:
nginx
:
Always
name
:
...
2/2
  ...
  ...
.
key
field of
PodSpec
of
merge
:
type
PodSpec
struct
{
...
...
}
:
{
...,
{
},
"name"
,
"merge"
}
.
spec
:
template
:
spec
:
:
-
effect
:
key
:
disktype
value
:
ssd
kubectl patch deployment patch-demo --patch-file patch-

file-tolerations.yaml
:
-
effect
:
key
:
disktype
value
:
ssd
replace
.
type
PodSpec
struct
{
...
...
}
.
The
type
Merge type
json
merge
strategic
.
type
strategic
spec
:
template
:
spec
:
:
-
name
:
image
:
type
to
merge
:
The
list.
spec:
...
1/1
0
1m
0
1m
strategy:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
strategy
:
:
maxSurge
:
30
%
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
image
:
nginx
strategy.
spec
:
strategy
:
type
:
Recreate
type
as
Recreate
:
type
spec
:
strategy
:
:
- type
type
:
Recreate
type
key of the
strategy
Patch your Deployment again with this new pat

ch:
spec
:
strategy
:
type
:
Recreate
template
:
list.
The
Strategy
of
:
type
struct
{
...
...
}
:
{
...,
{
"$ref":
,
,
},
....
}
.
The
{
"spec"
: {
: {
"spec"
: {
: [
{
"name"
:
,
"image"
:
"redis"
}
]
}
}
}
}
with
The flag
status
and
scale
status
or
scale
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
app
=
nginx
flag:
kubectl patch deployment ngi

nx-deployment --subresource
=
'scale'
--type
=
'merge'
-p
app
=
nginx
...
spec
:
replicas
:
3
...
status
:
...
:
3
:
3
replicas
:
3
Note:
If you run
Summary
,
,
,
,
and
.
Note:
6 - Managing Secrets
6.1 - Managing Secrets using kubectl
Secrets
using the
kubectl
minikube
A
Secret
admin
.
\
=
username
=
admin
\
=
password
=
''
$
,
\
,
*
,
=
, and
!
echo
-n
'admin'
echo
-n
The
-n
kubectl
kubectl
command:
\
=
\
=
using
\
=
username
=
\
=
password
=
Verify the Secre

t

Data
====
and
of a
Secret
Secret
jsonpath
=
'{.data}'
{
:
,
:
}
Decode the
password
data:
echo
Caution:
this method could cause the command with the encoded data to be stored in
jsonpath
=
Secret
immutable
. To edit a
data
#
:
v1
data
:
password
:
username
:
YWRtaW4=
kind
:
Secret
metadata
:
:
name
:
namespace
:
default
:
uid
:
type
:
Opaque
Clean up
6.2 - Managing Secrets using Configuration File
minikube
Secret
Secret
data
and
.
The
data
The keys of
data
and
-
,
_
or
.
.
data
field.
echo
-n
'admin'
| base64
echo
-n
| base64
Note:
base64
-b
should
-w 0
to
base64
if the
-w
YWRtaW4=
apiV

ersion
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
password
:
.
:
.
apiUrl
:
username
:
password
:
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
:
:
|
.
:
v1
data
:
:
kind
:
Secret
metadata
:
:
name
:
mysecret
namespace
:
default
:
"7225"
uid
:
type
:
Opaque
data
and
data
and
is used.
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
:
username
:
The
Secret
:
v1
data
:
username
:
kind
:
Secret
metadata
:
:
name
:
mysecret
namespace
:
default
:
"7579"
uid
:
type
:
Opaque
decodes to
.
data
or
Secret
immutable
.
echo
-n
| base64
Update the
data
:
v1
kind
:
Secret
metadata
:
name
:
mysecret
type
:
Opaque
data
:
username
:
YWRtaW4=
password
:
Secret
kubectl
tool
Secret
kubectl
Secret
instead,
kubectl
uses
instead.
Clean up
6.3 - Managing Secrets using Kustomize
kubectl

kubectl
.
minikube
in a
.env
files, or
admin
.
Literals
Files
.env files
:
-
name
:
literals
:
echo
-n
'admin'
echo
-n
The
-n
files.
Create the
file:
:
-
name
:
files
:
file by
providing
.env
file
file:
:
-
name
:
envs
:
file
must
be
or
.
.
In your
password
.
Secret
existing
Secret
your Pods.
Clean up
kubectl
:
7 - Inject Data Into Applications
7.1 - Define a Command and Arguments for a Container
in a
Pod
.
minikube
.
command
the
args
field in the configuration file. The command and arguments tha

t
Note:
The
command
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
purpose
:
spec
:
:
-
name
:
image
:
debian
command
:
[
]
args
:
[
,
]
:
OnFailure
env
:
-
name
:
MESSAGE
value
:
command
:
[
]
args
:
[
]
and
Secrets
.
Note:
"$(VAR)"
. This is
command
or
args
field.
command:
[
"/bin/sh"
]
args:
[
"-c"
,
]
.
.
See
Container
.
7.2 - Define Dependent Environment Variables
minikube
value
of
env
Pod:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
args
:
command
:
- sh
- -c
image
:
env
:
-
name
:
value
:
"80"
-
name
:
value
:
-
name
:
value
:
-
name
:
PROTOCOL
value
:
"https"
-
name
:
SER

VICE_ADDRESS
value
:
-
name
:
value
:
.
case.
env
The
$
, ie:
.
.
See
.
7.3 - Define Environment Variables for a Container
minikube
env
or
envFrom
and
value
Pod:
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
purpose
:
spec
:
:
-
name
:
image
:
env
:
-
name
:
value
:
-
name
:
value
:
purpose
=
kubectl
exec
...
Note:
env
or
envFrom
field
Note:
Environment variables may reference each other, however orderin

g is important.
GREETING
,
HONORIFIC
, and
NAME
,
, and
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
bash
env
:
-
name
:
GREETING
value
:
-
name
:
HONORIFIC
value
:
-
name
:
NAME
value
:
command
:
[
"echo"
]
args
:
[
]
.
.
See
.
7.4 - Expose Pod Information to Containers Through Environment Variables
.
minikube
Use Pod fields as values for environment variables
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
sleep 10;
done;
env
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
MY_POD_IP
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
:
Never
env
Note:
minikube
default
default
To see why these values

 are in the log, look at the
command
and
args
fields
kubectl
exec
printenv
...
...
...
container
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
sleep 10;
done;
resources
:
requests
:
memory
:
"32Mi"
cpu
:
"125m"
limits
:
memory
:
"64Mi"
cpu
:
"250m"
env
:
-
name
:
valueFrom
:
:
:
resource
:
-
name
:
valueFrom
:
:
:
resource
:
-
name
:
valueFrom
:
:
:
resource
:
-
name
:
valueFrom
:
:
:
resource
:
:
Never
env
field of a container named
1
1
33554432
67108864
Read
Read the
spec
that you
PodSpec
Container
EnvVar
7.5 - Expose Pod Information to Containers Through Files
volume
,
A
.
minikube
or you can use one of thes

e Kubernetes playgrounds:
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
zone
:
cluster
:
rack
:
rack-22
:
build
:
two
builder
:
john-doe
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
sleep 5;
done;
:
-
name
:
podinfo
mountPath
:
volumes
:
-
name
:
podinfo
:
items
:
-
path
:
"labels"
fieldRef
:
fieldPath
:
-
path
:
fieldRef
:
fieldPath
:
Volume,
.
Look at the
items
defines a
volume.
labels
.
.
Note:
labels
file:

kubectl
exec
labels
file:
to the
labels
file:
cluster
=
rack
=
"rack-22"
zone
=
file:
labels
and
files
. In the
..data
is
labels
and

total 8
..data
rename(2)
.
Note:
subPath
/#
exit
container
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
comma

nd
:
[
"sh"
,
"-c"
]
args
:
sleep 5;
done;
resources
:
requests
:
memory
:
"32Mi"
cpu
:
"125m"
limits
:
memory
:
"64Mi"
cpu
:
"250m"
:
-
name
:
podinfo
mountPath
:
volumes
:
-
name
:
podinfo
:
items
:
-
path
:
:
:
resource
:
divisor
:
1m
-
path
:
:
:
resource
:
divisor
:
1m
-
path
:
:
:
resource
:
divisor
:
1Mi
-
path
:
:
:
resource
:
divisor
:
1Mi
volume
,
.
Look at the
items
,
1m
should be
cpu_limit
. The
divisor
1
cpu
bytes for
memory
kubectl
exec
cpu_limit
file:
,
mem_limit
and
files.
Secrets
.
Read the
spec
that you
Check the
Volume
Check the
Check the
Check the
7.6 - Distribute Credentials Securely Using Secrets
minikube
my-app
echo
-n
'my-app'
| base64
echo
-n
| base64
bXktYXBw
,
and 

the base-64 representation of your password is
.
Caution:
:
v1
kind
:
Secret
metadata
:
name
:
data
:
username
:
bXktYXBw
password
:
Output:
Output:

Data
====
=
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
:
-
name
:
mountPath
:
volumes
:
-
name
:
secret
:
:
Output:
kubectl
exec
.
username
and
password
files:
echo
"
$(
)
"
echo
"
$(
)
"
my-app
=
=
Assign the
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
env
:
-
name
:
valueFrom
:
:
name
:
key
:
kubectl
exec
=
=

=
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
env
:
-
name
:
valueFrom
:
:
name
:
key
:
-
name
:
valueFrom
:
:
name
:
db-user
key
:
kubectl
exec
Note:
=
username
=
'my-app'
=
password
=
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
envFrom
:
-
secretRef
:
name
:
username
and
password
kubectl
exec
Secret
Volume
Pod
Secrets
.
Volumes
.
8 - Run Applications
8.1 - Run a Stateless Application Using a Deployment
minikube
.
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
 Pod Template

:
    nginx:
 No events.
where
specifies that the deployment should be updated to use nginx 1.16.1.
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
replicas
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
4
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.
.
8.2 - Run a Single-Instance Stateful Application
minikube
Killerc

oda
.
,
or
used here.
:
v1
kind
:
Service
metadata
:
name
:
mysql
spec
:
ports
:
-
port
:
3306
selector
:
app
:
mysql
clusterIP
:
None
---
:
apps/v1
kind
:
metadata
:
name
:
mysql
spec
:
selector
:
:
app
:
mysql
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
mysql
spec
:
:
-
image
:
mysql:5.6
name
:
mysql
env
:
-
name
:
value
:
password
ports
:
-
:
3306
name
:
mysql
:
-
name
:
mountPath
:
volumes
:
-
name
:
:
claimName
:
:
v1
kind
:
metadata
:
name
:
labels
:
type
:
local
spec
:
:
manual
capacity
:
storage
:
20Gi
:
hostPath
:
path
:
---
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
20Gi
    mysql:
     Mounts:
   Volumes:
 Events:
 mysql-63082529-2

z3ki   1/1       Running   0          3m

mysql&gt;
Updating
with the
.
Use
strategy:
not
Recreate
.
Volumes
and
8.3 - Run a Replicated Stateful Application
.
Note:
minikube
,
or
used here.
and
,
Pods
,
Services
, and
.
Some familiarity with MySQL helps, but this tutorial

 aims to present
:
v1
kind
:
ConfigMap
metadata
:
name
:
mysql
labels
:
app
:
mysql
:
mysql
data
:
:
|
[mysqld]
log-bin
:
|
[mysqld]
my.cnf
:
v1
kind
:
Service
metadata
:
name
:
mysql
labels
:
app
:
mysql
:
mysql
spec
:
ports
:
-
name
:
mysql
port
:
3306
clusterIP
:
None
selector
:
app
:
mysql
---
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
mysql
:
mysql
readonly
:
"true"
spec
:
ports
:
-
name
:
mysql
port
:
3306
selector
:
app
:
mysql
mysql
resolving
replicas.
writes.
:
apps/v1
kind
:
metadata
:
name
:
mysql
spec
:
selector
:
:
app
:
mysql
:
mysql
:
mysql
replicas
:
3
template
:
metadata
:
labels
:
app
:
mysql
:
mysql
spec
:
:
-
name
:
image
:
mysql:5.7
command
:
- bash
-
"-c"
- |
set -ex
else
fi
volume

Mounts
:
-
name
:
conf
mountPath
:
-
name
:
mountPath
:
-
name
:
image
:
command
:
- bash
-
"-c"
- |
set -ex
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
:
-
name
:
mysql
image
:
mysql:5.7
env
:
-
name
:
value
:
"1"
ports
:
-
name
:
mysql
:
3306
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
resources
:
requests
:
cpu
:
500m
memory
:
1Gi
:
exec
:
command
:
[
,
"ping"
]
:
30
:
10
:
5
:
exec
:
command
:
[
"mysql"
,
"-h"
,
,
"-e"
,
"SELECT 1"
]
:
5
:
2
:
1
-
name
:
image
:
ports
:
-
name
:
:
3307
command
:
- bash
-
"-c"
- |
set -ex
fi
fi
:
-
name
:
data
mountPath
:
subPath
:
mysql
-
name
:
conf
mountPath
:
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
volumes
:
-
name
:
conf
emptyDir
:
{}
-
name
:
configMap
:
name
:
mysql
:
-
metadata
:
name
:
data
spec
:
:
[
]
resources
:
requests
:
storage
:
10Gi
app
=
Running
:
Press
Ctrl+C
Note:
If

 you don't see any progress, make sure you have a dynamic PersistentVolume
.
mysql-0
,
mysql-1
, and
mysql-2
.
hostname
command.
conf.d
or
conf.d
.
0
,
N
is
N+1
.
mysql
mysqld
sidecar
.
The
mysqld
and
(
), they automatically find the primary

 server even if it gets a new
)
mysql:5.7
mysql
=
=
Never --
\
EOF
=
=
Never --
\
| message |
| hello   |
in a loop:
=
=
Never --
\
bash -ic
Ctrl+C
The
for the
mysql
kubectl
exec
mysql-2
and
mysql
Look for
1/2
in the
READY
column:
102
anymore.
server-id
as
,
102
mysql-2
.
kubectl
exec
mysql-2

102
drain
.
Replace
Caution:
cluster.
[...]
102
=
5
app
=
103
and
104
the
existed:
=
=
Never --
\
| message |
| hello   |
=
3
Note:
app
=
mysql
data-mysql-4   Bound     pvc-500a9957-b1c5-11e6-93fa-42010a800002   10Gi       RWO  

         2m
Cancel the
Ctrl+C
app
=
mysql
app
=
mysql
.
.
.
.
Look in the
8.4 - Scale a StatefulSet
.
or
=
,
update
:
:
Or use
:
Learn

 more about
.
8.5 - Delete a StatefulSet
.
kubectl
.
=
orphan
By passing
to
=
MyApp
Note:
grace
=
$(
)
=
MyApp
sleep
$grace
=
MyApp
.
8.6 - Force Delete StatefulSet Pods
never
at most one
Manual force deletion should be undertaken with caution, as it has the potential to violate the at most one semantics inherent to StatefulSet. StatefulSets may be used to run distributed and clustered applications which have a need for a stable network identity and stable storage. These applications often have configuration which relies on an ensemble of a fixed number of members with fixed identities. Having multiple members with the same identity 

can be disastrous and may lead to data loss (e.g. split brain scenario in quorum-based systems).
must not
specify a
timeout
.
).
The kubelet on the unresponsive Node starts responding, kills the Pod and removes the entry from the apiserver.
do not
=
0
--force
--force
=
0
Unknown
.
8.7 - Horizontal Pod Autoscaling
a
or
), with the
Pods
.
vertical
Horizontal pod autoscaling does not apply to objects that can't be scaled (for exa

mple:
a
DaemonSet
.)
.
There is
of using
graph BT
scale
end
enabled
,
API
(
,
, or
). The
API is
.
and StatefulSet). These resources each have a subre

source named
scale
.
value:
200m
is
100m
50m
When a
or
the
are
Ready
.
or
The
original
used.
into a desired replica count (e.g. due t

o an error fetching the metrics
API Object
the
.
.
.
thrashing
,
or
flapping
replicas
replicas
cpu
and
memory
should
like this:
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
60
Algorithm
Note:
use of the web application

, ignoring the sidecar container and its resource use.
Algorithm
type
:
:
name
:
cpu
container
:
target
:
type
:
:
60
in the
Note:
(the
API.
See
(the
The
.
the HPA V2
,
and
.
and
.
(the
v2
behavior
field
(see the
)
You specify these behaviours by 

setting
scaleUp
and / or
scaleDown
under the
behavior
field.
flapping
behavior
behavior
:
scaleDown
:
policies
:
-
type
:
Pods
value
:
4
:
60
-
type
:
Percent
value
:
10
:
60
(Pods)
(Percent)
(Pods)
is applied
Min
Disabled
flapping
of
scale.
scaleDown
.
behavior
:
scaleDown
:
:
300
behavior
:
scaleDown
:
:
300
policies
:
-
type
:
Percent
value
:
100
:
15
scaleUp
:
:
0
policies
:
-
type
:
Percent
value
:
100
:
15
-
type
:
Pods
value
:
4
:
15
:
Max
300
running replicas will be added every 15 seconds till the HPA reaches its ste

ady state.
behavior
:
scaleDown
:
:
60
behavior
:
scaleDown
:
policies
:
-
type
:
Percent
value
:
10
:
60
to
Min
means
behavior
:
scaleDown
:
policies
:
-
type
:
Percent
value
:
10
:
60
-
type
:
Pods
value
:
5
:
60
:
Min
The
value of
Disabled
behavior
:
scaleDown
:
:
Disabled
kubectl
.
command.
.
.
foo
80%
to
false
of
).
kubectl
cluster-level au

toscaler such as
.
Read a
.
Read the
8.8 - HorizontalPodAutoscaler Walkthrough
A
a
or
), with the
Pods
.
vertical
minikube
.
).
the
kubelets
,
using an
to add
.
Service
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
run
:
template
:
metadata
:
labels
:
run
:
spec
:
:
-
name
:
image
:
ports
:
-
:
80
resources
:
limits
:
cpu
:
500m
requests
:
cpu
:
200m
---
:
v1
kind
:
Service
metadata
:
name
:
labels
:
run
:
spec
:
ports
:
-
port
:
80
selector
:
run
:
kubectl
. There is
part of
kubectl
and then the ReplicaSet either adds or removes Pods based on the change to i

ts
.spec
.
See
=
50
--min
=
1
--max
=
10
(the
TARGET
=
=
Now run:
Note:
busybox
.
form:
Open the
:

/v2
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
metrics
:
-
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
50
status
:
:
1
:
:
1
:
1
:
-
type
:
Resource
resource
:
name
:
cpu
current
:
:
0
:
0
metrics
.
of
instead of
, and
.
only
support a
target
type of
.
type
:
Pods
pods
:
metric
:
name
:
target
:
type
:
:
1k
target
types of
both
Value
and
. With
Value
metric.
type
:
Object
object
:
metric
:
name
:
:
:
kind
:
Ingress
name
:
target
:
type
:
Value
value
:
2k
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
metrics
:
-
type
:
Resource
resource
:
name
:
cpu
target
:
type
:
:
50
-
type
:
Pods
pods
:
metric
:
name
:
target
:
type
:
:
1k
-
type
:
Object
object
:
metric
:
name
:
:
:
kind
:
Ingress
name
:
target
:
type
:
Value
value
:
10k
status
:
:
1
:
:
1
:
1
:
-
type
:
Resource
resource
:
name
:
cpu
current
:
:
0
:
0
-
type
:
Object
object
:
metric
:
name
:
requests-per-sec

ond
:
:
kind
:
Ingress
name
:
current
:
value
:
10k
labels
with the
verb
type
:
Object
object
:
metric
:
name
:
selector
:
{
:
{
verb
:
GET}}
not
Pods
Object
type).
with
.
metric
name
and
selector
External
Object
.
,
Value
and
Object
type.
-
type
:
External
external
:
metric
:
name
:
selector
:
:
queue
:
target
:
type
:
:
30
in any way.
we can use
:
Min replicas:              

     1
Events:
,
False
quantity
10500m
10.5
between
1
and
1500m
, or
1
and
1.5
:
kind
:
metadata
:
name
:
spec
:
:
:
apps/v1
kind
:
name
:
:
1
:
10
:
50
8.9 - Specifying a Disruption Budget for your Application
nodes.
.
and/or
.
.
; the same
.
From version 1.15 PDBs

 support custom controllers where the
.
Values for
or
"50%"
to
"50%"
you set
to
"50%"
code
A
Note:
and
in a single
.
is the
scale
.
5 or more
healthy
selector
.
of 30%, evictions are allowed as long as at

 least 30%
Note:
If you set
.
.
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
:
2
selector
:
:
app
:
zookeeper
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
:
1
selector
:
:
app
:
zookeeper
zk-pdb
:
policy/v1
kind
:
metadata
:
:
…
:
:
1
name
:
zk-pdb
…
status
:
:
3
:
2
:
1
:
3
:
1
item with
and
.
Note:
on the
API server
.
does 

not fall below the number specified in
By using
to the
policy.
Policies:
).
Ready
Ready
Note:
Pods in
Pending
,
Succeeded
or
Failed
only
.
8.10 - Accessing the Kubernetes API from a Pod
minikube
. These
.
The
See
.
.
The
See
.
page.
secure

ly with the API server.
in the
default
at
.
at
as the
command
localhost
APISERVER
=
=
NAMESPACE
=
$(
cat
${
}
)
TOKEN
=
$(
cat
${
}
/token
)
CACERT
=
${
}
/ca.crt
${
CACERT
}
--header
${
TOKEN
}
"
-X GET
${
APISERVER
}
/api
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
9 - Run Jobs
9.1 - Running Automated Tasks with a CronJob
CronJob
to run
Jobs
Cron
CronJobs
.
minikube
:
batch/v1
kind
:
CronJob
metadata
:
name
:
hello
spec
:
schedule
:
"* * * * *"
:
spec
:
template
:
spec
:
:
-
name
:
hello
image
:
:
command
:
- /bin/sh
- -c
:
OnFailure
cronjo

b.batch/hello created
jobs
--watch
hello
Note:
pods
=
$(
=
job-name
=
=
jsonpath
={
.items
[
*
]
}
)
$pods
:
.
,
kind
, and
metadata
fields.
manifests
, see the
,
and
.spec
section.
Note:
not
Schedule
The
.spec
.
It takes a
Cron
0 * * * *
or
@hourly
,
:
with
0-23/2
*/2
.
Note:
?
*
, that is,
The
Job
or
kind
.
.spec
, see
.
The
Jobs that do not meet their deadline in

 this way count as failed jobs.
If the
200
The
Allow
Forbid
Replace
Suspend
The
true
Caution:
When
true
to
false
The
and
0
9.2 - Coarse Parallel Processing Using a Work Queue
Job
.
minikube
replicationcontroller "rabbitmq-controller" creat

ed
.
....
[
]
....
....
[
]
....

export
=
=
-q foo -d
foo
=
=
-q foo -c
1
cat
echo
Hello
-c 1
)
cat
printed.
=
-q job1  -d
job1
for
do
=
$f
done
# Just prints standard out and s

leeps for 10 seconds.
import
sys
import
time
print
(
+
sys
.
stdin
.
0
])
time
.
sleep(
10
)
.
,
and
worker.py
For the
Docker Hub
Registry
, tag
.
:
batch/v1
kind
:
Job
metadata
:
name
:
job-wq-1
spec
:
:
8
:
2
template
:
metadata
:
name
:
job-wq-1
spec
:
:
-
name
:
c
image
:
env
:
-
name
:
value
:
-
name
:
QUEUE
value
:
job1
:
OnFailure
kubectl
wait
--for
=
condition
=
complete
--timeout
=
   c:
    Port:
Events:
This approach has t

he advantage that you
.
example
A
Caveats
9.3 - Fine Parallel Processing Using a Work Queue
minikube
Job
.
See the
job.yaml
worker.py
printed.
"/bin/sh"
Waiting
for
false
Hit enter
for
command
prompt
redis:6379

&gt; rpush job2 "apple"
1) "apple"
4) "date"
5) "fig"
6) "grape"
7) "lemon"
8) "melon"
job2
.
Download
).
import
time
import
rediswq
host
=
"redis"
# import os
q
=
rediswq
.
=
"job2"
, host
=
host)
print
(
+
q
.
print
(
+
str
(q
.
empty()))
while
not
q
.
empty():
item
=
q
.
=
10
, block
=
True
, timeout
=
2
)
if
item
is
not
None
:
itemstr
=
item
.
decode(
"utf-8"
)
print
(
+
itemstr)
time
.
sleep(
10
)
q
.
else
:
print
(
)
print
(
)
worker.py
,
, and
the image:
For the
Docker Hub
.
Registry
, tag
:
batch/v1
kind
:
Job
metadata
:
name
:
job-wq-2
spec
:
:
2
template
:
metadata
:
name
:
job-wq-2
spec
:
:
-
name
:
c
image
:
:
OnFailure
change
too.
=
=
job-name
=
job-wq-2
2
11
Jan
2016
1
Running /

0
0
Failed
=
job-name
=
job-wq-2
c:
Port:
Events:
33s          33s
1
{
}
kubectl
wait
--for
=
condition
=
complete
--timeout
=
.
instead,
.
9.4 - Indexed Job for Parallel Processing with Static Work Assignment
.
Start an
Indexed
.
Job
.
minikube
.
Read the
index.
rev
rev
busybox
string). In a real workload you might, for example, create a Job th

at represents
the
Indexed
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
5
:
3
:
Indexed
template
:
spec
:
:
Never
:
-
name
:
'input'
image
:
command
:
-
"bash"
-
"-c"
- |
:
-
mountPath
:
/input
name
:
input
:
-
name
:
'worker'
image
:
command
:
-
"rev"
-
:
-
mountPath
:
/input
name
:
input
volumes
:
-
name
:
input
emptyDir
:
{}
.
API
from a
.
,
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
5
:
3
:
Indexed
template
:
spec
:
:
Never
:
-
name
:
'worker'
image
:
command
:
-
"rev"
-
:
-
mountPath
:
/input
name
:
input
volumes
:
-
name
:
input
:
items
:
-
path
:
fieldRef
:
fieldPath
:
Because
kubectl
wait
--for
=
condition
=
complete
--timeout
=
   input:
    Command:
      bash
      -c

    Mounts:
   worker:
    Image:      docker.io/li

brary/busybox
    Command:
      rev
    Mounts:
  Volumes:
   input:
    Medium:
Events:
xuq
9.5 - Job with Pod-to-Pod Communication
without
Job
.
minikube
.
Note:
Set up a
job-name
subdomain
:
Example
Note:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
clusterIP
:
None
se

lector
:
job-name
:
---
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
3
:
3
:
Indexed
template
:
spec
:
subdomain
:
:
Never
:
-
name
:
image
:
command
:
- bash
- -c
- |
for i in 0 1 2
do
do
sleep 1
fi
done
done
using:
Note:
None
or
Default
.
here
.
9.6 - Parallel Processing using Expansions
Jobs
based on a common template. You can use this approach to process batches of work in
parallel.
apple
,
banana
, and
cherry
.
See
Job
.
minikube
sed
.
Python
.
:
batch/v1
kind
:
Job
metadata
:
name
:
labels
:
jobgroup
:
spec
:
template
:
metadata
:
name
:
labels
:
jobgroup
:
spec
:
:
-
name
:
c
image
:
command
:
[
"sh"
,
"-c"
,
]
:
Never
manifest
.
$ITEM
sed
$ITEM
jobs
for
do
$i
/"
$i
.yaml
done
ls jobs/
w

rite a program to generate the Job manifests.
jobs
-l
jobgroup
=
Using the
-l
:
jobgroup
=
jobgroup
=
Clean up
jobgroup
=
In the
, each instance of the template had one
names
:
%}
---
kind: Job
metadata:
  labels:
spec:
  template:
    metadata:
      labels:
    spec:
      - name: c
for
---
on a line
by itself.
kubectl
alias
=
Use
jobs.yaml
kubectl
:
Clean up
# Your cluster automatically cleans

 up their Pods
jobgroup
=
$ITEM
$ITEM
use HTTP to
PUT
frame.
labels
that
.
jobgroup
In the
Note:
jobgroup
There are
objects.
9.7 - Handling retriable and non-retriable pod failures with Pod failure policy
,
,
within a
Job
.
,
or
taint
Job
.
minikube
.
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
8
:
2
template
:
spec
:
:
Never
:
-
name
:
main
image
:

command
:
[
"bash"
]
args
:
- -c
:
6
:
rules
:
-
action
:
FailJob
:
:
main
operator
:
In
values
:
[
42
]
jobs
=
Failed
reason
equal
message
.
Clean up
limit.
Caution:
:
batch/v1
kind
:
Job
metadata
:
name
:
spec
:
:
4
:
2
template
:
spec
:
:
Never
:
-
name
:
main
image
:
command
:
[
"bash"
]
args
:
- -c
:
0
:
rules
:
-
action
:
Ignore
:
-
type
:
nodeName
nodeName
=
$(
=
jsonpath
=
)
$nodeName
=
0
jobs
=
$nodeName
,
10 - Access Applications in a Cluster
10.1 - Deploy and Access the Kubernetes Dashboard
as well as for creating or modifying individual Kubernetes re

sources
.
Warning:
kubectl
.
The UI can
only
Note:
not
namespace
manifest
Click the
CREATE
App name
A
label
namespace
.
A
Service
Service
Note:
If needed, you can expa

nd the
Labels
: Default
labels
to be used
Example:
Namespace
.
:
.
file.
and
:
and
:
.
.
syntax.
API
default
Workloads
Shows all appli

cations running in the selected namespace.
Services
Storage
.
10.2 - Accessing Clusters
kubectl
.
a
,
Many of the
examples
kubectl
.
=
8080
See
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Use
and
metadata:
EOF
while
'^token'
&gt;/dev

/null;
do
echo
&gt;&amp;
2
sleep
1
done
APISERVER
=
$(
":"
| tr -d
" "
)
TOKEN
=
$(
'^token'
| cut -f2 -d
':'
| tr -d
" "
)
curl
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
Using
jsonpath
:
APISERVER
=
$(
jsonpath
=
)
TOKEN
=
$(
jsonpath
=
)
curl
$TOKEN
"
{
"kind"
:
,
: [
"v1"
],
: [
{
:
,
:
}
]
}
~/.kube
for this.
Go
and
Python
Go client
, see
example
.
.
To use
. See
example
.
There are
.
The
:
The
:
client to proxy uses HTTPS (or http if

 apiserver so configured)
The
kube proxy
:
10.3 - Configure Access to Multiple Clusters
command.
Note:
a
.
Warning:
minikube
kubectl
run
of your
Define clusters, users, and contexts
In the
frontend
,
storage
. In your
scratch
cluster,
. In your
:
v1
kind
:
Config
:
{}
clusters
:
-
cluster
:
name
:
-
cluster
:
name
:
scratch
users
:
-
name
:
developer
-
name
:
contexts
:
-
context
:
name
:
-
context
:
name
:
-
context
:
name
:
file
Go to your
=
=
=
=
=
Caution:
Storing passwords in Kubernetes client config is risky. A better alternative would be to use a credential 

plugin and store them separately. See:
=
=
=
=
=
=
Note:
=
=
=
=
developer
=
=
=
=
developer
=
=
=
=
Open your
command.
=
:
v1
clusters
:
-
cluster
:
:
server
:
name
:
-
cluster
:
:
true
server
:
name
:
scratch
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
storage
user
:
developer
name
:
-
context
:
cluster
:
scratch
namespace
:
default
user
:
name
:
:
""
kind
:
Config
:
{}
users
:
-
name
:
developer
user
:
:
:
-
name
:
user
:
password
:
username
:
exp
The
,
and
-data
,
,
.
developer
frontend
cluster".
=
kubectl
context.
--minify
flag.
=
context:
:
v1
clusters
:
-
cluster
:
:
server
:
name
:
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
:
kind
:
Config
:
{}
users
:
-
name
:
developer
user
:
:
:
Now suppose you want to work for a while i

n the scratch cluster.
:
=
Now any
kubectl
the
scratch
context.
.
=
storage
cluster.
:
=
.
=
In your
:
v1
kind
:
Config
:
{}
contexts
:
-
context
:
cluster
:
namespace
:
ramp
user
:
developer
name
:
.
Linux
export
=
"
"
=
The
a
Linux
export
=
"
${
}
=(
)
In your
the
file:
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
ramp
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
storage
user
:
developer
name
:
-
context
:
cluster
:
scratch
namespace
:
default
user
:
name
:
kubectl
config
in the
Go to
config
Linux
export
=
"
${
}
:
${
HOME
}
=
in your
Clean up
Linux
export
=
"
"
=
It is not alway

s obvious what attributes (username, groups) you will get after authenticating to the cluster.
There is a
kubectl
.
Read
10.4 - Use Port Forwarding to Access Applications in a Cluster
minikube
.
Install
.
The output of a successful command verifies that the Service was created:
=
27017
or
or
or
Note:
28015
ping
comm

and:
{ ok: 1 }
kubectl
kubectl
The
kubectl
Note:
.
.
10.5 - Use a Service to Access an Application in a Cluster
minikube
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
run
:
replicas
:
2
template
:
metadata
:
labels
:
run
:
spec
:
:
-
name
:
image
:
ports
:
-
:
8080
protocol
:
TCP
.
Pods
=
=
Labels:
run
=
Selector:
run
=
=
--output
=
wide
NAME                      

     READY   STATUS    ...  IP           NODE
. If you are
nodes.
where
and
Follow the
tutorial.
10.6 - Connect a Frontend to a Backend Using Services
frontend
and a
backend
Service
object.
hello
object.
nginx
minikube
.
, which
NodePort
instead.
---
:
apps/v1
kind
:
metadata
:
name
:
backend
spec
:
selector
:
:
app
:
hello
tier
:
backend
track
:
stable
replicas
:
3
template
:
metadata
:
labels
:
app
:
hello
tier
:
backend
track
:
stable
spec
:
:
-
name
:
hello
image
:
ports
:
-
name
:
http
:
80
...
                               
 tier=backend
   hello:
Events:
...
hello
selectors
to find
---
:
v1
kind
:
Service
metadata
:
name
:
hello
spec
:
selector
:
app
:
hello
tier
:
backend
ports
:
-
protocol
:
TCP
port
:
80
:
http
...
hello
routes
app: hello
and
.
backend
hello
hello
of the
name
hello
}

server {
    listen 80;

    }
}
---
:
v1
kind
:
Service
metadata
:
name
:
frontend
spec
:
selector
:
app
:
hello
tier
:
frontend
ports
:
-
protocol
:
"TCP"
port
:
80
:
80
type
:
...
---
:
apps/v1
kind
:
metadata
:
name
:
frontend
spec
:
selector
:
:
app
:
hello
tier
:
frontend
track
:
stable
replicas
:
1
template
:
metadata
:
labels
:
app
:
hello
tier
:
frontend
track
:
stable
spec
:
:
-
name
:
nginx
image
:
"gcr.io/google-samples/hello

-frontend:1.0"
lifecycle
:
preStop
:
exec
:
command
:
[
,
"-s"
,
"quit"
]
...
Note:
be to use a
ConfigMap
,
frontend
:
heading:
frontend
cluster.
${
}
{
"message"
:
"Hello"
}
Services
10.7 - Create an External Load Balancer
Service
, you have
nodes,
.
Ingress
Ingress
minikube
type
:
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
example
ports
:
-
port
:
8765
:
9376
type
:
its
flag:
=
8765
=
9376
\
--name
=
=
named
example
).
For more information, including opt

ional flags, refer to the
reference
.
kubectl
:
.
Note:
source IP
.spec
options:
Cluster
Local
.
Cluster
Local
Service
type
is set
to
Local
.
Setting
:
v1
kind
:
Service
metadata
:
name
:
spec
:
selector
:
app
:
example
ports
:
-
port
:
8765
:
9376
:
Local
type
:
Where
or
type
service.

kubernetes.io/load-balancer-cleanup
.
type
type
Follow the
tutorial
Read about
Service
Read about
Ingress
10.8 - List All Container Images Running in a Cluster
minikube
.
using
image
See the
tr
,
sort
,
uniq
Use
tr
Use
sort
Use
uniq
to aggregate image counts
jsonpath
=
|
\
tr -s
'\n'
|
\
sort |
\
uniq -c
.items[*]
.spec
.image
Note:
,
the
.items[*]
range
jsonpath
=
|
\
sort
app=nginx
.
jsonpath
=
-l
app
=
nginx
jsonpath
=
=
"{{range .items}}{{range .spe

c.containers}}{{.image}} {{end}}{{end}}"
Reference
Jsonpath
10.9 - Set up Ingress on Minikube with the NGINX Ingress Controller
An
Ingress
minikube
.
Locally
enable
ingress
Note:
Note:
kubernetes-dashboard-5498ccf677-b8p5h       1/1       Running   0          2m
.
=
=
=
8080
Note:
31637
.
Create
service/netw

orking/example-ingress.yaml
:
kind
:
Ingress
metadata
:
name
:
:
:
/$1
spec
:
rules
:
-
host
:
http
:
paths
:
-
path
:
/
pathType
:
Prefix
backend
:
service
:
name
:
web
port
:
number
:
8080
Note:
file on
Note:
Note:
=
=
8080
--type
=
NodePort
-
path
:
/v2
pathType
:
Prefix
backend
:
service
:
name
:
web2
port
:
number
:
8080
Note:
Ingress
Services
10.10 - Communicate Between Containers in the Same Pod Using a Shared Volume
minikube
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
Never
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
image
:
nginx
:
-
name
:
mountPath
:
-
name
:
image
:
debian
:
-
name
:
shared

-data
mountPath
:
/pod-data
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
]
.
.
/pod-data
kind: Pod
metadata:
  ...
  ...
spec:
  ...

    ...
        ...
    ...

    ...
    ...
    state:
      running:
    ...
curl
.
.
See
.
See
See
Volume
.
See
Pod
.
10.11 - Configure DNS for a Cluster
.
10.12 - Access Services Running on Clusters
be configured to communicate with your cluster. It is recommended to run this tutorial on a cluster with at least two nodes that are not acting as control plane hosts. If you do not already have

 a
minikube
.
nodes
,
pods
and
services
all have
NodePort
or
services
and
Described
here
.
.
with the
command:
at
.
Note:
See
http://
/proxy
port_name
port_name
for both named and un

named ports.
https:
:
Examples
{
:
,
"status"
:
"yellow"
,
:
false
,
:
1
,
:
1
,
:
5
,
:
5
,
:
0
,
:
0
,
:
5
}
https
11 - Extend Kubernetes
11.1 - Configure the Aggregation Layer
minikube
.
Note:
Caution:
.
This section describes how the authentication a

nd authorization flows work, and how to configure them.
.
.
.
.
Note:
any
all
extension apiservers. It does not create a client certificate per extension apiser

ver, but rather a single one to authenticate as the Kubernetes apiserver. This same one is reused for all extension apiserver requests.
in the
named
If the
: When a request arrives to the Kubernetes apiserver, if this option is enabled, the Kubernetes apiserver checks the certificate of the request. If it is signed by one of the CA certificates in 

the file referenced by
CN=
O=
. See the
.
CN=
If
both
and
same
CN=
will
not
Warning:
Do
not
flag:
:
kind
:
metadata
:
name
:
spec
:
group
:
version
:
:
:
service
:
namespace
:
name
:
caBundle
:
.
The
service
:
kind
:
...
spec
:
...
service
:
namespace
:
name
:
port
:
1234
caBundle
:
...
Extending the Kubernetes API with the aggregation

 layer
.
.
11.2 - Use Custom Resources
11.2.1 - Extend the Kubernetes API with CustomResourceDefinitions
.
minikube
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
The
kind
CronTab
cronSpec
and
image
CronTab
. The kind
CronTab
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
You can then manage your CronTab objects

 using kubectl. For example:
NAME                 AGE
cronSpec
and
image
fields
:
v1
items
:
-
:
kind
:
CronTab
metadata
:
:
:
|
{
:
,
"kind"
:
"CronTab"
,
:{
:{},
"name"
:
,
:
"default"
},
"spec"
:{
:
"* * * * */5"
,
"image"
:
}}
:
:
1
name
:
namespace
:
default
:
"1326"
uid
:
spec
:
cronSpec
:
'* * * * */5'
image
:
kind
:
List
metadata
:
:
""
selfLink
:
""
fields
,
kind
and
metadata
With
which:
type
(via
or
(via
items
a node with
a node with
allOf
,
anyOf
,
oneOf
or
not
,
type
,
default
,
,
nullable
within an
allOf
,
anyOf
,
oneOf
or
not
if
metadata
and
allOf:
    foo:
      ...
  foo:
    ...
allOf:
    foo:
      ...
allOf:
- items:
      foo:
        ...
items:
    foo:
      ...
allOf:
- items:
      foo:
        ...
  foo:
  metadata:
      name:
        items:
anyOf:
    bar:
the type at the root is

 missing (rule 1).
the type of
foo
bar
inside of
anyOf
bar
's
type
is within
anyOf
(rule 3).
anyOf
(rule 3).
type
:
object
:
:
foo
:
type
:
string
pattern
:
"abc"
bar
:
type
:
integer
metadata
:
type
:
object
:
name
:
type
:
string
pattern
:
"^a"
anyOf
:
-
:
bar
:
minimum
:
42
required
:
[
"bar"
]
etcd
.
ConfigMap
,
pruned
to
might lack
might be
true
.
with
set to
true
Set
to
false
.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
:
42
=
false
:
kind
:
CronTab
metadata
:
:
:
1
name
:
namespace
:
default
:
"285"
uid
:
spec
:
cronSpec
:
'* * * * */5'
image
:
the
kubectl
in the
.
type
:
object
:
json
:
:
true
The field
json
type
:
object
:
json
:
:
true
type
:
object
:
object
):
type
:
object
:
json
:
:
true
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
bar
:
type
:
string
json
:
spec
:
foo
:
abc
bar
:
def
something
:
x
status
:
something
:
x
json
:
spec
:
foo
:
abc
bar
:
def
status
:
something
:
x
something
spec
type
:
object
:
foo
:
:
true
Also those nodes are partially excluded from rule 3 in the sense that th

e following two patterns are allowed
anyOf:
...
and
allOf:
  - anyOf:
...
In
,
)
and
kind
fields.
by setting
type
:
object
:
foo
:
:
true
:
true
foo
foo:
  kind: Pod
  spec:
    ...
Because
The use of
With
, the
,
kind
and
metadata
See
:
kind
:
CronTab
metadata
:
:
When the
,
.
,
,
,
,
id
,
,
readOnly
,
writeOnly
,
xml
,
$ref
.
The field
true
.
The field
false
.
The field
.
The
.
spec.repli

cas
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * *"
image
:
replicas
:
15
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
5
crontab
created
false
, for the
.
And
self
...
      spec:
          ...
          replicas:
        required:
      
    - replicas
:
kind
:
CronTab
metadata
:
name
:
spec
:
:
0
replicas
:
20
:
10
The
rule
under
The
message
has(self)
Rule
Purpose
Xref:
,
kind
,
metadata.

name
and
spec
and
status
in the
...
        spec:
            ...
        status:
via
...
      spec:
          ...
          foo:
and all
.
...
      spec:
          ...
            foo:
self[i]
and
...
      ...
      foo:
        items:
self
...
      spec:
          ...
          foo:
Examples:
string
The
,
kind
,
and
is not
.
__
__dot__
.
__dash__
-
__slash__
/
namespace
x-prop
redact__d
string
of
set
or
map
ignor

es element order,
i.e.,
set
:
X + Y
X
Y
map
:
X + Y
X
Y
X
and
Y
in
Y
CEL type
map
'array
list
'boolean'
boolean
double
int (64)
'null'
null_type
'string'
string
bytes
xref:
CEL types
,
,
.
macros
CEL
oldSelf
type
:
string
enum
:
[
"low"
,
"medium"
,
"high"
]
:
-
rule
:
message
:
path
rule
array
;
any
set
or
atomic
self
with
oldSelf
.
Use Case
Rule
Prevent modification/removal onc

e assigned
or
But if
foo
foo
is.
foo
.
foo
is not
maxItems
,
, and
maxLength
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
string
:
-
rule
:
self.all
foo
,
'a string'
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
maxLength
:
10
:
-
rule
:
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
:
-
rule
:
maxLength
:
10
self.all
lower limits set on both nested lists in order to be allowed. For 

example, even without having limits set,
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
integer
:
-
rule
:
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
array
items
:
type
:
integer
:
-
rule
:
foo
self.all
.
Note:
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
default
:
"5 0 * * *"
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
default
:
1
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
cronSpec
and
replicas
:
kind
:
CronTab
metadata
:
name
:
spec
:
image
:
leads to
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"5 0 * * *"
image
:
replicas
:
1
metadata
metadata
fields of
metadata
false
true
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
nullable
:
false
default
:
"default"
bar
:
type
:
string
nullable
:
true
baz
:
type
:
string
foo
and
bar
and
baz
spec
:
foo
:
null
bar
:
null
baz
:
null
leads to
spec
:
foo
:
"default"
bar
:
null
with
foo
bar
, and
baz
default.
which are
and
The
kubectl
and
)
The OpenAPI v3

 validation schema is converted to OpenAPI v2 schema, and
show up in
and
paths
.
The fields
allOf
,
anyOf
,
oneOf
and
not
If
type
,
nullable
,
items
and
Spec
,
Replicas
, and
Age
columns.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
:
-
name
:
Spec
type
:
string
:
jsonPath
:
-
name
:
Replicas
type
:
integer
:
jsonPath
:
-
name
:
Age
type
:
date
jsonPath
:
Notice the
NAME
,
SPEC
,
REPLICAS
, and
AGE
Note:
The
NAME
Priority
priority
-o wide
flag).
0
0
Type
A column's
type
):
integer
number
string
– strings
boolean
–
true
or
false
date
Format
A column's
format
int32
int64
float
double
byte
date
date-time
password
format
kubectl
/status
and
/scale
/status
.status
and
.spec
JSONPaths
PUT
/status
PUT

 the
/status
resource.
PUT
/
POST
/
PATCH
The
.metadata
or
.status
.
example
format
items
maximum
maxItems
maxLength
minimum
minItems
minLength
pattern
required
title
type
/scale
The
/scale
.
.
.spec
the
/scale
.
.status
/scale
.
.status
or
.spec
/scale
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
status
:
type
:
object
:
replicas
:
type
:
integer
:
type
:
string
:
status
:
{}
scale
:
:
:
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
3
and
A custom resource can be s

caled using the
command.
of the
=
5
crontabs
scaled
jsonpath
=
5
all
).
You can use
all
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
- all
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
:
CronTab
:
NAME                          AGE
Read about
.
See
.
Serve
of a
11.2.2 - Versions in CustomResourceDefinitions
minikube
.
.
Overview
v1beta1
v1beta1
API.
v1
.
version they are stored at and the ve

rsion they are served at. If the
None
the
list with
. Also, set
field
See the
section.
Set
served
to
false
list. If
Ensure the
storage
is set to
true
.
list.
versions
Note:
In
version
versions
. The
version
versions
field.
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
conversio

n
:
strategy
:
None
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
-
name
:
v1
served
:
true
storage
:
false
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
None
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
and
.
name
v
beta
or
alpha
like
v2
or
v2beta1
do not.
beta
or
alpha
beta
or
alpha
beta
, or
alpha
foo1
foo10
This might make sense if y

ou look at the following sorted version list:
- v10
- v2
- v1
- v11beta2
- v10beta3
- v3beta1
- foo1
- foo10
, the
v1
v1beta1
v1
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
:
true
:
schema
:
...
-
name
:
v1beta1
served
:
true
:
true
schema
:
...
-
name
:
v1
served
:
true
storage
:
true
schema
:
...
:
kind
:
metadata
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
:
...
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
:
true
:
-
name
:
v1beta1
served
:
true
:
true
-
name
:
v1
served
:
true
storage
:
true
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1beta1
served
:
false
schema
:
...
-
name
:
v1
served
:
true
storage
:
true
schema
:
...
Ku

bernetes v1.16 [stable]
Note:
field
server
Note:
field
empty
,
how to
.
metadata
other than
labels
and
.
name
,
UID
and
namespace
.
named
in
default
.
Note:
The
None
spec
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false

schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
Webhook
webhook
:
:
[
"v1"
,
"v1beta1"
]
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
kind
:
metadata
:
name
:
spec
:
group
:
:
false
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
Webhook
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
URL
url
(
).
The
host
a service refe

rence by specifying the
service
(i.e.,
host
localhost
or
127.0.0.1
as a
host
is
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
url
:
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
url
:
...
The
service
service
instead of
url
.
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
caBundle
:
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
caBundle
:
...
Request
,
with a
API group
with the
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
[
"v1"
,
"v1beta1"
]
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
[
"v1"
,
"v1beta1"
]
...
If no
v1beta1
.
con

versionReviewVersions
object
CronTab
objects to
:
{
,
"kind":
,
{
"uid":
,
,
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
}
]
}
}
{
,
"kind":
,
{
"uid":
,
,
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
}
]
}
}
Response
,
with the
response
response
uid
result
, set to
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Success"
},
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},

"host":
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
,
"port":
"2345"
}
]
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
},
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
"host":
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
,
"port":
"2345"
}
]
}
}
response
uid
result
, set to
Warning:
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
,
}
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
,
}
}
}
:
strategy
value
None
pruning
represent

ed in different fields between versions.
if
v1beta1
v1beta1
v1
v1
and
v1beta1
v1beta1
version
v1
field.
v1
. You now
v1beta1
v1
.
v1
since that
Option 1:
Run the
field.
Option 2:
v1beta1
to
v1
.
Set
v1
is now
.
v1
.
Remove
v1beta1
field.
Note:
The flag
status
and
scale
status
kubectl
tool
status
.
status
kubectl
:
=
'status'
--type
=
'merge'
-p
11.3 - Set up an Extension API Server
minikube
.
You must
Set up an extens

ion api-server to work with the aggregation layer
Create a Kubernetes secret with the server cert/key in your namespace.
.
.
11.4 - Configure Multiple Schedulers
here
.

minikube
.
Clone the
cd
make
FROM
busybox
ADD
.
.
configuration for it and run it in your Kubernetes cluster. But instead of creating a pod
manages a
:
:
v1
kind
:
metadata
:
name
:
namespace
:
---
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
---
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
---
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
data
:
:
|
profiles:
---
:
apps/v1
kind
:
metadata
:
labels
:
component
:
scheduler
tier
:
name
:
namespace
:
spec
:
selector
:
:
component
:
scheduler
tier
:
replicas
:
1
template
:
metadata
:
labels
:
component
:
scheduler
tier
:
version
:
second
spec
:
:
:
-
command
:
image
:
:
httpGet
:
path
:
/healthz
port
:
10259
scheme
:
HTTPS
:
15
name
:
:
httpGet
:
path
:
/healthz
port
:
10259
scheme
:
HTTPS
resources
:
requests
:
cpu
:
'0.1'
:
:
false
:
-
name
:
mountPath
:
:
false
hostPID
:
false
volumes
:
-
name
:
configMap
:
name
:
the
--config
my-sche

duler-config
a
.
Note:
field in a
.
.
for
for
=
....
...
You should see a "Running" my-scheduler pod, in addition to the default kube-scheduler
to
true
to
to
Note:
endpoints
and
leases
:
kind
:
metadata
:
:
:
"true"
labels
:
:
name
:
rules
:
-
apiGroups
:
resources
:
- leases
verbs
:
- create
-
apiGroups
:
:
resources
:
- leases
verbs
:
- get
- update
-
apiGroups
:
-
""
:
resources
:
verbs
:
- delete
- get
- patch
- update
:
v1
kind
:
Pod
metadata
:
name
:
no
labels
:
name
:
spec
:
:
-
name
:
image
:
pod1.yaml
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
name
:
spec
:
:
:
-
name
:
image
:
A scheduler is specified by supplying the

 scheduler name as a value to
.
pod2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
name
:
spec
:
:
:
-
name
:
image
:
deployed -
in the
.
pod3.yaml
while the other two pods get scheduled. Once we submit the scheduler deployment config
pod gets
11.5 - Use an HTTP Proxy to Access the Kubernetes API
minikube
.
=
=
8080
curl
,
wget
,
{
    "v1"
  ],
    {
    }
  ]
}
{
  },
  "items": [
    {
        },
        ...
}
Learn more

 about
.
11.6 - Use a SOCKS5 Proxy to Access the Kubernetes API
minikube
.
ssh
Note:
SOCKS5
proxies.
graph LR;
end
SOCKS5
sshd[SSH
end
enabled
ssh -D
1080
-D 1080
-q
-N
export
=
curl
Note:
localhost
localhost
.
The
curl
kubectl
proxy-ur

l
element
cluster
:
v1
clusters
:
-
cluster
:
:
LRMEMMW2
server
:
proxy-url
:
name
:
default
contexts
:
-
context
:
cluster
:
default
user
:
default
name
:
default
:
default
kind
:
Config
:
{}
users
:
-
name
:
default
user
:
:
:
kubectl
Clean up
CTRL+C
Type
11.7 - Set up Konnectivity service
minikube
.
:
kind
:
:
-
name
:
cluster
:
:
GRPC
transport
:
uds
:
udsName
:
.
Set the
spec
:
:
:
-
name
:
mountPath
:
readOnly
:
false
volumes
:
-
name
:
hostPath
:
path
:
type
:
/etc/ku

bernetes/pki/ca.crt
375
-sha256
SERVER
=
$(
jsonpath
=
)
=
true
"
$SERVER
"
=
true
static Pod
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
:
true
:
-
name
:
image
:
command
:
[
]
args
:
[
,
,
,
,
,
,
,
,
,
,
,
,
]
:
httpGet
:
scheme
:
HTTP
host
:
127.0.0.1
port
:
8134
path
:
/healthz
:
30
:
60
ports
:
-
name
:
agentport
:
8132
hostPort
:
8132
-
name
:
adminport
:
8133
hostPort
:
8133
-
name
:
:
8134
hostPort
:
8134
:
-
name
:
k8s-certs
mountPath
:
readOnly
:
true
-
name
:
mountPath
:
readOnly
:
true
-
name
:
mountPath
:
readOnly
:
false
volumes
:
-
name
:
k8s-certs
hostPath
:
path
:
-
name
:
hostPath
:
path
:
type
:
-
name
:
hostPath
:
path
:
type
:
:
apps/v1
kind
:
DaemonSet
metadata
:
labels
:
:
Reconcile
k8s-app
:
namespace
:
name
:
spec
:
selector
:
:
k8s-app

:
template
:
metadata
:
labels
:
k8s-app
:
spec
:
:
:
-
key
:
operator
:
"Exists"
:
-
image
:
name
:
command
:
[
]
args
:
[
,
,
,
,
,
,
]
:
-
mountPath
:
name
:
:
httpGet
:
port
:
8134
path
:
/healthz
:
15
:
15
:
volumes
:
-
name
:
projected
:
sources
:
-
:
path
:
audience
:
:
kind
:
metadata
:
name
:
labels
:
:
"true"
:
Reconcile
roleRef
:
apiGroup
:
kind
:
name
:
subjects
:
-
apiGroup
:
kind
:
User
name
:
---
:
v1
kind
:
metadata
:
name
:
namespace
:
labels
:
:
"true"
:
Reconcile
12 - TLS
12.1 - Configure Certificate Rotation for the Kubelet
Overview
rotation
,
The
kubelet
The
(
status of
Pending
a status of
Approved
The kubelet will retrieve the signed certificate from the Ku

bernetes API and
12.2 - Manage TLS Certificates in a Cluster
ACME
draft
.
Note:
CA for this purpose, but you should never rely on this. Do not assume that
minikube
cfssl
cfssl
from
.
jq
jq
, you can
.
custom CA
RootCAs
struct.
Note:
ConfigMap
),
ConfigMap
Note:
click here
cat
{
"hosts": [
],
"key": {
}
}
EOF
Where
10.0.34.2
2022/02/01 1

1:45:32 [INFO] received CSR
encoded
PKCS#10
and
cat
metadata:
spec:
usages:
EOF
signer.
A specific
Subject:
                        10.0.34.2
kubectl
,
cat
{
"key": {
}
}
EOF
ca.pem
).
{
"signing"
: {
"default"
: {
"

usages"
: [
,
,
],
"expiry"
:
"876000h"
,
: {
"is_ca"
:
false
}
}
}
}
Use a
jsonpath
=
|
\
\
\
.
\
jq
$(
'\n'
)
'"'
|
\
Note:
jq
field.
jq
Once the CSR is approved and the signed certificate is uploaded, run:
jsonpath
=
\
and
in a
Secret
ca.pem
into a
ConfigMap
=
ca.pem
and
Caution:
and
approve
approver
is
the
Configurin

g your cluster to provide signing
and
12.3 - Manual Rotation of CA Certificates
minikube
.
.
Caution:
This approach assumes operation of the Kubernetes control plane in a HA configuration with multiple API servers.
ca.crt
,
ca.key
,
,
and
Update the
to include
Any
Note:
and
ca.crt
and
Once
ca.crt
ca.crt
Issue 1350
ca.crt
and
flag in the
flag in the
and
.
Update the
to include
both old and new CA, then restart the cloud-controller-man

ager.
Note:
or
in
and
in
.
,
and
to use and
Note:
To generate certificates and private keys for your cluster using the
openssl
see
openssl
)
.
cfssl
.
for
$(
jsonpath
=
)
;
do
for
name in
$(
-o
jsonpath
=
)
;
do
${
namespace
}
${
name
}
-p
;
done
for
name in
$(
-o
jsonpath
=
)
;
do
${
namespace
}
${
name
}
-p
;
done
done
Note:
see
.
in the
=
"
$(
)
"
\
/bin/sed
${
}
/"
|
\
,
flag
13 - Manage Cluster Daemons
13.1 - Perform a Rolling Update on a DaemonSet
You need to have a Kubernetes cluster, 

and the kubectl command-line tool must
minikube
OnDelete
: With
OnDelete
only
before.
With
to
.
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
spec
:
selector
:
:
name
:
:
type
:
:
:
1
template
:
metadata
:
labels
:
name
:
spec
:
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
-
name
:
image
:
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
mountPath
:
readOnly
:
true
:
30
volumes
:
-
name
:
varlog
hostPath
:
path
:
/var/log
-
name
:
hostPath
:
path
:
.
:
=
=
=
DaemonSet
kubectl
commands.
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
fl

uentd-logging
spec
:
selector
:
:
name
:
:
type
:
:
:
1
template
:
metadata
:
labels
:
name
:
spec
:
:
-
key
:
operator
:
Exists
effect
:
-
key
:
operator
:
Exists
effect
:
:
-
name
:
image
:
resources
:
limits
:
memory
:
200Mi
requests
:
cpu
:
100m
memory
:
200Mi
:
-
name
:
varlog
mountPath
:
/var/log
-
name
:
mountPath
:
readOnly
:
true
:
30
volumes
:
-
name
:
varlog
hostPath
:
path
:
/var/log
-
name
:
hostPath
:
path
:
,
use
:
,
use
:
, use
:
kubectl
set
=
daemonset
causes:
.
name
=
Note:
either.
Clock skew
If
progress.
Clean up
See
See
13.2 - Perform a Rollback on a DaemonSet
DaemonSet
.
minikube
Your Kubernetes server must be at or la

ter than version 1.7.
.
DaemonSet
.
history
1               ...
2               ...
...
in
kubectl
history
=
1
app:
 Image:        ...
 Port:         ...
=
Note:
If
.
=
revision.
or
.
Note:
.revision
.
See
update
.
14 - Networking
14.1 - Adding entries to Pod /etc/hosts with HostAliases
pod/ngi

nx created
=
wide
kubectl
exec
hosts
localhost
hosts
file.
foo.local
,
bar.local
to
127.0.0.1
and
,
to
10.1.2.3
:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
Never
:
-
ip
:
hostnames
:
-
-
-
ip
:
hostnames
:
-
-
:
-
name
:
cat-hosts
image
:
command
:
- cat
args
:
-
=
wide
The
hosts

hosts
Caution:
14.2 - Validate IPv4/IPv6 dual-stack
A
cluster
.
Note:
:
=
The

re should be one IPv4 block and one IPv6 block allocated.
:
=
pod01
:
=
env:
            fieldRef:
kubectl
exec
set
kubectl
exec
and set the
to
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Use
kubectl
set to
and
:
v1
kind
:
Service
metadata
:
name
:
namespace
:
default
spec
:
clusterIP
:
:
-
:
- IPv4
:
ports
:
-
port
:
80
protocol
:
TCP
:
9376
selector
:
:
MyApp
:
None
type
:
ClusterIP
status
:
:
{}
IPv6
and set the
to
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
- IPv6
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Use
kubectl
set to
and
:
v1
kind
:
Service
metadata
:
labels
:
:
MyApp
name
:
spec
:
clusterIP
:
2001
:
-
2001
:
- IPv6
ipFamil

yPolicy
:
ports
:
-
port
:
80
protocol
:
TCP
:
80
selector
:
:
MyApp
:
None
type
:
ClusterIP
status
:
:
{}
in
array.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
Note:
The
field.
=
MyApp
(
S
)
AGE
=
MyApp
in
,
IPv6
type
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
:
MyApp
spec
:
:
:
- IPv6
type
:
selector
:
:
MyApp
ports
:
-
protocol
:
TCP
port
:
80
=
MyApp
(
S
)
AGE
15 - Extend kubectl with plugins
kubectl
kubectl
kubectl
kubectl
.
kubectl
kubectl-
PATH
.
using
Krew
Caution:

kubectl
PATH
PATH
kubectl-
PATH
kubectl-
that are
not
You can use
Krew
kubectl
.
kubectl
not
kubectl
There is no plugin installation or pre-loading required. Plugin executables receive
kubectl
binary.
. You must
PATH
.
if
[[
"
$1
"
==
"version"
]]
then
echo
"1.0.0"
exit
0
fi
if
[[
"
$1
"
==
"config"
]]
then
echo
"
"
exit
0
fi
echo
PATH
:
kubectl
command:
1.0.0
export
=
=
$0
-
).
.
Note:
not
See
If you run
would be
. Upon not finding that plugin, kubectl then treats the last dash-separated 

value as an argument (
arg1
.
Example:
echo
-e

kubectl
kubectl
-
command containing dashes in its commandline invocation by using underscores (
_
Example:
echo
-e
.
-
_
):
PATH
.
PATH
and
,
PATH
=

PATH
,
kubectl
kubectl
PATH
kubectl
command
PATH
to be
.
PATH
:
and
, the
kubectl
plugin mechanism will always choose the longest possible 

plugin name for a given user command. Some examples below, clarify this further:
kubectl
kubectl
command.

error: 2 plugin warnings were found
of the
See the
for
Krew
Krew
plugin
index
apt
or
yum
PATH
.
of a
.
Read about
Krew
16 - Manage HugePages
sizes.
API
Huge pages can be consumed via container level resource requirem

ents using the
, where
resources
and
must use
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
example
image
:
command
:
- sleep
- inf
:
-
mountPath
:
name
:
-
mountPath
:
name
:
resources
:
limits
:
:
100Mi
:
2Gi
memory
:
100Mi
requests
:
memory
:
100Mi
volumes
:
-
name
:
emptyDir
:
medium
:
-
name
:
emptyDir
:
medium
:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
example
image
:
command
:
- sleep
- inf
:
-
mountPath
:
name
:
hugepage
resources
:
limits
:
:
100Mi
memory
:
100Mi
requests
:
memory
:
100Mi
volumes
:
-
name
:
hugepage
emptyDir
:
medium
:
HugePages
shmget()
with
must
.
cpu
or
memory
using the
token.
17 - Schedule GPUs
stable
.
Note:
AMD
Intel
NVIDIA
or
.
cpu
or
memory
.
limits
limits
requests
, because
limits
and
requests
requests
limits
.
ap

iVersion
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
OnFailure
:
-
name
:
image
:
resources
:
limits
:
:
1
can use
=
=
If you're using AMD GPU devices, you can deploy
.
KV - Kaveri
RV - Raven
…
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
OnFailure
:
-
name
:
image
:
resources
:
limits
:
:
1
affinity
:
:
:
:
:
– key
:
operator
:
Exist
Home
Blog
Training
Partners
Community
CC BY 4.0

Client Authentication (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Set Up kubectl on macOS
Debug Pods
Get a Shell to a Runn

ing Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Contai

ner's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Configurat

ion (v1)
Reference
Field
string
kind
string
spec
status
Cluster
Field
server
string
string
bool
[]byte
proxy-url
string
cluster.
bool
config
clusters:
cluster:
...
In some enviro

nments, the user config may be exactly the same across many clusters
Field
cluster
Cluster
bool
Field
token
string
string
string
Feedback
Yes
No
.
or
.
Cluster
Home
Blog
Training
Partners
Community
CC BY 4.0

kubelet | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

lpha1)
Reference
kubelet
kubelet
Synopsis
Options
here
.)
0.0.0.0
or
::
--config
flag. See
*
--config
flag. See
here
.)
--config
flag. See
Use the
--config
flag. See
kubelet-config-fi

le
2m0s
--config
flag. See
5m0s
--config
flag. See
30s
--config
flag. See
.
and
cgroupfs
cgroupfs
,
systemd
--config
flag. See
''
true
--config
flag. See
Comma-separated list of DNS server IP add

ress. This value is used for containers DNS server in case of Pods with "dnsPolicy=ClusterFirst".
Note:
--config
flag. See
--config
flag. See
10Mi
10Mi
--config
flag. See
remote
docker
,
remote
,
.
--config
flag. See
true
--config
flag. See
100ms
--config
flag. See
none
none
,
static
--config
flag. See
--config
flag. See
10s
10s
, or
1m
--config
flag. See
for more in

formation.)
true
--config
flag. See
true
--config
flag. See
true
--config
flag. See
pods
none
,
pods
,
, and
and
none
here
--config
flag. See
--config
flag. See
--config
flag. See
--config
flag. See
--config
flag. See
5m0s
--config
flag. See
--config
flag. See
--eviction-soft-grace-period mapStringS

tring
--config
flag. See
false
When set to
true
here
--config
flag. See
mount
mount
true
--config
flag. See
A set of
key=value
InTreePlugin

AzureDiskUnregister=true|false (ALPHA - default=false)
--config
flag. See
20s
--config
flag. See
,
and
none
--config
flag. See
--healthz-bind-address string  

   Default:
127.0.0.1
0.0.0.0
or
::
--config
flag. See
0
--config
flag. See
-h, --help
20s
--config
flag. See
--config
flag. See
--config
flag. See
,
fwmark
--config
flag. See
fwmark
--config
flag. See
--config
flag. See
--config
flag. See
application/vnd

.kubernetes.protobuf
--config
flag. See
--config
flag. See
A set of
(e.g.
cpu
,
memory
and local
here
--config
flag. See
''
flag. Ex.
--config
flag. See
--config
flag. See
":0"
:
here
.)
here
.)
here
.)
here
.)
5s
'0'
--config
flag. See
--config
flag. See
text
text
,
json
.
,
,
,
--log-dir
,
,
,
,
,
,
,
.
Non-default choices are cur

rently alpha and subject to change without warning. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's
--config
flag. See
true
here
.)
true
iptables
--config
flag. See
--config
flag. See
--config
flag. See
default
--config
flag. See
--config
flag. See
or
or
None
'None'
,
'Static'
--config
flag. See
'300ms'
,
'10s'
or
'2h45m'
or
2m0s
'300ms'
,
'10s'
or
'2h45m'
--config
flag. See
'::'
','
,
) or be in the specifica

lly allowed set (
,
,
,
,
,
,
,
,
,
,
)
. If
-1
--config
flag. See
10s
--config
flag. See
here
.)
The
--config
flag. See
--config
flag. See
docker
--config
flag. See
-1
--config
flag. See
, so
0
--config
flag. See
--config
flag. See
--config
flag. See
--config
flag. See

-file
(e.g.
memory
--config
flag. See
0
--config
flag. See
true
true
is
false
). No-op if
is
false
--config
flag. See
0
--config
flag. See
and
--config
flag. See
,
and
. See
here
--config
flag. See
--config
flag. See
--config
flag. See
when the certificate

 expiration approaches. Requires the
--config
flag. See
--runonce
If
true
2m0s
pull
,
logs
,
exec
and
attach
--config
flag. See
true
aufs
--config
flag. See
If
true
here
.)
If
true
here
.)
here
.)
4h0m0s
0
5m
--config
flag. See
1m0s
--config
flag. See
'/'
--config
flag. See
A set of
(e.g.
cpu
and
memory
here
--config
flag. See
''
flag. Ex.
--config
flag. See
--tls-cert-file s

tring
and
--config
flag. See
,
,
,
--config
flag. See
--config
flag. See
'none'
'none'
,
,
,
--config
flag. See
container
,
'pod'
pattern=N
--config
flag. See
1m0s
0
--config
flag. See
Feedback
Yes
No
Thanks for th

e feedback. If you have a specific, answerable question about how to use Kubernetes, ask it on
.
or
.
Last modified October 20, 2022 at 11:59 AM PST:
see https://github.com/kubernetes/kubernetes/pull/3015 This whole functionality got removed from kubelet. So please remove the entire statement (a375935e53)
Synopsis
Options
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
1 - Reference Documentation Quickstart
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
website
remote on
GitHub (
main
website
fork.
cd
.
Note:
see the
.
The
The
kubectl
The
under
/tmp
and
GOPATH
K8S_ROOT
)
1.17
field.
The
from
. The
variable
The
default is
.
Updates
kubectl
kubectl
the secti

ons in the
kubectl
to
.
repos
:
-
name
:
community
remote
:
branch
:
master
files
:
-
src
:
dst
:
-
src
:
dst
:
the
.
Open
.
Note:
The
In
,
files
src
and
dst
fields.
The
src
dst
repos
:
-
name
:
remote
:
files
:
-
src
:
dst
:
...
src
dst
.
files
:
-
src
:
dst
:
cd
The
true
.
:
cd
git status
static/docs/reference/generated/kubectl/s

tylesheet.css
Run
git add
and
git commit
.
2 - Contributing to the Upstream Kubernetes Code
project.
kubeadm
,
, and
.
API or the
kube-*
Git
Golang
Docker
etcd
make
Your
GOPATH
etcd
PATH
and
.
kube-*
such as
,
.
mkdir
$GOPATH
/src
cd
$GOPATH
/src
For example, if you followed the precedi

ng step to get the repository, your
.
.
kube-*
Note:
cd
types.go
git status
types.go
...
Run
git add
and
git commit
Go to
Run
git status
...
.
Run
git add
and
git commit
types.go
to the
PR 57758
Note:
staging
directory
in the
staging
directory
README
files in
.
In the preceding section, you edited a file in t

he master branch and then ran scripts
types.go
types.go
.
Note:
types.go
types.go
in the
The
.
3 - Generating Reference Documentation for the Kubernetes API
using the
.
OpenAPI
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
GOPATH
.
mkdir -p
$HOME
export
GOPATH
=
$HOME
Get a local clone of the foll

owing repositories:
$GOPATH
$GOPATH
.
.
.
.
Set
K8S_ROOT
to
.
Set
to
.
Set
to 1.17.0.
export
=
${
GOPATH
}
export
K8S_ROOT
=
${
GOPATH
}
export
=
1.17.0
The
.
In the
cd
The
copyapi
.
:
cd
[
-e
]
echo
||
echo
[
-e
]
echo
||
echo
, and
cd
git status
with

 the new
Open
---
---

Open
Verify the
.
cd
1
In
run
git add
and
git commit
to the
4 - Generating Reference Documentation for kubectl Commands
kubectl
Note:
like
and
.
kubectl
.
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
GOPATH
.
mkdir -p
$HOME
export
GOPATH
=
$HOME
$GOPATH
$GOPATH
.
rm -rf
$GOPATH
kubectl
and
kustomize
.
.

.
cd
kubectl
.
.
PR 56673
1.26
.
Note:
Go to
Set
K8S_ROOT
to
.
Set
to
.
Set
to 1.25.
export
=
$GOPATH
export
K8S_ROOT
=
$GOPATH
export
=
1.25
The
.
In the
cd
v1.25
cd
, run the
copycli
root
:
cd
The
copycli
.
[
-e
]
echo
||
echo
[
-e
]
echo
||
echo
Verify that all generated files have been c

opied to your
:
cd
git status
.
cd
1
View the
.
Run
git add
and
git commit
.
5 - Generating Reference Pages for Kubernetes Components and Tools
Follow the
6 -
Python
v3.7.x
Git
Golang
Pip
PyYAML
v5.1.2
make
Docker
kubectl
Your
PATH
Go
binary and
python
.
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Assign Extended Resources to a Container | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
I

Tasks
minikube
.
.
where
foo
is a
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
:
3
limits
:
:
3
Limits
:
:
3
Requests
:
:
3
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
:
2
limits
:
:
2
kubectl apply -f https://k8s.io

/examples/pods/resource/extended-resource-pod-2.yaml
...
Events:
  ...
Clean up
Feedback
Yes
No
.
or
.
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

1alpha1)
Tasks
Run Jobs
Run Jobs
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

LocalSubjectAccessReview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

(v1)
Reference
kind
metadata
(
)
spec
(
status
(
)
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
Feedback
Yes
No
.
or
.
create
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses tradema

rks. For a list of trademarks of The Linux Foundation, please see our

kube-scheduler | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Synopsis
See
Options
--bind-address string  

   Default: 0.0.0.0
JobMutableNodeSchedulingDirectives=

true|false (BETA - default=true)
-h, --help
--kube-api-qps float   

  Default: 50
Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. 

WARNING: generally do not depend on authorization being already done for incoming requests.
-v, --v int
If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. Th

e fix may need to happen elsewhere in the Kubernetes project.
Feedback
Yes
No
.
or
.
Synopsis
Options
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes Deprecation Policy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

Reference
fall
Example
Track
v1
v1beta1
v1alpha1
includes:
API group.
Note:
might have to a

dd an equivalent field or represent it as an annotation.
may not
may not
,
Note:
There are no current plans for a major version revision of Kubernetes that removes GA APIs.
Note:
Until
#52185
is
objects.
Release
Notes
X
v1alpha1
v1alpha1
X+1
v1alpha2
v1alpha2
X+2
v1beta1
v1beta1
X+3
v1beta1
X+4
v1beta2
X+5
v1beta2
X+6
v1
X+7
v1
X+8
v1
X+9
v1
X+10
v1
X+11
v1
X+12
v1
X+13
v2
X+14
v2
X+15
v2
announce
the
exist and function in relea

ses up to and including X+8. Only in release X+9,
Returns a
Warning
Adds a
Sets an
1
in the
group
,
version
,
resource
,
to the
{
=
"
1.22
"}
*
on
(
group
,
version
,
resource
,
)
()
must exist and function until API v1 is removed. Unlike whole resources, the
follows:
when used.
are not intended to be long-term APIs. As such

, they are expected to be deprecated
When removing a feature gate for a GA feature also requires considerable time, calls to
/metrics
process.
ALPHA
and
STABLE
);
is equal to
hidden
metric.
Unlike
no longer
be
(
escape h

atch to properly migrate off of a deprecated metric, if they were not
Feedback
Yes
No
.
or
.
Last modified September 07, 2022 at 12:41 PM PST:
Clarify API version stability and guidelines (d308cbb35a)
Home
Blog
Training
Partners
Community
CC BY 4.0

Use Port Forwarding to Access Applications in a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API Aggregation Lay

er
Tasks
Debug Pods
Determine t

he Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

 the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
minikube
.
Install
.
=
'{{(index (index .spec.containers 0).ports

 0).containerPort}}{{"\n"}}'
27017
or
or
or
Note:
28015
ping
command:
{ ok: 1 }
kubectl
kubectl
The
kubectl
Note:
.
.
Feedback
Yes
No
.
or
.
Last modified April 27, 2022 at 10:20 AM PST:
Tweak indentation on portforward page (7359d60de8)
kubectl
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
1 - Understanding Kubernetes Objects
.yaml
format.
cluster's
.
kubectl
.
spec
status
.
spec
its
.
The
status
supplied.
spec
.
kubectl
kubectl
kubectl
request.
.yaml
file that

 shows the required fields and object spec for a Kubernetes Deployment:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.yaml
command
in the
kubectl
.yaml
In the
.yaml
kind
metadata
name
string,
UID
namespace
spec
spec
spec
field
.spec
spec
field
.spec
Within the
.spec
template
.status
.status
Pods
objects.
kubectl
and
.
2 - Kubernetes Object Management
The
kubectl
for
Warning:
1+
Lowest
1
Moderate
1+
Highest
the
kubectl
Examples
Commands do not provide an audit trail associated

 with changes.
See the
Warning:
replace
Examples
kubectl
Note:
patch
replace
Examples
configs
diff

3 - Object Names and IDs
Name
UID
namespace
.
labels
and
.
Names
.
Note:
RFC 1123
.
RFC 1123
.
RFC 1035
.
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Note:
UIDs
Read about
labels
and
See the
4 - Labels and Selectors
Labels
:
{
"labels"
: {
"key1"
:
"value1"
,
"key2"
:
"value2"
}
}
.
Service deployments and batc

h processing pipelines are often multi-dimensional entities (e.g., multiple partitions or deployments, multiple release tracks, multiple tiers, multiple micro-services per tier). Management often requires cross-cutting operations, which breaks encapsulation of strictly hierarchical representations, especially rigid hierarchies determined by the infrastructure rather than by users.
,
,
,
,
,
,
,
Labels
/
-
_
), dots (
.
.
/
).
,
,
,
kubectl
The
and
k8s.io/
reserved
),
-
_
), dots (
.
and
app: nginx
:
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
Unlike
Via a
and
set-based
.
AND
(
them.
Note:
Caution:
OR
(
||
Equality-
or
=
,
==
,
!=
equality
.
tier
front

end
tier
key.
excluding
frontend
the label "
".
:
v1
kind
:
Pod
metadata
:
name
:
cuda-test
spec
:
:
-
name
:
cuda-test
image
:
resources
:
limits
:
:
1
:
:
Set-based
Set-based
in
,
notin
and
exists
partition
or
qa
.
tier
frontend
and
backend
tier
key.
partition
partition
AND
partition
qa
.
The
set-based
!=
and
notin
.
Set-based
.
API
set-based
apiserver
with
kubectl
and using
=
=
frontend
or using
set-based
set-based
OR
exists
operator:
services
and
,
pods
.
service
json
or
yaml
:
{
:
"redis"
,
}
or
selector
:
component
:
redis
json
or
yaml
or
.
Job
,
,
, and
DaemonSet
,
support
set-based
selector
:
:
component
:
redis
:
- {
:
[cache]}
- {
:
[dev]}
m

atchLabels
is a map of
in the
, whose
key
operator
values
and
Find
See
5 - Namespaces
.
).
labels
Note:
not
using the
default
default
Lease
all
.
Note:
kube-
flag.
=
=
=
context.
kubectl config set-c

ontext --current --namespace
=
Service
,
DNS entry
.
.
Warning:
domains
will
as
admission
webhooks
,
public
TLDs
.
nodes
and
=
true
=
false
label
.
.
6 - Annotations
:
{
: {
"key1"
:
"value1"
,
"key2"
:
"value2"
}
}
Note:
Directives from the end-

user to the implementations to modify behavior or
/
-
_
), dots (
.
.
/
).
,
,
,
kubectl
The
and
k8s.io/
:
:
v1
kind
:
Pod
metadata
:
name
:
:
:
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.
7 - Field Selectors
let you
This
kubectl
field is
Running
:
=
Running
Note:
filters
kubectl
queries
and
and
=
baz
=
,
==
, and
!=
=
and
==
kubectl
default
=
default
As with
label
kubectl
Running
and the
Always
:
=
=
=
Always
kubectl
default
=
default
8 - Finalizers
When you tell Kubernetes to delete an object that has finaliz

ers specified for
,
202
your own.
the
field
Returns a
202
field set
using the
Like
labels
,
Job
Note:
to resources for 

a reason, so forcefully removing them can lead to issues in
Read
9 - Owners and Dependents
owners
creates
field that
field to
true
if a
Note:
must
,
and an
.
has the
Kubernetes a

lso adds finalizers to an owner resource when you use either
.
orphan
object.
.
.
.
10 - Recommended Labels
Note:
. Labels
Labels
Key
Example
Type
mysql
string
string
SemVer 1.0
5.7.21
string
database
string
wordpress
string
helm
string
object:
:
apps/v1
kind
:
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
database
:
wordpress
:
helm
of
wordpress
Examples
and
Service
The
:
apps/v1
kind
:
metadata
:
labels
:
:
myservice
:
...
The
Service
:
v1
kind
:
Service
metadata
:
labels
:

/name
:
myservice
:
...
:
apps/v1
kind
:
metadata
:
labels
:
:
wordpress
:
:
"4.9.4"
:
helm
:
server
:
wordpress
...
The
Service
:
v1
kind
:
Service
metadata
:
labels
:
:
wordpress
:
:
"4.9.4"
:
helm
:
server
:
wordpress
...
:
apps/v1
kind
:
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
helm
:
database
:
wordpress
...
The
Service
:
v1
kind
:
Service
metadata
:
labels
:
:
mysql
:
:
"5.7.21"
:
helm
:
database
:
wordpress
...
and
Service
Home
Blog
Training
Partners
Community
CC BY 4.0

Other Tools | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl on

 macOS
Debug Pods
Tro

ubleshooting Clusters
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a Container's Access to R

esources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-pro

xy Configuration (v1alpha1)
Reference
crictl
crictl
CRI
Dashboard
Dashboard
Helm
Helm
.
Kompose
Kompose
yaml
files or
Kui
Kui
kubectl
kubectl
Type in
kubectl
kubectl
itself
Query a
Job
Minikube
minikube
Feedback
Yes
No
.
or
.
Updates page weights

 in reference docs section (98f310ab58)
crictl
Dashboard
Helm
Kompose
Kui
Minikube
Home
Blog
Training
Partners
Community
CC BY 4.0

Proxies in Kubernetes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Concepts
Proxies
The
:
The
:
The
kube proxy
:
Feedback
Yes
No
.
or
.
Last modified October 24, 2022 at 1:43 PM PST:
KubeCon Docs Sprint: Update page weights for content/en/docs/concepts/cluster-administration. (ac5e7c0bd0)
Proxies
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linu

x Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

ClusterCIDR v1alpha1 | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kind
metadata
(
)
spec
(
)
ipv4
(string)
ipv6
(string)
([]NodeSelecto

rTerm), required
([]
)
([]
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): inte

ger
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Set up an Extension API Server | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install and Set Up kubectl 

on Linux
Debug Pods

Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespace Le

vel
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Encryption Configuration (v1)
Tasks
minikube
.
You must
Create a Kubernetes deployment for the extension api-server and make sure you are loading the secret as a volume. It should contain a reference to a working image of your extension api-server. The deployment should also be in your n

amespace.
.
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator p

attern
Tasks
Debug Pods
Determine the Reason for

 Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Cluster 

Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
k

ube-apiserver Configuration (v1alpha1)
Tasks
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Encrypting Secret Data at Rest | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Tasks
minikube
.
The
.
Caution:
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
aesgcm
:
keys
:
-
name
:
key1
secret
:
-
name
:
key2
secret
:
-
aescbc
:
keys
:
-
name
:
key1
secret
:
-
name
:
key2
secret
:
-
secretbox
:
keys
:
-
name
:
key1
secret
:
Each
resources
resource
or
A

ny custom resources that existed in etcd prior to that version and configuration will be unencrypted
See the
section.
The
providers
identity
or
aescbc
.
Caution:
Providers
Name
Strength
Speed
Key Length
identity
None
N/A
N/A
N/A
secretbox
Strong
Faster
32-byte
aesgcm
Fastest
aescbc
PKCS#7
padding
Weak
Fast
32-byte
kms
PKCS#7
Strongest
Fast
32-bytes
.
Caution:
kms
identity
keys.
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
aescbc
:
keys
:
-
name
:
key1
secret
:
-
identity
:
{}
To create a new Secret, perform 

the following steps:
head -c
32
secret
struct.
Set the
flag on the
to point to
:
v1
kind
:
Pod
metadata
:
:
:
:
6443
:
null
labels
:
component
:
tier
:
name
:
namespace
:
spec
:
:
-
command
:
...
:
...
-
name
:
enc
mountPath
:
readonly
:
true
...
volumes
:
...
-
name
:
enc
hostPath
:
path
:
type
:
...
Caution:
etcdctl
secret1
in the
default
=
mykey
=
mydata
Using the
etcdctl
where
[...]
=
3
etcdctl
\
--cacert
=
\
--cert
=
\
--key
=
\
00000000
2f
72
65
67
69
73
74
72
79
2f
73
65
63
72
65
74
|
|
00000010
73
2f
64
65
66
61
75
6c
74
2f
73
65
63
72
65
74
|
|
00000020
31
0a
6b
38
73
3a
65
6e
63
3a
61
65
73
63
62
63
|
|
00000030
3a
76
31
3a
6b
65
79
31
3a
c7
6c
e7
d3
09
bc
06
|
|
00000040
25
51
91
e4
e0
6c
e5
b1
4d
7a
8b
3d
b9
c2
7c
6e
|
|
00000050
b4
79
df
05
28
ae
0d
8e
5f
35
13
2c
c0
18
99
3e
|
|
[...]
00000110
23
3a
0d
fc
28
ca
48
2d
6b
2d
46
cc
72
0b
70
4c
|
|
00000120
a5
fc
35
43
12
4e
60
ef
bf
6f
fe
cf
df
0b
ad
1f
|
|
00000130
82
c4
88
53
02
da
3e
66
ff
0a
|
|
0000013a
the
aescbc
etcd
key1
is used in
etcd
and in
.
mydata
Since Secrets 

are encrypted on write, performing an update on a Secret will encrypt that content.
Note:
keys
Run
identity
:
kind
:
resources
:
-
resources
:
- secrets
providers
:
-
identity
:
{}
-
aescbc
:
keys
:
-
name
:
key1
secret
:
.
Feedback
Yes
No
.
or
.
Last modified December 18, 2022 at 9:48 PM PST:
Fix punctuations in headings of encrypt-data.md (d8e90b4c6b)
Providers
Home
Blog
Training
Partners
Community
CC BY 4.0

Change the Reclaim Policy of a PersistentVolume | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install

Tasks
minikube
.
where
Note:
double
quote any JSONPath templ

ate that contains spaces (not single
"{\"
spec\
":{\"
":\"
Retain\
"}}"
Retain
.
.
.
field
Feedback
Yes
No
.
or
.
Last modified March 14, 2022 at 8:22 PM PST:
Reformat the indentation of lists and sample outputs (e9a8cc6eda)
Home
Blog
Training
Partners
Community
CC BY 4.0

Status | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

ha1)
Reference
Status
Status
(string)
code
(int32)
details
(string)
Examples:
(string)
(string)
(string)
(string)
(string)
The name attribute of the

 resource associated with the status StatusReason (when there is a single name which can be described).
(int32)
(string)
kind
(string)
message
(string)
metadata
(
ListMeta
)
reason
(string)
status
(string)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-apiserver Admission (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extending Kubernet

es
Tasks
Migrate from PodSecurityPolicy to the Built-In 

PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfile, 

Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Event Ra

te Limit Configuration (v1alpha1)
Reference
Field
string
kind
string
request
response
Field
uid
kind
resource
string
and
,
with
and
m

eta/v1.GroupVersionResource
and
,
with
and
string
name
string
namespace
string
operation
Operation
userInfo
object
oldObject
dryRun
bool
options
e.g.
or
.
Field
uid
allowed
bool
status
patch
[]byte
patchType
PatchType
warnings
[]string
Limit warnings to 120 chara

cters if possible.
Operation
(Alias of
string
)
PatchType
(Alias of
string
)
Feedback
Yes
No
.
or
.
Last modified December 17, 2022 at 4:27 PM PST:
Add admission.k8s.io/v1 API and fix references to it (3362aa9701)
Operation
PatchType
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Network Pl

ha1)
Note:
.
CNCF
page
.
.
git
and
GitHub
.
.
and
.
U[ ] -.-
end
T[ ] -.-
end
S[ ] -.-
end
enabled
Sign up
and
Review

Open PR
Some tasks require more trust and more access in the Kubernetes organization.
See
S[ ] -.-
end
T[ ] -.-
end
enabled
Read the
to
Check
from other
content
and
and
.
Next steps
Learn to
Document
.
SIG Docs
.
.
SIG Docs
Join
#sig-docs
Join the
,
Join the
#sig-docs
#sig-docs
Visit the
Read the
and
.
Submit a
.
Feedback
Yes
No
.
or
.
Last modified June 18, 2022 at 3:30 PM PST:
Fix links in some pages (9b5e2aec1b)
Next steps
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has register

ed trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Enforce Pod Security Standards with Namespace Labels | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install To

Tasks
,
baseline
and
.
.
Ensure the
baseline
that:
Blocks
baseline
meet the
baseline
and
:
v1
kind
:
Namespace
metadata
:
name
:
labels
:
:
baseline
:
v1.26
:
:
v1.26
:
:
v1.26
Note:
When an
enforce
--dry-run
dry run
=
\
=
baseline
baseline
:
\
=
baseline
\
=
baseline

=
policy to
\
=
\
=
v1.26
Feedback
Yes
No
.
or
.
Last modified June 17, 2022 at 4:17 PM PST:
Replace skew shortcode parameters (e7caadc564)
baseline
Home
Blog
Training
Partners
Community
CC BY 4.0

ComponentStatus | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
: v1
kind
metadata
(
)
type
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
If you plan to report an issue with this page, mention that the page is auto-generated in your issue description. The fix may need to happen elsewher

e in the Kubernetes project.
Feedback
Yes
No
.
or
.
get
list
Home
Blog
Training
Partners
Community
CC BY 4.0

Enforce Pod Security Standards by Configuring the Built-in Admission Controller | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
.
.
.
Ensure the
:
kind
:
plugins
:
-
name
:
:
:
kind
:
#
#
defaults
:
enforce
:
:
"latest"
audit
:
:
"latest"
warn
:
:
"latest"
:
usernames
:
[]
:
[]
:
[]
Note:
v1beta1
.
v1alpha1
.
Feedback
Yes
No
.
or
.
Last modified September 01, 2022 at 8:35 AM PST:
Update pod security configuration docs for v1 (7eba57d4de)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Service
Endpoints
Ingress
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Insta

lpha1)
Reference
Example:
).
One of the
.
Example:
).
Note:
Example:
).
.
One of the
.
Example:
).
One of the
.
Example:
).
One of the
.
Example:
).
One of the
.
Example:
).
the Git
One of the
.
Example:
"true"
"false"

Example:
Example:
Example:
)
selector
.
Example:
.
instead.
instead.
Example:
The
Note:
onstart
true
Example:
flag to the
kubelet
.
Example:
--record
to a
kubectl
Example:
Example:
true
secrets
field.
Example:
you can add labels to particular worker nodes to exclude them from the 

list of backend servers.
Example:
int32
type.
Example:
"true"
"false"
Note:
Note:
bandwidth
) and
).
Example:
Quantity
.
10M
Note:
bandwidth
) and
).
Example:
Quantity
.
10M
Note:
.
Example:
.
).
See
.
Note:
.
See
.
Note:
Starting in v1.17, t

his label is deprecated in favor of
.
Example:
See
in the
Example:
The
Example:
See
.
Example:
kubelet
.
.
If
adding the labels manually (o

r adding support for
). With
Example:
Example :
by the
.
Example:
Example:
Example:
with
Services
name
of the
Example:
name
of the
Example:
unique ID
of the
Example:
.
Example:
Example:
"true"
Example:
Th

e kube-proxy has this label for custom proxy, which delegates service control to custom proxy.
Example:
.
Note:
Example:
"true"
Note:
.
Example:
"true"
Example:
--node-ip
Example:
.
Example:
or
without
-c
or
Example:
The
Endpoints
Service
Example:
.
You should
not
Note:
Example:
scheduler.alpha.kuberne

tes.io/preferAvoidPods (deprecated)
Use
instead.
Example:
Example:
Ready
being
Unknown
.
Example:
Example:
and
Example:
,
,
and
Example:
Example:
Example:
Caution:
Refer to
Example:
Example:
.
Example:
Value
must
be one of
,
baseline
, or
the
enforce
label
prohibits
the creation of any Pod in the labeled Namespace whic

h does not meet
See
Example:
Value
must
be
latest
.
See
Example:
Value
must
be one of
,
baseline
, or
the
audit
See
Example:
Value
must
be
latest
.
See
Example:
Value
must
be one of
,
baseline
, or
the
warn
See
Example:
Value
must
be
latest
.
See
Example:
.
The
.spec
When you
,
takes
you through the steps you follow to apply a seccomp profile t

o a Pod or to one of
.spec
.
Example:
true
or
false
.
is being
Refer to
and the
Example:
Note:
.
page.
kubeadm
Example:
init
/
join
Example:
Example:
Example:
Example:
Example:
; kubeadm
Feedback
Yes
No
.
or
.
Last modified November 29, 2022 at 10:50 PM PST:
adding annotation for kube-aggregator automanaged in labels-md (0f6cdedf54)
Labels, annotations and taints used on API ob

jects
kubeadm
Home
Blog
Training
Partners
Community
CC BY 4.0

Instrumentation | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
1 - Kubernetes Component SLI Metrics
# HELP kubernetes_healthcheck [ALPHA] This metric records the result of a single healthcheck.
2 - Node metrics data
The
kubelet
.
Kubernetes API s

erver.
minikube
:
curl
:
Note:
0.6.x,
.
cAdvisor
in your
discuss
3 - Kubernetes Metrics Reference
Name
Type
Help
Labels
STABLE
Histogram
name
operation
rejected
type
None
STABLE
Histogram
operation
rejected
type
None
STABLE
Histogram
name
operation
rejected
type
None
STABLE
Gauge
None
STABLE
Gauge
component
group
resource
scope
verb
version
None
STABLE
Histogram
component
dry_run
group
resource
scope
verb
version
None
STABLE
Counter
code
component
dry_run
group
resource
scope
verb
version
None
STABLE
Gauge
group
resource
version
None
STABLE
Histogram
component
group
resource
scope
verb
version
None
STABLE
Gauge
resource
None
STABLE
Counter
zone
None
STABLE
Histogram
Latency for running all plugins 

of a specific extension point.
profile
status
None
STABLE
Gauge
queue
None
STABLE
Histogram
None
None
STABLE
Histogram
attempts
None
STABLE
Counter
None
None
STABLE
Histogram
None
None
STABLE
Counter
event
queue
None
STABLE
Counter
profile
result
None
STABLE
Histogram
profile
result
None
Name
Type
Help
Labels
ALPHA
Counter
reason
None
ALPHA
Gauge
reason
None
ALPHA
Custom
name
None
ALPHA
Counter
name
reason
None
ALPHA
Counter
crd
reason
None
ALPHA
Counter
crd
group
reason
version
None
ALPHA
Summary
operation
rejected
type
None
ALPHA
Counter
name
type
None
ALPHA
Counter
name
operation
type
None
ALPHA
Counter
code
name
operation
rejected
type
None
ALPHA
Counter
plugin
None
ALPHA
Coun

ter
None
None
ALPHA
Counter
level
None
ALPHA
Counter
None
None
ALPHA
Counter
index
None
ALPHA
Counter
None
ALPHA
Counter
index
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Histogram
None
None
ALPHA
Histogram
crd_name
succeeded
None
ALPHA
Gauge
None
ALPHA
Histogram
code
None
ALPHA
Counter
code
None
ALPHA
Histogram
code
None
ALPHA
Counter
code
None
ALPHA
Histogram
protocol
transport
None
ALPHA
Counter
protocol
stage
transport
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
ALPHA
Gauge
None
ALPHA
Gauge
None
ALPHA
Gauge
None
ALPHA
Gauge
None
ALPHA
Counter
None
ALPHA
Counter
success
None
apiserver_flowcontrol_lat

est_s
ALPHA
Gauge
None
ALPHA
Gauge
bound
None
ALPHA
Gauge
bound
None
ALPHA
phase
None
ALPHA
ALPHA
phase
None
ALPHA
Counter
reason
None
ALPHA
Gauge
None
ALPHA
Gauge
None
ALPHA
Counter
None
ALPHA
Histogram
type
None
ALPHA
Histogram
None
ALPHA
Histogram
execute
None
ALPHA
Histogram
None
ALPHA
Histogram
None
ALPHA
Counter
resource
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Counter
group
resource
scope
verb
version
None
ALPHA
Histogram
resource

verb
None
ALPHA
Histogram
filter
None
ALPHA
Counter
source
status
None
ALPHA
Histogram
component
group
resource
scope
verb
version
None
ALPHA
Counter
code
component
group
resource
scope
verb
version
None
ALPHA
Histogram
code_path
None
ALPHA
Counter
resource
verb
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
None
ALPHA
Gauge
endpoint
None
ALPHA
Counter
None
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Histogram
None
ALPHA
Counter
status
None
ALPHA
Counter
resource
None
ALPHA
Counter
None
None
ALPHA
Histogram
params
policy
state
None
ALPHA
Counter
params
policy
state
None
ALPHA
Counter
state
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Histogram
group
kind
version

None
ALPHA
Counter
group
kind
version
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Custom
state
None
ALPHA
Counter
username
None
ALPHA
Counter
result
None
ALPHA
Histogram
result
None
ALPHA
Gauge
status
None
ALPHA
Counter
status
None
ALPHA
Histogram
status
None
ALPHA
Counter
status
None
ALPHA
Histogram
request
None
ALPHA
Counter
request
None
ALPHA
Counter
None
ALPHA
Histogram
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Histogram
request
source
None
ALPHA
Counter
request
source
None
ALPHA
Histogram
region
request
version
zone
None
ALPHA
Counter
region
request
version
zone
None
ALPHA
Histogram
request
None
ALPHA
Counter
request
None
ALPHA
Histogram
operation
None
ALPHA
Counter
operation
None
ALPHA
Custom
hostname
version
build
None
ALPHA
Custom
container
pod
namespace
None
ALPHA
Custom
container
pod
namespace
None
ALPHA
Custom
container
pod
namespace
None
ALPHA
Histogram
Time between when a cronjob is scheduled to be run, and when the correspon

ding job is created
None
None
ALPHA
Histogram
migrated
None
ALPHA
Counter
operation
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
topology
None
ALPHA
Gauge
None
None
ALPHA
Counter
result
None
ALPHA
Histogram
None
None
ALPHA
Counter
operation
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Gauge
resource
None
ALPHA
Histogram
None
None
ALPHA
Histogram
operation
type
None
ALPHA
Gauge
None
ALPHA
Histogram
enabled
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Counter
reason
result
None
ALPHA
Counter
result
None
ALPHA
Histogram
action
result
None
ALPHA
Counter
action
resul

t
None
ALPHA
Counter
action
None
ALPHA
Counter
event
None
ALPHA
Gauge
cidr
None
ALPHA
Counter
cidr
scope
None
ALPHA
Counter
cidr
scope
None
ALPHA
Gauge
cidr
None
ALPHA
Counter
None
None
ALPHA
Counter
usage
None
ALPHA
Custom
namespace
pod
node
scheduler
priority
resource
unit
None
ALPHA
Custom
namespace
pod
node
scheduler
priority
resource
unit
None
ALPHA
Counter
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
ALPHA
Custom
uid
namespace
pod
container
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Histogram
Time between when stats are co

llected, and when pod is evicted based on those stats by eviction signal
None
ALPHA
Counter
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
method
path
None
ALPHA
Histogram
method
path
None
ALPHA
Counter
method
path
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
node
None
ALPHA
Counter
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
ALPHA
Histogram
Duration in seconds of the run_podsandbox operations. Broken down by RuntimeClass.Handl

er.
None
ALPHA
Counter
None
ALPHA
Gauge
None
ALPHA
Gauge
None
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Counter
None
None
ALPHA
Counter
code
None
ALPHA
Counter
None
ALPHA
Counter
code
None
ALPHA
Counter
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Histogram
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Custom
namespace
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Gauge
table
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
kubeproxy_

sync_proxy_rules_service_changes_pending
ALPHA
Gauge
None
None
ALPHA
Counter
None
None
ALPHA
Gauge
compiler
major
minor
platform
None
ALPHA
Gauge
name
stage
None
ALPHA
Gauge
name
type
None
ALPHA
Counter
name
status
type
None
ALPHA
Gauge
name
None
ALPHA
Histogram
operation
None
ALPHA
Counter
zone
None
ALPHA
Gauge
zone
None
ALPHA
Gauge
zone
None
ALPHA
Gauge
zone
None
ALPHA
Custom
None
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Gauge
None
ALPHA
Histogram
None
ALPHA
Counter
None
ALPHA
Counter
None
ALPHA
Gauge
None
ALPHA
Custom
None
None
ALPHA
Gauge
feature
None
ALPHA
Custom
state
None
ALPHA
Custom
pod
namespace
None
ALPHA
Custom
pod
namespace
None
ALPHA
Counter
fatal
resource
None
ALPHA
Counter
decision
mode
resource
None
ALPHA
Counter
Number of exempt requests, not counting ignored or out of

 scope requests.
resource
None
ALPHA
Histogram
container
namespace
pod
None
ALPHA
Counter
container
namespace
pod
pod_uid
result
None
ALPHA
Custom
None
ALPHA
Custom
namespace
None
ALPHA
Custom
None
ALPHA
Custom
None
ALPHA
Custom
namespace
None
ALPHA
Histogram
None
None
ALPHA
Counter
code
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Histogram
host
verb
None
ALPHA
Histogram
host
verb
None
ALPHA
Histogram
host
verb
None
ALPHA
Counter
code
host
method
None
ALPHA
Histogram
host
verb
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Histogram
code
None
ALPHA
Counter
code
None
ALPHA
Gauge
manager
name
None
ALPHA
Histogram
profile
result
None
ALPHA
Gauge
operation
None

ALPHA
Histogram
result
None
ALPHA
Histogram
plugin
status
None
ALPHA
Gauge
type
None
ALPHA
Gauge
work
None
ALPHA
Histogram
None
None
ALPHA
Gauge
plugin
profile
None
ALPHA
Counter
operation
None
ALPHA
Counter
operation
None
ALPHA
Custom
None
None
ALPHA
Histogram
None
None
ALPHA
Histogram
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Counter
None
None
ALPHA
Custom
node
None
ALPHA
Histogram
migrated
status
None
ALPHA
Histogram
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
None
None
ALPHA
Gauge
Number of errors when a Pod uses a volume that is already mounted with a different SELinux context than th

e Pod needs. They are not errors yet, but they will become real errors when SELinuxMountReadWriteOncePod feature is expanded to all volume access modes.
None
None
ALPHA
Gauge
None
None
ALPHA
Custom
state
None
ALPHA
Counter
None
ALPHA
Histogram
None
ALPHA
Gauge
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
resource
None
ALPHA
Counter
name
None
ALPHA
Gauge
name
None
ALPHA
Gauge
name
None
ALPHA
Histogram
name
None
ALPHA
Counter
name
None
ALPHA
Gauge
name
None
ALPHA
Histogram
name
None
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm upgrade phase | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Kubeadm
phases.
apply
phase
preflight
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help

.
Synopsis
Options
--dry-run
-h, --help
or
Feedback
Yes
No
.
or
.
Last modified October 14, 2020 at 11:32 AM PST:
Add links to dangling kubeadm reference pages (0c3f9a53e0)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
crictl
crictl
CRI
Dashboard
Dashboard
Helm
Helm
.
Kompose
Kompose
yaml
files or
Kui
Kui
kubectl
kubectl
Type in
kubectl
kubectl
itself
Query a
Job
Minikube
minikube
1 - Mapping from dockercli to crictl
Note:
Note:
crictl
CRI
crictl
cri-tools
docker
crictl
.
docker
crictl
docker
Note:
crictl
docker
docker cli
crictl
attach
attach
,
exec
exec
,
--user
,
images
images
info
info
inspect
inspect
,
inspecti
logs
logs
--details
ps
ps

stats
stats
version
version
docker cli
crictl
create
create
kill
stop
--signal
pull
pull
,
rm
rm
rmi
rmi
run
run
start
start
stop
stop
update
update
--restart
,
crictl
inspectp
pods
List pods
runp
rmp
stopp
Home
Blog
Training
Partners
Community
CC BY 4.0

Reviewing for approvers and reviewers | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install Tool

s
Debug Pods
Debug Init C

ontainers
Auditing
Run Jobs
TLS
Tutorials
Security

Standards at the Namespace Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

on (v1alpha1)
SIG Docs
Reviewers
and
Approvers
.
Using the
/assign
Note:
Look at the
reviewers
Content
and
Style
/approve
or
/lgtm
true:
Note:
Prow
is
the Kubernetes-based CI/CD system that runs jobs against pull requests (PRs). Pr

ow
format.
/lgtm
/approve
Approvers
/assign
Anyone
/close
/hold
Anyone
Adds the
Anyone
label.
.
filter
Add the
and
labels.
Label
help
or
#sig-docs
on Slack or
the
.
)
or
)
)
)
Label
command.
label.

kubectl
We expect

Feedback
Yes
No
.
or
.
Last modified September 20, 2022 at 12:21 AM PST:
Update some contributor page (36899cd93c)
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Multiple Schedulers | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 Linux
Debug Pods
Ge

t a Shell to a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Namespace Leve

l
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver En

cryption Configuration (v1)
Tasks
here
.
minikube
.
Clone the
cd
make
FROM
busybox
ADD
.
.
manages a
:
:
v1
kind
:
metadata
:
name
:
namespace
:
kube-s

ystem
---
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
---
:
kind
:
metadata
:
name
:
subjects
:
-
kind
:
name
:
namespace
:
roleRef
:
kind
:
name
:
apiGroup
:
---
:
v1
kind
:
ConfigMap
metadata
:
name
:
namespace
:
data
:
:
|
profiles:
---
:
apps/v1
kind
:
metadata
:
labels
:
component
:
scheduler
tier
:
name
:
namespace
:
spec
:
selector
:
:
component
:
scheduler
tier
:
replicas
:
1
template
:
metadata
:
labels
:
component
:
scheduler
tier
:
version
:
second
spec
:
:
:
-
command
:
image
:
:
httpGet
:
path
:
/healthz
port
:
10259
scheme
:
HTTPS
:
15
name
:
:
httpGet
:
path
:
/healthz
port
:
10259
scheme
:
HTTPS
resources
:
requests
:
cpu
:
'0.1'
:
:
false
:
-
name
:
mountPath
:
:
false
hostPID
:
false
volumes
:
-
name
:
configMap
:
name
:
the
--config
a
.
Note:
field in a
.
.
for
for
=
....
...
to
true
to
to
&lt;lock-

object-name&gt;
Note:
endpoints
and
leases
:
kind
:
metadata
:
:
:
"true"
labels
:
:
name
:
rules
:
-
apiGroups
:
resources
:
- leases
verbs
:
- create
-
apiGroups
:
:
resources
:
- leases
verbs
:
- get
- update
-
apiGroups
:
-
""
:
resources
:
verbs
:
- delete
- get
- patch
- update
:
v1
kind
:
Pod
metadata
:
name
:
no
labels
:
name
:
spec
:
:
-
name
:
image
:
pod1.yaml
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
name
:
spec
:
:
:
-
name
:
image
:
.
pod2.yaml
:
v1
kind
:
Pod
metadata
:
name
:
labels
:
name
:
spec
:
:
:
-
name
:
image
:
deployed -
in the
.
pod3.yaml
pod gets
or a custom container image for the clust

er's main scheduler by modifying its static pod manifest
Feedback
Yes
No
.
or
.
Last modified December 08, 2021 at 3:46 PM PST:
Configuration API for v1.23 (b198199336)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Reference
ConfigMap
Secret
Volume
CSIDriver
CSINode
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

ClusterRoleBinding | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kind
metadata
(
)
roleRef
subjects
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
HTTP R

equest
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
ge

t
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm kubeconfig | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Insta

ll and Set Up kubectl on macOS
Debug Pods
Get a Shell to

 a Running Container
Auditing
Run Jobs
TLS
Tutorials
Security
Restrict a

 Container's Access to Resources with AppArmor
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver Encryption Con

Reference
Kubeadm
see
.
overview
Synopsis
Options
-h, --help
user
Synopsis
Examples
Options
-h, --help
Feedback
Yes
No
.
or
.
Last modified February 02, 2022 at 9:41 PM PST:
kubeadm: add missing guide for "kubeconfig user" (75a33b2650)
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes Components | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks

oting Applications
Debug Pods
Auditing
Run Jobs
TLS
Tutorials

Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-apiserver A

dmission (v1)
Concepts
Overview
nodes
,
Pods
that are
pod
replicas
.
etcd
back up
plan
.
Pods
node
to run on.
individual and collective resource

 requirements, hardware/software/policy
component
kubelet
node
Pod
.
node
Service
concept.
,
CRI-O
,
.
Addons
DaemonSet
,
, etc)
see
Addons
.
DNS
Dashboard
is a general purpose, web-based UI for Kubernetes clusters. It allows users to manage and troubleshoot 

applications running in the cluster, as well as the cluster itself.
A
Nodes
Feedback
Yes
No
.
or
.
etcd
kubelet
Addons
DNS
Home
Blog
Training
Partners
Community
CC BY 4.0

Attach Handlers to Container Lifecycle Events | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

Tasks
minikube
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
nginx
lifecycle
:
postStart
:
exec
:
command
:
[
"/bin/sh"
,
"-c"
,
]
preStop
:
exec
:
command
:
[
"/bin/sh"
,
"-c"
,
]
message
postStart
message
file:
relative to the Container's code, but Kubernetes' manage

ment of the container
.
Note:
.
completed
.
.
.
Reference
Lifecycle
Container
See
in
PodSpec
Feedback
Yes
No
.
or
.
Last modified August 01, 2022 at 3:22 PM PST:
fix doc about preStop hook (e7a0f694bf)
Reference
Home
Blog
Training
Partners
Community
CC BY 4.0

API Overview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

Reference
API
.
The
.
The
.
Alpha:
alpha
v1alpha1
).
Beta:
beta
v2beta3
).
(
except
Built-in beta API versions have a maximum lifetime of 9 months or 3 minor releases (whichever is longer) from introduction
a subsequent beta or stable API

 version. When this happens, migration
Note:
Stable:
vX
where
X
API groups
API groups
field of a
The
core
legacy
/api/v1
.
field, for
example,
.
and use
).
.
pairs
=true
to disable
batch/v1
, set
to enable
, set
, set
Note:
changes.
etcd
.
Feedback
Yes
No
.
or
.
API groups
Home
Blog
Training
Partners
Community
CC BY 4.0

Documentation Style Guide | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

.
Note:
to support
Language
(see
).
.
.
Code Style
.
Do
Don't
hostPath
field.
default
Use bold for user interface elemen

ts
Do
Don't
Click
Fork
.
Select
Other
.
Do
Don't
A
cluster
.
.
Do
Don't
Open the
file.
Go to the
Open the
file.
Do
Don't
`
).
Do
Don't
The
Pod
.
Lease
…
A
.
.
**
_
.
Note:
Do
Don't
replicas
exec
Do
Don't
The
kubelet
The
kubectl
.
Do
Don't
The
kubeadm
kubeadm
Do and Don't - U

se a general descriptor over a component name
Do
Don't
Do
Don't
to Always.
image
image
to
.
replicas
field to 2.
replicas
field to
2
.
pod
,
pods
,
)
pod
,
secret
)
Secret
Secret
".
`
).
and
.
.
Do
Don't
=
wide
or the
.
---
title
:
:
v1.8
---
Be careful to not include incorrect statements in your examples as comments

, such as:
:
v1
kind
:
Pod
...
Term
Usage
Docker
SIG Docs
Hugo
Note
,
Caution
, and
Warning
.
Note:
Note
Use
Note:
You can
still
in a list:

Note:
.
Caution
Use
Caution:
Warning
Use
Beware.
Warning:
Beware.
Caution:

Note:
Use a single newline to separate block-level content like headings, lists, images, code blocks, and others. The exception is second-level headings, where it should be two newlines. Second-level headings follow the first-l

evel (or the title) without any preceding paragraphs or texts. A two line spacing helps visualize the overall structure of content in a code editor better.
Do
Don't
#
---
or
===
Do
Don't
---
Links
Do
Don't
here
.
Lists
Note:
1.
Use (
+
), (
*
), or (
-
Tables
The semantic purpose of a data table is to present tabular data. Sighted users can q

uickly scan the table but a screen reader goes through line by line. A table caption is used to create a descriptive title for a data table. Assistive technologies (AT) use the HTML table caption element to identify the table contents to the user within the page structure.
Do
Don't
meaning.
Do
Don't
Do
Don't
Do
Don't
Do
Don't
e.g., ...
i.e., ...
Do
Don't
Do
Don't
is the
.
Do
Don't
Do
Don't
.
.
.
Feedback
Yes
No
.
or
.
Update

 page weights for pages in content/en/docs/contribute (35ebbd1e41)
Language
Note
Caution
Warning
Links
Lists
Tables
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Liveness, Readiness and Startup Probes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubern

)
Tasks
The
kubelet
minikube
:
v1
kind
:
Pod
metadata
:
labels
:
test
:
liveness
name
:
spec
:
:
-
name
:
liveness
image
:
args
:
- /bin/sh
- -c
:
exec
:
command
:
- cat
:
5
:
5
Container
.
The
field specifies that 

the kubelet should perform a liveness
/bin/sh -c
file.
RESTARTS
RESTARTS
image.
:
v1
kind
:
Pod
metadata
:
labels
:
test
:
liveness
name
:
spec
:
:
-
name
:
liveness
image
:
args
:
- /server
:
httpGet
:
path
:
/healthz
port
:
8080
:
-
name
:
value
:
Awesome
:
3
:
3
The
kubelet sends an HTTP GET request to the server that is running in the co

ntainer
/healthz
path
server.go
.
/healthz
handler
http.
(
,
func
*
duration
:=
time.
Now
().
Sub
(started)
if
duration.
Seconds
() &gt;
10
{
w.
(
500
)
w.
Write
([]
byte
(fmt.
Sprintf
(
Seconds
())))
}
else
{
w.
(
200
)
w.
Write
([]
byte
(
"ok"
))
}
})
(or
:
v1
kind
:
Pod
metadata
:
name
:
goproxy
labels
:
app
:
goproxy
spec
:
:
-
name
:
goproxy
image
:
ports
:
-
:
8080
:
tcpSocket
:
port
:
8080
:
5
:
10
:
tcpSocket
:
port
:
8080
:
15
:
20
goproxy
seconds.
goproxy
,
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
etcd
image
:
command
:
[
,
,
,
,
,
,
,
"--log-leve

l"
,
"debug"
]
ports
:
-
:
2379
:
grpc
:
port
:
2379
:
10
port
service
.
Note:
,
.
127.0.0.1
.
-tls
).
If
false
not
port
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
:
1
:
10
:
httpGet
:
path
:
/healthz
port
:
:
30
:
10
.
Services.
Note:
Caution:
do not
Readiness probes are configured similarly to liveness probes. The on

ly difference
field.
:
exec
:
command
:
- cat
:
5
:
5
Probes
checks:
try
Note:
(set it to
false
)
value set.
1.20+
with the
Caution:
httpGet
:
host
scheme
path
port
host
field in
httpGet
. If
scheme
HTTPS
host
field.
host
, under
httpGet
host
Host
header in
.
Host
head

er:
, and
Accept
(where
1.26
*/*
:
httpGet
:
:
-
name
:
Accept
value
:
:
httpGet
:
:
-
name
:
value
:
:
httpGet
:
:
-
name
:
Accept
value
:
""
:
httpGet
:
:
-
name
:
value
:
""
TCP probes
host
was used
was set.
Note:
The
field if
spec
:
:
3600
:
-
name
:
test
image
:
...
ports
:
-
name
:
:
8080
hostPort
:
8080
:
httpGet
:
path
:
/healthz
port
:
:
1
:
60
:
60
.
Pod
probe(s)
Feedback
Yes
No
.
or
.
Last modified September 23, 2022 at 11:24 AM PST:
Make scope for `Configure Probes` more clear (491036a847)
TCP probes
Home
Blog
Training
Partners
Community
CC BY 4.0
ICP license: 京ICP备17

kubeadm Configuration (v1beta3) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
Overview
Basics
--config
---
).
:
kind
:
:
kind
:
:
kind
:
:
kind
:
:
kin

d
:
init
or
join
--config
:
kind
:
:
...
:
...
:
kind
:
:
...
etcd
:
...
apiServer
:
extraArgs
:
...
:
...
...
etcd
:
kind
:
...
:
kind
:
...
run.
:
kind
:
:
-
token
:
"9a08jv.c0izixklcxtmnze7

"
:
ttl
:
"24h"
-
token
:
:
usages
:
- signing
groups
:
:
name
:
criSocket
:
taints
:
-
key
:
value
:
effect
:
:
v
:
4
:
:
:
:
bindPort
:
6443
:
:
---
:
kind
:
etcd
:
local
:
:
imageTag
:
"3.2.24"
dataDir
:
extraArgs
:
:
:
-
:
-
:
:
podSubnet
:
dnsDomain
:
:
"v1.21.0"
:
apiServer
:
extraArgs
:
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
certSANs
:
-
-
:
4m0s
:
extraArgs
:
"20"
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
scheduler
:
extraArgs
:
address
:
:
-
name
:
hostPath
:
mountPath
:
readOnly
:
false
pathType
:
File
:
:
:
---
:
kind
:
---
:
kind
:
with the
--config
:
kind
:
...
Field
token
token
string
ttl
ttl
24h
.
expires
and
ttl
expires
expires
ttl
.
expires
and
ttl
are mutually

 exclusive.
usages
[]string
usages
groups
[]string
groups
Field
-
string
-
string
Field
string
kind
string
etcd
Etcd
etcd
string
string
In case the
+
bindPort
the
bindPort
is used.
could
apiServer
APIServer
apiServer
scheduler
scheduler
dns
DNS
dns
string
string
If empty,
ci/
)
string
runs.
ConfigMap
Field
string
kind
string
is the global endpoint for the cluster, w

hich then
string
phase.
[]string
command.
patches
Patches
patches
.
Field
string
kind
string
string
discovery
Discovery
discovery
[]string
command.
The flag
patches
Patches
patches
.
Field
string
bindPort
int32
bindPort
APIServer
Field
certSANs
[]string
certSANs
Discovery
Field
token
string
token
string
[]string
bool
via
impersonate the control-p

lane.
APIServer
Field
extraArgs
extraArgs
DNS
Field
ImageMeta
ImageMeta
ImageMeta
imageMeta
Discovery
Field
and
file
file
file
and
file
string
If
, but
file
must be set
timeout
timeout
Etcd
Field
local
LocalEtcd
local
local
and
external
external
external
local
and
external
Etcd
Field
endpoints
[]string
endpoints
caFile
string
caFile
certFile
string
certFile
keyFile
string
keyFile
Discovery
Field
string
Field
name
string
name
hostPath
string
hostPath
mountPath
string
mountPath
hostPath
readOnly
bool
readOnly
pathType
pathType
hostPath
.
ImageMeta
DNS
LocalEtcd
Field
string
imageTag
string
imageTag
In case this value is set, kubeadm does not change automatically

 the version of
Field
string
LocalEtcd
Etcd
Field
ImageMeta
ImageMeta
ImageMeta
dataDir
string
dataDir
extraArgs
extraArgs
[]string
[]string
Field
string
podSubnet
string
podSubnet
dnsDomain
string
dnsDomain
Field
name
string
name
is the
or
criSocket
string
criSocket
taints
taints
i.e.
taints: []
[]string
Patches
Patches contains options related to applyin

g patches to components deployed by kubeadm.
Field
directory
string
directory
kubectl.
Feedback
Yes
No
.
or
.
APIServer
DNS
Discovery
Etcd
ImageMeta
LocalEtcd
Patches
Home
Blog
Training
Partners
Community
CC BY 4.0

TypedLocalObjectReference | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

n (v1)
Reference
kind
name
apiGroup
(string)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Certificate Signing Requests | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Reference
X.509
X.509
the
field.
Note that
.
600
by a
.
the
Once the
In order to reduce the number of old CertificateSigningReq

uest resources left in a cluster, a garbage collection
Signers
field, etc
field.
field. In
use the
(the
and
is
used.
Note:
The
:
.
The
,
.
Expiration/certificate lifetime - for the kube-controller-manager implementation of this signer, set to th

e minimum
of the
.
".
.
of the
.
".
.
of the
and later)
as
.
.
of the
Note:
Note:
The
).
Verbs:
create
,
get
,
list
,
w

atch
, group:
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
resources
:
verbs
:
- create
- get
- list
- watch
Verbs:
get
,
list
,
watch
, group:
Verbs:
update
, group:
Verbs:
approve
, group:
signers
or
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
resources
:
verbs
:
- get
- list
- watch
-
apiGroups
:
resources
:
verbs
:
- update
-
apiGroups
:
resources
:
- signers
:
verbs
:
- approve
Verbs:
get
,
list
,
watch
, group:
Verbs:
update
, group:
Verbs:
sign
, group:
signers
or
:
kind
:
metadata
:
name
:
rules
:
-
apiGroups
:
resources
:
verbs
:
- get
- list
- watch
-
apiGroups
:
resources
:
verbs
:
- update
-
apiGroups
:
resources
:
- signers
:
verbs
:
- sign
RBAC
2048
cat
metadata:
spec:
request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ1ZqQ0NBVDRDQVFBd0VURVBNQTBHQTFVRUF3d0dZVzVuWld4aE1JSUJJakFOQmdrcWhraUc5dzBCQVFFRgpBQU9DQVE4QU1JSUJDZ0tDQVFFQTByczhJTHRHdTYxakx2dHhWTTJSVlRWMDNHWlJTWWw0dWluVWo4RElaWjBOCnR2MUZtRVFSd3VoaUZsOFEzcWl0Qm0wMUFSMkNJVXBGd2ZzSjZ4MXF3ckJzVkhZbGlBNVhwRVpZM3ExcGswSDQKM3Z3aGJlK1o2MVNrVHF5SVBYUUwrTWM5T1Nsbm0xb0R2N0NtSkZNMUlMRVI3QTVGZnZKOEdFRjJ6dHBoaUlFMwpub1dtdHNZb3JuT2wzc2lHQ2ZGZzR4Zmd4eW8ybmlneFNVekl1bXNnVm9PM2ttT0x1RVF6cXpkakJ3TFJXbWlECklmMXBMWnoyalVnald4UkhCM1gyWnVVV1d1T09PZnpXM01LaE8ybHEvZi9DdS8wYk83c0x0MCt3U2ZMSU91TFcKcW90blZtRmxMMytqTy82WDNDKzBERHk5aUtwbXJjVDBnWGZLemE1dHJRSURBUUFCb0FBd0RRWUpLb1pJaHZjTgpBUUVMQlFBRGdnRUJBR05WdmVIOGR4ZzNvK21VeVRkbmFjVmQ1N24zSkExdnZEU1JWREkyQTZ1eXN3ZFp1L1BVCkkwZXpZWFV0RVNnSk1IRmQycVVNMjNuNVJsSXJ3R0xuUXFISUh5VStWWHhs

dnZsRnpNOVpEWllSTmU3QlJvYXgKQVlEdUI5STZXT3FYbkFvczFqRmxNUG5NbFpqdU5kSGxpT1BjTU1oNndLaTZzZFhpVStHYTJ2RUVLY01jSVUyRgpvU2djUWdMYTk0aEpacGk3ZnNMdm1OQUxoT045UHdNMGM1dVJVejV4T0dGMUtCbWRSeEgvbUNOS2JKYjFRQm1HCkkwYitEUEdaTktXTU0xMzhIQXdoV0tkNjVoVHdYOWl4V3ZHMkh4TG1WQzg0L1BHT0tWQW9FNkpsYWFHdTlQVmkKdjlOSjVaZlZrcXdCd0hKbzZXdk9xVlA3SVFjZmg3d0drWm89Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
usages:
EOF
usages
has to be '
'
864000
3600
request
.
jsonpath
=
=
=
get --verb
=
=
=
=
pods
=
=
myuser
=
=
=
true
=
=
myuser
myuser
:
kubectl
and
commands.
approval
operator
Approved
or
Denied
For
Approved
CSRs:
:
kind
:
...
status
:
:
-
:
:
message
:
reason
:
type
:
Approved
For
Denied
CSRs:
:
kind
:
...
status
:
:
-
:
:
"2020-02-08T11:37:35Z

"
message
:
reason
:
type
:
Denied
field.
Signing
,
Note:
Note:
The
status
.
.
:
kind
:
...
status
:
:
Read
signer
approver
RFC 5280
RFC 2986
Feedback
Yes
No
.
or
.
Last modified November 25, 2022 at 4:56 PM PST:
Added shell for code snippet. (e100cf80c4)
Signers
Approve certificate signi

ng request
kubectl
Signing
Home
Blog
Training
Partners
Community
CC BY 4.0

Quantity | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1alpha1)
Reference
Quantity
Quantity

Examples:

This format is intended to make it difficult to use these numbers without writing some sort of special handling code in the hopes that that will cause i

mplementors to also use a fixed point implementation.

Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Server-Side Apply | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
objects
combined
"
result in a
conflict
kubectl
POST
,
PUT
PATCH
change a field which is managed by someone else will result i

n a rejected
Conflicts
).
metadata
.
:
v1
kind
:
ConfigMap
metadata
:
name
:
test-cm
namespace
:
default
labels
:
:
test
:
-
manager
:
kubectl
operation
:
Apply
:
v1
time
:
:
FieldsV1
fieldsV1
:
:
f:labels
:
:
{}
f:data
:
f:key
:
{}
data
:
key
:
some value
. The
Note:
the user.
through an
Update
API
.
Conflicts
Apply
force
using the
Managers
query
kubectl
Apply
(
PATCH
with
) and
Update
Note:
as the
operations. Finally, when using the 

apply operation you cannot have
:
v1
kind
:
ConfigMap
metadata
:
name
:
test-cm
namespace
:
default
labels
:
:
test
:
-
manager
:
kubectl
operation
:
Apply
:
v1
fields
:
:
f:labels
:
:
{}
-
manager
:
operation
:
Update
:
v1
time
:
fields
:
f:data
:
f:key
:
{}
data
:
key
:
new value
Update
.
Apply
applied config if it is specified in both places. If the set of items present in
.
CRD
:
atomic
/
set
/
map
set
map
atomic
atomic
set
or
map
1.16
1.16
atomic
/
granular
atomic
granular
1.17
atomic
/
granular
.
1.17
If
listType
marker as a
and the
marker as a
.
The
atomic
map
/
set
/
granular
to
atomic

When the
listType
,
mapType
, or
map
/
set
/
granular
to
atomic
atomic
to
map
/
set
/
granular
atomic
to
map
/
set
/
granular
.
:
kind
:
Foo
metadata
:
name
:
:
-
manager
:
operation
:
Apply
:
fields
:
f:spec
:
f:data
:
{}
spec
:
data
:
key1
:
val1
key2
:
val2
Before
spec.data
atomic
to
granular
,
spec.data
(
key1
and
key2
spec.data
granular
,
spec.data
data
key1
and
key2
By default, Server-Side Apply treats custom resources as unstructured data. All
schema
type.
,
replicas
replicas
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
nginx
spec
:
replicas
:
3
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
=
50
--min
=
1
--max
=
10
replicas
replicas
, and if
replicas
replicas
is not what the user wants to happen, even temporarily

.
replicas
replicas
from their
replicas
field:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
3
:
\
=
\
=
false
replicas
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
nginx
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
replicas
case.
can start
[
--dry-run
=
server
]
Caution:
Keep the
.
kubectl
kubectl
.
kubec

tl apply --server-side --field-manager
=
[
--dry-run
=
server
]
,
.
.
kubectl
kubectl
.
=
[
--dry-run
=
server
]
PATCH
PATCH
PATCH
CREATE
using
,
,
JSONPatch
, or
Update
, so every
Feedback
Yes
No
.
or
.
Last modified September 30, 2022 at 5:18 AM PST:
Various Server-Side Apply improvements (#36293) (26848881f0)
Conflicts
Managers
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Scheduler Configuration
in the
.
,
using the
v1beta3
or
v1
)
struct.
:
kind
:
:
:
Note:
v1beta2
v1beta3
or
v1
Profiles
in the
.
.
Plugins
to run
.
queueSort
preFilter
filter
Pod
preScore
score
reserve
Unreserve
Reserve
.
permit
preBind
bind
bind
postBind
:
kind
:
profiles
:
-
plugins
:
score
:
disabled
:
-
name
:
enabled
:
-
name
:
weight
:
2
-
name
:

weight
:
1
You can use
*
desired.
Pod runs.
score
.
.
filter
,
preScore
,
score
.
NodeName
filter
.
NodePorts
preFilter
,
filter
.
and
.
filter
,
score
.
.
preFilter
,
filter
,
preScore
,
score
.
set to
true.
filter
.
and
.
preFilter
,
filter
,
score
.
score
.
volumes
.
preFilter
,
filter
,
reserve
,
preBind
,
score
.
Note:
score
feature is
size.
filter
.
filter
.
node.
filter
.
EBSLimits
filter
.
filter
.
the node.
filter
.
.
preFilter
,
filter
,
preScore
,
score
.
queueSort
.
bind
.
.
filter
.
.
disabled.
:
kind
:
profiles
:
-
:
-
:
no
plugins
:
preScore
:
disabled
:
-
name
:
'*'
score
:
disabled
:
-
name
:
'*'
.
than one profile, a unique scheduler name for eac

h of them is required.
Note:
list.
Note:
queueSort
MyPlugin
preScore
,
score
,
preFilter
,
and
filter
MyPlugin
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
MyPlugin
MyPlugin
:
kind
:
profiles
:
-
:
plugins
:
preScore
:
enabled
:
-
name
:
MyPlugin
score
:
enabled
:
-
name
:
MyPlugin
preFilter
:
enabled
:
-
name
:
MyPlugin
filter
:
enabled
:
-
name
:
MyPlugin
MyPlugin
the
disabled
'*'
Score
and
PreScore
, would be:
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
preScore
:
disabled
:
-
name
:
'*'
score
:
disabled
:
-
name
:
'*'
, all
.
and
, each with
a weight of
1
:
kind
:
profiles
:
-
:
plugins
:
score
:
enabled
:
-
name
:
weight
:
5
Score
is
.
over
Plugin
QueueSort
QueueSort
Score
,
Filter
Score
Score
,
Filter
Score
,
Fil

ter
:
kind
:
profiles
:
-
:
plugins
:
:
enabled
:
-
name
:
-
name
:
weight
:
3
-
name
:
disabled
:
-
name
:
filter
:
disabled
:
-
name
:
score
:
enabled
:
-
name
:
queueSort
Enables
and
Disables
filter
Reorders
score
v1beta3
, without
:
kind
:
profiles
:
-
:
plugins
:
queueSort
:
enabled
:
-
name
:
disabled
:
-
name
:
filter
:
enabled
:
-
name
:
-
name
:
-
name
:
disabled
:
-
name
:
score
:
enabled
:
-
name
:
weight
:
1
-
name
:
weight
:
3
config
plugin.
,
and
plugins.
with a
:
kind
:
profiles
:
-
:
-
args
:
:
resources
:
-
name
:
cpu
weight
:
1
type
:
name
:
NodeLabel
Invalid
host
or
port
from 1 to 2
from 1 to 2
from 1 to 3
Read the
Read the
reference
Read the
reference
Read the
reference
2 - Scheduling Policies
and
running
or
kube-scheduler --poli

cy-configmap &lt;ConfigMap&gt;
.
,
,
and
Read the
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Version Skew Policy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Kubernetes Release Cycle
Patch Releases
Release Managers
Release Notes
Release History
The maximum version skew supported between various Kubernetes components.
This document describes the maximum version skew supported between various Kubernetes components.
Specific cluster deployment tools may place additional restrictions on version skew.
Supported versions
Kubernetes versions are expressed as
x.y.z
, where
x
is the major version,
y
is the minor version, and
z
is the patch version, following
Semantic Versioning
terminology.
Kubernetes Release Versioning
.
The Kubernetes project maintains release branches for the most recent three minor releases (1.26, 1.25, 1.24). Kubernetes 1.19 and newer receive
approximately 1 year of patch support
. Kubernetes 1.18 and older received approximately 9 months of patch support.
Applicable fixes, including security fixes, may be backported to those three release branches, depending on severity and feasibility.
Patch releases are cut from those branches at a
regular cadence
, plus additional urgent releases, when required.
The
group owns this decision.
For more information, see the Kubernetes
patch releases
page.
Supported version skew
In
highly-available (HA) clusters
, the newest and oldest
instances must be within one minor version.
Example:
newest
is at
1.26
other
instances are supported at
1.26
and
1.25
kubelet
kubelet
must not be newer than
, and may be up to two minor versions older.
Example:
is at
1.26
kubelet
is supported at
1.26
,
1.25
, and
1.24
Note:
If version skew exists between
instances in an HA cluster, this narrows the allowed
kubelet
versions.
Example:
instances are at
1.26
and
1.25
kubelet
1.25
, and
1.24
(
1.26
is not supported because that would be newer than the
instance at version
1.25
)
kube-controller-manager, kube-scheduler, and cloud-controller-manager
,
, and
must not be newer than the
instances they communicate with. They are expected to match the
minor version, but may be up to one minor version older (to allow live upgrades).
Example:
is at
1.26
,
, and
are supported at
1.26
and
1.25
Note:
instances in an HA cluster, and these components can communicate with any
instance in the cluster (for example, via a load balancer), this narrows the allowed versions of these components.
Example:
1.26
and
1.25
,
, and
communicate with a load balancer that can route to any
instance
,
, and
1.25
(
1.26
1.25
)
kubectl
kubectl
is supported within one minor version (older or newer) of
.
Example:
is at
1.26
kubectl
1.27
,
1.26
, and
1.25
Note:
instances in an HA cluster, this narrows the supported
kubectl
versions.
Example:
1.26
and
1.25
kubectl
1.26
and
1.25
(other versions would be more than one minor version skewed from one of the
components)
Supported component upgrade order
The supported version skew between

 components has implications on the order in which components must be upgraded.
This section describes the order in which components must be upgraded to transition an existing cluster from version
1.25
to version
1.26
.
Optionally, when preparing to upgrade, the Kubernetes project recommends that
you do the following to benefit from as many regression and bug fixes as
possible during your upgrade:
Ensure that components are on the most recent patch version of your current
minor version.
Upgrade components to the most recent patch version of the target minor
version.
For example, if you're running version 1.25,
ensure that you're on the most recent patch version. Then, upgrade to the most
recent patch version of 1.26.
Pre-requisites:
In a single-instance cluster, the existing
instance is
1.25
In an HA cluster, all
1.25
or
1.26
(this ensures maximum skew of 1 minor version between the oldest and newest
instance)
The
,
, and
instances that communicate with this server are at version
1.25
(this ensures they are not newer than the existing API server version, and are within 1 minor version of the new API server version)
kubelet
instances on all nodes are at version
1.25
or
1.24
(this ensures they are not newer than the existing API server version, and are within 2 minor versions of the new API server version)
Registered admission webhooks are able to handle the data the new
instance will send them:
and
objects are updated to include any new versions of REST resources added in
1.26
(or use the
option
available in v1.15+)
The webhooks are able to handle any new versions of REST resources that will be sent to them, and any new fields added to existing versions in
1.26
Upgrade
to
1.26
Note:
Project policies for
API deprecation
and
API change guidelines
require
to not skip minor versions when upgrading, even in single-instance clusters.
The
instances these components communicate with are at
1.26
(in HA clusters in which these control plane components can communicate with any
instance in the cluster, all
instances must be upgraded before upgrading these components)
Upgrade
,
, and
to
1.26
. There is no
required upgrade order between
,
, and
. You can upgrade these components in any order, or
even simultaneously.
kubelet
The
instances the
kubelet
communicates with are at
1.26
Optionally upgrade
kubelet
instances to
1.26
(or they can be left at
1.25
or
1.24
)
Note:
Before performing a minor version
kubelet
upgrade,
drain
pods from that node.
In-place minor version
kubelet
upgrades are not supported.
Warning:
Running a cluster with
kubelet
instances that are persistently two minor versions behind
is not recommended:
they must be upgraded within one minor version of
before the control plane can be upgraded
it increases the likelihood of running
kubelet
versions older than the three maintained minor releases
must be the same minor version as
kubelet
on the node.
.
must be at most two minor versions older than
kube-apiserver.
Example:
If
version is
1.24
:
kubelet
version must be at the same minor version as
1.24
.
version must be between
1.24
and
1.26
, inclusive.
Feedback
Yes
No
.
Open an issue in the GitHub repo i

f you want to
or
.
Last modified August 13, 2022 at 5:53 AM PST:
Revert wrong skew shortcode parameters (480bf89f15)
kubelet
kubectl
kubelet
Home
Blog
Training
Partners
Community
CC BY 4.0

Declare Network Policy | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Note:
minikube
.
Antrea
Calico
Cilium
Romana
Weave Net
Create an
nginx
nginx
nginx
.
nginx
. The
nginx
default

nginx
nginx
default
=
1
nginx
Limit acc

ess to the
nginx
service
nginx
:
kind
:
metadata
:
name
:
spec
:
:
:
app
:
nginx
ingress
:
-
from
:
-
:
:
access
:
"true"
.
Note:
app=nginx
nginx
file:
nginx
=
1
nginx
=
1
nginx
Feedback
Yes
No
.
or
.
Last modified March 01, 2022 at 9:31 PM PST:
fix busybox image to 1.28 (issues with `nslookup`). (7122a4498a)
Create an
nginx
nginx
service
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Access to Multiple Clusters | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

ion (v1)
Tasks
command.
Note:
a
.
Warning:
minikube
kubectl
run
of your
In the
frontend
,
storage
. In your
scratch
cluster,
. In your
:
v1
kind
:
Config
:
{}
clusters
:
-
cluster
:
name
:
-
cluster
:
name
:
scratch
users
:
-
name
:
developer
-
name
:
contexts
:
-
context
:
name
:
-
context
:
name
:
-
context
:
name
:
file
has the framework to describe two clusters, two use

rs, and three contexts.
Go to your
=
=
=
=
=
Caution:
=
=
=
=
=
=
Note:
=
=
=
=
developer
=
=
=
=
developer
=
=
=
=
Open your
command.
=
:
v1
clusters
:
-
cluster
:
:
server
:
name
:
-
cluster
:
:
true
server
:
name
:
scratch
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
storage
user
:
developer
name
:
-
context
:
cluster
:
scratch
namespace
:
default
user
:
name
:
:
""
kind
:
Config
:
{}
users
:
-
name
:
developer
user
:
:
:
-
name
:
user
:
password
:
username
:
exp
The
,
and
-data
,
,
.
developer
frontend
cluster".
=
kubectl
context.
--minify
flag.
kubectl config --kubeco

nfig
=
context:
:
v1
clusters
:
-
cluster
:
:
server
:
name
:
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
:
kind
:
Config
:
{}
users
:
-
name
:
developer
user
:
:
:
:
=
Now any
kubectl
the
scratch
context.
.
=
storage
cluster.
:
=
.
=
In your
:
v1
kind
:
Config
:
{}
contexts
:
-
context
:
cluster
:
namespace
:
ramp
user
:
developer
name
:
.
Linux
export
=
"
"
=
The
a
Linux
export
=
"
${
}
=(
)
In your
the
file:
contexts
:
-
context
:
cluster
:
namespace
:
frontend
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
ramp
user
:
developer
name
:
-
context
:
cluster
:
namespace
:
storage
user
:
developer
name
:
-
context
:
cluster
:
scratch
namespace
:
default
user
:
name
:
kubectl
config
in the
Go to
config
Linux
export
=
"
${

}
:
${
HOME
}
=
in your
Clean up
Linux
export
=
"
"
=
There is a
kubectl
.
Read
Feedback
Yes
No
.
or
.
Last modified October 30, 2022 at 1:43 AM PST:
Apply suggestions from code review (5d61921a11)
Linux
Linux
Linux
Clean up
Linux
Home
Blog
Training
Partners
Community
CC BY 4.0

Manual Rotation of CA Certificates | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
TLS
minikube
.
.
Caution:
ca.crt
,
ca.key
,
,
and
Update the
to include
Any
Note:
and
ca.crt
and
Once
ca.crt
ca.crt
Issue 1350
ca.crt
old and n

ew CAs.
and
flag in the
flag in the
and
.
Update the
to include
Note:
or
in
and
in
.
,
and
to use and
Note:
openssl
see
openssl
)
.
cfssl
.
for
$(
jsonpath
=
)
;
do
for
name in
$(
-o
jsonpath
=
)
;
do
${
namespace
}
${
name
}
-p
;
done
for
name in
$(
-o
jsonpath
=
)
;
do
${
namespace
}
${
name
}
-p
;
done
done
Note:
see
.
in the
=
"
$(
)
"
\
/bin/sed
${
}
/"
|
\
Validate

 logs from any aggregated api servers and pods using in-cluster config.
,
flag
Feedback
Yes
No
.
or
.
Last modified April 14, 2022 at 2:47 PM PST:
Remove unnecessary step to manually update the service account secrets (b1a5f31dd5)
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubelet CredentialProvider (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extending Kube

rnetes
Tasks
Migrate from PodSecurityPolicy to the Built

-In PodSecurity Admission Controller
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Externalizing config using MicroProfi

le, ConfigMaps and Secrets
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
Even

t Rate Limit Configuration (v1alpha1)
Reference
Field
string
kind
string
image
string
Field
string
kind
string
auth
Globs can be used in the domai

n, but not in the port or the path. Globs are supported
'.
gcr.io
.
Field
username
string
password
string
(Alias of
string
)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Running ZooKeeper, A Distributed System Coordinator | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tutorials
on
,
,
and
.
Pods
tutorial.
ZooKeeper
is a
Zab
The ensemble uses the Zab protocol to elect a leader, and the ensemble cannot write data until that election is complete. Once complete, the ensemble uses Zab to ensure that it replicates all writes to a quorum before it acknowledges and makes them visible to clients. Without respect to weighted quorums, a quorum is a majority component of the ensemble containing the current leader. For instance, if the ensemble has three servers, a component that contains the leader and one other server constitutes a quorum. If the 

ensemble can not achieve a quorum, the ensemble cannot write data.
,
a
Service
,
a
,
and a
.
:
v1
kind
:
Service
metadata
:
name
:
zk-hs
labels
:
app
:
zk
spec
:
ports
:
-
port
:
2888
name
:
server
-
port
:
3888
name
:
clusterIP
:
None
selector
:
app
:
zk
---
:
v1
kind
:
Service
metadata
:
name
:
zk-cs
labels
:
app
:
zk
spec
:
ports
:
-
port
:
2181
name
:
client
selector
:
app
:
zk
---
:
policy/v1
kind
:
metadata
:
name
:
zk-pdb
spec
:
selector
:
:
app
:
zk
:
1
---
:
apps/v1
kind
:
metadata
:
name
:
zk
spec
:
selector
:
:
app
:
zk
:
zk-hs
replicas
:
3
:
type
:
:
template
:
metadata
:
labels
:
app
:
zk
spec
:
affinity
:
:
:
-
:
:
-
key
:
"app"
operator
:
In
values
:
- zk
:
:
-
name
:
:
Always
image
:
resources
:
requests
:
memory
:
"1Gi"
cpu
:
"0.5"
ports
:
-
:
2181
name
:
client
-
:
2888
name
:
server
-
:
3888
name
:
command
:
- sh
- -c
-
:
exec
:
command
:
- sh
- -c
-
:
10
:
5
:
exec
:
command
:
- sh
- -c
-
:
10
:
5
:
-
name
:
datadir
mountPath
:
:
runAsUser
:
1000
fsGroup
:
1000
:
-
metadata
:
name
:
datadir
spec
:
:
[
]
resources
:
requests
:
storage
:
10Gi
manifest.
zk-hs
zk-cs
Service,
the
zk-pdb
zk
Use
app
=
zk
Once the
zk-2
CTRL-C
zk-2      0/1       Pending   0    

     0s
a
ZooKeeper
server.
Use
zk
for
i in
0
1
2;
do
kubectl
exec
zk-
$i
done
replicas
zk
3
zk-0
,
zk-1
, and
zk-2
.
zk-0
zk-1
zk-2
myid
myid
for
i in
0
1
2;
do
echo
"myid zk-
$i
"
;kubectl
exec
zk-
$i
done
myid zk-0
1
myid zk-1
2
myid zk-2
3
zk
for
i in
0
1
2;
do
kubectl
exec
zk-
$i
done
The
zk-hs
.
zoo.cfg
. Use
zoo.cfg
file in the
zk-0
Pod.
kubectl
exec
In the
server.1
,
server.2
, and
server.3
1
,
2
, and
3
myid
the
zk
app
=
zk
zk-1      1/1       Runn

ing   0         40s
in its
myid
file.
servers
zoo.cfg
files
zkCli.sh
world
to the path
/hello
on the
zk-0
kubectl
exec
WATCHER::

zk-1
kubectl
exec
zk-0
ensemble.
WATCHER::

world
section,
Use the
zk
app
=
zk
When
zk-0
CTRL-C
.
zk
app
=
zk
Once the
zk-2
CTRL-C
zk-1      0/1       P

ending   0         0s
,
from the
zk-2
Pod.
kubectl
exec
zk
WATCHER::

world
The
zk
spec
:
-
metadata
:
name
:
datadir
:
:
anything
spec
:
:
[
]
resources
:
requests
:
storage
:
20Gi
The
the
.
's
.
app
=
zk
When the
The
template
:
-
name
:
datadir
mountPath
:
zk
same
and
Get the
zk
…
command:
      - sh
      - -c
…
Log4j
Use the command below to get the lo

gging configuration from one of Pods in the
zk
.
kubectl
exec
Use
20
2016-12-06 19:34:46,230 [myid:1] - INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192]

 - Accepted socket connection from /127.0.0.1:52768
The
zk
's Pod
template
contains a
.
:
runAsUser
:
1000
fsGroup
:
1000
zk-0
Pod.
kubectl
exec
As the
runAsUser
zk-0
Pod.
kubectl
exec
fsGroup
The
The
zk
You can use
cpus
=
'json'
-p
=
Use
waiting for statefulset rolling upd

ate to complete 0 pods at revision zk-5db4499664...
Use the
history
sts/zk
REVISION
1
2
Use the
never
override
zk-0
Pod.
kubectl
exec
zk
app
=
zk
zk-0
kubectl
exec
The Pod
template
for the
zk
specifies

 a liveness probe.
:
exec
:
command
:
- sh
- -c
-
:
15
:
5
ruok
    exit 0
else
    exit 1
fi
zk
app
=
zk
zk-0
.
kubectl
exec
app
=
zk
:
exec
:
command
:
- sh
- -c
-
:
15
:
5
.
zk
.
for
i in
0
1
2;
do
$i
{{
}}
;
echo
""
;
done
zk
zk
have a
affinity
:
:
:
-
:
:
-
key
:
"app"
operator
:
In
values
:
- zk
:
The
app
label
as
zk

. The
zk-pdb
.
The
zk
zk
.
app
=
zk
for
i in
0
1
2;
do
$i
{{
}}
;
echo
""
;
done
Use
zk-0
$(
{{
}}
)

zk-0
zk-1
$(
{{
}}
)
The
zk-1
zk
contains a
app
=
zk
zk-0      1/1      

 Running   2          1h
zk-2
$(
{{
}}
)

pod/zk-2
Use
CTRL-C
zk-2
zk-budget
Use
zkCli.sh
zk-0
.
kubectl
exec
world
Use
zk-1
zk-1
app
=
zk
zk-2
$(
{{
}}
)
node "kubernetes-node-i4c4" alr

eady cordoned
This time
succeeds.
zk-2
You can use
Use
Feedback
Yes
No
.
or
.
Last modified November 22, 2022 at 4:49 PM PST:
zookeeper.md - remove extra "to" (4b07c88fad)
ZooKeeper
Home
Blog
Training
Partners
Community
CC BY 4.0

kubectl for Docker Users | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kubectl
kubectl
command.
docker run
.
docker:
=
always -e
DOMAIN
=
docker ps
kubectl:
=
kubectl
set
DOMAIN
=
cluster
Note:
kubectl
=
80
--name
=
service
.
[
-i
]
[
--tty
]
=
Unlike
--attach
stdin
,
stdout
and
stderr
).
docker ps
.
docker:
14636241935f        ubuntu:16.04 

       "echo test"              5 seconds ago        Exited (0) 5 seconds ago                        cocky_fermi
kubectl:
.
docker:
docker ps
...
kubectl:
...
.
docker:
docker ps
docker
exec
kubectl:
kubectl
exec
docker:
docker
exec
# exit
kubectl:
kubectl
exec
# exit
.
.
docker:
kubectl:
.
.
docker:
docker ps
kubectl:
app
=
nginx-app
deploym

ent "nginx-app" deleted
app
=
nginx-app
Note:
.
.
docker:
kubectl:
.
docker:
 Dirs: 248
CPUs: 12
kubectl:
Feedback
Yes
No
.
or
.
docker run
docker ps
Home
Blog
Training
Partners
Community
CC BY 4.0

Advanced contributing | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install a

1)
and
Propose improvements
SIG Docs
members
, the
.
#sig-docs
SIG Docs
approvers
refer to
.
Mentor other SIG 

Docs contributors who want to learn how to do this role in
SIG Docs
approvers
.
SIG Docs
reviewers
on the
or on the
.
If you feel confident about the applicant's work, you volunteer to sponsor them.
SIG Docs
members
, the
,
plugins in
.
works.
CNCF
Uphold the
:
:
For weekly meetings, copypaste the previous week's notes

 into the "Past meetings" section of the notes
:
:
:
:
:
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
1 - Extend the Kubernetes API with CustomResourceDefinitions
.
minikube
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
The
kind
CronTab
cronSpec
and
image
CronTab
. The kind
CronTab
CustomResourceDef

inition object you created above.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
NAME                 AGE
cronSpec
and
image
fields
:
v1
items
:
-
:
kind
:
CronTab
metadata
:
:
:
|
{
:
,
"kind"
:
"CronTab"
,
:{
:{},
"name"
:
,
:
"default"
},
"spec"
:{
:
"* * * * */5"
,
"image"
:
}}
:
:
1
name
:
namespace
:
default
:
"1326"
uid
:
spec
:
cronSpec
:
'* * * * */5'
image
:
kind
:
List
metadata
:
:
""
selfLink
:
""
fields
,
kind
and
metadata
With
which:
type
(via
or
(via
items
a node with
a node with
allOf
,
anyOf
,
oneOf
or
not
,
type
,
default
,
,
nullable
within an
allOf
,
anyOf
,
oneOf
or
not
if
metadata
and
allOf:
    foo:
      ...
  foo:
    ...
allOf:
    foo:
      ...
allOf:
- items:
      foo:
        ...
items:
    foo:
      ...
allOf:
- items:
      foo:
        ...
  foo:
  metadata:
      name:
        patter

n: "^a"
        items:
anyOf:
    bar:
the type of
foo
bar
inside of
anyOf
bar
's
type
is within
anyOf
(rule 3).
anyOf
(rule 3).
type
:
object
:
:
foo
:
type
:
string
pattern
:
"abc"
bar
:
type
:
integer
metadata
:
type
:
object
:
name
:
type
:
string
pattern
:
"^a"
anyOf
:
-
:
bar
:
minimum
:
42
required
:
[
"bar"
]
etcd
.
ConfigMap
,
pruned
to
might lack
might be
true
.
with
set to
true
Set
to
false
.
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
:
42
=
false
:
kind
:
CronTab
metadata
:
:
:
1
name
:
namespace
:
default
:
"285"
uid
:
spec
:
cronSpec
:
'* * * * */5'
image
:
the
kubectl
in the
.
type
:
object
:
json
:
:
true
The field
json
type
:
object
:
json
:
:
true
type
:
object
:
object
):
type
:
object
:
json
:
:
true
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
bar
:
type
:
string
json
:
spec
:
foo
:
abc
bar
:
def
something
:
x
status
:
something
:
x
json
:
spec
:
foo
:
abc
bar
:
def
status
:
something
:
x
something
s

pec
type
:
object
:
foo
:
:
true
anyOf:
...
and
allOf:
  - anyOf:
...
In
,
)
and
kind
fields.
by setting
type
:
object
:
foo
:
:
true
:
true
foo
foo:
  kind: Pod
  spec:
    ...
Because
The use of
With
, the
,
kind
and
metadata
See
:
kind
:
CronTab
metadata
:
:
When the
,
.
,
,
,
,
id
,
,
readOnly
,
writeOnly
,
xml
,
$ref
.
The field
true
.
The field
false
.
The field
.
The
.
section for o

ther
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * *"
image
:
replicas
:
15
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
5
crontab
created
false
, for the
.
And
self
...
      spec:
          - rule: "self.replicas &lt;=

 self.maxReplicas"
          ...
          replicas:
        required:
          - replicas
:
kind
:
CronTab
metadata
:
name
:
spec
:
:
0
replicas
:
20
:
10
The
rule
under
The
message
has(self)
Rule
Purpose
size(self.clusters.filter(c, c.name == s

elf.primary)) == 1
Xref:
,
kind
,
and
spec
and
status
in the
...
        spec:
            ...
        status:
via
...
      spec:
          ...
          foo:
and all
.
...
      spec:
          ...
            foo:
self[i]
and
...
      ...
      foo:
        items:
self
...
      spec:
          ...
          foo:
Examples:
string
The
,
kind
,
and
is not
.
__
__dot__
.
__dash__
-
__slash__
/
Note: CEL RESERVED keyword needs to match the exact property name to be escaped (e.g. 

int in the word sprint would not be escaped).
namespace
x-prop
redact__d
string
of
set
or
map
i.e.,
set
:
X + Y
X
Y
map
:
X + Y
X
Y
X
and
Y
in
Y
CEL type
map
'array
list
'boolean'
boolean
double
int (64)
'null'
null_type
'string'
string
bytes
xref:
CEL types
,
,
.
macros
CEL
oldSelf
type
:
string
enum
:
[
"low"
,
"medium"
,
"high"
]
:
-
rule
:
message
:
path
rule
A portion of the schema is correlatable if al

l
array
;
any
set
or
atomic
self
with
oldSelf
.
Use Case
Rule
or
But if
foo
foo
is.
foo
.
foo
is not
maxItems
,
, and
maxLength
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
string
:
-
rule
:
self.all
foo
,
'a string'
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
maxLength
:
10
:
-
rule
:
:
type
:
object
:
foo
:
type
:
array
maxItems
:
25
items
:
type
:
string
:
-
rule
:
"se

lf.contains('a string'))"
maxLength
:
10
self.all
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
integer
:
-
rule
:
:
type
:
object
:
foo
:
type
:
array
items
:
type
:
array
items
:
type
:
integer
:
-
rule
:
foo
self.all
.
Note:
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
pattern
:
default
:
"5 0 * * *"
image
:
type
:
string
replicas
:
type
:
integer
minimum
:
1
maximum
:
10
default
:
1
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
cronSpec
and
replicas
:
kind
:
CronTab
metadata
:
name
:
spec
:
image
:
leads to
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"5 0 * * *"
image
:
replicas
:
1
metadata
metadata
fields of
metadata
false
true
type
:
object
:
spec
:
type
:
object
:
foo
:
type
:
string
nullable
:
false
default
:
"default"
bar
:
type
:
string
nullable
:
true
baz
:
type
:
string
foo
and
bar
and
baz
spec
:
foo
:
null
bar
:
null
baz
:
null
leads to
spec
:
foo
:
"default"
bar
:
null
with
foo
bar
, and
baz
default.
which are
and
from Kubernetes

 API server.
The
kubectl
and
)
show up in
and
paths
.
The fields
allOf
,
anyOf
,
oneOf
and
not
If
type
,
nullable
,
items
and
Spec
,
Replicas
, and
Age
columns.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
:
-
name
:
Spec
type
:
string
:
jsonPath
:
-
name
:
Replicas
type
:
integer
:
jsonPath
:
-
name
:
Age
type
:
date
jsonPath
:
Notice the
NAME
,
SPEC
,
REPLICAS
, and
AGE
Note:
The
NAME
Priority
priority
-o wide
flag).
0
0
Type
A column's
type
):
integer
number
string
– strings
boolean
–
true
or
false
date
Format
A column's
format
int32
int64
float
double
byte
date
date-time
password
format
kubectl
/status
and
/scale
When the status subresource is enabled, th

e
/status
.status
and
.spec
JSONPaths
PUT
/status
PUT
/status
resource.
PUT
/
POST
/
PATCH
The
.metadata
or
.status
.
example
format
items
maximum
maxItems
maxLength
minimum
minItems
minLength
pattern
required
title
type
/scale
The
/scale
.
.
.spec
the
/scale
.
.status
/scale
.
.status
or
.spec
/scale
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
status
:
type
:
object
:
replicas
:
type
:
integer
:
type
:
string
:
status
:
{}
scale
:
:
:
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
:
kind
:
CronTab
metadata
:
name
:

spec
:
cronSpec
:
"* * * * */5"
image
:
replicas
:
3
and
command.
of the
=
5
crontabs
scaled
jsonpath
=
5
all
).
You can use
all
.
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
spec
:
type
:
object
:
cronSpec
:
type
:
string
image
:
type
:
string
replicas
:
type
:
integer
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
- all
:
:
kind
:
CronTab
metadata
:
name
:
spec
:
cronSpec
:
"* * * * */5"
image
:
:
CronTab
:
NAME                          AGE
Read about
.
See
.
Serve
of a
2 - Versions in CustomResourceDefinitions
minikube
.
.
Overview
v1beta1
v1beta1
API.
v1
.

None
the
list with
. Also, set
field
See the
section.
Set
served
to
false
list. If
Ensure the
storage
is set to
true
.
list.
versions
Note:
In
version
versions
. The
version
versions
field.
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
host

:
type
:
string
port
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
None
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
-
name
:
v1
served
:
true
storage
:
false
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
None
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
and
.
name
v
beta
or
alpha
like
v2
or
v2beta1
do not.
beta
or
alpha
beta
or
alpha
beta
, or
alpha
Stri

ngs that don't fit the above format are sorted alphabetically and the
foo1
foo10
- v10
- v2
- v1
- v11beta2
- v10beta3
- v3beta1
- foo1
- foo10
, the
v1
v1beta1
v1
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
:
true
:
schema
:
...
-
name
:
v1beta1
served
:
true
:
true
schema
:
...
-
name
:
v1
served
:
true
storage
:
true
schema
:
...
:
kind
:
metadata
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
:
...
versions
:
-
name
:
v1alpha1
served
:
true
storage
:
false
:
true
:
-
name
:
v1beta1
served
:
true
:
true
-
name
:
v1
served
:
true
storage
:
true
:
kind
:
name
:
spec
:
group
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
scope
:
versions
:
-
name
:
v1beta1
# This indicates the v

1beta1 version of the custom resource is no longer served.
served
:
false
schema
:
...
-
name
:
v1
served
:
true
storage
:
true
schema
:
...
Note:
field
server
Note:
field
empty
,
how to
.
metadata
other than
labels
and
.
name
,
UID
and
namespace
.
named
in
default
.
Note:
The
None
spec
:
:
kind
:
metadata
:
name
:
spec
:
group
:
versions
:
-
name
:
v1beta1
# Each version can be enabled/disabled b

y Served flag.
served
:
true
storage
:
true
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
Webhook
webhook
:
:
[
"v1"
,
"v1beta1"
]
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
:
kind
:
metadata
:
name
:
spec
:
group
:
:
false
versions
:
-
name
:
v1beta1
served
:
true
storage
:
true
schema
:
:
type
:
object
:
hostPort
:
type
:
string
-
name
:
v1
served
:
true
storage
:
false
schema
:
:
type
:
object
:
host
:
type
:
string
port
:
type
:
string
:
strategy
:
Webhook
:
service
:
namespace
:
default
name
:
path
:
caBundle
:
scope
:
names
:
plural
:
crontabs
singular
:
crontab
kind
:
CronTab
:
- ct
Conversion w

ebhooks can either be called via a URL or a service reference,
URL
url
(
).
The
host
service
(i.e.,
host
localhost
or
127.0.0.1
as a
host
is
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
url
:
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
url
:
...
The
service
service
instead of
url
.
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
caBundle
:
...
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
:
service
:
namespace
:
name
:
path
:
/my-path
port
:
1234
caBundle
:
...
Request
,
with a
API group
with the
:
kind
:
...
spec
:
...
:
strategy
:
Webhook
webhook
:
:
[
"v1"
,
"v1beta1"
]
...
:
kind
:
CustomRe

sourceDefinition
...
spec
:
...
:
strategy
:
Webhook
:
[
"v1"
,
"v1beta1"
]
...
If no
v1beta1
.
object
CronTab
objects to
:
{
,
"kind":
,
{
"uid":
,
,
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
}
]
}
}
{
,
"kind":
,
{
"uid":
,
,
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
}
]
}
}
Response
,
with the
response
response
uid
result
, set to
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Success"
},
# All other

 changes to metadata fields by the webhook are ignored.
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
"host":
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
,
"port":
"2345"
}
]
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
},
[
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"default"
,
"143"
,
"uid":
},
"host":
,
"port":
"1234"
},
{
"kind":
"CronTab"
,
,
{
,
"name":
,
"12893"
,
"uid":
},
"host":
,
"port":
"2345"
}
]
}
}
response
uid
result
, set to
Warning:
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
,
}
}
}
{
,
"kind":
,
{
"uid":
,
"result":
{
"status":
"Failed"
,
}
}
}
:
strategy
value
None
is specified, the only modif

ications
pruning
if
v1beta1
v1beta1
v1
v1
and
v1beta1
v1beta1
version
v1
field.
v1
. You now
v1beta1
v1
.
v1
since that
Option 1:
Run the
field.
Option 2:
v1beta1
to
v1
.
Set
v1
is now
.
v1
.
Remove
v1beta1
field.
Note:
The flag
status
and
scale
status
kubectl
tool
status
.
status
kubectl
:
=
'status'
--type
=
'merge'
-p
Home
Blog
Training
Partners
Community
CC BY 4.0

Services | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tutorials
Services
Services
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Utilizing the NUMA-aware Memory Manager | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
QoS class
.
minikube
.
See
;
See
.
.
kubelet
.
cgroups
(i.e.
) for pods.
The complete flow diagram concerning pod admission and depl

oyment process is illustrated in
and below:
Startup
(section
).
.
flag when
Static
Runtime
Reference
.
Static
).
).
Policies
kubelet
flag
:
None
(default)
Static
The
None
pod, the
Static
NodeMap
object.
or
Burstable
pod, the
Static
NodeMap
object.
The
mechanism
,
and
.
A new
The Memory Manager will not use this reserved memory for the allocation of c

ontainer workloads.
10Gi
the
1Gi
9Gi
.
In fact,
100Mi
if
Static
memory
or
Syntax:
N
0
memory
or
value
1Gi
or
or
,
where
i
),
we reserve
3Gi
Note:
not
zero.
=
=
true
=
cpu
=
4,memory
=
4Gi
=
cpu
=
1,memory
=
1Gi
=
Static
None
QoS class.
pod.
QoS class.
requests
limits
:
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"2"
:
"1"
requests
limits
.
spec
:
:
-
name

:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
requests
:
memory
:
"200Mi"
cpu
:
"300m"
:
"1"
(includes
)
can be used
Use
or
cgroups
and
updates.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
consumer
:
Never
resources
:
limits
:
cpu
:
"2"
memory
:
150Gi
requests
:
cpu
:
"2"
memory
:
150Gi
command
:
[
"sleep"
,
]
:
{
:
"Static"
,
:{
"0"
:{
:
1
,
:{
:{
"total"
:
0
,
:
0
,
:
0
,
:
0
,
"free"
:
0
},
"memory"
:{
"total"
:
,
:
,
:
,
:
,
"free"
:
0
}
},
"nodes"
:[
0
,
1
]
},
"1"
:{
:
1
,
:{
:{
"total"
:
0
,
:
0
,
:
0
,
:
0
,
"free"
:
0
},
"memory"
:{
"total"
:
,
:
,
:
,
:
,
"free"
:
}
},
"nodes"
:[
0
,
1
]
}
},
"entries"
:{
:{
:[
{
:[
0
,
1
],
"type"
:
"memory"
,
"size"
:
}
]
}
},
:
}
:
[
0
,
1
]
,
cgroups
0
and
1
this
and
this
sectio

ns.
"memory"
0
(
"free":0
1
(
).
bytes.
The line
3Gi
0
,
by using
flag.
,
message.
Feedback
Yes
No
.
or
.
Last modified November 29, 2022 at 7:08 PM PST:
Update memory-manager.md (00425ef91f)
Startup
Runtime
Policies
Home
Blog
Training
Partners
Community
CC BY 4.0

Manage TLS Certificates in a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
TLS
ACME
draft
.
Note:
minikube
cfssl
cfssl
from
.
jq
jq
, you can
.
custom CA
RootCAs
struct.
Note:
ConfigMap
),
ConfigMap
Note:
click here
to kn

ow more.
cat
{
"hosts": [
],
"key": {
}
}
EOF
Where
10.0.34.2
encoded
PKCS#10
and
cat
metadata:
spec:
usages:
EOF
signer.
A specific
Subject:
                        10.0.34.2
kubectl
,
First, create a signing certificate by running the foll

owing:
cat
{
"key": {
}
}
EOF
ca.pem
).
{
"signing"
: {
"default"
: {
"usages"
: [
,
,
],
"expiry"
:
"876000h"
,
: {
"is_ca"
:
false
}
}
}
}
Use a
jsonpath
=
|
\
\
\
.
\
jq
$(
'\n'
)
'"'
|
\
Note:
jq
field.
jq
jsonpath
=
\
and
in a
Secret
ca.pem
into a
ConfigMap
=
ca.pem
and
Caution:
and
approve
approver
is
The subject of the CSR controls the private key

 used to sign the CSR. This
the
and
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

ServiceAccount | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
: v1
kind
metadata
(
)
(boolean)
([]
)
secrets
([]
)
name
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
na

mespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in

 query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
Services
1:
2:
1 - Connecting Applications with Services
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
selector
:
:
run
:
my-nginx
replicas
:
2
template
:
metadata
:
labels
:
run
:
my-nginx
spec
:
:
-
name
:
my-nginx
image
:
nginx
ports
:
-
:
80
run
=
run
=
=
POD_IP
[
map
[
]]
[
map
[
]]
curl
not
:
kubectl apply 

-f
:
v1
kind
:
Service
metadata
:
name
:
my-nginx
labels
:
run
:
my-nginx
spec
:
ports
:
-
port
:
80
protocol
:
TCP
selector
:
run
:
my-nginx
with the
(
port
: is the
Service).
View
Service
.
labels
.
=
my-nginx
from
about the
.
.
Note:
flag to
false
on the
pod spec
.
kubectl
exec
before
the
=
=
2;
run
=
my-nginx-3800858182-e9ihh   1/1       Running   0          5s      10.244.2.7    kubernetes

-minion-ljyd
kubectl
exec
DNS
=
kube-dns
or
=
:
[
]
A
secret
make keys
KEY
=
CERT
=
=
365
:
"v1"
kind
:
"Secret"
metadata
:
name
:
namespace
:
"default"
type
:
data
:
tls.crt
:
"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURIekNDQWdlZ0F3SUJBZ0lKQUp5M3lQK0pzMlpJTUEwR0NTcUdTSWIzRFFFQkJRVUFNQ1l4RVRBUEJnTlYKQkFNVENHNW5hVzU0YzNaak1SRXdEd1lEVlFRS0V3aHVaMmx1ZUhOMll6QWVGdzB4TnpFd01qWXdOekEzTVRKYQpGdzB4T0RFd01qWXdOekEzTVRKYU1DWXhFVEFQQmdOVkJBTVRDRzVuYVc1NGMzWmpNUkV3RHdZRFZRUUtFd2h1CloybHVlSE4yWXpDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSjFxSU1SOVdWM0IKMlZIQlRMRmtobDRONXljMEJxYUhIQktMSnJMcy8vdzZhU3hRS29GbHlJSU94NGUrMlN5ajBFcndCLzlYTnBwbQppeW1CL3JkRldkOXg5UWhBQUxCZkVaTmNiV3NsTVFVcnhBZW50VWt1dk1vLzgvMHRpbGhjc3paenJEYVJ4NEo5Ci82UVRtVVI3a0ZTWUpOWTVQZkR3cGc3dlVvaDZmZ1Voam92VG42eHN

VR0M2QURVODBpNXFlZWhNeVI1N2lmU2YKNHZpaXdIY3hnL3lZR1JBRS9mRTRqakxCdmdONjc2SU90S01rZXV3R0ljNDFhd05tNnNTSzRqYUNGeGpYSnZaZQp2by9kTlEybHhHWCtKT2l3SEhXbXNhdGp4WTRaNVk3R1ZoK0QrWnYvcW1mMFgvbVY0Rmo1NzV3ajFMWVBocWtsCmdhSXZYRyt4U1FVQ0F3RUFBYU5RTUU0d0hRWURWUjBPQkJZRUZPNG9OWkI3YXc1OUlsYkROMzhIYkduYnhFVjcKTUI4R0ExVWRJd1FZTUJhQUZPNG9OWkI3YXc1OUlsYkROMzhIYkduYnhFVjdNQXdHQTFVZEV3UUZNQU1CQWY4dwpEUVlKS29aSWh2Y05BUUVGQlFBRGdnRUJBRVhTMW9FU0lFaXdyMDhWcVA0K2NwTHI3TW5FMTducDBvMm14alFvCjRGb0RvRjdRZnZqeE04Tzd2TjB0clcxb2pGSW0vWDE4ZnZaL3k4ZzVaWG40Vm8zc3hKVmRBcStNZC9jTStzUGEKNmJjTkNUekZqeFpUV0UrKzE5NS9zb2dmOUZ3VDVDK3U2Q3B5N0M3MTZvUXRUakViV05VdEt4cXI0Nk1OZWNCMApwRFhWZmdWQTRadkR4NFo3S2RiZDY5eXM3OVFHYmg5ZW1PZ05NZFlsSUswSGt0ejF5WU4vbVpmK3FqTkJqbWZjCkNnMnlwbGQ0Wi8rUUNQZjl3SkoybFIrY2FnT0R4elBWcGxNSEcybzgvTHFDdnh6elZPUDUxeXdLZEtxaUMwSVEKQ0I5T2wwWW5scE9UNEh1b2hSUzBPOStlMm9KdFZsNUIyczRpbDlhZ3RTVXFxUlU9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
tls.key
:
"LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQ2RhaURFZlZsZHdkbFIKd1V5eFpJWmVEZWNuTkFhbWh4d1NpeWF5N1AvOE9ta3NVQ3FCWmNpQ0RzZUh2dGtzbzlCSzhBZi9WemFhWm9zcApnZjYzUlZuZmNmVUlRQUN3WHhHVFhHMXJKVEVGSzhRSHA3VkpMcnpLUC9QOUxZcFlYTE0yYzZ3MmtjZUNmZitrCkU1bEVlNUJVbUNUV09UM3c4S1lPNzFLSWVuNEZJWTZMMDUrc2JGQmd1Z0ExUE5JdWFubm9UTWtlZTRuMG4rTDQKb3NCM01ZUDhtQmtRQlAzeE9JNHl3YjREZXUraURyU2pKSHJzQmlIT05Xc0RadXJFaXVJMmdoY1kxeWIyWHI2UAozVFVOcGNSbC9pVG9zQngxcHJHclk4V09HZVdPeGxZZmcvbWIvNnBuOUYvNWxlQlkrZStjSTlTMkQ0YXBKWUdpCkwxeHZzVWtGQWdNQkFBRUNnZ0VBZFhCK0xkbk8ySElOTGo5bWRsb25IUGlHWWVzZ294RGQwci9hQ1Zkank4dlEKTjIwL3FQWkUxek1yall6Ry9kVGhTMmMwc0QxaTBXSjdwR1lGb0xtdXlWTjltY0FXUTM5SjM0VHZaU2FFSWZWNgo5TE1jUHhNTmFsNjRLMFRVbUFQZytGam9QSFlhUUxLOERLOUtnNXNrSE5pOWNzMlY5ckd6VWlVZWtBL0RBUlBTClI3L2ZjUFBacDRuRWVBZmI3WTk1R1llb1p5V21SU3VKdlNyblBESGtUdW1vVlVWdkxMRHRzaG9reUxiTWVtN3oKMmJzVmpwSW1GTHJqbGtmQXlpNHg0WjJrV3YyMFRrdWtsZU1jaVlMbjk4QWxiRi9DSmRLM3QraTRoMTVlR2ZQegpoTnh3bk9QdlVTaDR2Q0o3c2Q5TmtEUGJvS2JneVVHOXBYamZhRGR2UVFLQmdRRFFLM01nUkhkQ1pKNVFqZWFKClFGdXF4cHdnNzhZTjQyL1NwenlUYmtGcVFoQWtyczJxWGx1MDZBRzhrZzIzQkswaHkzaE9zSGgxcXRVK3NHZVAKOWRERHBsUWV0ODZsY2FlR3hoc0V0L1R6cEdtNGFKSm5oNzVVaTVGZk9QTDhPTm1FZ3MxMVRhUldhNzZxelRyMgphRlpjQ2pWV1g0YnRSTHVwSkgrMjZnY0FhUUtCZ1FEQmxVSUUzTnNVOFBBZEYvL25sQVB5VWs1T3lDdWc3dmVyClUycXlrdXFzYnBkSi9hODViT1JhM05IVmpVM25uRGpHVHBWaE9JeXg5TEFrc2RwZEFjVmxvcG9HODhXYk9lMTAKMUdqbnkySmdDK3JVWUZiRGtpUGx1K09IYnRnOXFYcGJMSHBzUVpsMGhucDBYSFNYVm9CMUliQndnMGEyOFVadApCbFBtWmc2d1BRS0JnRHVIUVV2SDZHYTNDVUsxNFdmOFhIcFFnMU16M2VvWTBPQm5iSDRvZUZKZmcraEppSXlnCm9RN3hqWldVR3BIc3AyblRtcHErQWlSNzdyRVhsdlhtOElVU2FsbkNiRGlKY01Pc29RdFBZNS9NczJMRm5LQTQKaENmL0pWb2FtZm1nZEN0ZGtFMXNINE9MR2lJVHdEbTRpb0dWZGIwMllnbzFyb2htNUpLMUI3MkpBb0dBUW01UQpHNDhXOTVhL0w1eSt5dCsyZ3YvUHM2VnBvMjZlTzRNQ3lJazJVem9ZWE9IYnNkODJkaC8xT2sybGdHZlI2K3VuCnc1YytZUXRSTHlhQmd3MUtpbGhFZDBKTWU3cGpUSVpnQWJ0LzVPbnlDak9OVXN2aDJjS2lrQ1Z2dTZsZlBjNkQKckliT2ZIaHhxV0RZK2Q1TGN1YSt2NzJ0RkxhenJsSlBsRzlOZHhrQ2dZRUF5elIzT3UyMDNRVVV6bUlCRkwzZAp4Wm5XZ0JLSEo3TnNxcGFWb2RjL0d5aGVycjFDZzE2MmJaSjJDV2RsZkI0VEdtUjZZdmxTZEFOOFRwUWhFbUtKCnFBLzVzdHdxNWd0WGVLOVJmMWxXK29xNThRNTBxMmk1NVdUTThoSDZhTjlaMTltZ0FGdE5VdGNqQUx2dFYxdEYKWSs4WFJkSHJaRnBIWll2NWkwVW1VbGc9Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K"
:
v1
kind
:
Service
metadata
:
name
:
my-nginx
labels
:
run
:
my-nginx
spec
:
type
:
NodePort
ports
:
-
port
:
8080
:
80
protocol
:
TCP
name
:
http
-
port
:
443
protocol
:
TCP
name
:
https
selector
:
run
:
my-nginx
---
:
apps/v1
kind
:
metadata
:
name
:
my-nginx
spec
:
selector
:
:
run
:
my-nginx
replicas
:
1
templa

te
:
metadata
:
labels
:
run
:
my-nginx
spec
:
volumes
:
-
name
:
secret
:
:
-
name
:
configMap
:
name
:
:
-
name
:
image
:
ports
:
-
:
443
-
:
80
:
-
mountPath
:
name
:
-
mountPath
:
name
:
The
.
before
run
=
=
POD_IP
[
map
[
]]
...
-k
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
curlpod
replicas
:
1
template
:
metadata
:
labels
:
app
:
curlpod
spec
:
volumes
:
-
name
:
secret
:
:
:
-
name
:
curlpod
command
:
- sh
- -c
image
:
:
-
mountPath
:
name
:
app
=
curlpod
kubectl
exec
...
...
NodePort
,
5
spec:
ports:
nodePort:
31704
port:
8080
80
nodePort:
32453
port:
443
443
selector:
1
--
...
...
Type
of
my-nginx
NodePort
to
:
...
creates an ELB, which uses 

a (long)
to
...
...
2 - Using Source IP
NAT
Source NAT
Pod
VIP
Service
minikube
=
,
iptables
=
=
80
=
8080
ClusterIP
=
=
Never --rm
ip addr
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:0

0
…then use
wget
...
The
NodePort
Service:
=
=
80
=
8080
--type
=
NodePort
NODEPORT
=
$(
jsonpath
=
)
NODES
=
$(
jsonpath
=
)
for the
for
node in
$NODES
;
do
curl -s
$node
:
$NODEPORT
done
node2
node2
Visually:
.
If you set
Local
,
endpoint.
Set the
for
node in
$NODES
;
do
1
-s
$node
:
$NODEPORT
done
right
, which
does
Visually:
Ready
with
an
k

ubectl expose deployment source-ip-app --name
=
=
80
=
8080
--type
=
...
field to
Local
without
Visually:
:
32122
The
/healthz
app
=
Use
curl
/healthz
is
then use
curl
...
.
Forwarded
or
.
the
app
=
Read how to
Home
Blog
Training
Partners
Community
CC BY 4.0
ICP license: 京ICP备17074

Run a Stateless Application Using a Deployment | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernetes API 

Tasks
Debug Pods
Debug a Stateful

Set
Auditing
Run Jobs
TLS
Tutorials
Security

ty Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
kube-ap

iserver Configuration (v1)
Tasks
minikube
.
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
    nginx:
 OldReplicaSets:   &

lt;none&gt;
 No events.
where
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
replicas
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
4
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
ports
:
-
:
80
.
.
Feedback
Yes
No
.
or
.
Last modified February 10, 2021 at 9:33 PM PST:
remove command from output block (a1f5566dc0)
Home
Blog
Training
Partners
Community
CC BY 4.0

Deployment | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Reference
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
template
(
replicas
(int32)
(int32)
strategy
(string)
IntOrString is a type that can hold an int32 or a string. When use

d in JSON or YAML marshalling and unmarshalling, it produces or consumes the inner type. This allows you to have, for example, a JSON field that can accept a name or number.
(int32)
(int32)
paused
(boolean)
replicas
(int32)
(int32)
(int32)
(int32)
(int32)
(int32)
type
(Time)
(Time)
(string)
(string)
(int64)
DeploymentList is a list of De

ployments.
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(

in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Adding entries to Pod /etc/hosts with HostAliases | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Operator pa

ttern
Tasks
Debug Pods
Determine the Reason for 

Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod Security Standards at the Cluster L

evel
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
ku

be-apiserver Configuration (v1alpha1)
Tasks
=
wide
kubectl
exec
hosts
localhost
hosts
file.
foo.local
,
bar.local
to
127.0.0.1
and
,
to
10.1.2.3
:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
Never
:
-
ip
:
hostnames
:
-
-
-
ip
:
hostnames
:
-
-
:
-
name
:
cat-hosts
image
:
command
:
- cat
args
:
-
=
wide
The
hosts

hosts
Caution:
Feedback
Yes
No
.
or
.
Last modified April 19, 2022 at 2:50 PM PST:
revert changes from PR 32941 that merged before 1.24 release (28cea36e93)
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubelet Configuration (v1beta1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
Field
string
kind
string
providers
Field
string
kind
string
bool
string
new data.
string
address
string
port
int32
int32
tlsC

ertFile
string
string
[]string
string
bool
bool
Defaults:
webhook:
Defaults:
webhook:
int32
Default: 5
int32
int32
Default: 5
int32
bool
bool
int32
string
int32
string
[]string

int32
int32
int32
string
string
string
bool
string
string
Requires the CPUManager feature gate to be

 enabled.
string
string
none
string
container
pod
string
maxPods
int32
podCIDR
string
int64
string
runOnce
bool
bool
.
int32
int64
maxOpenFiles is Number

 of files that can be opened by Kubelet process.
string
int32
Default: 5
int32
bool
not
.
Default:
.
.
int32
Default: 0
.
int32
Default: 0
bool
bool
bool
int32

int32
bool
string
int32
Default: 5
Get
Cache
Watch
string
string
, e.g.:
1.16
.
string
to enforce
Refer to
string
to enforce
Refer to
[]string
none
,
pods
,
and
.
If
none
When
When
is set to true

.
Refer to
[]string
*
).
,
,
,
,
and
net.*
"
string
string
bool
logging
Refer to
Default:
bool
by the
.
If at least one node al

locatable parameter has a non-zero value, you will need
bool
bool
bool
float64
bool
tracing
bool
Field
string
kind
string
source
Field
name
string
[]string
Globs can be used in the domain, but not in the port or the path. 

Globs are supported
'.
gcr.io
.
string
args
[]string
env
Field
name
string
value
string
Field
enabled
bool
.
Field
x509
webhook
anonymous
Field
mode
and
Webhook
.
webhook
(Alias of
string
)
Field
enabled
bool
cacheTTL
Field
Field
string
MemoryReservation specif

ies the memory reservation of different types for each NUMA node
Field
numaNode
int32
limits
Field
string
(Alias of
string
)
Field
priority
int32
int64
Field
json
Field
bool
Field
format
string
text
verbosity
vmodule
options
Field
endpoint
string
int32
(Alias of
)
Loggin

gConfiguration
(Alias of
uint32
)
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Kubernetes Component SLI Metrics | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kubernetes_healthchecks_total{name="informer-sync",status="su

ccess",type="readyz"} 14
Feedback
Yes
No
.
or
.
Last modified November 08, 2022 at 11:24 AM PST:
Add documentation for Component SLIs feature (#37767) (1591d7d224)
Home
Blog
Training
Partners
Community
CC BY 4.0

PodDisruptionBudget | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
)
(string)
IfHealthyBudget policy means that running pods (status.phase="R

unning"), but not yet healthy can be evicted only if the guarded application is not disrupted (status.currentHealthy is at least equal to status.desiredHealthy). Healthy pods will be subject to the PDB for eviction.
type
(int64)
DisruptedPods contains information about pods whose eviction was processed by the API server eviction subresource handler but has not yet been observed by the 

PodDisruptionBudget controller. A pod will be in this map from the time when the API server processed the eviction request to the time when the pod is seen by PDB controller as having been marked for deletion (or after a timeout). The key in the map is the name of the pod and the value is the time when the API server processed the eviction request. If the deletion didn't occur and a pod is still there it will be removed from the list automatically by PodDisruptionBudget controller after some time. If everything goes smooth this map should be empty for the most of the time. Large number of entries in the map may indicate problems with pod deletions.
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
PodDisruptio

nBudget
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
Thanks for the feedback. If you

 have a specific, answerable question about how to use Kubernetes, ask it on
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

pha1)
Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm init phase | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Kubeadm
,
preflight
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patchtype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string t

hat can be used to determine which patches are applied first alpha-numerically.
certs
all
ca
apiserver
etcd-ca
etcd-peer
sa
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
help for ca
Synopsis
Options
--dry-run
-h, --help
Choose a specific Kuberne

tes version for the control plane.
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Generate the certificate for servin

g etcd
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Alpha Disclaimer: this comman

d is currently alpha.
Options
-h, --help
help for sa
all
all
admin
kubelet
scheduler
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
only
Synopsis
only
Options
--dry-

run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
all
apiserver
scheduler
Synopsis
Options
-h, --help
Synopsis
Examples
  
Options
Port for the API Ser

ver to bind to.
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Path to a directory that contains files named "target[suffix][+patchtype].extension". For example, "kube-apiserver0+merge.yaml" or just "etcd.json". "target" can be one of "kube-apiserver", "kube-controller-manager", "kube-scheduler", "etcd", "kubeletconfiguration". "patcht

ype" can be one of "strategic", "merge" or "json" and they match the patch formats supported by kubectl. The default "patchtype" is "strategic". "extension" must be either "json" or "yaml". "suffix" is an optional string that can be used to determine which patches are applied first alpha-numerically.
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
etcd
local
Synopsis
This command is not

 meant to be run on its own. See list of available subcommands.
Options
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
.
all
kubeadm
kubelet
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Upload the k

ubelet component config to a ConfigMap
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
Synopsis
Examples
  
Options
--dry-run
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Skip printing of the default bootstrap token generated by 'kubeadm in

it'.
all
phases.
Synopsis
Examples
Options
-h, --help
Synopsis
Examples
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
all
addon
all
coredns
Synopsis
Options
-h, --help
Synopsis
Options
--dry-run
-h, --help
--s

ervice-dns-domain string     Default: "cluster.local"
Synopsis
Options
--dry-run
-h, --help
Synopsis
Options
--dry-run
-h, --help
v1beta3
or
Feedback
Yes
No
.
or
.
Last modified January 18, 2022 at 6:08 PM PST:
kubeadm: update references of "master" label/taint for 1.24 (806518df13)
kubeadm ini

t phase preflight
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
Debug Pods
2:
3:
4:
5:
6:
7:
1 - Debug Pods
This is
not
this guide
.
${
POD_NAME
}
Running
Pending
hostPort
hostPort
hostPort
hostPort
Waiting
Waiting
.
command
as
commnd
option.
kube

ctl apply --validate -f mypod.yaml
.
command
as
commnd
46757
]
46757
]
false
and then
endpoints
${
}
...
spec
:
-
selector
:
name
:
nginx
type
:
frontend
=
name
=
=
frontend
Please see
Service
Endpoints
, and your
Pods
are
2 - Debug Services
Note:
kubectl
exec
Setup
=
registry.k8s.io/serve_ho

stname
kubectl
=
3
YAML:
:
apps/v1
kind
:
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
:
app
:
hostnames
replicas
:
3
template
:
metadata
:
labels
:
app
:
hostnames
spec
:
:
-
name
:
hostnames
image
:
app
=
hostnames
app
=
hostnames
\
=
for
do
wget -qO-
$ep
done
there.
=
80
=
9376
:
v1
kind
:
Service
metadata
:
labels
:
app
:
hostnames
name
:
hostnames
spec
:
selector
:
app
:
hostnames
ports
:
-
name
:
default
protocol
:
TCP
port
:
80
:
9376
traffic to
One of 

the most common ways that clients consume a Service is through a DNS
name.

Note:

The
kubelet
with the
flag.
The
search
kubelet
with the
commands.
The
options
ndots

section
above).
for
i in
$(
seq
1
3
)
;
do
done
It m

ight sound silly, but you should really double and triple check that your
{
"kind"
:
"Service"
,
:
"v1"
,
: {
"name"
:
,
:
"default"
,
"uid"
:
,
:
"347189"
,
:
,
"labels"
: {
"app"
:
}
},
"spec"
: {
"ports"
: [
{
"name"
:
"default"
,
:
"TCP"
,
"port"
:
80
,
:
9376
,
:
0
}
],
: {
"app"
:
},
:
,
"type"
:
,
:
"None"
},
"status"
: {
: {}
}
}
?
Is the
protocol
app
=
hostnames
The
debug pods
.
ENDPOINTS
column is
the
, but the
the
above.
Note:
for
do
wget -qO-
$ep
done
The default implementation of Services, and the one used on most clust

ers, is
required
conntrack
conntrack
on Ubuntu)
line
-A KUBE-SVC-NWV5X2332I4OT4T3 -m comment --comment "default/hostnames:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-X3P2623AG

DH6CDF3
and
one
and one
chain with
IPVS mode
...
...
,
,
).
with the
-v
iptables
Kubelet
exposes a
flag
or
.
Confirm
is set to
or
.
is set to
match
hairpin
Kubelet
has
/sys
for
do
cat
$intf
done
1
1
1
1
, ensure
Kubelet
cbr0
bridge is
used and configured p

roperly, you should see:
Seek help
Slack
or
Forum
or
GitHub
.
Visit the
3 - Debug a StatefulSet
=
MyApp
Unknown
or
task for
guide.
.
4 - Determine the Reason for Pod Failure
.
minikube
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
debian
command
:
[
"/bin/sh"
]
args
:
[
"-c"
,
]
command
and
args
the
=
yaml
:
v1
kind
:
Pod
...
lastState
:
:
:
...
exitCode
:
0
:
...
message
:
|
...
=
=
'{{range .status.containerStatuses}}{{printf "%s:\n%s\n\n" .name .lastState.

terminated.message}}{{end}}'
value of
or
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
debian
:
File
to "
See the
field in
Container
.
.
.
5 - Debug Init Containers
and
.
minikube
.
.
.
Init:1/2
See
    ...
    ...
    ...
    ...
You can also access the Init Container statuses programm

atically by reading the
set -x
Init:
Status
Meaning
Init:N/M
The Pod has
M
N
Pending
or
Running
6 - Debug Running Pods
Your
Pod
Debugging
Pods
.
kubectl
.
Using
:
apps/v1
kind
:
metadata
:
name
:
spec
:
selector
:
:
app
:
nginx
replicas
:
2
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"128Mi"
cpu
:
"500m"
ports
:
-
:
80
IPs:
  nginx:
    Limits:
      cpu:     500m
    Requests:
      cpu:        500m
    Mounts:
  Ready             True 
Volumes:
    Type:                    Projected (a volume that contains injected data from multiple sour

ces)
Events:
  Node:		/
  IP:
  Controllers:	ReplicaSet/nginx-deployment-1370807

587
    nginx:
      QoS Tier:
      Limits:
        cpu:	1
      Requests:
        cpu:	1
  Volumes:
  Events:
=
argument.
-o yaml
:
v1
kind
:
Pod
metadata
:
:
:
labels
:
app
:
nginx
:
name
:
namespace
:
default
:
-
:
apps/v1
:
true
:
true
kind
:
name
:
uid
:
:
"1364"
uid
:
spec
:
:
-
image
:
nginx
:
Always
name
:
nginx
ports
:
-
:
80
protocol
:
TCP
resources
:
limits
:
cpu
:
500m
memory
:
128Mi
requests
:
cpu
:
500m
memory
:
128Mi
:
:
File
:
-
mountPath
:
name
:
readOnly
:
true
dnsPolicy
:
:
true
nodeName
:
:
priority
:
0
:
Always
:
:
{}
:
default
:
default
:
30
:
-
effect
:
NoExecute
key
:
operator
:
Exists
:
300
-
effect
:
NoExecute
key
:
operator
:
Exists
:
300
volumes
:
-
name
:
kube-api-a

ccess-bgsgp
projected
:
:
420
sources
:
-
:
:
3607
path
:
token
-
configMap
:
items
:
-
key
:
ca.crt
path
:
ca.crt
name
:
-
:
items
:
-
fieldRef
:
:
v1
fieldPath
:
path
:
namespace
status
:
:
-
:
null
:
status
:
"True"
type
:
-
:
null
:
status
:
"True"
type
:
Ready
-
:
null
:
status
:
"True"
type
:
-
:
null
:
status
:
"True"
type
:
:
-
:
image
:
imageID
:
lastState
:
{}
name
:
nginx
ready
:
true
:
0
started
:
true
state
:
running
:
startedAt
:
hostIP
:
phase
:
Running
podIP
:
10.88.0.3
podIPs
:
-
ip
:
10.88.0.3
-
ip
:
2001
:db8::1
qosClass
:
startTime
:
${
POD_NAME
}
${
}
${
POD_NAME
}
${
}
If the
includes
:
kubectl
exec
${
POD_NAME
}
-c
${
}
--
${
CMD
}
${
ARG1
}
${
ARG2
}
...
${
ARGN
}
Note:
kubectl
exec
-i
and
-t
kubectl
exec
.
.
=
=
Never
pause
images.
kubectl
exec
. If you
-i
/
argument,
kubectl
=
=
/ #
--target
creates.
Note:
The
--target
parameter must be supp

orted by the
ps
:
...
...
Use
to create a
busybox
busybox
:
=
=
myapp
named
that adds a
=
=
Note:
flag.
The
-i
.
The
works, see
.
  myapp:
    ...
    Args:
      false
You can use
/ #
Note:
or
create a new container to run the command you speci

fied.
The
-i
.
:
Now use
to
ubuntu
:
syntax as
.
*=ubuntu
to
ubuntu
.
, run:
=
ubuntu
/host
.
and
will fail.
7 - Get a Shell to a Running Container
minikube
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
volumes
:
-
name
:
emptyDir
:
{}
:
-
name
:
nginx
image
:
nginx
:
-
name
:
mountPath
:
:
true
dnsPolicy
:
Default
kubectl
exec
Note:
--
ls /
ls /
tcpdump
lsof
ps aux
has an
emptyDir
at
.
file in the
/usr/s

hare/nginx/html
echo
file:
exit
.
exit
kubectl
exec
kubectl
exec
kubectl
exec
kubectl
exec
or
-c
to
named
main-app
and
main-app
kubectl
exec
Note:
-i
and
-t
--stdin
and
--tty
Read about
Home
Blog
Training
Partners
Community
CC BY 4.0

Share Process Namespace between Containers in a Pod | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
minikube
field of
.spec
:
v1
kind
:
Pod
metadata
:
name
:
nginx
spec
:
:
true
:
-
name
:
nginx
image
:
nginx
-
name
:
shell
image
:
:
:
add
:
stdin
:
true
tty
:
true
nginx
shell
ps
:
ps ax
SIGHUP
to
nginx
kill
-HUP
8
ps ax
It's even possible to access the file system of another container using th

e
link.

events {
systemd
) or run
kill -HUP 1
kill -HUP 1
(
/pause
/proc
link.
Feedback
Yes
No
.
or
.
Last modified March 13, 2022 at 1:59 PM PST:
Tweak the share process namespace page (060b924df8)
Home
Blog
Training
Partners
Community
CC BY 4.0

Example: Deploying WordPress and MySQL with Persistent Volumes | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tutorials
A
. A
Warning:
Note:
Create a
with
Clean up
minikube
.
kubectl
When a PersistentVolumeClaim is

 created, a PersistentVolume is dynamically provisioned based on the StorageClass configuration.
Warning:
hostPath
hostPath
hostPath
/tmp
Note:
hostPath
Note:
this guide
.
A
Secret
kubectl
.
cat
literals:
EOF
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
ports
:
-
port
:
3306
selector
:
app
:
wordpress
tier
:
mysql
clusterIP
:
None
---
:
v1
kind
:
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
:
resources
:
requests
:
storage
:
20Gi
---
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
selector
:
:
app
:
wordpress
tier
:
mysql
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
wordpress
tier
:
mysql
spec
:
:
-
image
:
mysql:5.6
name
:
mysql
env
:
-
name
:
valueFrom
:
:
name
:
key
:
password
ports
:
-
:
3306
name
:
mysql
:
-
name
:
mountPath
:
volumes
:
-
name
:
:
claimName
:
:
v1
kind
:
Service
metadata
:
name
:
wordpress
labels
:
app
:
wordpress
spec
:
ports
:
-
port
:
80
selector
:
app
:
wordpress
tier
:
frontend
type
:
---
:
v1
kind
:
metadata
:
name
:
labels
:
app
:
wordpress
spec
:
:
resources
:
requests
:
storage
:
20Gi
---
:
apps/v1
kind
:
metadata
:
name
:
wordpress
labels
:
app
:
wordpress
spec
:
selector
:
:
app
:
wordpress
tier
:
frontend
strategy
:
type
:
Recreate
template
:
metadata
:
labels
:
app
:
wordpress
tier
:
frontend
spec
:
:
-
image
:
name
:
wordpress
env
:
-
name
:
value
:
-
name
:
valueFrom
:
:
name
:
key
:
password
ports
:
-
:
80
name
:
wordpress
:
-
name
:
mountPath
:
vol

umes
:
-
name
:
:
claimName
:
Add them to
file.
cat
EOF
The
1
9s
Note:
Note:
RUNNING
.
Note:
NodePort
Warning:
Jobs
Feedback
Yes
No
.
or
.
Last modified May 11, 2022 at 7:02 PM PST:
Update chart link (e21eafb9e8)
Home
Blog
Training
Partners
Community
CC BY 4.0
Trademark Usa

Expose Pod Information to Containers Through Environment Variables | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
.
minikube
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
sleep 10;
done;
env
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
MY_POD_IP
valueFrom
:
fieldRef
:
fieldPath
:
-
name
:
valueFrom
:
fieldRef
:
fieldPath
:
:
Never
env
Note:
kubectl apply -f ht

tps://k8s.io/examples/pods/inject/dapi-envars-pod.yaml
minikube
default
default
command
and
args
fields
kubectl
exec
printenv
...
...
...
container
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
-
name
:
image
:
command
:
[
"sh"
,
"-c"
]
args
:
sleep 10;
done;
resources
:
requests
:
memory
:
"32Mi"
cpu
:
"125m"
limits
:
memory
:
"64Mi"
cpu
:
"250m"
env
:
-
name
:
valueFrom
:
:
:
resource
:
-
name
:
valueFrom
:
:
:
resource
:
-
name
:
valueFrom
:
:
:
resource
:
-
name
:
valueFrom
:
:
:
resource
:
:
Never
env
1
1
33554432
67108864
Read
Read the
spec
that you
PodSpec
Container
EnvVar
Feedback
Yes
No
.
or
.
Last modified March 21, 2022 at 10:29 PM PST:
Revise downward API documentation (7f3604a949)

Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm upgrade | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Kubeadm
.
You can use
and
will also
.
Note:
and
--config
Synopsis
Options
-h, --help
The kubeconfig file to use when talking to the cluster. If the flag is not set, a 

set of standard locations can be searched for an existing kubeconfig file.
Synopsis
Options
--dry-run
-h, --help
-y, --yes
Synopsis
Options
-h, --help
--scheduler-manifest string     Default: "/etc/kubernetes/manife

sts/kube-scheduler.yaml"
Synopsis
Options
--dry-run
-h, --help
Feedback
Yes
No
.
or
.
Last modified October 15, 2021 at 3:07 PM PST:
fixed a typo (cbd7a7ab74)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install To

Tasks
TLS
TLS
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure a Pod to Use a PersistentVolume for Storage | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
O

on (v1)
Tasks
kubectl
Minikube
.
.
.
/mnt/data
In the
/mnt/data
file:
sudo sh -c
Note:
sudo
, you can
sudo
hostPath
use
to set up
.
pods/storage/pv-vol

ume.yaml
:
v1
kind
:
metadata
:
name
:
labels
:
type
:
local
spec
:
:
manual
capacity
:
storage
:
10Gi
:
hostPath
:
path
:
/mnt/data
on the
manual
STATUS
of
Available
. This
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
3Gi
STATUS
of
Bound
.
.
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
volumes
:
-
name
:
:
claimName
:
:
-
name
:
image
:
nginx
ports
:
-
:
80
name
:
:
-
mountPath
:
name
:
kubectl
exec
apt update
ind

ex.html
file on the
Clean up
:
v1
kind
:
Pod
metadata
:
name
:
test
spec
:
:
-
name
:
test
image
:
nginx
:
-
name
:
config
mountPath
:
subPath
:
html
-
name
:
config
mountPath
:
subPath
:
volumes
:
-
name
:
config
:
claimName
:
Use the
:
v1
kind
:
metadata
:
name
:
pv1
:
:
"1234"
Note:
.
Read the
.
Reference
Feedback
Yes
No
.
or
.
Last modified September 21, 2022 at 1:32 PM PST:
Link to Dynamic Provisioning doc instead of blog (4be5a30965)
Clean up
Reference
Home
Blog
Training
Partners
Community
CC BY 4.0

Force Delete StatefulSet Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
never
at most one
must not
specify a
timeout
.
).
Force deletion of th

e Pod by the user.
do not
=
0
--force
--force
=
0
Unknown
.
Feedback
Yes
No
.
or
.
Last modified June 24, 2022 at 9:58 PM PST:
Update content/en/docs/tasks/run-application/force-delete-stateful-set-pod.md (7055a7efb3)
Home
Blog
Training
Partners
Community
CC BY 4.0

RoleBinding | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Instal

Reference
kind
metadata
(
)
roleRef
subjects
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in q

uery
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): s

tring
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Protocols for Services | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Service
,
SCTP
TCP
UDP
HTTP
and
HTTPS
TLS
protocol
SCTP
TCP
UDP
Services,
HTTP
TCP
and add an
Note:
Ingress
to expose
http
or
https
. Use
http
https
if the

.
TCP
.
TLS
as
TCP
Feedback
Yes
No
.
or
.
SCTP
TCP
UDP
HTTP
TLS
Home
Blog
Training
Partners
Community
CC BY 4.0

LimitRange | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install 

Reference
: v1
kind
metadata
(
)
spec
(
)
limits
Quantity
)
Quantity
)
Quantity
)
Quantity
)
Quantity
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespa

ce
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
If you plan to report an issue with this page, mention t

hat the page is auto-generated in your issue description. The fix may need to happen elsewhere in the Kubernetes project.
Feedback
Yes
No
.
or
.
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

PersistentVolumeClaim | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
: v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
)
resources
Quantity
)
reso

urces.requests
Quantity
)
(string)
(string)
(string)
Beta level
(
)
**
(string)
(string)
accessModes contains the actual access modes the volume bac

king the PVC has. More info:
Quantity
)
capacity
Quantity
)
type
(Time)
(Time)
(string)
(string)
phase
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace

(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 

(
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Beta level
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

CronJob | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

lpha1)
Reference
CronJob
CronJob
CronJob
: batch/v1
kind
: CronJob
metadata
(
)
spec
(
)
status
(
)
(
)
(
JobSpec
)
schedule
.
timeZone
(string)
(string)
Specifies how to treat concurrent executions of a Job. Valid values are: - "Allow" (default): all

ows CronJobs to run concurrently; - "Forbid": forbids concurrent runs, skipping next run if previous run hasn't finished yet; - "Replace": cancels currently running job and replaces it with a new one
(int64)
suspend
(boolean)
(int32)
(int32)
active
([]
)
(Time)
(Time)
: batch/v1
kind
metadata
(
ListMeta
)
items
([]
CronJob
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
fie

ldManager
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
202 (
CronJob
update
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
suggest an

 improvement
.
CronJob
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

StorageClass | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Inst

Reference
kind
metadata
(
)
(boolean)
allowedTopologies.matchLabelExpressions
(string)
(string)
VolumeBindingMode indicates h

ow PersistentVolumeClaims should be provisioned and bound. When unset, VolumeBindingImmediate is used. This field is only honored by servers that enable the VolumeScheduling feature.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
This page is automatically

 generated.
Feedback
Yes
No
.
or
.
get
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Configure Quotas for API Objects | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
object.
minikube
.
:
v1
kind
:
metadata
:
name
:
spec
:
hard
:
:
"1"
:
"2"
:
"0"
=
=
=
yaml
status
:
hard
:
:
"1"
:
"2"
:
"0"
used
:
:
"0"
:
"0"
:
"0"
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
3Gi
=
=
NA

ME             STATUS
:
v1
kind
:
metadata
:
name
:
spec
:
:
manual
:
resources
:
requests
:
storage
:
4Gi
=
Notes
by quotas:
String
API Object
"pods"
Pod
Service
"secrets"
Secret
ConfigMap
Clean up
Feedback
Yes
No
.
or
.
Notes
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

kubeadm certs | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

 (v1)
Reference
Kubeadm
.
overview
Synopsis
Options
-h, --help
all
.
renew
all
apiserver
etcd-peer
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Renewal by default tries to use the certificate authority in the local PKI managed by kubeadm; as alternative it is possible to use K8s certificate API for certificate renewal, or as a last option, 

to generate a CSR request.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
After renewal, in order to make changes effective, is required to restar

t control-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
After renewal, in order to make changes effective, is required to restart contr

ol-plane components and eventually re-distribute the renewed certificate in case the file is used elsewhere.
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
Options
-h, --help
Synopsis
kubeadm certs rene

w scheduler.conf [flags]
Options
-h, --help
to
and
Synopsis
Options
-h, --help
.
Synopsis
Options
-h, --help
Synopsis
Examples
Options
-h, --help
to revert any chan

ges made to this host by
or
Feedback
Yes
No
.
or
.
Last modified November 12, 2020 at 9:28 PM PST:
kubeadm: promote the "kubeadm certs" command to GA (#24410) (d0c6d303c3)
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
1.1:
Pod
1.2:
1.3:
1.4:
1.5:
1.6:
1.7:
1.8:
DaemonSet
1.9:
Job
1.10:
CronJob
1.11:
1.12:
1.13:
1.14:
1.15:
1.16:
1.17:
2:
2.1:
Service
2.2:
Endpoints
2.3:
2.4:
Ingress
2.5:
3:
3.1:
ConfigMap
3.2:
Secret
3.3:
Volume
3.4:
3.5:
3.6:
3.7:
3.8:
CSIDriver
3.9:
CSINode
3.10:
4:
4.1:
4.2:
4.3:
4.4:
5:
5.1:
5.2:
5.3:
5.4:
5.5:
5.6:
5.7:
5.8:
Role
5.9:
6:
6.1:
6.2:
6.3:
6.4:
7:
7.1:
7.2:
7.3:
7.4:
8:
8.1:
Node
8.2:
Namespace
8.3:
Event
8.4:
8.5:
Lease
8.6:
8.7:
8.8:
8.9:
Binding
8.10:
8.11:
9:
9.1:
9.2:
9.3:
ListMeta
9.4:
9.5:
9.6:
9.7:
9.8:
9.9:
Patch
9.10:
Quantity
9.11:
9.12:
Status
9.13:
10:
10.1:
11:
kubectl
1 - Workload Resources
1.1 - Pod
Pod
: v1
kind
: Pod
metadata
(
)
spec
(
PodSpec
)
status
(
PodStatus
)
PodSpec
([]
Container
name
([]
Container
)
Patch strategy: merge on k

ey
name
([]
)
name
([]
)
name
(boolean)
os
(PodOS)
os.name
Volumes
volumes
([]
Volume
)
name
nodeName

(string)
affinity
(
)
(
)
(
)
.
(string)
(string)
(string)
(string)
(int64)
(string)
(string)
(string)
priority
(int32)
(string)
TopologySpreadC

onstraint specifies how to spread matching pods among the given topology.
skew.
more
(
)
(int32)
MinDomains indicates a minimum number of eligible domains. When the number of eligible domains with matching topology keys is less than minDomains, Pod Topology Spread treats "global minimum" as 0, and then the calculation of Skew is performed. And when the number of eligible domains with matching topology keys equals or greater than minDomains, this value has no effect on scheduling. As a result,

 when the number of eligible domains is less than minDomains, scheduler won't schedule more than maxSkew Pods to those domains. If value is nil, the constraint behaves as if MinDomains is equal to 1. Valid values are integers greater than 0. When value is not nil, WhenUnsatisfiable must be DoNotSchedule.
(string)
(string)
overhead
Quantity
)
Lifecycle
(string)
(int64)
(int64)
https://git.k8s.io/enhance

ments/keps/sig-network/580-pod-readiness-gates
hostname
(string)
(boolean)
subdomain
(string)
ip
(string)
dnsConfig
(string)
Required.
(string)
dnsPolicy
(string)
(boolean)
hostPID
(boolean)
hostIPC
(boolean)
(boolean)
(string)
(boolean)
AutomountServiceAccountToken indicates whether a servic

e account token should be automatically mounted.
(int64)
(boolean)
(int64)
([]int64)
(int64)
(string)
(string)
localhostProfile indicates a profile defined in a file on the n

ode should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must only be set if type is "Localhost".
(string)
(string)
(string)
(string)
(string)
(string)
(boolean)
(string)
hostUsers
(boolean)
([]PodR

esourceClaim)
name
(string)
(string)
name
.
(string)
Container
name
Image
image
(string)
(string)
command
Entrypoint array. Not executed within a shell. The container image's ENTRYPOINT is used if this is not provided. Variable references $(VAR_NAME) are expanded using the container's environment. If a variable cannot be resolved, the reference in the input string will be unchanged. Double $$ are reduced to a single $, which allows for esc

aping the $(VAR_NAME) syntax: i.e. "$$(VAR_NAME)" will produce the string literal "$(VAR_NAME)". Escaped references will never be expanded, regardless of whether the variable exists or not. Cannot be updated. More info:
args
(string)
Ports
ports
(string)
(int32)
(string)
(string)
env
name
env.name
env.value
(string)
(string)
(boolean)
(
Obj

ectFieldSelector
)
,
(
)
(string)
(boolean)
envFrom
(string)
(boolean)
(string)
(string)
(boolean)
Volumes
mountPath
(string)
(boolean)
(string)
(string)
volumeDevice describes a mapping of a raw blo

ck device within a container.
Resources
resources
Quantity
)
Quantity
)
Lifecycle
lifecycle
(
)
(
)
(string)
(string)
Indicate how the termination message should be populated. File will use the contents of terminationMessagePath to populate the container status message on both success and failure. FallbackToLogsOnError will use the last chunk of container log output if the termination message file is empty and the container exited with an error. The log output is l

imited to 2048 bytes or 80 lines, whichever is smaller. Defaults to File. Cannot be updated.
(
Probe
)
(
Probe
)
(
Probe
)
(int64)
(boolean)
(int64)
(boolean)
(string)
(boolean)
(boolean)
security

Context.capabilities
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(boolean)
(string)
The UserName in Windows to run the entrypoint of the container process. Defaults to the user specified in image metadata if unspecified. May also be set in PodSecurityCont

ext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence.
Debugging
stdin
(boolean)
stdinOnce
(boolean)
tty
(boolean)
name
(string)
Image
image
(string)
(string)
command
args
(string)
Container's working directory. If not spe

cified, the container runtime's default will be used, which might be configured in the container image. Cannot be updated.
env
name
env.name
env.value
(string)
(string)
(boolean)
(
)
,
(
)
(string)
(boolean)
envFrom
(string)
(boolean)
(string)
(string)
Name of th

e referent. More info:
(boolean)
Volumes
mountPath
(string)
(boolean)
(string)
(string)
Lifecycle
(string)
(string)
Debugging
stdin
(boolean)
stdinOnce
(boolean)
tty
(boolean)
Optional: SecurityContext defines the security options the ephemeral container should be run with. If set, the fields of SecurityContext override the equivalent fields of PodSecurityContex

t.
(int64)
(boolean)
(int64)
(boolean)
(string)
(boolean)
(boolean)
(string)
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must only be set if type i

s "Localhost".
(string)
(string)
(string)
(string)
(string)
(string)
(boolean)
(string)
ports
(string)
(int32)
(string)
(string)
resources
Claims lists the names of 

resources, defined in spec.resourceClaims, that are used by this container.
Quantity
)
Quantity
)
lifecycle
(
)
(
)
(
Probe
)
(
Probe
)
(
Probe
)
exec
httpGet
(string)
Host name to connect to, defaults to the pod IP. You proba

bly want to set "Host" in httpHeaders instead.
(string)
(string)
tcpSocket
(string)
([]
)
([]
)
([]
Nod

eSelectorRequirement
)
([]
)
(
)
(
)
matches that 

of any node on which a pod of the set of pods is running
(
)
(
)
(
)
(
)
namespaces specifies a static list of namespace names that the term applies to. The term is applied to the union of the namespa

ces listed in this field and the ones selected by namespaceSelector. null or empty namespaces list and null namespaceSelector means "this pod's namespace".
(
)
(
)
Probe
exec
httpGet
(string)
HTTPHeader describes a custom header to be used in HTTP p

robes
(string)
(string)
tcpSocket
(string)
(int32)
(int64)
(int32)
(int32)
(int32)
(int32)
grpc
**
grpc.port
(string)
.
PodStatus
(string)
hostIP
(string)
startTime
(Time)
RFC 3339 date and time at which the obj

ect was acknowledged by the Kubelet. This is before the Kubelet pulled the container image(s) for the pod.
phase
(string)
More info:
message
(string)
reason
(string)
podIP
(string)
podIPs
([]PodIP)
ip
podIPs.ip
(string)
type
(Time)
(Time)
(string)
(string)
qosClass
(string)
The list has one entry per init container in the

 manifest. The most recent successful init container will have ready = true, the most recently started container will have startTime set. More info:
.
(string)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
ContainerStateTerminated is a terminated st

ate of a container.
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(boolean)
.
(string)
(Time)
(string)
(Time)
Time at which previous execution of the conta

iner started
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(boolean)
ep

hemeralContainerStatuses
.
(string)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
e

phemeralContainerStatuses.lastState.terminated.containerID
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(boolean)
PodList
items
([]
Pod
(string)
kind
(string)
metadata
(
ListMeta
)
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
name
(
in path
namespace
(
in path
namespace
container
(
in query
): string
The container for which to stream logs. Defaults to only container if there i

s one container in the pod.
follow
(
in query
): boolean
(
in query
): boolean
(
in query
): integer
pretty
(
in query
): string
pretty
previous
(
in query
): boolean
(
in query
): integer
tailLines
(
in query
): integer
(
in query
): boolean
Response
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
create
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
202 (
Pod
update
name
(
in path
namespace
(
in path

namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Pod
): OK
202 (
Pod
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer

nds
Response
200 (
Status
): OK
1.2 - PodTemplate
: v1
kind
metadata
(
)
template
(
)
metadata
(
)
spec
(
PodSpec
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
401: Unautho

rized
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.3 - ReplicationController
: v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
template
(
)
replicas
(int32)
https://kubernetes.io/docs/concepts/workloads/controllers/rep

licationcontroller#what-is-a-replicationcontroller
(int32)
replicas
(int32)
(int32)
(int32)
type
(Time)
(string)
(string)
(int64)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
Parame

ters
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection of Replica

tionController
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.4 - ReplicaSet
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
template
(
)
replicas
(int32)
(int32)
replicas
(int32)
(int32)
(int32)
type
condi

tions.lastTransitionTime
(Time)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
PUT /apis/apps/v1/namespaces/{namespace}

/replicasets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.5 - Deployment
: apps/v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
template
(
replicas
(int32)
(int32)
Minimum number of seconds for which a newly created pod should be ready without any of its co

ntainer crashing, for it to be considered available. Defaults to 0 (pod will be considered available as soon as it is ready)
strategy
(string)
(int32)
(int32)
paused
(boolean)
replicas
(int32)
(int32)
(int32)
(int32)
(int32)

(int32)
type
(Time)
(Time)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
up

date
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.6 - StatefulSet
: apps/v1
kind
metadata
(
)
https://git.k8s.io/community/contributors/devel/sig-a

rchitecture/api-conventions.md#metadata
spec
(
)
status
(
)
selector
(
template
(
replicas
(int32)
(string)
(int32)
(string)
, where pods are created in increasing order (pod-0, then pod-1, etc) and the controller will wait until each pod is ready before continuing. When scaling do

wn, the pods are removed in the opposite order. The alternative policy is
Parallel
(int32)
([]
)
(int32)
(string)
Retain
Delete
(string)
Retain
Delete
ordinals
(int32)
replicas
(int32)
(int32)
(int32)
updatedReplicas is the number

 of Pods created by the StatefulSet controller from the StatefulSet version indicated by updateRevision.
(int32)
(int32)
type
(Time)
(string)
(string)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
timeo

utSeconds
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
timeoutSec

onds
Response
200 (
Status
): OK
1.7 - ControllerRevision
: apps/v1
kind
metadata
(
)
revision
data
}
}
}
}
{
},
}
: apps/v1
kind
metadata
(
ListMeta
)
More info:
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): O

K
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.8 - DaemonSet
DaemonSet
: apps/v1
kind
metadata
(
)
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventio

ns.md#metadata
spec
(
)
status
(
)
selector
(
template
(
(int32)
(string)
The maximum number of DaemonSet pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of total number of DaemonSet pods at the start of the update (ex: 10%). Absolute number is calculated from percentage by rounding up. This cannot be 0 if MaxSurge is 0 Default value is 1. Example: when this is set to 30%, at most 30% of the total number of nodes that should be running the daemon pod (i.e. status.desiredNumberScheduled) can have their pods stopped for an upda

te at any given time. The update starts by stopping at most 30% of those DaemonSet pods and then brings up new DaemonSet pods in their place. Once the new pods are available, it then proceeds onto other DaemonSet pods, thus ensuring that at least 70% of original number of DaemonSet pods are available at all times during the update.
(int32)
(int32)
(int32)
(int32)
(int32)
type
(Time)
(string)
(string)
(int64)
: apps/v1
kind
metadata
(
ListMeta
)
items
([]
DaemonSet
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
get
Par

ameters
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
202 (
DaemonSet
update
name
(
in path
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
DaemonSet
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in quer

y
): string
pretty
Response
200 (
DaemonSet
): OK
201 (
DaemonSet
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.9 - Job
Job
: batch/v1
kind
: Job
metadata
(
)
spec
(
JobSpec
)
status
(
JobStatus
)
JobSpec
Replicas
template
(
(int32)
Lifecycle
(int32)
(string)
Indexed
.
Indexed
Indexed
.
Mor

e completion modes can be added in the future. If the Job controller observes a mode that it doesn't recognize, which is possible during upgrades due to version skew, the controller skips updates for the Job.
(int32)
(int64)
(int32)
suspend
(boolean)
Selector
selector
(
)
(boolean)
Specifies the action taken on a pod failure when the requirements are satisfi

ed. Possible values are:
(string)
JobStatus
startTime
(Time)
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the fac

tory methods that the time package offers.
(Time)
active
(int32)
failed
(int32)
succeeded
(int32)
(string)
type
(Time)
(Time)
(string)
(string)
counter.
Beta level
ready
(int32)
JobList
: batch/v1
kind
: JobList
metadata
(
ListMeta
)
https://git.k8s.io/community/contributors/devel/sig-architectur

e/api-conventions.md#metadata
items
([]
Job
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
JobList
): OK
create
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
202 (
Job
update
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Job
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
patch
name
(
in path

namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Job
): OK
201 (
Job
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.10 - CronJob
CronJob
: batch/v1
kind
: CronJob
metadata
(
)
spec
(
)
status
(
)
(
)
(
JobSpec
)
schedule
.
timeZone
(string)
https://kubernetes.io/docs/concepts/workloads/controllers/cron-

jobs/#time-zones
(string)
(int64)
suspend
(boolean)
(int32)
(int32)
active
([]
)
(Time)
(Time)
: batch/v1
kind
metadata
(
ListMeta
)
items
([]
CronJob
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
HTTP R

equest
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
202 (
CronJob
update
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
CronJob
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CronJob
): OK
201 (
CronJob
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.11 - HorizontalPodAutoscaler
kin

d
metadata
(
)
spec
(
)
.
status
(
)
(string)
(int32)
(int32)
(int32)
(Time)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path

d
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string

(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.12 - HorizontalPodAutoscaler
kind
metadata
(
)
spec
(
)
.
status
(
)
(string)
(int32)
behavior
(HorizontalPodAutoscalerBeh

avior)
(string)
(int32)
(string)
selectPolicy is used to speci

fy which policy should be used. If not set, the default value Max is used.
(int32)
metrics
type
(int32)
(
Quantity
)
(
Quantity
)
external refers to a global metric that is not associated with any Kubernetes object. It allows autoscaling based on information coming from components running outside of cluster (for example length of queue in cloud messaging service, or QPS from loadbalancer running outside of cl

uster).
(
)
(int32)
(
Quantity
)
(
Quantity
)
(string)
(
)
(int32)
(
Quantity
)
(
Quanti

ty
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
(int32)
(
Quantity
)
(
Quantity
)
desiredReplicas is the desired number of replicas of pods managed by this autoscaler, as last calculated by the auto

scaler.
type
(Time)
(string)
(string)
(int32)
(
Quantity
)
(
Quantity
)

.averageUtilization
(int32)
(
Quantity
)
(
Quantity
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
(string)
(
)
(int32)
curre

ntAverageUtilization is the current value of the average of the resource metric across all relevant pods, represented as a percentage of the requested value of the resource for the pods.
(
Quantity
)
(
Quantity
)
(
)
(int32)
(
Quantity
)
(
Quantity
)
(int32)
(Time)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
GET /apis/autoscaling/v2/namespaces/{namespace}/horizontalpodautos

calers/{name}/status
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
HorizontalPodAutos

caler
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.13 - PriorityClass
kind
metadata
(
)
value
(string)
(boolean)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string

pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.14 - PodScheduling v1alpha1
kind
metadata
(
)
spec
(
status
(
)
Status desc

ribes where resources for the Pod can be allocated.
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
POST /apis/resource.k8s.io

/v1alpha1/namespaces/{namespace}/podschedulings
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
R

esponse
200 (
Status
): OK
1.15 - ResourceClaim v1alpha1
kind
metadata
(
)
spec
(
status
(
)
(string)
(string)
([]
)
([]
)
(string)
(boolean)
Sha

reable determines whether the resource supports more than one consumer at a time.
(boolean)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in p

ath
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.16 - ResourceClaimTemplate v

1alpha1
kind
metadata
(
)
spec
(
spec
(
metadata
(
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
): s

namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
1.17 - ResourceClass v1alpha1
kind
metadata
(
)
(string)
(string)
Namespace that contains the 

referenced resource. Must be empty for cluster-scoped resources and non-empty for namespaced resources.
([]
)
([]
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
prett

y
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2 - Service Resources
2.1 - Service
Service
: v1
kind
: Service
metadata
(
)
spec
(
)
status
(
)
selector
ports
port
(string)
(string)
The name of this port within the service. This must be a DNS_LABEL. All ports within a ServiceSpec must have unique names. When considering the endpoi

nts for a Service, this must match the 'name' field in the EndpointPort. Optional if only one ServicePort is defined on this service.
(int32)
(string)
type
(string)
(string)
clusterIP
(string)
clusterIP is the IP address of the service and is usually assigned randomly. If an address is specified manually, is in-range (as per system configuration), and is not in use, it will be allocated t

o the service; otherwise creation of the service will fail. This field may not be changed through updates unless the type field is also being changed to ExternalName (which requires this field to be blank) or the type field is being changed from ExternalName (in which case this field may optionally be specified, as describe above). Valid values are "None", empty string (""), or a valid IP address. Setting this to "None" makes a "headless service" (no virtual IP), which is useful when direct endpoint connections are preferred and proxying is not required. Only applies to types ClusterIP, NodePort, and LoadBalancer. If this field is specified when creating a Service of type ExternalName, creation will fail. This field will be wiped when updating a Service to type ExternalName. More info:
(string)
(string)
https://kubernetes.io/docs/tasks/access-application-

cluster/create-external-load-balancer/
(string)
(string)
type
(string)
(string)
(int32)
(boolean)
publishNotReadyAddresses indicates that any agent which deals with endpoints for this Service should disregard any indications of ready/not-ready. The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery. The Kubernetes controllers that generate Endpoints and EndpointSlice resources for Services interpret this to mean that all endpoints are considered "ready" even if the Pods themselves are not. Agents which consume only Kubernetes

 generated endpoints through the Endpoints or EndpointSlice resources can safely assume this behavior.
(int32)
(boolean)
type
(int64)
(string)
(string)
Ports is a list of records of service ports If used, every port defined in the service should have an entry in 

it
**
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Service
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
202 (
Service
update
name
(
in path
namespace
(
in path
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
update
name
(
in path
namespace
(
in path
): string, re

quired
namespace
body
:
Service
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Service
): OK
201 (
Service
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Service
): OK
202 (
Service
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2.2 - Endpoints
Endpoints
 Subsets: [
   {
   },
   {
   },
]
: v1
kind
metadata
(
)
subsets
{
Addresses: [{"ip": "

10.10.1.1"}, {"ip": "10.10.2.2"}],
}
(string)
(string)
(
)
(string)
(string)
(
)
(string)
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Endpoints
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
resou

rceVersionMatch
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
202 (
Endpoints
update
name
(
in path
namespace
(
in path
namespace
body
:
Endpoints
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Endpoints
): OK
201 (
Endpoints
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2.3 - EndpointSlice
kind
metadata
(
)
Standard object's metadata

.
endpoints
(boolean)
(boolean)
(boolean)
(string)
(string)
(
)
(string)
ports
ports specifies the list of network ports exposed by each endpoint in this slice. Each port must have a unique n

ame. When ports is empty, it indicates that there are no defined ports. When a port is defined with a nil port value, it indicates "all ports". Each slice may include a maximum of 100 ports.
(int32)
(string)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace

body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2.4 - Ingress
Ingress
kind
: Ingress
metadata
(
)
spec
(
)
status
(
)
(
)
(string)
annotation (simple constant name) was never formally defined, it was widely supported by Ingress contro

llers to create a direct binding between Ingress controller and Ingress resources. Newly created Ingress resources should prefer using the field. However, even though the annotation is officially deprecated, for backwards compatibility reasons, ingress controllers should still honor that annotation if present.
rules
(string)
2. The
:
/
?
(
(string)
Path is matched against the path of an incoming request. Currently it can contain characters disallowed from the conventional "path" part of a URL as defined by RFC 3986. Paths must begin with a '/' and must be present when using PathType with

 value "Exact" or "Prefix".
tls
tls.hosts
(string)
resource
(
)
service
(string)
(int32)
(string)
(string)
(string)
items
([]
Ingress
(string)
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schema

s to the latest internal value, and may reject unrecognized values. More info:
kind
(string)
metadata
(
ListMeta
)
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
202 (
Ingress
update
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Ingress
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
prett

y
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Ingress
): OK
201 (
Ingress
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
2.5 - IngressClass
kind
metadata
(
)
spec
(
)
(string)

 is a link to a custom resource containing additional configuration for the controller. This is optional if the controller does not require extra parameters.
(string)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
401: Unauth

orized
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3 - Config and Storage Resources
3.1 - ConfigMap
ConfigMap
: v1
kind
metadata
(
)
data
immutable
(boolean)
: v1
kind
metadata
(
ListMeta
)
More info:
items
([]
ConfigMap
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
create

namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
202 (
ConfigMap
update
name
(
in path
namespace
(
in path
namespace
body
:
ConfigMap
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
ConfigMap
): OK
201 (
ConfigMap
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.2 - Secret
Secret
: v1
kind
: Secret
metadata
(
)
data
immutable
(boolean)
stringData allows specifying non-binary secret data in string form. It is provided as a write-only input field for convenience. All keys and values are merged into the data field on write, overwriting any existing values. The stringData field is never output when reading from the AP

I.
type
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Secret
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
202 (
Secret
update
name
(
in path
namespace
(
in path
namespace
body
:
Secret
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Secret
): OK
201 (
Secret
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status

namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.3 - Volume
Volume
name
(boolean)
configMap
(string)
(boolean)
(int32)
([]
KeyToPath
)
secret
*Adapts a Secret into a

 volume.
(string)
(boolean)
(int32)
([]
KeyToPath
)
(int32)
([]
)
projected
(int32)
(string)
(boolean)
([]
KeyToPath
)
items if unspecified, each key-value pair 

in the Data field of the referenced ConfigMap will be projected into the volume as a file whose name is the key and content is the value. If specified, the listed keys will be projected into the specified paths, and unlisted keys will not be present. If a key is specified which is not present in the ConfigMap, the volume setup will error unless it is marked optional. Paths must be relative and may not contain the '..' path or start with '..'.
([]
)
(string)
(boolean)
([]
KeyToPath
)
(string)
(int64)
emptyDir
(string)
(
Quantity
)
sizeLimit is the total amount of local storage required for this EmptyDir volume. 

The size limit is also applicable for memory medium. The maximum usage on memory medium EmptyDir would be the minimum value between the SizeLimit specified here and the sum of memory limits of all containers in a pod. The default is nil which means that the limit is undefined. More info:
hostPath
(string)
(string)
(int32)
(boolean)
azureDisk
(string)
(string)
(string)
(boolean)
azureFile
A

zureFile represents an Azure File Service mount on the host and bind mount to the pod.
(boolean)
cephfs
(string)
(boolean)
(string)
(
)
(string)
cinder
(string)
(boolean)
(
)
csi
(string)
(
)
(boolean)
csi.v

olumeAttributes
ephemeral
where
not
(
(
)
fc
fc.fsType
(string)
fc.lun
(int32)
(boolean)
fc.wwids
flexVo

lume represents a generic volume resource that is provisioned/attached using an exec based plugin.
(string)
(boolean)
(
)
flocker
(string)
(string)
(string)
(int32)
(boolean)
iscsi
iscsi.iqn
iscsi.lun
targetPortal is iSCSI Target Portal. The

 Portal is either an IP or ip_addr:port if the port is other than default (typically TCP ports 860 and 3260).
(boolean)
(boolean)
(string)
(string)
(string)
(boolean)
(
)
nfs
nfs.path
(boolean)
(string)
(string)
(boolean)
quobyte
(string)
group to map volum

e access to Default is no group
(boolean)
(string)
(string)
rbd
rbd.image
(string)
(string)
rbd.pool
(string)
(boolean)
(
)
rbd.user
(string)
scaleIO
(
(string)
(string)
(boolean)
(boolean)
(string)
(string)
(string)
storageos
(string)
fsType is the filesystem type to mount. Must be a filesystem type supported by the host operating system. Ex. "ext4", "xfs", "ntfs". Implicitly inferred to be "ext4" if u

nspecified.
(boolean)
(
)
(string)
(string)
(string)
(string)
(string)
gitRepo
(string)
(string)
path
fieldRef
(
)
mode
(int32)
(
)
KeyToPath
key
path
(string),

 required
mode
(int32)
3.4 - PersistentVolumeClaim
: v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
(
)
resources
Quantity
)
Quantity
)
(string)
(string)
(string)
Beta level
(
)
dataSource field can be used

 to specify either: * An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot) * An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.
**
(string)
(string)
Quantity
)
capacity
(map[str

ing]
Quantity
)
type
(Time)
(Time)
(string)
(string)
phase
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
)

: integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
)

: string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.5 - PersistentVolume
: v1
kind
metadata
(
)
spec
(
)
status
(
)
capacity
Quantity
)
claimRef
(
)
([]
)
([]
)
(string)
persistentVolumeReclaimPolicy defines what happens to a persistent volume when released from its claim. Valid options are Retain (default for manually created PersistentVolumes), Delete (default for dynamically provisioned PersistentVolumes), and Recycle (deprecated). R

ecycle must be supported by the volume plugin underlying this PersistentVolume. More info:
(string)
(string)
Local
hostPath
(string)
local
(string)
(string)
(int32)
(boolean)
azureDisk
(string)
cach

ingMode is the Host Caching mode: None, Read Only, Read Write.
(string)
(string)
(boolean)
azureFile
(boolean)
(string)
cephfs
(string)
(boolean)
(string)
(string)
(string)
(string)
cinder
(string)
(boolean)
https://examples.k8s.io/mysql-ci

nder-pd/README.md
(string)
(string)
csi
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
namespace defines the space withi

n which the secret name must be unique.
(string)
(string)
(boolean)
fc
fc.fsType
(string)
fc.lun
(int32)
(boolean)
fc.wwids
(string)
(boolean)
(string)
(string)
flocker
(string)
(string)
datasetUUID is the UUID of the dataset. Thi

s is unique identifier of a Flocker dataset
(string)
(int32)
(boolean)
iscsi
iscsi.iqn
iscsi.lun
(boolean)
(boolean)
(string)
(string)
(string)
(boolean)
(string)
(string)
namespac

e defines the space within which the secret name must be unique.
nfs
nfs.path
(boolean)
(string)
(string)
(boolean)
quobyte
(string)
(boolean)
(string)
(string)
rbd
rbd.image
(string)
(string)
keyring is the path to key r

ing for RBDUser. Default is /etc/ceph/keyring. More info:
rbd.pool
(string)
(boolean)
(string)
(string)
rbd.user
(string)
scaleIO
(string)
(string)
(string)
(string)
(boolean)
(boolean)
(string)
(string)
(string)
storageos
(string)
(boolean)
(
)
(string)
(string)
volumeNamespace specifies the scope of the volume within StorageOS. If no namespace is specified then the P

od's namespace will be used. This allows the Kubernetes name scoping to be mirrored within StorageOS for tighter integration. Set VolumeName to any name to override the default behaviour. Set to "default" if you are not using namespaces within StorageOS. Namespaces that do not pre-exist within StorageOS will be created.
(string)
(string)
(string)
message
(string)
phase
(string)
reason
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Resp

onse
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.6 - StorageClass
kind
metadata
(
)
(boolean)
A topology selector term represent

s the result of label queries. A null or empty topology selector term matches no objects. The requirements of them are ANDed. It provides a subset of functionality as NodeSelectorTerm. This is an alpha feature and may change in the future.
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boole

an
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.7 - VolumeAttachment
kind
metadata
(
)
spec
(
status
(
)
attacher
nodeName
source
(
)
(string)
attached
(string)
String detailing the error encountered during Attach or Detach operation. This string may be logged, so it should not contain sensitive

 information.
(Time)
(string)
(Time)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun

(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.8 - CSIDriver
CSIDriver
kind
metadata
(
)
spec
(
(boolean)
attachRequired indicates this CSI volume driver requires an attach operation (because it implements the CSI ControllerPublishVolume() method), and that the Kubernetes attach detach controller should call the attach volume interface which checks the volumeattachment status and waits until the volume is attach

ed before proceeding to mounting. The CSI external-attacher coordinates with CSI volume driver and updates the volumeattachment status when the attach operation is complete. If the CSIDriverRegistry feature gate is enabled and the value is specified to false, the attach operation will be skipped. Otherwise the attach operation will be called.
(string)
(boolean)
(boolean)
(boolean)
-o context
(boolean)
The check can be enabled immediately when deploying a driver. In that case, provisioning new volumes with late binding will pause until the driver 

deployment has published some suitable CSIStorageCapacity object.
},
...
}
(int64)
kind
metadata
(
ListMeta
)
items
([]
CSIDriver
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
202 (
CSIDriver
update
PUT /apis/storage.k8s.io/v1/csidrivers

/{name}
name
(
in path
body
:
CSIDriver
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSIDriver
): OK
201 (
CSIDriver
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
CSIDriver
): OK
202 (
CSIDriver
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.9 - CSINode
CSINode
kind
: CSINode
metadata
(
)
spec
(
drivers
name
nodeID of the node from the driver point of view. This field enables Kubernetes to communicate with storage systems that do not share the same nomenclature for nodes. For example, Kubernetes may refer to a given node as "node1", but the storage system may refer to the same node as "nodeA". When Kubernetes issues a command to the storage system to attach a volume to a specific node, it can use this field to refer to the node name using the ID that the storage system w

ill understand, e.g. "nodeA" instead of "node1". This field is required.
(int32)
kind
metadata
(
ListMeta
)
items
([]
CSINode
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
202 (
CSINode
update
name
(
in path
body
:
CSINode
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
CSINode
): OK
201 (
CSINode
): Created
delete
name
(
in path
nam

e of the CSINode
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
CSINode
): OK
202 (
CSINode
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
3.10 - CSIStorageCapacity
kind
metadata
(
)
More info:
capacity
(
Quantity
)
(
Quantity
)
This is defined since CSI spec 1.4.0 as the largest size that may be used in a CreateVolumeRequest.capac

ity_range.required_bytes field to create a volume with the same parameters as those in GetCapacityRequest. The corresponding value in the Kubernetes API is ResourceRequirements.Requests in a volume claim.
(
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path

namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
4 - Authentication Resources
4.1 - ServiceAccount
: v1
kind
metadata
(
)
(boolean)
([]
)
secrets
([]
)
name
: v1
kind
metadata
(
ListMeta
)
items
([]
https://ku

bernetes.io/docs/tasks/configure-pod-container/configure-service-account/
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path

namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
4.2 - TokenRequest
kind
metadata
(
)
spec
(
status
(
)
audiences
(string)
(string)
(string)
(string)
(int64)
token
create
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
4.3 - TokenReview
kind
: T

okenReview
metadata
(
)
spec
(
status
(
)
audiences
token
(string)
audiences
(boolean)
error
(string)
user
user.uid
(string)
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
4.4 - CertificateSigningRequest
kind
metadata
(
)
spec
(
spec contains the certificate request, and is immutable after creation. Only the request, signerName, expirationSeconds, 

and usages fields can be set on creation. Other fields are derived by Kubernetes and cannot be modified by users.
status
(
)
request
(int32)
extra
groups
uid
(string)
uid contains the uid of the user that created the CertificateSigningRequest. Populated by th

e API server on creation and immutable.
usages
username
(string)
([]byte)
base64(
...
)
(Time)
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the 

factory methods that the time package offers.
(Time)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Certific

ateSigningRequest
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5 - Authorization Resources
5.1 - LocalSubjectAccessReview
LocalSubjectAccessReview checks whether or not a user or group can perform an action in a given namespace. Having a namespace sc

oped resource makes it much easier to grant namespace scoped policy that includes permissions checking.
kind
metadata
(
)
spec
(
status
(
)
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.2 - SelfSubjectAccessReview
kind
metadata
(
)
spec
(
status
(
)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
Version is the API Vers

ion of the Resource. "*" means all.
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.3 - SelfSubjectRulesReview
kind
metadata
(
)
spec
(
status
ResourceNa

mes is an optional white list of names that the rule applies to. An empty set means that everything is allowed. "*" means all.
"
(string)
namespace
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.4 - SubjectAccessReview
kind
metadata
(
)
spec
(
status
(
)
extra
groups
(string)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
Verb is a kubernetes resource API verb, like: get, list, watch, create, update, delete, proxy. "*" means a

ll.
(string)
uid
(string)
user
(string)
allowed
denied
(boolean)
(string)
reason
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.5 - SelfSubjectReview v1alpha1
kind
metadata
(
)
status
(
)
userInfo
(string)
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
5.6 - ClusterRole
kind
metadata

(
)
([]
)
rules
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK

201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.7 - ClusterRoleBinding
kind
metadata
(
)
roleRef
subjects
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
GET /apis/rb

ac.authorization.k8s.io/v1/clusterrolebindings/{name}
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.8 - Role
Role
kind
: Role
metadata
(
)

rules
RoleList
kind
: RoleList
metadata
(
ListMeta
)
items
([]
Role
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
RoleList
): OK
create
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
202 (
Role
update
name
(
in path
namespace
(
in path
namespace
body
:
Role
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
fiel

(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Role
): OK
201 (
Role
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
5.9 - RoleBinding
kind
metadata
(
)
roleRef
subjects
(string)
(string)
Namespace of the referenced object. If the object kind is non-namespace, such as "User" or "Group", and this value is not empty th

e Authorizer should report an error.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
propaga

tionPolicy
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
6 - Policy Resources
6.1 - LimitRange
: v1
kind
metadata
(
)
spec
(
)
limits
Quantity
)
Quantity
)
Quantity
)
Quantity
)
Quantity
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watc

h
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
6.2 - ResourceQuota
: v1
kind
metadata
(
)
spec
(
)
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-

status
status
(
)
hard
Quantity
)
scopes
hard
Quantity
)
used
Quantity
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
list or watch

 objects of kind ResourceQuota
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(

in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
6.3 - NetworkPolicy
kind
metadata
(
)
spec
(
)
status
(
)
(
ingress
(IPBlock)
IPBlock defines policy on a particular IPBlock. If this field is set then neither of the oth

er fields can be.
(
)
(
)
(int32)
(string)
egress
egress.to
List of destinations for outgoing traffic of pods selected for this rule. Items in this list are combined using a logical OR operation. If this field is empty or missing, this rule matches all destinations (traffic not restricted by destination). If this field is present and contains at least one item, this rule allows 

traffic only if the traffic matches at least one item in the to list.
(IPBlock)
(
)
(
)
(int32)
(string)
type
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of the factory methods that the time package of

fers.
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
PUT /apis/n

etworking.k8s.io/v1/namespaces/{namespace}/networkpolicies/{name}
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
6.4 - PodDisruptionBudget
kind
metadata
(
)
https://git.k8s.io/community/contributors/devel/sig-archite

spec
(
)
status
(
)
selector
(
)
(string)
type
InsufficientPods: The numbe

r of pods are either at or below the number
(int64)
(int64)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
GET /apis/policy/v1/namespaces/{namespace}/poddisruptionbudgets/{name}/sta

tus
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
PATCH /apis/policy/v1/namespaces/{namespace}/poddisrupti

onbudgets/{name}/status
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7 - Extend Resources
7.1 - CustomResourceDefinition
kind
metadata
(
)
spec
(
status
(
)
group
).
names
kind
.
(string)
listKind is th

e serialized kind of the list for this resource. Defaults to "
kind
List".
(string)
kind
.
scope
Cluster
and
.
versions
if
served
is true.
(string)
(string)
(int32)
(boolean)
(string)
is tr

ue. The default warning indicates this version is deprecated and recommends use of the newest served version of equal or greater stability, if one exists.
(
)
/scale
.spec
/scale
.status
/scale
(string)
.status
or
.spec
/scale
/status
status
/status
status
.status
None
Webhook
is needed for this option. This requires spec.preserveUnknownFields to be false, and spec.conversion.we

bhook to be set.
strategy
is set to
Webhook
.
Webhook
.
([]byte)
service
.
(string)
(int32)
port
(string)
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
(boolean)
to true in
. See
.
$ref
(string)
$schema
(string)
additionalProp

erties
allOf
([]
)
anyOf
([]
)
default
(JSON)
)
(string)
enum
([]JSON)
example
(JSON)
(boolean)
(boolean)
(string)
(string)
format
(string)
id
(string)
items
maxItems
(int64)
maxLength
(int64)
(int64)
maximum
(double)
minItems
(int64)
minLength
(int64)
(int64)
minimum
(double)
(double)
not
(
)
nullable
(boolean)
oneOf
([]
)
pattern
(string)

)
)
required
title
(string)
type
(string)
(boolean)
(boolean)
(boolean)
anyOf:
allOf:
anyOf:
map
(string)
atomic
set
:
atomic
or an
atomic
.
map
:
(string)
granular
:
atomic
(boolean)
rule
self
variable in the CEL expression is bound to the scoped value. Example: - Rule scoped to the root of a resource with a status subresourc

e: {"rule": "self.status.actual &lt;= self.spec.maxDesired"}
self[i]
self
The
,
kind
,
and
dot
dash
slash
Examples:
namespace
&gt; 0"}
'set':
X + Y
X
Y
'map':
X + Y
X
Y
X
and
Y
Y
with
(string)

ired
kind
.
(string)
kind
List".
(string)
kind
.
(Time)
(string)
(string)
items
([]
(string)
kind
(string)
metadata
(
ListMeta
)
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
read status of the spec

ified CustomResourceDefinition
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): intege

r
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7.2 - MutatingWebhookConfiguration
kind
metadata
(
)
.
webhooks
name
([]byte)
caBundle
service
service
or
url
service
.
name
namespace
(string)
path
(int32)
port
(string)
url
url
or
service
The
host
service
host
may also

 be an IP address.
localhost
or
127.0.0.1
as a
host
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
ObjectSelector decides whether to run the webhook based on if the object has matching labels. objectSelector is evaluated against both the oldObject and newObject that would be sent to the webhook, and is considered to match if either object matches the selector. A null object (oldObject in the case of create, or newObject in th

e case of delete) or an object that cannot have labels (like a DeploymentRollback or a PodProxyOptions object) is not considered to match. Use the object selector only if the webhook is opt-in, because end users may skip the admission webhook by setting the labels. Default to the empty LabelSelector, which matches everything.
(string)
must
any
(string)
(int32)
TimeoutSeconds specifies the timeout for this webhook. After the timeout passes, the webhook call will be ignored or the API call will fail based on the failure policy.

 The timeout value must be between 1 and 30 seconds. Default to 10 seconds.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryR

un
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7.3 - ValidatingWebhookConfiguration
kind
metadata
(
)
.
webhooks
name
([]byte)
caBundle
service
service
or
url
service
.
name
namespace
(string)
path
(int32)
port
(string)
url
url
or
service
The
host
service
host
localhost
or
127.0.0.1
as a
host
A path is optional, and if present

 may be any string permissible in a URL. You may use the path to pass an arbitrary string to the webhook, for example, a cluster identifier.
(string)
(string)
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
Rules describes what operations on wh

at resources/subresources the webhook cares about. The webhook cares about an operation if it matches
any
(string)
(int32)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string

(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
7.4 - ValidatingAdmissionPolicy v1alpha1
kind
metadata
(
)
.
spec
ValidatingAdmissionPolicySpec is the specification of the desired behav

ior of the AdmissionPolicy.
ref
The
,
kind
,
and
dot
dash
slash
Examples:
namespace
&gt; 0"}
'set':
X + Y
X
Y
'map':
X + Y
X
Y
X
and
Y
Y
with
Required.
(string)
(string)
(string)
MatchConstraints specifies what r

esources this policy is designed to validate. The AdmissionPolicy cares about a request if it matches
all
(string)
(string)
, a request to apps/v1beta1 or extensions/v1beta1 would be

 converted to apps/v1 and sent to the ValidatingAdmissionPolicy.
(
)
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
any
Rule.
(string)
scope specifies the scope of t

his rule. Valid values are "Cluster", "Namespaced", and "
(string)
(string)
(string)
items
([]
)
kind
(string)
metadata
(
ListMeta
)
(string)
kind
(string)
metadata
(
)
.
spec
MatchResources decides whether to run the admission control policy on an object based on whether 

it meets the match criteria. The exclude rules take precedence over include rules (if a resource matches both, it is excluded)
(string)
(string)
(
)
NamespaceSelector decides whether to run the admission control policy on an object based on whether the namespace for that object matches the selector. If the object itself is a namespace, the matching is performed on object.metadata.labels. If the object is another cluster scoped resource, it never skips the policy

.
{
"0",
"1"
]
}
]
}
{
"prod",
"staging"
]
}
]
}
See
(
)
any
Rule.
(string)
ParamRef specifies the parameter resource used to configure the admission control policy. 

It should point to a resource of the type specified in ParamKind of the bound ValidatingAdmissionPolicy. If the policy specifies a ParamKind and the resource referred to by ParamRef does not exist, this binding is considered mis-configured and the FailurePolicy of the ValidatingAdmissionPolicy applied.
(string)
(string)
(string)
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
delete collection o

f ValidatingAdmissionPolicy
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8 - Cluster Resources
8.1 - Node
Node
: v1
kind
: Node
metadata
(
)
spec
(
NodeSpec
)
status
(
)
NodeSpec
(string)
(string)
(string)
podCIDR
(string)
podCIDRs
(string)
taints
([]Taint)
Required. 

The taint key to be applied to a node.
(Time)
(string)
(boolean)
addresses
type
Quantity
)
capacity
Quantity
)
type
(Time)
(Time)
(string)
(string)
config
Name is the metadata.name of the referenced

 ConfigMap. This field is required in all cases.
(string)
(string)
(string)
(string)
(string)
LastKnownGood reports the checkpointed config the node will fall back to when it encounters an error attempting to use the Assigned config. The Assigned config becomes the LastKnownGood config when the node determines that the Assigned config is stable and correct. This is currently implemented as a 10-minute soak period starting when the local record of Assigned config is updated. If the Assigned confi

g is Active at the end of this period, it becomes the LastKnownGood. Note that if Spec.ConfigSource is reset to nil (use local defaults), the LastKnownGood is also immediately reset to nil, because the local default config is always assumed good. You should not make assumptions about the node's method of determining config stability and correctness, as this may change or become configurable in the future.
(string)
(string)
images
(int64)
nodeInfo
phase

(string)
NodeList
: v1
kind
: NodeList
metadata
(
ListMeta
)
items
([]
Node
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
NodeList
): OK
create
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
202 (
Node
update
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
update
name
(
in path
body
:
Node
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Node
): OK
201 (
Node
): Created
401: Unauthorize

d
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8.2 - Namespace
Namespace
: v1
kind
metadata
(
)
spec
(
)
status
(
)
type
(Time)
(string)
(string)
phase
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
Namespace
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
list
(
in query
): boolean
a

continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
202 (
Namespace
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
update
name
(
in path
body
:
Namespace
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Namespace
): OK
201 (
Namespace
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
8.3 - Event
Event
Event is a report of an event somewhere in the cluster. It generally denotes some state change in the system. Events have a limited retention time and triggers and messages may evolve with time. Event consumers should not rely on the timing of an event with a given Reason reflecting a consistent underlying trigger, or the continued exi

stence of events with that Reason. Events should be treated as informative, best-effort, supplemental data.
kind
: Event
metadata
(
)
eventTime
action
(string)
(int32)
(Time)
(Time)
(string)
(string)
note
(string)
reason
(string)
regarding
(
)
related
(
)
(string)
(string)
series
type
(string)
EventList
kind
metadata
(
ListMeta
)
ht

tps://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
items
([]
Event
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
EventList
): OK
create
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
202 (
Event
update
name
(
in path
namespace
(
in path
namespace
body
:
Event
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Event
): OK
201 (
Event
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dry

Run
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8.4 - APIService
kind
metadata
(
)
spec
(
)
status
(
)
caBundle
([]byte)
group
(string)
(boolean)
service
(string)
(string)
service

.port
(int32)
port
version
(string)
type
(Time)
(string)
(string)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Resp

onse
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8.5 - Lease
Lease
kind
: Lease
metadata
(
)
More info:
spec
(
LeaseSpec
)
LeaseSpec
(string)
(int32)
(int32)
renewTime
LeaseList
kind
metadata
(
ListMeta
)
items
([]
Lease
get
name
(
in path
namespace
(
in path
): string,

 required
namespace
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
LeaseList
): OK
create
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
202 (
Lease
update
name
(
in path
namespace
(
in path
namespace
body
:
Lease
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Lease
): OK
201 (
Lease
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string

(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8.6 - RuntimeClass
kind
metadata
(
)
More info:
handler
overhead
Quantity
)
.
(string)
(string)
(string)
(string)
(int64)
TolerationSeconds 

represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system.
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
timeoutSe

conds
Response
200 (
Status
): OK
8.7 - FlowSchema v1beta3
kind
metadata
(
)
metadata
spec
(
)
spec
status
(
)
status
name
nil
type
(int32)
rules
rules
kind
group
GroupSubject holds detailed information for group-

kind subject.
name
namespace
user
name
"
verbs
and
apiGroups
resources
verbs
(boolean)
indicates whether to match requests that do not specify a na

mespace (which happens either because the resource is not namespaced or the request targets all namespaces). If this field is omitted or false then the
is true.
type
(Time)
(string)
message
(string)
reason
(string)
status
(string)
type
kind
metadata
(
ListMeta
)
metadata
items
([]
items
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): str

ing
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
8.8 - PriorityLevelConfiguration v1beta3
kind
metadata
(
)
metadata
spec
(
)
spec
status
(
)
status
type
(string

type
"Exempt"
"Limited"
are
limited
limited
type
is
"Limited"
.
(int32)
nil
(int32)
type
queuing
type
is
"Queue"
.
(int32)
handSize
handSize
queues
(int32)
(int32)
queues
is the number of queues for this priority level. The queu

es exist independently at each apiserver. The value must be positive. Setting it to 1 effectively precludes shufflesharding and thus makes the distinguisher method of associated flow schemas irrelevant. This field has a default value of 64.
(int32)
type
(Time)
(string)
message
(string)
reason
(string)
status
(string)
type
kind
metadata
(
ListMeta
)
metadata
items
([]
items
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
POST /api

s/flowcontrol.apiserver.k8s.io/v1beta3/prioritylevelconfigurations
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response

200 (
Status
): OK
8.9 - Binding
Binding
: v1
kind
: Binding
metadata
(
)
target
(
create
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
create
name
(
in path
namespace
(
in path
namespace
body
:
Binding
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Binding
): OK
201 (
Binding
): Created
202 (
Binding
8.10 - ComponentStatus
: v1
kind
metadata
(
)
type
(string)
(string)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
8.11 - Clu

sterCIDR v1alpha1
kind
metadata
(
)
spec
(
)
ipv4
(string)
ipv6
(string)
([]
)
([]
)
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
l

imit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
9 - Common Definitions
9.1 - DeleteOptions
(string)
dryRun
(int64)
kind
(string)
K

ind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info:
(boolean)
(string)
(string)
(string)
9.2 - LabelSelector
key
9.3 - ListMeta
continue
(string)
(int64)
remainingItemCount is the number of subsequent items in the list which are not included in this list response. If the list request contained labe

l or field selectors, then the number of remaining items is unknown and the field will be left unset and omitted during serialization. If the list is complete (either because it is not chunking or because this is the last chunk), then there are no more remaining items and this field will be left unset and omitted during serialization. Servers older than v1.15 do not set this field. The intended use of the remainingItemCount is
(string)
selfLink
(string)
9.4 - LocalObjectReference
name
(string)
9.5 - NodeSelectorRequirement
key
operator
values
9.6 - ObjectFieldSelector
fieldPath
(string)
9.7 - ObjectMeta
name
(string)
(string)
https://git.k8s.io/community/contri

butors/devel/sig-architecture/api-conventions.md#idempotency
namespace
(string)
labels
System
(string)
(string)
', where
', where
', where
', where
(string)
(string)
Operation is the type of operation which lead to this ManagedFieldsEntry being created. The only valid values for thi

s field are 'Apply' and 'Update'.
(string)
(Time)
uid
(boolean)
(boolean)
Read-only
(Time)
(int64)
(Time)
DeletionTimestamp is RFC 3339 date and time at which this resource will be deleted. This field is set by the server when a graceful deletion is requested by the user, and is not directly settable by a client. The resource is expected to be deleted (no longer visible from resource lists, and not reachable by name) after the time in this field, once the finalizers list is empty. As long as th

e finalizers list contains items, deletion is blocked. Once the deletionTimestamp is set, this value may not be unset or be set further into the future, although it may be shortened or the resource may be deleted prior to this time. For example, a user may request that a pod is deleted in 30 seconds. The Kubelet will react by sending a graceful termination signal to the containers in the pod. After that 30 seconds, the Kubelet will send a hard termination signal (SIGKILL) to the container and after cleanup, remove the pod from the API. In the presence of network partitions, this object may still exist after this timestamp, until an administrator or automated process can determine the resource is fully terminated. If not set, graceful deletion of the object has not been requested.
(int64)
(string)
selfLink
(string)
uid
(string)
9.8 - ObjectReference
(string)
fieldPath
(string)
kind
(string)
name
(string)
namespace
(string)
(string)
uid
(string)
9.9 - Patch
Patch is provided to give a concrete n

ame and type to the Kubernetes PATCH request body.
9.10 - Quantity

Examples:

9.11 - ResourceFieldSelector
resource
(string)
divisor
(
Quantity
)
9.12 - Status
(string)
code
(int32)
details
Extended data associated with the reason. Each reason may define its own extend

ed details. This field is optional and the data returned is not guaranteed to conform to any schema except that defined by the reason type.
(string)
Examples:
(string)
(string)
(string)
(string)
(string)
(int32)
(string)
kind
(string)
message
(string)
metadata
(
ListMeta
)
reason
(string)
status
(string)
9.13 - TypedLocalObjectReference
kind
name
apiGroup
(string)
APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the cor

e API group. For any other third-party types, APIGroup is required.
10 - Other Resources
10.1 - ValidatingAdmissionPolicyBindingList v1alpha1
11 - Common Parameters
continue
dryRun
.
force
The duration in seconds before the object should be deleted. Value must be non-negative integer. The value zero indicates delete i

mmediately. If this value is nil, the default grace period for the specified type will be used. Defaults to a per object value if not specified. zero means delete immediately.
limit
continue
namespace
pretty
watch
Home
Blog
Training
Partners
Community
CC BY 4.0

SelfSubjectAccessReview | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
kind
metadata
(
)
spec
(
status
(
)
(string)
(string)
(string)
(string)
(string)
(string)
(string)
Subresource is one of the existing res

ource types. "" means none.
(string)
(string)
create
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
Feedback
Yes
No
.
or
.
create
Home
Blog
Training
Partners
Community
CC BY 4.0

Using sysctls in a Kubernetes Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install To

Tasks
sysctl
Note:
/
or
.
using a
the page
from
minikube
virtual
kernel.
)
net.
)
vm.
)
dev.
)
.
safe
and
unsafe
safe
isolated
safe
of a pod.
safe
.
safe
set:
,
,
,
(since Kubernetes 1.

22).
Note:
All
safe
All
unsafe
unsafe
Unsafe
\
...
For
Minikube
flag:
=
...
Only
,
,
,
,
net.*
and
and
safe
and
unsafe
sysctls in
Warning:
:
v1
kind
:
Pod
metadata
:
name
:
spec
:
:
sysctls
:
-
name
:
value
:
"0"
-
name
:
value
:
"1024"
-
name
:
value
:
"65536"
...
Warning:
unsafe
, the use of
unsafe
sysctls
tainted
feature
unsafe
unsafe
sysctls it
feature
or
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation h

as registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Parallel Processing using Expansions | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
Run Jobs
Jobs
parallel.
apple
,
banana
, and
cherry
.
See
Job
.
minikube
sed
.
Python
.
:
batch/v1
kind
:
Job
metadata
:
name
:
labels
:
jobgroup
:
spec
:
template
:
metadata
:
name
:
labels
:
jobgroup
:
spec
:
:
-
name
:
c
image
:
command
:
[
"sh"
,
"-c"
,
]
:
Never
manifest
.
$ITEM
sed
$ITEM
jobs
for
do
$i
/"
$i
.yaml
done
ls jobs/
job-cher

ry.yaml
jobs
-l
jobgroup
=
Using the
-l
:
jobgroup
=
jobgroup
=
Clean up
jobgroup
=
In the
names
:
%}
---
kind: Job
metadata:
  labels:
spec:
  template:
    metadata:
      labels:
    spec:
      - name: c
for
---
on a line
by itself.
kubectl
alias
=
Use
jobs.yaml
kubectl
:
Kubernetes accepts and runs the Jobs you

 created.
Clean up
jobgroup
=
$ITEM
$ITEM
use HTTP to
PUT
frame.
labels
that
.
jobgroup
In the
Note:
jobgroup
There are
objects.
Feedback
Yes
No
.
or
.
Clean up
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
Auditing
6:
7:
8:
the
Ready
state.
and
Lease:
  DiskPressure         Unknown   Thu, 17 Feb 2022 17:12:40 -0500   Thu, 17 Feb 2022 17:13:52 -0500   NodeStatusUnknown   Kubelet stoppe

d posting node status.
Capacity:
  cpu:                2
  pods:               110
  cpu:                2
  pods:               110
Events:
...
:
v1
kind
:
Node
metadata
:
:
:
:
"0"
:
"true"
:
labels
:
:
amd64
:
linux
:
amd64
:
:
linux
name
:
:
"4026"
uid
:
spec
:
{}
status
:
addresses
:
-
address
:
type
:
-
address
:
type
:
Hostname
:
cpu
:
"2"
:
:
"0"
memory
:
1922788Ki
pods
:
"110"
capacity
:
cpu
:
"2"
:
:
"0"
memory
:
2025188Ki
pods
:
"110"
:
-
:
:
message
:
reason
:
WeaveIsUp
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"False"
type
:
-
:
:
message
:
kubelet has sufficient PID av

ailable
reason
:
status
:
"False"
type
:
-
:
:
message
:
reason
:
status
:
"True"
type
:
Ready
:
:
Port
:
10250
nodeInfo
:
:
amd64
bootID
:
22333234
:
:
5.13.0-28
-generic
:
v1.23.3
:
v1.23.3
machineID
:
:
linux
osImage
:
:
- logs from
Results
Results
Results
Results
Results
Results
etc.
Mitigates: 

API server backing storage (i.e., etcd's data directory) lost
Use
to
Use
crictl
to
Use
to
1 - Resource metrics pipeline
The
(HPA) and
(VPA)
command.
Note:
.
S[
]
Server]
end
L[API
server]
W[HPA]
end
top]
enabled
cAdvisor
kubelet
and
/stats
kubelet
/stats
endpoint.
the
Note:
FEA

TURE STATE:
minikube
jq
for easier
reading:
| jq
'.'
curl
:
{
"kind"
:
,
:
,
: {
"name"
:
,
:
,
:
},
:
,
"window"
:
"30s"
,
"usage"
: {
"cpu"
:
,
"memory"
:
}
}
jq
| jq
'.'
curl
:
{
"kind"
:
,
:
,
: {
"name"
:
,
:
,
:
,
:
},
:
,
"window"
:
"30s"
,
: [
{
"name"
:
,
"usage"
: {
"cpu"
:
,
"memory"
:
"22244Ki"
}
}
]
}
for the
API.
,
the
and the
.
Note:
CPU
.
Memory
.
using the
command.
is available via the extension API t

hat the metrics-server makes available.
kubelet
API
.
.
2 - Tools for Monitoring Resources
pods
,
services
, and
or
utility.
and
API.
kubelet
for CPU and
cAdvisor
).
or
API.
Logging
exec
3 - Monitor Node Health
DaemonSet
and reports these conditions to the API server as N

ode
Condition
s
or as
Event
s.
.
minikube
.
Addon
.
kubectl
kubectl
:
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
Note:
kubectl
:
Create
directory
The
ConfigMap
config/
Create the
ConfigMap
:
=
config/
Change the
to use the
ConfigMap
:
:
apps/v1
kind
:
DaemonSet
metadata
:
name
:
namespace
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
selector
:
:
k8s-app
:
version
:
v0.1
:
"true"
template
:
metadata
:
labels
:
k8s-app
:
version
:
v0.1
:
"true"
spec
:
:
true
:
-
name
:
image
:
:
:
true
resources
:
limits
:
cpu
:
"200m"
memory
:
"100Mi"
requests
:
cpu
:
"20m"
memory
:
"20Mi"
:
-
name
:
log
mountPath
:
/log
readOnly
:
true
-
name
:
config
mountPath
:
/config
readOnly
:
tr

ue
volumes
:
-
name
:
log
hostPath
:
path
:
/var/log/
-
name
:
config
configMap
:
name
:
Note:
kubectl
.
ConfigMap
.
A
such as
filelog
,
kmsg
,
kernel
,
abrt
,
and
systemd
.
A
.
A
.
A
.
.
Exporter
.
.
4 - Debugging Kubernetes nodes with crictl
crictl
crictl
cri-tools
crictl
crictl
.
General usag

e
The
crictl
or
crictl
Set the
and
flags.
Set the
and
crictl
.
Note:
crictl
timeout
or
debug
--timeout
and
--debug
crictl
crictl
.
crictl
Warning:
If you use
crictl
crictl
is not a
List pods
run
=
nginx
CONTAINER ID        IMA

GE                                                                                                             CREATED             STATE               NAME                       ATTEMPT
crictl ps
crictl
exec
N
=
1
Using
crictl
{
: {
"name"
:
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
},
:
"/tmp"
,
"linux"
: {
}
}
Use the
Using
crictl
Pod config
:
{
: {
"name"
:
,
:
"default"
,
"attempt"
:
1
,
"uid"
:
},
:
"/tmp"
,
"linux"
: {
}
}
:
{
: {
"name"
:
"busybox"
},
"image"
:{
"image"
:
"busybox"
},
"command"
: [
"top"
],
:
,
"linux"
: {
}
}
container config file, and the pod conf

ig file. The ID of the container is
returned.
Created
.
:
Running
.
crictl ps
crictl
.
Map
docker
crictl
.
5 - Auditing
auditing
stage
chain.
Panic
Note:
Event
API group
.
None
Metadata
Request
using the
rules
field
must
:
kind
:
Policy
:
-

rules
:
-
level
:
resources
:
-
group
:
""
resources
:
[
"pods"
]
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
,
]
-
level
:
None
resources
:
-
group
:
""
resources
:
[
]
:
[
]
-
level
:
None
users
:
[
]
verbs
:
[
"watch"
]
resources
:
-
group
:
""
resources
:
[
,
]
-
level
:
None
:
[
]
:
-
"/api*"
-
-
level
:
Request
resources
:
-
group
:
""
resources
:
[
]
:
[
]
-
level
:
Metadata
resources
:
-
group
:
""
resources
:
[
"secrets"
,
]
-
level
:
Request
resources
:
-
group
:
""
-
group
:
-
level
:
Metadata
:
-
Metadata
level:
:
kind
:
Policy
rules
:
-
level
:
Metadata
Policy
API group
.
Note:
request to
:
[
{
"op"
:
"replace"
,
"path"
:
,
"value"
:
0
},
{
"op"
:
"remove"
,
"path"
:
}
]
JSONlines
format.
flags:
-
hostPath
to the location of the policy file and log file, so t

hat audit records are persisted. For example:
=
\
=
...
:
-
mountPath
:
name
:
audit
readOnly
:
true
-
mountPath
:
name
:
audit-log
readOnly
:
false
hostPath
:
...
volumes
:
-
name
:
audit
hostPath
:
path
:
type
:
File
-
name
:
audit-log
hostPath
:
path
:
type
:
.
webhook
with
log
in the flag
webhook
log
webhook
log
.
batch
blocking
batch
mode:
on
and
whether event

 and batch truncating is enabled.
webhook
and
log
or
.
Event
and the
Policy
6 - Debugging Kubernetes Nodes With Kubectl
node
command.
minikube
.
Use the
=
ubuntu
ip
,
ifconfig
,
nc
,
ping
, and
ps
mtr
,
tcpdump
, and
curl
Note:
/host
in the Pod.
kubelet
Logs from
/host
.
will fail.
Note:
The
ku

bectl debug node
Check
7 - Developing and debugging services locally using telepresence
Note:
kubectl
, run
 
...
Use the
Where:
And
.
8 - Windows debugging tips
See
Note:
plugins.plugins

.cri.sandbox_image
or
this guide
.
enabled
on all
commands.
cni.conf
(see
) is
from the
in
cni.conf
            ]
NodePort
C:\k\
| ? Name
-ieq
"cbr0"
| ? Name
-Like
start.ps1
section
.
.
wincat
.
,
,
,
,
With Flannel, my nodes 

are having issues after rejoining a cluster
issue
;
start.ps1
,
)
-file
to restart
=
=
=
1500
=
true
topic
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
6:
API
.
The
.
The
.
Alpha:
alpha
v1alpha1
).
Beta:
beta
v2beta3
).
(
except
Note:
Stable:
vX
where
X
and there are no current plans for a major version revision of Kubernetes that removes 

stable APIs.
API groups
API groups
field of a
The
core
legacy
/api/v1
.
field, for
example,
.
and use
).
.
pairs
=true
to disable
batch/v1
, set
to enable
, set
, set
Note:
changes.
etcd
.
1 - Kubernetes API Concepts
GET).
watches
.
online,
A
pods
,
,
services
)
kind
resource
object
objects
:
virtual
in
to the
eviction
).
name
Within a
namespace
example: Nodes), and so their names must be unique across th

e whole cluster.
API verbs
list
of
a
get
?watch
watch
and not a
get
(see
create
or
update
update
patch
; the
patch
is PATCH.
) or to a
/api
instead of
/apis
Examples:
.
see the
watch
. Clients
can send a
list
or a
get
watch
request.
use that
watch
watch
create
,
delete
,
and
update
to the
watch
watch
If a client
watch
watch
from
the last 

returned
get
/
list
---
200 OK
{
}
(such as
create
,
delete
,
apply
or
update
test
(served as
---
200 OK

{
}
{
}
...
watch
410 Gone
get
or
list
watch
from the
list
-then-
watch
Reflector
package.)
BOOKMARK
to a given
BOOKMARK
---
200 OK
{
}
...
{
}
BOOKMARK
watch
BOOKMARK
, the
list
chunks

limit
and
continue
continue
list
metadata
limit
limit
continue
continue
chunk
continue
Like a
watch
continue
410 Gone
limit
---
200 OK
{
...
},
}
---
200 OK
{
...
},
}
---
200 OK
{
...
},
}
10245
list
continue
watch
operation
list
selectors
If the
list
list
is
a
List
appended.
list
has
kind
set to
{
"kind":
,
"v1"
,
{
"2947301"
},
"items":
[
{
{
"name":
,
"default"
,
...
{
"name":
,
,
...
There are dozens of collection types (

such as
PodList
,
,
and
NodeList
kubectl
kubectl
multiple
list
kubectl
kind: List
:
v1
kind
:
List
metadata
:
:
""
selfLink
:
""
items
:
-
:
v1
kind
:
Service
metadata
:
:
labels
:
component
:
apiserver
provider
:
name
:
namespace
:
default
...
-
:
v1
kind
:
Service
metadata
:
:
:
"9153"
:
"true"
:
labels
:
k8s-app
:
kube-dns
:
"true"
:
CoreDNS
name
:
kube-dns
namespace
:
Note:
kind
named
List
.
kind: List
kind: List
kubectl
an
Accept
with a
GET
type.
---
200 OK
{
...
...
]
}
name
and
fields.
---
200 OK
...
{
...
{
...
},
{
...
}
]
}
Accept
The Kubernetes API implements standard HTTP co

ntent type negotiation: passing an
Accept
GET
PUT
or
POST
---
200 OK
in JSON.
---
200 OK
{
...
}
.
Accept
}
}
Note:
that does
When you
delete
removal
{
"kind":
,
"v1"
,
{
{
,
},
nil,
}
}
delete
Once the
Order is
not
.
The
If the finalizer list were processed

 in order, then this might lead to a situation
get
,
create
,
apply
,
update
,
patch
,
delete
and
proxy
list
and
watch
PUT
request).
CRDs
fields via
).
POST
,
PUT
, and
PATCH
), field
Ignore
,
Warn
, and
Strict
.
Note:
Ignore
Warn
Warning:
.
Strict
detected.
Tools that submit requests to the server

 (such as
kubectl
Warn
The
kubectl
ignore
,
warn
,
and
strict
true
strict
) and
false
ignore
,
Dry-run
POST
,
PUT
,
PATCH
, and
DELETE
dry run
dryRun
?dryRun
or
All
PATCH
field to
None
.
Note:
DryRun
:
name
: if
is set,
name
/
UID
:
For the
Service
Authorization for dry-run and non-

dry-run requests is identical. Thus, to make
patch
patch
RBAC
rules
:
-
apiGroups
:
[
"apps"
]
resources
:
[
]
verbs
:
[
"patch"
]
See
.
Apply
and
Update
.
apply
.
See
watch
list
- The
- The
list
The
get
,
list
, and
watch
list
requests.
. If you set
get
and
list
For
get
and
list
are:
get:
Any
list:
parameter
on
list
and
, the
.
on a
list
and
a known
and
list
and
:
unset
Any
unset
Any
Exact
unset
400

 Bad Request
Invalid
Invalid
Exact
Invalid
Invalid
Exact
Invalid
Any
Invalid
Any
Note:
get
and
list
Any
may be
list
of any
Exact
is
list
list
continue
tokens
list
list
.
Note:
When you
list
metadata
well as
When using
newer
.
When using
and
limit
matches
limit
set.
watch
For
watch
watch:
watch
Caution:
Start a
watch
watch
configurations, due to partitions or stale caches. Cli

ents that cannot tolerate
watch
watch
watch
Start a
watch
the
watch
watch
Start a
watch
watch
410 (Gone)
410 (Gone)
410 (Gone)
410 Gone
list
or
get
watch
2 - Server-Side Apply
objects
combined
"
result in a
conflict
is either deleted from the live object or r

eset to its default value, if
kubectl
POST
,
PUT
PATCH
Conflicts
).
metadata
.
:
v1
kind
:
ConfigMap
metadata
:
name
:
test-cm
namespace
:
default
labels
:
:
test
:
-
manager
:
kubectl
operation
:
Apply
:
v1
time
:
:
FieldsV1
fieldsV1
:
:
f:labels
:
:
{}
f:data
:
f:key
:
{}
data
:
key
:
some value
. The
Note:
the user.
through an
Update
API
.
Conflicts
Apply
force
using the
on the server, a

nd make the request again. This leaves the value unchanged,
Managers
query
kubectl
Apply
(
PATCH
with
) and
Update
Note:
as the
:
v1
kind
:
ConfigMap
metadata
:
name
:
test-cm
namespace
:
default
labels
:
:
test
:
-
manager
:
kubectl
operation
:
Apply
:
v1
fields
:
:
f:labels
:
:
{}
-
manager
:
operation
:
Update
:
v1
time
:
fields
:
f:data
:
f:key
:
{}
data
:
key
:
new value
Update
.
Apply
.
CRD
:
atomic
/
set
/
map
set
map
atomic
atomic
set
or
map
1.16
. A list of field names whose values uniquely identify entries in the list. Wh

ile there can be multiple keys,
1.16
atomic
/
granular
atomic
granular
1.17
atomic
/
granular
.
1.17
If
listType
marker as a
and the
marker as a
.
The
atomic
map
/
set
/
granular
to
atomic
When the
listType
,
mapType
, or
map
/
set
/
granular
to
atomic
atomic
to
map
/
set
/
granular
atomic
to
map
/
set
/
granular
.
:
kind
:
Foo
metadata
:
name
:
:
-
manager
:
operation
:
Apply
:
fields
:
f:spec
:
f:data
:
{}
spec
:
data
:
key1
:
val1
key2
:
val2
Before
spec.data
atomic
to
granular
,
spec.data
(
key1
and
key2
spec.data
granular
,
spec.data
data
key1
and
key2
schema
type.
,
replicas
field from a user to a controller wh

ile enabling
replicas
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
nginx
spec
:
replicas
:
3
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
=
50
--min
=
1
--max
=
10
replicas
replicas
, and if
replicas
replicas
replicas
replicas
from their
replicas
field:
:
apps/v1
kind
:
metadata
:
name
:
spec
:
replicas
:
3
:
\
=
\
=
false
replicas
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
nginx
spec
:
selector
:
:
app
:
nginx
template
:
metadata
:
labels
:
app
:
nginx
spec
:
:
-
name
:
nginx
image
:
replicas
applier has no way of knowing whether their next appl

y will overwrite other
case.
can start
[
--dry-run
=
server
]
Caution:
Keep the
.
kubectl
kubectl
.
=
[
--dry-run
=
server
]
,
.
.
kubectl
kubectl
.
=
[
--dry-run
=
server
]
PATCH
PATCH
PATCH
CREATE
using
,
,
JSONPatch
, or
Update
, so every
3 - Client Libraries
To write applications usi

ng the
,
.
Language
C
browse
dotnet
browse
Go
browse
Haskell
browse
Java
browse
browse
Perl
browse
Python
browse
Ruby
browse
Note:
Language
Clojure
DotNet
Elixir
Elixir
Go
Java
Lisp
Lisp
Node.js
Node.js
Node.js
Perl
PHP
PHP
PHP
PHP
Python
Python
Python
Python
Python
Ruby
Ruby
Ruby
Rust
Rust
Scala
Scala
Scala
Swift
4 - Kubernetes Deprecation Policy
fall
Example
Track
v1
v1beta1
v1alpha1
number o

f versions of each.
includes:
API group.
Note:
may not
may not
,
Note:
Note:
Until
#52185
is
objects.
Release
Notes
X
v1alpha1
v1alpha1
X+1
v1alpha2
v1alpha2
X+2
v1beta1
v1b

eta1
X+3
v1beta1
X+4
v1beta2
X+5
v1beta2
X+6
v1
X+7
v1
X+8
v1
X+9
v1
X+10
v1
X+11
v1
X+12
v1
X+13
v2
X+14
v2
X+15
v2
announce
the
Returns a
Warning
Adds a
Sets an
1
in the
group
,
version
,
resource
,
to the
{
=
"
1.22
"}
*
on
(
group
,
version
,
resource
,
)
()
not versioned in the same way as the REST API, the rules for deprecation a

re as
follows:
when used.
/metrics
to determine SLOs, these tend to have 

greater import. Other metrics are more
process.
ALPHA
and
STABLE
);
is equal to
hidden
metric.
Unlike
no longer
be
(
5 - Deprecated API Migration Guide
v1.29
The
v1.29
The
:
v1.27
The
v1.27
The
v1.26
The
v1.26
The

The
v1.25
The
v1.25
CronJob
The
batch/v1
The
:
nodeName
field
zone
field
topology
Event
The
:
type
Normal
and
Warning
regarding
action
,
reason
,
, and
Events
use
eventTime
Events)
use
Events)
use
count
Events)
use
Events)
use
Events)
The
The
policy/v1
policy/v1
:
an empty
(
{}
policy/v1
an empty
Migrate to
or a
.
.
.
v1.22
The
v1.22
release stopped

 serving the following deprecated API versions:
The
Ignore
to
Fail
for v1
Exact
to
for v1
30s
to
10s
for v1
None
and
v1
and
v1beta1
)
The
instead
instead
instead
instead
is moved to
in v1
is moved to
in v1
In
items, the
JSONPath
jsonPath
#66531
)
The
The
The
#32709
)
The
:
API
may not contain dupli

cate types
blocks
Lease
The
Ingress
The
and
pathType
Prefix
,
Exact
, and
v1beta1
.
The
The
The
The
since v1.6
v1.13
v1.16
The
v1.16
The
DaemonSet
The
and
apps/v1
is removed
was
OnDelete
)
The
,
, and
apps/v1
is removed
600
exte

nsions/v1beta1
10
was
2
maxSurge
and
25%
was
1
)
The
and
apps/v1
was
OnDelete
)
The
,
, and
apps/v1
The
What to do
Use
command (
.
apps/v1
.
6 - Kubernetes API health endpoints
API server
healthz
,
livez
and
readyz
The
healthz
livez
and
readyz
The
livez
flag
flag
with the
/readyz
endpoint.
healthz
/
livez
/
readyz
200
healthy
/
live
/
ready
verbose
curl -

k https://localhost:6443/livez?verbose
=
[+]ping ok
[+]log ok
[+]etcd ok
curl -k
etcd
[+]ping ok
[+]log ok
where
livez
and
readyz
The
verbose
[+]
and
ok
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Control CPU Management Policies on the Node | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Tasks
minikube
.
CFS quota
kubelet
flag or the
field in
.
none
static
.
flag.
key=value
policy 

options.
and
Drain
the node.
The
none
and
The
static
requests
using the
.
Note:
Note:
state file
or
and
and
Burstable
CPU
requests
requests
Note:
and/or
or
when
As
placed in the cpuset for the container. CFS 

quota is not used to bound
CPU
limit
workload.
spec
:
:
-
name
:
nginx
image
:
nginx
requests
or
limits
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
Burstable
requests
do not
equal
limits
and the
cpu
pool.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
requests
:
memory
:
"100Mi"
cpu
:
"1"
Burstable
requests
do not
equal
limits
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
requests
:
memory
:
"200Mi"
cpu
:
"2"
requests
limits
.
nginx
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"1.5"
requests
:
memory
:
"200Mi"
cpu
:
"1.5"
requests
limits
.
spec
:
:
-
name
:
nginx
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"2"
limits
and
requests
limits
nginx
policy:
If the
to the
.
.
If the
This can cause undesired bottlenecks in parallel code relying on barri

ers (and
If the
minimum
The
to
cores.
The
to the
and
Feedback
Yes
No
.
or
.
Last modified October 12, 2022 at 6:53 PM PST:
node: cpumgr: docs: Kubelet CPU Manager GA graduation (25771e94f0)
Home
Blog
Training
Partners
Community
CC BY 4.0

Pod | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Pod
Pod
Pod
: v1
kind
: Pod
metadata
(
)
spec
(
PodSpec
)
status
(
PodStatus
)
PodSpec
([]
Container
name
([]
Container
)
name
([]
)
name
([]
)
name
ImagePullSecrets is an optional list of r

eferences to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. More info:
(boolean)
os
(PodOS)
os.name
Volumes
volumes
([]
Volume
)
name
nodeName
(string)
affinity
(
)
(
)
(
)
.
(string)
(string)
(string)
Value is 

the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string.
(string)
(int64)
(string)
(string)
(string)
priority
(int32)
(string)
TopologyKey is the key of node labels. Nodes that have a label with this key and identical values are considered to be in the same topology. We consider each &lt;key, value&gt; as a "bucket", and try to put balanced number of pods into each bucket. We define a dom

ain as a particular instance of a topology. Also, we define an eligible domain as a domain whose nodes meet the requirements of nodeAffinityPolicy and nodeTaintsPolicy. e.g. If TopologyKey is "kubernetes.io/hostname", each Node is a domain of that topology. And, if TopologyKey is "topology.kubernetes.io/zone", each zone is a domain of that topology. It's a required field.
skew.
more
(
)
(int32)
(string)
topologyS

preadConstraints.nodeTaintsPolicy
(string)
overhead
Quantity
)
Lifecycle
(string)
(int64)
(int64)
hostname
(string)
(boolean)
subdomain
(string)
ip
HostAlias holds the mapping between IP and hostnames that will be injected as an entry in the pod's host

s file.
(string)
dnsConfig
(string)
Required.
(string)
dnsPolicy
(string)
(boolean)
hostPID
(boolean)
hostIPC
(boolean)
(boolean)
(string)
(boolean)
(int64)
(boolean)
(int64)
The GID to run the entrypoint of the container proces

s. Uses runtime default if unset. May also be set in SecurityContext. If set in both SecurityContext and PodSecurityContext, the value specified in SecurityContext takes precedence for that container. Note that this field cannot be set when spec.os.name is windows.
([]int64)
(int64)
(string)
(string)
(string)
(string)
(string)
(string)
(str

ing), required
(string)
(string)
(boolean)
(string)
hostUsers
(boolean)
name
(string)
(string)
The template will be used to create a new ResourceClaim, which will be b

ound to this pod. When this pod is deleted, the ResourceClaim will also be deleted. The name of the ResourceClaim will be &lt;pod name&gt;-&lt;resource name&gt;, where &lt;resource name&gt; is the PodResourceClaim.Name. Pod validation will reject the pod if the concatenated name is not valid for a ResourceClaim (e.g. too long).
name
.
(string)
Container
name
Image
image
(string)
(string)
command
args
(string)
Ports
ports
List of ports to expose from the container. Not specifying a port here DOES NOT prevent 

that port from being exposed. Any port which is listening on the default "0.0.0.0" address inside a container will be accessible from the network. Modifying this array with strategic merge patch may corrupt the data. For more information See
(string)
(int32)
(string)
(string)
env
name
env.name
env.value
(string)
(string)
(boolean)
(
)
,
(
)
(string)
(boolean)
envFrom
List of sources to populate environment variables in the container. The keys defined within a source must be a C_IDENTIFIER. All invalid keys will be reported as an event when the container is starting. When a key exists in multiple sources, the value associated with the last source will take precedence. Values defined by an Env with a duplicate key will take precedence. Cannot

 be updated.
(string)
(boolean)
(string)
(string)
(boolean)
Volumes
mountPath
(string)
(boolean)
(string)
(string)
Resources
resources
Quantity
)
Quantity
)
Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defau

lts to Limits if that is explicitly specified, otherwise to an implementation-defined value. More info:
Lifecycle
lifecycle
(
)
(
)
(string)
(string)
(
Probe
)
(
Probe
)
(
Probe
)
https://kubernetes.io/docs/tasks/configure-pod-container/security-context

/
(int64)
(boolean)
(int64)
(boolean)
(string)
(boolean)
(boolean)
(string)
localhostProfile indicates a profile defined in a file on the node should be used. The profile must be preconfigured on the node to work. Must be a descending path, relative to the kubelet's configured seccomp profile location. Must only be set if type is

 "Localhost".
(string)
(string)
(string)
(string)
(string)
(string)
(boolean)
(string)
Debugging
stdin
(boolean)
stdinOnce
(boolean)
tty
(boolean)
An EphemeralContainer is a temporary container that you may add to an existing Pod for user-initiated activities such as debugging. Ephemeral containers have no resource or scheduling guarantees, and they will not be restarted when they exit or when a Pod is removed or restarted. The kubelet may evict a Pod if an ephemeral container causes the Pod to exceed its resource al

location.
name
(string)
Image
image
(string)
(string)
command
args
(string)
env
name
env.name
env.value
(string)
(string)
https://kubernetes.io/docs/concepts/overview/working-wit

h-objects/names/#names
(boolean)
(
)
,
(
)
(string)
(boolean)
envFrom
(string)
(boolean)
(string)
(string)
(boolean)
Volumes
mountPath
(string)
(boolean)
(string)
(string)
Expanded path within the volume from which the container's volume should be mounted. Behaves similarly to SubPath but environment variable references $(VAR_NAME) are expanded using the container's environment. Defaults to "" (volume's root). SubPathExpr and SubPat

h are mutually exclusive.
Lifecycle
(string)
(string)
Debugging
stdin
(boolean)
stdinOnce
(boolean)
tty
(boolean)
(int64)
(boolean)
(int64)
(boolean)
Wheth

er this container has a read-only root filesystem. Default is false. Note that this field cannot be set when spec.os.name is windows.
(string)
(boolean)
(boolean)
(string)
(string)
(string)
(string)
(string)
(string)
GMSACredentialSpec i

s where the GMSA admission webhook (
(string)
(boolean)
(string)
ports
(string)
(int32)
(string)
(string)
resources
Quantity
)
Quantity
)
lifecycle
(
)
PostStart is called immediately a

fter a container is created. If the handler fails, the container is terminated and restarted according to its restart policy. Other management of the container blocks until the hook completes. More info:
(
)
(
Probe
)
(
Probe
)
(
Probe
)
exec
httpGet
(string)
(string)
(string)
tcpSocket
(string)
The sch

eduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. The node that is most preferred is the one with the greatest sum of weights, i.e. for each node that meets all of the scheduling requirements (resource request, requiredDuringScheduling affinity expressions, etc.), compute a sum by iterating through the elements of this field and adding "weight" to the sum if the node matches the corresponding matchExpressions; the node(s) with the highest sum are the most preferred.
([]
)
([]
)
([]
)
([]
)
Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (af

finity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key
(
)
(
)
(
)
(
)
Pod anti affinity is a group of inter pod anti affinity

 scheduling rules.
(
)
(
)
This pod should be co-located (affinity) or not co-located (anti-affinity) with the pods matching the labelSelector in the specified namespace

s, where co-located is defined as running on a node whose value of the label with key topologyKey matches that of any node on which any of the selected pods is running. Empty topologyKey is not allowed.
(
)
(
)
Probe
exec
httpGet
(string)
(string)
(string)
tcpSocket
(string)
(int32)
(int64)
Optional duration in seconds the pod needs to terminate gracefully upon probe failure. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set 

this value longer than the expected cleanup time for your process. If this value is nil, the pod's terminationGracePeriodSeconds will be used. Otherwise, this value overrides the value provided by the pod spec. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). This is a beta field and requires enabling ProbeTerminationGracePeriod feature gate. Minimum value is 1. spec.terminationGracePeriodSeconds is used if unset.
(int32)
(int32)
(int32)
(int32)
grpc
**
grpc.port
(string)
.
PodStatus
(string)
hostIP
(string)
startTime
(Time)
phase
(string)
More info:
https://kubernetes.

io/docs/concepts/workloads/pods/pod-lifecycle#pod-phase
message
(string)
reason
(string)
podIP
(string)
podIPs
([]PodIP)
ip
podIPs.ip
(string)
type
(Time)
(Time)
(string)
(string)
qosClass
(string)
.
(string)
(Time)
Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON. Wrappers are provided for many of t

he factory methods that the time package offers.
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
Message regarding why the container is n

ot yet running.
(string)
(boolean)
.
(string)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(ContainerS

tateRunning)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(boolean)
.
(string)
(Time)
(Co

ntainerStateTerminated)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
(string)
(Time)
(string)
(Time)
(Time)
(string)
(string)
(int32)
(string)
Message regardin

g why the container is not yet running.
(string)
(boolean)
PodList
items
([]
Pod
(string)
kind
(string)
metadata
(
ListMeta
)
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
get
name
(
in path
namespace
(
in path
namespace
container
(
in query
): string
follow
(
in query
): boolean
(
in query
): boolean
(
in query
): integer
pretty
(
in query
): string
pretty
previous
(
in query
): boolean
(
in query
): integer
A relative time in seconds before the current time from which to show logs. If this value precedes the time a pod was started, only logs since the pod start will be returned. If this value is in the future, no logs will be returned. Only one of sinceSeconds or sinceTime ma

y be specified.
tailLines
(
in query
): integer
(
in query
): boolean
Response
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
PodList
): OK
create
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
202 (
Pod
update
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
Pod
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path

namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
Pod
): OK
201 (
Pod
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Pod
): OK
202 (
Pod
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
Pod
PodSpec
Volumes
Lifecycle
Container
Image
Ports
Volumes
Resources
Lifecycle
Debugging
Image
Volumes
Lifecycle
Debugging
Probe
PodStatus
PodList
get
get
get
get
list
list
create
update
upda

te
update
patch
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-scheduler Configuration (v1beta3) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Field
string
kind
string
int32
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
TODO: We might wanna make this a substruct like Debugging componentba

seconfigv1alpha1.DebuggingConfiguration
int32
int64
int64
profiles
extenders
Field
string
kind
string
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
Field
string
kind
string
.
must be empty, as they ar

e
Field
string
kind
string
int64
shape
Extender
Field
urlPrefix
string
string
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool
will skip chec

king the resource in predicates.
ignorable
bool
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
Field
string
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
Field
name
string
args
PluginSet
Plugins
Field
enabled
[]Plugin
be ove

rridden accordingly.
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSet
PluginSet
preScore
PluginSet
score
PluginSet
reserve
PluginSet
permit
PluginSet
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
set in both
and
including
and
(Alias of
string
)
Sco

ringStrategy
Field
shape
Field
name
string
weight
int64
Field
type
resources
(Alias of
string
)
Field
int32
score
int32
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
enabled.
retryPeriod is the duration the clients should wait 

between attempting
string
string
string
Feedback
Yes
No
.
or
.
Extender
Plugin
PluginSet
Plugins
Home
Blog
Training
Partners
Community
CC BY 4.0

Feature Gates | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install an

ation (v1)
Reference
See
Overview
Use
-h
=
=
true
in the
.
.
The
Note:
.
Feature
Default
Stage
Since
Until
false
Alpha
1.8
1.8
true
Beta
1.9
false
Alpha
1.18
1.19
true
Beta
1.20
false
Alpha
1.7
1.15
true
Beta
1.16
false
Alpha
1.26
false
Alpha
1.20
1.25
true
Beta
1.26
false
Alpha
1.22
false
Alpha
1.26
false
Alpha
1.18
1.23
true
Beta
1.24
AppArmor
true
Beta
1.4
false
Alpha
1.23
true
Beta
1.23
false
Alpha
1.22
1.22
true
Beta
1.23
false
Alpha
1.23
1.24
false
Beta
1.25
false
Alpha
1.23
false
Alpha
1.25
false
Alpha
1.21
false
Alpha
1.26
ContainerChe

ckpoint
false
Alpha
1.25
false
Alpha
1.24
false
Alpha
1.24
1.24
true
Beta
1.25
false
Alpha
1.26
false
Alpha
1.12
false
Alpha
1.23
1.24
true
Beta
1.25
false
Alpha
1.22
false
Alpha
1.23
false
Alpha
1.20
1.20
false
Beta
1.21
1.21
true
Beta
1.22
false
Alpha
1.26
false
Alpha
1.26
-
false
Alpha
1.22
1.25
true
Beta
1.26
false
Beta
1.5
false
Alpha
1.23
1.23
true
Beta
1.24
false
Alpha
1.20
1.20
true
Beta
1.21
false
Alpha
1.23
1.23
true
Beta
1.24
false
Alpha
1.20
false
Alpha
1.16
false
Alpha
1.23
false
Alpha
1.25
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.21
false
Alpha
1.23
false
Alpha
1.23
false
Alpha
1.21
true
Beta
1.23
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.23
1.23
true
Beta
1.24
KMSv2
false
Alpha
1.25
false
Alpha
1.22
false
Alpha
1.13
1.14
true
Beta
1.15
false
Alpha
1.21
1.22
true
Beta
1.23
false
Alpha
1.25
false
Alpha
1.25
false
Alpha
1.15
-
false
Alpha
1.21
1.21
true
Beta
1.22
false
Alpha
1.24
-
true
Beta
1.24
-
false
Alpha
1.25
false
Alpha
1.24
false
Alpha
1.21
1.21
true
Beta
1.22
MemoryQoS
false
Alpha
1.22
false
Alpha
1.24
1.24
false
Beta
1.25
false
Alpha
1.26
-
false
Alpha
1.25
false
Alpha
1.24
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.24
1.25
true
Beta
1.26
NodeSwap
false
Alpha
1.22
false
Alpha
1.23
1.23
true
Beta
1.24
OpenAPIV3
false
Alpha
1.23
1.23
OpenAPIV3
true
Beta
1.24
false
Alpha
1.26
false
Alpha
1.23
false
Alpha
1.21
1.21
true
Beta
1.22
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.25
false
Alpha
1.26
false
Alpha
1.21
1.21
false
Beta
1.22
1.24
true
Beta
1.25
false
Alpha
1.12
false
Alpha
1.22
1.25
true
Beta
1.26
false
Alpha
1.11
false
Alpha
1.22
false
Alpha
1.23
false
Alpha
1.15
1.15
true
Beta
1.16
false
Alpha
1.25
1.25
true
Beta
1.26
false
Alpha
1.7
1.11
true
Beta
1.

12
false
Alpha
1.25
false
Alpha
1.22
1.24
true
Beta
1.25
false
Alpha
1.23
1.24
true
Beta
1.25
false
Alpha
1.20
1.21
true
Beta
1.22
false
Alpha
1.22
false
Alpha
1.26
false
Alpha
1.20
false
Alpha
1.14
1.14
true
Beta
1.15
false
Alpha
1.21
1.22
false
Beta
1.23
1.23
true
Beta
1.24
false
Alpha
1.16
1.17
true
Beta
1.18
false
Alpha
1.26
false
Beta
1.26
false
Alpha
1.26
false
Alpha
1.25
false
Alpha
1.26
false
Alpha
1.21
-
WinDSR
false
Alpha
1.14
false
Alpha
1.14
1.19
true
Beta
1.20
false
Alpha
1.26
Feature
Default
Stage
Since
Until
false
Alpha
1.7
1.7
true
Beta
1.8
1.11
true
GA
1.12
-
false
Alpha
1.8
1.9
true
Beta
1.10
1.25
true
GA
1.26
-
false
Alpha
1.15
1.15
true
Beta
1.16
1.24
true
GA
1.25
-
false
Alpha
1.14
1.16
true
Beta
1.17
1.24
true
GA
1.25
-
false
Alpha
1.14
1.16
false
Beta
1.17
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
false
Alpha
1.15
1.18
false
Beta
1.19
1.22
true
Beta
1.23
1.23
true
GA
1.24
false
Alpha
1.15
1.20
false
Beta
1.21
1.23
true
Beta
1.24
1.25
true
GA
1.26
false
Alpha
1.14
1.16
false
Beta
1.17
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
false
Alpha
1.18
1.18
false
Beta
1.19
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
false
Alpha
1.19
1.20
true
Beta
1.21
1.23
true
GA
1.24
-
true
GA
1.25
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.23
true
GA
1.24
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.24
true
GA
1.25
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.25
true
GA
1.26
-
false
Alpha
1.8
1.9
true
Beta
1.10
1.25
true
GA
1.26
-
false
Alpha
1.19
1.19
true
Beta
1.20
1.24
true
GA
1.25
-
DryRun
false
Alpha
1.12
1.12
DryRun
true
Beta
1.13
1.18
DryRun
true
GA
1.19
-
false
Alpha
1.20
1.20
true
Beta
1.21
1.23
true
GA
1.24
-
false
Alpha
1.20
1.21
true
Beta
1.22
1.25
true
GA
1.26
false
Alpha
1.16
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
true
GA
1.20
-
false
Alpha
1.14
1.15
true
Beta
1.16
1.23
true
GA
1.24
-
false
Alpha
1.11
1.14
true
Beta
1.15
1.23
true
GA
1.24
-
false
Alpha
1.8
1.10
Ex

pandPersistentVolumes
true
Beta
1.11
1.23
true
GA
1.24
-
false
Alpha
1.23
1.23
true
Beta
1.24
1.24
true
GA
1.25
-
false
Alpha
1.22
1.22
false
Beta
1.23
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
false
Alpha
1.20
1.23
true
Beta
1.24
1.25
true
GA
1.26
-
true
Beta
1.24
1.25
true
GA
1.26
-
false
Alpha
1.7
1.9
true
Beta
1.10
1.24
true
GA
1.25
-
false
Alpha
1.20
1.23
true
Beta
1.24
1.25
true
GA
1.26
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.24
true
GA
1.25
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.24
true
GA
1.25
false
Alpha
1.16
1.19
true
Beta
1.20
1.23
true
GA
1.24
-
false
Alpha
1.14
1.15
true
Beta
1.16
1.21
true
GA
1.22
-
false
Alpha
1.24
1.24
true
Beta
1.25
1.25
true
GA
1.26
-
false
Alpha
1.21
1.21
true
Beta
1.22
1.25
true
GA
1.26
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.24
true
GA
1.25
-
false
Alpha
1.15
1.15
true
Beta
1.16
1.16
true
GA
1.17
-
false
Alpha
1.22
1.22
true
Beta
1.23
1.25
true
GA
1.26
-
Alpha
,
Beta
or
GA
stage.
An
Alpha
A
Beta
Note:
Beta
A
stable
LIST
or
GET
)
)
: Compress the A

PI responses for
LIST
or
GET
requests.
See
See
: Enable
which
of a
PVC
.
AppArmor
See
API.
See
and
.
a group
a group
plugin must be installed and configured. Requires CSIMigration featur

e flag
enabled.
enabled.
during a
.
Check the
csi
and
.
timeZone
CronJobs
field of a
in
.
the
See
.
fsGroup
passing
fsGroup
:
.
,
and
kubelet
.
DryRun
dry run
requests
and
serving
: Allows for storage-

originated bookmark (progress
via
CRI
.
.
.
.
.
MKNODE
,
See
.
Delete
to 0 for
are
windows
and
linux
.
InTreePluginvSphereU

nregister
Job
.
Ready
condition
.
Ready
status
of a
Job
status.
Job
KMSv2
.
See
.
See
.
.
sizeLimit
.
: When
field for
.
MemoryQoS
: Enable
in
.
type
endPort
status
and
in
when calculating pod topology spread 

skew.
NodeSwap
OpenAPIV3
field of a
.
: Enable
procMount
.
See
.
: Sets the
See
volume.
ServerSideFieldValid

ation
or
emptyDir
volumes).
: Allows
.
Hints
for more
details.
.
a group
a group
: Enable
WinDSR
The
, set
.
See
Feedback
Yes
No
.
or
.
Last modified January 03, 2023 at 5:02 PM PST:
Add ComponentSLIs feature gate (50028e56de)
Overview
Home
Blog
Training
Partners
Community
CC BY 4.0

List All Container Images Running in a Cluster | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
minikube
.
using
image
See the
tr
,
sort
,
uniq
Use
tr
Use
sort
Use
uniq
jsonpath
=
|
\
tr -s
'\n'
|
\
sort |
\
uniq -c
.items[*]
.spec
.image
Note:
,
the
.items[*]
range
jsonpath
=
|
\
sort
app=nginx
.
jsonpath
=
-l
app
=
nginx
List Container images filtering by Pod namesp

ace
jsonpath
=
=
Reference
Jsonpath
Feedback
Yes
No
.
or
.
Last modified September 29, 2022 at 8:52 AM PST:
Remove the description of the old jsonpath ..image, Retain the explanation of the new jsonpath (f22034ad40)
Reference
Home
Blog
Training
Partners
Community
CC BY 4.0

Documentation Content Guide | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

and ask!
.
.
Overview
project
.
and
, and
logging
.
Note:
.
Read the
.
Feedback
Yes
No
.
or
.
Overview
More inf

ormation
Home
Blog
Training
Partners
Community
CC BY 4.0

ReplicationController | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
: v1
kind
metadata
(
)
spec
(
)
status
(
)
selector
template
(
)
replicas
(int32)
(int32)
ReplicationControllerStatus represents the current status of a re

plication controller.
replicas
(int32)
(int32)
(int32)
type
(Time)
(string)
(string)
(int64)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string

ch
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): stri

ng
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Content organization | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install a

1)
Note:
Hugo Tip:
Page Lists
Page Order
title
:
My Page
weight
:
10
Note:
The
docs/
with the
main_menu
_index.md
main_menu
:
true
linkTitle
main_menu
:
true
title
:
Page Title
linkTitle
:
Note:
_index.md
docs/
.
toc_hide
flag to
true
toc_hide
:
true
_index.md
.
toc_hide
flag to
true
toc_hide
:
true
In addition to standalone content pages

 (Markdown files), Hugo supports
.
index.md
includes
Resources
.
of a
Resource
.
Styles
The
SASS
Feedback
Yes
No
.
or
.
Page Lists
Page Order
Styles
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Kubernet

es API Aggregation Layer
Tasks
Debug Pods
Debug a S

tatefulSet
Auditing
Run Jobs
TLS
Tutorials
Security
Apply Pod

 Security Standards at the Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tasks
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Device Plu

gins
Tasks
Troub

leshooting Applications
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configuring a Java Microser

vice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

ver Admission (v1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

DeleteOptions | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
(string)
dryRun
(int64)
kind
(string)
(boolean)
(string)
(string)
(string)
This pag

e is automatically generated.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Tutorials
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Network

alpha1)
Tutorials
task
Basics
Services
Security
AppArmor
seccomp
Feedback
Yes
No
.
or
.
Basics
Services
Security
Home
Blog
Training
Partners
Community
CC BY 4.0

kube-scheduler Configuration (v1) | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
Field
string
kind
string
int32
int32
Field
string
kind
string
int32
Field
string
kind
string
int32
TODO: We might wanna make this a substruct like Debugging componentbaseconfigv1alpha1.DebuggingConfigura

tion
int32
int64
int64
profiles
extenders
Field
string
kind
string
match).
Field
string
kind
string
resources
Field
string
kind
string
[]string
[]string
Field
string
kind
string
.
must be empty, as th

ey are
Field
string
kind
string
int64
shape
Extender
Field
urlPrefix
string
string
string
string
weight
int64
bindVerb
string
bool
tlsConfig
bool

ignorable
bool
Extender
Field
name
string
bool
Extender
Field
insecure
bool
string
certFile
string
keyFile
string
caFile
string
certData
[]byte
keyData
[]byte
caData
[]byte
Field
string
int32
plugins
Plugins
Plugin
PluginSet
Field
name
string
weight
int32
A plugin that is invoked at multiple extension points is initialized once. Args can 

have arbitrary structure.
Field
name
string
args
PluginSet
Plugins
Field
enabled
[]Plugin
disabled
[]Plugin
Plugins
Field
PluginSet
queueSort
PluginSet
preFilter
PluginSet
filter
PluginSet
PluginSet
preScore
PluginSet
score
PluginSet
reserve
PluginSet
permit
PluginSet
preBind
PluginSet
bind
PluginSet
postBind
PluginSet
PluginSet
This implies that a higher precedence plugin will run first and overwrite any sett

ings within MultiPoint.
set in both
and
including
and
(Alias of
string
)
Field
shape
Field
name
string
weight
int64
Field
type
resources
(Alias of
string
)
Field
int32
score
int32
Field
string
string
client.
string
qps
float32
burst
int32
Field
bool
bool
Field
bool
before exec

uting the main loop. Enable this when running replicated
enabled.
string
string
string
Feedback
Yes
No
.
or
.
Extender
Plugin
PluginSet
Plugins
Home
Blog
Training
Partners
Community
CC BY 4.0

Best practices | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

Feedback
Yes
No
.
or
.
Last modified June 12, 2019 at 5:27 PM PST:
Restructure the left navigation pane of setup (#14826) (55ac801bc4)
Home
Blog
Training
Partners
Community
CC BY 4.0

ControllerRevision | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

Reference
: apps/v1
kind
metadata
(
)
revision
data
}
}
}
}
{
},
}
ControllerRevisio

nList is a resource containing a list of ControllerRevision objects.
: apps/v1
kind
metadata
(
ListMeta
)
More info:
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string

dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
Status
): OK
202 (
Status
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
list
list
create
update
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0

Example: Deploying PHP Guestbook application with Redis | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
Debug Pods
Determine the

 Reason for Pod Failure
Auditing
Run Jobs
TLS
Tutorials
Security

he Cluster Level
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

tion (v1)
Tutorials
ready)
Docker
Redis
Clean up.
minikube
.
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
replicas
:
1
selector
:
:
app
:
redis
template
:
metadata
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
:
-
name
:
leader
image
:
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
6379
file:
The guestbook application needs to communicat

e to the Redis to write its data.
Service
to
Pods.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
redis
role
:
leader
tier
:
backend
spec
:
ports
:
-
port
:
6379
:
6379
selector
:
app
:
redis
role
:
leader
tier
:
backend
file:
Note:
:
apps/v1
kind
:
metadata
:
name
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
replicas
:
2
selector
:
:
app
:
redis
template
:
metadata
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
:
-
name
:
follower
image
:
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
6379
file:
Service
.
:
v1
kind
:
Service
metadata
:
name
:
labels
:
app
:
redis
role
:
follower
tier
:
backend
spec
:
ports
:
-
port
:
6379
selector
:
app
:
redis
role
:
follower
tier
:
backend
file:
Note:
the guestbook web server

s. Like the Redis followers, the frontend is deployed
:
apps/v1
kind
:
metadata
:
name
:
frontend
spec
:
replicas
:
3
selector
:
:
app
:
guestbook
tier
:
frontend
template
:
metadata
:
labels
:
app
:
guestbook
tier
:
frontend
spec
:
:
-
name
:
php-redis
image
:
env
:
-
name
:
value
:
"dns"
resources
:
requests
:
cpu
:
100m
memory
:
100Mi
ports
:
-
:
80
file:
app
=
tier
=
frontend
The
Redis
ClusterIP
.
ClusterIP
ClusterIP
.
Note:
.
:
v1
kind
:
Service
metadata
:
name
:
frontend
labels
:
app
:
guestbook
tier
:
frontend
spec
:
ports
:
-
port
:
80
selector
:
app
:
guestbook
tier
:
frontend
file:
8080
80
manifest 

with type:
Note:
=
5
=
2
app
=
redis
app
=
redis
Feedback
Yes
No
.
or
.
Creating the Fr

ontend Service
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
.
.
1:
2:
3:
4:
5:
1 - Considerations for large clusters
nodes
(physical
.
all
consider:
CPUs
A
A
A
A
API server
See
and
Set up a High Availability etcd cluster with kubeadm
addon
...
:
-
name
:
image
:
resources
:
limits
:
cpu
:
100m
memory
:
200Mi
may continuously get killed because they keep 

hitting the memory limit.
DaemonSet
The
The
2 - Running in multiple zones
region
All
Note:
or
)
labels
.
scheduler
to place
without explicitly defining which availability zones are in use for
each workloa

d.
to
zone. The
scheduler
see
.
Service
and
Ingress
visit
.
3 - Validate node setup
Kubelet
.
use
=
host
\
:
$CONFIG

_DIR
-v
$LOG_DIR
\
Arch
Image
amd64
arm
arm64
FOCUS
with the
=
host
\
:
-v
$LOG_DIR
\
-e
FOCUS
=
MirrorPod
\
SKIP
with the
=
host
\
:
-v
$LOG_DIR
\
-e
SKIP
=
MirrorPod
\
.
test
Caveats
test.
4 - Enforcing Pod Security Standards
.
The
example
The
audit
and
warn
desired
level
enforce
warn
audit
mode.
enforce
By setting
warn
enforce
to create Pods (or resources that have Pod templates) 

that do not pass validation. This will help
enforce
audit
and
warn
enforce
, but to the
latest
Note:
.
Kyverno
.
.
built-in
any
5 - PKI certificates and requirements
kubeadm
Kubelet
Note:
.
.
.
path
Default CN
etcd-ca
For the
sa.key
and
sa.pub
.
The following example illustrates the CA key and certificate files shown in the previous table:
Req

uired certificates:
Default CN
Parent CA
kind
kube-etcd
etcd-ca
,
,
localhost
,
127.0.0.1
etcd-ca
,
,
localhost
,
127.0.0.1
etcd-ca
client
etcd-ca
client
server
,
,
,
[1]
client
client
kubeadm
,
,
,
,
)
where
kind
of a
type:
kind
Key usage
server
client
Note:
Note:
kube-etcd
,
and
kubeadm
).
Default CN
command
etcd-ca
ca.key
ca.crt
ca.key
ca.crt
etcd-ca
etcd
kube-etcd
etcd
etcd
etcd-ca
etcdctl
--cacert
etcdctl
--key
--cert
command
argument
sa.key
sa.pub
/etc/kubernetes/pki/apise

rver-kubelet-client.crt
filename
Default CN
(see note)
Note:
for
must
.
Run
kubectl
filename
command
comment
kubectl
kubelet
Home
Blog
Training
Partners
Community
CC BY 4.0

ResourceQuota | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
In

Reference
: v1
kind
metadata
(
)
spec
(
)
status
(
)
hard
Quantity
)
scopes
hard
Quantity
)
Hard is the set of enforced hard limits for each na

med resource. More info:
used
Quantity
)
: v1
kind
metadata
(
ListMeta
)
items
([]
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
get
name
(
in path
namespace
(
in path
namespace
pretty
(
in query
): string
pretty
Response
200 (
): OK
list
namespace
(
in path
namespace
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
list
(
in query
): boolean
continue
(
in query
): string
continue
(
in query
): string
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): integer
watch
(
in query
): boolean
watch
Response
200 (
): OK
create
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
202 (
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
update
name
(
in path
namespace
(
in path
namespace
body
:
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string

ion
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
patch
name
(
in path
namespace
(
in path
namespace
body
:
Patch
, required
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): string
force
(
in query
): boolean
force
pretty
(
in query
): string
pretty
Response
200 (
): OK
201 (
): Created
delete
name
(
in path
namespace
(
in path
namespace
body
:
dryRun
(
in query
): string
dryRun
(
in query
): integer
pretty
(
in query
): string
pretty
(
in query
): string
Response
200 (
): OK
202 (
namespace
(
in path
namespace
body
:
continue
(
in query
): string
continue
dryRun
(
in query
): string
dryRun
(
in query
): string
(
in query
): integer
(
in query
): string
limit
(
in query
): integer
limit
pretty
(
in query
): string
pretty
(
in query
): string
(
in query
): string
(
in query
): string
(
in query
): integer
Response
200 (
Status
): OK
Feedback
Yes
No
.
or
.
get
get
list
list
create
update
update
patch
patch
delete
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Linux Foundation has registered trademarks and us

es trademarks. For a list of trademarks of The Linux Foundation, please see our

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Extending the Ku

bernetes API
Tasks
Troubleshooting Applicatio

ns
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Configuring Redi

s using a ConfigMap
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet
ku

be-apiserver Audit Configuration (v1)
Tutorials
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Patch | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

1)
Reference
Patch
Patch
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Managing Secrets using kubectl | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Ins

Tasks
Secrets
using the
kubectl
minikube
A
Secret
admin
.
\
=
username
=
admin
\
=
password
=
''
$
,
\
,
*
,
=
, and
!
echo
-n
'admin'
echo
-n
The
-n
kubectl
kubectl
command:
\
=
\
=
using
\
=
username
=
\
=
password
=
db-user-pass      Opaqu

e     2         51s

Data
====
and
of a
Secret
Secret
jsonpath
=
'{.data}'
{
:
,
:
}
Decode the
password
data:
echo
Caution:
jsonpath
=
Secret
immutable
. To edit a
data
#
:
v1
data
:
password
:
username
:
YWRtaW4=
kind
:
Secret
metadata
:
:
name
:
namespace
:
default
:
uid
:
type
:
Opaque
Clean up
Feedback
Yes
No
.
or
.
Last modified December 10, 2022 at 8:00 AM PST:
Remove Base64'ing of file contents (92eb40073f)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0

Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks

(v1)
Tasks
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Set up Ingress on Minikube with the NGINX Ingress Controller | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
An
Ingress
minikube
.
Locally
enable
ingress
Note:
Note:
storage-provision

er                         1/1       Running   0          2m
.
=
=
=
8080
Note:
31637
.
Create
:
kind
:
Ingress
metadata
:
name
:
:
:
/$1
spec
:
rules
:
-
host
:
http
:
paths
:
-
path
:
/
pathType
:
Prefix
backend
:
service
:
name
:
web
port
:
number
:
8080
Note:
file on
Note:
Note:
=
=
8080
--type
=
NodePort
-
path
:
/v2
pathType
:
Prefix
backend
:
service
:
name
:
web2
port
:
number
:
8080
Note:
If you are running Minikube locally, you can visit

 hello-world.info and hello-world.info/v2 from your browser.
Ingress
Services
Feedback
Yes
No
.
or
.
Last modified March 21, 2022 at 9:15 PM PST:
second yaml snippet was not alligned to first one (2176c33e62)
Home
Blog
Training
Partners
Community
CC BY 4.0

Guaranteed Scheduling For Critical Add-On Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Tasks
Install T

Tasks
or
.
.
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0

Running Multiple Instances of Your App | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies
Network Plugin

s
Tasks
Monitoring, Logging, and Deb

ugging
Debug Pods
Auditing
Run Jobs
TLS
Tutorials
Interactive Tutorial - Configuring a J

ava Microservice
Security
Services
Reference
Glossary
Pod
DaemonSet
Job
CronJob
Service
Endpoints
Ingress
ConfigMap
Secret
Volume
CSIDriver
CSINode
Role
Node
Namespace
Event
Lease
Binding
ListMeta
Patch
Quantity
Status
Kubeadm
kubectl
kubelet

Tutorials
Service
Scaling
Summary:
Previous
Next
›
Feedback
Yes
No
.
or
.
Home
Blog
Training
Partners
Community
CC BY 4.0
Copyright © 2023 The Linux Foundation ®. All rights reserved. The Li

nux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our

Configure Quality of Service for Pods | Kubernetes
Training
Partners
Community
Versions
v1.26
v1.25
v1.24
v1.23
v1.22
English
Home
Concepts
Overview
Nodes
Leases
Images
Workloads
Pods
DaemonSet
Jobs
CronJob
Service
Ingress
Storage
Volumes
Secrets
Security
Policies

Tasks
minikube
.
Burstable
:
v1
kind
:
Pod
metadata
:
name
:
qos-demo
namespace
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
cpu
:
"700m"
requests
:
memory
:
"200Mi"
cpu
:
"700m"
=
=
=
yaml
spec
:
:
...
resources
:
limits
:
cpu
:

700m
memory
:
200Mi
requests
:
cpu
:
700m
memory
:
200Mi
...
status
:
qosClass
:
Note:
the limit.
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
nginx
resources
:
limits
:
memory
:
"200Mi"
requests
:
memory
:
"100Mi"
=
=
=
yaml
spec
:
:
-
image
:
nginx
:
Always
name
:
resources
:
limits
:
memory
:
200Mi
requests
:
memory
:
100Mi
...
status
:
qosClass
:
Burstable
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
nginx
=
=
=
yaml
spec
:
:
...
resources
:
{}
...
status
:
qosClass
:
=
:
v1
kind
:
Pod
metadata
:
name
:
namespace
:
spec
:
:
-
name
:
image
:
nginx
resources
:
requests
:
memory
:
"200Mi"
-
name
:
image
:
redis
=
=
=
yaml
spec
:
:
...
name
:
resources
:
requests
:
memory
:
200Mi
...
name
:
resources
:
{}
...
status
:
qosClass
:
Burstable
=
Clean up
Configure Default

 CPU Requests and Limits for a Namespace
Feedback
Yes
No
.
or
.
Last modified April 02, 2022 at 1:09 PM PST:
Update quality-service-pod.md (82456a84de)
Clean up
Home
Blog
Training
Partners
Community
CC BY 4.0