<commit_before>---
layout:       post
title:        "kubernetes管理经验"
subtitle:     "Kubernetes Management Experience"
date:         2019-01-25
author:       "Zeusro"
header-img:   "img/oYYBAFHlDveICOlTAAWdBpjTP2sAAAvzgB9mBEABZ0e231.jpg"
header-mask:  0.3
catalog:      true
tags:
    - Kubernetes
---

## 集群管理经(教)验(训)

### 节点问题


#### taint别乱用

```bash
kubectl taint nodes xx  elasticsearch-test-ready=true:NoSchedule
kubectl taint nodes xx  elasticsearch-test-ready:NoSchedule-
```

master节点本身就自带taint,所以才会导致我们发布的容器不会在master节点上面跑.但是如果自定义`taint`的话就要注意了!所有`DaemonSet`和kube-system,都需要带上相应的`tolerations`.不然该节点会驱逐所有不带这个`tolerations`的容器,甚至包括网络插件,kube-proxy,后果相当严重,请注意

`taint`跟`tolerations`是结对对应存在的,操作符也不能乱用

##### NoExecute


```yml
      tolerations:
        - key: "elasticsearch-exclusive"
          operator: "Equal"
          value: "true"
          effect: "NoExecute"
```

  kubectl taint node cn-shenzhen.xxxx  elasticsearch-exclusive=true:NoExecute

NoExecute是立刻驱逐不满足容忍条件的pod,该操作非常凶险,请务必先行确认系统组件有对应配置tolerations.

特别注意用`Exists`这个操作符是无效的,必须用`Equal`

##### NoSchedule

```yml
      tolerations:
        - key: "elasticsearch-exclusive"
          operator: "Exists"
          effect: "NoSchedule"
        - key: "elasticsearch-exclusive"
          operator: "Equal"
          value: "true"
          effect: "NoExecute"
```

  kubectl taint node cn-shenzhen.xxxx  elasticsearch-exclusive=true:NoSchedule

是尽量不往这上面调度,但实际上还是会有pod在那上面跑

`Exists`和`Exists`随意使用,不是很影响

值得一提的是,同一个key可以同时存在多个effect

```yml
Taints:             elasticsearch-exclusive=true:NoExecute
                    elasticsearch-exclusive=true:NoSchedule
```

其他参考链接：

1. [Kubernetes中的Taint和Toleration（污点和容忍）](https://jimmysong.io/posts/kubernetes-taint-and-toleration/)
1. [kubernetes的调度机制](https://segmentfault.com/a/1190000012709117#articleHeader8)


#### 删除节点的正确步骤

```bash
# SchedulingDisabled,确保新的容器不会调度到该节点
kubectl cordon $node
# 驱逐除了ds以外所有的pod
kubectl drain $node   --ignore-daemonsets
kubectl delete $node
```

#### 维护节点的正确步骤

```bash
# SchedulingDisabled,确保新的容器不会调度到该节点
kubectl cordon $node
# 驱逐除了ds以外所有的pod
kubectl drain $node --ignore-daemonsets
# 维护完成,恢复其正常状态
kubectl uncordon $node
```

#### 节点出现磁盘压力(DiskPressure)

```
--eviction-hard=imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5%
```

kubelet在启动时指定了磁盘压力,以阿里云为例,`imagefs.available<15%`意思是说容器的读写层少于15%的时候,节点会被驱逐.节点被驱逐的后果就是产生DiskPressure这种状况,并且节点上再也不能运行任何镜像,直至磁盘问题得到解决.如果节点上容器使用了宿主目录,这个问题将会是致命的.因为你不能把目录删除掉,但是真是这些宿主机的目录堆积,导致了节点被驱逐.

所以,平时要养好良好习惯,容器里面别瞎写东西(容器里面写文件会占用ephemeral-storage,ephemeral-storage过多pod会被驱逐),多使用无状态型容器,谨慎选择存储方式,尽量别用hostpath这种存储

出现状况时,真的有种欲哭无泪的感觉.

```
Events:
  Type     Reason                 Age                   From                                            Message
  ----     ------                 ----                  ----                                            -------
  Warning  FreeDiskSpaceFailed    23m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 5182058496 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed    18m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 6089891840 bytes, but freed 0 bytes<commit_msg>维护节点的正确步骤<commit_after>---
layout:       post
title:        "kubernetes管理经验"
subtitle:     "Kubernetes Management Experience"
date:         2019-01-25
author:       "Zeusro"
header-img:   "img/oYYBAFHlDveICOlTAAWdBpjTP2sAAAvzgB9mBEABZ0e231.jpg"
header-mask:  0.3
catalog:      true
tags:
    - Kubernetes
---

## 集群管理经(教)验(训)

### 节点问题


#### taint别乱用

```bash
kubectl taint nodes xx  elasticsearch-test-ready=true:NoSchedule
kubectl taint nodes xx  elasticsearch-test-ready:NoSchedule-
```

master节点本身就自带taint,所以才会导致我们发布的容器不会在master节点上面跑.但是如果自定义`taint`的话就要注意了!所有`DaemonSet`和kube-system,都需要带上相应的`tolerations`.不然该节点会驱逐所有不带这个`tolerations`的容器,甚至包括网络插件,kube-proxy,后果相当严重,请注意

`taint`跟`tolerations`是结对对应存在的,操作符也不能乱用

##### NoExecute


```yml
      tolerations:
        - key: "elasticsearch-exclusive"
          operator: "Equal"
          value: "true"
          effect: "NoExecute"
```

  kubectl taint node cn-shenzhen.xxxx  elasticsearch-exclusive=true:NoExecute

NoExecute是立刻驱逐不满足容忍条件的pod,该操作非常凶险,请务必先行确认系统组件有对应配置tolerations.

特别注意用`Exists`这个操作符是无效的,必须用`Equal`

##### NoSchedule

```yml
      tolerations:
        - key: "elasticsearch-exclusive"
          operator: "Exists"
          effect: "NoSchedule"
        - key: "elasticsearch-exclusive"
          operator: "Equal"
          value: "true"
          effect: "NoExecute"
```

  kubectl taint node cn-shenzhen.xxxx  elasticsearch-exclusive=true:NoSchedule

是尽量不往这上面调度,但实际上还是会有pod在那上面跑

`Exists`和`Exists`随意使用,不是很影响

值得一提的是,同一个key可以同时存在多个effect

```yml
Taints:             elasticsearch-exclusive=true:NoExecute
                    elasticsearch-exclusive=true:NoSchedule
```

其他参考链接：

1. [Kubernetes中的Taint和Toleration（污点和容忍）](https://jimmysong.io/posts/kubernetes-taint-and-toleration/)
1. [kubernetes的调度机制](https://segmentfault.com/a/1190000012709117#articleHeader8)


#### 删除节点的正确步骤

```bash
# SchedulingDisabled,确保新的容器不会调度到该节点
kubectl cordon $node
# 驱逐除了ds以外所有的pod
kubectl drain $node   --ignore-daemonsets
kubectl delete $node
```

#### 维护节点的正确步骤

```bash
# SchedulingDisabled,确保新的容器不会调度到该节点
kubectl cordon $node
# 驱逐除了ds以外所有的pod
kubectl drain $node --ignore-daemonsets --delete-local-data
# 维护完成,恢复其正常状态
kubectl uncordon $node
```

--delete-local-data 是忽略 `emptyDir`这类的临时存储的意思



#### 节点出现磁盘压力(DiskPressure)

```
--eviction-hard=imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5%
```

kubelet在启动时指定了磁盘压力,以阿里云为例,`imagefs.available<15%`意思是说容器的读写层少于15%的时候,节点会被驱逐.节点被驱逐的后果就是产生DiskPressure这种状况,并且节点上再也不能运行任何镜像,直至磁盘问题得到解决.如果节点上容器使用了宿主目录,这个问题将会是致命的.因为你不能把目录删除掉,但是真是这些宿主机的目录堆积,导致了节点被驱逐.

所以,平时要养好良好习惯,容器里面别瞎写东西(容器里面写文件会占用ephemeral-storage,ephemeral-storage过多pod会被驱逐),多使用无状态型容器,谨慎选择存储方式,尽量别用hostpath这种存储

出现状况时,真的有种欲哭无泪的感觉.

```
Events:
  Type     Reason                 Age                   From                                            Message
  ----     ------                 ----                  ----                                            -------
  Warning  FreeDiskSpaceFailed    23m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 5182058496 bytes, but freed 0 bytes
  Warning  FreeDiskSpaceFailed    18m                   kubelet, node.xxxx1     failed to garbage collect required amount of images. Wanted to free 6089891840 bytes, but freed 0 bytes
{'fraction_non_alphanumeric': 0.10898021308980213, 'fraction_numerical': 0.0228310502283105, 'mean_word_length': 3.261348897535668, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1447533', 'n_tokens_mistral': 3219, 'n_tokens_neox': 3268, 'n_words': 349}