<commit_before>=== 测试、生产环境安装配置

[IMPORTANT]
====
本文会给出使用代理与不使用代理的安装、配置方式，但强烈推荐使用代理方式，详见 <<proxies>> 。
====

*以 Centos7 为例，做好ssh免密互访、关闭防火墙、关闭swap、禁用SELINUX*

[source,bash]
----
systemctl stop firewalld.service
systemctl disable firewalld.service
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config
----

.服务列表
|===
|主机名 |IP |服务 | 备注

|devops | 10.200.10.10 | Gitlab | Gitlab及其CI/CD独立一台服务器部署，实际环境应该做HA
|middleware | 10.200.10.11 | Docker、RabbitMQ、PostgreSql、Redis、NFS、dnsmasq | 各类中间件，实际环境应该做HA
|k8s0 | 10.200.10.12 | Docker、kubernetes master、Helm |
|k8s1 | 10.200.10.13 | Docker、kubernetes node |
|k8s2 | 10.200.10.14 | Docker、kubernetes node |
|k8s3 | 10.200.10.15 | Docker、kubernetes node |
|k8s4 | 10.200.10.16 | Docker、kubernetes node |
|…… | …… | Docker、kubernetes node |
|===

[source,bash]
.各节点Host
----
# 除middleware外的各节点

cat >>/etc/hosts <<EOF
10.200.10.10 devops
10.200.10.11 middleware
10.200.10.12 k8s0
10.200.10.13 k8s1
10.200.10.14 k8s2
10.200.10.15 k8s3
10.200.10.16 k8s4
EOF

# middleware节点和客户端节点

cat >>/etc/hosts <<EOF
10.200.10.10 devops gitlab.dew.ms
10.200.10.11 middleware rabbitmq.dew.ms redis.dew.ms nfs.dew.ms postgre.dew.ms
10.200.10.12 k8s0
10.200.10.13 k8s1 harbor.dew.ms notary.dew.ms dashboard.dew.ms es.dew.ms jaeger.dew.ms kibana.dew.ms prometheus.dew.ms grafana.dew.ms
10.200.10.14 k8s2
10.200.10.15 k8s3
10.200.10.16 k8s4
EOF
----

[[podAssignment]]
==== pod的调度
TIP: 此处简单说明pod的调度配置，方便后文中各组件的安装配置时的参考。
     更详细的配置细节请参考以下官方文档：
     https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
     https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

===== node亲和性和调度
.nodeName
nodeName是最简单的pod节点选择的方法，直接根据节点的名字来选择pod要调度的节点：
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
  nodeName: devops
----

.nodeSelector
使用此功能需要先给要被调度的node加上标签： +
`kubectl label nodes <node-name> <label-key>=<label-value>` +
eg: +
`kubectl label nodes devops tag=devops`

查看node的标签： +
`kubectl get nodes --show-labels`

给pod添加nodeSelector（推荐在controller部分加，eg:Deployment,StatefulSet等）
[source,yaml]
----
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    tag: devops
----
移除标签：
`kubectl label nodes devops tag-`


.Tolerations
给node添加污点： +
`kubectl taint nodes devops key=value:NoSchedule`
给pod添加tolerations：
[source,yaml]
----
tolerations:
- key: "key"
  operator: "Equal"
  value: "value"
  effect: "NoSchedule"
----
以上配置是指：当node满足pod的tolerations的条件 `key=value:NoSchedule` 时，pod才会被调度到该节点上。

移除node的污点： +
`kubectl taint nodes devops key:NoSchedule-`

.node的亲和性和反亲和性
eg:
[source,bash]
----
kubectl label nodes k8s1 devops=1
kubectl label nodes k8s2 devops=2
kubectl label nodes k8s3 devops=3
----

[source,yaml]
----
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: devops
            operator: In
            values:
            - 1
            - 2
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: devops
            operator: In
            values:
            - 3
----
以上例子说明，pod将会被调度到打有 `devops=1` 和 `devops=2` 标签的节点上去， `devops=3`的节点为备选节点。 +
operator的值可以为：`In, NotIn, Exists, DoesNotExist, Gt, Lt.` +
此外，可以通过给 `operator`设值 `NotIn` 或 `DoesNotExist`，来实现node的反亲和性功能。

===== pod亲和性与反亲和性调度
.Inter-pod affinity and anti-affinity
[source,yaml]
----
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: security
            operator: In
            values:
            - S1
        topologyKey: failure-domain.beta.kubernetes.io/zone
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: security
              operator: In
              values:
              - S2
          topologyKey: kubernetes.io/hostname
----

pod的亲和性调度，主要是根据pod及node的标签来进行调度的。 +
当一个节点上已有至少一个正在运行的，并带有满足该pod的podAffinity条件的标签的pod1、且该节点有`failure-domain.beta.kubernetes.io/zone`的标签，则该pod会被调度到pod1的节点上去。 +
反亲和性与之相反，当某一节点上至少有一个正在运行的、具有满足podAntiAffinity条件的pod2，则该pod不会被调度到pod2的节点上去。


==== docker

TIP: https://kubernetes.io/docs/setup/cri/#docker

[source,bash]
----
yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

yum update -y && yum install -y docker-ce-18.06.2.ce

mkdir /etc/docker

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# 添加代理
cat >>/etc/systemd/system/docker.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://<代理host>:<代理端口>" "HTTPS_PROXY=http://<代理host>:<代理端口>" "NO_PROXY=localhost,127.0.0.1,dew.ms"
EOF

systemctl daemon-reload
systemctl restart docker
systemctl enable docker.service
----

==== kubernetes

TIP: https://kubernetes.io/docs/setup/independent/install-kubeadm/

[source,bash]
.安装
----
# 使用阿里云镜像
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable --now kubelet
----

TIP: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

[source,bash]
.Master配置
----
# 安装Git，后续会用到
yum install -y git

# 初始化Kubernetes，二选一，使用代理方式
kubeadm init \
    --kubernetes-version v1.13.4 \
    --pod-network-cidr=10.244.0.0/16

# 初始化Kubernetes，二选一，不使用代理方式，通过image-repository 及 --kubernetes-version 避免被墙
kubeadm init \
    --image-repository registry.aliyuncs.com/google_containers \
    --kubernetes-version v1.13.4 \
    --pod-network-cidr=10.244.0.0/16

# 记录上述操作输出中的kubeadm join
# e.g. kubeadm join 10.200.131.18:6443 --token <PASSWORD> --discovery-token-ca-cert-hash sha256:cabc90823a8e0bcf6e3bf719abc569a47c186f6cfd0e156ed5a3cd5a8d85fab0

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 查看集群状态
kubectl get cs

# 安装flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

# 都为Running后表示完成
kubectl get pods --all-namespaces
----

[NOTE]
.Master做为Node
====
默认情况下 master 不会做为 node 节点，可通过此命令强制启用（不推荐）
``kubectl taint nodes --all node-role.kubernetes.io/master-``
====

TIP: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

[source,bash]
.Node配置
----
# 执行上一步输出的 kubeadm join ...

# 完成后在master上执行情况如下（以1.13.4版本为例）
kubectl get no
NAME        STATUS     ROLES    AGE   VERSION
test1.k8s   Ready   master   11m   v1.13.4
test2.k8s   Ready   <none>   70s   v1.13.4
test3.k8s   Ready   <none>   52s   v1.13.4
test4.k8s   Ready   <none>   43s   v1.13.4
test5.k8s   Ready   <none>   34s   v1.13.4
----

==== helm

TIP: https://docs.helm.sh/using_helm/#installing-helm

[source,bash]
----

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
EOF

helm init --service-account tiller

# 不使用代理方式，需要修改镜像
kubectl set image deployment/tiller-deploy tiller=registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.12.3 -n kube-system

kubectl get pod -n kube-system -l app=helm
----

==== dns

[source,bash]
----
# 在middleware节点上执行安装
yum install -y dnsmasq
systemctl enable dnsmasq
systemctl start dnsmasq

# 编辑各节点，加上middleware节点的IP
vi /etc/resolv.conf
-
nameserver 10.200.10.11
-

# 编辑Kubernetes的DNS，加上dew.ms的代理
kubectl -n kube-system edit cm coredns
-
data:
  Corefile: |
    ...
    dew.ms:53 {
        errors
        cache 30
        proxy . 10.200.10.11
    }
-
----

==== nfs

[source,bash]
----
yum install -y nfs-utils
mkdir -p /data/nfs
chmod 755 /data/nfs

mkdir -p /data/nfs/gitlab

vi /etc/exports

    /data/nfs/gitlab     *(rw,sync,no_root_squash,no_all_squash)

systemctl enable rpcbind
systemctl enable nfs-server
systemctl start rpcbind
systemctl start nfs-server

showmount -e localhost
----

==== postgreSql

[source,bash]
----
wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm

rpm -Uvh pgdg-redhat96-9.6-3.noarch.rpm
yum install -y postgresql96-server

/usr/pgsql-9.6/bin/postgresql96-setup initdb

vi /var/lib/pgsql/9.6/data/postgresql.conf
-
listen_addresses='*'
-

vi /var/lib/pgsql/9.6/data/pg_hba.conf
-
host  all  all 0.0.0.0/0 md5
-

systemctl enable postgresql-9.6.service
systemctl start postgresql-9.6.service

su - postgres
psql -U postgres
-
ALTER USER postgres WITH PASSWORD '<PASSWORD>';
-
----

==== redis

[source,bash]
----
yum install -y epel-release
yum -y install redis
vi /etc/redis.conf
-
# 注释
# bind 127.0.0.1
# 开启密码
requirepass <PASSWORD>
-
systemctl start redis
----

==== gitlab

TIP: https://docs.gitlab.com/omnibus/README.html#installation-and-configuration-using-omnibus-package

[source,bash]
----
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash
yum install -y gitlab-ce

# 按需修改，可修改说明见: https://docs.gitlab.com/omnibus/settings/
vi /etc/gitlab/gitlab.rb
-
external_url 'http://gitlab.dew.ms'
...
-

gitlab-ctl reconfigure

# 浏览器访问并修改root密码

# 安装 gitlab runner，Helm方式，在k8s0节点上执行
helm repo add gitlab https://charts.gitlab.io
helm fetch --untar gitlab/gitlab-runner
cd gitlab-runner

# 添加账号绑定关系
vi templates/role-binding.yaml
-
    - kind: ServiceAccount
      name: default
      namespace: "{{ .Release.Namespace }}"
-

# 添加PVC，使用DooD方式
# 注意添加的位置在 “# Start the runner” 前
# DooD方式由于直接使用宿主机的Docker，存在一定的安全风险，但DinD模式下Kubernetes无法很好地处理镜像缓存，导致每次都要全量拉取
# 详见 https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-in-your-builds
vi templates/configmap.yaml
-
    cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF
            [[runners.kubernetes.volumes.pvc]]
              name = "gitlab-runner-cache"
              mount_path = "{{ .Values.runners.cache.cachePath }}"
            [[runners.kubernetes.volumes.host_path]]
              name = "docker-socket"
              mount_path = "/var/run/docker.sock"
    EOF
    # Start the runner
-

# 创建PV
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv_gitlab
  labels:
    pv: pv_gitlab
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/gitlab
    server: nfs.dew.ms
EOF

# 创建PVC
cat <<EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gitlab-runner-cache
  namespace: devops
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  selector:
    matchLabels:
      pv: pv_gitlab
EOF

# runnerRegistrationToken 需要从gitlab页面上获取
# 特别说明的是这里定义了cachePath，使用PV
helm install --name dew-gitlab-runner --namespace devops \
    --set gitlabUrl=http://gitlab.dew.ms/ \
    --set runnerRegistrationToken=<...> \
    --set concurrent=20 \
    --set rbac.create=true \
    --set rbacWideAccess=true \
    --set runners.tags=general \
    --set runners.cache.cachePath=/opt/cache \
    --set runners.privileged=true \
    .
----

==== nginx Ingress Controller

[source,bash]
----
# 使用如下方式将80 443暴露出来
helm install stable/nginx-ingress --name dew-nginx --namespace ingress-nginx \
    --set controller.kind=DaemonSet \
    --set controller.hostNetwork=true \
    --set controller.stats.enabled=true \
    --set controller.metrics.enabled=true
----

==== harbor

TIP: https://github.com/goharbor/harbor-helm

[source,bash]
----
git clone https://github.com/goharbor/harbor-helm
cd harbor-helm
git checkout 1.0.0

# 创建Postgres数据库
-
CREATE DATABASE  registry;
CREATE DATABASE  clair;
CREATE DATABASE  notary_server;
CREATE DATABASE  notary_signer;
-

# 创建3个PV
for i in {0..2}; do
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv00${i}
  namespace: devops
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/d${i}
    server: nfs.dew.ms
EOF
done

# TBD 使用PVC

helm install --name dew-harbor --namespace devops \
    --set externalURL=https://harbor.dew.ms \
    --set harborAdminPassword=<PASSWORD> \
    --set expose.ingress.hosts.core=harbor.dew.ms \
    --set expose.ingress.hosts.notary=notary.dew.ms \
    --set database.type=external \
    --set database.external.host=postgre.dew.ms \
    --set database.external.port=5432 \
    --set database.external.username=postgres \
    --set database.external.password=<PASSWORD> \
    --set redis.type=external \
    --set redis.external.host=redis.dew.ms \
    --set redis.external.port=6379 \
    --set redis.external.password=<PASSWORD> \
    --set redis.external.coreDatabaseIndex=10 \
    --set redis.external.jobserviceDatabaseIndex=11 \
    --set redis.external.registryDatabaseIndex=12 \
    --set redis.external.chartmuseumDatabaseIndex=13\
    .

# 初始用户名/密码 admin/H<PASSWORD>

# 访问 https://harbor.dew.ms

# 获取证书
kubectl -n devops get secrets/dew-harbor-harbor-ingress -o jsonpath="{.data.ca\.crt}" | base64 --decode

# 以下操作在每台服务上执行

mkdir -p /etc/docker/certs.d/harbor.dew.ms
cat <<EOF > /etc/docker/certs.d/harbor.dew.ms/ca.crt
<上一步获取的证书>
EOF

systemctl daemon-reload
systemctl restart docker

# 登录，用户名/密码 admin/Dew!12345
docker login harbor.dew.ms -u admin -p Dew!12345

# 测试
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 harbor.dew.ms/test/pause:3.1
docker push harbor.dew.ms/test/pause:3.1
----

==== dashboard

[source,bash]
----
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kube-system
type: Opaque
EOF

# 安装，不使用代理方式需要加上 --set image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64
helm install stable/kubernetes-dashboard --name dew-dashboard --namespace kube-system \
    --set rbacAdminRole=true \
    --set ingress.enabled=true \
    --set-string ingress.annotations."nginx\.ingress\.kubernetes\.io/backend-protocol"="HTTPS" \
    --set ingress.hosts={dashboard.dew.ms} \
    --set ingress.tls[0].hosts={dashboard.dew.ms},ingress.tls[0].secretName=kubernetes-dashboard-certs

# 获取Token
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep dew-dashboard-kubernetes-dashboard | awk '{print $1}')

# 使用Firefox访问
----

==== elasticsearch

TIP: https://github.com/elastic/helm-charts/blob/master/elasticsearch 注意仔细查看各参数设值的说明。

[source,bash]
----

# 在NFS服务器上加上文件路径,并创建需要的目录

vi /etc/exports

/data/nfs/elasticsearch    *(rw,sync,no_root_squash,no_all_squash)
# 重启NFS
systemctl restart nfs-server
# 查看结果
showmount -e localhost

# 创建PV，PV可以加上和helm安装时一致的label,以便和PVC绑定，例： app: dew-elasticsearch-client
app=dew-elasticsearch-client
size=100Gi
# 请根据replicas的个数来决定下面PV的创建个数
for i in {0..1}; do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    app: ${app}
  name: ${app}-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/elasticsearch/${app}-${i}
    server: 10.200.131.182
EOF
done

# TIP：如果pod没有启动成功，报错和路径权限问题有关，可尝试给PV的存储路径添加权限,如：
chmod 777 /data/nfs/elasticsearch

# 使用helm安装
helm repo add elastic https://helm.elastic.co

helm install --name dew-elasticsearch elastic/elasticsearch --namespace devops \
    --set imageTag=6.6.1 \
    --set clusterName=dew-elasticsearch \
    --set nodeGroup=client \
    --set masterService=dew-elasticsearch-client \
    --set replicas=2 \
    --set minimumMasterNodes=2 \
    --set volumeClaimTemplate.storageClassName="" \
    --set volumeClaimTemplate.resources.requests.storage=200Gi \
    --set volumeClaimTemplate.selector.matchlabels."app"="dew-elasticsearch-client" \
    --set fsGroup=0 \
    --set clusterHealthCheckParams="" \
    --set ingress.enabled=true \
    --set ingress.hosts={elasticsearch.dew.ms}

    # pod调度相关配置,请根据需要进行设值
    --set nodeSelector."tag"="devops" \
    --set tolerations[0].key="key" \
    --set tolerations[0].operator="Equal" \
    --set tolerations[0].value="value" \
    --set tolerations[0].effect="NoSchedule"

    # 若使用xpack security,请加上以下参数
    --set-string extraEnvs[0]."name"="xpack\.security\.enabled" \
    --set-string extraEnvs[0]."value"="true" \
    --set-string extraEnvs[1]."name"="xpack\.security\.authc\.accept_default_password" \
    --set-string extraEnvs[1]."value"="true" \
    --set-string extraEnvs[2]."name"="ELASTIC_USERNAME" \
    --set-string extraEnvs[2]."value"="elastic" \
    --set-string extraEnvs[3]."name"="ELASTIC_PASSWORD" \
    --set-string extraEnvs[3]."value"="123456"

    xpack安装相关文档说明：https://github.com/elastic/helm-charts/blob/master/elasticsearch/README.md#security

    * 开启xpack security的简单例子：
      . 进入容器内部
        kubectl exec -it dew-elasticsearch-client-0 -n devops /bin/sh
      . 激活30天试用license
        curl -H "Content-Type:application/json" -XPOST  http://localhost:9200/_xpack/license/start_trial?acknowledge=true
      . 修改密码：
        bin/elasticsearch-setup-passwords interactive
      . 测试：
        curl -u elastic -XGET 'localhost:9200/_cat/health?v&pretty'

----

TIP: 其他elasticsearch的helm chart : https://github.com/helm/charts/tree/master/stable/elasticsearch

==== fluentd

TIP: https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch +
     https://kiwigrid.github.io/

[source,bash]
----
helm repo add kiwigrid https://kiwigrid.github.io

# 不使用代理要加上  --set image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch
helm install kiwigrid/fluentd-elasticsearch --name dew-fluentd-es --namespace devops \
    --set elasticsearch.host=dew-elasticsearch-client \
    --set elasticsearch.logstash_prefix=logstash \
    # 若 ES 启用 xpack 的 security，加上以下参数
    --set elasticsearch.user=elastic \
    --set elasticsearch.password=<PASSWORD>
    # Prometheus 相关设置(需配合prometheus-operator使用)
    --set service.type=ClusterIP \
    --set service.ports[0].name="monitor-agent" \
    --set service.ports[0].port=24231 \
    --set prometheusRule.enabled=true \
    --set prometheusRule.prometheusNamespace=devops \
    --set prometheusRule.labels.app=prometheus-operator \
    --set prometheusRule.labels.release=dew-prometheus-operator \
    --set serviceMonitor.enabled=true \
    --set serviceMonitor.labels.release=dew-prometheus-operator
    # 不使用代理要加上
    --set image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch

    # pod调度相关配置,请根据实际需要进行设值
    --set nodeSelector."tag"="devops" \
    --set tolerations[0].key="key" \
    --set tolerations[0].operator="Equal" \
    --set tolerations[0].value="value" \
    --set tolerations[0].effect="NoSchedule"
----


==== kibana

TIP: https://github.com/helm/charts/tree/master/stable/kibana

[source,bash]
----

# 在NFS服务器上加上文件路径,并创建需要的目录

vi /etc/exports

/data/nfs/kibana    *(rw,sync,no_root_squash,no_all_squash)
# 重启NFS
systemctl restart nfs-server
# 查看结果
showmount -e localhost

使用PVC
apps=("kibana")
size=10Gi

for i in ${apps[@]};do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: ${size}
  selector:
    matchLabels:
      app: ${i}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/${apps}
    server: 10.200.131.182
EOF
done

helm install --name dew-kibana  stable/kibana  --namespace devops \
    --set image.tag="6.6.1" \
    --set env."ELASTICSEARCH_URL"="http://dew-elasticsearch-client:9200" \
    --set service.internalPort=5601 \
    --set ingress.enabled=true,ingress.hosts={kibana.dew.ms} \
    --set-string ingress.annotations."kubernetes\.io/ingress\.class"=nginx \
    --set-string ingress.annotations."kubernetes\.io/tls-acme"="true" \
    --set ingress.tls[0].hosts={kibana.dew.ms},ingress.tls[0].secretName=kibana-certs \
    --set dashboardImport.enabled=true \
    --set dashboardImport.dashboards."k8s"="https://raw.githubusercontent.com/monotek/kibana-dashboards/master/k8s-fluentd-elasticsearch.json" \
    --set serviceAccount.create=true,serviceAccountName=kibana \
    --set plugins.enabled=true \
    --set persistentVolumeClaim.enabled=true \
    --set persistentVolumeClaim.existingClaim=true \
    --set securityContext.enabled=true \
    --set securityContext.allowPrivilegeEscalation=true \
    --set securityContext.runAsUser=0 \
    --set securityContext.fsGroup=0

    # pod调度相关配置，请根据实际情况设值
    --set nodeSelector."tag"="devops" \
    --set tolerations[0].key="key" \
    --set tolerations[0].operator="Equal" \
    --set tolerations[0].value="value" \
    --set tolerations[0].effect="NoSchedule" \
    --set affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key="key" \
    --set affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator=In \
    --set affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]=target-host-name \

    # xpack security相关参数：
    --set image.repository=docker.elastic.co/kibana/kibana \
    --set env."XPACK_SECURITY_ENABLED"="true" \
    --set env."ELASTICSEARCH_USERNAME"="kibana" \
    --set env."ELASTICSEARCH_PASSWORD"="<PASSWORD>" \
    --set dashboardImport.xpackauth.enabled=true \
    --set dashboardImport.xpackauth.username=kibana\
    --set dashboardImport.xpackauth.password=<PASSWORD>

----

==== jaeger

TIP: https://github.com/jaegertracing/jaeger-operator

[source,bash]
----
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing_v1_jaeger_crd.yaml
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -

# 使用elasticsearch作为jaeger的数据源
    # 若ES启用Xpack Security，则需要创建secret
    ELASTICSEARCH_USERNAME=elastic
    ELASTICSEARCH_PASSWORD=<PASSWORD>
    cat <<EOF | kubectl -n devops apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: jaeger-es-secrets
    type: Opaque
    data:
      ES_USERNAME: `echo -n $ELASTICSEARCH_USERNAME | base64`
      ES_PASSWORD: `echo -n $ELASTICSEARCH_PASSWORD | base64`
    EOF

    # 创建Jaeger实例
    cat <<EOF | kubectl apply -n devops -f -
    apiVersion: jaegertracing.io/v1
    kind: Jaeger
    metadata:
      name: jaeger
    spec:
      strategy: production
      storage:
        type: elasticsearch
        options:
          es:
            server-urls: http://dew-elasticsearch-client:9200
        secretName: jaeger-es-secrets  # 若ES启用Xpack Security，需要设置此项及创建secret
    EOF

    TIP: Jaeger实例可在不同namespace下创建使用，使用中请注意namespace的问题。
    使用sidecar的方式部署项目：https://github.com/jaegertracing/jaeger-operator#auto-injection-of-jaeger-agent-sidecars
    使用daemonset的方式部署项目：https://github.com/jaegertracing/jaeger-operator#agent-as-daemonset

# 修改Ingress
cat <<EOF | kubectl -n devops apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: jaeger-query
spec:
  rules:
    - host: jaeger.dew.ms
      http:
        paths:
          - backend:
              serviceName: jaeger-query
              servicePort: 16686
            path: /
EOF

----

.pod的调度
目前jaeger-operator暂不支持直接设置，请关注该项目的更新情况。
可以自行给需要调度的pod的deployment添加限制条件。可参考： <<podAssignment>>


==== prometheus-operator 和 grafana
TIP: https://github.com/helm/charts/tree/master/stable/prometheus-operator

  prometheus-operator结构：
    |--- prometheus-operator
    |--- prometheus
    |--- alertmanager
    |--- node-exporter
    |--- kube-state-metrics
    |--- service monitors to scrape internal kubernetes components
    |     |---kube-apiserver
    |     |---kube-scheduler
    |     |---kube-controller-manager
    |     |---etcd
    |     |---kube-dns/coredns
    |
    |--- grafana

===== prometheus-operator 使用PV

[source,bash]
----
# 在NFS服务器上加上文件路径,并创建需要的目录

vi /etc/exports

/data/nfs/prometheus-operator    *(rw,sync,no_root_squash,no_all_squash)

# 重启NFS
systemctl restart nfs-server
# 查看结果
showmount -e localhost


# 创建PV,注意label的对应

app=prometheus-operator
components=("alertmanager" "prometheus")
size=100Gi

for i in ${components[@]};do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    component: ${i}
  name: dew-${app}-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/${app}/${i}
    server: 10.200.131.182
EOF
done

----

===== 创建grafana的PVC和PV
[source,bash]
----
apps=("grafana")
size=50Gi

for i in ${apps[@]};do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: ${size}
  selector:
    matchLabels:
      app: ${i}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/prometheus-operator/${i}
    server: 10.200.131.182
EOF
done

----


===== 使用helm 安装

注意安装前先更新chart仓库
`helm repo update`

[source,yaml]
----
# 若需要对etcd进行监控，则需要先创建secret
kubectl -n devops create secret generic dew-prometheus-operator-etcd  --from-file=/etc/kubernetes/pki/etcd/ca.crt  --from-file=/etc/kubernetes/pki/etcd/peer.crt  --from-file=/etc/kubernetes/pki/etcd/peer.key

helm install stable/prometheus-operator --name dew-prometheus-operator --namespace devops \
    --set kubelet.serviceMonitor.https=true \
    --set prometheus.ingress.enabled=true \
    --set prometheus.ingress.hosts={prometheus.dew.ms} \
    --set alertmanager.ingress.enabled=true \
    --set alertmanager.ingress.hosts={prometheus.alertmanager.ms} \
    --set prometheusOperator.securityContext.runAsNonRoot=false \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi \
    --set alertmanager.alertmanagerSpec.storage.volumeClaimTemplate.spec.resources.requests.storage=100Gi \
    --set alertmanager.alertmanagerSpec.storage.volumeClaimTemplate.spec.selector.matchLabels."component"="alertmanager" \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.selector.matchLabels."component"="prometheus"
    # 对etcd监测相关参数
    --set prometheus.prometheusSpec.secrets[0]=dew-prometheus-operator-etcd
    --set kubeEtcd.serviceMonitor.scheme=https \
    --set kubeEtcd.serviceMonitor.insecureSkipVerify=true \
    --set kubeEtcd.serviceMonitor.caFile="/etc/prometheus/secrets/dew-prometheus-operator-etcd/ca.crt" \
    --set kubeEtcd.serviceMonitor.certFile="/etc/prometheus/secrets/dew-prometheus-operator-etcd/peer.crt" \
    --set kubeEtcd.serviceMonitor.keyFile="/etc/prometheus/secrets/dew-prometheus-operator-etcd/peer.key"
    # 直接使用prometheus-operator的grafana，添加以下设置
    --set grafana.enabled=true \
    --set grafana.adminPassword=<PASSWORD> \
    --set grafana.defaultDashboardsEnabled=true \
    --set grafana.ingress.enabled=true \
    --set grafana.ingress.hosts={grafana.dew.ms} \
    --set grafana.ingress.tls[0].host={grafana.dew.ms},ingress.tls[0].secretName=dew-grafana \
    --set grafana.sidecar.dashboards.enabled=true \
    --set grafana.sidecar.dashboards.searchNamespace="devops"\
    --set grafana.sidecar.dashboards.label=grafana_dashboard \
    --set grafana.sidecar.datasources.enabled=true \
    --set grafana.sidecar.datasources.searchNamespace="devops" \
    --set grafana.sidecar.datasources.label=grafana_datasource \
    --set grafana.'grafana\.ini'.smtp.enabled="true" \
    --set grafana.'grafana\.ini'.smtp.host="smtp.163.com:25" \
    --set grafana.'grafana\.ini'.smtp.user=<EMAIL> \
    --set grafana.'grafana\.ini'.smtp.password=<PASSWORD> \
    --set grafana.'grafana\.ini'.smtp.from_address="<EMAIL>" \
    --set grafana.'grafana\.ini'.smtp.skip_verify=true \
    --set grafana.persistence.enabled=true \
    --set grafana.persistence.existingClaim=dew-grafana
    # 可选设置
    --set grafana.'grafana\.ini'.server.root_url="https://grafana.dew.ms"

    如不使用代理,更换以下镜像仓库
    --set kube-state-metrics.image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-state-metrics

 TIP: grafana默认用户名：admin,
 查看密码：
 kubectl get secret --namespace devops dew-prometheus-operator-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

 grafana重置密码：进入grafana的容器内部后
 grafana-cli admin reset-admin-password <PASSWORD>

INFO: 若有pod启动失败,报文件权限拒绝相关问题，很可能和PV的文件目录的权限有关，检查下权限是否一致，设置对应的securityContext进行排查。
例：
kubectl edit statefulset prometheus-dew-prometheus-operator-prometheus -n devops
设置securityContext为以下内容
      securityContext:
        fsGroup: 0
        runAsNonRoot: false
        runAsUser: 0

INFO: 若验证prometheus的target，kube-scheduler、kube-controller处于down状态，是因为它们只能在宿主机上通过127.0.0.1访问，可使用以下操作：
    . 如果使用kubeadm启动的集群，初始化时的config.yml里可以加入如下参数
        controllerManagerExtraArgs:
          address: 0.0.0.0
        schedulerExtraArgs:
          address: 0.0.0.0
    . 已经启动后的使用下面命令更改就会滚动更新
        sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml
        sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml
      或者全部替换：
        sed -ri '/--address/s#=.+#=0.0.0.0#' /etc/kubernetes/manifests/kube-*
    参考文章： http://www.servicemesher.com/blog/prometheus-operator-manual/
              https://github.com/coreos/prometheus-operator/blob/master/Documentation/troubleshooting.md

# 监控APP
  1.首先需要将项目instrument
    参考文章：https://prometheus.io/docs/instrumenting/clientlibs/
  2.部署项目及创建进行监控的ServiceMonitor。 +
    注意ServiceMonitor的labels要含有Prometheus-operator创建的Prometheus的serviceMonitorSelector的label。 +
    详细文章：https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#related-resources
----

===== pod的调度
pod调度相关配置，以下配置仅供参考
[source,yaml]
----<commit_msg>Format docs<commit_after>=== 测试、生产环境安装配置

[IMPORTANT]
====
本文会给出使用代理与不使用代理的安装、配置方式，但强烈推荐使用代理方式，详见 <<proxies>> 。
====

*以 Centos7 为例，做好ssh免密互访、关闭防火墙、关闭swap、禁用SELINUX*

[source,bash]
----
systemctl stop firewalld.service
systemctl disable firewalld.service
swapoff -a
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sed -i s/^SELINUX=.*$/SELINUX=disabled/ /etc/selinux/config
----

.服务列表
|===
|主机名 |IP |服务 | 备注

|devops | 10.200.10.10 | Gitlab | Gitlab及其CI/CD独立一台服务器部署，实际环境应该做HA
|middleware | 10.200.10.11 | Docker、RabbitMQ、PostgreSql、Redis、NFS、dnsmasq | 各类中间件，实际环境应该做HA
|k8s0 | 10.200.10.12 | Docker、kubernetes master、Helm |
|k8s1 | 10.200.10.13 | Docker、kubernetes node |
|k8s2 | 10.200.10.14 | Docker、kubernetes node |
|k8s3 | 10.200.10.15 | Docker、kubernetes node |
|k8s4 | 10.200.10.16 | Docker、kubernetes node |
|…… | …… | Docker、kubernetes node |
|===

[source,bash]
.各节点Host
----
# 除middleware外的各节点

cat >>/etc/hosts <<EOF
10.200.10.10 devops
10.200.10.11 middleware
10.200.10.12 k8s0
10.200.10.13 k8s1
10.200.10.14 k8s2
10.200.10.15 k8s3
10.200.10.16 k8s4
EOF

# middleware节点和客户端节点

cat >>/etc/hosts <<EOF
10.200.10.10 devops gitlab.dew.ms
10.200.10.11 middleware rabbitmq.dew.ms redis.dew.ms nfs.dew.ms postgre.dew.ms
10.200.10.12 k8s0
10.200.10.13 k8s1 harbor.dew.ms notary.dew.ms dashboard.dew.ms es.dew.ms jaeger.dew.ms kibana.dew.ms prometheus.dew.ms grafana.dew.ms
10.200.10.14 k8s2
10.200.10.15 k8s3
10.200.10.16 k8s4
EOF
----

==== docker

TIP: https://kubernetes.io/docs/setup/cri/#docker

[source,bash]
----
yum install -y yum-utils \
  device-mapper-persistent-data \
  lvm2

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

yum update -y && yum install -y docker-ce-18.06.2.ce

mkdir /etc/docker

cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

mkdir -p /etc/systemd/system/docker.service.d

# 添加代理
cat >>/etc/systemd/system/docker.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://<代理host>:<代理端口>" "HTTPS_PROXY=http://<代理host>:<代理端口>" "NO_PROXY=localhost,127.0.0.1,dew.ms"
EOF

systemctl daemon-reload
systemctl restart docker
systemctl enable docker.service
----

==== kubernetes

TIP: https://kubernetes.io/docs/setup/independent/install-kubeadm/

[source,bash]
.安装
----
# 使用阿里云镜像
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
        http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF

sysctl --system

yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
systemctl enable --now kubelet
----

TIP: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

[source,bash]
.Master配置
----
# 安装Git，后续会用到
yum install -y git

# 初始化Kubernetes，二选一，使用代理方式
kubeadm init \
    --kubernetes-version v1.13.4 \
    --pod-network-cidr=10.244.0.0/16

# 初始化Kubernetes，二选一，不使用代理方式，通过image-repository 及 --kubernetes-version 避免被墙
kubeadm init \
    --image-repository registry.aliyuncs.com/google_containers \
    --kubernetes-version v1.13.4 \
    --pod-network-cidr=10.244.0.0/16

# 记录上述操作输出中的kubeadm join
# e.g. kubeadm join 10.200.131.18:6443 --token <PASSWORD> --discovery-token-ca-cert-hash sha256:cabc90823a8e0bcf6e3bf719abc569a47c186f6cfd0e156ed5a3cd5a8d85fab0

mkdir -p $HOME/.kube
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

# 查看集群状态
kubectl get cs

# 安装flannel
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml

# 都为Running后表示完成
kubectl get pods --all-namespaces
----

[NOTE]
.Master做为Node
====
默认情况下 master 不会做为 node 节点，可通过此命令强制启用（不推荐）
``kubectl taint nodes --all node-role.kubernetes.io/master-``
====

TIP: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/

[source,bash]
.Node配置
----
# 执行上一步输出的 kubeadm join ...

# 完成后在master上执行情况如下（以1.13.4版本为例）
kubectl get no
NAME        STATUS     ROLES    AGE   VERSION
test1.k8s   Ready   master   11m   v1.13.4
test2.k8s   Ready   <none>   70s   v1.13.4
test3.k8s   Ready   <none>   52s   v1.13.4
test4.k8s   Ready   <none>   43s   v1.13.4
test5.k8s   Ready   <none>   34s   v1.13.4
----

==== helm

TIP: https://docs.helm.sh/using_helm/#installing-helm

[source,bash]
----

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
EOF

helm init --service-account tiller

# 不使用代理方式，需要修改镜像
kubectl set image deployment/tiller-deploy tiller=registry.cn-hangzhou.aliyuncs.com/google_containers/tiller:v2.12.3 -n kube-system

kubectl get pod -n kube-system -l app=helm
----

==== dns

[source,bash]
----
# 在middleware节点上执行安装
yum install -y dnsmasq
systemctl enable dnsmasq
systemctl start dnsmasq

# 编辑各节点，加上middleware节点的IP
vi /etc/resolv.conf
-
nameserver 10.200.10.11
-

# 编辑Kubernetes的DNS，加上dew.ms的代理
kubectl -n kube-system edit cm coredns
-
data:
  Corefile: |
    ...
    dew.ms:53 {
        errors
        cache 30
        proxy . 10.200.10.11
    }
-
----

==== nfs

[source,bash]
----
yum install -y nfs-utils
mkdir -p /data/nfs
chmod 755 /data/nfs

mkdir -p /data/nfs/gitlab

vi /etc/exports

    /data/nfs/gitlab     *(rw,sync,no_root_squash,no_all_squash)

systemctl enable rpcbind
systemctl enable nfs-server
systemctl start rpcbind
systemctl start nfs-server

showmount -e localhost
----

==== postgreSql

[source,bash]
----
wget https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm

rpm -Uvh pgdg-redhat96-9.6-3.noarch.rpm
yum install -y postgresql96-server

/usr/pgsql-9.6/bin/postgresql96-setup initdb

vi /var/lib/pgsql/9.6/data/postgresql.conf
-
listen_addresses='*'
-

vi /var/lib/pgsql/9.6/data/pg_hba.conf
-
host  all  all 0.0.0.0/0 md5
-

systemctl enable postgresql-9.6.service
systemctl start postgresql-9.6.service

su - postgres
psql -U postgres
-
ALTER USER postgres WITH PASSWORD '<PASSWORD>';
-
----

==== redis

[source,bash]
----
yum install -y epel-release
yum -y install redis
vi /etc/redis.conf
-
# 注释
# bind 127.0.0.1
# 开启密码
requirepass <PASSWORD>
-
systemctl start redis
----


[[podAssignment]]
==== pod的调度
TIP: 此处简单说明pod的调度配置，方便后文中各组件的安装配置时的参考。
     更详细的配置细节请参考以下官方文档：
     https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
     https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

===== node亲和性和调度
.nodeName
nodeName是最简单的pod节点选择的方法，直接根据节点的名字来选择pod要调度的节点：
[source,yaml]
----
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
  nodeName: devops
----

.nodeSelector
使用此功能需要先给要被调度的node加上标签： +
`kubectl label nodes <node-name> <label-key>=<label-value>` +
eg: +
`kubectl label nodes devops tag=devops`

查看node的标签： +
`kubectl get nodes --show-labels`

给pod添加nodeSelector（推荐在controller部分加，eg:Deployment,StatefulSet等）
[source,yaml]
----
spec:
  containers:
  - name: nginx
    image: nginx
    imagePullPolicy: IfNotPresent
  nodeSelector:
    tag: devops
----
移除标签：
`kubectl label nodes devops tag-`


.Tolerations
给node添加污点： +
`kubectl taint nodes devops key=value:NoSchedule`
给pod添加tolerations： +
[source,yaml]
----
tolerations:
- key: "key"
  operator: "Equal"
  value: "value"
  effect: "NoSchedule"
----
以上配置是指：当node满足pod的tolerations的条件 `key=value:NoSchedule` 时，pod才会被调度到该节点上。

移除node的污点： +
`kubectl taint nodes devops key:NoSchedule-`

.node的亲和性和反亲和性
eg:
[source,bash]
----
kubectl label nodes k8s1 devops=1
kubectl label nodes k8s2 devops=2
kubectl label nodes k8s3 devops=3
----

[source,yaml]
----
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: devops
            operator: In
            values:
            - 1
            - 2
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 1
        preference:
          matchExpressions:
          - key: devops
            operator: In
            values:
            - 3
----
以上例子说明，pod将会被调度到打有 `devops=1` 和 `devops=2` 标签的节点上去， `devops=3` 的节点为备选节点。 +
operator的值可以为：`In, NotIn, Exists, DoesNotExist, Gt, Lt.` +
此外，可以通过给 `operator` 设值 `NotIn` 或 `DoesNotExist`，来实现node的反亲和性功能。

===== pod亲和性与反亲和性调度
.Inter-pod affinity and anti-affinity
[source,yaml]
----
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: security
            operator: In
            values:
            - S1
        topologyKey: failure-domain.beta.kubernetes.io/zone
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: security
              operator: In
              values:
              - S2
          topologyKey: kubernetes.io/hostname
----

pod的亲和性调度，主要是根据pod及node的标签来进行调度的。 +
当一个节点上已有至少一个正在运行的，并带有满足该pod的podAffinity条件的标签的pod1、且该节点有 `failure-domain.beta.kubernetes.io/zone` 的标签，则该pod会被调度到pod1的节点上去。 +
反亲和性与之相反，当某一节点上至少有一个正在运行的、具有满足podAntiAffinity条件的pod2，则该pod不会被调度到pod2的节点上去。

==== gitlab

TIP: https://docs.gitlab.com/omnibus/README.html#installation-and-configuration-using-omnibus-package

[source,bash]
----
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.rpm.sh | sudo bash
yum install -y gitlab-ce

# 按需修改，可修改说明见: https://docs.gitlab.com/omnibus/settings/
vi /etc/gitlab/gitlab.rb
-
external_url 'http://gitlab.dew.ms'
...
-

gitlab-ctl reconfigure

# 浏览器访问并修改root密码

# 安装 gitlab runner，Helm方式，在k8s0节点上执行
helm repo add gitlab https://charts.gitlab.io
helm fetch --untar gitlab/gitlab-runner
cd gitlab-runner

# 添加账号绑定关系
vi templates/role-binding.yaml
-
    - kind: ServiceAccount
      name: default
      namespace: "{{ .Release.Namespace }}"
-

# 添加PVC，使用DooD方式
# 注意添加的位置在 “# Start the runner” 前
# DooD方式由于直接使用宿主机的Docker，存在一定的安全风险，但DinD模式下Kubernetes无法很好地处理镜像缓存，导致每次都要全量拉取
# 详见 https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-in-your-builds
vi templates/configmap.yaml
-
    cat >>/home/gitlab-runner/.gitlab-runner/config.toml <<EOF
            [[runners.kubernetes.volumes.pvc]]
              name = "gitlab-runner-cache"
              mount_path = "{{ .Values.runners.cache.cachePath }}"
            [[runners.kubernetes.volumes.host_path]]
              name = "docker-socket"
              mount_path = "/var/run/docker.sock"
    EOF
    # Start the runner
-

# 创建PV
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv_gitlab
  labels:
    pv: pv_gitlab
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/gitlab
    server: nfs.dew.ms
EOF

# 创建PVC
cat <<EOF | kubectl apply -f -
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gitlab-runner-cache
  namespace: devops
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  selector:
    matchLabels:
      pv: pv_gitlab
EOF

# runnerRegistrationToken 需要从gitlab页面上获取
# 特别说明的是这里定义了cachePath，使用PV
helm install --name dew-gitlab-runner --namespace devops \
    --set gitlabUrl=http://gitlab.dew.ms/ \
    --set runnerRegistrationToken=<...> \
    --set concurrent=20 \
    --set rbac.create=true \
    --set rbacWideAccess=true \
    --set runners.tags=general \
    --set runners.cache.cachePath=/opt/cache \
    --set runners.privileged=true \
    .
----

==== nginx Ingress Controller

[source,bash]
----
# 使用如下方式将80 443暴露出来
helm install stable/nginx-ingress --name dew-nginx --namespace ingress-nginx \
    --set controller.kind=DaemonSet \
    --set controller.hostNetwork=true \
    --set controller.stats.enabled=true \
    --set controller.metrics.enabled=true
----

==== harbor

TIP: https://github.com/goharbor/harbor-helm

[source,bash]
----
git clone https://github.com/goharbor/harbor-helm
cd harbor-helm
git checkout 1.0.0

# 创建Postgres数据库
-
CREATE DATABASE  registry;
CREATE DATABASE  clair;
CREATE DATABASE  notary_server;
CREATE DATABASE  notary_signer;
-

# 创建3个PV
for i in {0..2}; do
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv00${i}
  namespace: devops
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/d${i}
    server: nfs.dew.ms
EOF
done

# TBD 使用PVC

helm install --name dew-harbor --namespace devops \
    --set externalURL=https://harbor.dew.ms \
    --set harborAdminPassword=<PASSWORD> \
    --set expose.ingress.hosts.core=harbor.dew.ms \
    --set expose.ingress.hosts.notary=notary.dew.ms \
    --set database.type=external \
    --set database.external.host=postgre.dew.ms \
    --set database.external.port=5432 \
    --set database.external.username=postgres \
    --set database.external.password=<PASSWORD> \
    --set redis.type=external \
    --set redis.external.host=redis.dew.ms \
    --set redis.external.port=6379 \
    --set redis.external.password=<PASSWORD> \
    --set redis.external.coreDatabaseIndex=10 \
    --set redis.external.jobserviceDatabaseIndex=11 \
    --set redis.external.registryDatabaseIndex=12 \
    --set redis.external.chartmuseumDatabaseIndex=13\
    .

# 初始用户名/密码 admin/Harbor<PASSWORD>

# 访问 https://harbor.dew.ms

# 获取证书
kubectl -n devops get secrets/dew-harbor-harbor-ingress -o jsonpath="{.data.ca\.crt}" | base64 --decode

# 以下操作在每台服务上执行

mkdir -p /etc/docker/certs.d/harbor.dew.ms
cat <<EOF > /etc/docker/certs.d/harbor.dew.ms/ca.crt
<上一步获取的证书>
EOF

systemctl daemon-reload
systemctl restart docker

# 登录，用户名/密码 admin/Dew!<PASSWORD>
docker login harbor.dew.ms -u admin -p Dew!12345

# 测试
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 harbor.dew.ms/test/pause:3.1
docker push harbor.dew.ms/test/pause:3.1
----

==== dashboard

[source,bash]
----
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kube-system
type: Opaque
EOF

# 安装，不使用代理方式需要加上 --set image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64
helm install stable/kubernetes-dashboard --name dew-dashboard --namespace kube-system \
    --set rbacAdminRole=true \
    --set ingress.enabled=true \
    --set-string ingress.annotations."nginx\.ingress\.kubernetes\.io/backend-protocol"="HTTPS" \
    --set ingress.hosts={dashboard.dew.ms} \
    --set ingress.tls[0].hosts={dashboard.dew.ms},ingress.tls[0].secretName=kubernetes-dashboard-certs

# 获取Token
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep dew-dashboard-kubernetes-dashboard | awk '{print $1}')

# 使用Firefox访问
----

==== elasticsearch

TIP: https://github.com/elastic/helm-charts/blob/master/elasticsearch 注意仔细查看各参数设值的说明。

[source,bash]
----

# 在NFS服务器上加上文件路径,并创建需要的目录

vi /etc/exports

/data/nfs/elasticsearch    *(rw,sync,no_root_squash,no_all_squash)
# 重启NFS
systemctl restart nfs-server
# 查看结果
showmount -e localhost

# 创建PV，PV可以加上和helm安装时一致的label,以便和PVC绑定，例： app: dew-elasticsearch-client
app=dew-elasticsearch-client
size=100Gi
# 请根据replicas的个数来决定下面PV的创建个数
for i in {0..1}; do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    app: ${app}
  name: ${app}-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/elasticsearch/${app}-${i}
    server: 10.200.131.182
EOF
done

# TIP：如果pod没有启动成功，报错和路径权限问题有关，可尝试给PV的存储路径添加权限,如：
chmod 777 /data/nfs/elasticsearch

# 使用helm安装
helm repo add elastic https://helm.elastic.co

helm install --name dew-elasticsearch elastic/elasticsearch --namespace devops \
    --set imageTag=6.6.1 \
    --set clusterName=dew-elasticsearch \
    --set nodeGroup=client \
    --set masterService=dew-elasticsearch-client \
    --set replicas=2 \
    --set minimumMasterNodes=2 \
    --set volumeClaimTemplate.storageClassName="" \
    --set volumeClaimTemplate.resources.requests.storage=200Gi \
    --set volumeClaimTemplate.selector.matchlabels."app"="dew-elasticsearch-client" \
    --set fsGroup=0 \
    --set clusterHealthCheckParams="" \
    --set ingress.enabled=true \
    --set ingress.hosts={elasticsearch.dew.ms}

    # pod调度相关配置,请根据需要进行设值
    --set nodeSelector."tag"="devops" \
    --set tolerations[0].key="key" \
    --set tolerations[0].operator="Equal" \
    --set tolerations[0].value="value" \
    --set tolerations[0].effect="NoSchedule"

    # 若使用xpack security,请加上以下参数
    --set-string extraEnvs[0]."name"="xpack\.security\.enabled" \
    --set-string extraEnvs[0]."value"="true" \
    --set-string extraEnvs[1]."name"="xpack\.security\.authc\.accept_default_password" \
    --set-string extraEnvs[1]."value"="true" \
    --set-string extraEnvs[2]."name"="ELASTIC_USERNAME" \
    --set-string extraEnvs[2]."value"="elastic" \
    --set-string extraEnvs[3]."name"="ELASTIC_PASSWORD" \
    --set-string extraEnvs[3]."value"="123456"

    xpack安装相关文档说明：https://github.com/elastic/helm-charts/blob/master/elasticsearch/README.md#security

    * 开启xpack security的简单例子：
      . 进入容器内部
        kubectl exec -it dew-elasticsearch-client-0 -n devops /bin/sh
      . 激活30天试用license
        curl -H "Content-Type:application/json" -XPOST  http://localhost:9200/_xpack/license/start_trial?acknowledge=true
      . 修改密码：
        bin/elasticsearch-setup-passwords interactive
      . 测试：
        curl -u elastic -XGET 'localhost:9200/_cat/health?v&pretty'

----

TIP: 其他elasticsearch的helm chart : https://github.com/helm/charts/tree/master/stable/elasticsearch

==== fluentd

TIP: https://github.com/kiwigrid/helm-charts/tree/master/charts/fluentd-elasticsearch +
     https://kiwigrid.github.io/

[source,bash]
----
helm repo add kiwigrid https://kiwigrid.github.io

# 不使用代理要加上  --set image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch
helm install kiwigrid/fluentd-elasticsearch --name dew-fluentd-es --namespace devops \
    --set elasticsearch.host=dew-elasticsearch-client \
    --set elasticsearch.logstash_prefix=logstash \
    # 若 ES 启用 xpack 的 security，加上以下参数
    --set elasticsearch.user=elastic \
    --set elasticsearch.password=<PASSWORD>
    # Prometheus 相关设置(需配合prometheus-operator使用)
    --set service.type=ClusterIP \
    --set service.ports[0].name="monitor-agent" \
    --set service.ports[0].port=24231 \
    --set prometheusRule.enabled=true \
    --set prometheusRule.prometheusNamespace=devops \
    --set prometheusRule.labels.app=prometheus-operator \
    --set prometheusRule.labels.release=dew-prometheus-operator \
    --set serviceMonitor.enabled=true \
    --set serviceMonitor.labels.release=dew-prometheus-operator
    # 不使用代理要加上
    --set image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/fluentd-elasticsearch

    # pod调度相关配置,请根据实际需要进行设值
    --set nodeSelector."tag"="devops" \
    --set tolerations[0].key="key" \
    --set tolerations[0].operator="Equal" \
    --set tolerations[0].value="value" \
    --set tolerations[0].effect="NoSchedule"
----


==== kibana

TIP: https://github.com/helm/charts/tree/master/stable/kibana

[source,bash]
----

# 在NFS服务器上加上文件路径,并创建需要的目录

vi /etc/exports

/data/nfs/kibana    *(rw,sync,no_root_squash,no_all_squash)
# 重启NFS
systemctl restart nfs-server
# 查看结果
showmount -e localhost

使用PVC
apps=("kibana")
size=10Gi

for i in ${apps[@]};do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: ${size}
  selector:
    matchLabels:
      app: ${i}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/${apps}
    server: 10.200.131.182
EOF
done

helm install --name dew-kibana  stable/kibana  --namespace devops \
    --set image.tag="6.6.1" \
    --set env."ELASTICSEARCH_URL"="http://dew-elasticsearch-client:9200" \
    --set service.internalPort=5601 \
    --set ingress.enabled=true,ingress.hosts={kibana.dew.ms} \
    --set-string ingress.annotations."kubernetes\.io/ingress\.class"=nginx \
    --set-string ingress.annotations."kubernetes\.io/tls-acme"="true" \
    --set ingress.tls[0].hosts={kibana.dew.ms},ingress.tls[0].secretName=kibana-certs \
    --set dashboardImport.enabled=true \
    --set dashboardImport.dashboards."k8s"="https://raw.githubusercontent.com/monotek/kibana-dashboards/master/k8s-fluentd-elasticsearch.json" \
    --set serviceAccount.create=true,serviceAccountName=kibana \
    --set plugins.enabled=true \
    --set persistentVolumeClaim.enabled=true \
    --set persistentVolumeClaim.existingClaim=true \
    --set securityContext.enabled=true \
    --set securityContext.allowPrivilegeEscalation=true \
    --set securityContext.runAsUser=0 \
    --set securityContext.fsGroup=0

    # pod调度相关配置，请根据实际情况设值
    --set nodeSelector."tag"="devops" \
    --set tolerations[0].key="key" \
    --set tolerations[0].operator="Equal" \
    --set tolerations[0].value="value" \
    --set tolerations[0].effect="NoSchedule" \
    --set affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key="key" \
    --set affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator=In \
    --set affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]=target-host-name \

    # xpack security相关参数：
    --set image.repository=docker.elastic.co/kibana/kibana \
    --set env."XPACK_SECURITY_ENABLED"="true" \
    --set env."ELASTICSEARCH_USERNAME"="kibana" \
    --set env."ELASTICSEARCH_PASSWORD"="<PASSWORD>" \
    --set dashboardImport.xpackauth.enabled=true \
    --set dashboardImport.xpackauth.username=kibana\
    --set dashboardImport.xpackauth.password=<PASSWORD>

----

==== jaeger

TIP: https://github.com/jaegertracing/jaeger-operator

[source,bash]
----
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing_v1_jaeger_crd.yaml
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -
curl https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/operator.yaml \
    | sed "s/namespace: observability/namespace: devops/g" \
    | kubectl create -f -

# 使用elasticsearch作为jaeger的数据源
    # 若ES启用Xpack Security，则需要创建secret
    ELASTICSEARCH_USERNAME=elastic
    ELASTICSEARCH_PASSWORD=<PASSWORD>
    cat <<EOF | kubectl -n devops apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: jaeger-es-secrets
    type: Opaque
    data:
      ES_USERNAME: `echo -n $ELASTICSEARCH_USERNAME | base64`
      ES_PASSWORD: `echo -n $ELASTICSEARCH_PASSWORD | base64`
    EOF

    # 创建Jaeger实例
    cat <<EOF | kubectl apply -n devops -f -
    apiVersion: jaegertracing.io/v1
    kind: Jaeger
    metadata:
      name: jaeger
    spec:
      strategy: production
      storage:
        type: elasticsearch
        options:
          es:
            server-urls: http://dew-elasticsearch-client:9200
        secretName: jaeger-es-secrets  # 若ES启用Xpack Security，需要设置此项及创建secret
    EOF

    TIP: Jaeger实例可在不同namespace下创建使用，使用中请注意namespace的问题。
    使用sidecar的方式部署项目：https://github.com/jaegertracing/jaeger-operator#auto-injection-of-jaeger-agent-sidecars
    使用daemonset的方式部署项目：https://github.com/jaegertracing/jaeger-operator#agent-as-daemonset

# 修改Ingress
cat <<EOF | kubectl -n devops apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  name: jaeger-query
spec:
  rules:
    - host: jaeger.dew.ms
      http:
        paths:
          - backend:
              serviceName: jaeger-query
              servicePort: 16686
            path: /
EOF

----

.pod的调度
目前jaeger-operator暂不支持直接设置，请关注该项目的更新情况。
可以自行给需要调度的pod的deployment添加限制条件。可参考： <<podAssignment>>


==== prometheus-operator 和 grafana
TIP: https://github.com/helm/charts/tree/master/stable/prometheus-operator

  prometheus-operator结构：
    |--- prometheus-operator
    |--- prometheus
    |--- alertmanager
    |--- node-exporter
    |--- kube-state-metrics
    |--- service monitors to scrape internal kubernetes components
    |     |---kube-apiserver
    |     |---kube-scheduler
    |     |---kube-controller-manager
    |     |---etcd
    |     |---kube-dns/coredns
    |
    |--- grafana

===== prometheus-operator 使用PV

[source,bash]
----
# 在NFS服务器上加上文件路径,并创建需要的目录

vi /etc/exports

/data/nfs/prometheus-operator    *(rw,sync,no_root_squash,no_all_squash)

# 重启NFS
systemctl restart nfs-server
# 查看结果
showmount -e localhost


# 创建PV,注意label的对应

app=prometheus-operator
components=("alertmanager" "prometheus")
size=100Gi

for i in ${components[@]};do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    component: ${i}
  name: dew-${app}-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/${app}/${i}
    server: 10.200.131.182
EOF
done

----

===== 创建grafana的PVC和PV
[source,bash]
----
apps=("grafana")
size=50Gi

for i in ${apps[@]};do
cat <<EOF | kubectl -n devops apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: ${size}
  selector:
    matchLabels:
      app: ${i}
---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    app: ${i}
  name: dew-${i}
spec:
  capacity:
    storage: ${size}
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  nfs:
    path: /data/nfs/prometheus-operator/${i}
    server: 10.200.131.182
EOF
done

----


===== 使用helm 安装

注意安装前先更新chart仓库
`helm repo update`

[source,yaml]
----
# 若需要对etcd进行监控，则需要先创建secret
kubectl -n devops create secret generic dew-prometheus-operator-etcd  --from-file=/etc/kubernetes/pki/etcd/ca.crt  --from-file=/etc/kubernetes/pki/etcd/peer.crt  --from-file=/etc/kubernetes/pki/etcd/peer.key

helm install stable/prometheus-operator --name dew-prometheus-operator --namespace devops \
    --set kubelet.serviceMonitor.https=true \
    --set prometheus.ingress.enabled=true \
    --set prometheus.ingress.hosts={prometheus.dew.ms} \
    --set alertmanager.ingress.enabled=true \
    --set alertmanager.ingress.hosts={prometheus.alertmanager.ms} \
    --set prometheusOperator.securityContext.runAsNonRoot=false \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=100Gi \
    --set alertmanager.alertmanagerSpec.storage.volumeClaimTemplate.spec.resources.requests.storage=100Gi \
    --set alertmanager.alertmanagerSpec.storage.volumeClaimTemplate.spec.selector.matchLabels."component"="alertmanager" \
    --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.selector.matchLabels."component"="prometheus"
    # 对etcd监测相关参数
    --set prometheus.prometheusSpec.secrets[0]=dew-prometheus-operator-etcd
    --set kubeEtcd.serviceMonitor.scheme=https \
    --set kubeEtcd.serviceMonitor.insecureSkipVerify=true \
    --set kubeEtcd.serviceMonitor.caFile="/etc/prometheus/secrets/dew-prometheus-operator-etcd/ca.crt" \
    --set kubeEtcd.serviceMonitor.certFile="/etc/prometheus/secrets/dew-prometheus-operator-etcd/peer.crt" \
    --set kubeEtcd.serviceMonitor.keyFile="/etc/prometheus/secrets/dew-prometheus-operator-etcd/peer.key"
    # 直接使用prometheus-operator的grafana，添加以下设置
    --set grafana.enabled=true \
    --set grafana.adminPassword=<PASSWORD> \
    --set grafana.defaultDashboardsEnabled=true \
    --set grafana.ingress.enabled=true \
    --set grafana.ingress.hosts={grafana.dew.ms} \
    --set grafana.ingress.tls[0].host={grafana.dew.ms},ingress.tls[0].secretName=dew-grafana \
    --set grafana.sidecar.dashboards.enabled=true \
    --set grafana.sidecar.dashboards.searchNamespace="devops"\
    --set grafana.sidecar.dashboards.label=grafana_dashboard \
    --set grafana.sidecar.datasources.enabled=true \
    --set grafana.sidecar.datasources.searchNamespace="devops" \
    --set grafana.sidecar.datasources.label=grafana_datasource \
    --set grafana.'grafana\.ini'.smtp.enabled="true" \
    --set grafana.'grafana\.ini'.smtp.host="smtp.163.com:25" \
    --set grafana.'grafana\.ini'.smtp.user=<EMAIL> \
    --set grafana.'grafana\.ini'.smtp.password=<PASSWORD> \
    --set grafana.'grafana\.ini'.smtp.from_address="<EMAIL>" \
    --set grafana.'grafana\.ini'.smtp.skip_verify=true \
    --set grafana.persistence.enabled=true \
    --set grafana.persistence.existingClaim=dew-grafana
    # 可选设置
    --set grafana.'grafana\.ini'.server.root_url="https://grafana.dew.ms"

    如不使用代理,更换以下镜像仓库
    --set kube-state-metrics.image.repository=registry.cn-hangzhou.aliyuncs.com/google_containers/kube-state-metrics

 TIP: grafana默认用户名：admin,
 查看密码：
 kubectl get secret --namespace devops dew-prometheus-operator-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

 grafana重置密码：进入grafana的容器内部后
 grafana-cli admin reset-admin-password passwordvalue

INFO: 若有pod启动失败,报文件权限拒绝相关问题，很可能和PV的文件目录的权限有关，检查下权限是否一致，设置对应的securityContext进行排查。
例：
kubectl edit statefulset prometheus-dew-prometheus-operator-prometheus -n devops
设置securityContext为以下内容
      securityContext:
        fsGroup: 0
        runAsNonRoot: false
        runAsUser: 0

INFO: 若验证prometheus的target，kube-scheduler、kube-controller处于down状态，是因为它们只能在宿主机上通过127.0.0.1访问，可使用以下操作：
    . 如果使用kubeadm启动的集群，初始化时的config.yml里可以加入如下参数
        controllerManagerExtraArgs:
          address: 0.0.0.0
        schedulerExtraArgs:
          address: 0.0.0.0
    . 已经启动后的使用下面命令更改就会滚动更新
        sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml
        sed -e "s/- --address=127.0.0.1/- --address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml
      或者全部替换：
        sed -ri '/--address/s#=.+#=0.0.0.0#' /etc/kubernetes/manifests/kube-*
    参考文章： http://www.servicemesher.com/blog/prometheus-operator-manual/
              https://github.com/coreos/prometheus-operator/blob/master/Documentation/troubleshooting.md

# 监控APP
  1.首先需要将项目instrument
    参考文章：https://prometheus.io/docs/instrumenting/clientlibs/
  2.部署项目及创建进行监控的ServiceMonitor。
    注意ServiceMonitor的labels要含有Prometheus-operator创建的Prometheus的serviceMonitorSelector的label。
    详细文章：https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md#related-resources
----

===== pod的调度
pod调度相关配置，以下配置仅供参考
[source,yaml]
----
{'fraction_non_alphanumeric': 0.138692864720262, 'fraction_numerical': 0.024153252463754746, 'mean_word_length': 3.3158383650719925, 'pattern_counts': {'":': 12, '<': 149, '<?xml version=': 0, '>': 93, 'https://': 86, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 82, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6265975', 'n_tokens_mistral': 27391, 'n_tokens_neox': 26609, 'n_words': 3913}