kudbernetes安装 Metrics-server

从 v1.8 开始，资源使用情况的度量（如容器的 CPU 和内存使用）可以通过 Metrics API 获取。，用户可以直接获取这些metrics信息（例如通过执行kubect top命令），HPA使用这些metics信息来实现动态伸缩。
1 Metrics server是K8S集群资源使用情况的聚合器
2 从1.8版本开始，Metrics server默认可以通过kube-up.sh 脚本以deployment的方式进行部署，也可以通过yaml文件的方式进行部署
3 Metrics server收集所有node节点的metrics信息

需要注意的是：
1 Metrics API 只可以查询当前的度量数据，并不保存历史数据
2 Metrics API URI 为 /apis/metrics.k8s.io/，在 k8s.io/metrics 维护
3 必须部署 metrics-server 才能使用该 API，metrics-server 通过调用 Kubelet Summary API 获取数据

集群准备

  • 集群基本环境
    docker版本: 17.03.2
    k8s版本 1.9.6
    操作系统 centos7 64位
    内核版本 3.10.0-862.9.1.el7.x86_64
    完成k8s集群的基本安装：
    master：kube-apiserver，kube-controller-manger，kube-scheduler
    node：kubelet，kube-proxy
    api-server认证方式：node，rbac
    集群间的认证采用的是CFSSL + CA认证

安装 Metrics-server 方式一

生成认证文件 (需要更新个人搭建集群的情况来配置)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# vi front-proxy-ca-csr.json 
{
    "CN": "kubernetes",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}

######################

# vi front-proxy-client-csr.json 
{
    "CN": "front-proxy-client",
    "key": {
        "algo": "rsa",
        "size": 2048
    }
}

生成认证文件

1
2
3
4
5
6
7
8
9
10
# cfssl gencert   -initca front-proxy-ca-csr.json | cfssljson -bare front-proxy-ca
# cfssl gencert   -ca=front-proxy-ca.pem  \
  -ca-key=front-proxy-ca-key.pem \
-config=/etc/kubernetes/ca-config.json   \
-profile=kubernetes   \
front-proxy-client-csr.json | cfssljson -bare front-proxy-client

# mv front-proxy* /etc/ssl/kubernetes
# scp -rp /etc/ssl/kubernetes vm2:/etc/ssl/
# scp -rp /etc/ssl/kubernetes vm3:/etc/ssl/

开启API Aggregation

修改kube-apiserver服务配置文件

1
2
3
4
5
6
7
8
9
10
11
# vi /usr/lib/systemd/system/kube-apiserver.service 
# 添加如下配置
  --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt \   # 前文生成的证书
  --requestheader-allowed-names=aggregator \
  --requestheader-extra-headers-prefix=X-Remote-Extra- \
  --requestheader-group-headers=X-Remote-Group \
  --requestheader-username-headers=X-Remote-User \
  --proxy-client-cert-file=/etc/ssl/kubernetes/front-proxy-client.pem \  # 前文生成的证书
  --proxy-client-key-file=/etc/ssl/kubernetes/front-proxy-client-key.pem \ # 前文生成的证书
  --runtime-config=api/all=true \
  --enable-aggregator-routing=true \

修改kube-controller-manager服务配置文件

1
2
3
# vi /usr/lib/systemd/system/kube-controller-manager.service 
# 添加如下配置
  --horizontal-pod-autoscaler-use-rest-clients=true

重启服务

systemctl daemon-reload
systemctl  restart kube-apiserver.service
systemctl  restart kube-controller-manager

部署metrics-server

获取镜像文件

需要在所有节点执行
docker pull gcr.io/google_containers/metrics-server-amd64:v0.2.1
如果无法成功下拉镜像可以使用阿里云镜像
docker pull registry.cn-hangzhou.aliyuncs.com/k8s-kernelsky/metrics-server-amd64:v0.2.1

获取yaml文件并修改

git clone https://github.com/stefanprodan/k8s-prom-hpa
cd k8s-prom-hpa/

修改metrics-server-deployment.yaml 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# vi metrics-server/metrics-server-deployment.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: registry.cn-hangzhou.aliyuncs.com/k8s-kernelsky/metrics-server-amd64:v0.2.1
        imagePullPolicy: Always
        volumeMounts:
        - mountPath: /etc/kubernetes/pki/
          name: ca-ssl
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''
        - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      volumes:
       - name: ca-ssl
         hostPath:
          path: /etc/kubernetes/pki/

修改metrics-server-service.yaml 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# vi metrics-server/metrics-server-service.yaml 
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/name: "Metrics-server"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
    nodePort: 8499
  type: NodePort

执行yaml文件

kubectl create -f ./metrics-server
kubectl get pod,svc -n kube-system

验证Metrics-server服务

查看服务是否成功注册apiserver

kubectl get apiservice


kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

通过kubectl工具测试获取metrics数据

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"


为了方便观察我们可以安装jq
wget http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
rpm -ivh epel-release-latest-7.noarch.rpm
yum -y install jq
kubectl get –raw “/apis/metrics.k8s.io/v1beta1/nodes” | jq

可能出现的错误

  • 错误描述
    执行 kubectl get –raw “/apis/metrics.k8s.io/v1beta1/nodes” 时
    error: You must be logged in to the server (Unauthorized)
    pod的日志为：
    authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kubernetes”), x509: certificate signed by unknown authority]

  • 错误分析
    metrics-server连接api-server时 认证失败

  • 解决方法
    重启k8s集群重新加载认证相关信息

安装Metrics Server 方式二

直接部署整合Yaml文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
# rbac
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-server:system:auth-delegator
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: metrics-server-auth-reader
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

---
apiVersion: apiregistration.k8s.io/v1beta1
kind: APIService
metadata:
  name: v1beta1.metrics.k8s.io
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  service:
    name: metrics-server
    namespace: kube-system
  group: metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100


---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metrics-server-config
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: EnsureExists
data:
  NannyConfiguration: |-
    apiVersion: nannyconfig/v1alpha1
    kind: NannyConfiguration
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server-v0.2.1
  namespace: kube-system
  labels:
    k8s-app: metrics-server
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    version: v0.2.1
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
      version: v0.2.1
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
        version: v0.2.1
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      containers:
      - name: metrics-server
        image: gcr.azk8s.cn/google_containers/metrics-server-amd64:v0.2.1
        command:
        - /metrics-server
        - --source=kubernetes.summary_api:''
        ports:
        - containerPort: 443
          name: https
          protocol: TCP
      - name: metrics-server-nanny
        image: gcr.azk8s.cn/google_containers/addon-resizer:1.8.1
        resources:
          limits:
            cpu: 100m
            memory: 300Mi
          requests:
            cpu: 5m
            memory: 50Mi
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        volumeMounts:
        - name: metrics-server-config-volume
          mountPath: /etc/config
        command:
          - /pod_nanny
          - --config-dir=/etc/config
          - --cpu=40m
          - --extra-cpu=0.5m
          - --memory=40Mi
          - --extra-memory=4Mi
          - --threshold=5
          - --deployment=metrics-server-v0.2.1
          - --container=metrics-server
          - --poll-period=300000
          - --estimator=exponential
      volumes:
        - name: metrics-server-config-volume
          configMap:
            name: metrics-server-config
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"


---
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: "Metrics-server"
spec:
  selector:
    k8s-app: metrics-server
  ports:
  - port: 443
    protocol: TCP
    targetPort: https


---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats # 新增nodes/stats 的权限
  - namespaces
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
  - deployments
  verbs:
  - get
  - list
  - update
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: system:metrics-server
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system

参考

https://kubernetes.feisky.xyz/zh/addons/metrics.html
http://blog.51cto.com/ylw6006/2114338
https://github.com/kubernetes/metrics