1. 概述
1.1 监控指标
一个好的系统，主要监控以下内容
- 集群监控
- 节点资源利用率
- 节点数
- 运行Pods
- Pod监控
- 容器指标
- 应用程序【程序占用多少CPU、内存】
1.2 监控平台
使用普罗米修斯【prometheus】 + Grafana 搭建监控平台
- prometheus【定时搜索被监控服务的状态】
- 开源的
- 监控、报警、数据库
- 以HTTP协议周期性抓取被监控组件状态
- 不需要复杂的集成过程，使用http接口接入即可
- Grafana
- 开源的数据分析和可视化工具
- 支持多种数据源
2. 搭建监控平台
2.1 部署Prometheus
首先需要部署一个守护进程 node-exporter.yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: node-exporter
namespace: kube-system
labels:
k8s-app: node-exporter
spec:
selector:
matchLabels:
k8s-app: node-exporter
template:
metadata:
labels:
k8s-app: node-exporter
spec:
containers:
- image: prom/node-exporter
name: node-exporter
ports:
- containerPort: 9100
protocol: TCP
name: http
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: node-exporter
name: node-exporter
namespace: kube-system
spec:
ports:
- name: http
port: 9100
nodePort: 31672
protocol: TCP
type: NodePort
selector:
k8s-app: node-exporter
执行下列命令
kubectl create -f node-exporter.yaml
[root@k8s-master monitor]# kubectl create -f node-exporter.yaml
daemonset.apps/node-exporter created
service/node-exporter created
通过yaml的方式部署prometheus
[root@k8s-master prometheus]# ls
configmap.yaml prometheus.deploy.yml prometheus.svc.yml rbac-setup.yaml
- configmap：定义一个configmap：存储一些配置文件【不加密】
- prometheus.deploy.yaml：部署一个deployment【包括端口号，资源限制】
- prometheus.svc.yaml：对外暴露的端口
- rbac-setup.yaml：分配一些角色的权限
下面我们进入目录下，首先部署 rbac-setup.yaml
[root@k8s-master prometheus]# kubectl create -f rbac-setup.yaml
clusterrole.rbac.authorization.k8s.io/prometheus created
serviceaccount/prometheus created
clusterrolebinding.rbac.authorization.k8s.io/prometheus created
然后分别部署
# 部署configmap
[root@k8s-master prometheus]# kubectl create -f configmap.yaml
configmap/prometheus-config created
# 部署deployment
[root@k8s-master prometheus]# kubectl create -f prometheus.deploy.yml
deployment.apps/prometheus created
# 部署svc
[root@k8s-master prometheus]# kubectl create -f prometheus.svc.yml
service/prometheus created
部署完成后，我们使用下面命令查看
kubectl get pods -n kube-system
[root@k8s-master prometheus]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-6jdxf 1/1 Running 0 3d1h
coredns-7ff77c879f-bt9mt 1/1 Running 0 3d1h
etcd-k8s-master 1/1 Running 0 3d1h
kube-apiserver-k8s-master 1/1 Running 1 3d1h
kube-controller-manager-k8s-master 1/1 Running 5 3d1h
kube-flannel-ds-bv727 1/1 Running 0 3d
kube-flannel-ds-fm7fz 1/1 Running 0 3d
kube-flannel-ds-rdj26 1/1 Running 0 3d
kube-proxy-5rqfw 1/1 Running 0 3d1h
kube-proxy-fqqtz 1/1 Running 0 3d
kube-proxy-ljndp 1/1 Running 0 3d
kube-scheduler-k8s-master 1/1 Running 5 3d1h
node-exporter-6x75f 1/1 Running 0 9m12s
node-exporter-6xnwn 1/1 Running 0 9m12s
prometheus-7486bf7f4b-csdts 1/1 Running 0 2m8s
在我们部署完成后，即可看到 prometheus 的 pod了，然后通过下面命令，能够看到对应的端口
kubectl get svc -n kube-system
[root@k8s-master prometheus]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 3d1h
node-exporter NodePort 10.108.230.146 <none> 9100:31672/TCP 10m
prometheus NodePort 10.104.207.74 <none> 9090:30003/TCP 2m50s
通过这个，我们可以看到Prometheus对外暴露的端口为 30003，访问页面 http://10.211.55.15:30003 即可对应的图形化界面
2.2 部署Grafana
通过yaml的方式部署grafana
[root@k8s-master grafana]# ls
grafana-deploy.yaml grafana-ing.yaml grafana-svc.yaml
部署
# 创建deployment
kubectl create -f grafana-deploy.yaml
[root@k8s-master grafana]# kubectl create -f grafana-deploy.yaml
deployment.apps/grafana-core created
# 创建svc
kubectl create -f grafana-svc.yaml
[root@k8s-master grafana]# kubectl create -f grafana-svc.yaml
service/grafana created
# 创建 ing
kubectl create -f grafana-ing.yaml
[root@k8s-master grafana]# kubectl create -f grafana-ing.yaml
ingress.extensions/grafana created
部署完成后，我们使用下面命令查看
[root@k8s-master grafana]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-7ff77c879f-6jdxf 1/1 Running 0 3d1h
coredns-7ff77c879f-bt9mt 1/1 Running 0 3d1h
etcd-k8s-master 1/1 Running 0 3d1h
grafana-core-768b6bf79c-x9kcf 1/1 Running 0 100s
kube-apiserver-k8s-master 1/1 Running 1 3d1h
kube-controller-manager-k8s-master 1/1 Running 5 3d1h
kube-flannel-ds-bv727 1/1 Running 0 3d
kube-flannel-ds-fm7fz 1/1 Running 0 3d
kube-flannel-ds-rdj26 1/1 Running 0 3d
kube-proxy-5rqfw 1/1 Running 0 3d1h
kube-proxy-fqqtz 1/1 Running 0 3d1h
kube-proxy-ljndp 1/1 Running 0 3d1h
kube-scheduler-k8s-master 1/1 Running 5 3d1h
node-exporter-6x75f 1/1 Running 0 21m
node-exporter-6xnwn 1/1 Running 0 21m
prometheus-7486bf7f4b-csdts 1/1 Running 0 14m
2.3 打开Grafana，配置Prometheus数据源，导入显示模板
下面需要开始打开 Grafana，然后配置数据源，导入数据显示模板
1、通过端口号访问Grafana
[root@k8s-master grafana]# kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana NodePort 10.102.196.188 <none> 3000:31583/TCP 6m37s
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 3d1h
node-exporter NodePort 10.108.230.146 <none> 9100:31672/TCP 26m
prometheus NodePort 10.104.207.74 <none> 9090:30003/TCP 19m
访问http://10.211.55.15:31583
然后输入账号和密码：admin admin
2、进入后，我们就需要配置 prometheus 的数据源
IP填写用命令查看出的IP
3、设置显示数据的模板
选择Dashboard，导入我们的模板
然后选择 prometheus数据源 mydb，导入即可
导入后的效果如下所示
