云計(jì)算 一、簡(jiǎn)介 Prometheus
Prometheus 是一個(gè)開(kāi)源監(jiān)控系統(tǒng),它本身已經(jīng)成為了云原生中指標(biāo)監(jiān)控的事實(shí)標(biāo)準(zhǔn),幾乎所有 Kubernetes 的核心組件以及其它云原生系統(tǒng)都以 Prometheus 的指標(biāo)格式輸出自己的運(yùn)行時(shí)監(jiān)控信息。

主要特性:
使用指標(biāo)名稱及鍵值對(duì)標(biāo)識(shí)的多維度數(shù)據(jù)模型。 采用彈性查詢語(yǔ)言PromQL。 不依賴分布式存儲(chǔ),為自治的單點(diǎn)服務(wù)。 使用http完成對(duì)監(jiān)控?cái)?shù)據(jù)的拉取。 通過(guò)網(wǎng)關(guān)支持時(shí)序數(shù)據(jù)的推送。 支持多種圖形和Dashboard的展示。另外在Prometheus的整個(gè)生態(tài)系統(tǒng)中有各種可選組件,用于功能的擴(kuò)充。
Prometheus-OperatorCoreOS提供了一種名為Operator的管理工具,他是管理特定應(yīng)用程序的控制器。通過(guò)擴(kuò)展Kubernetes API以軟件的方式幫助用戶創(chuàng)建、配置和管理復(fù)雜的或又狀態(tài)的應(yīng)用程序?qū)嵗ㄈ鏴tcd、Redis、mysql、Prometheus等)。
它通過(guò)Kubernetes的CRD(Custom Resource Definition,自定義資源定義)對(duì)Prometheus和Prometheus需要監(jiān)控的服務(wù)進(jìn)行部署和配置。
Prometheus-Operator使用下面兩種資源來(lái)配置Prometheus及其要監(jiān)控的服務(wù)。
Prometheus:為Prometheus的deployment。 ServiceMonitor:用于描述Prometheus監(jiān)控的服務(wù)。首先我們先來(lái)了解下 Prometheus-Operator 的架構(gòu)圖:
上圖是 Prometheus-Operator 官方提供的架構(gòu)圖,其中 Operator 是最核心的部分,作為一個(gè)控制器,他會(huì)去創(chuàng)建 Prometheus 、 ServiceMonitor 、 AlertManager 以及 PrometheusRule 4個(gè) CRD 資源對(duì)象,然后會(huì)一直監(jiān)控并維持這4個(gè)資源對(duì)象的狀態(tài)。
其中創(chuàng)建的 prometheus 這種資源對(duì)象就是作為 Prometheus Server 存在,而 ServiceMonitor 就是 exporter 的各種抽象, exporter是用來(lái)提供專門提供 metrics 數(shù)據(jù)接口的工具, Prometheus 就是通過(guò) ServiceMonitor 提供的 metrics 數(shù)據(jù)接口去 pull 數(shù)據(jù)的。
當(dāng)然 alertmanager 這種資源對(duì)象就是對(duì)應(yīng)的 AlertManager 的抽象,而 PrometheusRule 是用來(lái)被 Prometheus 實(shí)例使用的報(bào)警規(guī)則文件。
二、說(shuō)明本文檔基于Prometheus Operator對(duì)Prometheus監(jiān)控系統(tǒng)而進(jìn)行,完整的配置文件請(qǐng)參考https://github.com/coreos/prometheus-operator
本文使用的k8s集群是基于kubeadm搭建的,具體搭建的詳細(xì)請(qǐng)參考https://blog.51cto.com/billy98/2350660 三、部署 1. 環(huán)境準(zhǔn)備配置Prometheus-Operator之前需要先準(zhǔn)備以下幾個(gè)環(huán)境:
Helm環(huán)境:請(qǐng)參考https://blog.51cto.com/billy98/2338415 gluster-heketi環(huán)境:因?yàn)閜rometheus和alertmanager都是有狀態(tài)的statefulsets集群,所以需要使用到公共存儲(chǔ)。請(qǐng)參考https://blog.51cto.com/billy98/2337874 Ingress環(huán)境:請(qǐng)參考https://blog.51cto.com/billy98/2337874 2. 配置域名解析將alert.cnlinux.club、grafana.cnlinux.club、prom.cnlinux.club三個(gè)域名的A記錄解析到負(fù)責(zé)均衡的IP10.31.90.200。
修改/etc/kubernetes/manifests/ 目錄下kube-controller-manager.yaml和kube-scheduler.yaml
將監(jiān)聽(tīng)地址改成--address=0.0.0.0,重啟kubelet服務(wù)
systemctl restart kubelet.service
4. 創(chuàng)建Namespace
kubectl create ns monitoring
5. 創(chuàng)建Secret因?yàn)閑tcd是使用https訪問(wèn)的,所以prometheus的容器中也必須要etcd的證書(shū)去監(jiān)控etcd集群,創(chuàng)建Secret就是將證書(shū)掛載到prometheus容器中,后續(xù)還需要在Prometheus-Operator的配置文件中使用此Secret。
kubectl -n monitoring create secret generic etcd-certs --from-file=/etc/kubernetes/pki/etcd/healthcheck-clien t.crt --from-file=/etc/kubernetes/pki/etcd/healthcheck-client.key --from-file=/etc/kubernetes/pki/etcd/ca.crt
6. 配置Prometheus-Operator
1)首先使用helm下載Prometheus-Operator文件包。
helm fetch stable/prometheus-operator
2)解壓下載的壓縮包
tar zxf prometheus-operator-1.8.0.tgz并修改prometheus-operator目錄下的values.yaml。
具體修改的配置如下(配置過(guò)多,其他未修改的就不再展示了):
nameOverride: p
alertmanager:
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
labels: {}
hosts:
- alert.cnlinux.club
tls: []
alertmanagerSpec:
storage:
volumeClaimTemplate:
spec:
storageClassName: gluster-heketi
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 20Gi
selector: {}
grafana:
enabled: true
adminPassword: admin
#grafana登錄密碼
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
labels: {}
hosts:
- grafana.cnlinux.club
kubeApiServer:
enabled: true
tlsConfig:
serverName: kubernetes
insecureSkipVerify: true
serviceMonitor:
jobLabel: component
selector:
matchLabels:
component: apiserver
provider: kubernetes
kubelet:
enabled: true
namespace: kube-system
serviceMonitor:
https: true
kubeControllerManager:
enabled: true
endpoints: []
service:
port: 10252
targetPort: 10252
selector:
component: kube-controller-manager
coredns:
enabled: true
service:
port: 9153
targetPort: 9153
selector:
k8s-app: kube-dns
kubeEtcd:
enabled: true
endpoints: []
service:
port: 2379
targetPort: 2379
selector:
component: etcd
serviceMonitor:
scheme: https
insecureSkipVerify: false
serverName:
caFile: /etc/prometheus/secrets/etcd-certs/ca.crt
certFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.crt
keyFile: /etc/prometheus/secrets/etcd-certs/healthcheck-client.key
#secret etcd-certs掛載在prometheus的路徑是/etc/prometheus/secrets/etcd-certs,證書(shū)文件名和secret一樣
kubeScheduler:
enabled: true
endpoints: []
service:
port: 10251
targetPort: 10251
selector:
component: kube-scheduler
prometheus:
ingress:
enabled: true
annotations:
kubernetes.io/ingress.class: nginx
labels: {}
hosts:
- prom.cnlinux.club
prometheusSpec:
secrets: [etcd-certs]
#上面步驟創(chuàng)建etcd證書(shū)的secret
storageSpec:
volumeClaimTemplate:
spec:
storageClassName: gluster-heketi
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 20Gi
selector: {}
3) 安裝prometheus-operator
[root@node-01 ~]# helm install --name p --namespace monitoring ./prometheus-operator
NAME: p
LAST DEPLOYED: Tue Feb 26 14:30:52 2019
NAMESPACE: monitoring
STATUS: DEPLOYED
RESOURCES:
==> v1beta1/DaemonSet
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
p-prometheus-node-exporter 6 6 1 6 1 <none> 5s
==> v1beta2/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
p-grafana 1 1 1 0 5s
==> v1/PrometheusRule
NAME AGE
p-alertmanager.rules 4s
p-etcd 4s
p-general.rules 4s
p-k8s.rules 4s
p-kube-apiserver.rules 4s
p-kube-prometheus-node-alerting.rules 4s
p-kube-prometheus-node-recording.rules 4s
p-kube-scheduler.rules 4s
p-kubernetes-absent 4s
p-kubernetes-apps 4s
p-kubernetes-resources 4s
p-kubernetes-storage 4s
p-kubernetes-system 4s
p-node.rules 4s
p-prometheus-operator 4s
p-prometheus.rules 4s
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
p-prometheus-node-exporter-48lw9 0/1 Running 0 5s
p-prometheus-node-exporter-7lpvx 0/1 Running 0 5s
p-prometheus-node-exporter-8q577 1/1 Running 0 5s
p-prometheus-node-exporter-ls8cx 0/1 Running 0 5s
p-prometheus-node-exporter-nbl2g 0/1 Running 0 5s
p-prometheus-node-exporter-v7tb5 0/1 Running 0 5s
p-grafana-fcf4dc6bb-9c6pg 0/3 ContainerCreating 0 5s
p-kube-state-metrics-57d788d69-vmh52 0/1 Running 0 5s
p-operator-666b958c4f-wvd4h 1/1 Running 0 5s
==> v1beta1/ClusterRole
NAME AGE
p-kube-state-metrics 6s
psp-p-prometheus-node-exporter 6s
==> v1/Service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
p-grafana ClusterIP 10.245.103.159 <none> 80/TCP 5s
p-kube-state-metrics ClusterIP 10.245.150.181 <none> 8080/TCP 5s
p-prometheus-node-exporter ClusterIP 10.245.98.70 <none> 9100/TCP 5s
p-alertmanager ClusterIP 10.245.10.5 <none> 9093/TCP 5s
p-coredns ClusterIP None <none> 9153/TCP 5s
p-kube-controller-manager ClusterIP None <none> 10252/TCP 5s
p-kube-etcd ClusterIP None <none> 2379/TCP 5s
p-kube-scheduler ClusterIP None <none> 10251/TCP 5s
p-operator ClusterIP 10.245.31.238 <none> 8080/TCP 5s
p-prometheus ClusterIP 10.245.109.85 <none> 9090/TCP 5s
==> v1/ClusterRoleBinding
NAME AGE
p-grafana-clusterrolebinding 6s
p-alertmanager 6s
p-operator 6s
p-operator-psp 6s
p-prometheus 6s
p-prometheus-psp 6s
==> v1beta1/ClusterRoleBinding
NAME AGE
p-kube-state-metrics 6s
psp-p-prometheus-node-exporter 6s
==> v1beta1/Role
NAME AGE
p-grafana 6s
==> v1/RoleBinding
NAME AGE
p-prometheus-config 5s
p-prometheus 4s
p-prometheus 4s
==> v1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
p-operator 1 1 1 1 5s
==> v1/Alertmanager
NAME AGE
p-alertmanager 5s
==> v1/Secret
NAME TYPE DATA AGE
p-grafana Opaque 3 6s
alertmanager-p-alertmanager Opaque 1 6s
==> v1/ServiceAccount
NAME SECRETS AGE
p-grafana 1 6s
p-kube-state-metrics 1 6s
p-prometheus-node-exporter 1 6s
p-alertmanager 1 6s
p-operator 1 6s
p-prometheus 1 6s
==> v1beta1/Deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
p-kube-state-metrics 1 1 1 0 5s
==> v1beta1/Ingress
NAME HOSTS ADDRESS PORTS AGE
p-grafana grafana.cnlinux.club 80 5s
p-alertmanager alert.cnlinux.club 80 5s
p-prometheus prom.cnlinux.club 80 5s
==> v1beta1/PodSecurityPolicy
NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES
p-grafana false RunAsAny RunAsAny RunAsAny RunAsAny false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
p-prometheus-node-exporter false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
p-alertmanager false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
p-operator false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
p-prometheus false RunAsAny RunAsAny MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
==> v1/ConfigMap
NAME DATA AGE
p-grafana-config-dashboards 1 6s
p-grafana 1 6s
p-grafana-datasource 1 6s
p-etcd 1 6s
p-grafana-coredns-k8s 1 6s
p-k8s-cluster-rsrc-use 1 6s
p-k8s-node-rsrc-use 1 6s
p-k8s-resources-cluster 1 6s
p-k8s-resources-namespace 1 6s
p-k8s-resources-pod 1 6s
p-nodes 1 6s
p-persistentvolumesusage 1 6s
p-pods 1 6s
p-statefulset 1 6s
==> v1beta1/RoleBinding
NAME AGE
p-grafana 5s
==> v1/Prometheus
NAME AGE
p-prometheus 4s
==> v1/ServiceMonitor
NAME AGE
p-alertmanager 4s
p-coredns 4s
p-apiserver 4s
p-kube-controller-manager 4s
p-kube-etcd 4s
p-kube-scheduler 4s
p-kube-state-metrics 4s
p-kubelet 4s
p-node-exporter 4s
p-operator 4s
p-prometheus 4s
==> v1/ClusterRole
NAME AGE
p-grafana-clusterrole 6s
p-alertmanager 6s
p-operator 6s
p-operator-psp 6s
p-prometheus 6s
p-prometheus-psp 6s
==> v1/Role
NAME AGE
p-prometheus-config 6s
p-prometheus 4s
p-prometheus 4s
NOTES:
The Prometheus Operator has been installed. Check its status by running:
kubectl --namespace monitoring get pods -l release=p
Visit https://github.com/coreos/prometheus-operator for instructions on how
to create & configure Alertmanager and Prometheus instances using the Operator.
6. 部署中遇到的問(wèn)題在部署中有幾個(gè)坑,我在此列舉一下,大家配置的時(shí)候需要注意一下
1)名稱過(guò)長(zhǎng)導(dǎo)致pvc創(chuàng)建失敗由于alertmanager和prometheus都是有狀態(tài)的statefulsets,所以我們使用了gluster的存儲(chǔ),并通過(guò) prometheus-operator自動(dòng)創(chuàng)建pvc,如果charts的release 名稱過(guò)長(zhǎng)會(huì)導(dǎo)致pvc創(chuàng)建失敗。
所以在上面的安裝中指定了release的名字為phelm install --name p --namespace monitoring ./prometheus-operator,并且在配置文件中也修改了namenameOverride: p。
Warning ProvisioningFailed 3s (x2 over 40s) persistentvolume-controller Failed to provision volume with StorageClass gluster-heketi: failed to create volume: failed to create endpoint/service
default/glusterfs-dynamic-72488422-3428-11e9-a74b-005056824bdc: failed to create endpoint: Endpoints glusterfs-dynamic-72488422-3428-11e9-a74b-005056824bdc is invalid:
metadata.labels: Invalid value: alertmanager-prom-alertmanager-db-alertmanager-prom-alertmanager-0: must be no more than 63 characters
2)配置文件中l(wèi)abels首先要查看pod的標(biāo)簽,然后修改修改prometheus-operator目錄下的values.yaml對(duì)應(yīng)的標(biāo)簽。
[root@node-01 ~]# kubectl -n kube-system get pod --show-labels
NAME READY STATUS RESTARTS AGE LABELS
coredns-7f65654f74-6gxps 1/1 Running 8 5d22h k8s-app=kube-dns,pod-template-hash=7f65654f74
etcd-node-01 1/1 Running 1 32d component=etcd,tier=control-plane
kube-controller-manager-node-01 1/1 Running 0 39h component=kube-controller-manager,tier=control-plane
kube-scheduler-node-01 1/1 Running 0 23h component=kube-scheduler,tier=control-plane
...需要注意的是一定要修改prometheus-operator目錄下values.yaml對(duì)應(yīng)的標(biāo)簽,不能在安裝的時(shí)候指定外部的配置文件來(lái)覆蓋labels值,這可能是個(gè)bug,指定外部配置時(shí)無(wú)法覆蓋labels而是追加,會(huì)導(dǎo)致prometheus無(wú)法抓取到數(shù)據(jù)。
7. 更新和刪除如果在修改了配置文件values.yaml,可以使用以下命令更新prometheus-operator
helm upgrade RELEASE_NAME ./prometheus-operator如果需要?jiǎng)h除,可以使用以下命令
helm del --purge RELEASE_NAME
kubectl -n monitoring delete crd prometheuses.monitoring.coreos.com
kubectl -n monitoring delete crd prometheusrules.monitoring.coreos.com
kubectl -n monitoring delete crd servicemonitors.monitoring.coreos.com
kubectl -n monitoring delete crd alertmanagers.monitoring.coreos.com
8. 驗(yàn)證部署完后可以在瀏覽器訪問(wèn)prometheushttp://prom.cnlinux.club/targets,可以看到如下圖,所有的項(xiàng)都有數(shù)據(jù),并且是UP狀態(tài)的。
瀏覽器訪問(wèn)grafana http://grafana.cnlinux.club/,可以看到各種資源的監(jiān)控圖。
用戶名為admin,密碼為values.yaml配置文件中指定的密碼
瀏覽器訪問(wèn)Alertmanagerhttp://alert.cnlinux.club/,可以看到報(bào)警項(xiàng)。
至此所有的安裝完成,下一篇將詳細(xì)說(shuō)明使用prometheus監(jiān)控自定義服務(wù),以及報(bào)警設(shè)置。如有問(wèn)題歡迎在下面留言交流。希望大家多多關(guān)注和點(diǎn)贊,謝謝!
網(wǎng)頁(yè)名稱:使用PrometheusOperator監(jiān)控kubetnetes集群
標(biāo)題路徑:http://chinadenli.net/article14/cjshge.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供域名注冊(cè)、網(wǎng)站維護(hù)、網(wǎng)站收錄、移動(dòng)網(wǎng)站建設(shè)、App開(kāi)發(fā)、定制網(wǎng)站
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來(lái)源: 創(chuàng)新互聯(lián)