prometheus-stack的数据持久化需要通过prometheus-opreate 配置。

prometheus 通过污点、容忍调度到master-01 上 。然后master-01 本地目录保持promethues数据。

1、创建pv

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

2、创建2个pvc，一个promethues , 一个alertmanager

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prom-local-pv
  labels:
    app: prometheus
spec:
  capacity:
    storage: 200Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/monitoring/prometheus
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-master-01

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: alert-local-pv
  labels:
    app: alert
spec:
  capacity:
    storage: 200Gi
  volumeMode: Filesystem
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  local:
    path: /data/monitoring/alertmanager
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-master-01

创建pvc 的时候，要使用nodeselector 来确保本地的pvc 创建在master-01上。
3、修改promethues CRD 挂载 pvc， 并配置选择器和容忍。

---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  annotations:
    meta.helm.sh/release-name: kube-prometheus-stack
    meta.helm.sh/release-namespace: monitoring
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: kube-prometheus-stack
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 59.1.0
    chart: kube-prometheus-stack-59.1.0
    heritage: Helm
    release: kube-prometheus-stack
  name: kube-prometheus-stack-prometheus
  namespace: monitoring
  resourceVersion: '11789793'
spec:
  alerting:
    alertmanagers:
      - apiVersion: v2
        name: kube-prometheus-stack-alertmanager
        namespace: monitoring
        pathPrefix: /
        port: http-web
  enableAdminAPI: false
  evaluationInterval: 30s
  externalUrl: 'http://kube-prometheus-stack-prometheus.monitoring:9090'
  hostNetwork: false
  image: 'quay.io/prometheus/prometheus:v2.52.0'
  listenLocal: false
  logFormat: logfmt
  logLevel: info
#################################
  nodeSelector:
    kubernetes.io/hostname: k8s-master-01
#################################
  paused: false
  podMonitorNamespaceSelector: {}
  podMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack
  portName: http-web
  probeNamespaceSelector: {}
  probeSelector:
    matchLabels:
      release: kube-prometheus-stack
  replicas: 1
  retention: 10d
  routePrefix: /
  ruleNamespaceSelector: {}
  ruleSelector:
    matchLabels:
      release: kube-prometheus-stack
  scrapeConfigNamespaceSelector: {}
  scrapeConfigSelector:
    matchLabels:
      release: kube-prometheus-stack
  scrapeInterval: 30s
  securityContext:
    fsGroup: 2000
    runAsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
    seccompProfile:
      type: RuntimeDefault
  serviceAccountName: kube-prometheus-stack-prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector:
    matchLabels:
      release: kube-prometheus-stack
  shards: 1
#################################
  storage:
    volumeClaimTemplate:
      spec:
        resources:
          requests:
            storage: 200Gi
        selector:
          matchLabels:
            app: prometheus
        storageClassName: local-storage
  tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/control-plane
      operator: Exists
#######################################
  tsdb:
    outOfOrderTimeWindow: 0s
  version: v2.52.0
  walCompression: true

配置选择器、容忍的目的是将promethues 和 alertmanager 的pod 调度到master-01上，因为pvc也在master-01。pvc模版通过matchlabel , 来选择使用哪个pvc。