前言:

​ 目前项目上了rancher的K8S,rancher自带的应用商店可以一键部署EFK集群,但是生产环境有安全性的需求,这里需要对这个EFK集群进行改造,增加用户名密码的验证登陆.

1.efk基础设置

这里采用的是rancher自带的应用商店里的efk，并自定义了镜像地址（harbor转储）

所有镜像均取自elastic的官方源，镜像版本均为7.7.1：

镜像下载地址：https://www.docker.elastic.co/

由于日志数据不太重要，就没有选择持久化数据，这样性能也会相对好一点，缺点是如果重新部署，elasticsearch的数据都会清空。目前rancher自己的分布式存储longhorn也正式发布了,配置也简单,有条件的可以考虑将数据存放到分布式存储上.

2.配置信息变更

2.1 elasticsearch 的StatefulSet配置变更：

变更的参数：

env： ES_JAVA_OPTS跟认证无关，默认配置资源太少，容易oom；ELASTIC_USERNAME，ELASTIC_PASSWORD是为了elasticsearch集群的状态检测准备的

        - name: ES_JAVA_OPTS
          value: -Xmx4g -Xms4g
        - name: xpack.security.enabled
          value: "true"
        - name: ELASTIC_USERNAME
          value: elastic
        - name: ELASTIC_PASSWORD
          value: elasticpassword

resource：跟开启用户认证无关，默认配置资源太少，容易oom

        resources:
          limits:
            cpu: "4"
            memory: 8Gi
          requests:
            cpu: 100m
            memory: 8Gi

附上rancher上完整的yaml文件：

apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    esMajorVersion: "7"
    field.cattle.io/publicEndpoints: '[{"addresses":["10.1.99.51"],"port":80,"protocol":"HTTP","serviceName":"efk:elasticsearch-master-headless","ingressName":"efk:elastic-ingress","hostname":"elastic-prod.hlet.com","allNodes":true}]'
  creationTimestamp: "2020-06-03T08:34:13Z"
  generation: 4
  labels:
    app: elasticsearch-master
    chart: elasticsearch-7.3.0
    heritage: Tiller
    io.cattle.field/appId: efk
    release: efk
  name: elasticsearch-master
  namespace: efk
  resourceVersion: "22963322"
  selfLink: /apis/apps/v1/namespaces/efk/statefulsets/elasticsearch-master
  uid: 03f40362-4e89-4bd1-b8d3-285a36cbce35
spec:
  podManagementPolicy: Parallel
  replicas: 5
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: elasticsearch-master
  serviceName: elasticsearch-master-headless
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: elasticsearch-master
        chart: elasticsearch-7.3.0
        heritage: Tiller
        release: efk
      name: elasticsearch-master
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - elasticsearch-master
            topologyKey: kubernetes.io/hostname
      containers:
      - env:
        - name: node.name
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: cluster.initial_master_nodes
          value: elasticsearch-master-0,elasticsearch-master-1,elasticsearch-master-2,elasticsearch-master-3,elasticsearch-master-4,
        - name: discovery.seed_hosts
          value: elasticsearch-master-headless
        - name: cluster.name
          value: elasticsearch
        - name: network.host
          value: 0.0.0.0
        - name: ES_JAVA_OPTS
          value: -Xmx4g -Xms4g
        - name: node.data
          value: "true"
        - name: node.ingest
          value: "true"
        - name: node.master
          value: "true"
        - name: xpack.security.enabled
          value: "true"
        - name: ELASTIC_USERNAME
          value: elastic
        - name: ELASTIC_PASSWORD
          value: elasticpassword
        image: 10.1.99.42/ranchercharts/elasticsearch-elasticsearch:7.7.1
        imagePullPolicy: IfNotPresent
        name: elasticsearch
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              # If the node is starting up wait for the cluster to be ready (request params: 'wait_for_status=green&timeout=1s' )
              # Once it has started only check that the node itself is responding
              START_FILE=/tmp/.es_start_file

              http () {
                  local path="${1}"
                  if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                    BASIC_AUTH="-u ${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                  else
                    BASIC_AUTH=''
                  fi
                  curl -XGET -s -k --fail ${BASIC_AUTH} http://127.0.0.1:9200${path}
              }

              if [ -f "${START_FILE}" ]; then
                  echo 'Elasticsearch is already running, lets check the node is healthy'
                  http "/"
              else
                  echo 'Waiting for elasticsearch cluster to become cluster to be ready (request params: "wait_for_status=green&timeout=1s" )'
                  if http "/_cluster/health?wait_for_status=green&timeout=1s" ; then
                      touch ${START_FILE}
                      exit 0
                  else
                      echo 'Cluster is not yet ready (request params: "wait_for_status=green&timeout=1s" )'
                      exit 1
                  fi
              fi
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "4"
            memory: 8Gi
          requests:
            cpu: 100m
            memory: 8Gi
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      initContainers:
      - command:
        - sysctl
        - -w
        - vm.max_map_count=262144
        image: 10.1.99.42/ranchercharts/elasticsearch-elasticsearch:7.7.1
        imagePullPolicy: IfNotPresent
        name: configure-sysctl
        resources: {}
        securityContext:
          privileged: true
          runAsUser: 0
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
      terminationGracePeriodSeconds: 120
  updateStrategy:
    type: RollingUpdate
status:
  collisionCount: 0
  currentReplicas: 5
  currentRevision: elasticsearch-master-85f58497dd
  observedGeneration: 4
  readyReplicas: 5
  replicas: 5
  updateRevision: elasticsearch-master-85f58497dd
  updatedReplicas: 5

配置完后点击保存，elasticsearch集群会自动重新部署

注意：如果集群一直不能初始化完成，建议一次性删除所有elastic节点，让节点完全重新初始化

待重新部署完成后，我们需要初始化一下elastic内置的账户密码：

登陆任意一台elastic，执行命令：

elasticsearch-setup-passwords interactive

至此，elasticsearch集群初始化完成

2.2 kibana 配置变更

因为是使用的应用商店自动部署的，所以会自动生成两个service，分别是efk-kibana和kibana-http，

在实际配置中，将service应用到ingress的时候，出现了无法访问的问题，具体的问题是在kibana本地访问http://0.0.0.0:5601 是可以访问的，但是使用http://efk-kibana:5601 访问就不通，后来就重新加了一个efk-kibana-headless的无头服务，并应用至kibana的ingress配置上去就好了。后来晚些时候service自己又恢复正常了。。。

[root@hlet-prod-k8s-rancher ~]# kubectl get svc -n efk
NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
efk-kibana                      ClusterIP   10.43.127.11    <none>        5601/TCP            17d
efk-kibana-headless             ClusterIP   None            <none>        5601/TCP            130m
elasticsearch-apm               ClusterIP   10.43.238.31    <none>        8200/TCP            52d
elasticsearch-heartbeat         ClusterIP   10.43.172.214   <none>        9200/TCP            2d
elasticsearch-master            ClusterIP   10.43.21.168    <none>        9200/TCP,9300/TCP   17d
elasticsearch-master-headless   ClusterIP   None            <none>        9200/TCP,9300/TCP   17d
kibana-http                     ClusterIP   10.43.71.157    <none>        80/TCP              174m

ingress配置：

image-20200612173912785.png

svc配置自带的就不贴了

kibana的yaml主要修改了两块：

ENV：两组用户名密码分别是连接elastic集群的用户名密码和存活检测脚本调用

        - name: xpack.security.enabled
          value: "true"
        - name: ELASTICSEARCH_USERNAME
          value: kibana
        - name: ELASTIC_USERNAME
          value: kibana
        - name: ELASTICSEARCH_PASSWORD
          value: elasticpassword
        - name: ELASTIC_PASSWORD
          value: elasticpassword

存活检测：就改了最后一行，默认的地址在开启认证后没有登陆会一直报404

        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              http () {
                  local path="${1}"
                  set -- -XGET -s --fail

                  if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                    set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                  fi

                  curl -k "$@" "http://localhost:5601${path}"
              }

              http "/login"

附上完整的Deployment的yaml配置：

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "23"
    field.cattle.io/publicEndpoints: '[{"addresses":["10.1.99.51"],"port":80,"protocol":"HTTP","serviceName":"efk:kibana-http","ingressName":"efk:kibana-ingress","hostname":"kibana-prod.hlet.com","allNodes":true}]'
  creationTimestamp: "2020-05-26T00:53:53Z"
  generation: 49
  labels:
    app: kibana
    io.cattle.field/appId: efk
    release: efk
  name: efk-kibana
  namespace: efk
  resourceVersion: "23026049"
  selfLink: /apis/apps/v1/namespaces/efk/deployments/efk-kibana
  uid: 85017148-3738-46f9-8e29-65d072549a92
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: kibana
      release: efk
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        cattle.io/timestamp: "2020-06-09T00:17:32Z"
        field.cattle.io/ports: '[[{"containerPort":80,"dnsName":"efk-kibana","kind":"ClusterIP","name":"http","protocol":"TCP"}],[{"containerPort":5601,"dnsName":"efk-kibana","kind":"ClusterIP","name":"5601tcp2","protocol":"TCP"}]]'
        field.cattle.io/publicEndpoints: '[{"addresses":["10.1.99.51"],"allNodes":true,"hostname":"kibana-prod.hlet.com","ingressId":"efk:kibana-ingress","port":80,"protocol":"HTTP","serviceId":"efk:kibana-http"}]'
      creationTimestamp: null
      labels:
        app: kibana
        release: efk
    spec:
      containers:
      - args:
        - nginx
        - -g
        - daemon off;
        - -c
        - /nginx/nginx.conf
        image: rancher/nginx:1.15.8-alpine
        imagePullPolicy: IfNotPresent
        name: kibana-proxy
        ports:
        - containerPort: 80
          name: http
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /nginx/
          name: kibana-nginx
      - env:
        - name: ELASTICSEARCH_HOSTS
          value: http://elasticsearch-master:9200
        - name: I18N_LOCALE
          value: zh-CN
        - name: LOGGING_QUIET
          value: "true"
        - name: SERVER_HOST
          value: 0.0.0.0
        - name: xpack.security.enabled
          value: "true"
        - name: ELASTICSEARCH_USERNAME
          value: kibana
        - name: ELASTIC_USERNAME
          value: kibana
        - name: ELASTICSEARCH_PASSWORD
          value: elasticpassword
        - name: ELASTIC_PASSWORD
          value: elasticpassword
        image: 10.1.99.42/ranchercharts/kibana-kibana:7.7.1
        imagePullPolicy: IfNotPresent
        name: kibana
        ports:
        - containerPort: 5601
          name: 5601tcp2
          protocol: TCP
        readinessProbe:
          exec:
            command:
            - sh
            - -c
            - |
              #!/usr/bin/env bash -e
              http () {
                  local path="${1}"
                  set -- -XGET -s --fail

                  if [ -n "${ELASTIC_USERNAME}" ] && [ -n "${ELASTIC_PASSWORD}" ]; then
                    set -- "$@" -u "${ELASTIC_USERNAME}:${ELASTIC_PASSWORD}"
                  fi

                  curl -k "$@" "http://localhost:5601${path}"
              }

              http "/login"
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 3
          timeoutSeconds: 5
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: 100m
            memory: 500m
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          runAsUser: 1000
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
      terminationGracePeriodSeconds: 30
      volumes:
      - configMap:
          defaultMode: 420
          items:
          - key: nginx.conf
            mode: 438
            path: nginx.conf
          name: efk-kibana-nginx
        name: kibana-nginx
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2020-06-12T07:46:09Z"
    lastUpdateTime: "2020-06-12T07:46:09Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2020-06-12T07:29:26Z"
    lastUpdateTime: "2020-06-12T07:46:09Z"
    message: ReplicaSet "efk-kibana-9884bd66b" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 49
  readyReplicas: 1
  replicas: 1
  updatedReplicas: 1

到这里就可以尝试登陆kibana了登陆界面:

image-20200721112715807.png

2.3 apm 配置变更

由于我们elastic的组件还使用到了apm,继续修改apm相关设置

原始部署相关步骤：

apm是不包含在应用商店中的，部署相关yaml：

部署顺序：

kubectl create configmap elasticsearch-apm --from-file=apm-server.docker.yml -n efk 
kubectl apply -f elasticsearch-apm-server.yaml 

apm-server.docker.yml：

apm-server:
  host: "0.0.0.0:8200"
  kibana.enabled: true
  kibana.host: "efk-kibana:5601"
  kibana.protocol: "http"
logging.level: warning
output.elasticsearch:
  hosts: ["elasticsearch-master-headless:9200"]

apm.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  labels:
    app: elasticsearch-apm
  name: elasticsearch-apm
  namespace: efk
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: elasticsearch-apm
  template:
    metadata:
      labels:
        app: elasticsearch-apm
    spec:
      containers:
      - image: 10.1.99.42/docker.elastic.co/apm/apm-server:7.7.1
        imagePullPolicy: IfNotPresent
        name: elasticsearch-apm
        ports:
        - containerPort: 8200
          protocol: TCP
        resources:
          limits:
            cpu: "1"
          requests:
            cpu: 25m
            memory: 512Mi
        volumeMounts:
        - mountPath: /usr/share/apm-server/apm-server.yml
          name: config
          subPath: apm-server.docker.yml
      volumes:
      - configMap:
          defaultMode: 420
          name: elasticsearch-apm
        name: config
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: elasticsearch-apm
  name: elasticsearch-apm
  namespace: efk
spec:
  ports:
  - name: elasticsearch-apm
    port: 8200
    protocol: TCP
  selector:
    app: elasticsearch-apm

修改配置文件，适配用户认证

修改elasticsearch-apm这个configmap

apm-server.docker.yml

apm-server:
  host: "0.0.0.0:8200"
  kibana.enabled: true
  kibana.host: "efk-kibana-headless:5601"
  kibana.username: "elastic"
  kibana.password: "elasticpassword"
  kibana.protocol: "http"
logging.level: warning
#logging.level: info
output.elasticsearch:
  hosts: ["elasticsearch-master-headless:9200"]
  username: "elastic"
  password: "elasticpassword"

修改完成后，重新部署一下即可。

2.4 filebeat 配置变更

应用商店自带的，直接修改相应的configmap即可

修改efk-filebeat-config这个configmap

filebeat.yml：

filebeat.inputs:
- type: docker
  containers.ids:
  - '*'
  processors:
  - add_kubernetes_metadata:
      in_cluster: true

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
  username: "elastic"
  password: "elasticpassword"