docker-compose搭建es集群+EFK搭建分布式日志监控系统

PART1

通过docker-compose容器编排一键搭建elasticsearch(ik中文分词)集群

Elasticsearch

Elasticsearch（ES）是一个基于Lucene构建的开源、分布式、RESTful接口的全文搜索引擎。Elasticsearch还是一个分布式文档数据库，其中每个字段均可被索引，而且每个字段的数据均可被搜索，ES能够横向扩展至数以百计的服务器存储以及处理PB级的数据。可以在极短的时间内存储、搜索和分析大量的数据。通常作为具有复杂搜索场景情况下的核心发动机。
————————————————
版权声明：本文为CSDN博主「small小阮」的原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接及本声明。
原文链接：https://blog.csdn.net/deliciousion/article/details/78050251

Elasticsearch-analysis-ik

IK Analysis插件将Lucene IK分析器（http://code.google.com/p/ik-analyzer/）集成到elasticsearch中，支持自定义词典。
github地址https://github.com/medcl/elasticsearch-analysis-ik

项目上考虑使用es是因为es强大的全文索引和分词能力，如果你也有这种需求，恭喜你，不用重复踩坑了

话不多说直接上代码

version: '3.1'

services:
  es1_test:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: es1_test
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - es-cluster
    volumes:
      - ./plugins:/usr/share/elasticsearch/plugins/analysis-ik
    environment:
      - node.name=es1_test
      - discovery.seed_hosts=es1_test
      - cluster.initial_master_nodes=es1_test
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
  es2_test:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: es2_test
    networks:
      - es-cluster
    environment:
      - node.name=es2_test
      - discovery.seed_hosts=es1_test
      - cluster.initial_master_nodes=es1_test
      - cluster.name=es-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - ./plugins:/usr/share/elasticsearch/plugins/analysis-ik
    ulimits:
      memlock:
        soft: -1
        hard: -1

  kibana_test:
    image: docker.elastic.co/kibana/kibana:7.3.2
    container_name: kibana_test
    environment:
      - I18N_LOCALE=zh-CN #汉化
    ports:
      - "5601:5601"
    links:
      - es1_test:elasticsearch
    depends_on:
      - es1_test
      - es2_test
    networks:
      - es-cluster

networks:
  es-cluster:
    driver: bridge

保存为docker-compose.yml
启动即可

启动成功

参数详解

1.discovery.seed_hosts和cluster.initial_master_nodes

https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#discovery-settings
这两个参数关系可被发现的主机和集群master节点初始化

2.bootstrap.memory_lock=true

https://www.elastic.co/guide/en/elasticsearch/reference/master/setup-configuration-memory.html
此参数旨在关闭操作系统对es应用程序的强制内存交换操作，以提升es稳定和性能

3.volumes:
- ./plugins:/usr/share/elasticsearch/plugins/analysis-ik
优先下载ik插件压缩包

https://github.com/medcl/elasticsearch-analysis-ik/releases 自行选择适应版本

解压缩至对应目录并挂载

可以看到启动的时候自动安装好了ik插件

1.查看集群状态
get http://localhost:9200/_cluster/health

查看集群状态

2.创建索引用于分词测试
curl -XPUT http://localhost:9200/index_test

创建索引用于分词测试

3.创建对应的mapping映射

curl -XPOST http://localhost:9200/index_test/_mapping -H 'Content-Type:application/json' -d'
{
        "properties": {
            "content": {
                "type": "text",
                "analyzer": "ik_max_word",
                "search_analyzer": "ik_smart"
            }
        }

}'

ik_max_word:尽可能多的分词
ik_smart:自行理解

分词设置成功

4.测试环节

构造数据
curl -XPOST http://localhost:9200/index/_create/1 -H 'Content-Type:application/json' -d'
{"content":"美国留给伊拉克的是个烂摊子吗"}
'
curl -XPOST http://localhost:9200/index/_create/2 -H 'Content-Type:application/json' -d'
{"content":"公安部：各地校车将享最高路权"}
'
curl -XPOST http://localhost:9200/index/_create/3 -H 'Content-Type:application/json' -d'
{"content":"中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"}
'
curl -XPOST http://localhost:9200/index/_create/4 -H 'Content-Type:application/json' -d'
{"content":"中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"}
'
查询
curl -XPOST http://localhost:9200/index/_search  -H 'Content-Type:application/json' -d'
{
    "query" : { "match" : { "content" : "中国" }},
    "highlight" : {
        "pre_tags" : ["<tag1>", "<tag2>"],
        "post_tags" : ["</tag1>", "</tag2>"],
        "fields" : {
            "content" : {}
        }
    }
}
'
结果
{
    "took": 14,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 2,
        "max_score": 2,
        "hits": [
            {
                "_index": "index",
                "_type": "fulltext",
                "_id": "4",
                "_score": 2,
                "_source": {
                    "content": "中国驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首"
                },
                "highlight": {
                    "content": [
                        "<tag1>中国</tag1>驻洛杉矶领事馆遭亚裔男子枪击 嫌犯已自首 "
                    ]
                }
            },
            {
                "_index": "index",
                "_type": "fulltext",
                "_id": "3",
                "_score": 2,
                "_source": {
                    "content": "中韩渔警冲突调查：韩警平均每天扣1艘中国渔船"
                },
                "highlight": {
                    "content": [
                        "均每天扣1艘<tag1>中国</tag1>渔船 "
                    ]
                }
            }
        ]
    }
}

数据构造参考自

https://github.com/medcl/elasticsearch-analysis-ik

哦还有一个kibana

kibana

PART2

es+filebeat+kibana搭建分布日志监控

先上docker-compose.yml文件

version: '3.1'

services:
  es1:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: es1
    ports:
      - "9200:9200"
      - "9300:9300"
    networks:
      - docker-network
    volumes:
      - ./plugins:/usr/share/elasticsearch/plugins/analysis-ik
    #      - /home/xhwh/docker/EFK/data/es1/data:/usr/share/elasticsearch/data
    environment:
      - node.name=es1
      - discovery.seed_hosts=es1
      - cluster.initial_master_nodes=es1
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
  es2:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: es2
    networks:
      - docker-network
    environment:
      - node.name=es2
      - discovery.seed_hosts=es1
      - cluster.initial_master_nodes=es1
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - ./plugins:/usr/share/elasticsearch/plugins/analysis-ik
    #      - /home/xhwh/docker/EFK/data/es2/data:/usr/share/elasticsearch/data
    ulimits:
      memlock:
        soft: -1
        hard: -1
  es3:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: es3
    networks:
      - docker-network
    environment:
      - node.name=es3
      - discovery.seed_hosts=es1
      - cluster.initial_master_nodes=es1
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - ./plugins:/usr/share/elasticsearch/plugins/analysis-ik
    #      - /home/xhwh/docker/EFK/data/es3/data:/usr/share/elasticsearch/data
    ulimits:
      memlock:
        soft: -1
        hard: -1

  filebeat:
    image: docker.elastic.co/beats/filebeat:7.3.2
    container_name: filebeat
    volumes:
      - ./logs/strategy.log:/usr/share/strategy_log/strategy.log
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml
    links:
      - es1:elasticsearch
      - kibana:kibana
    networks:
      - docker-network
    depends_on:
      - es1
      - es2
      - es3
      - kibana

  filebeat-test:
    image: docker.elastic.co/beats/filebeat:7.3.2
    container_name: filebeat-test
    volumes:
      - ./logs:/usr/share/logs-test
      - ./filebeat_test.yml:/usr/share/filebeat/filebeat.yml
    links:
      - es1:elasticsearch
      - kibana:kibana
    networks:
      - docker-network
    depends_on:
      - es1
      - es2
      - kibana

  kibana:
    image: docker.elastic.co/kibana/kibana:7.3.2
    container_name: kibana
    environment:
      - I18N_LOCALE=zh-CN #汉化
    ports:
      - "5601:5601"
    links:
      - es1:elasticsearch
    depends_on:
      - es1
      - es2
      - es3
    networks:
      - docker-network
networks:
  docker-network:
    driver: bridge

在docker-compose中挂载好对应的目录,这里的./logs内部的日志作者使用的是springboot+log4j生成的日志

下面贴一下filebeat示例，这里配置2个日志源一个指向strategy索引一个指向test索引

filebeat.yml

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /usr/share/strategy_log/strategy.log
    multiline:
      pattern: '^\s*(\d{4}|\d{2})\-(\d{2}|[a-zA-Z]{3})\-(\d{2}|\d{4})'
      negate: true
      match: after
      timeout: 15s
      max_lines: 500

setup.kibana:
  host: "kibana:5601"

setup.ilm.enabled: false
setup.template.name: "strategy"
setup.template.pattern: "strategy-*"
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "strategy-%{+yyyy.MM.dd}" #指定index name

filebeat-test.yml

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /usr/share/logs-test/*.log
    multiline:
      pattern: '^\s*(\d{4}|\d{2})\-(\d{2}|[a-zA-Z]{3})\-(\d{2}|\d{4})'
      negate: true
      match: after
      timeout: 15s
      max_lines: 500

setup.kibana:
  host: "kibana:5601"

setup.ilm.enabled: false
setup.template.name: "test"
setup.template.pattern: "test-*"
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  index: "test-%{+yyyy.MM.dd}"

新增了filebeat服务和对应的yml配置文件

    volumes:
      - ./logs:/usr/share/logs-test
      - ./filebeat_test.yml:/usr/share/filebeat/filebeat.yml

使用docker-compose up启动成功后

打开kibana

选择日志模块，点击更改数据源配置

将日志索引改为test-*

这里将日志索引改为test-*或者strategy-*就是前面配置的两个索引
点击更新源

可以看到日志已经上去了，也可以查看es索引直接看到
这里我们修改一下日志文件

image.png

1.对应目录新增一个test.log文件
2.写入以下内容
3.kibana查看

接着我们切换以下数据源查看以下strategy日志

因为配置strategy这个日志的时候我们只推了strategy.log这个文件
而test这个日志推了整个目录，所以strategy日志是没有多余的test.log日志的

下次再见