elk结合beats收集日志

elk结合beats收集日志

beats简介

https://www.elastic.co/guide/en/beats/libbeat/current/beats-reference.html#beats-reference

​ beats为一系列轻量级的数据采集工具,可以直接汇总数据到es,或经由logstash进一步处理,常用的有filebeat,auditbeat,packetbeat

image-20201214170712331

elk结合filebeat

节点架构

与elk架构相比,efk架构只是将最初的采集端从logstash换成了专用的beats系列,如filebeat,

image-20201214174124980

安装filebeat

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.8.1-x86_64.rpm
rpm -ivh filebeat-6.8.1-x86_64.rpm 

filebeat 收集日志写入到kafka

修改filebeat配置,为yaml格式,读取本地的系统日志,然后写入到kafka节点;



读取本地的系统日志
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/messages
  fields:
    host: "192.168.80.105"
    type: "filebeat-syslog-80-105"
    app: "syslog"

输出到kafka
output.kafka:
 hosts: ["192.168.80.106:9092","192.168.80.107:9092"]
 topic: "filebeat-syslog-80-105"
 partition.round_robin:
  reachable_only: true
 required_acks: 1  # 本地写入完成
 compression: gzip  # 开启压缩
 max_message_bytes: 1000000  # 消息最大值

[root@logstash-105 ~]# systemctl start filebeat
[root@logstash-105 ~]# systemctl status filebeat

kafka查看是否写入了响应的topic

[root@es2 ~]# /usr/local/kafka/bin/kafka-topics.sh --zookeeper 192.168.80.107:2181 --list
filebeat-syslog-80-105
hello

logstash从kafka读取数据写入es

编辑logstash108的配置文件,从kafka读取数据,写入到es中;

[root@logstash-108 conf.d]# cat from-kafka.conf 
input {

	kafka {
		topics => "filebeat-syslog-80-105"
		bootstrap_servers => "192.168.80.107:9092"
		codec => "json"
	}
}


output {
	if [fields][app] == "syslog" {
		elasticsearch {
			hosts => ["192.168.80.107:9200"]
			index => "filebeat-syslog-80-105-%{+YYYY.MM.dd}"
		}
	}		

}

语法测试,启动,启动无报错,但kibana界面查不到该索引

[root@logstash-108 conf.d]# logstash -f from-kafka.conf


[INFO ] 2020-12-14 19:02:37.180 [Ruby-0-Thread-11: :1] ConsumerCoordinator - [Consumer clientId=logstash-0, groupId=logstash] Revoking previously assigned partitions []
[INFO ] 2020-12-14 19:02:37.181 [Ruby-0-Thread-11: :1] AbstractCoordinator - [Consumer clientId=logstash-0, groupId=logstash] (Re-)joining group
[INFO ] 2020-12-14 19:02:37.284 [Ruby-0-Thread-11: :1] AbstractCoordinator - [Consumer clientId=logstash-0, groupId=logstash] Successfully joined group with generation 1
[INFO ] 2020-12-14 19:02:37.287 [Ruby-0-Thread-11: :1] ConsumerCoordinator - [Consumer clientId=logstash-0, groupId=logstash] Setting newly assigned partitions [filebeat-syslog-80-105-0]
[INFO ] 2020-12-14 19:02:37.316 [Ruby-0-Thread-11: :1] Fetcher - [Consumer clientId=logstash-0, groupId=logstash] Resetting offset for partition filebeat-syslog-80-105-0 to offset 3777.

es中查不到此索引

[root@es2 ~]# curl 'localhost:9200/_cat/indices?v'
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   messagelog-2020.12.11       71PHxMv1T56xqEAsEJ95Vg   5   1          2            0     28.2kb         14.1kb
green  open   tcp-log-2020.12.12          YZ4nkU3WSRyxr_XRmhP7QA   5   1          2            0     25.6kb         12.8kb
green  open   messagelog-7-100-2020.12.11 Al2uqt1wRe2gHWEQuOkB5w   5   1        419            0    593.8kb        296.8kb
green  open   .kibana_1                   rj1dL_VMTYSTIeg2Pvx3oA   1   1         28            2    242.3kb        124.1kb
green  open   nginx2020.12.11             g2AHe4WoSEaCNcwY4-p1bg   5   1          5            0     67.8kb         33.9kb
green  open   nginx-access-log-2020.12.12 BBNARQCuQl62Ad-HlAJC6A   5   1         11            0    116.5kb         58.2kb
green  open   rsyslog-80-106-2020.12.12   8HLfzyvIQDaBDHflK1fUBw   5   1          6            0      105kb         52.5kb
green  open   .kibana_task_manager        9ckx4q-3ROWMxev8mQHcwA   1   1          2            0       15kb          7.4kb
green  open   linux-test-2020.12.11       uzzSUWISQZ6ucwn8TGkDXQ   5   1          1            0     11.9kb          5.9kb
green  open   tomcat-log-2020.12.12       GAk7ia6iSA-b5cHcRLYa4A   5   1          8            0    100.3kb         50.1kb
green  open   mes-105-2020.12.13          m0PRk3BdTHit228H-I0tEg   5   1      13050            0      6.2mb          3.1mb
green  open   logstash-log-2020.12.12     wj_XsXvGR4qP9PZrE7e7cw   5   1         21            0     71.9kb         35.9kb

自然,kibana界面也无法创建索引

kibana创建索引

updatedupdated2020-12-172020-12-17
加载评论