Beats轻量级日志采集工具

Beats 平台集合了多种单一用途数据采集器。这些采集器安装后可用作轻量型代理，从成百上千或成千上万台机器向 Logstash 或 Elasticsearch 发送数据。常用的Beats有Filebeat(收集文件)、Metricbeat(收集服务、系统的指标数据)、Packetbeat(收集网络包)等。这里主要介绍Filebeat插件。

一、架构图

二、安装Filebeat

官网地址： https://www.elastic.co/cn/products/beats

1、下载并安装Filebeat

wget  https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.2-linux-x86_64.tar.gz
tar -xzf filebeat-6.3.2-linux-x86_64.tar.gz -C /usr/local/
cd /usr/local/
ln -s filebeat-6.3.2-linux-x86_64 filebeat

2、自定义配置文件
① 简单版本的配置文件

cd /usr/local/filebeat/
cat > test.yml << END
filebeat.inputs:
- type: stdinenabled: true
setup.template.settings:index.number_of_shards: 3
output.console:pretty: trueenable: trueEND#启动filebeat，启动filebeat的时候用户需要用filebeat用户或者root用户
./filebeat -e -c test.yml#测试
启动好后输入任意字符串，如hello，即可输出对应信息。#启动参数说明：./filebeat -e -c test.yml
-e：输出到标准输出，默认输出到syslog和logs下
-c：指定配置文件

②收集日志文件

cd /usr/local/filebeat/
cat > test.yml << END
filebeat.inputs:
- type: logenabled: truepaths:- /var/log/*.log- /var/log/messagesexclude_lines: ['^DBG',"^$",".gz$"]
setup.template.settings:index.number_of_shards: 3
output.console:pretty: trueenable: true
END#启动filebeat
./filebeat -e -c test.yml

③自定义字段收集日志文件

cd /usr/local/filebeat/
cat > test.yml << END
filebeat.inputs:
- type: logenabled: truepaths:- /var/log/*.log- /var/log/messagesexclude_lines: ['^DBG',"^$",".gz$"]tags: ["web","item"]       #自定义tagsfields:                    #添加自定义字段from: itcast_from        #值随便写fields_under_root: true    #true为添加到根节点中，false为添加到子节点中
setup.template.settings:index.number_of_shards: 3
output.console:pretty: trueenable: true
END#启动filebeat
./filebeat -e -c test.yml#如果有tags字段在logstash中的书写格式
if "web" in [tags] {  }

④收集nginx日志文件输出到ES或者logstash中

cd /usr/local/filebeat/
cat > nginx.yml << END
filebeat.inputs:
- type: logenabled: truepaths:- /usr/local/nginx/access/*.logexclude_lines: ['^DBG',"^$",".gz$"]document_type: filebeat-nginx_accesslogtags: ["web","nginx"] fields:from: nginx fields_under_root: true 
setup.template.settings:index.number_of_shards: 3
output.elasticsearch:hosts: ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]
#output.logstash:
#  hosts: ["192.168.0.117:5044"]
END#启动filebeat
./filebeat -e -c nginx.yml

三、Filebeat收集各个日志到logstash，然后由logstash将日志写到redis，然后再写入到ES

1、filebeat配置文件

cat > dashboard.yml << END
filebeat.inputs:
- input_type: logpaths:- /var/log/*.log- /var/log/messagesexclude_lines: ['^DBG',"^$",".gz$"]document_type: filebeat-systemlog
- input_type: logpaths:- /usr/local/tomcat/logs/tomcat_access_log.*.logexclude_lines: ['^DBG',"^$",".gz$"]document_type: filebeat-tomcat-accesslogmultiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'multiline.negate: truemultiline.match: after- type: logenabled: truepaths:- /usr/local/nginx/access/*.logexclude_lines: ['^DBG',"^$",".gz$"]document_type: filebeat-nginx-accesslog
output.logstash:hosts: ["192.168.0.117:5044"]enabled: trueworker: 3compression_level: 3END##启动
./filebeat -e -c dashboard.yml

2、logstash配置文件
①将beats收集的日志写入到logstash中

cat > beats.conf << END
input {beats {port => "5044"#host => "192.168.0.117"}
}
output {if [type] == "filebeat-systemlog" {redis {data_type => "list"host => "192.168.0.119"db => "3"port => "6379"password => "123456"key => "filebeat-systemlog"}}if [type] == "filebeat-tomcat-accesslog" {redis {data_type => "list"host => "192.168.0.119"db => "3"port => "6379"password => "123456"key => "filebeat-tomcat-accesslog"}}if [type] == "filebeat-nginx-accesslog" {redis {data_type => "list"host => "192.168.0.119"db => "3"port => "6379"password => "123456"key => "filebeat-nginx-accesslog"}}
}
END

②从redis中读取日志写入ES

cat > redis-es.conf << END
input {redis {data_type => "list"host => "192.168.0.119"db => "3"port => "6379"password => "123456"key => "filebeat-systemlog"type => "filebeat-systemlog"}redis {data_type => "list"host => "192.168.0.119"db => "3"port => "6379"password => "123456"key => "filebeat-tomcat-accesslog"type => "filebeat-tomcat-accesslog"}redis {data_type => "list"host => "192.168.0.119"db => "3"port => "6379"password => "123456"key => "filebeat-nginx-accesslog"type => "filebeat-nginx-accesslog"}
}output {if [type] == "filebeat-systemlog" {elasticsearch {hosts => ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]index => "logstash-systemlog-%{+YYYY.MM.dd}"}}if [type] == "filebeat-tomcat-accesslog" {elasticsearch {hosts => ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]index => "logstash-tomcat-accesslog-%{+YYYY.MM.dd}"}}if [type] == "filebeat-nginx-accesslog" {elasticsearch {hosts => ["192.168.0.117:9200","192.168.0.118:9200","192.168.0.119:9200"]index => "logstash-nginx-accesslog-%{+YYYY.MM.dd}"}}
}
END

备注：使用negate: true和match: after设置来指定任何不符合指定模式的行都属于上一行。更多多行匹配配置请参考

四、通过kibana展示即可