filebeat-daemonSet模式采集pod日志

1、准备filebeat包

filebeat-8.5.3-linux-x86_64.tar.gz

2、新增Dockerfile

将filebeat包放入容器/root目录下

FROM tomcat:2.0
ADD filebeat-8.5.3-linux-x86_64.tar.gz /root
RUN cd /root/filebeat-8.5.3-linux-x86_64
WORKDIR /root/filebeat-8.5.3-linux-x86_64
CMD ["./filebeat" ,"-e", "-c", "filebeat.yml"]

3、新增filebeat.yml文件

根据自己的服务配置加入以下内容，示例为采集服务日志到kafka

#filebeat自身日志配置
logging.level: info
logging.to_files: true
logging.files:path: /root/filebeat-8.5.3-linux-x86_64/logname: filebeatkeepfiles: 7permissions: 0644filebeat.inputs:
- type: logenabled: truepaths:- /data/docker/containers/19ee9c5ec9a9f8799f5d770cce23b1dea6c806ece1177d8fb1535fd049bb7afc/*.logfields:app_deployment_id: 759bfe5b-a7d3-9c4c-fb7e-ab41fde3abb8agent_id: application_name: app_name: KuberneteServicetopic: app-logfields_under_root: truetail_files: true
# -----------------------------  kafka  ---------------------------------------
output.kafka:hosts: ["134.64.15.155:9092"]topic: filebeat-container-logpartition.round_robin:reachable_only: trueprocessors:- add_host_metadata: ~logging.level: debug

4、新增start.sh启动脚本

根据自己的服务加入以下内容：
注意：一定要保证除最后一个服务外，其他服务必须后台运行；最后一个服务必须前台运行；如果都在后台运行的话，docker会以为程序都运行结束，自动退出。

#!/bin/bash
cd /root/KuberneteService/bin/
nohup sh KuberneteService >> nohup.out 2>&1 &
cd /root/filebeat-8.5.3-linux-x86_64/
./filebeat -e -c filebeat.yml

5、构建镜像+运行

准备好的环境如下：

1. 构建镜像

docker build -t filebeat:1.0 .

2. 运行镜像
   docker run -d --name filebeat -v /data/docker/containers:/data/docker/containers -v /app/tool/k8s/filebeatDocker/log:/root/filebeat-8.5.3-linux-x86_64/log -v /app/tool/k8s/filebeatDocker/filebeat.yml:/root/filebeat-8.5.3-linux-x86_64/filebeat.yml filebeat:1.0

3. 查看镜像是否启动成功
   docker ps -a
   

4. 查看容器日志
   docker logs 容器id -f

5. 进入容器内部，查看部署情况
   docker exec -it 容器id bash
   

6. 采集到的消息格式：

{"@timestamp": "2022-12-16T03:33:29.264Z","@metadata": {"beat": "filebeat","type": "_doc","version": "8.5.3"},"topic": "app-log","ecs": {"version": "8.0.0"},"message": "{\"log\":\"I1209 08:20:05.294990       1 shared_informer.go:247] Caches are synced for endpoint slice config \\n\",\"stream\":\"stderr\",\"time\":\"2022-12-09T08:20:05.295139982Z\"}","log": {"offset": 8873,"file": {"path": "/data/docker/containers/19ee9c5ec9a9f8799f5d770cce23b1dea6c806ece1177d8fb1535fd049bb7afc/19ee9c5ec9a9f8799f5d770cce23b1dea6c806ece1177d8fb1535fd049bb7afc-json.log"}},"input": {"type": "log"},"app_name": "KuberneteService","app_deployment_id": "759bfe5b-a7d3-9c4c-fb7e-ab41fde3abb8","host": {"architecture": "x86_64","os": {"codename": "focal","type": "linux","platform": "ubuntu","version": "20.04.5 LTS (Focal Fossa)","family": "debian","name": "Ubuntu","kernel": "3.10.0-957.el7.x86_64"},"containerized": true,"name": "6cd4de6fbf07","ip": ["172.17.0.4"],"mac": ["02-42-AC-11-00-04"],"hostname": "6cd4de6fbf07"},"agent": {"type": "filebeat","version": "8.5.3","ephemeral_id": "77432577-0e75-4d60-8582-ced5447979a3","id": "f9c219df-d1b2-44aa-ba9b-1b34e316382a","name": "6cd4de6fbf07"}
}