prometheus + alertmanager + blackbox_exporter 实现应用监控并发送告警邮件

1. 基本信息

2. 安装
2.1 blackbox_exporter 安装

$ pwd
/alidata1/admin/tools/exporter/blackbox$ ls  ## 配置文件blackbox.yml  使用默认即可
blackbox_exporter  blackbox.yml  LICENSE  NOTICE## 配置开机自启文件
$ sudo cat /etc/systemd/system/blackbox_exporter.service
[Unit]
Description=BlackBox Exporter[Service]
User=admin
Group=admin
ExecStart=/alidata1/admin/tools/exporter/blackbox/blackbox_exporter --config.file=/alidata1/admin/tools/exporter/blackbox/blackbox.yml --web.listen-address=192.168.242.205:20011 --log.level=warn 
Restart=on-failure[Install]
WantedBy=multi-user.target## 启动并配置开机自启
$ sudo systemctl enable --now blackbox_exporter 

2.2 prometheus安装
安装过程见 Prometheus安装 , 此处不再安装, 只是配置需要做些改动

## 目录结构
$ pwd
/alidata1/admin/tools/prometheus-2.37.5
$ $ ll
总用量 206448
drwxr-xr-x 2 admin admin        22 6月   6 17:02 auth
drwxrwxr-x 2 admin admin        90 6月   7 14:44 conf
drwxr-xr-x 2 admin admin        38 12月  9 21:06 console_libraries
drwxr-xr-x 2 admin admin       173 12月  9 21:06 consoles
-rw-r--r-- 1 admin admin     11357 12月  9 21:06 LICENSE
-rw-r--r-- 1 admin admin      3773 12月  9 21:06 NOTICE
-rwxr-xr-x 1 admin admin 109779661 12月  9 20:49 prometheus
-rw-r--r-- 1 admin admin      2173 6月   7 11:44 prometheus.yml
-rwxr-xr-x 1 admin admin 101601052 12月  9 20:52 promtool
drwxrwxr-x 2 admin admin        30 6月   7 14:33 rules## 1. 在配置文件prometheus.yml 中追加关于blackbox的配置
......
## 应用状态- job_name: services_statusmetrics_path: /probefile_sd_configs:- files: ['./conf/blackbox_service_status.yml']params:module: [tcp_connect]relabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- replacement: "192.168.242.205:20011"target_label: __address__## 2. 在配置文件prometheus.yml中新增告警配置
# Alertmanager configuration     ## 将告警打开
alerting:alertmanagers:- static_configs:- targets:- 192.168.242.205:9093   rule_files:    ## 告警配置文件- "rules/*.yml"## 3. 新增配置文件 ./conf/blackbox_service_status.yml 里面是关于应用的配置
$ cat ./conf/blackbox_service_status.yml
- targets: ['192.168.242.205:9090']labels: {appName: frontend, env: prd }
- targets: ['192.168.84.56:8635', '192.168.242.205:9091']labels: {appName: backend, env: prd }## 4. 新增告警规则配置文件 rules/blackbox_service.yml
groups:
- name: appstatusrules:- alert: 服务已停止expr: probe_success{job=~"services_status"} == 0 for: 1mlabels:severity: critical annotations:summery: "当前应用 {{ $labels.appName }} : {{ $labels.instance }} 服务已停止, 请尽快处理"## 5. 重新加载prometheus配置
$ curl -XPUT http://192.168.242.205:9090/-/reload

2.3 alertmanager安装

$ pwd
/alidata1/admin/tools/alertmanager-0.25.0
$ ls
alertmanager  alertmanager.yml  amtool  LICENSE  NOTICE  template## 1. 查看配置文件alertmanager.yml
global:resolve_timeout: 5msmtp_smarthost: "smtp.qq.com:465"smtp_from: "939545179@qq.com"smtp_auth_username: "939545179@qq.com"smtp_auth_password: "xxxxxxx"    ## qq邮箱授权码smtp_require_tls: falsetemplates:- './template/*.tmpl'route:group_by: ['Alert']group_wait: 10sgroup_interval: 10srepeat_interval: 5mreceiver: 'mail'routes:- receiver: 'mail'match_re:severity: critical|warning receivers:
- name: "mail"email_configs:- to: '123456789@163.com'html: '{{ template "email.html" . }}'headers: { Subject: '{{ .CommonLabels.appName }}: {{ .CommonLabels.alertname }} ' }## 2. 邮件内容配置
$ cat template/mail.tmpl 
{{ define "email.html" }}
{{ range .Alerts }}<pre>
<strong>应用名:</strong> {{ .Labels.appName }}
<strong>环境  :</strong> {{ .Labels.env }}
<strong>实例  :</strong> {{ if gt (len .Labels.instance) 0 -}} {{ .Labels.instance }} {{ else }}  grafana {{ end }}
<strong>信息  :</strong> {{ .Annotations.summery }}
<strong>时间  :</strong> {{ (.StartsAt.Add 28800e9).Format "2006-01-02 15:04:05" }}</pre>
{{ end }}
{{ end }}## 3.配置开机自启
$ cat /etc/systemd/system/alertmanager.service
[Unit]
Description=AlertManager Service
After=network.target[Service]
User=admin
Group=admin
ExecStart=/alidata1/admin/tools/alertmanager-0.25.0/alertmanager --config.file=/alidata1/admin/tools/alertmanager-0.25.0/alertmanager.yml --storage.path=/alidata1/admin/data/alertmanager --web.listen-address=192.168.242.205:9093 --log.level=warn
Restart=on-failure
RestartSec=10[Install]
WantedBy=multi-user.target$ sudo systemctl enable alertmanager --now

3. 验证