prometheus_blackbox（五、黑盒监测 ）

参考：基于blackbox_exporter实现对URL状态、IP可用性、端口状态、TLS证书的过期时间监控
Blackbox_exporter黑盒监测

一、blackbox_exporter介绍

blackbox_exporter 是 Prometheus 官方提供的一个 exporter， 可以监控 HTTP、HTTPS、DNS、TCP 、ICMP 等目标实例， 从而实现对被监控节点进行监控和数据采集。
HTTP/HTPPS： URL/API 可用性检测
TCP： 端口监听检测
ICMP： 主机存活检测
DNS： 域名解析

二、安装部署blackbox_exporter

https://github.com/prometheus/blackbox_exporter/releases/

cd /opt
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.22.0/blackbox_exporter-0.22.0.linux-amd64.tar.gz
tar -zxvf blackbox_exporter-0.22.0.linux-amd64.tar.gz
mv blackbox_exporter-0.22.0.linux-amd64 blackbox_exporter

vim /etc/systemd/system/blackbox-exporter.service
[Unit]
Description=Prometheus Blackbox Exporter
After=network.target
[Service]
Type=simple
User=root
Group=root
ExecStart=/opt/blackbox_exporter/blackbox_exporter \
--config.file=/opt/blackbox_exporter/blackbox.yml \
--web.listen-address=:9115
Restart=on-failure
[Install]
WantedBy=multi-user.target

systemctl enable --now blackbox-exporter.service
systemctl start blackbox-exporter.service
systemctl status blackbox-exporter.service
netstat -tnlp
#放通防火墙
firewall-cmd --zone=public --add-port=9115/tcp --permanent
firewall-cmd --reload
验证是否安装成功
cd /opt/blackbox_exporter/
./blackbox_exporter --version

查看 http://192.168.1.2:9115/

三、配置启用blackbox exporter

1.使用blackbox exporter 实现 URL 监控
配置Prometheus

[root@monitoring prometheus]# vim prometheus.yml
[root@monitoring prometheus]# grep http_status  -A10  prometheus.yml
  - job_name: 'http_status'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['http://www.xiaomi.com', 'http://www.magedu.com']
        labels:
          instance: http_status
          group: web
    relabel_configs:
      - source_labels: [__address__] # 将__address__(当前监控目标URL地址的标签)修改为__param_target,用于传递给blackbox_exporter
        target_label: __param_target #标签key为__param_target、value为www.xiaomi.mkey为__param_target、value为www.magedu.com
      - source_labels: [__param_target] #基于__param_target获取监控目标
        target_label: url #将监控目标的值与 url 创建一个label
      - target_label: __address__ #新添加一个目标__address__,指向blackbox_exporter服务器地址,用于将监控请求发送给指定的blackbox_exporter服务器
        replacement: 192.168.1.2:9115 #指定blackbox_exporter服务器地址

API Serevr节点发现

#检查配置文件是否书写正确
[root@monitoring prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax
#重启prometheus
[root@monitoring prometheus]# systemctl restart prometheus.service

2.使用blackbox exporter 实现 ICMP 监控
ICMP就是Ping使用的协议，可以探测IP是否在线：

[root@monitoring prometheus]# vim prometheus.yml
[root@monitoring prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax
[root@monitoring prometheus]# grep ping_status  -A10  prometheus.yml
  - job_name: 'ping_status'
    metrics_path: /probe
    params:
     module: [icmp]
    static_configs:
    - targets: ['172.16.88.254',"223.6.6.6"]
      labels:
        instance: 'ping_status'
        group: 'icmp'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: ip
      - target_label: __address__
        replacement: 192.168.1.2:9115

API Serevr节点发现

#检查配置文件是否书写正确
[root@monitoring prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax
#重启prometheus
[root@monitoring prometheus]# systemctl restart prometheus.service

3.使用blackbox exporter 实现 端口监控

[root@monitoring prometheus]# vim prometheus.yml
[root@monitoring prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax
[root@monitoring prometheus]# grep port_status  -A10  prometheus.yml
  - job_name: 'port_status'
    metrics_path: /probe
    params:
     module: [tcp_connect]
    static_configs:
      - targets: ['192.168.1.2:51234', '192.168.1.2:9256','192.168.1.2:22']
        labels:
          instance: 'port_status'
          group: 'port'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: ip
      - target_label: __address__
        replacement: 192.168.1.2:9115

API Serevr节点发现

#检查配置文件是否书写正确
[root@monitoring prometheus]# ./promtool check config prometheus.yml
Checking prometheus.yml
 SUCCESS: prometheus.yml is valid prometheus config file syntax
#重启prometheus
[root@monitoring prometheus]# systemctl restart prometheus.service

访问Prometheus的 http://127.0.0.1:9090/targets 查看加入的监控信息。

四、grafana中加入blackbox_exporter监控数据

此模板为9965号模板，数据源选择Prometheus 模板下载地址 https://grafana.com/grafana/dashboards/9965

此模板需要安装饼状图插件 下载地址 https://grafana.com/grafana/plugins/grafana-piechart-panel
安装插件，重启grafana生效。

grafana-cli plugins install grafana-piechart-panel
service grafana-server restart

访问grafana http://127.0.0.1:3000，查看数据。

prometheus 告警规则

#以下为Blackbox-exporer通用告警模板，可根据label自行调整
blackbox.rules: |
    groups:
    - name: blackbox.rules
      rules:
      - alert: blackbox_network_stats
        expr: probe_success == 0
        for: 1m
        labels:
          team: ops
          severity: critical
        annotations:
          description: "接口/主机/路径/端口：{{ $labels.instance }} 无法连通"