Prometheus. Alerts using Alertmanager

Configuring

Add Recording Rules. They are the kind of templates for big calculating values.

nano /etc/prometheus/prometheus_rules.yml

groups:
  - name: custom_rules
    rules:
      - record: node_cpuUsedTotal_percent
        expr: 100 - ((rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

      - record: node_memory_MemFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

      - record: node_memory_MemUsed_percent
        expr: (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}

Add alert rules for each event.

nano /etc/prometheus/alert_rules.yml

groups:

  - name: alert_rules
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance [{{ $labels.instance }}] down"
          description: "[{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 1 minute."

      - alert: NodeCPUUsage
        expr: node_cpuUsedTotal_percent > 30
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: High CPU usage detected CPU usage is above 30%

systemctl restart prometheus

Download and install Alertmanager service.

wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz

tar -xzf alertmanager-0.25.0.linux-amd64.tar.gz

cd alertmanager-0.25.0.linux-amd64

mkdir /etc/alertmanager

cp alertmanager amtool /usr/local/bin/

cp alertmanager.yml /etc/alertmanager/

mv /etc/alertmanager/alertmanager.yml /etc/alertmanager/alertmanager.yml.ORIG

Add alertmanager rules. In this example we are going to push notifications via Telegram.

nano /etc/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1m
  receiver: telegram

receivers:
- name: 'telegram'
  telegram_configs:
  - bot_token: 46***71:AAG***YCY # bot_token
    api_url: https://api.telegram.org
    chat_id: 25***73 # chat_id
    parse_mode: ''

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Test alertmanager rules

amtool check-config /etc/alertmanager/alertmanager.yml

Add alertmanager service

nano /etc/systemd/system/alertmanager.service

[Unit]
Description=Prometheus Alert Manager Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/alertmanager \
        --config.file=/etc/alertmanager/alertmanager.yml

[Install]
WantedBy=multi-user.target

systemctl daemon-reload

systemctl restart alertmanager

Add rule files into prometheus config.

nano /etc/prometheus/prometheus.yml

...
rule_files:
...
  - "prometheus_rules.yml"
  - "alert_rules.yml"

...
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
...

systemctl restart prometheus

Ok. Now, when some Instance (monitored server) will down or node_cpuUsedTotal_percent will greater than 30%, the trigger will be triggered and notification will come to Telegram.