Prometheus. Alerts using Alertmanager: различия между версиями

Материал из Wiki
Перейти к навигации Перейти к поиску
(Новая страница: «Категория:Prometheus __ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__»)
 
 
Строка 1: Строка 1:
[[Категория:Prometheus]]
[[Категория:Prometheus]]
__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__
__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__
= Configuring =
Add Recording Rules. They are the kind of templates for big calculating values.
<code>nano /etc/prometheus/prometheus_rules.yml</code>
groups:
  - name: custom_rules
     rules:
       - record: node_cpuUsedTotal_percent
         expr: 100 - ((rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)
       - record: node_memory_MemFree_percent
         expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
       - record: node_memory_MemUsed_percent
         expr: (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
       - record: node_filesystem_free_percent
         expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
Add alert rules for each event.
<code>nano /etc/prometheus/alert_rules.yml</code>
groups:
  - name: alert_rules
<nowiki> </nowiki>   rules:
<nowiki> </nowiki>     - alert: InstanceDown
<nowiki> </nowiki>       expr: up == 0
<nowiki> </nowiki>       for: 1m
<nowiki> </nowiki>       labels:
<nowiki> </nowiki>         severity: critical
<nowiki> </nowiki>       annotations:
<nowiki> </nowiki>         summary: "Instance <nowiki>[{{ $labels.instance }}]</nowiki> down"
<nowiki> </nowiki>         description: "<nowiki>[{{ $labels.instance }}]</nowiki> of job <nowiki>[{{ $labels.job }}]</nowiki> has been down for more than 1 minute."
<nowiki> </nowiki>     - alert: NodeCPUUsage
<nowiki> </nowiki>       expr: node_cpuUsedTotal_percent > 30
<nowiki> </nowiki>       for: 1m
<nowiki> </nowiki>       labels:
<nowiki> </nowiki>         severity: critical
<nowiki> </nowiki>       annotations:
<nowiki> </nowiki>         summary: High CPU usage detected CPU usage is above 30%
<code>systemctl restart prometheus</code>
Download and install Alertmanager service.
<code>wget <nowiki>https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz</nowiki></code>
<code>tar -xzf alertmanager-0.25.0.linux-amd64.tar.gz</code>
<code>cd alertmanager-0.25.0.linux-amd64</code>
<code>mkdir /etc/alertmanager</code>
<code>cp alertmanager amtool /usr/local/bin/</code>
<code>cp alertmanager.yml /etc/alertmanager/</code>
<code>mv /etc/alertmanager/alertmanager.yml /etc/alertmanager/alertmanager.yml.ORIG</code>
Add alertmanager rules. In this example we are going to push notifications via Telegram.
<code>nano /etc/alertmanager/alertmanager.yml</code>
global:
  resolve_timeout: 5m
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1m
  receiver: telegram
receivers:
- name: 'telegram'
  telegram_configs:
  - bot_token: 46***71:AAG***YCY # bot_token
<nowiki> </nowiki>   api_url: <nowiki>https://api.telegram.org</nowiki>
<nowiki> </nowiki>   chat_id: 25***73 # chat_id
<nowiki> </nowiki>   parse_mode: <nowiki>''</nowiki>
inhibit_rules:
  - source_match:
<nowiki> </nowiki>     severity: 'critical'
<nowiki> </nowiki>   target_match:
<nowiki> </nowiki>     severity: 'warning'
<nowiki> </nowiki>   equal: ['alertname', 'dev', 'instance']
Test alertmanager rules
<code>amtool check-config /etc/alertmanager/alertmanager.yml</code>
Add alertmanager service
<code>nano /etc/systemd/system/alertmanager.service</code>
[Unit]
Description=Prometheus Alert Manager Service
After=network.target
[Service]
Type=simple
ExecStart=/usr/local/bin/alertmanager \
         --config.file=/etc/alertmanager/alertmanager.yml
[Install]
WantedBy=multi-user.target
<code>systemctl daemon-reload</code>
<code>systemctl restart alertmanager</code>
Add rule files into prometheus config.
<code>nano /etc/prometheus/prometheus.yml</code>
...
rule_files:
...
  - "prometheus_rules.yml"
  - "alert_rules.yml"
...
alerting:
  alertmanagers:
  - static_configs:
     - targets:
       - localhost:9093
...
<code>systemctl restart prometheus</code>
Ok. Now, when some Instance (monitored server) will down or node_cpuUsedTotal_percent will greater than 30%, the trigger will be triggered and notification will come to Telegram.

Текущая версия на 21:14, 31 мая 2023


Configuring

Add Recording Rules. They are the kind of templates for big calculating values.

nano /etc/prometheus/prometheus_rules.yml 

groups:
  - name: custom_rules
    rules:
      - record: node_cpuUsedTotal_percent
        expr: 100 - ((rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

      - record: node_memory_MemFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

      - record: node_memory_MemUsed_percent
        expr: (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}

Add alert rules for each event.

nano /etc/prometheus/alert_rules.yml 

groups:

  - name: alert_rules
    rules:
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance [{{ $labels.instance }}] down"
          description: "[{{ $labels.instance }}] of job [{{ $labels.job }}] has been down for more than 1 minute."

      - alert: NodeCPUUsage
        expr: node_cpuUsedTotal_percent > 30
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: High CPU usage detected CPU usage is above 30%

systemctl restart prometheus

Download and install Alertmanager service.

wget https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz

tar -xzf alertmanager-0.25.0.linux-amd64.tar.gz

cd alertmanager-0.25.0.linux-amd64

mkdir /etc/alertmanager

cp alertmanager amtool /usr/local/bin/

cp alertmanager.yml /etc/alertmanager/

mv /etc/alertmanager/alertmanager.yml /etc/alertmanager/alertmanager.yml.ORIG

Add alertmanager rules. In this example we are going to push notifications via Telegram.

nano /etc/alertmanager/alertmanager.yml 

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1m
  receiver: telegram

receivers:
- name: 'telegram'
  telegram_configs:
  - bot_token: 46***71:AAG***YCY # bot_token
    api_url: https://api.telegram.org
    chat_id: 25***73 # chat_id
    parse_mode: ''

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Test alertmanager rules

amtool check-config /etc/alertmanager/alertmanager.yml

Add alertmanager service

nano /etc/systemd/system/alertmanager.service 

[Unit]
Description=Prometheus Alert Manager Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/alertmanager \
        --config.file=/etc/alertmanager/alertmanager.yml

[Install]
WantedBy=multi-user.target

systemctl daemon-reload

systemctl restart alertmanager

Add rule files into prometheus config.

nano /etc/prometheus/prometheus.yml 

...
rule_files:
...
  - "prometheus_rules.yml"
  - "alert_rules.yml"

...
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
...

systemctl restart prometheus

Ok. Now, when some Instance (monitored server) will down or node_cpuUsedTotal_percent will greater than 30%, the trigger will be triggered and notification will come to Telegram.

Источник — https://wiki.cless.pro/index.php?title=Prometheus._Alerts_using_Alertmanager&oldid=1556