Cless в 18:14, 31 мая 2023

2023-05-31T18:14:55Z

← Предыдущая		Версия 18:14, 31 мая 2023
Строка 1:		Строка 1:
	[[Категория:Prometheus]]		[[Категория:Prometheus]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

			= Configuring =
			Add Recording Rules. They are the kind of templates for big calculating values.

			<code>nano /etc/prometheus/prometheus_rules.yml</code>
			groups:
			- name: custom_rules
			rules:
			- record: node_cpuUsedTotal_percent
			expr: 100 - ((rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

			- record: node_memory_MemFree_percent
			expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

			- record: node_memory_MemUsed_percent
			expr: (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

			- record: node_filesystem_free_percent
			expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
			Add alert rules for each event.

			<code>nano /etc/prometheus/alert_rules.yml</code>
			groups:

			- name: alert_rules
			<nowiki> </nowiki> rules:
			<nowiki> </nowiki> - alert: InstanceDown
			<nowiki> </nowiki> expr: up == 0
			<nowiki> </nowiki> for: 1m
			<nowiki> </nowiki> labels:
			<nowiki> </nowiki> severity: critical
			<nowiki> </nowiki> annotations:
			<nowiki> </nowiki> summary: "Instance <nowiki>[{{ $labels.instance }}]</nowiki> down"
			<nowiki> </nowiki> description: "<nowiki>[{{ $labels.instance }}]</nowiki> of job <nowiki>[{{ $labels.job }}]</nowiki> has been down for more than 1 minute."

			<nowiki> </nowiki> - alert: NodeCPUUsage
			<nowiki> </nowiki> expr: node_cpuUsedTotal_percent > 30
			<nowiki> </nowiki> for: 1m
			<nowiki> </nowiki> labels:
			<nowiki> </nowiki> severity: critical
			<nowiki> </nowiki> annotations:
			<nowiki> </nowiki> summary: High CPU usage detected CPU usage is above 30%
			<code>systemctl restart prometheus</code>

			Download and install Alertmanager service.

			<code>wget <nowiki>https://github.com/prometheus/alertmanager/releases/download/v0.25.0/alertmanager-0.25.0.linux-amd64.tar.gz</nowiki></code>

			<code>tar -xzf alertmanager-0.25.0.linux-amd64.tar.gz</code>

			<code>cd alertmanager-0.25.0.linux-amd64</code>

			<code>mkdir /etc/alertmanager</code>

			<code>cp alertmanager amtool /usr/local/bin/</code>

			<code>cp alertmanager.yml /etc/alertmanager/</code>

			<code>mv /etc/alertmanager/alertmanager.yml /etc/alertmanager/alertmanager.yml.ORIG</code>

			Add alertmanager rules. In this example we are going to push notifications via Telegram.

			<code>nano /etc/alertmanager/alertmanager.yml</code>
			global:
			resolve_timeout: 5m

			route:
			group_by: ['alertname']
			group_wait: 10s
			group_interval: 10s
			repeat_interval: 1m
			receiver: telegram

			receivers:
			- name: 'telegram'
			telegram_configs:
			- bot_token: 46*71:AAG*YCY # bot_token
			<nowiki> </nowiki> api_url: <nowiki>https://api.telegram.org</nowiki>
			<nowiki> </nowiki> chat_id: 25***73 # chat_id
			<nowiki> </nowiki> parse_mode: <nowiki>''</nowiki>

			inhibit_rules:
			- source_match:
			<nowiki> </nowiki> severity: 'critical'
			<nowiki> </nowiki> target_match:
			<nowiki> </nowiki> severity: 'warning'
			<nowiki> </nowiki> equal: ['alertname', 'dev', 'instance']
			Test alertmanager rules

			<code>amtool check-config /etc/alertmanager/alertmanager.yml</code>

			Add alertmanager service

			<code>nano /etc/systemd/system/alertmanager.service</code>
			[Unit]
			Description=Prometheus Alert Manager Service
			After=network.target

			[Service]
			Type=simple
			ExecStart=/usr/local/bin/alertmanager \
			--config.file=/etc/alertmanager/alertmanager.yml

			[Install]
			WantedBy=multi-user.target
			<code>systemctl daemon-reload</code>

			<code>systemctl restart alertmanager</code>

			Add rule files into prometheus config.

			<code>nano /etc/prometheus/prometheus.yml</code>
			...
			rule_files:
			...
			- "prometheus_rules.yml"
			- "alert_rules.yml"

			...
			alerting:
			alertmanagers:
			- static_configs:
			- targets:
			- localhost:9093
			...
			<code>systemctl restart prometheus</code>

			Ok. Now, when some Instance (monitored server) will down or node_cpuUsedTotal_percent will greater than 30%, the trigger will be triggered and notification will come to Telegram.

Cless: Новая страница: «Категория:Prometheus __ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__»

2023-05-31T18:00:53Z

Новая страница: «Категория:Prometheus __ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__»

Новая страница

[[Категория:Prometheus]]
__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

Prometheus. Alerts using Alertmanager - История изменений

Cless в 18:14, 31 мая 2023

Cless: Новая страница: «Категория:Prometheus __ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__»