Prometheus. Monitor Linux server using Prometheus - История изменений

Cless в 16:46, 29 мая 2023

2023-05-29T16:46:04Z

← Предыдущая		Версия 16:46, 29 мая 2023
Строка 98:		Строка 98:
	<code>systemctl restart prometheus</code>		<code>systemctl restart prometheus</code>

	Add		Add Dashboard and Panel node_memory_MemFree_bytes for example. You can also add other panels with custom rules.
			[[Файл:Dashboard with record rules 1.jpg\|без\|мини]]
			[[Файл:Dashboard with record rules 2.jpg\|без\|мини]]

	== Case 2. Full expression in Grafana for every metric ==		== Case 2. Full expression in Grafana for every metric ==

Cless в 16:42, 29 мая 2023

2023-05-29T16:42:17Z

← Предыдущая		Версия 16:42, 29 мая 2023
Строка 1:		Строка 1:
	= At Client - Linux node (192.168.17.186) =		= At Client - Linux node (192.168.17.186) - add Node Exporter =
	Install linux component "'''node_exporter'''"		Install linux component "'''node_exporter'''"

Строка 38:		Строка 38:

	Look at '''centos7-test01''' metrics[[Файл:Linux node dashboard to grafana 1.jpg\|без\|мини]]		Look at '''centos7-test01''' metrics[[Файл:Linux node dashboard to grafana 1.jpg\|без\|мини]]
	= At Server - Prometheus =		= At Server - Prometheus - add Job for monitored host (Linux node 192.168.17.186) =
	Add linux '''centos7-test01''' as a Job		Add linux '''centos7-test01''' as a Job

Строка 55:		Строка 55:
	Look at '''centos7-test01''' in Prometheus[[Файл:Linux node dashboard to grafana 2.jpg\|без\|мини]]		Look at '''centos7-test01''' in Prometheus[[Файл:Linux node dashboard to grafana 2.jpg\|без\|мини]]

	= ~~Examples~~ =		= Metrics Monitoring =

	== ~~Load Average 5 minutes~~ ==		== Case 1. Recording rules ('''Recommended''') ==
			https://medium.com/javarevisited/create-recording-rules-in-prometheus-8a6a1c0b9e11

	=== At Grafana ===		Recording rules allows us to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. Querying the precomputed result will then often be much faster than executing the original expression every time it is needed.

			This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh. Recording and alerting rules exist in a rule group.

			Create rules

			<code>nano /etc/prometheus/prometheus_rules.yml</code>
			groups:
			- name: custom_rules
			rules:
			- record: node_cpuUsedTotal_percent
			expr: 100 - ((rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

			- record: node_memory_MemFree_percent
			expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

			- record: node_memory_MemUsed_percent
			expr: (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

			- record: node_filesystem_free_percent
			expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
			Verify new rules

			<code>promtool check rules /etc/prometheus/prometheus_rules.yml</code>

			Edit main config

			<code>nano /etc/prometheus/prometheus.yml</code>
			rule_files:
			...
			- "prometheus_rules.yml"
			...
			Restart service

			<code>systemctl daemon-reload</code>

			<code>systemctl restart prometheus</code>

			Add

			== Case 2. Full expression in Grafana for every metric ==

			=== Load Average 5 minutes ===

			==== At Grafana ====

	Add '''centos7-test01''' Dashboard to Grafana		Add '''centos7-test01''' Dashboard to Grafana
Строка 68:		Строка 113:
	[[Файл:Linux node dashboard to grafana 7.jpg\|без\|мини]]		[[Файл:Linux node dashboard to grafana 7.jpg\|без\|мини]]

	== CPU Load total % ==		=== CPU Load total % ===

	=== Understanding Machine CPU usage ===		==== Understanding Machine CPU usage ====
	High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.		High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.

Строка 113:		Строка 158:
	100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)		100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)

	=== At Client - Linux node (192.168.17.186) ===		==== At Client - Linux node (192.168.17.186) ====
	Let's simulate CPU Load of one CPU core at client		Let's simulate CPU Load of one CPU core at client

	<code>dd if=/dev/zero of=/dev/null</code>		<code>dd if=/dev/zero of=/dev/null</code>

	=== Prometheus ===		==== Prometheus ====
	Let's form '''Query''' in Web and '''Execute''' it:		Let's form '''Query''' in Web and '''Execute''' it:

Строка 125:		Строка 170:
	[[Файл:Prometheus cpu load total 2.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 2.jpg\|без\|мини]]

	=== Grafana ===		==== Grafana ====
	[[Файл:Prometheus cpu load total 3.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 3.jpg\|без\|мини]]
	[[Файл:Prometheus cpu load total 4.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 4.jpg\|без\|мини]]
	[[Файл:Prometheus cpu load total 5.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 5.jpg\|без\|мини]]

	== RAM Free ==		=== RAM Free ===

	=== RAM free in MB ===		==== RAM free in MB ====
	<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / 1024 / 1024</code>		<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / 1024 / 1024</code>

	=== RAM free in % ===		==== RAM free in % ====
	<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / (node_memory_MemTotal_bytes{job="centos7-test01"}) * 100</code>		<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / (node_memory_MemTotal_bytes{job="centos7-test01"}) * 100</code>

	== Disk I/O ==		=== Disk I/O ===

	=== Disk write MB/s ===		==== Disk write MB/s ====
	All disks		All disks

Строка 149:		Строка 194:
	<code>rate(node_disk_written_bytes_total{'''device="sda"''',job="centos7-test01"}[30s]) / 1024 / 1024</code>		<code>rate(node_disk_written_bytes_total{'''device="sda"''',job="centos7-test01"}[30s]) / 1024 / 1024</code>

	=== Disk read MB/s ===		==== Disk read MB/s ====
	All disks		All disks

Cless в 08:18, 6 августа 2022

2022-08-06T08:18:37Z

← Предыдущая		Версия 08:18, 6 августа 2022
Строка 147:		Строка 147:
	Disk "'''sda'''"		Disk "'''sda'''"

	rate(node_disk_written_bytes_total{device="sda",job="centos7-test01"}[30s]) / 1024 / 1024		<code>rate(node_disk_written_bytes_total{'''device="sda"''',job="centos7-test01"}[30s]) / 1024 / 1024</code>

	=== Disk read MB/s ===		=== Disk read MB/s ===
Строка 153:		Строка 153:

	<code>rate(node_disk_read_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>		<code>rate(node_disk_read_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>

	Disk "'''sda'''"		Disk "'''sda'''"

	rate(node_disk_read_bytes_total{device="sda",job="centos7-test01"}[30s]) / 1024 / 1024		<code>rate(node_disk_read_bytes_total{'''device="sda"''',job="centos7-test01"}[30s]) / 1024 / 1024</code>
	[[Категория:Prometheus]]		[[Категория:Prometheus]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

Cless в 08:14, 25 июля 2022

2022-07-25T08:14:55Z

← Предыдущая		Версия 08:14, 25 июля 2022
Строка 140:		Строка 140:
	== Disk I/O ==		== Disk I/O ==

	=== ~~Disks~~ write MB/s ===		=== Disk write MB/s ===
			All disks

	<code>rate(node_disk_written_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>		<code>rate(node_disk_written_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>

	=== ~~Disks~~ read MB/s ===		Disk "'''sda'''"

			rate(node_disk_written_bytes_total{device="sda",job="centos7-test01"}[30s]) / 1024 / 1024

			=== Disk read MB/s ===
			All disks

	<code>rate(node_disk_read_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>		<code>rate(node_disk_read_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>
			Disk "'''sda'''"

			rate(node_disk_read_bytes_total{device="sda",job="centos7-test01"}[30s]) / 1024 / 1024
	[[Категория:Prometheus]]		[[Категория:Prometheus]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

Cless в 08:05, 25 июля 2022

2022-07-25T08:05:10Z

← Предыдущая		Версия 08:05, 25 июля 2022
Строка 131:		Строка 131:

	== RAM Free ==		== RAM Free ==
	~~RAM free in MB~~

			=== RAM free in MB ===
	<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / 1024 / 1024</code>		<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / 1024 / 1024</code>

	RAM free in %		=== RAM free in % ===
			<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / (node_memory_MemTotal_bytes{job="centos7-test01"}) * 100</code>

			== Disk I/O ==

			=== Disks write MB/s ===
			<code>rate(node_disk_written_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>

	<code>(~~node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes~~{job="centos7-test01"}) / ~~(node_memory_MemTotal_bytes{job="centos7-test01"}) * 100~~</code>		=== Disks read MB/s ===
			<code>rate(node_disk_read_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024</code>
	[[Категория:Prometheus]]		[[Категория:Prometheus]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

Cless в 06:47, 25 июля 2022

2022-07-25T06:47:02Z

← Предыдущая		Версия 06:47, 25 июля 2022
Строка 129:		Строка 129:
	[[Файл:Prometheus cpu load total 4.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 4.jpg\|без\|мини]]
	[[Файл:Prometheus cpu load total 5.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 5.jpg\|без\|мини]]

			== RAM Free ==
			RAM free in MB

			<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / 1024 / 1024</code>

			RAM free in %

			<code>(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / (node_memory_MemTotal_bytes{job="centos7-test01"}) * 100</code>
	[[Категория:Prometheus]]		[[Категория:Prometheus]]
	__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__		__ОБЯЗАТЕЛЬНОЕ_ОГЛАВЛЕНИЕ__

Cless в 09:34, 24 июля 2022

2022-07-24T09:34:28Z

← Предыдущая		Версия 09:34, 24 июля 2022
Строка 57:		Строка 57:
	= Examples =		= Examples =

	== Load Average 5 ~~seconds~~ ==		== Load Average 5 minutes ==

	=== At Grafana ===		=== At Grafana ===

Cless в 19:41, 23 июля 2022

2022-07-23T19:41:30Z

← Предыдущая		Версия 19:41, 23 июля 2022
Строка 121:		Строка 121:
	Let's form '''Query''' in Web and '''Execute''' it:		Let's form '''Query''' in Web and '''Execute''' it:

	<code>~~sum~~ by (~~mode,~~ instance) (rate(node_cpu_seconds_total{~~mode!~~="~~idle~~",~~job~~="~~centos7-test01~~"}[10s]))</code>		<code>100 - (avg by (instance) (rate(node_cpu_seconds_total{job="centos7-test01",mode="idle"}[10s])) * 100)</code>
	[[Файл:Prometheus cpu load total 1.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 1.jpg\|без\|мини]]
	[[Файл:Prometheus cpu load total 2.jpg\|без\|мини]]		[[Файл:Prometheus cpu load total 2.jpg\|без\|мини]]

Cless в 19:38, 23 июля 2022

2022-07-23T19:38:17Z

← Предыдущая		Версия 19:38, 23 июля 2022
Строка 69:		Строка 69:

	== CPU Load total % ==		== CPU Load total % ==
	~~<blockquote>~~High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.
			=== Understanding Machine CPU usage ===
			High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.

	On a Node exporters' metrics page, part of the output is:		On a Node exporters' metrics page, part of the output is:
Строка 108:		Строка 110:
	We can aggregate this to get the overall value across all CPUs for the machine:		We can aggregate this to get the overall value across all CPUs for the machine:
	sum by (mode, instance) (rate(node_cpu_seconds_total{job="node"}[1m]))		sum by (mode, instance) (rate(node_cpu_seconds_total{job="node"}[1m]))


	As these values always sum to one second per second for each cpu, the per-second rates are also the ratios of usage. We can use this to calculate the percentage of CPU used, by subtracting the idle usage from 100%:		As these values always sum to one second per second for each cpu, the per-second rates are also the ratios of usage. We can use this to calculate the percentage of CPU used, by subtracting the idle usage from 100%:
	100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)		100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)
	~~</blockquote>~~

	=== At Client - Linux node (192.168.17.186) ===		=== At Client - Linux node (192.168.17.186) ===

Cless в 19:36, 23 июля 2022

2022-07-23T19:36:36Z

← Предыдущая		Версия 19:36, 23 июля 2022
Строка 69:		Строка 69:

	== CPU Load total % ==		== CPU Load total % ==
	<blockquote>		<blockquote>High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.
	~~== Understanding Machine CPU usage ==~~
	High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.

	On a Node exporters' metrics page, part of the output is:		On a Node exporters' metrics page, part of the output is: