Prometheus. Monitor Linux server using Prometheus

At Client - Linux node (192.168.17.186)

Install linux component "node_exporter"

wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz

tar -xvzf node_exporter-1.3.1.linux-amd64.tar.gz

useradd -rs /bin/false nodeusr

mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/

nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target
[Service]
User=nodeusr
Group=nodeusr
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

systemctl daemon-reload

systemctl enable node_exporter

systemctl start node_exporter

systemctl start firewalld

firewall-cmd --zone=public --add-port=9100/tcp --permanent

systemctl restart firewalld

Go to web

http://192.168.17.186:9100/metrics

Look at centos7-test01 metrics

At Server - Prometheus

Add linux centos7-test01 as a Job

nano /etc/prometheus/prometheus.yml

...
  - job_name: 'centos7-test01'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.17.186:9100']

systemctl restart prometheus

Web

http://<Prometheus-Server-IP>:9090/targets

Look at centos7-test01 in Prometheus

Examples

Load Average 5 seconds

At Grafana

Add centos7-test01 Dashboard to Grafana

CPU Load total %

Understanding Machine CPU usage

High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.

On a Node exporters' metrics page, part of the output is:

# HELP node_cpu Seconds the cpus spent in each mode.
# TYPE node_cpu counter
node_cpu_seconds_total{cpu="0",mode="guest"} 0
node_cpu_seconds_total{cpu="0",mode="idle"} 2.03442237e+06
node_cpu_seconds_total{cpu="0",mode="iowait"} 3522.37
node_cpu_seconds_total{cpu="0",mode="irq"} 0.48
node_cpu_seconds_total{cpu="0",mode="nice"} 515.56
node_cpu_seconds_total{cpu="0",mode="softirq"} 953.06
node_cpu_seconds_total{cpu="0",mode="steal"} 0
node_cpu_seconds_total{cpu="0",mode="system"} 6605.46
node_cpu_seconds_total{cpu="0",mode="user"} 23343.01
node_cpu_seconds_total{cpu="1",mode="guest"} 0
node_cpu_seconds_total{cpu="1",mode="idle"} 2.03471439e+06
node_cpu_seconds_total{cpu="1",mode="iowait"} 3633.5
node_cpu_seconds_total{cpu="1",mode="irq"} 0.58
node_cpu_seconds_total{cpu="1",mode="nice"} 542.05
node_cpu_seconds_total{cpu="1",mode="softirq"} 880.49
node_cpu_seconds_total{cpu="1",mode="steal"} 0
node_cpu_seconds_total{cpu="1",mode="system"} 6581.92
node_cpu_seconds_total{cpu="1",mode="user"} 23171.06

This metric comes from /proc/stat and tell us how many seconds each CPU spent doing each type of work:

• user: The time spent in userland
• system: The time spent in the kernel
• iowait: Time spent waiting for I/O
• idle: Time the CPU had nothing to do
• irq&softirq: Time servicing interrupts
• guest: If you are running VMs, the CPU they use
• steal: If you are a VM, time other VMs "stole" from your CPUs

These modes are mutually exclusive. A high iowait means that you are disk or network bound, high user or system means that you are CPU bound.

These are counters, so to calculate the per-second values we use the rate function in the expression browser:

rate(node_cpu_seconds_total{job="node"}[1m])

We can aggregate this to get the overall value across all CPUs for the machine:

sum by (mode, instance) (rate(node_cpu_seconds_total{job="node"}[1m]))

As these values always sum to one second per second for each cpu, the per-second rates are also the ratios of usage. We can use this to calculate the percentage of CPU used, by subtracting the idle usage from 100%:

100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)

At Client - Linux node (192.168.17.186)

Let's simulate CPU Load of one CPU core at client

dd if=/dev/zero of=/dev/null

Prometheus

Let's form Query in Web and Execute it:

100 - (avg by (instance) (rate(node_cpu_seconds_total{job="centos7-test01",mode="idle"}[10s])) * 100)

Grafana