At Client - Linux node (192.168.17.186) - add Node Exporter

Install linux component "node_exporter"

wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz

tar -xvzf node_exporter-1.3.1.linux-amd64.tar.gz

useradd -rs /bin/false nodeusr

mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/

nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target
[Service]
User=nodeusr
Group=nodeusr
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target

systemctl daemon-reload

systemctl enable node_exporter

systemctl start node_exporter

systemctl start firewalld

firewall-cmd --zone=public --add-port=9100/tcp --permanent

systemctl restart firewalld

Go to web

http://192.168.17.186:9100/metrics

Look at centos7-test01 metrics

At Server - Prometheus - add Job for monitored host (Linux node 192.168.17.186)

Add linux centos7-test01 as a Job

nano /etc/prometheus/prometheus.yml

...
  - job_name: 'centos7-test01'
    scrape_interval: 5s
    static_configs:
      - targets: ['192.168.17.186:9100']

systemctl restart prometheus

Web

http://<Prometheus-Server-IP>:9090/targets

Look at centos7-test01 in Prometheus

Metrics Monitoring

Case 1. Recording rules (Recommended)

https://medium.com/javarevisited/create-recording-rules-in-prometheus-8a6a1c0b9e11

Recording rules allows us to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. Querying the precomputed result will then often be much faster than executing the original expression every time it is needed.

This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh. Recording and alerting rules exist in a rule group.

Create rules

nano /etc/prometheus/prometheus_rules.yml

groups:
  - name: custom_rules
    rules:
      - record: node_cpuUsedTotal_percent
        expr: 100 - ((rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

      - record: node_memory_MemFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

      - record: node_memory_MemUsed_percent
        expr: (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}

Verify new rules

promtool check rules /etc/prometheus/prometheus_rules.yml

Edit main config

nano /etc/prometheus/prometheus.yml

rule_files:
  ...
  - "prometheus_rules.yml"
...

Restart service

systemctl daemon-reload

systemctl restart prometheus

Add

Case 2. Full expression in Grafana for every metric

Load Average 5 minutes

At Grafana

Add centos7-test01 Dashboard to Grafana

CPU Load total %

Understanding Machine CPU usage

High CPU load is a common cause of issues. Let's look at how to dig into it with Prometheus and the Node exporter.

On a Node exporters' metrics page, part of the output is:

# HELP node_cpu Seconds the cpus spent in each mode.
# TYPE node_cpu counter
node_cpu_seconds_total{cpu="0",mode="guest"} 0
node_cpu_seconds_total{cpu="0",mode="idle"} 2.03442237e+06
node_cpu_seconds_total{cpu="0",mode="iowait"} 3522.37
node_cpu_seconds_total{cpu="0",mode="irq"} 0.48
node_cpu_seconds_total{cpu="0",mode="nice"} 515.56
node_cpu_seconds_total{cpu="0",mode="softirq"} 953.06
node_cpu_seconds_total{cpu="0",mode="steal"} 0
node_cpu_seconds_total{cpu="0",mode="system"} 6605.46
node_cpu_seconds_total{cpu="0",mode="user"} 23343.01
node_cpu_seconds_total{cpu="1",mode="guest"} 0
node_cpu_seconds_total{cpu="1",mode="idle"} 2.03471439e+06
node_cpu_seconds_total{cpu="1",mode="iowait"} 3633.5
node_cpu_seconds_total{cpu="1",mode="irq"} 0.58
node_cpu_seconds_total{cpu="1",mode="nice"} 542.05
node_cpu_seconds_total{cpu="1",mode="softirq"} 880.49
node_cpu_seconds_total{cpu="1",mode="steal"} 0
node_cpu_seconds_total{cpu="1",mode="system"} 6581.92
node_cpu_seconds_total{cpu="1",mode="user"} 23171.06

This metric comes from /proc/stat and tell us how many seconds each CPU spent doing each type of work:

user: The time spent in userland
system: The time spent in the kernel
iowait: Time spent waiting for I/O
idle: Time the CPU had nothing to do
irq&softirq: Time servicing interrupts
guest: If you are running VMs, the CPU they use
steal: If you are a VM, time other VMs "stole" from your CPUs

These modes are mutually exclusive. A high iowait means that you are disk or network bound, high user or system means that you are CPU bound.

These are counters, so to calculate the per-second values we use the rate function in the expression browser:

rate(node_cpu_seconds_total{job="node"}[1m])

We can aggregate this to get the overall value across all CPUs for the machine:

sum by (mode, instance) (rate(node_cpu_seconds_total{job="node"}[1m]))

As these values always sum to one second per second for each cpu, the per-second rates are also the ratios of usage. We can use this to calculate the percentage of CPU used, by subtracting the idle usage from 100%:

100 - (avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[1m])) * 100)

At Client - Linux node (192.168.17.186)

Let's simulate CPU Load of one CPU core at client

dd if=/dev/zero of=/dev/null

Prometheus

Let's form Query in Web and Execute it:

100 - (avg by (instance) (rate(node_cpu_seconds_total{job="centos7-test01",mode="idle"}[10s])) * 100)

Grafana

RAM Free

RAM free in MB

(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / 1024 / 1024

RAM free in %

(node_memory_MemTotal_bytes{job="centos7-test01"} - node_memory_MemFree_bytes{job="centos7-test01"}) / (node_memory_MemTotal_bytes{job="centos7-test01"}) * 100

Disk I/O

Disk write MB/s

All disks

rate(node_disk_written_bytes_total{job="centos7-test01"}[30s]) / 1024 / 1024

Disk "sda"

rate(node_disk_written_bytes_total{device="sda",job="centos7-test01"}[30s]) / 1024 / 1024

Disk read MB/s