The Proxmox web UI shows CPU, RAM, and network per-node in real time. What it doesn’t show is historical trends, cross-node comparisons, threshold-based alerting, or per-container resource breakdowns over time. That’s what Prometheus and Grafana provide — and the data pipeline for a 4-node Proxmox cluster is simpler to set up than most guides make it look.
This post covers the complete metrics stack: node_exporter for OS-level metrics on every node, pve-exporter for Proxmox API metrics (QEMU/LXC status, cluster health), Prometheus for collection, and two community Grafana dashboards that make the data usable immediately.
Architecture
Task 1: Install node_exporter on all four Proxmox nodes
cd /tmp
curl -sSL https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz | tar xz
cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
chmod +x /usr/local/bin/node_exporter
useradd -r -s /bin/false node_exporter
cat > /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Prometheus Node Exporter
After=network.target
[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter --collector.systemd --collector.processes
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now node_exporter
curl -s http://localhost:9100/metrics | head -20
# Should output lines like: node_cpu_seconds_total{...}
Run the same three commands on each remaining node — the download, the service file, and systemctl enable --now node_exporter. Each node exposes metrics on port 9100.
If your Proxmox cluster nodes share an SSH key, you can loop this efficiently from pvelab04:
for node in 10.0.0.70 10.0.0.71 10.0.0.72; do
ssh root@$node 'bash -s' < /root/install_node_exporter.sh
doneSave the three install commands above as /root/install_node_exporter.sh first.
Task 2: Install pve-exporter
pve-exporter queries the Proxmox API and exposes cluster-level metrics: node status, VM/CT status, storage usage, and more.
apt-get update && apt-get install -y python3-pip python3-venv
python3 -m venv /opt/pve-exporter
/opt/pve-exporter/bin/pip install prometheus-pve-exporter
pve-exporter needs a read-only Proxmox API token. On any Proxmox node:
pveum user add monitoring@pve --comment "Prometheus pve-exporter"
pveum aclmod / -user monitoring@pve -role PVEAuditor
pveum user token add monitoring@pve exporter --privsep=0
# Note the token secret that is printed — you need it in the next step
mkdir -p /etc/pve-exporter
cat > /etc/pve-exporter/config.yml << 'EOF'
default:
user: monitoring@pve
token_name: exporter
token_value: PASTE_TOKEN_SECRET_HERE
verify_ssl: false
EOF
chmod 600 /etc/pve-exporter/config.yml
cat > /etc/systemd/system/pve-exporter.service << 'EOF'
[Unit]
Description=Proxmox VE Exporter for Prometheus
After=network.target
[Service]
ExecStart=/opt/pve-exporter/bin/pve_exporter --config.file=/etc/pve-exporter/config.yml --web.listen-address=0.0.0.0:9221
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now pve-exporter
curl -s "http://localhost:9221/pve?target=10.0.0.73&module=default" | head -30
# Should output pve_* metrics
Task 3: Configure Prometheus
cd /tmp
curl -sSL https://github.com/prometheus/prometheus/releases/download/v2.53.1/prometheus-2.53.1.linux-amd64.tar.gz | tar xz
cp prometheus-2.53.1.linux-amd64/{prometheus,promtool} /usr/local/bin/
mkdir -p /etc/prometheus /var/lib/prometheus
cp -r prometheus-2.53.1.linux-amd64/{consoles,console_libraries} /etc/prometheus/
useradd -r -s /bin/false prometheus
chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
cat > /etc/prometheus/prometheus.yml << 'EOF'
global:
scrape_interval: 30s
evaluation_interval: 30s
scrape_configs:
- job_name: node_exporter
static_configs:
- targets:
- 10.0.0.70:9100 # pvelab01
- 10.0.0.71:9100 # pvelab02
- 10.0.0.72:9100 # pvelab03
- 10.0.0.73:9100 # pvelab04
labels:
cluster: myofficelab
- job_name: pve
metrics_path: /pve
params:
module: [default]
static_configs:
- targets:
- 10.0.0.70 # pvelab01
- 10.0.0.71 # pvelab02
- 10.0.0.72 # pvelab03
- 10.0.0.73 # pvelab04
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: localhost:9221 # pve-exporter address
EOF
cat > /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus
After=network.target
[Service]
User=prometheus
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention.time=30d --web.listen-address=0.0.0.0:9090
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now prometheus
Open http://10.0.0.68:9090/targets (or whatever IP CT 105 has) in your browser. You should see:
- 4
node_exportertargets — all green - 4
pvetargets — all green
If any show as “down”, click the error link for details — usually a firewall or wrong IP.
Task 4: Import Grafana dashboards
In Grafana: Dashboards → Import → Enter ID: 1860
This is the most downloaded Grafana dashboard — it shows CPU, memory, disk, network, and load for every node. Select your Prometheus data source when prompted.
Dashboards → Import → Enter ID: 10347
This dashboard uses pve-exporter metrics: node status, VM/CT counts, CPU and memory per node, storage pool usage. It gives a bird’s-eye cluster view that the Proxmox web UI doesn’t provide.
Task 5: Add alert rules for critical metrics
cat > /etc/prometheus/alerts.yml << 'EOF'
groups:
- name: homelab
rules:
- alert: NodeDown
expr: up{job="node_exporter"} == 0
for: 2m
annotations:
summary: "Node {{ $labels.instance }} is down"
- alert: HighCPU
expr: 100 - (avg by(instance)(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
for: 5m
annotations:
summary: "High CPU on {{ $labels.instance }}: {{ $value | printf "%.0f" }}%"
- alert: LowMemory
expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
for: 5m
annotations:
summary: "Low memory on {{ $labels.instance }}: {{ $value | printf "%.0f" }}% free"
- alert: DiskAlmostFull
expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
for: 1m
annotations:
summary: "Disk almost full on {{ $labels.instance }}: {{ $value | printf "%.0f" }}% free"
EOF
Add the alert file reference to prometheus.yml:
sed -i '/^global:/i rule_files:
- alerts.yml
' /etc/prometheus/prometheus.yml
systemctl reload prometheus
The full Grafana and Alertmanager setup — including routing alerts to email and Slack — is covered in the Grafana alerting post.
Related posts:
- Proxmox Monitoring with Prometheus and Grafana: Full Stack Setup — Alertmanager, email alerts, and dashboard configuration built on these metrics
- Building a 4-Node Proxmox Cluster on HP EliteDesk Mini PCs — the cluster these exporters are scraping
- Proxmox Security Hardening — harden the nodes you’re now monitoring
- Tailscale Subnet Router: Remote Access to Your Entire Homelab — access Grafana dashboards remotely