A Proxmox cluster without monitoring is flying blind. This post walks through deploying a full observability stack: Prometheus collecting metrics from all four nodes plus the Proxmox API, Grafana visualizing everything, and alerting that emails you when something goes wrong.
Stack architecture
All monitoring services run in a single LXC container (CT 105) on pvelab02, keeping the stack isolated and easy to back up.
Task 1: Create the monitoring container
| Field | Value |
|---|---|
| VMID | 105 |
| Hostname | monitoring |
| OS | Debian 12 LXC (unprivileged) |
| Host | pvelab02 (10.25.144.71) |
| IP | 10.25.144.68 |
| Cores | 2 |
| RAM | 2048 MB |
| Disk | 20 GB |
pvelab02 is the right host: pvelab01 hosts CT 100 and CT 107; pvelab03 already has three containers. Spreading the load matters on 8 GB nodes.
Task 2: Install node_exporter on all four Proxmox hosts
Run this on each Proxmox host — not in containers. node_exporter must run on the bare metal to collect hardware metrics (CPU temp, memory, disk).
apt-get update
apt-get install -y prometheus-node-exporter
systemctl enable --now prometheus-node-exporter
Verify it’s running and exposing metrics:
curl http://localhost:9100/metrics | grep node_cpu_seconds | head -5
The AMD Ryzen 5 2400G exposes temperature via the k10temp kernel module. Verify it’s showing up:
curl -s http://localhost:9100/metrics | grep hwmon | grep temp
You should see entries for k10temp (CPU die temp) and amdgpu (iGPU edge temp). These are the metrics that power the temperature alerts.
If no hwmon metrics appear, ensure prometheus-node-exporter is version ≥ 1.5 and that the hwmon collector is not explicitly disabled. Debian Bookworm ships 1.5.0 which enables hwmon by default.
Task 3: Set up pve-exporter for Proxmox API metrics
In the Proxmox web UI: Datacenter → Permissions → API Tokens → Add. Create:
| Field | Value |
|---|---|
| User | prometheus@pve (create user first if needed) |
| Token ID | prometheus |
| Privilege Separation | Yes |
| Role | PVEAuditor |
Copy the token secret — you’ll only see it once.
apt-get update && apt-get install -y python3-pip python3-venv
python3 -m venv /opt/pve-exporter
/opt/pve-exporter/bin/pip install prometheus-pve-exporter
# Create config
cat > /etc/pve-exporter.yml {'<<'} 'EOF'
default:
user: prometheus@pve
token_name: prometheus
token_value: YOUR_TOKEN_SECRET_HERE
verify_ssl: false
EOF
cat > /etc/systemd/system/pve-exporter.service {'<<'} 'EOF'
[Unit]
Description=Proxmox VE Prometheus Exporter
After=network.target
[Service]
ExecStart=/opt/pve-exporter/bin/pve_exporter --config.file /etc/pve-exporter.yml
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now pve-exporter
Task 4: Install and configure Prometheus
apt-get install -y prometheus
Replace /etc/prometheus/prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'node'
static_configs:
- targets:
- '10.25.144.70:9100' # pvelab01
- '10.25.144.71:9100' # pvelab02
- '10.25.144.72:9100' # pvelab03
- '10.25.144.73:9100' # pvelab04
- job_name: 'pve'
static_configs:
- targets: ['10.25.144.68:9221']
metrics_path: /pve
params:
module: [default]
cluster: ['1']
node: ['1']
systemctl restart prometheus
curl http://localhost:9090/-/healthy
Task 5: Install Grafana and import dashboards
apt-get install -y apt-transport-https software-properties-common
wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" \
> /etc/apt/sources.list.d/grafana.list
apt-get update && apt-get install -y grafana
systemctl enable --now grafana-server
Open http://10.25.144.68:3000. Log in (default: admin/admin, change immediately).
Connections → Data Sources → Add → Prometheus. URL: http://localhost:9090. Click Save & Test — should show “Data source is working.”
Dashboards → New → Import. Import by ID:
| Dashboard | ID | What it shows |
|---|---|---|
| Node Exporter Full | 1860 | Per-host CPU, RAM, disk, network, temperatures |
| Proxmox via Prometheus | 10347 | VM/CT resource usage, cluster storage |
Task 6: Configure alerting
Edit /etc/grafana/grafana.ini inside CT 105:
[smtp]
enabled = true
host = smtp.gmail.com:587
user = your@gmail.com
password = YOUR_APP_PASSWORD
from_address = your@gmail.com
from_name = MyOfficeLab Grafana
startTLS_policy = MandatoryStartTLS
Use a Gmail App Password (Google Account → Security → App Passwords), not your real Gmail password. App Passwords work even with 2FA enabled and can be revoked individually.
Restart Grafana after editing: systemctl restart grafana-server
Alerting → Alert Rules → New Alert Rule. Create one rule per condition:
| Rule | PromQL | Fires after |
|---|---|---|
| Node Down | up{job="node"} == 0 |
2m |
| Container Not Running | pve_up{type="lxc"} == 0 |
2m |
| Disk Usage High | (1 - node_filesystem_avail_bytes/node_filesystem_size_bytes) > 0.8 |
5m |
| CPU Temp High | node_hwmon_temp_celsius{chip=~".*k10temp.*",sensor="tdie"} > 80 |
3m |
| RAM Low | node_memory_MemAvailable_bytes < 1073741824 |
5m |
Set noDataState: OK on all rules that use threshold-filter PromQL (like the disk and RAM rules). If the metric returns an empty result because a node is healthy, noDataState: Alerting fires a spurious alert. This is a common misconfiguration.
Monitoring is now live
You have:
- ✅ Real-time metrics from all four Proxmox hosts
- ✅ Cluster-level VM/CT stats via pve-exporter
- ✅ CPU, RAM, disk, and temperature visibility
- ✅ Email alerts for node-down, high CPU temp, low RAM, and full disks
The next post covers Tailscale: making every cluster service accessible remotely through a single subnet router, without opening any firewall ports.