Homelab Capacity Planning: What If a Node Dies Tonight?

Every dashboard in my rack could tell me what was happening right now, and not one of them could answer the question that actually keeps me up: if one of these four machines dies tonight, do the containers it was running fit on the three that are left — and how long until they’re back? I’d been answering that with a shrug and a vague sense that it was “probably fine.” So I spent a day building something that answers it properly: a read-only simulator that reads my real cluster, does the arithmetic, and tells me plainly when the answer is no.

It told me no on day one. Here’s how it works, and what it found.

Why “just migrate it” isn’t the answer

The tempting mental model is the one from the enterprise world: a node dies, the cluster notices, and your workloads reappear elsewhere a minute later. That’s real, and Proxmox does it — but it rests on a requirement most home clusters don’t meet. The Proxmox VE high availability documentation is blunt about the prerequisites: you need “at least three cluster nodes”, “shared storage for VMs and containers”, “hardware redundancy (everywhere)”, and reliable server components.

Shared storage is the one that bites. It means the guest’s disk lives somewhere both machines can reach — a SAN, a Ceph pool, an NFS share — so a survivor can simply start a guest whose disk it already has. My four nodes each have their own local NVMe and nothing shared. When one dies, its disks die with it. There is no “start it elsewhere,” because elsewhere doesn’t have the data.

So the honest recovery path on a cluster like mine is: restore the guest from a backup onto a surviving node. That reframes the whole planning question. I don’t need to know whether a guest could migrate. I need to know two things:

1. Does it fit? Does some survivor have enough free memory to run this guest once it’s back?
2. How long? How many minutes of restore stand between “node died” and “service answering again”?

Those are both arithmetic. Arithmetic is exactly the kind of thing a computer should be doing for me at 2am instead of me.

The number that makes the model honest

My first instinct was to plan against configured memory — the maxmem value each container is set to. It’s right there in the API, it’s one field, done.

It’s also close to useless for this. I’d sized most of those containers by guessing generously and never revisiting, which is what everyone does. Planning against those numbers made my cluster look nearly full when it was mostly idle, and it would have told me a node loss was catastrophic when it wasn’t. The configured limit is a safety ceiling — it stops a runaway process from eating the host. It is not a statement about what the workload needs.

What the workload needs is what it actually used, over long enough to have caught its busy moments. I already had that: Prometheus has been scraping my cluster for months, and the Proxmox VE exporter publishes per-guest memory as pve_memory_usage_bytes. So demand became a seven-day peak, and node capacity became a seven-day typical.

Two Prometheus range queries do the whole job. max_over_time returns “the maximum value of all float samples in the specified interval” — the busiest that guest got all week. quantile_over_time returns the φ-quantile over the interval, and at φ = 0.95 that’s a node’s typical-but-busy baseline with brief spikes ignored:

# Per-guest demand: the busiest each guest got in seven days
max_over_time(pve_memory_usage_bytes{id=~"(lxc|qemu)/.+"}[7d])

# Per-node baseline: what the node itself is typically carrying
quantile_over_time(0.95, pve_memory_usage_bytes{id=~"node/.+"}[7d])

Why the asymmetry — peak for guests, 95th percentile for nodes? Because they’re answering different questions. A guest’s demand has to cover its worst moment, or the restore fills memory and something gets killed. A node’s existing load is a baseline I’m subtracting from its total, and using that node’s worst moment there would double-count the same spikes and make every node look fuller than it is. Peak what you’re placing; percentile what you’re placing it onto.

Free capacity then comes out as one line, with a gigabyte held back so I’m never planning to fill a hypervisor to the last byte:

def node_free(node, p95_used, reserve=1 * 1024**3):
    """Placement capacity: total - observed p95 (fallback: current) - reserve."""
    used = p95_used if p95_used is not None else node["used_mem_now"]
    return max(0, node["total_mem"] - used - reserve)

The fallback in that line matters more than it looks. If Prometheus is unreachable, the model doesn’t crash and it doesn’t lie — it drops to the node’s current memory reading and carries on with a worse number. Same for guests: no seven-day history means fall back to configured maxmem, and every result labels which source it used, so I can see at a glance whether I’m reading measurement or guesswork.

Packing the survivors in fifteen lines

Once every guest has a demand number and every survivor has a free number, “does it fit” is a bin-packing problem: fit a set of items into as few containers as possible. It’s a genuinely hard problem in the general case, but the good approximations are almost insultingly simple.

I used first-fit decreasing: sort the guests largest-first, then walk the list and drop each one into the first node it fits in. The sorting is the whole trick — place the awkward large guests while there’s still somewhere to put them, instead of filling every node with small fry and discovering the 13 GiB container has nowhere to go.

It’s also a well-studied choice rather than a hunch. The asymptotic quality of first-fit decreasing was established by Johnson in 1973, and the exact tight bound — that it never needs more than 11/9 of the optimal number of bins plus 6/9 — was proved by Dósa in 2007 and given a complete proof in Theoretical Computer Science in 2013. For deciding whether three home servers can absorb a fourth, being provably within about 22% of a perfect packing is far more rigour than the question needs.

def pack(items, bins):
    """First-fit-decreasing bin packing. Inputs are not mutated."""
    order = sorted(items, key=lambda g: (-g["demand"], g.get("vmid", 0)))
    state = [dict(b) for b in sorted(bins, key=lambda b: (-b["free"], b["name"]))]
    placements, stranded = [], []
    for g in order:
        for b in state:
            if b["free"] >= g["demand"]:
                b["free"] -= g["demand"]
                p = dict(g)
                p["target"] = b["name"]
                placements.append(p)
                break
        else:
            stranded.append(dict(g))
    return placements, stranded, state

That for...else is one of Python’s odder pieces of syntax and it’s perfect here: the else branch runs only if the inner loop finished without hitting break, which is precisely the case where no node had room. Anything landing in stranded is a guest that has nowhere to go — the answer I actually built this to find.

The tie-breakers exist for a reason too. Sorting by demand and then by guest ID means the same cluster state always produces the same plan; without it, two guests of equal size could swap targets between runs and I’d never be sure whether the model or the lab had changed.

How long until it’s back?

A plan that says “it fits” but not “in twenty minutes” is only half an answer. Restore time is the other half, and it’s the part where it’s easiest to quietly invent a number.

The arithmetic is trivial — backup size divided by restore throughput. Getting an honest throughput figure is not, because it depends on your disks, your network, and how compressible your data is. So I measured mine: during a disaster-recovery drill I restored real guests from Proxmox Backup Server and timed them, which gave a range of 110–119 MiB/s. The model uses 110 — the pessimistic end, on the principle that a recovery estimate should disappoint you in the right direction.

The simulator asks the backup server for the latest snapshot of each affected guest, sums the sizes, divides, and reports both the total and a per-guest breakdown — because knowing that the whole node takes 40 minutes is less useful than knowing DNS is back in 30 seconds and the big one is the long pole. It also reports each backup’s age, which turned out to be an accidental audit: if a guest’s most recent backup is 200 hours old, the restore estimate is the least of my problems.

Day one, it told me something I didn’t want to hear

I built this expecting reassurance. The first full run gave me a red result instead: losing my first node strands the large language model container. Its observed seven-day peak is 12.85 GiB, and no survivor has anywhere near that much free memory. The other guests on that node place fine. That one has nowhere to go.

This was the moment the tool paid for itself. I’d been vaguely intending to add memory to the cluster “at some point,” in the way you vaguely intend to do most homelab things. What I had instead, in one query, was a specific sentence: this specific container cannot come back after this specific failure, and it needs 12.85 GiB of headroom somewhere for that to change. A vague intention became a sized purchase with a reason attached.

That’s the real argument for building something like this. Not the dashboard — the forced honesty. A model that only ever agrees with you isn’t telling you anything.

The scenario with no ETA

There’s one node in my cluster whose loss doesn’t get a number, and refusing to give it one was a deliberate design decision.

That node runs the backup server itself. If it dies, I don’t just lose its guests — I lose the backups I’d restore them from. The fast recovery path and the thing that failed are the same machine. Two other things live there too: the Tailscale subnet router that gets me into the lab remotely, and a mirror of my documentation vault. Losing that node means losing guests, backups, and my remote way in, all at once.

The simulator flags this case explicitly rather than smoothing it over. It sets a pbs_lost marker, sets the recovery estimate to null instead of a number, and returns notes in plain language: recovery falls back to the weekly archive that lives on separate hardware at a different location, which is hours rather than minutes; remote access is down until the router is replaced.

The bug that taught me the most

Alongside the failure scenarios I added a “can I fit a new container?” form: type in cores, memory, and disk, get back which node it should go on. I wired it up, tested it, shipped it.

Then I watched the access log while someone else used it. Every request was arriving fine. Every request was also coming back fits — 10 cores, fine. 15 cores, fine. 24 cores on an 8-core machine, fine. A 3.2 TB disk, fine. The verdict never changed, so the form looked like it wasn’t running at all.

It was running perfectly. That was the bug. I’d implemented every constraint except memory as a soft one — the response carried polite advisory notes about oversubscription that nobody reads, while the headline verdict stayed green. I’d optimised for not being annoying and landed on not being useful.

The fix was to decide which constraints are real. Cores and disk space are physical: a node with 8 cores cannot host a guest configured for 24, and a node with 412 GiB free cannot store a 3.2 TB root disk. Those became hard refusals. Only the oversubscription ratio — how many virtual CPUs are promised across all guests versus how many the node has — stayed advisory, because oversubscription is genuinely normal and usually fine.

Then each refusal got a sentence naming the node and the reason, so the answer is “pvelab02: only 8 cores (need 24)” rather than a bare no. And the form clears its verdict the moment you change any input, so a stale green result can never sit under a question you’ve since edited.

Four gotchas that cost me an evening

API tokens can’t outrank their user. I gave the simulator a read-only Proxmox API token with privilege separation, which is the right call — but it means the token’s “effective permissions are calculated by intersecting user and token permissions.” A permission the user doesn’t have cannot be granted to its token, no matter what you put in the token’s ACL. Worth knowing before you spend an hour debugging a 403 on a token you’re certain you configured correctly.

Backup server tokens work the same way, but need their own entry. Proxmox Backup Server states it plainly: “API tokens require their own ACL entries” and “API tokens can never do more than their corresponding user.” I granted the DatastoreAudit role — which “can view datastore metrics, settings and list content” but not read the actual data, exactly the access a size-reading tool should have — to the token and got 403s until I granted it to the user as well. Both entries, every time.

Filter non-finite values out of Prometheus results. A range query over a window where a target was down can hand back values that aren’t real numbers. Feed one into a subtraction and it propagates silently, turning a capacity figure into nonsense that still looks like a figure. Three characters fix it — for a float, v == v is false precisely when that float is NaN — and it’s the difference between a wrong answer and no answer.

Speak the vocabulary your interface already uses. My first refusal messages said “threads,” because on these particular CPUs the number in question is the thread count. But the field a reader fills in when creating a container is labelled Cores, and that’s the same pool the number is drawn from — so my form said cores, my error said threads, and both were individually defensible. I made them match the label the reader is actually looking at. Pick the vocabulary of the interface in front of your user, even when it’s not the word you’d choose.

What this is not

Honesty about the limits, since the whole point of the thing is honesty:

• It is not a production web service. It’s stdlib Python built on http.server, whose own documentation warns it “is not recommended for production” and “only implements basic security checks” (Python docs). It binds to my LAN, serves only GET requests, holds read-only credentials, and computes from a cached snapshot. That’s the entire threat model, and it’s why this is acceptable — put it on the internet and it stops being acceptable immediately.
• It models memory, cores, and disk. Nothing else. Not disk throughput, not network saturation, not the thermal reality of three nodes carrying four nodes’ work. A guest that “fits” on RAM can still run badly.
• It’s a point estimate, not a promise. It reports what would happen if the node died now, with the current backups and the current load. It’s a planning aid, not an SLA.
• It has no alerting and no opinions. It never pages anyone. Nothing it says is urgent; it answers when asked.

None of that undermines the value, because the value was never precision. It was replacing “probably fine” with a specific number I can act on — and, once, with a specific number I didn’t want to hear.

What’s next

If you want to build the same thing, the order that worked for me: get Prometheus scraping your cluster and leave it running for a week so you have real history to plan against, make sure your backups actually restore and time one while you’re there, then write the arithmetic. The simulator is the easy part — a few hundred lines. The week of history and the measured restore rate are what make its answers mean anything.

And run it before you think you need to. The useful result isn’t the one that confirms you’re fine.

Related posts:

• Prometheus and Grafana on Proxmox — the metrics stack this whole model reads from; build it first
• Node Exporter on Proxmox — adds the per-node hardware metrics that give the capacity numbers their baseline
• Proxmox Backup Server in an LXC — the restore path every one of these scenarios depends on, and where you measure your own restore rate
• Build a Proxmox Cluster — you need more than one node before “what if one dies” is an interesting question
• A 4-Node Proxmox Cluster on HP EliteDesks — the actual hardware being simulated here, and why it has no shared storage
• Tailscale Subnet Router — the remote access that disappears along with one specific node in these scenarios
• Uptime Kuma Monitoring — tells you a node died; this post is about what to do in the ten minutes after
• Track AI Token Spend in Grafana — the other “measure the thing you were guessing about” project in this lab
• Turn a Hand-Built Proxmox Cluster Into Code Without Breaking It — the other half of planning for failure: what code restores after a node dies, and what it can’t