Running local LLMs on commodity hardware is genuinely practical in 2026 — if you’re willing to accept CPU-speed inference (~2–12 tok/s) in exchange for zero cloud cost and complete privacy. This post covers deploying Ollama across three Proxmox LXC containers, wiring them to Open WebUI for load-balanced access, and the specific gotchas I hit along the way.
Architecture
Performance expectations
Before diving in: CPU inference is slow compared to a GPU. Here’s what to expect on Ryzen 5 2400G hardware:
| Model | Size | Tokens/sec | Use case |
|---|---|---|---|
| llama3.2:3b | ~2 GB | 8–12 tok/s | Quick queries, fast chat |
| qwen2.5-coder:14b | ~9 GB | 2–3 tok/s | Code gen, complex reasoning |
These speeds are per-node. Open WebUI’s round-robin distributes concurrent requests across all three nodes — parallel users each get a dedicated node.
Task 1: Create three Ubuntu 24.04 LXC containers
In the Proxmox web UI: select any node → local storage → CT Templates → Templates. Search for ubuntu-24.04 and click Download.
Datacenter → pvelab01 → Create CT. Settings:
| Field | Value |
|---|---|
| VMID | 100 |
| Hostname | ollama-node1 |
| Template | ubuntu-24.04 |
| Disk | 20 GB on local-lvm |
| Cores | 4 |
| RAM | 14336 MB (14 GB) |
| Network | DHCP → then set static to 10.25.144.65 |
| Unprivileged | Yes |
Allocate 14 GB RAM per node. qwen2.5-coder:14b loads ~9 GB into memory; you need headroom for the OS and concurrent requests. Nodes with only 8 GB host the smaller llama3.2 model only.
Create two more containers with the same spec. Assign:
- CT 101: pvelab02, IP 10.25.144.66, RAM 6144 MB (8 GB node — llama3.2 only)
- CT 102: pvelab03, IP 10.25.144.67, RAM 4096 MB (8 GB node — llama3.2 only)
Start all three containers.
Task 2: Install Ollama on each node
Run these steps inside each container. Access via pct exec <vmid> -- bash from the Proxmox host.
curl -fsSL https://ollama.com/install.sh | sh
systemctl enable --now ollama
By default, Ollama binds to localhost:11434. Open WebUI needs to reach it over the container network, so override the bind address:
mkdir -p /etc/systemd/system/ollama.service.d
cat > /etc/systemd/system/ollama.service.d/override.conf {'<<'} 'EOF'
[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_NUM_THREADS=8"
EOF
systemctl daemon-reload && systemctl restart ollama
OLLAMA_NUM_THREADS=8 pins Ollama to all 8 logical cores of the 2400G. Without this, Ollama may auto-detect fewer threads and leave performance on the table.
On nodes with 14+ GB RAM (pvelab01):
ollama pull llama3.2
ollama pull qwen2.5-coder:14b
On 8 GB nodes (pvelab02, pvelab03):
ollama pull llama3.2
curl http://10.25.144.65:11434/api/tags
curl http://10.25.144.66:11434/api/tags
curl http://10.25.144.67:11434/api/tags
Each should return a JSON object listing installed models. If any fails, check systemctl status ollama inside that container.
Task 3: Deploy Open WebUI natively (not Docker)
Docker inside an unprivileged LXC container requires nesting=1 and causes iptables/nftables rule conflicts. The Docker container also sets a FORWARD drop policy that blocks all traffic from your Ollama containers to the gateway. This is exactly what happened in my initial setup — it cost two sessions to diagnose and clean up. Use the native Python install instead.
| Field | Value |
|---|---|
| VMID | 103 |
| Hostname | open-webui |
| IP | 10.25.144.69 |
| RAM | 2048 MB |
| Cores | 2 |
| Unprivileged | Yes |
apt-get update && apt-get install -y python3-venv python3-pip
useradd -m -s /bin/bash openwebui
su - openwebui -c "python3 -m venv /home/openwebui/venv"
su - openwebui -c "/home/openwebui/venv/bin/pip install open-webui"
cat > /etc/systemd/system/open-webui.service {'<<'} 'EOF'
[Unit]
Description=Open WebUI
After=network.target
[Service]
User=openwebui
WorkingDirectory=/home/openwebui
Environment="OLLAMA_BASE_URLS=http://10.25.144.65:11434;http://10.25.144.66:11434;http://10.25.144.67:11434"
Environment="DATA_DIR=/home/openwebui/data"
ExecStart=/home/openwebui/venv/bin/open-webui serve
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now open-webui
The semicolon-separated OLLAMA_BASE_URLS gives Open WebUI its round-robin load balancer. Requests are distributed across all three Ollama nodes automatically.
Open http://10.25.144.69:8080 in your browser. After creating an admin account, go to Admin → Settings → Connections. You should see all three Ollama endpoints listed with green status indicators.
Routing policy: when to use local vs. cloud
With three local nodes running, the question becomes: when to use local inference vs. Claude/GPT?
The tl;dr: local inference is free and private; use it for everything that doesn’t require frontier-model reasoning quality.
Service management reference
# Check model list on all three nodes
for ct in 100 101 102; do
echo "=== CT $ct ==="; pct exec $ct -- ollama list
done
# Restart Ollama on a specific node
pct exec 100 -- systemctl restart ollama
# Watch inference logs in real time
pct exec 100 -- journalctl -u ollama -f
# Test a quick generation (expect ~2 tok/s)
curl http://10.25.144.65:11434/api/generate \
-d '{"model":"llama3.2","prompt":"Hello world","stream":false}'
Next in this series: wiring up Hermes Agent — a fully autonomous AI agent running on the cluster with Proxmox read access, web search, and automated weekly reports.