34 KiB

Raw Blame History

Incus Clustering Guide for IncusOS (via Remotes)

This guide covers forming an Incus cluster from IncusOS nodes using only the incus CLI and remotes. No SSH access to the nodes is needed (IncusOS is immutable and has no shell).

Tested with Incus client 6.21 and IncusOS nodes deployed via incusos-proxmox.

Overview

Incus clustering does not require a virtual IP (VIP), load balancer, or shared storage. Each node advertises its own IP address. Clients can connect to any cluster member -- requests are forwarded internally via the cluster's internal communication channel.

Cluster roles:

database-leader: holds the primary Dqlite database
database: database replica (for HA, need 3 database members)
database-standby: standby replica (promoted automatically if a database member goes down)

With 3 nodes, you get a fully HA database quorum.

Prerequisites

Incus client version 6.20+ (required for incus cluster join via remotes)
Init node deployed with apply_defaults: true; joining nodes with apply_defaults: false (recommended) or true (requires cleanup before join)
Remotes configured for all nodes (incus remote add <name> https://<ip>:8443)
Client certificate trusted on all nodes (injected via seed or added manually)

incus version                        # Verify 6.20+
incus info incus-lab-01: | head -5   # Verify remote connectivity
incus info incus-lab-02: | head -5
incus info incus-lab-03: | head -5

Step 1: Set Specific IP Addresses

Why this is needed

IncusOS nodes listen on :8443 by default (all interfaces). Clustering requires each node to have a specific routable IP so other members can address it. Without this, incus cluster enable either fails or prompts for an address.

How to discover node IPs

for node in incus-lab-01 incus-lab-02 incus-lab-03; do
  echo "=== $node ==="
  incus query "$node:/1.0" | python3 -c "
import sys, json
d = json.load(sys.stdin)
for a in d['environment']['addresses']:
    if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('['):
        print(a)
"
done

This filters out:

10.x.x.x -- internal bridge addresses (incusbr0)
fd42: -- ULA IPv6 bridge addresses
[...] -- IPv6 addresses in bracket notation

Set the addresses

incus config set incus-lab-01: core.https_address 192.168.1.250:8443
incus config set incus-lab-02: core.https_address 192.168.1.159:8443
incus config set incus-lab-03: core.https_address 192.168.1.78:8443

Verify connectivity after each change

incus config get incus-lab-01: core.https_address   # Should show IP:8443
incus info incus-lab-01: | head -3                   # Should still connect

What happens under the hood

The Incus daemon rebinds its HTTPS listener from 0.0.0.0:8443 to the specific IP. Since the client remote was already connecting to that IP, the connection continues to work seamlessly. Certificate trust is stored by fingerprint in the Incus database, independent of the listen address.

Step 2: Enable Clustering on the Init Node

incus cluster enable incus-lab-01: incus-lab-01

Syntax note: this is TWO arguments -- remote: (trailing colon, targeting the server) and member-name (the cluster member name). The help text shows [<remote>:] <name>. Using remote:name as a single arg causes "The incus daemon doesn't appear to be started" on client-only systems.

The TLS certificate problem

Enabling clustering causes the server to generate a new TLS certificate (the cluster certificate). This new cert typically only has SANs (Subject Alternative Names) for 127.0.0.1 and ::1, not the node's actual IP.

Your existing remote has the old (pre-clustering) certificate pinned, so the next connection fails:

Error: tls: failed to verify certificate: x509: certificate is valid for
127.0.0.1, ::1, not 192.168.1.250

The fix: re-add the remote

# Switch default remote first (can't remove the current default)
incus remote switch local

# Remove and re-add with the new certificate
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://192.168.1.250:8443 --accept-certificate

Incus remotes use fingerprint pinning (not CA-based SAN validation), so --accept-certificate pins the new cluster cert and future connections work despite the SAN mismatch.

Verify

incus cluster list incus-lab-01:

Expected output: one member (incus-lab-01), status ONLINE, role database-leader + database.

Step 3: Prepare Joining Nodes

Recommended: deploy joining nodes with apply_defaults: false

The upstream IncusOS clustering tutorial recommends deploying joining nodes without default settings. In the YAML config:

vms:
  - name: incus-lab-01
    app: incus
    apply_defaults: true    # init node: needs storage pool + network

  - name: incus-lab-02
    app: incus
    apply_defaults: false   # joining node

  - name: incus-lab-03
    app: incus
    apply_defaults: false   # joining node

With apply_defaults: false, the node still listens on port 8443, trusts preseed certificates, and the underlying ZFS dataset (local/incus) exists. But no Incus storage pool or network metadata is created, so the join process works cleanly without cleanup. Skip to Step 4 if using this approach.

Alternative: cleanup when apply_defaults: true was used

If joining nodes were deployed with apply_defaults: true, they already have:

A local ZFS storage pool (source: local/incus)
An incusbr0 network bridge
A default profile referencing both

When joining a cluster that already has a local pool defined, the join wizard asks for source and zfs.pool_name properties. However, it attempts to set these as cluster-wide config, which Incus rejects because they are member-specific keys:

Error: Failed to join cluster: Failed to initialize member: Failed to
initialize storage pools and networks: Failed to update storage pool
"local": Config key "zfs.pool_name" is cluster member specific

The fix: delete pool and network before joining

This must be done for each joining node (not the init node).

NODE=incus-lab-02   # repeat with incus-lab-03

# 1. Remove server config references to the pool
incus config unset "$NODE": storage.backups_volume
incus config unset "$NODE": storage.images_volume

# 2. Delete the backup and image custom volumes
incus storage volume delete "$NODE":local backups
incus storage volume delete "$NODE":local images

# 3. Remove default profile device references (pool shows "in use" otherwise)
incus profile device remove "$NODE":default root
incus profile device remove "$NODE":default eth0

# 4. Now delete the pool and network
incus storage delete "$NODE":local
incus network delete "$NODE":incusbr0

The underlying ZFS dataset (local/incus) still exists on disk -- only the Incus metadata is removed. The join process will re-register it.

Step 4: Join Nodes

Interactive method

# Generate a join token (on the cluster, for the new member name)
incus cluster add incus-lab-01:incus-lab-02

# Join (prompts for 5 values)
incus cluster join incus-lab-01: incus-lab-02:

The interactive prompts and correct answers:

#	Prompt	Answer
1	IP address or DNS name [default=192.168.1.159]	Press Enter (accept default)
2	Member name [default=incus-lab-02]	Press Enter (accept default)
3	All existing data is lost, continue? [default=no]	`yes`
4	Choose "source" property for storage pool "local"	`local/incus`
5	Choose "zfs.pool_name" property for storage pool "local"	`local/incus`

Automated (non-interactive) method

# Generate token
incus cluster add incus-lab-01:incus-lab-02

# Join with piped answers (two empty lines = accept defaults for IP + name)
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:

Fix the remote after join

Same certificate issue as the init node -- the joining node now uses the cluster certificate:

incus remote remove incus-lab-02
incus remote add incus-lab-02 https://192.168.1.159:8443 --accept-certificate

Verify after each join

incus cluster list incus-lab-01:
incus info incus-lab-02: | head -5

Repeat for remaining nodes

# Prepare node 3 (same cleanup as step 3)
# ...

# Generate token + join
incus cluster add incus-lab-01:incus-lab-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-03:

# Fix remote
incus remote remove incus-lab-03
incus remote add incus-lab-03 https://192.168.1.78:8443 --accept-certificate

Step 5: Verify the Cluster

# All 3 members should be ONLINE
incus cluster list incus-lab-01:

# Check storage pool exists on all members
incus storage show incus-lab-01:local
incus storage show incus-lab-01:local --target incus-lab-01
incus storage show incus-lab-01:local --target incus-lab-02
incus storage show incus-lab-01:local --target incus-lab-03

# Network should be on all members
incus network list incus-lab-01:

# All individual remotes should work
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3

Quick Reference: Full Automated Script

For copy-paste clustering of a 3-node lab (replace IPs with your values):

IP1=192.168.1.250
IP2=192.168.1.159
IP3=192.168.1.78

# --- Set specific IPs ---
incus config set incus-lab-01: core.https_address "$IP1":8443
incus config set incus-lab-02: core.https_address "$IP2":8443
incus config set incus-lab-03: core.https_address "$IP3":8443

# --- Enable clustering on init node ---
incus cluster enable incus-lab-01: incus-lab-01

# Fix remote (switch default first if needed)
incus remote switch local
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://"$IP1":8443 --accept-certificate

# --- Prepare and join node 2 ---
incus config unset incus-lab-02: storage.backups_volume
incus config unset incus-lab-02: storage.images_volume
incus storage volume delete incus-lab-02:local backups
incus storage volume delete incus-lab-02:local images
incus profile device remove incus-lab-02:default root
incus profile device remove incus-lab-02:default eth0
incus storage delete incus-lab-02:local
incus network delete incus-lab-02:incusbr0

incus cluster add incus-lab-01:incus-lab-02
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:

incus remote remove incus-lab-02
incus remote add incus-lab-02 https://"$IP2":8443 --accept-certificate

# --- Prepare and join node 3 ---
incus config unset incus-lab-03: storage.backups_volume
incus config unset incus-lab-03: storage.images_volume
incus storage volume delete incus-lab-03:local backups
incus storage volume delete incus-lab-03:local images
incus profile device remove incus-lab-03:default root
incus profile device remove incus-lab-03:default eth0
incus storage delete incus-lab-03:local
incus network delete incus-lab-03:incusbr0

incus cluster add incus-lab-01:incus-lab-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-03:

incus remote remove incus-lab-03
incus remote add incus-lab-03 https://"$IP3":8443 --accept-certificate

# --- Verify ---
incus cluster list incus-lab-01:

Command Syntax Reference

Command	Arguments	Notes
`incus cluster enable`	`remote: member-name`	TWO args: remote: + name
`incus cluster add`	`remote:member-name`	ONE arg, no space after colon
`incus cluster join`	`init-remote: joining-remote:`	TWO args, space between
`incus cluster list`	`remote:`	Trailing colon = target server
`incus config set`	`remote: key value`	Trailing colon + space + key
`incus storage show`	`remote:pool`	ONE arg, no space after colon
`incus storage show`	`remote:pool --target member`	Member-specific config
`incus profile device remove`	`remote:profile device`	ONE arg + device name
`incus cluster remove`	`remote:member-name --force`	ONE arg; prompts "yes/no" even with --force
`incus cluster evacuate`	`remote:member-name`	ONE arg, no space after colon
`incus cluster restore`	`remote:member-name`	ONE arg, no space after colon

General rule: remote:resource targets a resource on the remote. remote: (trailing colon, no resource) targets the server itself.

Troubleshooting

Symptom	Cause	Fix
`certificate is valid for 127.0.0.1, ::1, not <IP>`	Cluster cert regenerated	Remove and re-add remote with `--accept-certificate`
`Config key "zfs.pool_name" is cluster member specific`	Pool already exists on joining node	Delete pool/network before joining (step 3)
`Config key "source" is cluster member specific`	Same as above	Delete pool/network before joining (step 3)
`The storage pool is currently in use`	Default profile still references pool	Remove `root` and `eth0` from default profile first
`Invalid number of arguments` on `cluster add`	Space between remote and member name	Use `remote:member` not `remote: member`
Join hangs / EOF errors	Interactive prompts in non-interactive shell	Use `printf` pipe method
Token expired	Tokens are time-limited	Generate a fresh token immediately before joining
Node shows in cluster with wrong URL	`core.https_address` was wildcard	Set specific IP before enabling clustering

What Happens Under the Hood

Certificate lifecycle

Pre-clustering: each node has its own self-signed server certificate, generated at install time. Client remotes pin this cert's fingerprint.
Cluster enable: the init node generates a new cluster certificate (shared across all members). The old server cert is replaced. Client remotes must re-pin.
Node join: the joining node receives the cluster certificate from the init node. Its old server cert is replaced. Client remotes must re-pin.

Storage pool lifecycle during join

Pre-join: each node has its own standalone local ZFS pool with source local/incus (a ZFS dataset on the IncusOS system disk).
Delete before join: we remove the Incus metadata (pool definition, volumes, profile references). The underlying ZFS dataset remains on disk.
During join: the cluster tells the new member "you need a local pool". The join wizard asks for the source (local/incus) and creates a member-specific pool entry pointing to the existing ZFS dataset.
Post-join: the pool shows as Created on all members with matching names but potentially different underlying sources (member-specific config).

Why we delete the network too

The incusbr0 network bridge follows the same pattern as the storage pool. If it already exists on the joining node, the join process may conflict. Deleting it lets the cluster recreate it with the correct member-specific configuration.

Cluster Workload Testing

After forming the cluster, validate it by launching workloads, verifying cluster-wide visibility, and testing migration.

Launch a container on a specific node

# Launch Debian 12 container on node 2
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02

# Verify it's running and check placement
incus list incus-lab-01: --columns ntsLl4

# Exec into it
incus exec incus-lab-01:test-container -- cat /etc/os-release | head -2

Verify cluster-wide visibility

All cluster members should see the same instance list, regardless of which remote you query:

incus list incus-lab-01: --columns nLs
incus list incus-lab-02: --columns nLs
incus list incus-lab-03: --columns nLs

Container migration (stop/move/start)

Containers do not support live migration (CRIU is unreliable). Use stop/move/start:

# Check current location
incus list incus-lab-01: --columns nLs

# Migrate: stop, move to new node, start
incus stop incus-lab-01:test-container
incus move incus-lab-01:test-container --target incus-lab-01
incus start incus-lab-01:test-container

# Verify new location
incus list incus-lab-01: --columns nLs

What survives migration:

All persistent filesystem data (root disk contents)
Instance configuration

What does NOT survive:

Running processes (stopped with the container)
tmpfs contents (/tmp, /run)
In-memory state

Launch a VM on a specific node

# Launch Debian 12 VM on node 2
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02

# VMs take ~10-15s to boot — wait for agent
incus exec incus-lab-01:test-vm -- hostname   # retry until it works

# Verify
incus list incus-lab-01: --columns ntsLl4
incus exec incus-lab-01:test-vm -- cat /etc/os-release | head -2

VM migration

Live migration

Live migration preserves running state (no downtime). It works across heterogeneous hosts (different core counts) as long as limits.cpu is set correctly — see "VM Live Migration: The limits.cpu Fix" below.

# 1. Enable stateful migration (must be set while VM is stopped)
incus stop incus-lab-01:test-vm
incus config set incus-lab-01:test-vm migration.stateful=true

# 2. Set limits.cpu as a RANGE (critical for migration!)
incus config set incus-lab-01:test-vm limits.cpu=0-1

# 3. Add size.state to root disk (required for stateful operations)
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB

# 4. Start the VM
incus start incus-lab-01:test-vm

# 5. Live-migrate (no stop needed -- running state is preserved!)
incus move incus-lab-01:test-vm --target incus-lab-03

What survives live migration (unlike stop/move/start):

All persistent filesystem data
Running processes (PID state preserved)
tmpfs contents (/tmp, /run)
Network connections
In-memory state (full RAM checkpoint)

Stop/move/start migration (always works)

incus stop incus-lab-01:test-vm
incus move incus-lab-01:test-vm --target incus-lab-01
incus start incus-lab-01:test-vm

Same persistence rules as containers: filesystem data survives, running processes and tmpfs do not.

Cluster evacuation and restore

Evacuation drains all workloads from a node (e.g., for maintenance). Incus automatically selects the right strategy per workload type:

# Evacuate a node (--force skips confirmation prompt)
incus cluster evacuate incus-lab-01:incus-lab-02 --force

# Check: node shows EVACUATED, instances moved to other nodes
incus cluster list incus-lab-01:
incus list incus-lab-01: --columns nLs

# Restore: moves workloads back to original node
incus cluster restore incus-lab-01:incus-lab-02 --force

# Verify: node back ONLINE, instances restored
incus cluster list incus-lab-01:
incus list incus-lab-01: --columns nLs

Syntax note: incus cluster evacuate remote:member is ONE argument (like cluster enable and cluster add).

What happens during evacuation:

VMs with migration.stateful=true: live migration (state preserved)
VMs without stateful: stop/move/start
Containers: stop/move/start
Use --action stop to force stop/move for all workloads (useful on heterogeneous clusters without limits.cpu range fix)

Cleanup test instances

incus delete incus-lab-01:test-container --force
incus delete incus-lab-01:test-vm --force
incus list incus-lab-01:   # should be empty

VM Live Migration: The `limits.cpu` Fix

The problem

By default, live migration fails between cluster nodes with different host CPU counts:

qemu-system-x86_64: Missing section footer for 0000:00:1f.0/ICH9LPC
qemu-system-x86_64: load of migration failed: Invalid argument

Or, with limits.cpu set as an integer:

qemu-system-x86_64: Unknown section or instance 'apic' 3

Root cause: Incus sets QEMU `maxcpus` from host CPU count

In the Incus source (driver_qemu_templates.go), the qemuCPU function generates the QEMU -smp configuration. When limits.cpu is unset or set to a plain integer, it does:

maxCpus := 64
if int(cpu.Total) < maxCpus {
    maxCpus = int(cpu.Total)   // <-- host CPU count
}

This means:

On a 4-core host: QEMU gets maxcpus=4 → cpu possible: 0-3
On a 2-core host: QEMU gets maxcpus=2 → cpu possible: 0-1

The ICH9 ACPI CPU hotplug state (hw/acpi/cpu.c) is a variable-length array sized by maxcpus. When source and destination have different maxcpus, the vmstate sizes don't match and QEMU can't deserialize the migration stream.

How we discovered this

We observed the VM's cpu possible differing by host:

# On 4-core host:
cat /sys/devices/system/cpu/possible  → 0-3

# On 2-core host:
cat /sys/devices/system/cpu/possible  → 0-1

Even though the VM only uses 1 vCPU, QEMU allocates hotplug slots for all "possible" CPUs.

The fix: use `limits.cpu` as a range

When limits.cpu is set to a range (e.g., 0-1), Incus takes the CPU pinning code path, which generates a fixed topology with no maxcpus:

# Integer (broken): maxcpus varies by host
[smp-opts]
cpus = 2
maxcpus = 4     # on 4-core host

# Range (fixed): no maxcpus, deterministic topology
[smp-opts]
cpus = 2
sockets = 1
cores = 2
threads = 1

The one-line fix:

# Set limits.cpu as a range (while VM is stopped)
incus config set <instance> limits.cpu=0-1    # for 2 vCPUs
incus config set <instance> limits.cpu=0-3    # for 4 vCPUs

Configuration summary

`limits.cpu` value	QEMU `-smp` behavior	Migration across heterogeneous hosts
(not set)	`maxcpus = host_cores`	Fails
`2` (integer)	`maxcpus = host_cores`	Fails
`0-1` (range)	Fixed `sockets/cores/threads`, no `maxcpus`	Works

Alternative: `raw.qemu.conf` override

If you need maxcpus (e.g., for CPU hotplug), force it to a fixed value:

incus config set <instance> raw.qemu.conf='[smp-opts]
maxcpus = 2'

Do not combine this with the range syntax — they are mutually exclusive.

Complete setup for a live-migratable VM

# Create and configure (all while stopped)
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
incus stop incus-lab-01:test-vm

incus config set incus-lab-01:test-vm migration.stateful=true
incus config set incus-lab-01:test-vm limits.cpu=0-1
incus config device add incus-lab-01:test-vm root disk \
  path=/ pool=local size.state=2GiB

incus start incus-lab-01:test-vm

# Now live migration works across any cluster node
incus move incus-lab-01:test-vm --target incus-lab-03

Migration Test Results

Tested with a 3-node cluster on Proxmox VE (Intel i9-13900HK, nested VT-x):

incus-lab-01: 4 vCPUs, 8192 MB
incus-lab-02: 2 vCPUs, 4096 MB
incus-lab-03: 2 vCPUs, 4096 MB

Test VM: Debian 12, limits.cpu=0-1, migration.stateful=true, size.state=2GiB. Heartbeat process (1/s counter to tmpfs) verifies full state preservation.

#	Test	Time	Result
1	Live: node1→node2 (4-core host → 2-core host)	7.4s	PASS
2	Live: node2→node3 (2-core host → 2-core host)	7.2s	PASS
3	Live: node3→node1 (2-core host → 4-core host)	7.5s	PASS
4	Rapid round-trip: 1→2→3→1 (back-to-back)	~22s	PASS
5	Stateful stop/restore (same node)	7.6s	PASS
6	Stateful stop/move/restore (node1→node2, cross-topology)	10.0s	PASS
7a	Cluster evacuate node2 (container + VM)	9.0s	PASS
7b	Cluster restore node2 (moves workloads back)	9.7s	PASS

Key observations:

Transfer speed: ~140 MB/s for ~1 GB VM
Heartbeat counter continuous through all migrations (no lost seconds)
tmpfs, systemd services, and VM uptime all preserved through live migration
VM agent (incus exec) needs ~3-4 seconds to reconnect after migration
Cluster evacuation auto-selects strategy: live migration for VMs, stop/move/start for containers
Cluster restore returns workloads to original node

Automated migration test

Run this from your workstation to test all migration paths:

REMOTE=incus-lab-01   # any cluster member
VM=test-vm

# Setup: create heartbeat inside VM
incus exec "$REMOTE:$VM" -- mkdir -p /tmp/migration-test
incus exec "$REMOTE:$VM" -- bash -c \
  'echo "created-$(hostname)-$(date +%s)" > /tmp/migration-test/marker'
incus exec "$REMOTE:$VM" -- bash -c \
  'cat > /tmp/migration-test/heartbeat.sh << "EOF"
#!/bin/bash
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
EOF
chmod +x /tmp/migration-test/heartbeat.sh'
incus exec "$REMOTE:$VM" -- \
  systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh

sleep 3

# Test: live-migrate through all nodes
PASS=0; FAIL=0
for TARGET in incus-lab-02 incus-lab-03 incus-lab-01; do
  PRE=$(incus exec "$REMOTE:$VM" -- cat /tmp/migration-test/heartbeat)
  LOC=$(incus list "$REMOTE:" --format csv -c nL | grep "$VM" | cut -d, -f2)
  echo -n "[$LOC → $TARGET] heartbeat=$PRE ... "

  incus move "$REMOTE:$VM" --target "$TARGET" 2>&1 | tail -1
  sleep 2  # wait for agent reconnect

  POST=$(incus exec "$REMOTE:$VM" -- cat /tmp/migration-test/heartbeat 2>&1)
  SVC=$(incus exec "$REMOTE:$VM" -- systemctl is-active migration-heartbeat 2>&1)
  if [[ "$SVC" == "active" && "$POST" -gt "$PRE" ]]; then
    echo "PASS (heartbeat=$POST, service=$SVC)"
    PASS=$((PASS + 1))
  else
    echo "FAIL (heartbeat=$POST, service=$SVC)"
    FAIL=$((FAIL + 1))
  fi
done

# Cleanup heartbeat
incus exec "$REMOTE:$VM" -- systemctl stop migration-heartbeat 2>/dev/null

echo ""
echo "Results: $PASS passed, $FAIL failed"

The vnmi warning is cosmetic

The CPUID[eax=8000000Ah].EDX.vnmi warning appears because QEMU's CPU feature dependency checker fires before KVM filters out unsupported features. It does not cause migration failures and can be safely ignored.

Nested virtualization works

Live migration works fine in nested virtualization (IncusOS inside Proxmox on Intel VT-x). Tested with QEMU 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2). The limits.cpu range fix makes host topology irrelevant.

Debugging tips

incus start --stateless discards a saved state file if a restore fails, allowing a clean boot.
Check cpu possible inside the VM: cat /sys/devices/system/cpu/possible. If this differs across nodes for the same VM config, migration will fail.
incus monitor --type=logging on the destination shows QEMU errors.

References

Incus source: driver_qemu_templates.go — maxcpus calculation
QEMU source: hw/acpi/cpu.c — CPU hotplug state sizing
QEMU source: hw/isa/lpc_ich9.c — ICH9 vmstate subsections
Incus Instance Options — limits.cpu, migration.stateful
Incus issue #486 — cluster-wide CPU baseline

Multi-vCPU Migration Test Results

Tested with a 3-node cluster (6/4/4 cores, 8 GiB RAM each) to validate migration across different vCPU counts and heterogeneous host topologies.

Test environment

incus-adv-01: 6 vCPUs, 8192 MB
incus-adv-02: 4 vCPUs, 8192 MB
incus-adv-03/04: 4 vCPUs, 8192 MB

Three test VMs, all Debian 12, migration.stateful=true, size.state=4GiB:

VM	`limits.cpu`	vCPUs	`cpu possible`
test-vm-2cpu	`0-1`	2	`0-1`
test-vm-3cpu	`0-2`	3	`0-2`
test-vm-4cpu	`0-3`	4	`0-3`

Results

#	Test	Time	Result
1	2-vCPU full ring (3 migrations)	~7s each	PASS
2	3-vCPU full ring (3 migrations)	~7.5s each	PASS
3	4-vCPU full ring (3 migrations)	~7.3s each	PASS
4	Concurrent: 2 VMs from different nodes	8s total	PASS
5	4-vCPU VM to 4-core host (100% core usage)	7.3s	PASS
6	Rapid round-trip: 3 hops back-to-back	~31s total	PASS
7	Active disk I/O (dd 100 MiB) during migration	8.2s	PASS
8	Active network (ping) during migration	8.5s	PASS
9	Stateful stop/move/start (cross-node)	10s	PASS
10	Memory-loaded VM (512 MiB allocated)	7.7s	PASS
11	Move to same node (no-op)	instant	Clean error
12	Mixed evacuation (2 containers + 2 VMs)	18s	PASS
13	Restore after evacuation	18s	PASS

Key findings

Odd vCPU counts work: limits.cpu=0-2 (3 vCPUs) migrates just as reliably as even counts. The range syntax always produces deterministic QEMU topology regardless of count.
4-vCPU on 4-core host works: a VM using all host cores migrates successfully and remains responsive. No performance degradation observed.
Concurrent migrations: migrating 2 VMs simultaneously from different source nodes completes without interference. Transfer speed is maintained at ~140 MB/s per migration.
Active I/O survives: both disk writes (dd) and network activity (ping) continue uninterrupted through live migration. File integrity verified.
Memory-loaded migration: 512 MiB of allocated memory does not significantly increase migration time (QEMU's iterative pre-copy handles dirty pages efficiently).
Agent reconnect time is ~3-4 seconds, not 1-2s as initially measured. May vary with cluster size and vCPU count. Scripts should sleep 4 after migration before running incus exec.
Same-node move: Incus returns a clear error ("Requested target server is the same as current server") — no crash or corruption.
size.state=4GiB recommended for VMs with 3-4 vCPUs to ensure sufficient room for state dumps.

Cluster Lifecycle: Node Replacement

Full procedure for evacuating, removing, destroying, and replacing a cluster node. Tested on the 3-node advanced cluster.

Overview

Evacuate the target node (drains workloads)
Remove the node from the cluster
Destroy the VM in Proxmox
Deploy a fresh replacement node
Join the replacement to the cluster
Verify workload rebalancing

Step 1: Evacuate

REMOTE=incus-adv-01   # init node (not the target)
incus cluster evacuate $REMOTE:incus-adv-03 --force

This live-migrates VMs (with migration.stateful=true) and stop/move/starts containers. Verify with incus cluster list — target shows EVACUATED.

Step 2: Remove from cluster

printf "yes\n" | incus cluster remove $REMOTE:incus-adv-03 --force

Note: --force requires interactive confirmation ("yes"). Pipe it in for automation. The --force flag is needed because the node is still technically reachable (just evacuated).

After removal, incus cluster list shows only the remaining members.

Step 3: Destroy the VM

Use incusos-proxmox --cleanup with a config targeting just that VM, or use the Proxmox API directly. Do NOT try raw curl with API tokens containing ! characters — bash history expansion will silently mangle the token. Always use single quotes or the incusos-proxmox script.

# Method 1: incusos-proxmox (recommended)
incusos-proxmox --cleanup --yes <config-with-target-vm.yaml>

# Method 2: Proxmox API (note: single quotes around the token!)
TOKEN='automation@pve!deploy=<secret>'
curl -s -k -X POST -H "Authorization: PVEAPIToken=$TOKEN" \
  "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/stop"

Also remove the incus remote:

incus remote switch local       # if target is current default
incus remote remove incus-adv-03

Step 4: Deploy replacement

Create a config for the replacement node (e.g., lab-replace.yaml):

defaults:
  memory: 8192
  disk: 50
  start_vmid: 913

vms:
  - name: incus-adv-04
    app: incus
    apply_defaults: true
    cores: 4

incusos-proxmox --yes lab-replace.yaml

Step 5: Join to cluster

Same procedure as initial cluster formation:

NEW_IP=<replacement-node-ip>

# Set specific address
incus config set incus-adv-04: core.https_address $NEW_IP:8443

# Prepare (delete pre-existing pool/network)
incus config unset incus-adv-04: storage.backups_volume
incus config unset incus-adv-04: storage.images_volume
incus storage volume delete incus-adv-04:local backups
incus storage volume delete incus-adv-04:local images
incus profile device remove incus-adv-04:default root
incus profile device remove incus-adv-04:default eth0
incus storage delete incus-adv-04:local
incus network delete incus-adv-04:incusbr0

# Generate token and join
incus cluster add $REMOTE:incus-adv-04
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join $REMOTE: incus-adv-04:

# Fix remote (new cluster cert)
incus remote remove incus-adv-04
incus remote add incus-adv-04 https://$NEW_IP:8443 --accept-certificate

Step 6: Verify rebalancing

With cluster.rebalance.interval configured, Incus automatically migrates VMs to the new (empty) node:

# Check after 1-2 minutes
incus list $REMOTE: --columns ntsL

In testing, 2 VMs were automatically live-migrated to the replacement node within 90 seconds. Containers are NOT auto-rebalanced (Incus limitation) — move them manually with stop/move/start.

Rebalancing configuration

REMOTE=incus-adv-01
incus config set $REMOTE: cluster.rebalance.interval=1     # check every N minutes
incus config set $REMOTE: cluster.rebalance.threshold=10    # imbalance % to trigger
incus config set $REMOTE: cluster.rebalance.batch=2         # max VMs per rebalance run
incus config set $REMOTE: cluster.rebalance.cooldown=5m     # wait between runs

Key	Description	Tested value
`cluster.rebalance.interval`	Check interval in minutes	`1` (aggressive for testing)
`cluster.rebalance.threshold`	Imbalance percentage to trigger	`10`
`cluster.rebalance.batch`	Max VMs moved per run	`2`
`cluster.rebalance.cooldown`	Minimum time between runs	`5m`

Important: rebalancing only moves VMs with migration.stateful=true. Containers and non-migratable VMs are not touched.

Command syntax: `cluster remove`

# Works (pipe the confirmation)
printf "yes\n" | incus cluster remove remote:member --force

# Also note: cluster enable syntax is TWO args, not ONE
incus cluster enable remote: member-name    # correct
incus cluster enable remote:member-name     # WRONG (single arg)

34 KiB Raw Blame History

Incus Clustering Guide for IncusOS (via Remotes)

Overview

Prerequisites

Step 1: Set Specific IP Addresses

Why this is needed

How to discover node IPs

Set the addresses

Verify connectivity after each change

What happens under the hood

Step 2: Enable Clustering on the Init Node

The TLS certificate problem

The fix: re-add the remote

Verify

Step 3: Prepare Joining Nodes

Recommended: deploy joining nodes with apply_defaults: false

Alternative: cleanup when apply_defaults: true was used

The fix: delete pool and network before joining

Step 4: Join Nodes

Interactive method

Automated (non-interactive) method

Fix the remote after join

Verify after each join

Repeat for remaining nodes

Step 5: Verify the Cluster

Quick Reference: Full Automated Script

Command Syntax Reference

Troubleshooting

What Happens Under the Hood

Certificate lifecycle

Storage pool lifecycle during join

Why we delete the network too

Cluster Workload Testing

Launch a container on a specific node

Verify cluster-wide visibility

Container migration (stop/move/start)

Launch a VM on a specific node

VM migration

Live migration

Stop/move/start migration (always works)

Cluster evacuation and restore

Cleanup test instances

VM Live Migration: The limits.cpu Fix

The problem

Root cause: Incus sets QEMU maxcpus from host CPU count

How we discovered this

The fix: use limits.cpu as a range

Configuration summary

Alternative: raw.qemu.conf override

Complete setup for a live-migratable VM

Migration Test Results

Automated migration test

The vnmi warning is cosmetic

Nested virtualization works

Debugging tips

References

Multi-vCPU Migration Test Results

Test environment

Results

Key findings

Cluster Lifecycle: Node Replacement

Overview

Step 1: Evacuate

Step 2: Remove from cluster

Step 3: Destroy the VM

Step 4: Deploy replacement

Step 5: Join to cluster

Step 6: Verify rebalancing

Rebalancing configuration

Command syntax: cluster remove

34 KiB

Raw Blame History

VM Live Migration: The `limits.cpu` Fix

Root cause: Incus sets QEMU `maxcpus` from host CPU count

The fix: use `limits.cpu` as a range

Alternative: `raw.qemu.conf` override

Command syntax: `cluster remove`