incus-contrib/notes/clustering-guide.md

# Incus Clustering Guide for IncusOS (via Remotes)

This guide covers forming an Incus cluster from IncusOS nodes using only the
`incus` CLI and remotes. No SSH access to the nodes is needed (IncusOS is
immutable and has no shell).

Tested with Incus client 6.21 and IncusOS nodes deployed via `incusos-proxmox`.

---

## Overview

Incus clustering does **not** require a virtual IP (VIP), load balancer, or
shared storage. Each node advertises its own IP address. Clients can connect
to any cluster member -- requests are forwarded internally via the cluster's
internal communication channel.

**Cluster roles:**
- **database-leader**: holds the primary Dqlite database
- **database**: database replica (for HA, need 3 database members)
- **database-standby**: standby replica (promoted automatically if a database
  member goes down)

With 3 nodes, you get a fully HA database quorum.

---

## Prerequisites

- Incus client version **6.20+** (required for `incus cluster join` via remotes)
- All nodes deployed with `apply_defaults: true` (matching pool/network names)
- Remotes configured for all nodes (`incus remote add <name> https://<ip>:8443`)
- Client certificate trusted on all nodes (injected via seed or added manually)

```bash
incus version                        # Verify 6.20+
incus info incus-lab-01: | head -5   # Verify remote connectivity
incus info incus-lab-02: | head -5
incus info incus-lab-03: | head -5
```

---

## Step 1: Set Specific IP Addresses

### Why this is needed

IncusOS nodes listen on `:8443` by default (all interfaces). Clustering
requires each node to have a specific routable IP so other members can
address it. Without this, `incus cluster enable` either fails or prompts
for an address.

### How to discover node IPs

```bash
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
  echo "=== $node ==="
  incus query "$node:/1.0" | python3 -c "
import sys, json
d = json.load(sys.stdin)
for a in d['environment']['addresses']:
    if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('['):
        print(a)
"
done
```

This filters out:
- `10.x.x.x` -- internal bridge addresses (`incusbr0`)
- `fd42:` -- ULA IPv6 bridge addresses
- `[...]` -- IPv6 addresses in bracket notation

### Set the addresses

```bash
incus config set incus-lab-01: core.https_address 192.168.1.250:8443
incus config set incus-lab-02: core.https_address 192.168.1.159:8443
incus config set incus-lab-03: core.https_address 192.168.1.78:8443
```

### Verify connectivity after each change

```bash
incus config get incus-lab-01: core.https_address   # Should show IP:8443
incus info incus-lab-01: | head -3                   # Should still connect
```

### What happens under the hood

The Incus daemon rebinds its HTTPS listener from `0.0.0.0:8443` to the
specific IP. Since the client remote was already connecting to that IP, the
connection continues to work seamlessly. Certificate trust is stored by
fingerprint in the Incus database, independent of the listen address.

---

## Step 2: Enable Clustering on the Init Node

```bash
incus cluster enable incus-lab-01: incus-lab-01
```

**Syntax note**: this is TWO arguments -- `remote:` (trailing colon, targeting
the server) and `member-name` (the cluster member name). The help text shows
`[<remote>:] <name>`. Using `remote:name` as a single arg causes "The incus
daemon doesn't appear to be started" on client-only systems.

### The TLS certificate problem

Enabling clustering causes the server to generate a **new TLS certificate**
(the cluster certificate). This new cert typically only has SANs (Subject
Alternative Names) for `127.0.0.1` and `::1`, not the node's actual IP.

Your existing remote has the old (pre-clustering) certificate pinned, so
the next connection fails:

```
Error: tls: failed to verify certificate: x509: certificate is valid for
127.0.0.1, ::1, not 192.168.1.250
```

### The fix: re-add the remote

```bash
# Switch default remote first (can't remove the current default)
incus remote switch local

# Remove and re-add with the new certificate
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://192.168.1.250:8443 --accept-certificate
```

Incus remotes use **fingerprint pinning** (not CA-based SAN validation), so
`--accept-certificate` pins the new cluster cert and future connections work
despite the SAN mismatch.

### Verify

```bash
incus cluster list incus-lab-01:
```

Expected output: one member (`incus-lab-01`), status `ONLINE`, role
`database-leader` + `database`.

---

## Step 3: Prepare Joining Nodes

### The apply_defaults storage pool conflict

Nodes deployed with `apply_defaults: true` already have:
- A `local` ZFS storage pool (source: `local/incus`)
- An `incusbr0` network bridge
- A `default` profile referencing both

When joining a cluster that already has a `local` pool defined, the join
wizard asks for `source` and `zfs.pool_name` properties. However, it
attempts to set these as **cluster-wide** config, which Incus rejects because
they are **member-specific** keys:

```
Error: Failed to join cluster: Failed to initialize member: Failed to
initialize storage pools and networks: Failed to update storage pool
"local": Config key "zfs.pool_name" is cluster member specific
```

### The fix: delete pool and network before joining

This must be done for **each joining node** (not the init node).

```bash
NODE=incus-lab-02   # repeat with incus-lab-03

# 1. Remove server config references to the pool
incus config unset "$NODE": storage.backups_volume
incus config unset "$NODE": storage.images_volume

# 2. Delete the backup and image custom volumes
incus storage volume delete "$NODE":local backups
incus storage volume delete "$NODE":local images

# 3. Remove default profile device references (pool shows "in use" otherwise)
incus profile device remove "$NODE":default root
incus profile device remove "$NODE":default eth0

# 4. Now delete the pool and network
incus storage delete "$NODE":local
incus network delete "$NODE":incusbr0
```

The underlying ZFS dataset (`local/incus`) still exists on disk -- only the
Incus metadata is removed. The join process will re-register it.

---

## Step 4: Join Nodes

### Interactive method

```bash
# Generate a join token (on the cluster, for the new member name)
incus cluster add incus-lab-01:incus-lab-02

# Join (prompts for 5 values)
incus cluster join incus-lab-01: incus-lab-02:
```

The interactive prompts and correct answers:

| # | Prompt | Answer |
|---|--------|--------|
| 1 | IP address or DNS name [default=192.168.1.159] | Press Enter (accept default) |
| 2 | Member name [default=incus-lab-02] | Press Enter (accept default) |
| 3 | All existing data is lost, continue? [default=no] | `yes` |
| 4 | Choose "source" property for storage pool "local" | `local/incus` |
| 5 | Choose "zfs.pool_name" property for storage pool "local" | `local/incus` |

### Automated (non-interactive) method

```bash
# Generate token
incus cluster add incus-lab-01:incus-lab-02

# Join with piped answers (two empty lines = accept defaults for IP + name)
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:
```

### Fix the remote after join

Same certificate issue as the init node -- the joining node now uses the
cluster certificate:

```bash
incus remote remove incus-lab-02
incus remote add incus-lab-02 https://192.168.1.159:8443 --accept-certificate
```

### Verify after each join

```bash
incus cluster list incus-lab-01:
incus info incus-lab-02: | head -5
```

### Repeat for remaining nodes

```bash
# Prepare node 3 (same cleanup as step 3)
# ...

# Generate token + join
incus cluster add incus-lab-01:incus-lab-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-03:

# Fix remote
incus remote remove incus-lab-03
incus remote add incus-lab-03 https://192.168.1.78:8443 --accept-certificate
```

---

## Step 5: Verify the Cluster

```bash
# All 3 members should be ONLINE
incus cluster list incus-lab-01:

# Check storage pool exists on all members
incus storage show incus-lab-01:local
incus storage show incus-lab-01:local --target incus-lab-01
incus storage show incus-lab-01:local --target incus-lab-02
incus storage show incus-lab-01:local --target incus-lab-03

# Network should be on all members
incus network list incus-lab-01:

# All individual remotes should work
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3
```

---

## Quick Reference: Full Automated Script

For copy-paste clustering of a 3-node lab (replace IPs with your values):

```bash
IP1=192.168.1.250
IP2=192.168.1.159
IP3=192.168.1.78

# --- Set specific IPs ---
incus config set incus-lab-01: core.https_address "$IP1":8443
incus config set incus-lab-02: core.https_address "$IP2":8443
incus config set incus-lab-03: core.https_address "$IP3":8443

# --- Enable clustering on init node ---
incus cluster enable incus-lab-01: incus-lab-01

# Fix remote (switch default first if needed)
incus remote switch local
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://"$IP1":8443 --accept-certificate

# --- Prepare and join node 2 ---
incus config unset incus-lab-02: storage.backups_volume
incus config unset incus-lab-02: storage.images_volume
incus storage volume delete incus-lab-02:local backups
incus storage volume delete incus-lab-02:local images
incus profile device remove incus-lab-02:default root
incus profile device remove incus-lab-02:default eth0
incus storage delete incus-lab-02:local
incus network delete incus-lab-02:incusbr0

incus cluster add incus-lab-01:incus-lab-02
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:

incus remote remove incus-lab-02
incus remote add incus-lab-02 https://"$IP2":8443 --accept-certificate

# --- Prepare and join node 3 ---
incus config unset incus-lab-03: storage.backups_volume
incus config unset incus-lab-03: storage.images_volume
incus storage volume delete incus-lab-03:local backups
incus storage volume delete incus-lab-03:local images
incus profile device remove incus-lab-03:default root
incus profile device remove incus-lab-03:default eth0
incus storage delete incus-lab-03:local
incus network delete incus-lab-03:incusbr0

incus cluster add incus-lab-01:incus-lab-03
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-03:

incus remote remove incus-lab-03
incus remote add incus-lab-03 https://"$IP3":8443 --accept-certificate

# --- Verify ---
incus cluster list incus-lab-01:
```

---

## Command Syntax Reference

| Command | Arguments | Notes |
|---------|-----------|-------|
| `incus cluster enable` | `remote: member-name` | TWO args: remote: + name |
| `incus cluster add` | `remote:member-name` | ONE arg, no space after colon |
| `incus cluster join` | `init-remote: joining-remote:` | TWO args, space between |
| `incus cluster list` | `remote:` | Trailing colon = target server |
| `incus config set` | `remote: key value` | Trailing colon + space + key |
| `incus storage show` | `remote:pool` | ONE arg, no space after colon |
| `incus storage show` | `remote:pool --target member` | Member-specific config |
| `incus profile device remove` | `remote:profile device` | ONE arg + device name |
| `incus cluster remove` | `remote:member-name --force` | ONE arg; prompts "yes/no" even with --force |
| `incus cluster evacuate` | `remote:member-name` | ONE arg, no space after colon |
| `incus cluster restore` | `remote:member-name` | ONE arg, no space after colon |

**General rule**: `remote:resource` targets a resource on the remote.
`remote:` (trailing colon, no resource) targets the server itself.

---

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| `certificate is valid for 127.0.0.1, ::1, not <IP>` | Cluster cert regenerated | Remove and re-add remote with `--accept-certificate` |
| `Config key "zfs.pool_name" is cluster member specific` | Pool already exists on joining node | Delete pool/network before joining (step 3) |
| `Config key "source" is cluster member specific` | Same as above | Delete pool/network before joining (step 3) |
| `The storage pool is currently in use` | Default profile still references pool | Remove `root` and `eth0` from default profile first |
| `Invalid number of arguments` on `cluster add` | Space between remote and member name | Use `remote:member` not `remote: member` |
| Join hangs / EOF errors | Interactive prompts in non-interactive shell | Use `printf` pipe method |
| Token expired | Tokens are time-limited | Generate a fresh token immediately before joining |
| Node shows in cluster with wrong URL | `core.https_address` was wildcard | Set specific IP before enabling clustering |

---

## What Happens Under the Hood

### Certificate lifecycle

1. **Pre-clustering**: each node has its own self-signed server certificate,
   generated at install time. Client remotes pin this cert's fingerprint.
2. **Cluster enable**: the init node generates a new **cluster certificate**
   (shared across all members). The old server cert is replaced. Client
   remotes must re-pin.
3. **Node join**: the joining node receives the cluster certificate from the
   init node. Its old server cert is replaced. Client remotes must re-pin.

### Storage pool lifecycle during join

1. **Pre-join**: each node has its own standalone `local` ZFS pool with
   source `local/incus` (a ZFS dataset on the IncusOS system disk).
2. **Delete before join**: we remove the Incus metadata (pool definition,
   volumes, profile references). The underlying ZFS dataset remains on disk.
3. **During join**: the cluster tells the new member "you need a `local` pool".
   The join wizard asks for the source (`local/incus`) and creates a
   **member-specific** pool entry pointing to the existing ZFS dataset.
4. **Post-join**: the pool shows as `Created` on all members with matching
   names but potentially different underlying sources (member-specific config).

### Why we delete the network too

The `incusbr0` network bridge follows the same pattern as the storage pool.
If it already exists on the joining node, the join process may conflict.
Deleting it lets the cluster recreate it with the correct member-specific
configuration.

---

## Cluster Workload Testing

After forming the cluster, validate it by launching workloads, verifying
cluster-wide visibility, and testing migration.

### Launch a container on a specific node

```bash
# Launch Debian 12 container on node 2
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02

# Verify it's running and check placement
incus list incus-lab-01: --columns ntsLl4

# Exec into it
incus exec incus-lab-01:test-container -- cat /etc/os-release | head -2
```

### Verify cluster-wide visibility

All cluster members should see the same instance list, regardless of which
remote you query:

```bash
incus list incus-lab-01: --columns nLs
incus list incus-lab-02: --columns nLs
incus list incus-lab-03: --columns nLs
```

### Container migration (stop/move/start)

Containers do **not** support live migration (CRIU is unreliable). Use
stop/move/start:

```bash
# Check current location
incus list incus-lab-01: --columns nLs

# Migrate: stop, move to new node, start
incus stop incus-lab-01:test-container
incus move incus-lab-01:test-container --target incus-lab-01
incus start incus-lab-01:test-container

# Verify new location
incus list incus-lab-01: --columns nLs
```

**What survives migration:**
- All persistent filesystem data (root disk contents)
- Instance configuration

**What does NOT survive:**
- Running processes (stopped with the container)
- tmpfs contents (`/tmp`, `/run`)
- In-memory state

### Launch a VM on a specific node

```bash
# Launch Debian 12 VM on node 2
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02

# VMs take ~10-15s to boot — wait for agent
incus exec incus-lab-01:test-vm -- hostname   # retry until it works

# Verify
incus list incus-lab-01: --columns ntsLl4
incus exec incus-lab-01:test-vm -- cat /etc/os-release | head -2
```

### VM migration

#### Live migration

Live migration preserves running state (no downtime). It works across
heterogeneous hosts (different core counts) as long as `limits.cpu` is
set correctly — see "VM Live Migration: The `limits.cpu` Fix" below.

```bash
# 1. Enable stateful migration (must be set while VM is stopped)
incus stop incus-lab-01:test-vm
incus config set incus-lab-01:test-vm migration.stateful=true

# 2. Set limits.cpu as a RANGE (critical for migration!)
incus config set incus-lab-01:test-vm limits.cpu=0-1

# 3. Add size.state to root disk (required for stateful operations)
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB

# 4. Start the VM
incus start incus-lab-01:test-vm

# 5. Live-migrate (no stop needed -- running state is preserved!)
incus move incus-lab-01:test-vm --target incus-lab-03
```

**What survives live migration** (unlike stop/move/start):
- All persistent filesystem data
- Running processes (PID state preserved)
- tmpfs contents (`/tmp`, `/run`)
- Network connections
- In-memory state (full RAM checkpoint)

#### Stop/move/start migration (always works)

```bash
incus stop incus-lab-01:test-vm
incus move incus-lab-01:test-vm --target incus-lab-01
incus start incus-lab-01:test-vm
```

Same persistence rules as containers: filesystem data survives, running
processes and tmpfs do not.

### Cluster evacuation and restore

Evacuation drains all workloads from a node (e.g., for maintenance). Incus
automatically selects the right strategy per workload type:

```bash
# Evacuate a node (--force skips confirmation prompt)
incus cluster evacuate incus-lab-01:incus-lab-02 --force

# Check: node shows EVACUATED, instances moved to other nodes
incus cluster list incus-lab-01:
incus list incus-lab-01: --columns nLs

# Restore: moves workloads back to original node
incus cluster restore incus-lab-01:incus-lab-02 --force

# Verify: node back ONLINE, instances restored
incus cluster list incus-lab-01:
incus list incus-lab-01: --columns nLs
```

**Syntax note**: `incus cluster evacuate remote:member` is ONE argument (like
`cluster enable` and `cluster add`).

**What happens during evacuation:**
- VMs with `migration.stateful=true`: live migration (state preserved)
- VMs without stateful: stop/move/start
- Containers: stop/move/start
- Use `--action stop` to force stop/move for all workloads (useful on
  heterogeneous clusters without `limits.cpu` range fix)

### Cleanup test instances

```bash
incus delete incus-lab-01:test-container --force
incus delete incus-lab-01:test-vm --force
incus list incus-lab-01:   # should be empty
```

---

## VM Live Migration: The `limits.cpu` Fix

### The problem

By default, live migration fails between cluster nodes with different host
CPU counts:

```
qemu-system-x86_64: Missing section footer for 0000:00:1f.0/ICH9LPC
qemu-system-x86_64: load of migration failed: Invalid argument
```

Or, with `limits.cpu` set as an integer:

```
qemu-system-x86_64: Unknown section or instance 'apic' 3
```

### Root cause: Incus sets QEMU `maxcpus` from host CPU count

In the Incus source (`driver_qemu_templates.go`), the `qemuCPU` function
generates the QEMU `-smp` configuration. When `limits.cpu` is unset or set
to a plain integer, it does:

```go
maxCpus := 64
if int(cpu.Total) < maxCpus {
    maxCpus = int(cpu.Total)   // <-- host CPU count
}
```

This means:
- On a 4-core host: QEMU gets `maxcpus=4` → `cpu possible: 0-3`
- On a 2-core host: QEMU gets `maxcpus=2` → `cpu possible: 0-1`

The ICH9 ACPI CPU hotplug state (`hw/acpi/cpu.c`) is a variable-length
array sized by `maxcpus`. When source and destination have different
`maxcpus`, the vmstate sizes don't match and QEMU can't deserialize the
migration stream.

### How we discovered this

We observed the VM's `cpu possible` differing by host:

```bash
# On 4-core host:
cat /sys/devices/system/cpu/possible  → 0-3

# On 2-core host:
cat /sys/devices/system/cpu/possible  → 0-1
```

Even though the VM only uses 1 vCPU, QEMU allocates hotplug slots for all
"possible" CPUs.

### The fix: use `limits.cpu` as a range

When `limits.cpu` is set to a **range** (e.g., `0-1`), Incus takes the
**CPU pinning** code path, which generates a fixed topology with no
`maxcpus`:

```ini
# Integer (broken): maxcpus varies by host
[smp-opts]
cpus = 2
maxcpus = 4     # on 4-core host

# Range (fixed): no maxcpus, deterministic topology
[smp-opts]
cpus = 2
sockets = 1
cores = 2
threads = 1
```

**The one-line fix:**

```bash
# Set limits.cpu as a range (while VM is stopped)
incus config set <instance> limits.cpu=0-1    # for 2 vCPUs
incus config set <instance> limits.cpu=0-3    # for 4 vCPUs
```

### Configuration summary

| `limits.cpu` value | QEMU `-smp` behavior | Migration across heterogeneous hosts |
|---|---|---|
| (not set) | `maxcpus = host_cores` | **Fails** |
| `2` (integer) | `maxcpus = host_cores` | **Fails** |
| `0-1` (range) | Fixed `sockets/cores/threads`, no `maxcpus` | **Works** |

### Alternative: `raw.qemu.conf` override

If you need `maxcpus` (e.g., for CPU hotplug), force it to a fixed value:

```bash
incus config set <instance> raw.qemu.conf='[smp-opts]
maxcpus = 2'
```

Do **not** combine this with the range syntax — they are mutually exclusive.

### Complete setup for a live-migratable VM

```bash
# Create and configure (all while stopped)
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
incus stop incus-lab-01:test-vm

incus config set incus-lab-01:test-vm migration.stateful=true
incus config set incus-lab-01:test-vm limits.cpu=0-1
incus config device add incus-lab-01:test-vm root disk \
  path=/ pool=local size.state=2GiB

incus start incus-lab-01:test-vm

# Now live migration works across any cluster node
incus move incus-lab-01:test-vm --target incus-lab-03
```

---

## Migration Test Results

Tested with a 3-node cluster on Proxmox VE (Intel i9-13900HK, nested VT-x):
- **incus-lab-01**: 4 vCPUs, 8192 MB
- **incus-lab-02**: 2 vCPUs, 4096 MB
- **incus-lab-03**: 2 vCPUs, 4096 MB

Test VM: Debian 12, `limits.cpu=0-1`, `migration.stateful=true`, `size.state=2GiB`.
Heartbeat process (1/s counter to tmpfs) verifies full state preservation.

| # | Test | Time | Result |
|---|------|------|--------|
| 1 | Live: node1→node2 (4-core host → 2-core host) | 7.4s | PASS |
| 2 | Live: node2→node3 (2-core host → 2-core host) | 7.2s | PASS |
| 3 | Live: node3→node1 (2-core host → 4-core host) | 7.5s | PASS |
| 4 | Rapid round-trip: 1→2→3→1 (back-to-back) | ~22s | PASS |
| 5 | Stateful stop/restore (same node) | 7.6s | PASS |
| 6 | Stateful stop/move/restore (node1→node2, cross-topology) | 10.0s | PASS |
| 7a | Cluster evacuate node2 (container + VM) | 9.0s | PASS |
| 7b | Cluster restore node2 (moves workloads back) | 9.7s | PASS |

**Key observations:**
- Transfer speed: ~140 MB/s for ~1 GB VM
- Heartbeat counter continuous through all migrations (no lost seconds)
- tmpfs, systemd services, and VM uptime all preserved through live migration
- VM agent (`incus exec`) needs ~3-4 seconds to reconnect after migration
- Cluster evacuation auto-selects strategy: live migration for VMs, stop/move/start for containers
- Cluster restore returns workloads to original node

### Automated migration test

Run this from your workstation to test all migration paths:

```bash
REMOTE=incus-lab-01   # any cluster member
VM=test-vm

# Setup: create heartbeat inside VM
incus exec "$REMOTE:$VM" -- mkdir -p /tmp/migration-test
incus exec "$REMOTE:$VM" -- bash -c \
  'echo "created-$(hostname)-$(date +%s)" > /tmp/migration-test/marker'
incus exec "$REMOTE:$VM" -- bash -c \
  'cat > /tmp/migration-test/heartbeat.sh << "EOF"
#!/bin/bash
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
EOF
chmod +x /tmp/migration-test/heartbeat.sh'
incus exec "$REMOTE:$VM" -- \
  systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh

sleep 3

# Test: live-migrate through all nodes
PASS=0; FAIL=0
for TARGET in incus-lab-02 incus-lab-03 incus-lab-01; do
  PRE=$(incus exec "$REMOTE:$VM" -- cat /tmp/migration-test/heartbeat)
  LOC=$(incus list "$REMOTE:" --format csv -c nL | grep "$VM" | cut -d, -f2)
  echo -n "[$LOC → $TARGET] heartbeat=$PRE ... "

  incus move "$REMOTE:$VM" --target "$TARGET" 2>&1 | tail -1
  sleep 2  # wait for agent reconnect

  POST=$(incus exec "$REMOTE:$VM" -- cat /tmp/migration-test/heartbeat 2>&1)
  SVC=$(incus exec "$REMOTE:$VM" -- systemctl is-active migration-heartbeat 2>&1)
  if [[ "$SVC" == "active" && "$POST" -gt "$PRE" ]]; then
    echo "PASS (heartbeat=$POST, service=$SVC)"
    PASS=$((PASS + 1))
  else
    echo "FAIL (heartbeat=$POST, service=$SVC)"
    FAIL=$((FAIL + 1))
  fi
done

# Cleanup heartbeat
incus exec "$REMOTE:$VM" -- systemctl stop migration-heartbeat 2>/dev/null

echo ""
echo "Results: $PASS passed, $FAIL failed"
```

---

### The vnmi warning is cosmetic

The `CPUID[eax=8000000Ah].EDX.vnmi` warning appears because QEMU's CPU
feature dependency checker fires before KVM filters out unsupported
features. It does not cause migration failures and can be safely ignored.

### Nested virtualization works

Live migration works fine in nested virtualization (IncusOS inside Proxmox
on Intel VT-x). Tested with QEMU 10.2.1 on Intel i9-13900HK with
heterogeneous host core counts (4 vs 2). The `limits.cpu` range fix makes
host topology irrelevant.

### Debugging tips

- **`incus start --stateless`** discards a saved state file if a restore
  fails, allowing a clean boot.
- **Check `cpu possible`** inside the VM: `cat /sys/devices/system/cpu/possible`.
  If this differs across nodes for the same VM config, migration will fail.
- **`incus monitor --type=logging`** on the destination shows QEMU errors.

### References

- [Incus source: driver_qemu_templates.go](https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_qemu_templates.go) — `maxcpus` calculation
- [QEMU source: hw/acpi/cpu.c](https://github.com/qemu/qemu/blob/master/hw/acpi/cpu.c) — CPU hotplug state sizing
- [QEMU source: hw/isa/lpc_ich9.c](https://github.com/qemu/qemu/blob/master/hw/isa/lpc_ich9.c) — ICH9 vmstate subsections
- [Incus Instance Options](https://linuxcontainers.org/incus/docs/main/reference/instance_options/) — `limits.cpu`, `migration.stateful`
- [Incus issue #486](https://github.com/lxc/incus/issues/486) — cluster-wide CPU baseline

---

## Multi-vCPU Migration Test Results

Tested with a 3-node cluster (6/4/4 cores, 8 GiB RAM each) to validate
migration across different vCPU counts and heterogeneous host topologies.

### Test environment

- **incus-adv-01**: 6 vCPUs, 8192 MB
- **incus-adv-02**: 4 vCPUs, 8192 MB
- **incus-adv-03/04**: 4 vCPUs, 8192 MB

Three test VMs, all Debian 12, `migration.stateful=true`, `size.state=4GiB`:

| VM | `limits.cpu` | vCPUs | `cpu possible` |
|----|-------------|-------|----------------|
| test-vm-2cpu | `0-1` | 2 | `0-1` |
| test-vm-3cpu | `0-2` | 3 | `0-2` |
| test-vm-4cpu | `0-3` | 4 | `0-3` |

### Results

| # | Test | Time | Result |
|---|------|------|--------|
| 1 | 2-vCPU full ring (3 migrations) | ~7s each | PASS |
| 2 | 3-vCPU full ring (3 migrations) | ~7.5s each | PASS |
| 3 | 4-vCPU full ring (3 migrations) | ~7.3s each | PASS |
| 4 | Concurrent: 2 VMs from different nodes | 8s total | PASS |
| 5 | 4-vCPU VM to 4-core host (100% core usage) | 7.3s | PASS |
| 6 | Rapid round-trip: 3 hops back-to-back | ~31s total | PASS |
| 7 | Active disk I/O (dd 100 MiB) during migration | 8.2s | PASS |
| 8 | Active network (ping) during migration | 8.5s | PASS |
| 9 | Stateful stop/move/start (cross-node) | 10s | PASS |
| 10 | Memory-loaded VM (512 MiB allocated) | 7.7s | PASS |
| 11 | Move to same node (no-op) | instant | Clean error |
| 12 | Mixed evacuation (2 containers + 2 VMs) | 18s | PASS |
| 13 | Restore after evacuation | 18s | PASS |

### Key findings

- **Odd vCPU counts work**: `limits.cpu=0-2` (3 vCPUs) migrates just as
  reliably as even counts. The range syntax always produces deterministic
  QEMU topology regardless of count.
- **4-vCPU on 4-core host works**: a VM using all host cores migrates
  successfully and remains responsive. No performance degradation observed.
- **Concurrent migrations**: migrating 2 VMs simultaneously from different
  source nodes completes without interference. Transfer speed is maintained
  at ~140 MB/s per migration.
- **Active I/O survives**: both disk writes (dd) and network activity (ping)
  continue uninterrupted through live migration. File integrity verified.
- **Memory-loaded migration**: 512 MiB of allocated memory does not
  significantly increase migration time (QEMU's iterative pre-copy handles
  dirty pages efficiently).
- **Agent reconnect time is ~3-4 seconds**, not 1-2s as initially measured.
  May vary with cluster size and vCPU count. Scripts should `sleep 4` after
  migration before running `incus exec`.
- **Same-node move**: Incus returns a clear error ("Requested target server
  is the same as current server") — no crash or corruption.
- **`size.state=4GiB`** recommended for VMs with 3-4 vCPUs to ensure
  sufficient room for state dumps.

---

## Cluster Lifecycle: Node Replacement

Full procedure for evacuating, removing, destroying, and replacing a cluster
node. Tested on the 3-node advanced cluster.

### Overview

1. Evacuate the target node (drains workloads)
2. Remove the node from the cluster
3. Destroy the VM in Proxmox
4. Deploy a fresh replacement node
5. Join the replacement to the cluster
6. Verify workload rebalancing

### Step 1: Evacuate

```bash
REMOTE=incus-adv-01   # init node (not the target)
incus cluster evacuate $REMOTE:incus-adv-03 --force
```

This live-migrates VMs (with `migration.stateful=true`) and stop/move/starts
containers. Verify with `incus cluster list` — target shows `EVACUATED`.

### Step 2: Remove from cluster

```bash
printf "yes\n" | incus cluster remove $REMOTE:incus-adv-03 --force
```

**Note**: `--force` requires interactive confirmation ("yes"). Pipe it in for
automation. The `--force` flag is needed because the node is still technically
reachable (just evacuated).

After removal, `incus cluster list` shows only the remaining members.

### Step 3: Destroy the VM

Use `incusos-proxmox --cleanup` with a config targeting just that VM, or
use the Proxmox API directly. Do NOT try raw `curl` with API tokens
containing `!` characters — bash history expansion will silently mangle the
token. Always use single quotes or the `incusos-proxmox` script.

```bash
# Method 1: incusos-proxmox (recommended)
incusos-proxmox --cleanup --yes <config-with-target-vm.yaml>

# Method 2: Proxmox API (note: single quotes around the token!)
TOKEN='automation@pve!deploy=<secret>'
curl -s -k -X POST -H "Authorization: PVEAPIToken=$TOKEN" \
  "https://<host>:8006/api2/json/nodes/<node>/qemu/<vmid>/status/stop"
```

Also remove the incus remote:

```bash
incus remote switch local       # if target is current default
incus remote remove incus-adv-03
```

### Step 4: Deploy replacement

Create a config for the replacement node (e.g., `lab-replace.yaml`):

```yaml
defaults:
  memory: 8192
  disk: 50
  start_vmid: 913

vms:
  - name: incus-adv-04
    app: incus
    apply_defaults: true
    cores: 4
```

```bash
incusos-proxmox --yes lab-replace.yaml
```

### Step 5: Join to cluster

Same procedure as initial cluster formation:

```bash
NEW_IP=<replacement-node-ip>

# Set specific address
incus config set incus-adv-04: core.https_address $NEW_IP:8443

# Prepare (delete pre-existing pool/network)
incus config unset incus-adv-04: storage.backups_volume
incus config unset incus-adv-04: storage.images_volume
incus storage volume delete incus-adv-04:local backups
incus storage volume delete incus-adv-04:local images
incus profile device remove incus-adv-04:default root
incus profile device remove incus-adv-04:default eth0
incus storage delete incus-adv-04:local
incus network delete incus-adv-04:incusbr0

# Generate token and join
incus cluster add $REMOTE:incus-adv-04
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join $REMOTE: incus-adv-04:

# Fix remote (new cluster cert)
incus remote remove incus-adv-04
incus remote add incus-adv-04 https://$NEW_IP:8443 --accept-certificate
```

### Step 6: Verify rebalancing

With `cluster.rebalance.interval` configured, Incus automatically migrates
VMs to the new (empty) node:

```bash
# Check after 1-2 minutes
incus list $REMOTE: --columns ntsL
```

In testing, 2 VMs were automatically live-migrated to the replacement node
within 90 seconds. Containers are NOT auto-rebalanced (Incus limitation) —
move them manually with stop/move/start.

### Rebalancing configuration

```bash
REMOTE=incus-adv-01
incus config set $REMOTE: cluster.rebalance.interval=1     # check every N minutes
incus config set $REMOTE: cluster.rebalance.threshold=10    # imbalance % to trigger
incus config set $REMOTE: cluster.rebalance.batch=2         # max VMs per rebalance run
incus config set $REMOTE: cluster.rebalance.cooldown=5m     # wait between runs
```

| Key | Description | Tested value |
|-----|-------------|-------------|
| `cluster.rebalance.interval` | Check interval in minutes | `1` (aggressive for testing) |
| `cluster.rebalance.threshold` | Imbalance percentage to trigger | `10` |
| `cluster.rebalance.batch` | Max VMs moved per run | `2` |
| `cluster.rebalance.cooldown` | Minimum time between runs | `5m` |

**Important**: rebalancing only moves VMs with `migration.stateful=true`.
Containers and non-migratable VMs are not touched.

### Command syntax: `cluster remove`

```bash
# Works (pipe the confirmation)
printf "yes\n" | incus cluster remove remote:member --force

# Also note: cluster enable syntax is TWO args, not ONE
incus cluster enable remote: member-name    # correct
incus cluster enable remote:member-name     # WRONG (single arg)
```