Add advanced migration testing, cluster lifecycle, and clustering guide

- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
  lab-replace.yaml (single replacement node) for advanced migration and
  cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
  formation, storage pool conflict fix, join workflow, VM live migration
  with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
  edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
  cluster enable syntax fix (TWO args not one), cluster remove syntax,
  agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.gitignore

@@ -25,3 +25,6 @@ Thumbs.db
 
 # Go binaries (flasher-tool)
 flasher-tool
+
+# Proxmox connection config (contains credentials)
+proxmox.yaml

CLAUDE.md

@@ -19,9 +19,11 @@ incu-contrib/
 │   ├── incusos-seed       # Seed archive generator
 │   ├── incusos-proxmox    # Declarative Proxmox VM deployment
 │   ├── lab-test           # Guided lab validation (workloads, clustering, migration)
+│   ├── proxmox.yaml       # Proxmox connection config (gitignored, contains credentials)
 │   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
 │   └── examples/          # Example seed + Proxmox YAML files
 └── notes/                 # Research notes and reference material
+    └── clustering-guide.md  # Detailed Incus clustering walkthrough
 ```
 
 ## Key technical context
@@ -78,6 +80,11 @@ incu-contrib/
 - **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
   via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
   VMs with IncusOS-correct settings, and boots them through installation.
+- **Config separation**: Proxmox connection settings (host, credentials, pool)
+  live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
+  etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
+  `proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
+  for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
 - **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
   (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
 - **Minimum API privileges** for token-based access:
@@ -142,27 +149,198 @@ incu-contrib/
 
 - **Cluster formation** is done entirely through the `incus` CLI using remotes.
   No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
-- **Workflow:**
+- **No VIP needed**: each node advertises its own IP as its cluster address.
+  Clients can connect to any cluster member; requests are forwarded internally.
+
+#### Pre-clustering: fix core.https_address
+
+- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
+  interfaces). Clustering requires a **specific routable IP** so nodes can
+  address each other.
+- **Set the IP on every node BEFORE enabling clustering:**
   ```bash
-  # Enable clustering on first node
+  incus config set <remote>: core.https_address <NODE_IP>:8443
-  incus cluster enable <remote1>: <member-name>
-
-  # Generate join token (on cluster, for new member name)
-  incus cluster add <remote1>: <new-member-name>
-
-  # Join second node to cluster
-  incus cluster join <remote1>: <remote2>:
-
-  # Verify
-  incus cluster list <remote1>:
   ```
-- **`apply_defaults: true`** must be set on all nodes that will be clustered --
+- Get each node's routable IP via the API:
-  this ensures matching storage pool (`local`) and network bridge names, which
+  ```bash
-  are required for migration.
+  incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
+    [print(a) for a in d['environment']['addresses'] \
+     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
+  ```
+- This is safe to do while remotes are connected -- the remote already points
+  to the specific IP; we're just narrowing the bind address. Certificate trust
+  is fingerprint-based, not address-based.
+
+#### Cluster enable (init node)
+
+- ```bash
+  incus cluster enable <remote>: <member-name>
+  ```
+  Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
+  The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
+- **TLS certificate regeneration**: enabling clustering causes the server to
+  generate a new TLS certificate (cluster cert). The new cert may only have
+  SANs for `127.0.0.1` and `::1`, breaking the existing remote.
+- **Fix**: remove and re-add the remote to pin the new certificate:
+  ```bash
+  incus remote switch local         # if init remote is current default
+  incus remote remove <remote>
+  incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
+  ```
+- The cert trust on the server side (client → server) is unaffected -- it's
+  stored by fingerprint in the Incus database, independent of listen address.
+
+#### Joining nodes: the apply_defaults storage pool conflict
+
+- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0`
+  network bridge on each node at install time. When joining a cluster that
+  already has a `local` pool defined, the join wizard asks for `source` and
+  `zfs.pool_name` properties but then tries to set them as cluster-wide config.
+  Incus rejects this because they are **member-specific** keys.
+- **The fix**: delete the storage pool and network on the joining node before
+  joining. The join process then creates the member-specific pool entry
+  correctly.
+  ```bash
+  # 1. Remove config references
+  incus config unset <remote>: storage.backups_volume
+  incus config unset <remote>: storage.images_volume
+  # 2. Delete volumes
+  incus storage volume delete <remote>:local backups
+  incus storage volume delete <remote>:local images
+  # 3. Clear default profile references (pool is "in use" otherwise)
+  incus profile device remove <remote>:default root
+  incus profile device remove <remote>:default eth0
+  # 4. Delete pool and network
+  incus storage delete <remote>:local
+  incus network delete <remote>:incusbr0
+  ```
+
+#### Join workflow
+
+- **Generate token** (on init node, single argument `remote:member-name`):
+  ```bash
+  incus cluster add <init-remote>:<new-member-name>
+  ```
+- **Join** (interactive -- prompts for 5 values):
+  ```bash
+  incus cluster join <init-remote>: <joining-remote>:
+  ```
+  Interactive prompts and correct answers:
+  1. IP address → accept default (node's IP, already set via core.https_address)
+  2. Member name → accept default (matches the token)
+  3. "All existing data is lost" → `yes`
+  4. `source` property for storage pool "local" → `local/incus`
+  5. `zfs.pool_name` property for storage pool "local" → `local/incus`
+- **Automated (non-interactive):**
+  ```bash
+  printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
+  ```
+- **After join**: the joining node gets a new cluster certificate. Fix the
+  remote (same as init node):
+  ```bash
+  incus remote remove <joining-remote>
+  incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
+  ```
+
+#### Command syntax gotchas
+
+- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
+- `incus cluster add remote:member-name` -- ONE argument (no space)
+- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
+  "yes/no" even with `--force`, pipe `printf "yes\n"` for automation
+- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
+- `incus cluster restore remote:member-name` -- ONE argument (no space)
+- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
+- `incus storage show remote:pool` -- ONE argument (no space)
+- `incus storage show remote:pool --target member` -- target flag for
+  member-specific config
+- `incus config set remote: key value` -- remote with trailing colon + space
+- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
+  colon) for targeting the server itself
+
+#### Post-join state
+
 - After joining, the cluster is managed through the init node's remote. The
   individual node remotes still work for node-specific operations.
 - **`lab-test`** automates cluster formation, workload testing, and migration.
 
+#### Workload placement and migration
+
+- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
+- **Cluster-wide visibility**: `incus list` on any member shows all instances.
+- **Container migration**: stop/move/start only (CRIU live migration is
+  unreliable). Data persists, processes do not.
+  ```bash
+  incus stop <remote>:<instance>
+  incus move <remote>:<instance> --target <destination-member>
+  incus start <remote>:<instance>
+  ```
+- **VM live migration**: requires `migration.stateful=true` (must be set
+  while VM is stopped). Preserves running state with no downtime.
+  ```bash
+  incus move <remote>:<instance> --target <destination-member>
+  ```
+- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
+  not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
+  CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
+  Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
+  differently, causing `Missing section footer for ICH9LPC` on restore.
+  Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
+  fixed `sockets/cores/threads` topology — portable across all hosts.
+  ```bash
+  # WRONG: integer → maxcpus varies by host → migration fails
+  incus config set <instance> limits.cpu=2
+  # RIGHT: range → fixed topology → migration works everywhere
+  incus config set <instance> limits.cpu=0-1
+  ```
+- **VM live migration works in nested virtualization** (IncusOS inside
+  Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
+  10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
+- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
+  appears during migration is cosmetic. It fires from QEMU's feature
+  dependency checker before KVM filters out unsupported features and does
+  not affect migration.
+- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
+  requires the `limits.cpu` range fix. Use `incus start --stateless` to
+  discard a saved state file that cannot be restored.
+- **VM `size.state` config**: stateful operations require `size.state` on
+  the root disk (`incus config device add <instance> root disk path=/
+  pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
+- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
+  (ONE argument, like `cluster enable` and `cluster add`).
+  Use `--action stop` if VMs lack the `limits.cpu` range fix.
+  Restore with `incus cluster restore <remote>:<member> --force`.
+- **VM agent reconnect**: after live migration, the incus agent inside the VM
+  needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
+  after migration may fail with "VM agent isn't currently running". Scripts
+  should `sleep 4` after migration before running `incus exec`.
+- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
+  heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
+  work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
+  usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
+- **Concurrent migrations**: migrating multiple VMs simultaneously from
+  different source nodes works without interference. ~140 MB/s per migration.
+- **Active I/O during migration**: disk writes and network activity survive
+  live migration transparently. File integrity verified after migration.
+- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
+  joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
+  auto-rebalanced.
+  ```bash
+  incus config set <remote>: cluster.rebalance.interval=1     # minutes
+  incus config set <remote>: cluster.rebalance.threshold=10    # imbalance %
+  incus config set <remote>: cluster.rebalance.batch=2         # max VMs/run
+  incus config set <remote>: cluster.rebalance.cooldown=5m     # wait between runs
+  ```
+- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
+  join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
+  for step-by-step instructions.
+- **`incus cluster remove` requires confirmation**: even with `--force`, it
+  prompts "Are you really sure?". Pipe `yes` for automation:
+  ```bash
+  printf "yes\n" | incus cluster remove <remote>:<member> --force
+  ```
+- See `notes/clustering-guide.md` for full details and references.
+
 ### Lab validation (lab-test)
 
 - **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
@@ -187,11 +365,19 @@ incu-contrib/
 ### incusos-proxmox doctor and cleanup
 
 - **`--doctor`**: standalone environment check. No config file required.
-  Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
+  Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
-- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
+  Proxmox connectivity (from proxmox.yaml or config file).
-  removes incus remotes + clears local ISO cache.
+- **`--cleanup`**: destroys VMs defined in the config file.
-- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
+- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
-  `[incusos-lab:managed]` marker in the configured resource pool.
+  deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
+  all IncusOS ISOs (unlike the old behavior).
+- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
+  config required). Destroys all VMs with `[incusos-lab:managed]` marker.
+- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
+  and `seed-*.iso` from storage + remotes + cache.
+- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
+  output is concise (step names + results). `--quiet` suppresses everything
+  except warnings and errors.
 
 ## Coding conventions for scripts
 

incusos/README.md

@@ -296,13 +296,16 @@ This script ensures all settings are correct, preventing install failures.
     --no-cert           Skip certificate injection
     --vm NAME           Operate on a single VM only
 -y, --yes               Skip confirmation prompts
+    --proxmox FILE      Proxmox connection config (default: auto-discover)
     --doctor            Check tool versions and environment health
     --status            Check deployment status (shortcut for --phase status)
     --cleanup           Destroy VMs defined in config
-    --cleanup --deep    Also remove ISOs, seeds, remotes, and local cache
+    --cleanup --deep    Also remove deployment ISO, seeds, remotes, and cache
-    --cleanup-all       Destroy ALL managed VMs in the resource pool
+    --cleanup-all       Destroy ALL managed VMs in pool (no config file needed)
+    --cleanup-all --deep  Also remove ALL ISOs and seeds from storage
     --force-cleanup     Destroy VMs even without management marker
     --dry-run           Preview without executing
+-v, --verbose           Show detailed output (tool paths, API calls)
 -q, --quiet             Suppress info output
 -h, --help              Full help text
 -V, --version           Show version
@@ -314,14 +317,15 @@ Run `--doctor` to verify your environment without a config file:
 
 ```bash
 ./incusos-proxmox --doctor
+./incusos-proxmox --doctor -v   # verbose: show tool paths
 ```
 
 This checks: incus client, flasher-tool, operations-center CLI (optional),
-python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
+python3, curl, genisoimage, jq, incusos-seed sibling, IncusOS CDN
-availability.
+availability, and `proxmox.yaml` discovery.
 
-If a config file is also provided, additionally checks Proxmox connectivity,
+If `proxmox.yaml` is found or a config file is provided, additionally checks
-pool existence, and API token validity:
+Proxmox connectivity, pool existence, and API token validity:
 
 ```bash
 ./incusos-proxmox --doctor examples/lab-cluster.yaml
@@ -329,22 +333,28 @@ pool existence, and API token validity:
 
 ### Deep Cleanup
 
-Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
+Use `--deep` with `--cleanup` or `--cleanup-all` to remove artifacts beyond VMs:
 
 ```bash
-# VMs + ISOs + seeds + incus remotes + local cache
+# VMs + the IncusOS ISO used by this deployment + seeds + remotes + cache
 ./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
 
-# Pool-wide: destroy all managed VMs + artifacts
+# Pool-wide: destroy all managed VMs (uses proxmox.yaml, no config needed)
-./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
+./incusos-proxmox --cleanup-all
+./incusos-proxmox --cleanup-all --deep   # also nuke ALL ISOs + seeds
 ```
 
-`--deep` additionally:
+`--cleanup --deep`:
-- Deletes seed ISOs from Proxmox storage
+- Deletes per-VM seed ISOs from Proxmox storage
-- Deletes IncusOS ISOs from Proxmox storage
+- Deletes the specific IncusOS ISO used by this deployment (not all of them)
 - Removes matching `incus remote` entries
 - Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
 
+`--cleanup-all --deep`:
+- Deletes ALL `IncusOS_*.iso` and `seed-*.iso` from storage (intentionally aggressive)
+- Removes remotes for the destroyed VMs
+- Clears the local ISO cache
+
 ### Deployment Phases
 
 | Phase | What it does |
@@ -356,18 +366,41 @@ Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
 | `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
 | `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
 
-### Config File Format
+### Proxmox Connection Config
+
+Proxmox connection settings live in a separate `proxmox.yaml` file to avoid
+duplicating credentials across lab configs. Auto-discovered in the script
+directory and current directory. Override with `--proxmox FILE`.
+
+Copy the template to get started:
+
+```bash
+cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
+# Edit with your Proxmox host, credentials, and storage settings
+```
 
 ```yaml
+# proxmox.yaml (flat structure, no wrapping key)
+host: 192.168.1.10
+method: api
+api_token_id: automation@pve!deploy
+node: pve
+storage: local-lvm
+iso_storage: local
+bridge: vmbr0
+pool: IncusLab
+ssh_user: root
+```
+
+Lab configs can override any setting by including a `proxmox:` section.
+Merge priority: `proxmox.yaml` → lab config `proxmox:` → CLI flags.
+
+### Lab Config File Format
+
+```yaml
+# proxmox: section is optional — overrides proxmox.yaml if present
 proxmox:
-  host: 192.168.1.10       # Proxmox host (required)
+  storage: local-zfs        # Override storage for this deployment
-  node: pve                 # Node name (default: pve)
-  storage: local-lvm        # VM disk storage (default: local-lvm)
-  iso_storage: local        # ISO storage (default: local)
-  bridge: vmbr0             # Network bridge (default: vmbr0)
-  pool: IncusLab            # Resource pool (optional, for isolation)
-  method: ssh               # ssh (default) or api
-  ssh_user: root            # For ssh method
 
 defaults:
   cores: 4
@@ -602,10 +635,11 @@ See the [`examples/`](examples/) directory for sample seed configurations:
 - [`incus-cluster-node.yaml`](examples/incus-cluster-node.yaml) -- Cluster node (no defaults)
 - [`ops-center.yaml`](examples/ops-center.yaml) -- Operations Center
 - [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
-- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
+- [`proxmox.yaml.example`](examples/proxmox.yaml.example) -- Proxmox connection config template
-- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
+- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (self-contained, includes proxmox: section)
-- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
+- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM (self-contained)
-- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes
+- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster (uses proxmox.yaml)
+- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes (uses proxmox.yaml)
 
 ## Security Notes
 

incusos/TESTING.md

@@ -129,13 +129,23 @@ ssh root@<YOUR_PROXMOX_IP> pvesh get /version
 You should see JSON with the Proxmox VE version. If this fails, fix SSH
 key auth first (`ssh-copy-id root@<host>`).
 
-### Step 2: Create a Test Config
+### Step 2: Set Up Proxmox Connection
+
+```bash
+cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
+```
+
+Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
+auto-discovered by the scripts and avoids repeating credentials in each
+lab config.
+
+### Step 3: Create a Test Config
 
 ```bash
 cp incusos/examples/proxmox-minimal.yaml my-test.yaml
 ```
 
-Edit `my-test.yaml` -- change only the host IP:
+Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):
 
 ```yaml
 proxmox:
@@ -155,7 +165,7 @@ vms:
 
 Using VMID 900+ avoids clashing with existing VMs.
 
-### Step 3: Dry Run
+### Step 4: Dry Run
 
 ```bash
 ./incusos/incusos-proxmox --dry-run my-test.yaml
@@ -172,7 +182,7 @@ Using VMID 900+ avoids clashing with existing VMs.
 - `[dry-run]` prefix on all actions -- nothing actually executed
 - No errors
 
-### Step 4: Deploy (Single VM)
+### Step 5: Deploy (Single VM)
 
 ```bash
 ./incusos/incusos-proxmox my-test.yaml
@@ -196,7 +206,7 @@ Using VMID 900+ avoids clashing with existing VMs.
   disk, and reboot
 - After reboot you should see the IncusOS login prompt
 
-### Step 5: Check Deployment Status
+### Step 6: Check Deployment Status
 
 After the deploy completes, verify with the built-in status command:
 
@@ -212,7 +222,7 @@ After the deploy completes, verify with the built-in status command:
 - Port 8443 is open
 - Incus remote status shown (if configured)
 
-### Step 6: Verify the Deployed VM
+### Step 7: Verify the Deployed VM
 
 Once the script reports success with an IP address:
 
@@ -233,7 +243,7 @@ incus network list incus-test:
 ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
 check the Proxmox console — stale ARP entries can be misleading.
 
-### Step 7: Test Re-run Reconcile
+### Step 8: Test Re-run Reconcile
 
 Re-run the deploy to verify the reconcile menu:
 
@@ -252,7 +262,7 @@ Also test with `--yes` to verify it defaults to safe behavior:
 
 **Verify:** Runs post-deployment checks, does NOT destroy.
 
-### Step 8: Clean Up the Test VM
+### Step 9: Clean Up the Test VM
 
 ```bash
 ./incusos/incusos-proxmox --cleanup my-test.yaml
@@ -269,7 +279,7 @@ Remove the remote too:
 incus remote remove incus-test
 ```
 
-### Step 9: (Optional) Test API Method
+### Step 10: (Optional) Test API Method
 
 After single-VM success with SSH, switch to API tokens.
 
@@ -302,9 +312,9 @@ proxmox:
   api_token_id: automation@pve!deploy
 ```
 
-Repeat steps 3-6 using the API method.
+Repeat steps 4-7 using the API method.
 
-### Step 10: (Optional) Test with Resource Pool
+### Step 11: (Optional) Test with Resource Pool
 
 For API method with pool isolation:
 
@@ -331,7 +341,7 @@ Deploy and verify:
 ./incusos/incusos-proxmox --status my-test.yaml     # Status scoped to pool
 ```
 
-### Step 11: (Optional) Full Lab Deployment
+### Step 12: (Optional) Full Lab Deployment
 
 Once single-VM tests pass, deploy the full 4-VM lab:
 
@@ -392,32 +402,229 @@ incus remote add incus-lab-03 <IP3> --accept-certificate
 
 ### Manual Cluster Formation (Alternative)
 
-If you prefer to form the cluster manually instead of using `lab-test`:
+If you prefer to form the cluster manually instead of using `lab-test`,
+follow the steps below. See also `notes/clustering-guide.md` for the full
+reference guide with explanations of what happens under the hood.
+
+#### Prerequisites
 
 ```bash
-# Step 1: Enable clustering on first node
+# Check client version (need 6.20+ for remote cluster join)
-incus cluster enable incus-lab-01: incus-lab-01
+incus version
 
-# Step 2: Generate join token for second node
+# Verify all three remotes are working
-incus cluster add incus-lab-01: incus-lab-02
+incus info incus-lab-01: | head -5
+incus info incus-lab-02: | head -5
+incus info incus-lab-03: | head -5
+```
 
-# Step 3: Join second node
+#### Step 1: Set specific IPs on all nodes
-incus cluster join incus-lab-01: incus-lab-02:
 
-# Step 4: Repeat for third node
+IncusOS defaults to `core.https_address: :8443` (wildcard). Clustering
-incus cluster add incus-lab-01: incus-lab-03
+requires a specific routable IP on each node.
-incus cluster join incus-lab-01: incus-lab-03:
 
-# Step 5: Verify
+```bash
+# Discover each node's routable IP
+for node in incus-lab-01 incus-lab-02 incus-lab-03; do
+  echo "=== $node ==="
+  incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
+    [print(a) for a in d['environment']['addresses'] \
+     if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
+done
+
+# Set the specific IP on each node (replace with actual IPs)
+incus config set incus-lab-01: core.https_address <IP1>:8443
+incus config set incus-lab-02: core.https_address <IP2>:8443
+incus config set incus-lab-03: core.https_address <IP3>:8443
+
+# Verify connectivity still works after each change
+incus info incus-lab-01: | head -3
+incus info incus-lab-02: | head -3
+incus info incus-lab-03: | head -3
+```
+
+#### Step 2: Enable clustering on the init node
+
+```bash
+incus cluster enable incus-lab-01:incus-lab-01
+```
+
+> **Important**: This regenerates the server's TLS certificate. Your remote
+> will break with a certificate error. Fix it:
+
+```bash
+incus remote switch local          # if incus-lab-01 is the current default
+incus remote remove incus-lab-01
+incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate
+```
+
+Verify:
+
+```bash
 incus cluster list incus-lab-01:
+# Should show one member (incus-lab-01), status ONLINE, role database-leader
+```
+
+#### Step 3: Prepare joining nodes
+
+Nodes with `apply_defaults: true` have a pre-existing `local` storage pool
+and `incusbr0` network. These conflict with the cluster join process (the
+join wizard tries to set member-specific ZFS config keys at cluster level).
+Delete them before joining:
+
+```bash
+# For each joining node (incus-lab-02, incus-lab-03):
+NODE=incus-lab-02   # then repeat with incus-lab-03
+
+# Remove server-level references to the pool
+incus config unset "$NODE": storage.backups_volume
+incus config unset "$NODE": storage.images_volume
+
+# Delete the backup and image volumes
+incus storage volume delete "$NODE":local backups
+incus storage volume delete "$NODE":local images
+
+# Clear default profile references (otherwise pool is "in use")
+incus profile device remove "$NODE":default root
+incus profile device remove "$NODE":default eth0
+
+# Delete pool and network
+incus storage delete "$NODE":local
+incus network delete "$NODE":incusbr0
+```
+
+#### Step 4: Join nodes to the cluster
+
+**Interactive** (one node at a time):
+
+```bash
+# Generate join token
+incus cluster add incus-lab-01:incus-lab-02
+
+# Join (answer the interactive prompts)
+incus cluster join incus-lab-01: incus-lab-02:
+# Prompt 1 - IP address: press Enter (accept default)
+# Prompt 2 - Member name: press Enter (accept default)
+# Prompt 3 - "All existing data is lost": type "yes"
+# Prompt 4 - source for storage pool "local": type "local/incus"
+# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"
+```
+
+**Automated** (non-interactive):
+
+```bash
+# Generate token + join in one go
+incus cluster add incus-lab-01:incus-lab-02
+printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:
+```
+
+After each join, fix the remote (same certificate issue as the init node):
+
+```bash
+incus remote remove incus-lab-02
+incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate
+```
+
+Repeat for incus-lab-03.
+
+#### Step 5: Verify the cluster
+
+```bash
+# All 3 members should be ONLINE
+incus cluster list incus-lab-01:
+
+# Storage pool should show on all members
+incus storage show incus-lab-01:local
+incus storage show incus-lab-01:local --target incus-lab-01
+incus storage show incus-lab-01:local --target incus-lab-02
+incus storage show incus-lab-01:local --target incus-lab-03
+
+# All remotes should work
+incus info incus-lab-01: | head -3
+incus info incus-lab-02: | head -3
+incus info incus-lab-03: | head -3
 ```
 
 **Key points:**
 - `apply_defaults: true` on all nodes ensures matching storage pool and
-  network bridge configurations (required for migration)
+  network bridge names (required for migration) -- but the pools must be
+  deleted on joining nodes before the join
 - After joining, manage the cluster through the init node's remote
+- The ZFS pool source is `local/incus` on IncusOS (auto-created by
+  `apply_defaults`)
+- Each join generates a new token (tokens are single-use and time-limited)
 - VM migration requires matching storage pool names across nodes
 
+### Migration Testing (After Cluster Formation)
+
+Once the cluster is formed, validate container and VM migration:
+
+#### Container migration (stop/move/start)
+
+```bash
+# Launch on node 2
+incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
+incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'
+
+# Migrate to node 3
+incus stop incus-lab-01:test-container
+incus move incus-lab-01:test-container --target incus-lab-03
+incus start incus-lab-01:test-container
+
+# Verify: data survives, processes don't
+incus exec incus-lab-01:test-container -- cat /root/test.txt   # "hello"
+incus list incus-lab-01: --columns nLs                         # node 3
+```
+
+#### VM live migration
+
+```bash
+# Launch and configure for migration
+incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
+# Wait for VM agent...
+incus stop incus-lab-01:test-vm
+incus config set incus-lab-01:test-vm migration.stateful=true
+incus config set incus-lab-01:test-vm limits.cpu=0-1
+incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
+incus start incus-lab-01:test-vm
+
+# Set up heartbeat for state verification
+incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
+incus exec incus-lab-01:test-vm -- bash -c \
+  'cat > /tmp/migration-test/heartbeat.sh << "EOF"
+#!/bin/bash
+i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
+EOF
+chmod +x /tmp/migration-test/heartbeat.sh'
+incus exec incus-lab-01:test-vm -- \
+  systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh
+
+# Live-migrate (state preserved)
+sleep 3
+incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # note value
+incus move incus-lab-01:test-vm --target incus-lab-03
+sleep 2   # agent needs ~1-2s to reconnect
+incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat   # should be higher
+incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat  # "active"
+```
+
+#### Cluster evacuation
+
+```bash
+# Move both workloads to node 2, then evacuate
+incus cluster evacuate incus-lab-01:incus-lab-02 --force
+incus cluster list incus-lab-01:    # node 2 = EVACUATED
+incus list incus-lab-01: --columns nLs  # workloads on other nodes
+
+# Restore
+incus cluster restore incus-lab-01:incus-lab-02 --force
+incus cluster list incus-lab-01:    # node 2 = ONLINE
+incus list incus-lab-01: --columns nLs  # workloads back on node 2
+```
+
+See `notes/clustering-guide.md` for the full automated migration test script
+and detailed test results.
+
 ### Cleanup
 
 ```bash
@@ -427,8 +634,12 @@ incus cluster list incus-lab-01:
 # Destroy VMs only
 ./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
 
-# Full cleanup: VMs + ISOs + seeds + remotes + cache
+# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
 ./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
+
+# Pool-wide: destroy ALL managed VMs (no config file needed)
+./incusos/incusos-proxmox --cleanup-all --yes
+./incusos/incusos-proxmox --cleanup-all --deep --yes   # nuke everything
 ```
 
 ---
@@ -507,3 +718,7 @@ CLI is installed.
 | Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
 | Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
 | `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
+| VM live migration fails: "Missing section footer for ICH9LPC" | `limits.cpu` not set or set as integer | Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md` |
+| `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start |
+| `certificate is valid for 127.0.0.1, ::1, not <IP>` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` |
+| `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide |

incusos/examples/lab-advanced.yaml

@@ -0,0 +1,45 @@
+# lab-advanced.yaml - 3-node heterogeneous cluster for advanced migration testing
+#
+# Deploys 3 Incus nodes with a 6/4/4 core split to test VM live migration
+# across hosts with different CPU topologies. 8 GiB RAM provides headroom
+# for multiple concurrent VMs with size.state for stateful operations.
+#
+# Test matrix targets:
+#   - Multi-vCPU VMs (2, 3, 4 vCPUs) with limits.cpu ranges
+#   - Concurrent migrations from different source nodes
+#   - Edge cases: active I/O, memory-loaded VMs, 4-vCPU on 4-core host
+#   - Cluster lifecycle: evacuate, remove, replace, rejoin, rebalance
+#
+# Usage:
+#   incusos-proxmox --dry-run examples/lab-advanced.yaml   # Preview
+#   incusos-proxmox --yes examples/lab-advanced.yaml        # Deploy
+#   incusos-proxmox --status examples/lab-advanced.yaml     # Check status
+#   incusos-proxmox --cleanup examples/lab-advanced.yaml    # Tear down VMs
+#   incusos-proxmox --cleanup --deep examples/lab-advanced.yaml  # Full cleanup
+#
+# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
+# Add a proxmox: section here to override per-deployment.
+#
+# Note: start_vmid 910 may collide with lab-oc.yaml. Clean up old deployments
+# first or adjust the range.
+
+defaults:
+  memory: 8192
+  disk: 50
+  start_vmid: 910
+
+vms:
+  - name: incus-adv-01
+    app: incus
+    apply_defaults: true
+    cores: 6
+
+  - name: incus-adv-02
+    app: incus
+    apply_defaults: true
+    cores: 4
+
+  - name: incus-adv-03
+    app: incus
+    apply_defaults: true
+    cores: 4

incusos/examples/lab-cluster.yaml

@@ -13,16 +13,12 @@
 #
 # After deployment, form a cluster with lab-test:
 #   lab-test --phase cluster examples/lab-cluster.yaml
-
+#
-proxmox:
+# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
-  host: 192.168.1.29
+# Add a proxmox: section here to override per-deployment.
-  method: api
-  api_token_id: automation@pve!deploy
-  storage: local-zfs
-  pool: IncusLab
 
 defaults:
-  cores: 2
+  cores: 4
   memory: 4096
   disk: 50
   start_vmid: 900
@@ -31,9 +27,6 @@ vms:
   - name: incus-lab-01
     app: incus
     apply_defaults: true
-    cores: 4
-    memory: 8192
-    disk: 64
 
   - name: incus-lab-02
     app: incus

incusos/examples/lab-oc.yaml

@@ -15,13 +15,9 @@
 #   incusos-proxmox --yes examples/lab-oc.yaml
 #   incusos-proxmox --status examples/lab-oc.yaml
 #   incusos-proxmox --cleanup --deep examples/lab-oc.yaml
-
+#
-proxmox:
+# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
-  host: 192.168.1.29
+# Add a proxmox: section here to override per-deployment.
-  method: api
-  api_token_id: automation@pve!deploy
-  storage: local-zfs
-  pool: IncusLab
 
 defaults:
   cores: 2

incusos/examples/lab-replace.yaml

@@ -0,0 +1,24 @@
+# lab-replace.yaml - Single replacement node for cluster lifecycle testing
+#
+# Used in Part 4 of the advanced migration test plan: after evacuating,
+# removing, and destroying incus-adv-03, this config deploys a fresh
+# replacement node that gets joined back into the cluster.
+#
+# Usage:
+#   incusos-proxmox --dry-run examples/lab-replace.yaml
+#   incusos-proxmox --yes examples/lab-replace.yaml
+#   incusos-proxmox --status examples/lab-replace.yaml
+#   incusos-proxmox --cleanup examples/lab-replace.yaml
+#
+# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
+
+defaults:
+  memory: 8192
+  disk: 50
+  start_vmid: 913
+
+vms:
+  - name: incus-adv-04
+    app: incus
+    apply_defaults: true
+    cores: 4

incusos/examples/my-test.yaml

@@ -1,9 +1,5 @@
-proxmox:
+# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
-  host: 192.168.1.29
+# Add a proxmox: section here to override per-deployment.
-  method: api
-  api_token_id: automation@pve!deploy
-  storage: local-zfs              # VM disk storage (default: local-lvm)
-  pool: IncusLab
 
 defaults:
   cores: 2

incusos/examples/proxmox.yaml.example

@@ -0,0 +1,21 @@
+# proxmox.yaml.example - Proxmox connection config template
+#
+# Copy to incusos/proxmox.yaml and edit with your settings.
+# This file is auto-discovered by incusos-proxmox and lab-test.
+#
+# Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference
+# this for connection settings, so you only need to change credentials
+# in one place.
+#
+# Override any setting per-deployment by adding a proxmox: section
+# to the lab config file.
+
+host: 192.168.1.10             # Proxmox host IP or hostname
+method: ssh                    # ssh (default) or api
+# api_token_id: automation@pve!deploy   # For api method
+node: pve                      # Proxmox node name
+storage: local-lvm             # VM disk storage
+iso_storage: local             # ISO/image storage
+bridge: vmbr0                  # Network bridge
+# pool: IncusLab               # Resource pool (optional, for isolation)
+ssh_user: root                 # SSH user (for ssh method)