From 732d70b38efdc5de52898cd36c568ea090237bd6 Mon Sep 17 00:00:00 2001 From: Maarten Date: Fri, 20 Feb 2026 18:46:55 +0100 Subject: [PATCH] Add advanced migration testing, cluster lifecycle, and clustering guide - New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and lab-replace.yaml (single replacement node) for advanced migration and cluster lifecycle testing - New notes/clustering-guide.md with full clustering walkthrough: cluster formation, storage pool conflict fix, join workflow, VM live migration with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs), edge cases (I/O, memory, concurrent), and node replacement lifecycle - Updated CLAUDE.md with clustering docs, rebalancing config keys, cluster enable syntax fix (TWO args not one), cluster remove syntax, agent reconnect timing (3-4s not 1-2s), and lifecycle procedure - proxmox.yaml.example for Proxmox connection config reference Co-Authored-By: Claude Opus 4.6 --- .gitignore | 3 + CLAUDE.md | 226 +++++- incusos/README.md | 86 ++- incusos/TESTING.md | 265 ++++++- incusos/examples/lab-advanced.yaml | 45 ++ incusos/examples/lab-cluster.yaml | 15 +- incusos/examples/lab-oc.yaml | 10 +- incusos/examples/lab-replace.yaml | 24 + incusos/examples/my-test.yaml | 8 +- incusos/examples/proxmox.yaml.example | 21 + incusos/incusos-proxmox | 615 ++++++++++++--- notes/clustering-guide.md | 1020 +++++++++++++++++++++++++ 12 files changed, 2157 insertions(+), 181 deletions(-) create mode 100644 incusos/examples/lab-advanced.yaml create mode 100644 incusos/examples/lab-replace.yaml create mode 100644 incusos/examples/proxmox.yaml.example create mode 100644 notes/clustering-guide.md diff --git a/.gitignore b/.gitignore index 095a2a4..0160dc5 100644 --- a/.gitignore +++ b/.gitignore @@ -25,3 +25,6 @@ Thumbs.db # Go binaries (flasher-tool) flasher-tool + +# Proxmox connection config (contains credentials) +proxmox.yaml diff --git a/CLAUDE.md b/CLAUDE.md index d009a38..12d36b8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,9 +19,11 @@ incu-contrib/ │ ├── incusos-seed # Seed archive generator │ ├── incusos-proxmox # Declarative Proxmox VM deployment │ ├── lab-test # Guided lab validation (workloads, clustering, migration) +│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials) │ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test │ └── examples/ # Example seed + Proxmox YAML files └── notes/ # Research notes and reference material + └── clustering-guide.md # Detailed Incus clustering walkthrough ``` ## Key technical context @@ -78,6 +80,11 @@ incu-contrib/ - **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates VMs with IncusOS-correct settings, and boots them through installation. +- **Config separation**: Proxmox connection settings (host, credentials, pool) + live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`, + etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config + `proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks + for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`. - **Connection methods**: SSH (default, `ssh root@host qm ...`) or API (`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header). - **Minimum API privileges** for token-based access: @@ -142,27 +149,198 @@ incu-contrib/ - **Cluster formation** is done entirely through the `incus` CLI using remotes. No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access). -- **Workflow:** +- **No VIP needed**: each node advertises its own IP as its cluster address. + Clients can connect to any cluster member; requests are forwarded internally. + +#### Pre-clustering: fix core.https_address + +- IncusOS nodes default to `core.https_address: :8443` (wildcard / all + interfaces). Clustering requires a **specific routable IP** so nodes can + address each other. +- **Set the IP on every node BEFORE enabling clustering:** ```bash - # Enable clustering on first node - incus cluster enable : - - # Generate join token (on cluster, for new member name) - incus cluster add : - - # Join second node to cluster - incus cluster join : : - - # Verify - incus cluster list : + incus config set : core.https_address :8443 ``` -- **`apply_defaults: true`** must be set on all nodes that will be clustered -- - this ensures matching storage pool (`local`) and network bridge names, which - are required for migration. +- Get each node's routable IP via the API: + ```bash + incus query :/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \ + [print(a) for a in d['environment']['addresses'] \ + if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]" + ``` +- This is safe to do while remotes are connected -- the remote already points + to the specific IP; we're just narrowing the bind address. Certificate trust + is fingerprint-based, not address-based. + +#### Cluster enable (init node) + +- ```bash + incus cluster enable : + ``` + Note: this is TWO arguments: `:` (trailing colon) and ``. + The help text shows `[:] ` — NOT `remote:name` as a single arg. +- **TLS certificate regeneration**: enabling clustering causes the server to + generate a new TLS certificate (cluster cert). The new cert may only have + SANs for `127.0.0.1` and `::1`, breaking the existing remote. +- **Fix**: remove and re-add the remote to pin the new certificate: + ```bash + incus remote switch local # if init remote is current default + incus remote remove + incus remote add https://:8443 --accept-certificate + ``` +- The cert trust on the server side (client → server) is unaffected -- it's + stored by fingerprint in the Incus database, independent of listen address. + +#### Joining nodes: the apply_defaults storage pool conflict + +- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0` + network bridge on each node at install time. When joining a cluster that + already has a `local` pool defined, the join wizard asks for `source` and + `zfs.pool_name` properties but then tries to set them as cluster-wide config. + Incus rejects this because they are **member-specific** keys. +- **The fix**: delete the storage pool and network on the joining node before + joining. The join process then creates the member-specific pool entry + correctly. + ```bash + # 1. Remove config references + incus config unset : storage.backups_volume + incus config unset : storage.images_volume + # 2. Delete volumes + incus storage volume delete :local backups + incus storage volume delete :local images + # 3. Clear default profile references (pool is "in use" otherwise) + incus profile device remove :default root + incus profile device remove :default eth0 + # 4. Delete pool and network + incus storage delete :local + incus network delete :incusbr0 + ``` + +#### Join workflow + +- **Generate token** (on init node, single argument `remote:member-name`): + ```bash + incus cluster add : + ``` +- **Join** (interactive -- prompts for 5 values): + ```bash + incus cluster join : : + ``` + Interactive prompts and correct answers: + 1. IP address → accept default (node's IP, already set via core.https_address) + 2. Member name → accept default (matches the token) + 3. "All existing data is lost" → `yes` + 4. `source` property for storage pool "local" → `local/incus` + 5. `zfs.pool_name` property for storage pool "local" → `local/incus` +- **Automated (non-interactive):** + ```bash + printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join : : + ``` +- **After join**: the joining node gets a new cluster certificate. Fix the + remote (same as init node): + ```bash + incus remote remove + incus remote add https://:8443 --accept-certificate + ``` + +#### Command syntax gotchas + +- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name) +- `incus cluster add remote:member-name` -- ONE argument (no space) +- `incus cluster remove remote:member-name --force` -- ONE argument; prompts + "yes/no" even with `--force`, pipe `printf "yes\n"` for automation +- `incus cluster evacuate remote:member-name` -- ONE argument (no space) +- `incus cluster restore remote:member-name` -- ONE argument (no space) +- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space) +- `incus storage show remote:pool` -- ONE argument (no space) +- `incus storage show remote:pool --target member` -- target flag for + member-specific config +- `incus config set remote: key value` -- remote with trailing colon + space +- General rule: `remote:resource` for targeting a resource, `remote:` (trailing + colon) for targeting the server itself + +#### Post-join state + - After joining, the cluster is managed through the init node's remote. The individual node remotes still work for node-specific operations. - **`lab-test`** automates cluster formation, workload testing, and migration. +#### Workload placement and migration + +- **Targeted launch**: `incus launch images:debian/12 :name --target ` +- **Cluster-wide visibility**: `incus list` on any member shows all instances. +- **Container migration**: stop/move/start only (CRIU live migration is + unreliable). Data persists, processes do not. + ```bash + incus stop : + incus move : --target + incus start : + ``` +- **VM live migration**: requires `migration.stateful=true` (must be set + while VM is stopped). Preserves running state with no downtime. + ```bash + incus move : --target + ``` +- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`), + not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's + CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`). + Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays + differently, causing `Missing section footer for ICH9LPC` on restore. + Using a range (pinning syntax) eliminates `maxcpus` entirely and uses + fixed `sockets/cores/threads` topology — portable across all hosts. + ```bash + # WRONG: integer → maxcpus varies by host → migration fails + incus config set limits.cpu=2 + # RIGHT: range → fixed topology → migration works everywhere + incus config set limits.cpu=0-1 + ``` +- **VM live migration works in nested virtualization** (IncusOS inside + Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU + 10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2). +- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that + appears during migration is cosmetic. It fires from QEMU's feature + dependency checker before KVM filters out unsupported features and does + not affect migration. +- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also + requires the `limits.cpu` range fix. Use `incus start --stateless` to + discard a saved state file that cannot be restored. +- **VM `size.state` config**: stateful operations require `size.state` on + the root disk (`incus config device add root disk path=/ + pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails. +- **Cluster evacuation**: `incus cluster evacuate : --force` + (ONE argument, like `cluster enable` and `cluster add`). + Use `--action stop` if VMs lack the `limits.cpu` range fix. + Restore with `incus cluster restore : --force`. +- **VM agent reconnect**: after live migration, the incus agent inside the VM + needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately + after migration may fail with "VM agent isn't currently running". Scripts + should `sleep 4` after migration before running `incus exec`. +- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across + heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`) + work identically to even counts. A 4-vCPU VM on a 4-core host (100% core + usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs. +- **Concurrent migrations**: migrating multiple VMs simultaneously from + different source nodes works without interference. ~140 MB/s per migration. +- **Active I/O during migration**: disk writes and network activity survive + live migration transparently. File integrity verified after migration. +- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node + joins. Only moves VMs with `migration.stateful=true`. Containers are NOT + auto-rebalanced. + ```bash + incus config set : cluster.rebalance.interval=1 # minutes + incus config set : cluster.rebalance.threshold=10 # imbalance % + incus config set : cluster.rebalance.batch=2 # max VMs/run + incus config set : cluster.rebalance.cooldown=5m # wait between runs + ``` +- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh → + join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md` + for step-by-step instructions. +- **`incus cluster remove` requires confirmation**: even with `--force`, it + prompts "Are you really sure?". Pipe `yes` for automation: + ```bash + printf "yes\n" | incus cluster remove : --force + ``` +- See `notes/clustering-guide.md` for full details and references. + ### Lab validation (lab-test) - **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates @@ -187,11 +365,19 @@ incu-contrib/ ### incusos-proxmox doctor and cleanup - **`--doctor`**: standalone environment check. No config file required. - Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity. -- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox + - removes incus remotes + clears local ISO cache. -- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the - `[incusos-lab:managed]` marker in the configured resource pool. + Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally + Proxmox connectivity (from proxmox.yaml or config file). +- **`--cleanup`**: destroys VMs defined in the config file. +- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this + deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete + all IncusOS ISOs (unlike the old behavior). +- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab + config required). Destroys all VMs with `[incusos-lab:managed]` marker. +- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso` + and `seed-*.iso` from storage + remotes + cache. +- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default + output is concise (step names + results). `--quiet` suppresses everything + except warnings and errors. ## Coding conventions for scripts diff --git a/incusos/README.md b/incusos/README.md index 0d79dbf..a01a813 100644 --- a/incusos/README.md +++ b/incusos/README.md @@ -296,13 +296,16 @@ This script ensures all settings are correct, preventing install failures. --no-cert Skip certificate injection --vm NAME Operate on a single VM only -y, --yes Skip confirmation prompts + --proxmox FILE Proxmox connection config (default: auto-discover) --doctor Check tool versions and environment health --status Check deployment status (shortcut for --phase status) --cleanup Destroy VMs defined in config - --cleanup --deep Also remove ISOs, seeds, remotes, and local cache - --cleanup-all Destroy ALL managed VMs in the resource pool + --cleanup --deep Also remove deployment ISO, seeds, remotes, and cache + --cleanup-all Destroy ALL managed VMs in pool (no config file needed) + --cleanup-all --deep Also remove ALL ISOs and seeds from storage --force-cleanup Destroy VMs even without management marker --dry-run Preview without executing +-v, --verbose Show detailed output (tool paths, API calls) -q, --quiet Suppress info output -h, --help Full help text -V, --version Show version @@ -314,14 +317,15 @@ Run `--doctor` to verify your environment without a config file: ```bash ./incusos-proxmox --doctor +./incusos-proxmox --doctor -v # verbose: show tool paths ``` This checks: incus client, flasher-tool, operations-center CLI (optional), -python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN -availability. +python3, curl, genisoimage, jq, incusos-seed sibling, IncusOS CDN +availability, and `proxmox.yaml` discovery. -If a config file is also provided, additionally checks Proxmox connectivity, -pool existence, and API token validity: +If `proxmox.yaml` is found or a config file is provided, additionally checks +Proxmox connectivity, pool existence, and API token validity: ```bash ./incusos-proxmox --doctor examples/lab-cluster.yaml @@ -329,22 +333,28 @@ pool existence, and API token validity: ### Deep Cleanup -Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts: +Use `--deep` with `--cleanup` or `--cleanup-all` to remove artifacts beyond VMs: ```bash -# VMs + ISOs + seeds + incus remotes + local cache +# VMs + the IncusOS ISO used by this deployment + seeds + remotes + cache ./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml -# Pool-wide: destroy all managed VMs + artifacts -./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml +# Pool-wide: destroy all managed VMs (uses proxmox.yaml, no config needed) +./incusos-proxmox --cleanup-all +./incusos-proxmox --cleanup-all --deep # also nuke ALL ISOs + seeds ``` -`--deep` additionally: -- Deletes seed ISOs from Proxmox storage -- Deletes IncusOS ISOs from Proxmox storage +`--cleanup --deep`: +- Deletes per-VM seed ISOs from Proxmox storage +- Deletes the specific IncusOS ISO used by this deployment (not all of them) - Removes matching `incus remote` entries - Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`) +`--cleanup-all --deep`: +- Deletes ALL `IncusOS_*.iso` and `seed-*.iso` from storage (intentionally aggressive) +- Removes remotes for the destroyed VMs +- Clears the local ISO cache + ### Deployment Phases | Phase | What it does | @@ -356,18 +366,41 @@ Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts: | `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk | | `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks | -### Config File Format +### Proxmox Connection Config + +Proxmox connection settings live in a separate `proxmox.yaml` file to avoid +duplicating credentials across lab configs. Auto-discovered in the script +directory and current directory. Override with `--proxmox FILE`. + +Copy the template to get started: + +```bash +cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml +# Edit with your Proxmox host, credentials, and storage settings +``` ```yaml +# proxmox.yaml (flat structure, no wrapping key) +host: 192.168.1.10 +method: api +api_token_id: automation@pve!deploy +node: pve +storage: local-lvm +iso_storage: local +bridge: vmbr0 +pool: IncusLab +ssh_user: root +``` + +Lab configs can override any setting by including a `proxmox:` section. +Merge priority: `proxmox.yaml` → lab config `proxmox:` → CLI flags. + +### Lab Config File Format + +```yaml +# proxmox: section is optional — overrides proxmox.yaml if present proxmox: - host: 192.168.1.10 # Proxmox host (required) - node: pve # Node name (default: pve) - storage: local-lvm # VM disk storage (default: local-lvm) - iso_storage: local # ISO storage (default: local) - bridge: vmbr0 # Network bridge (default: vmbr0) - pool: IncusLab # Resource pool (optional, for isolation) - method: ssh # ssh (default) or api - ssh_user: root # For ssh method + storage: local-zfs # Override storage for this deployment defaults: cores: 4 @@ -602,10 +635,11 @@ See the [`examples/`](examples/) directory for sample seed configurations: - [`incus-cluster-node.yaml`](examples/incus-cluster-node.yaml) -- Cluster node (no defaults) - [`ops-center.yaml`](examples/ops-center.yaml) -- Operations Center - [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration -- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster) -- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM -- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation -- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes +- [`proxmox.yaml.example`](examples/proxmox.yaml.example) -- Proxmox connection config template +- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (self-contained, includes proxmox: section) +- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM (self-contained) +- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster (uses proxmox.yaml) +- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes (uses proxmox.yaml) ## Security Notes diff --git a/incusos/TESTING.md b/incusos/TESTING.md index cb7307d..864a493 100644 --- a/incusos/TESTING.md +++ b/incusos/TESTING.md @@ -129,13 +129,23 @@ ssh root@ pvesh get /version You should see JSON with the Proxmox VE version. If this fails, fix SSH key auth first (`ssh-copy-id root@`). -### Step 2: Create a Test Config +### Step 2: Set Up Proxmox Connection + +```bash +cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml +``` + +Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is +auto-discovered by the scripts and avoids repeating credentials in each +lab config. + +### Step 3: Create a Test Config ```bash cp incusos/examples/proxmox-minimal.yaml my-test.yaml ``` -Edit `my-test.yaml` -- change only the host IP: +Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml): ```yaml proxmox: @@ -155,7 +165,7 @@ vms: Using VMID 900+ avoids clashing with existing VMs. -### Step 3: Dry Run +### Step 4: Dry Run ```bash ./incusos/incusos-proxmox --dry-run my-test.yaml @@ -172,7 +182,7 @@ Using VMID 900+ avoids clashing with existing VMs. - `[dry-run]` prefix on all actions -- nothing actually executed - No errors -### Step 4: Deploy (Single VM) +### Step 5: Deploy (Single VM) ```bash ./incusos/incusos-proxmox my-test.yaml @@ -196,7 +206,7 @@ Using VMID 900+ avoids clashing with existing VMs. disk, and reboot - After reboot you should see the IncusOS login prompt -### Step 5: Check Deployment Status +### Step 6: Check Deployment Status After the deploy completes, verify with the built-in status command: @@ -212,7 +222,7 @@ After the deploy completes, verify with the built-in status command: - Port 8443 is open - Incus remote status shown (if configured) -### Step 6: Verify the Deployed VM +### Step 7: Verify the Deployed VM Once the script reports success with an IP address: @@ -233,7 +243,7 @@ incus network list incus-test: ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong, check the Proxmox console — stale ARP entries can be misleading. -### Step 7: Test Re-run Reconcile +### Step 8: Test Re-run Reconcile Re-run the deploy to verify the reconcile menu: @@ -252,7 +262,7 @@ Also test with `--yes` to verify it defaults to safe behavior: **Verify:** Runs post-deployment checks, does NOT destroy. -### Step 8: Clean Up the Test VM +### Step 9: Clean Up the Test VM ```bash ./incusos/incusos-proxmox --cleanup my-test.yaml @@ -269,7 +279,7 @@ Remove the remote too: incus remote remove incus-test ``` -### Step 9: (Optional) Test API Method +### Step 10: (Optional) Test API Method After single-VM success with SSH, switch to API tokens. @@ -302,9 +312,9 @@ proxmox: api_token_id: automation@pve!deploy ``` -Repeat steps 3-6 using the API method. +Repeat steps 4-7 using the API method. -### Step 10: (Optional) Test with Resource Pool +### Step 11: (Optional) Test with Resource Pool For API method with pool isolation: @@ -331,7 +341,7 @@ Deploy and verify: ./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool ``` -### Step 11: (Optional) Full Lab Deployment +### Step 12: (Optional) Full Lab Deployment Once single-VM tests pass, deploy the full 4-VM lab: @@ -392,32 +402,229 @@ incus remote add incus-lab-03 --accept-certificate ### Manual Cluster Formation (Alternative) -If you prefer to form the cluster manually instead of using `lab-test`: +If you prefer to form the cluster manually instead of using `lab-test`, +follow the steps below. See also `notes/clustering-guide.md` for the full +reference guide with explanations of what happens under the hood. + +#### Prerequisites ```bash -# Step 1: Enable clustering on first node -incus cluster enable incus-lab-01: incus-lab-01 +# Check client version (need 6.20+ for remote cluster join) +incus version -# Step 2: Generate join token for second node -incus cluster add incus-lab-01: incus-lab-02 +# Verify all three remotes are working +incus info incus-lab-01: | head -5 +incus info incus-lab-02: | head -5 +incus info incus-lab-03: | head -5 +``` -# Step 3: Join second node -incus cluster join incus-lab-01: incus-lab-02: +#### Step 1: Set specific IPs on all nodes -# Step 4: Repeat for third node -incus cluster add incus-lab-01: incus-lab-03 -incus cluster join incus-lab-01: incus-lab-03: +IncusOS defaults to `core.https_address: :8443` (wildcard). Clustering +requires a specific routable IP on each node. -# Step 5: Verify +```bash +# Discover each node's routable IP +for node in incus-lab-01 incus-lab-02 incus-lab-03; do + echo "=== $node ===" + incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \ + [print(a) for a in d['environment']['addresses'] \ + if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]" +done + +# Set the specific IP on each node (replace with actual IPs) +incus config set incus-lab-01: core.https_address :8443 +incus config set incus-lab-02: core.https_address :8443 +incus config set incus-lab-03: core.https_address :8443 + +# Verify connectivity still works after each change +incus info incus-lab-01: | head -3 +incus info incus-lab-02: | head -3 +incus info incus-lab-03: | head -3 +``` + +#### Step 2: Enable clustering on the init node + +```bash +incus cluster enable incus-lab-01:incus-lab-01 +``` + +> **Important**: This regenerates the server's TLS certificate. Your remote +> will break with a certificate error. Fix it: + +```bash +incus remote switch local # if incus-lab-01 is the current default +incus remote remove incus-lab-01 +incus remote add incus-lab-01 https://:8443 --accept-certificate +``` + +Verify: + +```bash incus cluster list incus-lab-01: +# Should show one member (incus-lab-01), status ONLINE, role database-leader +``` + +#### Step 3: Prepare joining nodes + +Nodes with `apply_defaults: true` have a pre-existing `local` storage pool +and `incusbr0` network. These conflict with the cluster join process (the +join wizard tries to set member-specific ZFS config keys at cluster level). +Delete them before joining: + +```bash +# For each joining node (incus-lab-02, incus-lab-03): +NODE=incus-lab-02 # then repeat with incus-lab-03 + +# Remove server-level references to the pool +incus config unset "$NODE": storage.backups_volume +incus config unset "$NODE": storage.images_volume + +# Delete the backup and image volumes +incus storage volume delete "$NODE":local backups +incus storage volume delete "$NODE":local images + +# Clear default profile references (otherwise pool is "in use") +incus profile device remove "$NODE":default root +incus profile device remove "$NODE":default eth0 + +# Delete pool and network +incus storage delete "$NODE":local +incus network delete "$NODE":incusbr0 +``` + +#### Step 4: Join nodes to the cluster + +**Interactive** (one node at a time): + +```bash +# Generate join token +incus cluster add incus-lab-01:incus-lab-02 + +# Join (answer the interactive prompts) +incus cluster join incus-lab-01: incus-lab-02: +# Prompt 1 - IP address: press Enter (accept default) +# Prompt 2 - Member name: press Enter (accept default) +# Prompt 3 - "All existing data is lost": type "yes" +# Prompt 4 - source for storage pool "local": type "local/incus" +# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus" +``` + +**Automated** (non-interactive): + +```bash +# Generate token + join in one go +incus cluster add incus-lab-01:incus-lab-02 +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02: +``` + +After each join, fix the remote (same certificate issue as the init node): + +```bash +incus remote remove incus-lab-02 +incus remote add incus-lab-02 https://:8443 --accept-certificate +``` + +Repeat for incus-lab-03. + +#### Step 5: Verify the cluster + +```bash +# All 3 members should be ONLINE +incus cluster list incus-lab-01: + +# Storage pool should show on all members +incus storage show incus-lab-01:local +incus storage show incus-lab-01:local --target incus-lab-01 +incus storage show incus-lab-01:local --target incus-lab-02 +incus storage show incus-lab-01:local --target incus-lab-03 + +# All remotes should work +incus info incus-lab-01: | head -3 +incus info incus-lab-02: | head -3 +incus info incus-lab-03: | head -3 ``` **Key points:** - `apply_defaults: true` on all nodes ensures matching storage pool and - network bridge configurations (required for migration) + network bridge names (required for migration) -- but the pools must be + deleted on joining nodes before the join - After joining, manage the cluster through the init node's remote +- The ZFS pool source is `local/incus` on IncusOS (auto-created by + `apply_defaults`) +- Each join generates a new token (tokens are single-use and time-limited) - VM migration requires matching storage pool names across nodes +### Migration Testing (After Cluster Formation) + +Once the cluster is formed, validate container and VM migration: + +#### Container migration (stop/move/start) + +```bash +# Launch on node 2 +incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02 +incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt' + +# Migrate to node 3 +incus stop incus-lab-01:test-container +incus move incus-lab-01:test-container --target incus-lab-03 +incus start incus-lab-01:test-container + +# Verify: data survives, processes don't +incus exec incus-lab-01:test-container -- cat /root/test.txt # "hello" +incus list incus-lab-01: --columns nLs # node 3 +``` + +#### VM live migration + +```bash +# Launch and configure for migration +incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02 +# Wait for VM agent... +incus stop incus-lab-01:test-vm +incus config set incus-lab-01:test-vm migration.stateful=true +incus config set incus-lab-01:test-vm limits.cpu=0-1 +incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB +incus start incus-lab-01:test-vm + +# Set up heartbeat for state verification +incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test +incus exec incus-lab-01:test-vm -- bash -c \ + 'cat > /tmp/migration-test/heartbeat.sh << "EOF" +#!/bin/bash +i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done +EOF +chmod +x /tmp/migration-test/heartbeat.sh' +incus exec incus-lab-01:test-vm -- \ + systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh + +# Live-migrate (state preserved) +sleep 3 +incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # note value +incus move incus-lab-01:test-vm --target incus-lab-03 +sleep 2 # agent needs ~1-2s to reconnect +incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # should be higher +incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat # "active" +``` + +#### Cluster evacuation + +```bash +# Move both workloads to node 2, then evacuate +incus cluster evacuate incus-lab-01:incus-lab-02 --force +incus cluster list incus-lab-01: # node 2 = EVACUATED +incus list incus-lab-01: --columns nLs # workloads on other nodes + +# Restore +incus cluster restore incus-lab-01:incus-lab-02 --force +incus cluster list incus-lab-01: # node 2 = ONLINE +incus list incus-lab-01: --columns nLs # workloads back on node 2 +``` + +See `notes/clustering-guide.md` for the full automated migration test script +and detailed test results. + ### Cleanup ```bash @@ -427,8 +634,12 @@ incus cluster list incus-lab-01: # Destroy VMs only ./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml -# Full cleanup: VMs + ISOs + seeds + remotes + cache +# Full cleanup: VMs + deployment ISO + seeds + remotes + cache ./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml + +# Pool-wide: destroy ALL managed VMs (no config file needed) +./incusos/incusos-proxmox --cleanup-all --yes +./incusos/incusos-proxmox --cleanup-all --deep --yes # nuke everything ``` --- @@ -507,3 +718,7 @@ CLI is installed. | Pool not found | Resource pool not created | Run `pveum pool add ` on Proxmox | | Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead | | `--status` shows "not found" | VM was cleaned up | Deploy first, then check status | +| VM live migration fails: "Missing section footer for ICH9LPC" | `limits.cpu` not set or set as integer | Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md` | +| `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start | +| `certificate is valid for 127.0.0.1, ::1, not ` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` | +| `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide | diff --git a/incusos/examples/lab-advanced.yaml b/incusos/examples/lab-advanced.yaml new file mode 100644 index 0000000..cb21d5a --- /dev/null +++ b/incusos/examples/lab-advanced.yaml @@ -0,0 +1,45 @@ +# lab-advanced.yaml - 3-node heterogeneous cluster for advanced migration testing +# +# Deploys 3 Incus nodes with a 6/4/4 core split to test VM live migration +# across hosts with different CPU topologies. 8 GiB RAM provides headroom +# for multiple concurrent VMs with size.state for stateful operations. +# +# Test matrix targets: +# - Multi-vCPU VMs (2, 3, 4 vCPUs) with limits.cpu ranges +# - Concurrent migrations from different source nodes +# - Edge cases: active I/O, memory-loaded VMs, 4-vCPU on 4-core host +# - Cluster lifecycle: evacuate, remove, replace, rejoin, rebalance +# +# Usage: +# incusos-proxmox --dry-run examples/lab-advanced.yaml # Preview +# incusos-proxmox --yes examples/lab-advanced.yaml # Deploy +# incusos-proxmox --status examples/lab-advanced.yaml # Check status +# incusos-proxmox --cleanup examples/lab-advanced.yaml # Tear down VMs +# incusos-proxmox --cleanup --deep examples/lab-advanced.yaml # Full cleanup +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). +# Add a proxmox: section here to override per-deployment. +# +# Note: start_vmid 910 may collide with lab-oc.yaml. Clean up old deployments +# first or adjust the range. + +defaults: + memory: 8192 + disk: 50 + start_vmid: 910 + +vms: + - name: incus-adv-01 + app: incus + apply_defaults: true + cores: 6 + + - name: incus-adv-02 + app: incus + apply_defaults: true + cores: 4 + + - name: incus-adv-03 + app: incus + apply_defaults: true + cores: 4 diff --git a/incusos/examples/lab-cluster.yaml b/incusos/examples/lab-cluster.yaml index 2c49b0e..5684106 100644 --- a/incusos/examples/lab-cluster.yaml +++ b/incusos/examples/lab-cluster.yaml @@ -13,16 +13,12 @@ # # After deployment, form a cluster with lab-test: # lab-test --phase cluster examples/lab-cluster.yaml - -proxmox: - host: 192.168.1.29 - method: api - api_token_id: automation@pve!deploy - storage: local-zfs - pool: IncusLab +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). +# Add a proxmox: section here to override per-deployment. defaults: - cores: 2 + cores: 4 memory: 4096 disk: 50 start_vmid: 900 @@ -31,9 +27,6 @@ vms: - name: incus-lab-01 app: incus apply_defaults: true - cores: 4 - memory: 8192 - disk: 64 - name: incus-lab-02 app: incus diff --git a/incusos/examples/lab-oc.yaml b/incusos/examples/lab-oc.yaml index ca0bd99..9362abc 100644 --- a/incusos/examples/lab-oc.yaml +++ b/incusos/examples/lab-oc.yaml @@ -15,13 +15,9 @@ # incusos-proxmox --yes examples/lab-oc.yaml # incusos-proxmox --status examples/lab-oc.yaml # incusos-proxmox --cleanup --deep examples/lab-oc.yaml - -proxmox: - host: 192.168.1.29 - method: api - api_token_id: automation@pve!deploy - storage: local-zfs - pool: IncusLab +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). +# Add a proxmox: section here to override per-deployment. defaults: cores: 2 diff --git a/incusos/examples/lab-replace.yaml b/incusos/examples/lab-replace.yaml new file mode 100644 index 0000000..afc4ce5 --- /dev/null +++ b/incusos/examples/lab-replace.yaml @@ -0,0 +1,24 @@ +# lab-replace.yaml - Single replacement node for cluster lifecycle testing +# +# Used in Part 4 of the advanced migration test plan: after evacuating, +# removing, and destroying incus-adv-03, this config deploys a fresh +# replacement node that gets joined back into the cluster. +# +# Usage: +# incusos-proxmox --dry-run examples/lab-replace.yaml +# incusos-proxmox --yes examples/lab-replace.yaml +# incusos-proxmox --status examples/lab-replace.yaml +# incusos-proxmox --cleanup examples/lab-replace.yaml +# +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). + +defaults: + memory: 8192 + disk: 50 + start_vmid: 913 + +vms: + - name: incus-adv-04 + app: incus + apply_defaults: true + cores: 4 diff --git a/incusos/examples/my-test.yaml b/incusos/examples/my-test.yaml index 0b0e004..e9ee826 100644 --- a/incusos/examples/my-test.yaml +++ b/incusos/examples/my-test.yaml @@ -1,9 +1,5 @@ -proxmox: - host: 192.168.1.29 - method: api - api_token_id: automation@pve!deploy - storage: local-zfs # VM disk storage (default: local-lvm) - pool: IncusLab +# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example). +# Add a proxmox: section here to override per-deployment. defaults: cores: 2 diff --git a/incusos/examples/proxmox.yaml.example b/incusos/examples/proxmox.yaml.example new file mode 100644 index 0000000..8342be4 --- /dev/null +++ b/incusos/examples/proxmox.yaml.example @@ -0,0 +1,21 @@ +# proxmox.yaml.example - Proxmox connection config template +# +# Copy to incusos/proxmox.yaml and edit with your settings. +# This file is auto-discovered by incusos-proxmox and lab-test. +# +# Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference +# this for connection settings, so you only need to change credentials +# in one place. +# +# Override any setting per-deployment by adding a proxmox: section +# to the lab config file. + +host: 192.168.1.10 # Proxmox host IP or hostname +method: ssh # ssh (default) or api +# api_token_id: automation@pve!deploy # For api method +node: pve # Proxmox node name +storage: local-lvm # VM disk storage +iso_storage: local # ISO/image storage +bridge: vmbr0 # Network bridge +# pool: IncusLab # Resource pool (optional, for isolation) +ssh_user: root # SSH user (for ssh method) diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index 154d78b..f93a6b7 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -43,7 +43,7 @@ success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET} warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; } error() { echo -e "${RED}[error]${RESET} $*" >&2; } step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; } -detail() { [[ "$QUIET" == true ]] && return; echo -e "${DIM} $*${RESET}"; } +detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; } # --------------------------------------------------------------------------- # Help @@ -91,25 +91,39 @@ ${BOLD}OPTIONS${RESET} ${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only) ${BOLD}--cleanup --deep${RESET} Also remove ISOs, seeds, remotes, and local cache ${BOLD}--cleanup-all${RESET} Destroy ALL managed VMs in the pool + ${BOLD}--proxmox${RESET} FILE Proxmox connection config (default: auto-discover) ${BOLD}--force-cleanup${RESET} Destroy VMs even without management marker ${BOLD}--dry-run${RESET} Preview actions without executing + ${BOLD}-v, --verbose${RESET} Show detailed output (tool paths, API calls) ${BOLD}-q, --quiet${RESET} Suppress informational output ${BOLD}-h, --help${RESET} Show this help message ${BOLD}-V, --version${RESET} Show version +${BOLD}PROXMOX CONNECTION CONFIG${RESET} + Proxmox connection settings can be stored in a separate proxmox.yaml file + to avoid duplicating credentials across lab configs. Auto-discovered in: + script directory, then current directory. Override with --proxmox FILE. + + ${BOLD}proxmox.yaml${RESET} (flat structure, no wrapping key): + host: 192.168.1.10 # Proxmox host (required) + method: api # ssh or api (default: ssh) + api_token_id: user!token # API token (for api method) + node: pve # Node name (default: pve) + storage: local-lvm # VM disk storage (default: local-lvm) + iso_storage: local # ISO storage (default: local) + bridge: vmbr0 # Network bridge (default: vmbr0) + pool: IncusLab # Resource pool (optional) + ssh_user: root # SSH user (default: root) + + Merge priority: proxmox.yaml → lab config proxmox: section → CLI flags + ${BOLD}CONFIG FILE${RESET} - YAML file defining the Proxmox connection and VM specifications. + YAML file defining VM specifications (and optional Proxmox overrides). See examples/proxmox-lab.yaml for a full example. - ${BOLD}proxmox:${RESET} - host: 192.168.1.10 # Proxmox host (required) - node: pve # Node name (default: pve) - storage: local-lvm # VM disk storage (default: local-lvm) - iso_storage: local # ISO storage (default: local) - bridge: vmbr0 # Network bridge (default: vmbr0) - pool: IncusLab # Resource pool (optional) - method: ssh # ssh or api (default: ssh) - ssh_user: root # SSH user (default: root) + ${BOLD}proxmox:${RESET} # Optional: overrides proxmox.yaml + host: 192.168.1.10 # Proxmox host + storage: local-zfs # Override storage for this deployment ${BOLD}defaults:${RESET} cores: 4 # Default CPU cores @@ -152,16 +166,20 @@ ${BOLD}EXAMPLES${RESET} # Destroy all VMs defined in config ${SCRIPT_NAME} --cleanup examples/proxmox-lab.yaml - # Deep cleanup: VMs + ISOs + seeds + remotes + cache + # Deep cleanup: VMs + the IncusOS ISO used + seeds + remotes + cache ${SCRIPT_NAME} --cleanup --deep examples/proxmox-lab.yaml - # Destroy ALL managed VMs in the pool (not config-specific) - ${SCRIPT_NAME} --cleanup-all examples/proxmox-lab.yaml + # Destroy ALL managed VMs in the pool (uses proxmox.yaml, no config needed) + ${SCRIPT_NAME} --cleanup-all + ${SCRIPT_NAME} --cleanup-all --deep # also remove all ISOs + seeds # Check tool versions and environment health ${SCRIPT_NAME} --doctor ${SCRIPT_NAME} --doctor examples/proxmox-lab.yaml # also check Proxmox + # Verbose output (show dependency paths, API calls, per-file actions) + ${SCRIPT_NAME} --doctor -v + # Use API method with token export PROXMOX_TOKEN_SECRET="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" ${SCRIPT_NAME} --method api examples/proxmox-lab.yaml @@ -228,6 +246,8 @@ DEEP_CLEANUP=false FORCE_CLEANUP=false DRY_RUN=false QUIET=false +VERBOSE=false +PROXMOX_CONFIG="" CONFIG_FILE="" parse_args() { @@ -290,10 +310,18 @@ parse_args() { FORCE_CLEANUP=true shift ;; + --proxmox) + PROXMOX_CONFIG="${2:?'--proxmox requires a file path'}" + shift 2 + ;; --dry-run) DRY_RUN=true shift ;; + -v|--verbose) + VERBOSE=true + shift + ;; -q|--quiet) QUIET=true shift @@ -329,11 +357,22 @@ parse_args() { # --------------------------------------------------------------------------- validate_args() { + # Universal checks (apply regardless of mode) + if [[ "$VERBOSE" == true ]] && [[ "$QUIET" == true ]]; then + error "Cannot use both --verbose and --quiet" + exit 1 + fi + # --doctor can run without a config file if [[ "$DOCTOR" == true ]] && [[ -z "$CONFIG_FILE" ]]; then return 0 fi + # --cleanup-all can run without a config file (uses proxmox.yaml) + if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then + return 0 + fi + if [[ -z "$CONFIG_FILE" ]]; then error "No configuration file specified" echo "Usage: ${SCRIPT_NAME} [OPTIONS] CONFIG_FILE" >&2 @@ -605,7 +644,19 @@ if versions: echo -e " Cached ISO: ${DIM}none${RESET}" fi - # -- Proxmox section (only if config file provided) -- + # -- Config section -- + echo "" + echo -e " ${BOLD}Config:${RESET}" + if find_proxmox_config; then + echo -e " proxmox.yaml: ${GREEN}${PROXMOX_CONFIG_RESOLVED} ✓${RESET}" + else + echo -e " proxmox.yaml: ${DIM}not found (optional)${RESET}" + fi + if [[ -n "$CONFIG_FILE" ]] && [[ -f "$CONFIG_FILE" ]]; then + echo -e " Lab config: ${CONFIG_FILE}" + fi + + # -- Proxmox section (if config file or proxmox.yaml provides connection info) -- echo "" if [[ -n "$CONFIG_FILE" ]] && [[ -f "$CONFIG_FILE" ]]; then echo -e " ${BOLD}Proxmox:${RESET}" @@ -713,9 +764,68 @@ json.dump(data, sys.stdout, indent=2) echo -e " ${RED}Failed to parse config file${RESET}" fi + rm -rf "$doctor_workdir" + elif [[ -n "$PROXMOX_CONFIG_RESOLVED" ]]; then + # Use proxmox.yaml for connectivity check (no lab config provided) + echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(from proxmox.yaml)${RESET}" + + local doctor_workdir + doctor_workdir=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-doctor-XXXXXX") + WORKDIR="$doctor_workdir" + + load_proxmox_config "$PROXMOX_CONFIG_RESOLVED" + [[ -z "$PVE_NODE" ]] && PVE_NODE="pve" + [[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh" + [[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root" + [[ -n "$METHOD" ]] && PVE_METHOD="$METHOD" + [[ -n "$HOST_OVERRIDE" ]] && PVE_HOST="$HOST_OVERRIDE" + + if [[ -z "$PVE_HOST" ]]; then + echo -e " Host: ${RED}not set in proxmox.yaml${RESET}" + else + echo -e " Host: ${PVE_HOST} (${PVE_METHOD})" + local pve_ok=false + if [[ "$PVE_METHOD" == "api" ]]; then + if [[ -n "$PVE_API_TOKEN_ID" ]] && [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]]; then + local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}" + local pve_ver + pve_ver=$(curl -fsSk --connect-timeout 5 -H "$auth" \ + "https://${PVE_HOST}:8006/api2/json/version" 2>/dev/null | \ + python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('version',''))" 2>/dev/null) || pve_ver="" + if [[ -n "$pve_ver" ]]; then + echo -e " Connectivity: ${GREEN}connected (PVE ${pve_ver}) ✓${RESET}" + pve_ok=true + else + echo -e " Connectivity: ${RED}API connection failed${RESET}" + fi + else + echo -e " Connectivity: ${YELLOW}API token not configured or PROXMOX_TOKEN_SECRET not set${RESET}" + fi + else + if ssh -o BatchMode=yes -o ConnectTimeout=5 "${PVE_SSH_USER}@${PVE_HOST}" "true" &>/dev/null; then + echo -e " Connectivity: ${GREEN}SSH connected ✓${RESET}" + pve_ok=true + else + echo -e " Connectivity: ${RED}SSH connection failed${RESET}" + fi + fi + if [[ -n "$PVE_POOL" ]]; then + if [[ "$pve_ok" == true ]] && [[ "$PVE_METHOD" == "api" ]]; then + local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}" + if curl -fsSk --connect-timeout 5 -H "$auth" \ + "https://${PVE_HOST}:8006/api2/json/pools/${PVE_POOL}" &>/dev/null; then + echo -e " Pool: ${GREEN}${PVE_POOL} exists ✓${RESET}" + else + echo -e " Pool: ${RED}${PVE_POOL} not found${RESET}" + fi + else + echo -e " Pool: ${PVE_POOL} (not verified)" + fi + fi + fi rm -rf "$doctor_workdir" else - echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(requires config file, skipped)${RESET}" + echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(no config file or proxmox.yaml found)${RESET}" fi echo "" @@ -1106,29 +1216,121 @@ declare -a VM_VMIDS=() declare -a VM_CLUSTERS=() declare -a VM_CLUSTER_ROLES=() +# Path to the resolved proxmox connection config file (set by find_proxmox_config) +PROXMOX_CONFIG_RESOLVED="" + +# Auto-discover proxmox.yaml: --proxmox flag > script directory > cwd +find_proxmox_config() { + if [[ -n "$PROXMOX_CONFIG" ]]; then + if [[ ! -f "$PROXMOX_CONFIG" ]]; then + error "Proxmox config file not found: ${PROXMOX_CONFIG}" + exit 1 + fi + PROXMOX_CONFIG_RESOLVED="$PROXMOX_CONFIG" + detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (--proxmox)" + return 0 + fi + + local script_dir + script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + + if [[ -f "${script_dir}/proxmox.yaml" ]]; then + PROXMOX_CONFIG_RESOLVED="${script_dir}/proxmox.yaml" + detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (auto-discovered)" + return 0 + fi + + if [[ -f "proxmox.yaml" ]]; then + PROXMOX_CONFIG_RESOLVED="$(pwd)/proxmox.yaml" + detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (auto-discovered)" + return 0 + fi + + # No proxmox.yaml found -- caller decides if this is an error + return 1 +} + +# Parse proxmox.yaml (flat key: value structure) and set PVE_* globals as base values +load_proxmox_config() { + local pxconfig="$1" + + local px_json + px_json=$(mktemp "${WORKDIR}/proxmox-config-XXXXXX.json") + + if ! python3 -c " +import json, sys, re + +result = {} +with open(sys.argv[1]) as f: + for line in f: + s = line.strip() + if not s or s.startswith('#'): + continue + m = re.match(r'^([^:]+?):\s*(.*?)$', s) + if m: + key = m.group(1).strip() + val = m.group(2).strip() + # Strip inline comments + if val and not val.startswith(('\"', \"'\")): + cm = re.match(r'^(.*?)\s+#.*$', val) + if cm: + val = cm.group(1).rstrip() + if (val.startswith('\"') and val.endswith('\"')) or (val.startswith(\"'\") and val.endswith(\"'\")): + val = val[1:-1] + result[key] = val + +json.dump(result, sys.stdout, indent=2) +" "$pxconfig" > "$px_json" 2>/dev/null; then + error "Failed to parse proxmox config: ${pxconfig}" + exit 1 + fi + + # Set base values from proxmox.yaml (lab config overlay and CLI flags applied later) + local val + val=$(json_get "$px_json" ".host" ""); [[ -n "$val" ]] && PVE_HOST="$val" + val=$(json_get "$px_json" ".node" ""); [[ -n "$val" ]] && PVE_NODE="$val" + val=$(json_get "$px_json" ".storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val" + val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val" + val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val" + val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val" + val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val" + val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val" + val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val" + + rm -f "$px_json" +} + load_config() { step "Loading configuration values" - # Proxmox settings - PVE_HOST=$(json_get "$CONFIG_JSON" ".proxmox.host" "") - PVE_NODE=$(json_get "$CONFIG_JSON" ".proxmox.node" "pve") - PVE_STORAGE=$(json_get "$CONFIG_JSON" ".proxmox.storage" "local-lvm") - PVE_ISO_STORAGE=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" "local") - PVE_BRIDGE=$(json_get "$CONFIG_JSON" ".proxmox.bridge" "vmbr0") - PVE_SSH_USER=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" "root") - PVE_API_TOKEN_ID=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" "") - PVE_POOL=$(json_get "$CONFIG_JSON" ".proxmox.pool" "") + # Merge priority: proxmox.yaml (base) → lab config proxmox: (overlay) → CLI flags + # Step 1: proxmox.yaml base values were already loaded by the caller via + # load_proxmox_config(). PVE_* globals may already be set. - # Method: CLI override > config > default - if [[ -z "$PVE_METHOD" ]]; then - PVE_METHOD=$(json_get "$CONFIG_JSON" ".proxmox.method" "ssh") - fi - # CLI override for method + # Step 2: overlay from lab config proxmox: section (if present) + local val + val=$(json_get "$CONFIG_JSON" ".proxmox.host" ""); [[ -n "$val" ]] && PVE_HOST="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.node" ""); [[ -n "$val" ]] && PVE_NODE="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.method" ""); [[ -n "$val" ]] && PVE_METHOD="$val" + + # Step 3: apply defaults for anything still unset + [[ -z "$PVE_NODE" ]] && PVE_NODE="pve" + [[ -z "$PVE_STORAGE" ]] && PVE_STORAGE="local-lvm" + [[ -z "$PVE_ISO_STORAGE" ]] && PVE_ISO_STORAGE="local" + [[ -z "$PVE_BRIDGE" ]] && PVE_BRIDGE="vmbr0" + [[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root" + [[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh" + + # Step 4: CLI flags override everything if [[ -n "$METHOD" ]]; then PVE_METHOD="$METHOD" fi - - # Host override if [[ -n "$HOST_OVERRIDE" ]]; then PVE_HOST="$HOST_OVERRIDE" fi @@ -1315,7 +1517,7 @@ pve_run() { local cmd="$1" if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] ${PVE_METHOD}: ${cmd}" + info "[dry-run] ${PVE_METHOD}: ${cmd}" return 0 fi @@ -1345,7 +1547,7 @@ pve_api() { local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}" if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] API ${method} ${endpoint}" + info "[dry-run] API ${method} ${endpoint}" if [[ -n "$data" ]]; then detail "[dry-run] data: ${data}" fi @@ -1387,7 +1589,7 @@ pve_upload() { local remote_filename="$2" if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] upload: ${local_path} -> ${PVE_ISO_STORAGE}:iso/${remote_filename}" + info "[dry-run] upload: ${local_path} -> ${PVE_ISO_STORAGE}:iso/${remote_filename}" return 0 fi @@ -1529,7 +1731,7 @@ pve_create_vm() { fi if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] ${cmd}" + info "[dry-run] ${cmd}" return 0 fi @@ -2498,6 +2700,11 @@ phase_validate() { step "Phase 1: Validate" echo "" + # Load proxmox.yaml base values (before lab config overlay) + if find_proxmox_config; then + load_proxmox_config "$PROXMOX_CONFIG_RESOLVED" + fi + parse_config load_config validate_config @@ -2527,7 +2734,7 @@ phase_prepare() { fi elif [[ "$DRY_RUN" == true ]]; then ISO_FILENAME="IncusOS_latest.iso" - detail "[dry-run] Would fetch latest version from CDN and download ISO" + info "[dry-run] Would fetch latest version from CDN and download ISO" else # Check if ISO already exists on Proxmox fetch_latest_version @@ -2581,7 +2788,7 @@ phase_prepare() { fi if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] ${seed_cmd[*]}" + info "[dry-run] ${seed_cmd[*]}" else if ! "${seed_cmd[@]}"; then error "Failed to generate seed for VM '${name}'" @@ -2708,6 +2915,78 @@ phase_create() { success "All VMs created" } +# --------------------------------------------------------------------------- +# Remote registration +# --------------------------------------------------------------------------- + +# Register an incus remote for a deployed VM. +# Handles stale remotes: if a remote exists but points to a different IP, +# it is removed and re-added with the correct IP (DHCP may reassign addresses +# between deploys). +# Usage: register_remote +register_remote() { + local name="$1" + local ip="$2" + + # Only register if incus client is available and a cert was injected + if ! command -v incus &>/dev/null; then + return + fi + if [[ -z "$CERT_FILE_RESOLVED" ]]; then + return + fi + + # Check if remote already exists + local existing_url="" + existing_url=$(incus remote list --format csv 2>/dev/null | \ + awk -F',' -v name="$name" '$1 == name {print $2; exit}') || true + + if [[ -n "$existing_url" ]]; then + # Remote exists -- check if the IP matches + if echo "$existing_url" | grep -q "${ip}"; then + detail "Remote '${name}' already configured at ${ip}" + return + fi + + # Stale remote -- IP changed (DHCP reassignment between deploys) + if [[ "$DRY_RUN" == true ]]; then + info "[dry-run] Would update stale remote '${name}': ${existing_url} -> ${ip}" + return + fi + + warn "Remote '${name}' points to ${existing_url} but VM is at ${ip} -- updating" + # Switch away first -- can't remove the current/default remote + local current_remote="" + current_remote=$(incus remote get-default 2>/dev/null) || true + if [[ "$current_remote" == "$name" ]]; then + incus remote switch local 2>/dev/null || true + fi + incus remote remove "$name" 2>/dev/null || true + fi + + if [[ "$DRY_RUN" == true ]]; then + info "[dry-run] Would add incus remote '${name}' at ${ip}" + return + fi + + # Wait for port 8443 to become reachable (up to 30s) + local attempt=0 + while [[ $attempt -lt 6 ]]; do + if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then + break + fi + sleep 5 + attempt=$((attempt + 1)) + done + + if incus remote add "$name" "$ip" --accept-certificate --quiet 2>/dev/null; then + success "Remote '${name}' added (${ip})" + else + warn "Could not add remote '${name}' at ${ip} -- add manually:" + warn " incus remote add ${name} ${ip} --accept-certificate" + fi +} + # --------------------------------------------------------------------------- # Phase 5: Install # --------------------------------------------------------------------------- @@ -2730,7 +3009,7 @@ phase_install() { if [[ "$DRY_RUN" == true ]]; then step "Starting VM '${name}' (VMID ${vmid})" - detail "[dry-run] Would start VM, swap boot order, wait for install, clean up" + info "[dry-run] Would start VM, swap boot order, wait for install, clean up" i=$((i + 1)) continue fi @@ -2743,12 +3022,13 @@ phase_install() { pve_vm_has_install_media "$vmid" || has_media=false if [[ "$cur_status" == "running" ]] && [[ "$has_media" == false ]]; then - # Already installed and running — detect IP and skip + # Already installed and running — detect IP, register remote, skip success "VM '${name}' (VMID ${vmid}) is already installed and running" local ip ip=$(pve_vm_ip "$vmid") || ip="" if [[ -n "$ip" ]]; then detail "IP: ${ip}" + register_remote "$name" "$ip" fi i=$((i + 1)) continue @@ -2763,6 +3043,7 @@ phase_install() { ip=$(pve_vm_ip "$vmid") || ip="" if [[ -n "$ip" ]]; then success "VM '${name}' started at ${ip}" + register_remote "$name" "$ip" else success "VM '${name}' started from disk" fi @@ -2805,7 +3086,7 @@ phase_install() { # installation is complete. We then stop the VM, remove install media, # and start cleanly from disk. step "Waiting for '${name}' to complete installation" - detail "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)" + info "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)" local timeout=600 local interval=5 @@ -2840,14 +3121,14 @@ phase_install() { local wr_mb=$(( wr_bytes / 1048576 )) # Show progress every 15s if [[ $((elapsed % 15)) -eq 0 ]]; then - detail "Installing... (${wr_mb} MiB written) [${elapsed}s]" + info "Installing... (${wr_mb} MiB written) [${elapsed}s]" fi elif [[ "$writes_started" == true ]]; then # Writes have stopped after being active stable_count=$((stable_count + 1)) if [[ $stable_count -ge $STABLE_THRESHOLD ]]; then local wr_mb=$(( wr_bytes / 1048576 )) - detail "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]" + info "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]" install_done=true break fi @@ -2873,7 +3154,7 @@ phase_install() { local cur_status cur_status=$(pve_vm_status "$vmid") || cur_status="unknown" if [[ "$cur_status" == "running" ]]; then - detail "Stopping VM to remove install media..." + info "Stopping VM to remove install media..." pve_stop_vm "$vmid" || true sleep 5 fi @@ -2892,7 +3173,7 @@ phase_install() { fi # 5. Wait for services to come up - detail "Waiting for services to start..." + info "Waiting for services to start..." sleep 30 # 6. Try to detect IP @@ -2902,12 +3183,10 @@ phase_install() { echo "" if [[ -n "$ip" ]]; then success "VM '${name}' installed and running at ${ip}" - if [[ -n "$CERT_FILE_RESOLVED" ]]; then - detail "Add remote: incus remote add ${name} ${ip}" - fi + register_remote "$name" "$ip" else success "VM '${name}' installed and running" - detail "Check Proxmox console for IP address" + info "Check Proxmox console for IP address" fi echo "" @@ -2988,7 +3267,7 @@ phase_status() { fi echo -e " Network: ${ip} port 8443: ${port_status}" - # Incus remote check + # Incus remote check + auto-register if command -v incus &>/dev/null; then local remote_name="" remote_name=$(incus remote list --format csv 2>/dev/null | \ @@ -2996,7 +3275,15 @@ phase_status() { if [[ -n "$remote_name" ]]; then echo -e " Remote: ${GREEN}incus remote '${remote_name}' configured${RESET}" else - echo -e " Remote: ${YELLOW}no incus remote for ${ip}${RESET}" + register_remote "$name" "$ip" + # Re-check after registration attempt + remote_name=$(incus remote list --format csv 2>/dev/null | \ + awk -F',' -v addr="$ip" '$2 ~ addr {print $1; exit}') || true + if [[ -n "$remote_name" ]]; then + echo -e " Remote: ${GREEN}incus remote '${remote_name}' configured${RESET}" + else + echo -e " Remote: ${YELLOW}no incus remote for ${ip}${RESET}" + fi fi fi else @@ -3032,7 +3319,7 @@ phase_post_deploy() { local app="${VM_APPS[$i]}" if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would check '${name}' (VMID ${vmid})" + info "[dry-run] Would check '${name}' (VMID ${vmid})" i=$((i + 1)) continue fi @@ -3089,13 +3376,22 @@ phase_post_deploy() { all_ok=false fi else - detail "VM '${name}': no incus remote configured" - detail " Add with: incus remote add ${name} ${ip} --accept-certificate" + # Auto-register the remote + register_remote "$name" "$ip" + # Verify + remote_name=$(incus remote list --format csv 2>/dev/null | \ + awk -F',' -v addr="$ip" '$2 ~ addr {print $1; exit}') || true + if [[ -n "$remote_name" ]]; then + success "VM '${name}': incus remote '${remote_name}' connected" + else + info "VM '${name}': no incus remote configured" + info " Add with: incus remote add ${name} ${ip} --accept-certificate" + fi fi elif [[ "$app" == "operations-center" ]]; then success "VM '${name}': Operations Center at https://${ip}:8443" - detail " Import PKCS#12 cert in browser for web UI access:" - detail " openssl pkcs12 -export -inkey ~/.config/incus/client.key -in ~/.config/incus/client.crt -out client.pfx" + info " Import PKCS#12 cert in browser for web UI access:" + info " openssl pkcs12 -export -inkey ~/.config/incus/client.key -in ~/.config/incus/client.crt -out client.pfx" fi i=$((i + 1)) @@ -3163,7 +3459,7 @@ do_cleanup() { step "Destroying VM '${name}' (VMID ${vmid})" if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would stop and destroy VM ${vmid}" + info "[dry-run] Would stop and destroy VM ${vmid}" destroyed=$((destroyed + 1)) i=$((i + 1)) continue @@ -3173,7 +3469,7 @@ do_cleanup() { local status status=$(pve_vm_status "$vmid") || status="unknown" if [[ "$status" == "running" ]]; then - detail "Stopping VM..." + info "Stopping VM..." pve_stop_vm "$vmid" || true sleep 5 fi @@ -3201,9 +3497,67 @@ do_cleanup() { success "$summary" } -# Deep cleanup: remove ISOs, seeds, remotes, and local cache +# Detect which IncusOS ISO was used by checking ide2 config on existing VMs. +# Sets DEPLOYMENT_ISO to the filename (e.g. "IncusOS_25.03.iso") or "" if unknown. +DEPLOYMENT_ISO="" + +detect_deployment_iso() { + DEPLOYMENT_ISO="" + + if [[ "$DRY_RUN" == true ]]; then + return + fi + + local count=${#VM_NAMES[@]} + local i=0 + while [[ $i -lt $count ]]; do + local vmid="${VM_VMIDS[$i]}" + if pve_vm_exists "$vmid"; then + local config="" + case "$PVE_METHOD" in + ssh) + config=$(pve_run "qm config ${vmid} 2>/dev/null" 2>/dev/null) || config="" + ;; + api) + config=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null) || config="" + ;; + esac + + if [[ -n "$config" ]]; then + local iso_name + # Extract IncusOS ISO name from ide2 line or JSON + iso_name=$(echo "$config" | python3 -c " +import sys, json, re +text = sys.stdin.read() +# Try JSON (API response) +try: + data = json.loads(text) + ide2 = data.get('data', data).get('ide2', '') +except (json.JSONDecodeError, AttributeError): + ide2 = '' + for line in text.splitlines(): + if line.startswith('ide2:'): + ide2 = line.split(':', 1)[1].strip() + break +m = re.search(r'(IncusOS_[^,]+\.iso)', ide2) +if m: + print(m.group(1)) +" 2>/dev/null) || iso_name="" + + if [[ -n "$iso_name" ]]; then + DEPLOYMENT_ISO="$iso_name" + detail "Detected deployment ISO: ${DEPLOYMENT_ISO} (from VM ${vmid})" + return + fi + fi + fi + i=$((i + 1)) + done +} + +# Deep cleanup: remove seeds and the specific IncusOS ISO used by this deployment cleanup_storage() { - step "Deep cleanup: removing ISOs and seeds from Proxmox storage" + step "Deep cleanup: removing seeds and deployment ISO from Proxmox storage" local count=${#VM_NAMES[@]} local i=0 @@ -3218,7 +3572,7 @@ cleanup_storage() { local seed_file="seed-${name}.iso" if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would delete ${PVE_ISO_STORAGE}:iso/${seed_file}" + info "[dry-run] Would delete ${PVE_ISO_STORAGE}:iso/${seed_file}" else case "$PVE_METHOD" in ssh) @@ -3228,18 +3582,46 @@ cleanup_storage() { pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${PVE_ISO_STORAGE}:iso/${seed_file}" &>/dev/null || true ;; esac - detail "Deleted seed: ${seed_file}" + info "Deleted seed: ${seed_file}" fi i=$((i + 1)) done - # Remove IncusOS ISOs from Proxmox storage + # Remove only the specific IncusOS ISO used by this deployment + if [[ -n "$DEPLOYMENT_ISO" ]]; then + local volid="${PVE_ISO_STORAGE}:iso/${DEPLOYMENT_ISO}" + if [[ "$DRY_RUN" == true ]]; then + info "[dry-run] Would delete ${volid}" + else + case "$PVE_METHOD" in + ssh) + pve_run "pvesm free ${volid} 2>/dev/null" &>/dev/null || true + ;; + api) + local encoded_volid + encoded_volid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${volid}', safe=''))" 2>/dev/null) + pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true + ;; + esac + info "Deleted ISO: ${DEPLOYMENT_ISO}" + fi + else + detail "No deployment ISO detected (VMs may already be destroyed)" + fi + + success "Storage cleanup complete" +} + +# Aggressive cleanup: remove ALL IncusOS ISOs and seed ISOs from storage +cleanup_storage_all() { + step "Deep cleanup: removing ALL IncusOS ISOs and seeds from Proxmox storage" + local iso_list="" if [[ "$DRY_RUN" != true ]]; then case "$PVE_METHOD" in ssh) - iso_list=$(pve_run "pvesm list ${PVE_ISO_STORAGE} --content iso 2>/dev/null" 2>/dev/null | grep "IncusOS_" | awk '{print $1}') || iso_list="" + iso_list=$(pve_run "pvesm list ${PVE_ISO_STORAGE} --content iso 2>/dev/null" 2>/dev/null | grep -E "(IncusOS_|seed-)" | awk '{print $1}') || iso_list="" ;; api) iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null | \ @@ -3248,7 +3630,7 @@ import json, sys data = json.load(sys.stdin) for item in data.get('data', []): volid = item.get('volid', '') - if 'IncusOS_' in volid: + if 'IncusOS_' in volid or 'seed-' in volid: print(volid) " 2>/dev/null) || iso_list="" ;; @@ -3259,7 +3641,7 @@ for item in data.get('data', []): while IFS= read -r volid; do [[ -z "$volid" ]] && continue if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would delete ${volid}" + info "[dry-run] Would delete ${volid}" else case "$PVE_METHOD" in ssh) @@ -3271,14 +3653,18 @@ for item in data.get('data', []): pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true ;; esac - detail "Deleted ISO: ${volid}" + info "Deleted: ${volid}" fi done <<< "$iso_list" + else + info "No IncusOS ISOs or seeds found in storage" fi success "Storage cleanup complete" } +# Remove incus remotes. Uses VM_NAMES (config-based cleanup) or +# DESTROYED_VM_NAMES (pool-wide cleanup) depending on what's populated. cleanup_remotes() { step "Deep cleanup: removing incus remotes" @@ -3287,27 +3673,40 @@ cleanup_remotes() { return fi - local count=${#VM_NAMES[@]} - local i=0 - - while [[ $i -lt $count ]]; do - local name="${VM_NAMES[$i]}" - if [[ -n "$VM_FILTER" ]] && [[ "$name" != "$VM_FILTER" ]]; then + # Determine which name list to use + local names_to_check=() + if [[ ${#DESTROYED_VM_NAMES[@]} -gt 0 ]]; then + # Pool-wide cleanup: use names of VMs actually destroyed + names_to_check=("${DESTROYED_VM_NAMES[@]}") + elif [[ ${#VM_NAMES[@]} -gt 0 ]]; then + # Config-based cleanup + local i=0 + while [[ $i -lt ${#VM_NAMES[@]} ]]; do + if [[ -n "$VM_FILTER" ]] && [[ "${VM_NAMES[$i]}" != "$VM_FILTER" ]]; then + i=$((i + 1)) + continue + fi + names_to_check+=("${VM_NAMES[$i]}") i=$((i + 1)) - continue - fi + done + fi + if [[ ${#names_to_check[@]} -eq 0 ]]; then + detail "No VM names to check for remote cleanup" + success "Remote cleanup complete" + return + fi + + for name in "${names_to_check[@]}"; do # Check if a remote with this name exists if incus remote list --format csv 2>/dev/null | grep -q "^${name},"; then if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would remove incus remote '${name}'" + info "[dry-run] Would remove incus remote '${name}'" else incus remote remove "$name" 2>/dev/null || true - detail "Removed incus remote: ${name}" + info "Removed incus remote: ${name}" fi fi - - i=$((i + 1)) done success "Remote cleanup complete" @@ -3319,10 +3718,10 @@ cleanup_local_cache() { local cache_base="${XDG_CACHE_HOME:-${HOME}/.cache}/${SCRIPT_NAME}/iso-cache" if [[ -d "$cache_base" ]]; then if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would clear ${cache_base}" + info "[dry-run] Would clear ${cache_base}" else rm -rf "$cache_base" - detail "Cleared cache: ${cache_base}" + info "Cleared cache: ${cache_base}" fi else detail "No local cache to clear" @@ -3331,13 +3730,17 @@ cleanup_local_cache() { success "Local cache cleanup complete" } +# Names of VMs destroyed during cleanup-all (used by cleanup_remotes) +declare -a DESTROYED_VM_NAMES=() + # Pool-wide cleanup: destroy ALL managed VMs in the resource pool do_cleanup_all() { step "Pool-wide cleanup: destroying all managed VMs" echo "" if [[ -z "$PVE_POOL" ]]; then - error "Pool-wide cleanup requires a resource pool (proxmox.pool) in the config" + error "Pool-wide cleanup requires a resource pool (proxmox.pool)" + error "Set pool in proxmox.yaml or pass a config with proxmox.pool" error "Without a pool, there is no safe way to scope which VMs to destroy" exit 1 fi @@ -3389,8 +3792,9 @@ do_cleanup_all() { step "Destroying VM '${name}' (VMID ${vmid})" if [[ "$DRY_RUN" == true ]]; then - detail "[dry-run] Would stop and destroy VM ${vmid}" + info "[dry-run] Would stop and destroy VM ${vmid}" destroyed=$((destroyed + 1)) + DESTROYED_VM_NAMES+=("$name") idx=$((idx + 1)) continue fi @@ -3398,7 +3802,7 @@ do_cleanup_all() { local status status=$(pve_vm_status "$vmid") || status="unknown" if [[ "$status" == "running" ]]; then - detail "Stopping VM..." + info "Stopping VM..." pve_stop_vm "$vmid" || true sleep 5 fi @@ -3406,6 +3810,7 @@ do_cleanup_all() { if pve_destroy_vm "$vmid"; then success "VM '${name}' (VMID ${vmid}) destroyed" destroyed=$((destroyed + 1)) + DESTROYED_VM_NAMES+=("$name") else error "Failed to destroy VM '${name}' (VMID ${vmid})" fi @@ -3546,6 +3951,40 @@ main() { setup_workdir + # Handle cleanup-all without a config file (only needs proxmox.yaml) + if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then + if ! find_proxmox_config; then + error "No proxmox.yaml found and no config file specified" + error "Create proxmox.yaml or use --proxmox FILE" + exit 1 + fi + load_proxmox_config "$PROXMOX_CONFIG_RESOLVED" + + # Apply defaults for anything still unset + [[ -z "$PVE_NODE" ]] && PVE_NODE="pve" + [[ -z "$PVE_STORAGE" ]] && PVE_STORAGE="local-lvm" + [[ -z "$PVE_ISO_STORAGE" ]] && PVE_ISO_STORAGE="local" + [[ -z "$PVE_BRIDGE" ]] && PVE_BRIDGE="vmbr0" + [[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root" + [[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh" + [[ -n "$METHOD" ]] && PVE_METHOD="$METHOD" + [[ -n "$HOST_OVERRIDE" ]] && PVE_HOST="$HOST_OVERRIDE" + + if [[ -z "$PVE_HOST" ]]; then + error "proxmox.yaml must contain 'host:'" + exit 1 + fi + + do_cleanup_all + if [[ "$DEEP_CLEANUP" == true ]]; then + echo "" + cleanup_storage_all + cleanup_remotes + cleanup_local_cache + fi + return + fi + if [[ "$DRY_RUN" != true ]]; then check_dependencies fi @@ -3554,12 +3993,12 @@ main() { # Always run validate first phase_validate - # Handle cleanup-all mode (pool-wide, not config-specific VMs) + # Handle cleanup-all mode (with config file for additional context) if [[ "$CLEANUP_ALL" == true ]]; then do_cleanup_all if [[ "$DEEP_CLEANUP" == true ]]; then echo "" - cleanup_storage + cleanup_storage_all cleanup_remotes cleanup_local_cache fi @@ -3568,6 +4007,10 @@ main() { # Handle cleanup mode if [[ "$CLEANUP" == true ]]; then + if [[ "$DEEP_CLEANUP" == true ]]; then + # Detect which ISO was used before destroying VMs + detect_deployment_iso + fi confirm "Destroy VMs listed above?" do_cleanup if [[ "$DEEP_CLEANUP" == true ]]; then diff --git a/notes/clustering-guide.md b/notes/clustering-guide.md new file mode 100644 index 0000000..9f8e666 --- /dev/null +++ b/notes/clustering-guide.md @@ -0,0 +1,1020 @@ +# Incus Clustering Guide for IncusOS (via Remotes) + +This guide covers forming an Incus cluster from IncusOS nodes using only the +`incus` CLI and remotes. No SSH access to the nodes is needed (IncusOS is +immutable and has no shell). + +Tested with Incus client 6.21 and IncusOS nodes deployed via `incusos-proxmox`. + +--- + +## Overview + +Incus clustering does **not** require a virtual IP (VIP), load balancer, or +shared storage. Each node advertises its own IP address. Clients can connect +to any cluster member -- requests are forwarded internally via the cluster's +internal communication channel. + +**Cluster roles:** +- **database-leader**: holds the primary Dqlite database +- **database**: database replica (for HA, need 3 database members) +- **database-standby**: standby replica (promoted automatically if a database + member goes down) + +With 3 nodes, you get a fully HA database quorum. + +--- + +## Prerequisites + +- Incus client version **6.20+** (required for `incus cluster join` via remotes) +- All nodes deployed with `apply_defaults: true` (matching pool/network names) +- Remotes configured for all nodes (`incus remote add https://:8443`) +- Client certificate trusted on all nodes (injected via seed or added manually) + +```bash +incus version # Verify 6.20+ +incus info incus-lab-01: | head -5 # Verify remote connectivity +incus info incus-lab-02: | head -5 +incus info incus-lab-03: | head -5 +``` + +--- + +## Step 1: Set Specific IP Addresses + +### Why this is needed + +IncusOS nodes listen on `:8443` by default (all interfaces). Clustering +requires each node to have a specific routable IP so other members can +address it. Without this, `incus cluster enable` either fails or prompts +for an address. + +### How to discover node IPs + +```bash +for node in incus-lab-01 incus-lab-02 incus-lab-03; do + echo "=== $node ===" + incus query "$node:/1.0" | python3 -c " +import sys, json +d = json.load(sys.stdin) +for a in d['environment']['addresses']: + if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('['): + print(a) +" +done +``` + +This filters out: +- `10.x.x.x` -- internal bridge addresses (`incusbr0`) +- `fd42:` -- ULA IPv6 bridge addresses +- `[...]` -- IPv6 addresses in bracket notation + +### Set the addresses + +```bash +incus config set incus-lab-01: core.https_address 192.168.1.250:8443 +incus config set incus-lab-02: core.https_address 192.168.1.159:8443 +incus config set incus-lab-03: core.https_address 192.168.1.78:8443 +``` + +### Verify connectivity after each change + +```bash +incus config get incus-lab-01: core.https_address # Should show IP:8443 +incus info incus-lab-01: | head -3 # Should still connect +``` + +### What happens under the hood + +The Incus daemon rebinds its HTTPS listener from `0.0.0.0:8443` to the +specific IP. Since the client remote was already connecting to that IP, the +connection continues to work seamlessly. Certificate trust is stored by +fingerprint in the Incus database, independent of the listen address. + +--- + +## Step 2: Enable Clustering on the Init Node + +```bash +incus cluster enable incus-lab-01: incus-lab-01 +``` + +**Syntax note**: this is TWO arguments -- `remote:` (trailing colon, targeting +the server) and `member-name` (the cluster member name). The help text shows +`[:] `. Using `remote:name` as a single arg causes "The incus +daemon doesn't appear to be started" on client-only systems. + +### The TLS certificate problem + +Enabling clustering causes the server to generate a **new TLS certificate** +(the cluster certificate). This new cert typically only has SANs (Subject +Alternative Names) for `127.0.0.1` and `::1`, not the node's actual IP. + +Your existing remote has the old (pre-clustering) certificate pinned, so +the next connection fails: + +``` +Error: tls: failed to verify certificate: x509: certificate is valid for +127.0.0.1, ::1, not 192.168.1.250 +``` + +### The fix: re-add the remote + +```bash +# Switch default remote first (can't remove the current default) +incus remote switch local + +# Remove and re-add with the new certificate +incus remote remove incus-lab-01 +incus remote add incus-lab-01 https://192.168.1.250:8443 --accept-certificate +``` + +Incus remotes use **fingerprint pinning** (not CA-based SAN validation), so +`--accept-certificate` pins the new cluster cert and future connections work +despite the SAN mismatch. + +### Verify + +```bash +incus cluster list incus-lab-01: +``` + +Expected output: one member (`incus-lab-01`), status `ONLINE`, role +`database-leader` + `database`. + +--- + +## Step 3: Prepare Joining Nodes + +### The apply_defaults storage pool conflict + +Nodes deployed with `apply_defaults: true` already have: +- A `local` ZFS storage pool (source: `local/incus`) +- An `incusbr0` network bridge +- A `default` profile referencing both + +When joining a cluster that already has a `local` pool defined, the join +wizard asks for `source` and `zfs.pool_name` properties. However, it +attempts to set these as **cluster-wide** config, which Incus rejects because +they are **member-specific** keys: + +``` +Error: Failed to join cluster: Failed to initialize member: Failed to +initialize storage pools and networks: Failed to update storage pool +"local": Config key "zfs.pool_name" is cluster member specific +``` + +### The fix: delete pool and network before joining + +This must be done for **each joining node** (not the init node). + +```bash +NODE=incus-lab-02 # repeat with incus-lab-03 + +# 1. Remove server config references to the pool +incus config unset "$NODE": storage.backups_volume +incus config unset "$NODE": storage.images_volume + +# 2. Delete the backup and image custom volumes +incus storage volume delete "$NODE":local backups +incus storage volume delete "$NODE":local images + +# 3. Remove default profile device references (pool shows "in use" otherwise) +incus profile device remove "$NODE":default root +incus profile device remove "$NODE":default eth0 + +# 4. Now delete the pool and network +incus storage delete "$NODE":local +incus network delete "$NODE":incusbr0 +``` + +The underlying ZFS dataset (`local/incus`) still exists on disk -- only the +Incus metadata is removed. The join process will re-register it. + +--- + +## Step 4: Join Nodes + +### Interactive method + +```bash +# Generate a join token (on the cluster, for the new member name) +incus cluster add incus-lab-01:incus-lab-02 + +# Join (prompts for 5 values) +incus cluster join incus-lab-01: incus-lab-02: +``` + +The interactive prompts and correct answers: + +| # | Prompt | Answer | +|---|--------|--------| +| 1 | IP address or DNS name [default=192.168.1.159] | Press Enter (accept default) | +| 2 | Member name [default=incus-lab-02] | Press Enter (accept default) | +| 3 | All existing data is lost, continue? [default=no] | `yes` | +| 4 | Choose "source" property for storage pool "local" | `local/incus` | +| 5 | Choose "zfs.pool_name" property for storage pool "local" | `local/incus` | + +### Automated (non-interactive) method + +```bash +# Generate token +incus cluster add incus-lab-01:incus-lab-02 + +# Join with piped answers (two empty lines = accept defaults for IP + name) +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02: +``` + +### Fix the remote after join + +Same certificate issue as the init node -- the joining node now uses the +cluster certificate: + +```bash +incus remote remove incus-lab-02 +incus remote add incus-lab-02 https://192.168.1.159:8443 --accept-certificate +``` + +### Verify after each join + +```bash +incus cluster list incus-lab-01: +incus info incus-lab-02: | head -5 +``` + +### Repeat for remaining nodes + +```bash +# Prepare node 3 (same cleanup as step 3) +# ... + +# Generate token + join +incus cluster add incus-lab-01:incus-lab-03 +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-03: + +# Fix remote +incus remote remove incus-lab-03 +incus remote add incus-lab-03 https://192.168.1.78:8443 --accept-certificate +``` + +--- + +## Step 5: Verify the Cluster + +```bash +# All 3 members should be ONLINE +incus cluster list incus-lab-01: + +# Check storage pool exists on all members +incus storage show incus-lab-01:local +incus storage show incus-lab-01:local --target incus-lab-01 +incus storage show incus-lab-01:local --target incus-lab-02 +incus storage show incus-lab-01:local --target incus-lab-03 + +# Network should be on all members +incus network list incus-lab-01: + +# All individual remotes should work +incus info incus-lab-01: | head -3 +incus info incus-lab-02: | head -3 +incus info incus-lab-03: | head -3 +``` + +--- + +## Quick Reference: Full Automated Script + +For copy-paste clustering of a 3-node lab (replace IPs with your values): + +```bash +IP1=192.168.1.250 +IP2=192.168.1.159 +IP3=192.168.1.78 + +# --- Set specific IPs --- +incus config set incus-lab-01: core.https_address "$IP1":8443 +incus config set incus-lab-02: core.https_address "$IP2":8443 +incus config set incus-lab-03: core.https_address "$IP3":8443 + +# --- Enable clustering on init node --- +incus cluster enable incus-lab-01: incus-lab-01 + +# Fix remote (switch default first if needed) +incus remote switch local +incus remote remove incus-lab-01 +incus remote add incus-lab-01 https://"$IP1":8443 --accept-certificate + +# --- Prepare and join node 2 --- +incus config unset incus-lab-02: storage.backups_volume +incus config unset incus-lab-02: storage.images_volume +incus storage volume delete incus-lab-02:local backups +incus storage volume delete incus-lab-02:local images +incus profile device remove incus-lab-02:default root +incus profile device remove incus-lab-02:default eth0 +incus storage delete incus-lab-02:local +incus network delete incus-lab-02:incusbr0 + +incus cluster add incus-lab-01:incus-lab-02 +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02: + +incus remote remove incus-lab-02 +incus remote add incus-lab-02 https://"$IP2":8443 --accept-certificate + +# --- Prepare and join node 3 --- +incus config unset incus-lab-03: storage.backups_volume +incus config unset incus-lab-03: storage.images_volume +incus storage volume delete incus-lab-03:local backups +incus storage volume delete incus-lab-03:local images +incus profile device remove incus-lab-03:default root +incus profile device remove incus-lab-03:default eth0 +incus storage delete incus-lab-03:local +incus network delete incus-lab-03:incusbr0 + +incus cluster add incus-lab-01:incus-lab-03 +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-03: + +incus remote remove incus-lab-03 +incus remote add incus-lab-03 https://"$IP3":8443 --accept-certificate + +# --- Verify --- +incus cluster list incus-lab-01: +``` + +--- + +## Command Syntax Reference + +| Command | Arguments | Notes | +|---------|-----------|-------| +| `incus cluster enable` | `remote: member-name` | TWO args: remote: + name | +| `incus cluster add` | `remote:member-name` | ONE arg, no space after colon | +| `incus cluster join` | `init-remote: joining-remote:` | TWO args, space between | +| `incus cluster list` | `remote:` | Trailing colon = target server | +| `incus config set` | `remote: key value` | Trailing colon + space + key | +| `incus storage show` | `remote:pool` | ONE arg, no space after colon | +| `incus storage show` | `remote:pool --target member` | Member-specific config | +| `incus profile device remove` | `remote:profile device` | ONE arg + device name | +| `incus cluster remove` | `remote:member-name --force` | ONE arg; prompts "yes/no" even with --force | +| `incus cluster evacuate` | `remote:member-name` | ONE arg, no space after colon | +| `incus cluster restore` | `remote:member-name` | ONE arg, no space after colon | + +**General rule**: `remote:resource` targets a resource on the remote. +`remote:` (trailing colon, no resource) targets the server itself. + +--- + +## Troubleshooting + +| Symptom | Cause | Fix | +|---------|-------|-----| +| `certificate is valid for 127.0.0.1, ::1, not ` | Cluster cert regenerated | Remove and re-add remote with `--accept-certificate` | +| `Config key "zfs.pool_name" is cluster member specific` | Pool already exists on joining node | Delete pool/network before joining (step 3) | +| `Config key "source" is cluster member specific` | Same as above | Delete pool/network before joining (step 3) | +| `The storage pool is currently in use` | Default profile still references pool | Remove `root` and `eth0` from default profile first | +| `Invalid number of arguments` on `cluster add` | Space between remote and member name | Use `remote:member` not `remote: member` | +| Join hangs / EOF errors | Interactive prompts in non-interactive shell | Use `printf` pipe method | +| Token expired | Tokens are time-limited | Generate a fresh token immediately before joining | +| Node shows in cluster with wrong URL | `core.https_address` was wildcard | Set specific IP before enabling clustering | + +--- + +## What Happens Under the Hood + +### Certificate lifecycle + +1. **Pre-clustering**: each node has its own self-signed server certificate, + generated at install time. Client remotes pin this cert's fingerprint. +2. **Cluster enable**: the init node generates a new **cluster certificate** + (shared across all members). The old server cert is replaced. Client + remotes must re-pin. +3. **Node join**: the joining node receives the cluster certificate from the + init node. Its old server cert is replaced. Client remotes must re-pin. + +### Storage pool lifecycle during join + +1. **Pre-join**: each node has its own standalone `local` ZFS pool with + source `local/incus` (a ZFS dataset on the IncusOS system disk). +2. **Delete before join**: we remove the Incus metadata (pool definition, + volumes, profile references). The underlying ZFS dataset remains on disk. +3. **During join**: the cluster tells the new member "you need a `local` pool". + The join wizard asks for the source (`local/incus`) and creates a + **member-specific** pool entry pointing to the existing ZFS dataset. +4. **Post-join**: the pool shows as `Created` on all members with matching + names but potentially different underlying sources (member-specific config). + +### Why we delete the network too + +The `incusbr0` network bridge follows the same pattern as the storage pool. +If it already exists on the joining node, the join process may conflict. +Deleting it lets the cluster recreate it with the correct member-specific +configuration. + +--- + +## Cluster Workload Testing + +After forming the cluster, validate it by launching workloads, verifying +cluster-wide visibility, and testing migration. + +### Launch a container on a specific node + +```bash +# Launch Debian 12 container on node 2 +incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02 + +# Verify it's running and check placement +incus list incus-lab-01: --columns ntsLl4 + +# Exec into it +incus exec incus-lab-01:test-container -- cat /etc/os-release | head -2 +``` + +### Verify cluster-wide visibility + +All cluster members should see the same instance list, regardless of which +remote you query: + +```bash +incus list incus-lab-01: --columns nLs +incus list incus-lab-02: --columns nLs +incus list incus-lab-03: --columns nLs +``` + +### Container migration (stop/move/start) + +Containers do **not** support live migration (CRIU is unreliable). Use +stop/move/start: + +```bash +# Check current location +incus list incus-lab-01: --columns nLs + +# Migrate: stop, move to new node, start +incus stop incus-lab-01:test-container +incus move incus-lab-01:test-container --target incus-lab-01 +incus start incus-lab-01:test-container + +# Verify new location +incus list incus-lab-01: --columns nLs +``` + +**What survives migration:** +- All persistent filesystem data (root disk contents) +- Instance configuration + +**What does NOT survive:** +- Running processes (stopped with the container) +- tmpfs contents (`/tmp`, `/run`) +- In-memory state + +### Launch a VM on a specific node + +```bash +# Launch Debian 12 VM on node 2 +incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02 + +# VMs take ~10-15s to boot — wait for agent +incus exec incus-lab-01:test-vm -- hostname # retry until it works + +# Verify +incus list incus-lab-01: --columns ntsLl4 +incus exec incus-lab-01:test-vm -- cat /etc/os-release | head -2 +``` + +### VM migration + +#### Live migration + +Live migration preserves running state (no downtime). It works across +heterogeneous hosts (different core counts) as long as `limits.cpu` is +set correctly — see "VM Live Migration: The `limits.cpu` Fix" below. + +```bash +# 1. Enable stateful migration (must be set while VM is stopped) +incus stop incus-lab-01:test-vm +incus config set incus-lab-01:test-vm migration.stateful=true + +# 2. Set limits.cpu as a RANGE (critical for migration!) +incus config set incus-lab-01:test-vm limits.cpu=0-1 + +# 3. Add size.state to root disk (required for stateful operations) +incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB + +# 4. Start the VM +incus start incus-lab-01:test-vm + +# 5. Live-migrate (no stop needed -- running state is preserved!) +incus move incus-lab-01:test-vm --target incus-lab-03 +``` + +**What survives live migration** (unlike stop/move/start): +- All persistent filesystem data +- Running processes (PID state preserved) +- tmpfs contents (`/tmp`, `/run`) +- Network connections +- In-memory state (full RAM checkpoint) + +#### Stop/move/start migration (always works) + +```bash +incus stop incus-lab-01:test-vm +incus move incus-lab-01:test-vm --target incus-lab-01 +incus start incus-lab-01:test-vm +``` + +Same persistence rules as containers: filesystem data survives, running +processes and tmpfs do not. + +### Cluster evacuation and restore + +Evacuation drains all workloads from a node (e.g., for maintenance). Incus +automatically selects the right strategy per workload type: + +```bash +# Evacuate a node (--force skips confirmation prompt) +incus cluster evacuate incus-lab-01:incus-lab-02 --force + +# Check: node shows EVACUATED, instances moved to other nodes +incus cluster list incus-lab-01: +incus list incus-lab-01: --columns nLs + +# Restore: moves workloads back to original node +incus cluster restore incus-lab-01:incus-lab-02 --force + +# Verify: node back ONLINE, instances restored +incus cluster list incus-lab-01: +incus list incus-lab-01: --columns nLs +``` + +**Syntax note**: `incus cluster evacuate remote:member` is ONE argument (like +`cluster enable` and `cluster add`). + +**What happens during evacuation:** +- VMs with `migration.stateful=true`: live migration (state preserved) +- VMs without stateful: stop/move/start +- Containers: stop/move/start +- Use `--action stop` to force stop/move for all workloads (useful on + heterogeneous clusters without `limits.cpu` range fix) + +### Cleanup test instances + +```bash +incus delete incus-lab-01:test-container --force +incus delete incus-lab-01:test-vm --force +incus list incus-lab-01: # should be empty +``` + +--- + +## VM Live Migration: The `limits.cpu` Fix + +### The problem + +By default, live migration fails between cluster nodes with different host +CPU counts: + +``` +qemu-system-x86_64: Missing section footer for 0000:00:1f.0/ICH9LPC +qemu-system-x86_64: load of migration failed: Invalid argument +``` + +Or, with `limits.cpu` set as an integer: + +``` +qemu-system-x86_64: Unknown section or instance 'apic' 3 +``` + +### Root cause: Incus sets QEMU `maxcpus` from host CPU count + +In the Incus source (`driver_qemu_templates.go`), the `qemuCPU` function +generates the QEMU `-smp` configuration. When `limits.cpu` is unset or set +to a plain integer, it does: + +```go +maxCpus := 64 +if int(cpu.Total) < maxCpus { + maxCpus = int(cpu.Total) // <-- host CPU count +} +``` + +This means: +- On a 4-core host: QEMU gets `maxcpus=4` → `cpu possible: 0-3` +- On a 2-core host: QEMU gets `maxcpus=2` → `cpu possible: 0-1` + +The ICH9 ACPI CPU hotplug state (`hw/acpi/cpu.c`) is a variable-length +array sized by `maxcpus`. When source and destination have different +`maxcpus`, the vmstate sizes don't match and QEMU can't deserialize the +migration stream. + +### How we discovered this + +We observed the VM's `cpu possible` differing by host: + +```bash +# On 4-core host: +cat /sys/devices/system/cpu/possible → 0-3 + +# On 2-core host: +cat /sys/devices/system/cpu/possible → 0-1 +``` + +Even though the VM only uses 1 vCPU, QEMU allocates hotplug slots for all +"possible" CPUs. + +### The fix: use `limits.cpu` as a range + +When `limits.cpu` is set to a **range** (e.g., `0-1`), Incus takes the +**CPU pinning** code path, which generates a fixed topology with no +`maxcpus`: + +```ini +# Integer (broken): maxcpus varies by host +[smp-opts] +cpus = 2 +maxcpus = 4 # on 4-core host + +# Range (fixed): no maxcpus, deterministic topology +[smp-opts] +cpus = 2 +sockets = 1 +cores = 2 +threads = 1 +``` + +**The one-line fix:** + +```bash +# Set limits.cpu as a range (while VM is stopped) +incus config set limits.cpu=0-1 # for 2 vCPUs +incus config set limits.cpu=0-3 # for 4 vCPUs +``` + +### Configuration summary + +| `limits.cpu` value | QEMU `-smp` behavior | Migration across heterogeneous hosts | +|---|---|---| +| (not set) | `maxcpus = host_cores` | **Fails** | +| `2` (integer) | `maxcpus = host_cores` | **Fails** | +| `0-1` (range) | Fixed `sockets/cores/threads`, no `maxcpus` | **Works** | + +### Alternative: `raw.qemu.conf` override + +If you need `maxcpus` (e.g., for CPU hotplug), force it to a fixed value: + +```bash +incus config set raw.qemu.conf='[smp-opts] +maxcpus = 2' +``` + +Do **not** combine this with the range syntax — they are mutually exclusive. + +### Complete setup for a live-migratable VM + +```bash +# Create and configure (all while stopped) +incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02 +incus stop incus-lab-01:test-vm + +incus config set incus-lab-01:test-vm migration.stateful=true +incus config set incus-lab-01:test-vm limits.cpu=0-1 +incus config device add incus-lab-01:test-vm root disk \ + path=/ pool=local size.state=2GiB + +incus start incus-lab-01:test-vm + +# Now live migration works across any cluster node +incus move incus-lab-01:test-vm --target incus-lab-03 +``` + +--- + +## Migration Test Results + +Tested with a 3-node cluster on Proxmox VE (Intel i9-13900HK, nested VT-x): +- **incus-lab-01**: 4 vCPUs, 8192 MB +- **incus-lab-02**: 2 vCPUs, 4096 MB +- **incus-lab-03**: 2 vCPUs, 4096 MB + +Test VM: Debian 12, `limits.cpu=0-1`, `migration.stateful=true`, `size.state=2GiB`. +Heartbeat process (1/s counter to tmpfs) verifies full state preservation. + +| # | Test | Time | Result | +|---|------|------|--------| +| 1 | Live: node1→node2 (4-core host → 2-core host) | 7.4s | PASS | +| 2 | Live: node2→node3 (2-core host → 2-core host) | 7.2s | PASS | +| 3 | Live: node3→node1 (2-core host → 4-core host) | 7.5s | PASS | +| 4 | Rapid round-trip: 1→2→3→1 (back-to-back) | ~22s | PASS | +| 5 | Stateful stop/restore (same node) | 7.6s | PASS | +| 6 | Stateful stop/move/restore (node1→node2, cross-topology) | 10.0s | PASS | +| 7a | Cluster evacuate node2 (container + VM) | 9.0s | PASS | +| 7b | Cluster restore node2 (moves workloads back) | 9.7s | PASS | + +**Key observations:** +- Transfer speed: ~140 MB/s for ~1 GB VM +- Heartbeat counter continuous through all migrations (no lost seconds) +- tmpfs, systemd services, and VM uptime all preserved through live migration +- VM agent (`incus exec`) needs ~3-4 seconds to reconnect after migration +- Cluster evacuation auto-selects strategy: live migration for VMs, stop/move/start for containers +- Cluster restore returns workloads to original node + +### Automated migration test + +Run this from your workstation to test all migration paths: + +```bash +REMOTE=incus-lab-01 # any cluster member +VM=test-vm + +# Setup: create heartbeat inside VM +incus exec "$REMOTE:$VM" -- mkdir -p /tmp/migration-test +incus exec "$REMOTE:$VM" -- bash -c \ + 'echo "created-$(hostname)-$(date +%s)" > /tmp/migration-test/marker' +incus exec "$REMOTE:$VM" -- bash -c \ + 'cat > /tmp/migration-test/heartbeat.sh << "EOF" +#!/bin/bash +i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done +EOF +chmod +x /tmp/migration-test/heartbeat.sh' +incus exec "$REMOTE:$VM" -- \ + systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh + +sleep 3 + +# Test: live-migrate through all nodes +PASS=0; FAIL=0 +for TARGET in incus-lab-02 incus-lab-03 incus-lab-01; do + PRE=$(incus exec "$REMOTE:$VM" -- cat /tmp/migration-test/heartbeat) + LOC=$(incus list "$REMOTE:" --format csv -c nL | grep "$VM" | cut -d, -f2) + echo -n "[$LOC → $TARGET] heartbeat=$PRE ... " + + incus move "$REMOTE:$VM" --target "$TARGET" 2>&1 | tail -1 + sleep 2 # wait for agent reconnect + + POST=$(incus exec "$REMOTE:$VM" -- cat /tmp/migration-test/heartbeat 2>&1) + SVC=$(incus exec "$REMOTE:$VM" -- systemctl is-active migration-heartbeat 2>&1) + if [[ "$SVC" == "active" && "$POST" -gt "$PRE" ]]; then + echo "PASS (heartbeat=$POST, service=$SVC)" + PASS=$((PASS + 1)) + else + echo "FAIL (heartbeat=$POST, service=$SVC)" + FAIL=$((FAIL + 1)) + fi +done + +# Cleanup heartbeat +incus exec "$REMOTE:$VM" -- systemctl stop migration-heartbeat 2>/dev/null + +echo "" +echo "Results: $PASS passed, $FAIL failed" +``` + +--- + +### The vnmi warning is cosmetic + +The `CPUID[eax=8000000Ah].EDX.vnmi` warning appears because QEMU's CPU +feature dependency checker fires before KVM filters out unsupported +features. It does not cause migration failures and can be safely ignored. + +### Nested virtualization works + +Live migration works fine in nested virtualization (IncusOS inside Proxmox +on Intel VT-x). Tested with QEMU 10.2.1 on Intel i9-13900HK with +heterogeneous host core counts (4 vs 2). The `limits.cpu` range fix makes +host topology irrelevant. + +### Debugging tips + +- **`incus start --stateless`** discards a saved state file if a restore + fails, allowing a clean boot. +- **Check `cpu possible`** inside the VM: `cat /sys/devices/system/cpu/possible`. + If this differs across nodes for the same VM config, migration will fail. +- **`incus monitor --type=logging`** on the destination shows QEMU errors. + +### References + +- [Incus source: driver_qemu_templates.go](https://github.com/lxc/incus/blob/main/internal/server/instance/drivers/driver_qemu_templates.go) — `maxcpus` calculation +- [QEMU source: hw/acpi/cpu.c](https://github.com/qemu/qemu/blob/master/hw/acpi/cpu.c) — CPU hotplug state sizing +- [QEMU source: hw/isa/lpc_ich9.c](https://github.com/qemu/qemu/blob/master/hw/isa/lpc_ich9.c) — ICH9 vmstate subsections +- [Incus Instance Options](https://linuxcontainers.org/incus/docs/main/reference/instance_options/) — `limits.cpu`, `migration.stateful` +- [Incus issue #486](https://github.com/lxc/incus/issues/486) — cluster-wide CPU baseline + +--- + +## Multi-vCPU Migration Test Results + +Tested with a 3-node cluster (6/4/4 cores, 8 GiB RAM each) to validate +migration across different vCPU counts and heterogeneous host topologies. + +### Test environment + +- **incus-adv-01**: 6 vCPUs, 8192 MB +- **incus-adv-02**: 4 vCPUs, 8192 MB +- **incus-adv-03/04**: 4 vCPUs, 8192 MB + +Three test VMs, all Debian 12, `migration.stateful=true`, `size.state=4GiB`: + +| VM | `limits.cpu` | vCPUs | `cpu possible` | +|----|-------------|-------|----------------| +| test-vm-2cpu | `0-1` | 2 | `0-1` | +| test-vm-3cpu | `0-2` | 3 | `0-2` | +| test-vm-4cpu | `0-3` | 4 | `0-3` | + +### Results + +| # | Test | Time | Result | +|---|------|------|--------| +| 1 | 2-vCPU full ring (3 migrations) | ~7s each | PASS | +| 2 | 3-vCPU full ring (3 migrations) | ~7.5s each | PASS | +| 3 | 4-vCPU full ring (3 migrations) | ~7.3s each | PASS | +| 4 | Concurrent: 2 VMs from different nodes | 8s total | PASS | +| 5 | 4-vCPU VM to 4-core host (100% core usage) | 7.3s | PASS | +| 6 | Rapid round-trip: 3 hops back-to-back | ~31s total | PASS | +| 7 | Active disk I/O (dd 100 MiB) during migration | 8.2s | PASS | +| 8 | Active network (ping) during migration | 8.5s | PASS | +| 9 | Stateful stop/move/start (cross-node) | 10s | PASS | +| 10 | Memory-loaded VM (512 MiB allocated) | 7.7s | PASS | +| 11 | Move to same node (no-op) | instant | Clean error | +| 12 | Mixed evacuation (2 containers + 2 VMs) | 18s | PASS | +| 13 | Restore after evacuation | 18s | PASS | + +### Key findings + +- **Odd vCPU counts work**: `limits.cpu=0-2` (3 vCPUs) migrates just as + reliably as even counts. The range syntax always produces deterministic + QEMU topology regardless of count. +- **4-vCPU on 4-core host works**: a VM using all host cores migrates + successfully and remains responsive. No performance degradation observed. +- **Concurrent migrations**: migrating 2 VMs simultaneously from different + source nodes completes without interference. Transfer speed is maintained + at ~140 MB/s per migration. +- **Active I/O survives**: both disk writes (dd) and network activity (ping) + continue uninterrupted through live migration. File integrity verified. +- **Memory-loaded migration**: 512 MiB of allocated memory does not + significantly increase migration time (QEMU's iterative pre-copy handles + dirty pages efficiently). +- **Agent reconnect time is ~3-4 seconds**, not 1-2s as initially measured. + May vary with cluster size and vCPU count. Scripts should `sleep 4` after + migration before running `incus exec`. +- **Same-node move**: Incus returns a clear error ("Requested target server + is the same as current server") — no crash or corruption. +- **`size.state=4GiB`** recommended for VMs with 3-4 vCPUs to ensure + sufficient room for state dumps. + +--- + +## Cluster Lifecycle: Node Replacement + +Full procedure for evacuating, removing, destroying, and replacing a cluster +node. Tested on the 3-node advanced cluster. + +### Overview + +1. Evacuate the target node (drains workloads) +2. Remove the node from the cluster +3. Destroy the VM in Proxmox +4. Deploy a fresh replacement node +5. Join the replacement to the cluster +6. Verify workload rebalancing + +### Step 1: Evacuate + +```bash +REMOTE=incus-adv-01 # init node (not the target) +incus cluster evacuate $REMOTE:incus-adv-03 --force +``` + +This live-migrates VMs (with `migration.stateful=true`) and stop/move/starts +containers. Verify with `incus cluster list` — target shows `EVACUATED`. + +### Step 2: Remove from cluster + +```bash +printf "yes\n" | incus cluster remove $REMOTE:incus-adv-03 --force +``` + +**Note**: `--force` requires interactive confirmation ("yes"). Pipe it in for +automation. The `--force` flag is needed because the node is still technically +reachable (just evacuated). + +After removal, `incus cluster list` shows only the remaining members. + +### Step 3: Destroy the VM + +Use `incusos-proxmox --cleanup` with a config targeting just that VM, or +use the Proxmox API directly. Do NOT try raw `curl` with API tokens +containing `!` characters — bash history expansion will silently mangle the +token. Always use single quotes or the `incusos-proxmox` script. + +```bash +# Method 1: incusos-proxmox (recommended) +incusos-proxmox --cleanup --yes + +# Method 2: Proxmox API (note: single quotes around the token!) +TOKEN='automation@pve!deploy=' +curl -s -k -X POST -H "Authorization: PVEAPIToken=$TOKEN" \ + "https://:8006/api2/json/nodes//qemu//status/stop" +``` + +Also remove the incus remote: + +```bash +incus remote switch local # if target is current default +incus remote remove incus-adv-03 +``` + +### Step 4: Deploy replacement + +Create a config for the replacement node (e.g., `lab-replace.yaml`): + +```yaml +defaults: + memory: 8192 + disk: 50 + start_vmid: 913 + +vms: + - name: incus-adv-04 + app: incus + apply_defaults: true + cores: 4 +``` + +```bash +incusos-proxmox --yes lab-replace.yaml +``` + +### Step 5: Join to cluster + +Same procedure as initial cluster formation: + +```bash +NEW_IP= + +# Set specific address +incus config set incus-adv-04: core.https_address $NEW_IP:8443 + +# Prepare (delete pre-existing pool/network) +incus config unset incus-adv-04: storage.backups_volume +incus config unset incus-adv-04: storage.images_volume +incus storage volume delete incus-adv-04:local backups +incus storage volume delete incus-adv-04:local images +incus profile device remove incus-adv-04:default root +incus profile device remove incus-adv-04:default eth0 +incus storage delete incus-adv-04:local +incus network delete incus-adv-04:incusbr0 + +# Generate token and join +incus cluster add $REMOTE:incus-adv-04 +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join $REMOTE: incus-adv-04: + +# Fix remote (new cluster cert) +incus remote remove incus-adv-04 +incus remote add incus-adv-04 https://$NEW_IP:8443 --accept-certificate +``` + +### Step 6: Verify rebalancing + +With `cluster.rebalance.interval` configured, Incus automatically migrates +VMs to the new (empty) node: + +```bash +# Check after 1-2 minutes +incus list $REMOTE: --columns ntsL +``` + +In testing, 2 VMs were automatically live-migrated to the replacement node +within 90 seconds. Containers are NOT auto-rebalanced (Incus limitation) — +move them manually with stop/move/start. + +### Rebalancing configuration + +```bash +REMOTE=incus-adv-01 +incus config set $REMOTE: cluster.rebalance.interval=1 # check every N minutes +incus config set $REMOTE: cluster.rebalance.threshold=10 # imbalance % to trigger +incus config set $REMOTE: cluster.rebalance.batch=2 # max VMs per rebalance run +incus config set $REMOTE: cluster.rebalance.cooldown=5m # wait between runs +``` + +| Key | Description | Tested value | +|-----|-------------|-------------| +| `cluster.rebalance.interval` | Check interval in minutes | `1` (aggressive for testing) | +| `cluster.rebalance.threshold` | Imbalance percentage to trigger | `10` | +| `cluster.rebalance.batch` | Max VMs moved per run | `2` | +| `cluster.rebalance.cooldown` | Minimum time between runs | `5m` | + +**Important**: rebalancing only moves VMs with `migration.stateful=true`. +Containers and non-migratable VMs are not touched. + +### Command syntax: `cluster remove` + +```bash +# Works (pipe the confirmation) +printf "yes\n" | incus cluster remove remote:member --force + +# Also note: cluster enable syntax is TWO args, not ONE +incus cluster enable remote: member-name # correct +incus cluster enable remote:member-name # WRONG (single arg) +```