maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add advanced migration testing, cluster lifecycle, and clustering guide 

- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
  lab-replace.yaml (single replacement node) for advanced migration and
  cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
  formation, storage pool conflict fix, join workflow, VM live migration
  with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
  edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
  cluster enable syntax fix (TWO args not one), cluster remove syntax,
  agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-20 18:46:55 +01:00
parent 2aca6fb33b
commit 732d70b38e
12 changed files with 2157 additions and 181 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@ -25,3 +25,6 @@ Thumbs.db
# Go binaries (flasher-tool)
flasher-tool
# Proxmox connection config (contains credentials)
proxmox.yaml

226
CLAUDE.md
View File

@ -19,9 +19,11 @@ incu-contrib/
│ ├── incusos-seed # Seed archive generator
│ ├── incusos-proxmox # Declarative Proxmox VM deployment
│ ├── lab-test # Guided lab validation (workloads, clustering, migration)
│ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials)
│ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test
│ └── examples/ # Example seed + Proxmox YAML files
└── notes/ # Research notes and reference material
└── clustering-guide.md # Detailed Incus clustering walkthrough
```
## Key technical context
@ -78,6 +80,11 @@ incu-contrib/
- **`incusos-proxmox`** reads a YAML config, generates per-VM SEED_DATA images
via `incusos-seed --format iso`, uploads the ISO + seeds to Proxmox, creates
VMs with IncusOS-correct settings, and boots them through installation.
- **Config separation**: Proxmox connection settings (host, credentials, pool)
live in `incusos/proxmox.yaml` (gitignored). Lab configs (`lab-cluster.yaml`,
etc.) only define VM specs. Merge priority: `proxmox.yaml` base → lab config
`proxmox:` overlay → CLI flags (`--host`, `--method`). Auto-discovery looks
for `proxmox.yaml` in script directory then cwd; override with `--proxmox FILE`.
- **Connection methods**: SSH (default, `ssh root@host qm ...`) or API
(`curl -k https://host:8006/api2/json/...` with `PVEAPIToken` header).
- **Minimum API privileges** for token-based access:
@ -142,27 +149,198 @@ incu-contrib/
- **Cluster formation** is done entirely through the `incus` CLI using remotes.
No SSH to the IncusOS nodes is needed (IncusOS is immutable, no shell access).
- **Workflow:**
- **No VIP needed**: each node advertises its own IP as its cluster address.
Clients can connect to any cluster member; requests are forwarded internally.
#### Pre-clustering: fix core.https_address
- IncusOS nodes default to `core.https_address: :8443` (wildcard / all
interfaces). Clustering requires a **specific routable IP** so nodes can
address each other.
- **Set the IP on every node BEFORE enabling clustering:**
```bash
# Enable clustering on first node
incus cluster enable <remote1>: <member-name>
# Generate join token (on cluster, for new member name)
incus cluster add <remote1>: <new-member-name>
# Join second node to cluster
incus cluster join <remote1>: <remote2>:
# Verify
incus cluster list <remote1>:
incus config set <remote>: core.https_address <NODE_IP>:8443
```
- **`apply_defaults: true`** must be set on all nodes that will be clustered --
this ensures matching storage pool (`local`) and network bridge names, which
are required for migration.
- Get each node's routable IP via the API:
```bash
incus query <remote>:/1.0 | python3 -c "import sys,json; d=json.load(sys.stdin); \
[print(a) for a in d['environment']['addresses'] \
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
```
- This is safe to do while remotes are connected -- the remote already points
to the specific IP; we're just narrowing the bind address. Certificate trust
is fingerprint-based, not address-based.
#### Cluster enable (init node)
- ```bash
incus cluster enable <remote>: <member-name>
```
Note: this is TWO arguments: `<remote>:` (trailing colon) and `<member-name>`.
The help text shows `[<remote>:] <name>` — NOT `remote:name` as a single arg.
- **TLS certificate regeneration**: enabling clustering causes the server to
generate a new TLS certificate (cluster cert). The new cert may only have
SANs for `127.0.0.1` and `::1`, breaking the existing remote.
- **Fix**: remove and re-add the remote to pin the new certificate:
```bash
incus remote switch local # if init remote is current default
incus remote remove <remote>
incus remote add <remote> https://<NODE_IP>:8443 --accept-certificate
```
- The cert trust on the server side (client → server) is unaffected -- it's
stored by fingerprint in the Incus database, independent of listen address.
#### Joining nodes: the apply_defaults storage pool conflict
- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0`
network bridge on each node at install time. When joining a cluster that
already has a `local` pool defined, the join wizard asks for `source` and
`zfs.pool_name` properties but then tries to set them as cluster-wide config.
Incus rejects this because they are **member-specific** keys.
- **The fix**: delete the storage pool and network on the joining node before
joining. The join process then creates the member-specific pool entry
correctly.
```bash
# 1. Remove config references
incus config unset <remote>: storage.backups_volume
incus config unset <remote>: storage.images_volume
# 2. Delete volumes
incus storage volume delete <remote>:local backups
incus storage volume delete <remote>:local images
# 3. Clear default profile references (pool is "in use" otherwise)
incus profile device remove <remote>:default root
incus profile device remove <remote>:default eth0
# 4. Delete pool and network
incus storage delete <remote>:local
incus network delete <remote>:incusbr0
```
#### Join workflow
- **Generate token** (on init node, single argument `remote:member-name`):
```bash
incus cluster add <init-remote>:<new-member-name>
```
- **Join** (interactive -- prompts for 5 values):
```bash
incus cluster join <init-remote>: <joining-remote>:
```
Interactive prompts and correct answers:
1. IP address → accept default (node's IP, already set via core.https_address)
2. Member name → accept default (matches the token)
3. "All existing data is lost" → `yes`
4. `source` property for storage pool "local" → `local/incus`
5. `zfs.pool_name` property for storage pool "local" → `local/incus`
- **Automated (non-interactive):**
```bash
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join <init-remote>: <joining-remote>:
```
- **After join**: the joining node gets a new cluster certificate. Fix the
remote (same as init node):
```bash
incus remote remove <joining-remote>
incus remote add <joining-remote> https://<NODE_IP>:8443 --accept-certificate
```
#### Command syntax gotchas
- `incus cluster enable remote: member-name` -- TWO arguments (remote: + name)
- `incus cluster add remote:member-name` -- ONE argument (no space)
- `incus cluster remove remote:member-name --force` -- ONE argument; prompts
"yes/no" even with `--force`, pipe `printf "yes\n"` for automation
- `incus cluster evacuate remote:member-name` -- ONE argument (no space)
- `incus cluster restore remote:member-name` -- ONE argument (no space)
- `incus cluster join init-remote: joining-remote:` -- TWO arguments (space)
- `incus storage show remote:pool` -- ONE argument (no space)
- `incus storage show remote:pool --target member` -- target flag for
member-specific config
- `incus config set remote: key value` -- remote with trailing colon + space
- General rule: `remote:resource` for targeting a resource, `remote:` (trailing
colon) for targeting the server itself
#### Post-join state
- After joining, the cluster is managed through the init node's remote. The
individual node remotes still work for node-specific operations.
- **`lab-test`** automates cluster formation, workload testing, and migration.
#### Workload placement and migration
- **Targeted launch**: `incus launch images:debian/12 <cluster-remote>:name --target <member>`
- **Cluster-wide visibility**: `incus list` on any member shows all instances.
- **Container migration**: stop/move/start only (CRIU live migration is
unreliable). Data persists, processes do not.
```bash
incus stop <remote>:<instance>
incus move <remote>:<instance> --target <destination-member>
incus start <remote>:<instance>
```
- **VM live migration**: requires `migration.stateful=true` (must be set
while VM is stopped). Preserves running state with no downtime.
```bash
incus move <remote>:<instance> --target <destination-member>
```
- **VM live migration requires `limits.cpu` as a range** (e.g., `0-1`),
not an integer. Without this, Incus sets QEMU's `maxcpus` to the host's
CPU count (`driver_qemu_templates.go`: `maxcpus = min(cpu.Total, 64)`).
Different `maxcpus` values size the ICH9 ACPI CPU hotplug state arrays
differently, causing `Missing section footer for ICH9LPC` on restore.
Using a range (pinning syntax) eliminates `maxcpus` entirely and uses
fixed `sockets/cores/threads` topology — portable across all hosts.
```bash
# WRONG: integer → maxcpus varies by host → migration fails
incus config set <instance> limits.cpu=2
# RIGHT: range → fixed topology → migration works everywhere
incus config set <instance> limits.cpu=0-1
```
- **VM live migration works in nested virtualization** (IncusOS inside
Proxmox on Intel). It is NOT limited to bare metal. Tested with QEMU
10.2.1 on Intel i9-13900HK with heterogeneous host core counts (4 vs 2).
- **The `vnmi` CPUID warning** (`CPUID[eax=8000000Ah].EDX.vnmi`) that
appears during migration is cosmetic. It fires from QEMU's feature
dependency checker before KVM filters out unsupported features and does
not affect migration.
- **Stateful stop/restore** (`incus stop --stateful` + `incus start`) also
requires the `limits.cpu` range fix. Use `incus start --stateless` to
discard a saved state file that cannot be restored.
- **VM `size.state` config**: stateful operations require `size.state` on
the root disk (`incus config device add <instance> root disk path=/
pool=local size.state=2GiB`). Without it, `incus stop --stateful` fails.
- **Cluster evacuation**: `incus cluster evacuate <remote>:<member> --force`
(ONE argument, like `cluster enable` and `cluster add`).
Use `--action stop` if VMs lack the `limits.cpu` range fix.
Restore with `incus cluster restore <remote>:<member> --force`.
- **VM agent reconnect**: after live migration, the incus agent inside the VM
needs ~3-4 seconds to reconnect. `incus exec` commands issued immediately
after migration may fail with "VM agent isn't currently running". Scripts
should `sleep 4` after migration before running `incus exec`.
- **Multi-vCPU migration**: tested with 2, 3, and 4 vCPU VMs across
heterogeneous hosts (6/4/4 cores). Odd vCPU counts (e.g., `limits.cpu=0-2`)
work identically to even counts. A 4-vCPU VM on a 4-core host (100% core
usage) migrates without issues. `size.state=4GiB` recommended for 3-4 vCPU VMs.
- **Concurrent migrations**: migrating multiple VMs simultaneously from
different source nodes works without interference. ~140 MB/s per migration.
- **Active I/O during migration**: disk writes and network activity survive
live migration transparently. File integrity verified after migration.
- **Cluster rebalancing**: Incus can auto-redistribute VMs when a new node
joins. Only moves VMs with `migration.stateful=true`. Containers are NOT
auto-rebalanced.
```bash
incus config set <remote>: cluster.rebalance.interval=1 # minutes
incus config set <remote>: cluster.rebalance.threshold=10 # imbalance %
incus config set <remote>: cluster.rebalance.batch=2 # max VMs/run
incus config set <remote>: cluster.rebalance.cooldown=5m # wait between runs
```
- **Node replacement lifecycle**: evacuate → remove → destroy → deploy fresh →
join → auto-rebalance. Full procedure tested. See `notes/clustering-guide.md`
for step-by-step instructions.
- **`incus cluster remove` requires confirmation**: even with `--force`, it
prompts "Are you really sure?". Pipe `yes` for automation:
```bash
printf "yes\n" | incus cluster remove <remote>:<member> --force
```
- See `notes/clustering-guide.md` for full details and references.
### Lab validation (lab-test)
- **`lab-test`** reads the same YAML config as `incusos-proxmox` and operates
@ -187,11 +365,19 @@ incu-contrib/
### incusos-proxmox doctor and cleanup
- **`--doctor`**: standalone environment check. No config file required.
Checks tool versions, IncusOS CDN, and optionally Proxmox connectivity.
- **`--cleanup --deep`**: destroys VMs + deletes ISOs/seeds from Proxmox +
removes incus remotes + clears local ISO cache.
- **`--cleanup-all`**: pool-wide cleanup. Destroys all VMs with the
`[incusos-lab:managed]` marker in the configured resource pool.
Checks tool versions, IncusOS CDN, proxmox.yaml discovery, and optionally
Proxmox connectivity (from proxmox.yaml or config file).
- **`--cleanup`**: destroys VMs defined in the config file.
- **`--cleanup --deep`**: also deletes the specific IncusOS ISO used by this
deployment + per-VM seed ISOs + incus remotes + local cache. Does NOT delete
all IncusOS ISOs (unlike the old behavior).
- **`--cleanup-all`**: pool-wide cleanup. Only needs `proxmox.yaml` (no lab
config required). Destroys all VMs with `[incusos-lab:managed]` marker.
- **`--cleanup-all --deep`**: aggressive blanket delete of ALL `IncusOS_*.iso`
and `seed-*.iso` from storage + remotes + cache.
- **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
output is concise (step names + results). `--quiet` suppresses everything
except warnings and errors.
## Coding conventions for scripts

86
incusos/README.md
View File

@ -296,13 +296,16 @@ This script ensures all settings are correct, preventing install failures.
--no-cert Skip certificate injection
--vm NAME Operate on a single VM only
-y, --yes Skip confirmation prompts
--proxmox FILE Proxmox connection config (default: auto-discover)
--doctor Check tool versions and environment health
--status Check deployment status (shortcut for --phase status)
--cleanup Destroy VMs defined in config
--cleanup --deep Also remove ISOs, seeds, remotes, and local cache
--cleanup-all Destroy ALL managed VMs in the resource pool
--cleanup --deep Also remove deployment ISO, seeds, remotes, and cache
--cleanup-all Destroy ALL managed VMs in pool (no config file needed)
--cleanup-all --deep Also remove ALL ISOs and seeds from storage
--force-cleanup Destroy VMs even without management marker
--dry-run Preview without executing
-v, --verbose Show detailed output (tool paths, API calls)
-q, --quiet Suppress info output
-h, --help Full help text
-V, --version Show version
@ -314,14 +317,15 @@ Run `--doctor` to verify your environment without a config file:
```bash
./incusos-proxmox --doctor
./incusos-proxmox --doctor -v # verbose: show tool paths
```
This checks: incus client, flasher-tool, operations-center CLI (optional),
python3, curl, genisoimage, jq, incusos-seed sibling, and IncusOS CDN
availability.
python3, curl, genisoimage, jq, incusos-seed sibling, IncusOS CDN
availability, and `proxmox.yaml` discovery.
If a config file is also provided, additionally checks Proxmox connectivity,
pool existence, and API token validity:
If `proxmox.yaml` is found or a config file is provided, additionally checks
Proxmox connectivity, pool existence, and API token validity:
```bash
./incusos-proxmox --doctor examples/lab-cluster.yaml
@ -329,22 +333,28 @@ pool existence, and API token validity:
### Deep Cleanup
Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
Use `--deep` with `--cleanup` or `--cleanup-all` to remove artifacts beyond VMs:
```bash
# VMs + ISOs + seeds + incus remotes + local cache
# VMs + the IncusOS ISO used by this deployment + seeds + remotes + cache
./incusos-proxmox --cleanup --deep examples/lab-cluster.yaml
# Pool-wide: destroy all managed VMs + artifacts
./incusos-proxmox --cleanup-all --deep examples/lab-cluster.yaml
# Pool-wide: destroy all managed VMs (uses proxmox.yaml, no config needed)
./incusos-proxmox --cleanup-all
./incusos-proxmox --cleanup-all --deep # also nuke ALL ISOs + seeds
```
`--deep` additionally:
- Deletes seed ISOs from Proxmox storage
- Deletes IncusOS ISOs from Proxmox storage
`--cleanup --deep`:
- Deletes per-VM seed ISOs from Proxmox storage
- Deletes the specific IncusOS ISO used by this deployment (not all of them)
- Removes matching `incus remote` entries
- Clears the local ISO cache (`~/.cache/incusos-proxmox/iso-cache/`)
`--cleanup-all --deep`:
- Deletes ALL `IncusOS_*.iso` and `seed-*.iso` from storage (intentionally aggressive)
- Removes remotes for the destroyed VMs
- Clears the local ISO cache
### Deployment Phases
| Phase | What it does |
@ -356,18 +366,41 @@ Use `--deep` with `--cleanup` or `--cleanup-all` to remove all artifacts:
| `install` | Start VMs, detect install completion via disk I/O monitoring, detach CD-ROMs, boot from disk |
| `status` | Check deployment status: Proxmox state, IP, port 8443, incus remotes, post-deploy checks |
### Config File Format
### Proxmox Connection Config
Proxmox connection settings live in a separate `proxmox.yaml` file to avoid
duplicating credentials across lab configs. Auto-discovered in the script
directory and current directory. Override with `--proxmox FILE`.
Copy the template to get started:
```bash
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
# Edit with your Proxmox host, credentials, and storage settings
```
```yaml
# proxmox.yaml (flat structure, no wrapping key)
host: 192.168.1.10
method: api
api_token_id: automation@pve!deploy
node: pve
storage: local-lvm
iso_storage: local
bridge: vmbr0
pool: IncusLab
ssh_user: root
```
Lab configs can override any setting by including a `proxmox:` section.
Merge priority: `proxmox.yaml` → lab config `proxmox:` → CLI flags.
### Lab Config File Format
```yaml
# proxmox: section is optional — overrides proxmox.yaml if present
proxmox:
host: 192.168.1.10 # Proxmox host (required)
node: pve # Node name (default: pve)
storage: local-lvm # VM disk storage (default: local-lvm)
iso_storage: local # ISO storage (default: local)
bridge: vmbr0 # Network bridge (default: vmbr0)
pool: IncusLab # Resource pool (optional, for isolation)
method: ssh # ssh (default) or api
ssh_user: root # For ssh method
storage: local-zfs # Override storage for this deployment
defaults:
cores: 4
@ -602,10 +635,11 @@ See the [`examples/`](examples/) directory for sample seed configurations:
- [`incus-cluster-node.yaml`](examples/incus-cluster-node.yaml) -- Cluster node (no defaults)
- [`ops-center.yaml`](examples/ops-center.yaml) -- Operations Center
- [`network-static.yaml`](examples/network-static.yaml) -- Static network configuration
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (ops-center + 3-node cluster)
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster for lab validation
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes
- [`proxmox.yaml.example`](examples/proxmox.yaml.example) -- Proxmox connection config template
- [`proxmox-lab.yaml`](examples/proxmox-lab.yaml) -- Proxmox: 4-VM lab (self-contained, includes proxmox: section)
- [`proxmox-minimal.yaml`](examples/proxmox-minimal.yaml) -- Proxmox: single standalone VM (self-contained)
- [`lab-cluster.yaml`](examples/lab-cluster.yaml) -- Proxmox: 3-node cluster (uses proxmox.yaml)
- [`lab-oc.yaml`](examples/lab-oc.yaml) -- Proxmox: Operations Center + 3 nodes (uses proxmox.yaml)
## Security Notes

265
incusos/TESTING.md
View File

@ -129,13 +129,23 @@ ssh root@<YOUR_PROXMOX_IP> pvesh get /version
You should see JSON with the Proxmox VE version. If this fails, fix SSH
key auth first (`ssh-copy-id root@<host>`).
### Step 2: Create a Test Config
### Step 2: Set Up Proxmox Connection
```bash
cp incusos/examples/proxmox.yaml.example incusos/proxmox.yaml
```
Edit `incusos/proxmox.yaml` with your Proxmox host settings. This is
auto-discovered by the scripts and avoids repeating credentials in each
lab config.
### Step 3: Create a Test Config
```bash
cp incusos/examples/proxmox-minimal.yaml my-test.yaml
```
Edit `my-test.yaml` -- change only the host IP:
Edit `my-test.yaml` -- change only the host IP (or rely on proxmox.yaml):
```yaml
proxmox:
@ -155,7 +165,7 @@ vms:
Using VMID 900+ avoids clashing with existing VMs.
### Step 3: Dry Run
### Step 4: Dry Run
```bash
./incusos/incusos-proxmox --dry-run my-test.yaml
@ -172,7 +182,7 @@ Using VMID 900+ avoids clashing with existing VMs.
- `[dry-run]` prefix on all actions -- nothing actually executed
- No errors
### Step 4: Deploy (Single VM)
### Step 5: Deploy (Single VM)
```bash
./incusos/incusos-proxmox my-test.yaml
@ -196,7 +206,7 @@ Using VMID 900+ avoids clashing with existing VMs.
disk, and reboot
- After reboot you should see the IncusOS login prompt
### Step 5: Check Deployment Status
### Step 6: Check Deployment Status
After the deploy completes, verify with the built-in status command:
@ -212,7 +222,7 @@ After the deploy completes, verify with the built-in status command:
- Port 8443 is open
- Incus remote status shown (if configured)
### Step 6: Verify the Deployed VM
### Step 7: Verify the Deployed VM
Once the script reports success with an IP address:
@ -233,7 +243,7 @@ incus network list incus-test:
ARP-based lookup (MAC from Proxmox VM config → ARP table). If the IP is wrong,
check the Proxmox console — stale ARP entries can be misleading.
### Step 7: Test Re-run Reconcile
### Step 8: Test Re-run Reconcile
Re-run the deploy to verify the reconcile menu:
@ -252,7 +262,7 @@ Also test with `--yes` to verify it defaults to safe behavior:
**Verify:** Runs post-deployment checks, does NOT destroy.
### Step 8: Clean Up the Test VM
### Step 9: Clean Up the Test VM
```bash
./incusos/incusos-proxmox --cleanup my-test.yaml
@ -269,7 +279,7 @@ Remove the remote too:
incus remote remove incus-test
```
### Step 9: (Optional) Test API Method
### Step 10: (Optional) Test API Method
After single-VM success with SSH, switch to API tokens.
@ -302,9 +312,9 @@ proxmox:
api_token_id: automation@pve!deploy
```
Repeat steps 3-6 using the API method.
Repeat steps 4-7 using the API method.
### Step 10: (Optional) Test with Resource Pool
### Step 11: (Optional) Test with Resource Pool
For API method with pool isolation:
@ -331,7 +341,7 @@ Deploy and verify:
./incusos/incusos-proxmox --status my-test.yaml # Status scoped to pool
```
### Step 11: (Optional) Full Lab Deployment
### Step 12: (Optional) Full Lab Deployment
Once single-VM tests pass, deploy the full 4-VM lab:
@ -392,32 +402,229 @@ incus remote add incus-lab-03 <IP3> --accept-certificate
### Manual Cluster Formation (Alternative)
If you prefer to form the cluster manually instead of using `lab-test`:
If you prefer to form the cluster manually instead of using `lab-test`,
follow the steps below. See also `notes/clustering-guide.md` for the full
reference guide with explanations of what happens under the hood.
#### Prerequisites
```bash
# Step 1: Enable clustering on first node
incus cluster enable incus-lab-01: incus-lab-01
# Check client version (need 6.20+ for remote cluster join)
incus version
# Step 2: Generate join token for second node
incus cluster add incus-lab-01: incus-lab-02
# Verify all three remotes are working
incus info incus-lab-01: | head -5
incus info incus-lab-02: | head -5
incus info incus-lab-03: | head -5
```
# Step 3: Join second node
incus cluster join incus-lab-01: incus-lab-02:
#### Step 1: Set specific IPs on all nodes
# Step 4: Repeat for third node
incus cluster add incus-lab-01: incus-lab-03
incus cluster join incus-lab-01: incus-lab-03:
IncusOS defaults to `core.https_address: :8443` (wildcard). Clustering
requires a specific routable IP on each node.
# Step 5: Verify
```bash
# Discover each node's routable IP
for node in incus-lab-01 incus-lab-02 incus-lab-03; do
echo "=== $node ==="
incus query "$node:/1.0" | python3 -c "import sys,json; d=json.load(sys.stdin); \
[print(a) for a in d['environment']['addresses'] \
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('[')]"
done
# Set the specific IP on each node (replace with actual IPs)
incus config set incus-lab-01: core.https_address <IP1>:8443
incus config set incus-lab-02: core.https_address <IP2>:8443
incus config set incus-lab-03: core.https_address <IP3>:8443
# Verify connectivity still works after each change
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3
```
#### Step 2: Enable clustering on the init node
```bash
incus cluster enable incus-lab-01:incus-lab-01
```
> **Important**: This regenerates the server's TLS certificate. Your remote
> will break with a certificate error. Fix it:
```bash
incus remote switch local # if incus-lab-01 is the current default
incus remote remove incus-lab-01
incus remote add incus-lab-01 https://<IP1>:8443 --accept-certificate
```
Verify:
```bash
incus cluster list incus-lab-01:
# Should show one member (incus-lab-01), status ONLINE, role database-leader
```
#### Step 3: Prepare joining nodes
Nodes with `apply_defaults: true` have a pre-existing `local` storage pool
and `incusbr0` network. These conflict with the cluster join process (the
join wizard tries to set member-specific ZFS config keys at cluster level).
Delete them before joining:
```bash
# For each joining node (incus-lab-02, incus-lab-03):
NODE=incus-lab-02 # then repeat with incus-lab-03
# Remove server-level references to the pool
incus config unset "$NODE": storage.backups_volume
incus config unset "$NODE": storage.images_volume
# Delete the backup and image volumes
incus storage volume delete "$NODE":local backups
incus storage volume delete "$NODE":local images
# Clear default profile references (otherwise pool is "in use")
incus profile device remove "$NODE":default root
incus profile device remove "$NODE":default eth0
# Delete pool and network
incus storage delete "$NODE":local
incus network delete "$NODE":incusbr0
```
#### Step 4: Join nodes to the cluster
**Interactive** (one node at a time):
```bash
# Generate join token
incus cluster add incus-lab-01:incus-lab-02
# Join (answer the interactive prompts)
incus cluster join incus-lab-01: incus-lab-02:
# Prompt 1 - IP address: press Enter (accept default)
# Prompt 2 - Member name: press Enter (accept default)
# Prompt 3 - "All existing data is lost": type "yes"
# Prompt 4 - source for storage pool "local": type "local/incus"
# Prompt 5 - zfs.pool_name for storage pool "local": type "local/incus"
```
**Automated** (non-interactive):
```bash
# Generate token + join in one go
incus cluster add incus-lab-01:incus-lab-02
printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join incus-lab-01: incus-lab-02:
```
After each join, fix the remote (same certificate issue as the init node):
```bash
incus remote remove incus-lab-02
incus remote add incus-lab-02 https://<IP2>:8443 --accept-certificate
```
Repeat for incus-lab-03.
#### Step 5: Verify the cluster
```bash
# All 3 members should be ONLINE
incus cluster list incus-lab-01:
# Storage pool should show on all members
incus storage show incus-lab-01:local
incus storage show incus-lab-01:local --target incus-lab-01
incus storage show incus-lab-01:local --target incus-lab-02
incus storage show incus-lab-01:local --target incus-lab-03
# All remotes should work
incus info incus-lab-01: | head -3
incus info incus-lab-02: | head -3
incus info incus-lab-03: | head -3
```
**Key points:**
- `apply_defaults: true` on all nodes ensures matching storage pool and
network bridge configurations (required for migration)
network bridge names (required for migration) -- but the pools must be
deleted on joining nodes before the join
- After joining, manage the cluster through the init node's remote
- The ZFS pool source is `local/incus` on IncusOS (auto-created by
`apply_defaults`)
- Each join generates a new token (tokens are single-use and time-limited)
- VM migration requires matching storage pool names across nodes
### Migration Testing (After Cluster Formation)
Once the cluster is formed, validate container and VM migration:
#### Container migration (stop/move/start)
```bash
# Launch on node 2
incus launch images:debian/12 incus-lab-01:test-container --target incus-lab-02
incus exec incus-lab-01:test-container -- bash -c 'echo "hello" > /root/test.txt'
# Migrate to node 3
incus stop incus-lab-01:test-container
incus move incus-lab-01:test-container --target incus-lab-03
incus start incus-lab-01:test-container
# Verify: data survives, processes don't
incus exec incus-lab-01:test-container -- cat /root/test.txt # "hello"
incus list incus-lab-01: --columns nLs # node 3
```
#### VM live migration
```bash
# Launch and configure for migration
incus launch images:debian/12 incus-lab-01:test-vm --vm --target incus-lab-02
# Wait for VM agent...
incus stop incus-lab-01:test-vm
incus config set incus-lab-01:test-vm migration.stateful=true
incus config set incus-lab-01:test-vm limits.cpu=0-1
incus config device add incus-lab-01:test-vm root disk path=/ pool=local size.state=2GiB
incus start incus-lab-01:test-vm
# Set up heartbeat for state verification
incus exec incus-lab-01:test-vm -- mkdir -p /tmp/migration-test
incus exec incus-lab-01:test-vm -- bash -c \
'cat > /tmp/migration-test/heartbeat.sh << "EOF"
#!/bin/bash
i=0; while true; do echo $i > /tmp/migration-test/heartbeat; i=$((i+1)); sleep 1; done
EOF
chmod +x /tmp/migration-test/heartbeat.sh'
incus exec incus-lab-01:test-vm -- \
systemd-run --unit=migration-heartbeat /tmp/migration-test/heartbeat.sh
# Live-migrate (state preserved)
sleep 3
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # note value
incus move incus-lab-01:test-vm --target incus-lab-03
sleep 2 # agent needs ~1-2s to reconnect
incus exec incus-lab-01:test-vm -- cat /tmp/migration-test/heartbeat # should be higher
incus exec incus-lab-01:test-vm -- systemctl is-active migration-heartbeat # "active"
```
#### Cluster evacuation
```bash
# Move both workloads to node 2, then evacuate
incus cluster evacuate incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01: # node 2 = EVACUATED
incus list incus-lab-01: --columns nLs # workloads on other nodes
# Restore
incus cluster restore incus-lab-01:incus-lab-02 --force
incus cluster list incus-lab-01: # node 2 = ONLINE
incus list incus-lab-01: --columns nLs # workloads back on node 2
```
See `notes/clustering-guide.md` for the full automated migration test script
and detailed test results.
### Cleanup
```bash
@ -427,8 +634,12 @@ incus cluster list incus-lab-01:
# Destroy VMs only
./incusos/incusos-proxmox --cleanup examples/lab-cluster.yaml
# Full cleanup: VMs + ISOs + seeds + remotes + cache
# Full cleanup: VMs + deployment ISO + seeds + remotes + cache
./incusos/incusos-proxmox --cleanup --deep --yes examples/lab-cluster.yaml
# Pool-wide: destroy ALL managed VMs (no config file needed)
./incusos/incusos-proxmox --cleanup-all --yes
./incusos/incusos-proxmox --cleanup-all --deep --yes # nuke everything
```
---
@ -507,3 +718,7 @@ CLI is installed.
| Pool not found | Resource pool not created | Run `pveum pool add <name>` on Proxmox |
| Re-run destroys VMs | Used option 3 in reconcile | Use option 1 (checks) or option 2 (continue) instead |
| `--status` shows "not found" | VM was cleaned up | Deploy first, then check status |
| VM live migration fails: "Missing section footer for ICH9LPC" | `limits.cpu` not set or set as integer | Use range syntax: `limits.cpu=0-1` (not `2`); see `notes/clustering-guide.md` |
| `migration.stateful` cannot be set while running | Incus requires VM to be stopped | Stop VM first, set config, then start |
| `certificate is valid for 127.0.0.1, ::1, not <IP>` | Cluster cert regenerated on enable/join | Remove and re-add remote with `--accept-certificate` |
| `Config key "zfs.pool_name" is cluster member specific` | Storage pool exists on joining node | Delete pool/network before joining; see clustering guide |

45
incusos/examples/lab-advanced.yaml Normal file
View File

@ -0,0 +1,45 @@
# lab-advanced.yaml - 3-node heterogeneous cluster for advanced migration testing
#
# Deploys 3 Incus nodes with a 6/4/4 core split to test VM live migration
# across hosts with different CPU topologies. 8 GiB RAM provides headroom
# for multiple concurrent VMs with size.state for stateful operations.
#
# Test matrix targets:
# - Multi-vCPU VMs (2, 3, 4 vCPUs) with limits.cpu ranges
# - Concurrent migrations from different source nodes
# - Edge cases: active I/O, memory-loaded VMs, 4-vCPU on 4-core host
# - Cluster lifecycle: evacuate, remove, replace, rejoin, rebalance
#
# Usage:
# incusos-proxmox --dry-run examples/lab-advanced.yaml # Preview
# incusos-proxmox --yes examples/lab-advanced.yaml # Deploy
# incusos-proxmox --status examples/lab-advanced.yaml # Check status
# incusos-proxmox --cleanup examples/lab-advanced.yaml # Tear down VMs
# incusos-proxmox --cleanup --deep examples/lab-advanced.yaml # Full cleanup
#
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
# Add a proxmox: section here to override per-deployment.
#
# Note: start_vmid 910 may collide with lab-oc.yaml. Clean up old deployments
# first or adjust the range.
defaults:
memory: 8192
disk: 50
start_vmid: 910
vms:
- name: incus-adv-01
app: incus
apply_defaults: true
cores: 6
- name: incus-adv-02
app: incus
apply_defaults: true
cores: 4
- name: incus-adv-03
app: incus
apply_defaults: true
cores: 4

15
incusos/examples/lab-cluster.yaml
View File

@ -13,16 +13,12 @@
#
# After deployment, form a cluster with lab-test:
# lab-test --phase cluster examples/lab-cluster.yaml
proxmox:
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
storage: local-zfs
pool: IncusLab
#
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
# Add a proxmox: section here to override per-deployment.
defaults:
cores: 2
cores: 4
memory: 4096
disk: 50
start_vmid: 900
@ -31,9 +27,6 @@ vms:
- name: incus-lab-01
app: incus
apply_defaults: true
cores: 4
memory: 8192
disk: 64
- name: incus-lab-02
app: incus

10
incusos/examples/lab-oc.yaml
View File

@ -15,13 +15,9 @@
# incusos-proxmox --yes examples/lab-oc.yaml
# incusos-proxmox --status examples/lab-oc.yaml
# incusos-proxmox --cleanup --deep examples/lab-oc.yaml
proxmox:
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
storage: local-zfs
pool: IncusLab
#
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
# Add a proxmox: section here to override per-deployment.
defaults:
cores: 2

24
incusos/examples/lab-replace.yaml Normal file
View File

@ -0,0 +1,24 @@
# lab-replace.yaml - Single replacement node for cluster lifecycle testing
#
# Used in Part 4 of the advanced migration test plan: after evacuating,
# removing, and destroying incus-adv-03, this config deploys a fresh
# replacement node that gets joined back into the cluster.
#
# Usage:
# incusos-proxmox --dry-run examples/lab-replace.yaml
# incusos-proxmox --yes examples/lab-replace.yaml
# incusos-proxmox --status examples/lab-replace.yaml
# incusos-proxmox --cleanup examples/lab-replace.yaml
#
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
defaults:
memory: 8192
disk: 50
start_vmid: 913
vms:
- name: incus-adv-04
app: incus
apply_defaults: true
cores: 4

8
incusos/examples/my-test.yaml
View File

@ -1,9 +1,5 @@
proxmox:
host: 192.168.1.29
method: api
api_token_id: automation@pve!deploy
storage: local-zfs # VM disk storage (default: local-lvm)
pool: IncusLab
# Connection settings from proxmox.yaml (see examples/proxmox.yaml.example).
# Add a proxmox: section here to override per-deployment.
defaults:
cores: 2

21
incusos/examples/proxmox.yaml.example Normal file
View File

@ -0,0 +1,21 @@
# proxmox.yaml.example - Proxmox connection config template
#
# Copy to incusos/proxmox.yaml and edit with your settings.
# This file is auto-discovered by incusos-proxmox and lab-test.
#
# Lab config files (lab-cluster.yaml, lab-oc.yaml, etc.) reference
# this for connection settings, so you only need to change credentials
# in one place.
#
# Override any setting per-deployment by adding a proxmox: section
# to the lab config file.
host: 192.168.1.10 # Proxmox host IP or hostname
method: ssh # ssh (default) or api
# api_token_id: automation@pve!deploy # For api method
node: pve # Proxmox node name
storage: local-lvm # VM disk storage
iso_storage: local # ISO/image storage
bridge: vmbr0 # Network bridge
# pool: IncusLab # Resource pool (optional, for isolation)
ssh_user: root # SSH user (for ssh method)

615
incusos/incusos-proxmox
View File

@ -43,7 +43,7 @@ success() { [[ "$QUIET" == true ]] && return; echo -e "${GREEN}[ok]${RESET}
warn() { echo -e "${YELLOW}[warn]${RESET} $*" >&2; }
error() { echo -e "${RED}[error]${RESET} $*" >&2; }
step() { [[ "$QUIET" == true ]] && return; echo -e "${CYAN}[step]${RESET} ${BOLD}$*${RESET}"; }
detail() { [[ "$QUIET" == true ]] && return; echo -e "${DIM} $*${RESET}"; }
detail() { [[ "$VERBOSE" != true ]] && return; echo -e "${DIM} $*${RESET}"; }
# ---------------------------------------------------------------------------
# Help
@ -91,25 +91,39 @@ ${BOLD}OPTIONS${RESET}
${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only)
${BOLD}--cleanup --deep${RESET} Also remove ISOs, seeds, remotes, and local cache
${BOLD}--cleanup-all${RESET} Destroy ALL managed VMs in the pool
${BOLD}--proxmox${RESET} FILE Proxmox connection config (default: auto-discover)
${BOLD}--force-cleanup${RESET} Destroy VMs even without management marker
${BOLD}--dry-run${RESET} Preview actions without executing
${BOLD}-v, --verbose${RESET} Show detailed output (tool paths, API calls)
${BOLD}-q, --quiet${RESET} Suppress informational output
${BOLD}-h, --help${RESET} Show this help message
${BOLD}-V, --version${RESET} Show version
${BOLD}PROXMOX CONNECTION CONFIG${RESET}
Proxmox connection settings can be stored in a separate proxmox.yaml file
to avoid duplicating credentials across lab configs. Auto-discovered in:
script directory, then current directory. Override with --proxmox FILE.
${BOLD}proxmox.yaml${RESET} (flat structure, no wrapping key):
host: 192.168.1.10 # Proxmox host (required)
method: api # ssh or api (default: ssh)
api_token_id: user!token # API token (for api method)
node: pve # Node name (default: pve)
storage: local-lvm # VM disk storage (default: local-lvm)
iso_storage: local # ISO storage (default: local)
bridge: vmbr0 # Network bridge (default: vmbr0)
pool: IncusLab # Resource pool (optional)
ssh_user: root # SSH user (default: root)
Merge priority: proxmox.yaml → lab config proxmox: section → CLI flags
${BOLD}CONFIG FILE${RESET}
YAML file defining the Proxmox connection and VM specifications.
YAML file defining VM specifications (and optional Proxmox overrides).
See examples/proxmox-lab.yaml for a full example.
${BOLD}proxmox:${RESET}
host: 192.168.1.10 # Proxmox host (required)
node: pve # Node name (default: pve)
storage: local-lvm # VM disk storage (default: local-lvm)
iso_storage: local # ISO storage (default: local)
bridge: vmbr0 # Network bridge (default: vmbr0)
pool: IncusLab # Resource pool (optional)
method: ssh # ssh or api (default: ssh)
ssh_user: root # SSH user (default: root)
${BOLD}proxmox:${RESET} # Optional: overrides proxmox.yaml
host: 192.168.1.10 # Proxmox host
storage: local-zfs # Override storage for this deployment
${BOLD}defaults:${RESET}
cores: 4 # Default CPU cores
@ -152,16 +166,20 @@ ${BOLD}EXAMPLES${RESET}
# Destroy all VMs defined in config
${SCRIPT_NAME} --cleanup examples/proxmox-lab.yaml
# Deep cleanup: VMs + ISOs + seeds + remotes + cache
# Deep cleanup: VMs + the IncusOS ISO used + seeds + remotes + cache
${SCRIPT_NAME} --cleanup --deep examples/proxmox-lab.yaml
# Destroy ALL managed VMs in the pool (not config-specific)
${SCRIPT_NAME} --cleanup-all examples/proxmox-lab.yaml
# Destroy ALL managed VMs in the pool (uses proxmox.yaml, no config needed)
${SCRIPT_NAME} --cleanup-all
${SCRIPT_NAME} --cleanup-all --deep # also remove all ISOs + seeds
# Check tool versions and environment health
${SCRIPT_NAME} --doctor
${SCRIPT_NAME} --doctor examples/proxmox-lab.yaml # also check Proxmox
# Verbose output (show dependency paths, API calls, per-file actions)
${SCRIPT_NAME} --doctor -v
# Use API method with token
export PROXMOX_TOKEN_SECRET="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
${SCRIPT_NAME} --method api examples/proxmox-lab.yaml
@ -228,6 +246,8 @@ DEEP_CLEANUP=false
FORCE_CLEANUP=false
DRY_RUN=false
QUIET=false
VERBOSE=false
PROXMOX_CONFIG=""
CONFIG_FILE=""
parse_args() {
@ -290,10 +310,18 @@ parse_args() {
FORCE_CLEANUP=true
shift
;;
--proxmox)
PROXMOX_CONFIG="${2:?'--proxmox requires a file path'}"
shift 2
;;
--dry-run)
DRY_RUN=true
shift
;;
-v|--verbose)
VERBOSE=true
shift
;;
-q|--quiet)
QUIET=true
shift
@ -329,11 +357,22 @@ parse_args() {
# ---------------------------------------------------------------------------
validate_args() {
# Universal checks (apply regardless of mode)
if [[ "$VERBOSE" == true ]] && [[ "$QUIET" == true ]]; then
error "Cannot use both --verbose and --quiet"
exit 1
fi
# --doctor can run without a config file
if [[ "$DOCTOR" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
return 0
fi
# --cleanup-all can run without a config file (uses proxmox.yaml)
if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
return 0
fi
if [[ -z "$CONFIG_FILE" ]]; then
error "No configuration file specified"
echo "Usage: ${SCRIPT_NAME} [OPTIONS] CONFIG_FILE" >&2
@ -605,7 +644,19 @@ if versions:
echo -e " Cached ISO: ${DIM}none${RESET}"
fi
# -- Proxmox section (only if config file provided) --
# -- Config section --
echo ""
echo -e " ${BOLD}Config:${RESET}"
if find_proxmox_config; then
echo -e " proxmox.yaml: ${GREEN}${PROXMOX_CONFIG_RESOLVED} ✓${RESET}"
else
echo -e " proxmox.yaml: ${DIM}not found (optional)${RESET}"
fi
if [[ -n "$CONFIG_FILE" ]] && [[ -f "$CONFIG_FILE" ]]; then
echo -e " Lab config: ${CONFIG_FILE}"
fi
# -- Proxmox section (if config file or proxmox.yaml provides connection info) --
echo ""
if [[ -n "$CONFIG_FILE" ]] && [[ -f "$CONFIG_FILE" ]]; then
echo -e " ${BOLD}Proxmox:${RESET}"
@ -713,9 +764,68 @@ json.dump(data, sys.stdout, indent=2)
echo -e " ${RED}Failed to parse config file${RESET}"
fi
rm -rf "$doctor_workdir"
elif [[ -n "$PROXMOX_CONFIG_RESOLVED" ]]; then
# Use proxmox.yaml for connectivity check (no lab config provided)
echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(from proxmox.yaml)${RESET}"
local doctor_workdir
doctor_workdir=$(mktemp -d "${TMPDIR:-/tmp}/${SCRIPT_NAME}-doctor-XXXXXX")
WORKDIR="$doctor_workdir"
load_proxmox_config "$PROXMOX_CONFIG_RESOLVED"
[[ -z "$PVE_NODE" ]] && PVE_NODE="pve"
[[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh"
[[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root"
[[ -n "$METHOD" ]] && PVE_METHOD="$METHOD"
[[ -n "$HOST_OVERRIDE" ]] && PVE_HOST="$HOST_OVERRIDE"
if [[ -z "$PVE_HOST" ]]; then
echo -e " Host: ${RED}not set in proxmox.yaml${RESET}"
else
echo -e " Host: ${PVE_HOST} (${PVE_METHOD})"
local pve_ok=false
if [[ "$PVE_METHOD" == "api" ]]; then
if [[ -n "$PVE_API_TOKEN_ID" ]] && [[ -n "${PROXMOX_TOKEN_SECRET:-}" ]]; then
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
local pve_ver
pve_ver=$(curl -fsSk --connect-timeout 5 -H "$auth" \
"https://${PVE_HOST}:8006/api2/json/version" 2>/dev/null | \
python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('version',''))" 2>/dev/null) || pve_ver=""
if [[ -n "$pve_ver" ]]; then
echo -e " Connectivity: ${GREEN}connected (PVE ${pve_ver}) ✓${RESET}"
pve_ok=true
else
echo -e " Connectivity: ${RED}API connection failed${RESET}"
fi
else
echo -e " Connectivity: ${YELLOW}API token not configured or PROXMOX_TOKEN_SECRET not set${RESET}"
fi
else
if ssh -o BatchMode=yes -o ConnectTimeout=5 "${PVE_SSH_USER}@${PVE_HOST}" "true" &>/dev/null; then
echo -e " Connectivity: ${GREEN}SSH connected ✓${RESET}"
pve_ok=true
else
echo -e " Connectivity: ${RED}SSH connection failed${RESET}"
fi
fi
if [[ -n "$PVE_POOL" ]]; then
if [[ "$pve_ok" == true ]] && [[ "$PVE_METHOD" == "api" ]]; then
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
if curl -fsSk --connect-timeout 5 -H "$auth" \
"https://${PVE_HOST}:8006/api2/json/pools/${PVE_POOL}" &>/dev/null; then
echo -e " Pool: ${GREEN}${PVE_POOL} exists ✓${RESET}"
else
echo -e " Pool: ${RED}${PVE_POOL} not found${RESET}"
fi
else
echo -e " Pool: ${PVE_POOL} (not verified)"
fi
fi
fi
rm -rf "$doctor_workdir"
else
echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(requires config file, skipped)${RESET}"
echo -e " ${BOLD}Proxmox:${RESET} ${DIM}(no config file or proxmox.yaml found)${RESET}"
fi
echo ""
@ -1106,29 +1216,121 @@ declare -a VM_VMIDS=()
declare -a VM_CLUSTERS=()
declare -a VM_CLUSTER_ROLES=()
# Path to the resolved proxmox connection config file (set by find_proxmox_config)
PROXMOX_CONFIG_RESOLVED=""
# Auto-discover proxmox.yaml: --proxmox flag > script directory > cwd
find_proxmox_config() {
if [[ -n "$PROXMOX_CONFIG" ]]; then
if [[ ! -f "$PROXMOX_CONFIG" ]]; then
error "Proxmox config file not found: ${PROXMOX_CONFIG}"
exit 1
fi
PROXMOX_CONFIG_RESOLVED="$PROXMOX_CONFIG"
detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (--proxmox)"
return 0
fi
local script_dir
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [[ -f "${script_dir}/proxmox.yaml" ]]; then
PROXMOX_CONFIG_RESOLVED="${script_dir}/proxmox.yaml"
detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (auto-discovered)"
return 0
fi
if [[ -f "proxmox.yaml" ]]; then
PROXMOX_CONFIG_RESOLVED="$(pwd)/proxmox.yaml"
detail "Using proxmox config: ${PROXMOX_CONFIG_RESOLVED} (auto-discovered)"
return 0
fi
# No proxmox.yaml found -- caller decides if this is an error
return 1
}
# Parse proxmox.yaml (flat key: value structure) and set PVE_* globals as base values
load_proxmox_config() {
local pxconfig="$1"
local px_json
px_json=$(mktemp "${WORKDIR}/proxmox-config-XXXXXX.json")
if ! python3 -c "
import json, sys, re
result = {}
with open(sys.argv[1]) as f:
for line in f:
s = line.strip()
if not s or s.startswith('#'):
continue
m = re.match(r'^([^:]+?):\s*(.*?)$', s)
if m:
key = m.group(1).strip()
val = m.group(2).strip()
# Strip inline comments
if val and not val.startswith(('\"', \"'\")):
cm = re.match(r'^(.*?)\s+#.*$', val)
if cm:
val = cm.group(1).rstrip()
if (val.startswith('\"') and val.endswith('\"')) or (val.startswith(\"'\") and val.endswith(\"'\")):
val = val[1:-1]
result[key] = val
json.dump(result, sys.stdout, indent=2)
" "$pxconfig" > "$px_json" 2>/dev/null; then
error "Failed to parse proxmox config: ${pxconfig}"
exit 1
fi
# Set base values from proxmox.yaml (lab config overlay and CLI flags applied later)
local val
val=$(json_get "$px_json" ".host" ""); [[ -n "$val" ]] && PVE_HOST="$val"
val=$(json_get "$px_json" ".node" ""); [[ -n "$val" ]] && PVE_NODE="$val"
val=$(json_get "$px_json" ".storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
val=$(json_get "$px_json" ".method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
rm -f "$px_json"
}
load_config() {
step "Loading configuration values"
# Proxmox settings
PVE_HOST=$(json_get "$CONFIG_JSON" ".proxmox.host" "")
PVE_NODE=$(json_get "$CONFIG_JSON" ".proxmox.node" "pve")
PVE_STORAGE=$(json_get "$CONFIG_JSON" ".proxmox.storage" "local-lvm")
PVE_ISO_STORAGE=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" "local")
PVE_BRIDGE=$(json_get "$CONFIG_JSON" ".proxmox.bridge" "vmbr0")
PVE_SSH_USER=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" "root")
PVE_API_TOKEN_ID=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" "")
PVE_POOL=$(json_get "$CONFIG_JSON" ".proxmox.pool" "")
# Merge priority: proxmox.yaml (base) → lab config proxmox: (overlay) → CLI flags
# Step 1: proxmox.yaml base values were already loaded by the caller via
# load_proxmox_config(). PVE_* globals may already be set.
# Method: CLI override > config > default
if [[ -z "$PVE_METHOD" ]]; then
PVE_METHOD=$(json_get "$CONFIG_JSON" ".proxmox.method" "ssh")
fi
# CLI override for method
# Step 2: overlay from lab config proxmox: section (if present)
local val
val=$(json_get "$CONFIG_JSON" ".proxmox.host" ""); [[ -n "$val" ]] && PVE_HOST="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.node" ""); [[ -n "$val" ]] && PVE_NODE="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.storage" ""); [[ -n "$val" ]] && PVE_STORAGE="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.method" ""); [[ -n "$val" ]] && PVE_METHOD="$val"
# Step 3: apply defaults for anything still unset
[[ -z "$PVE_NODE" ]] && PVE_NODE="pve"
[[ -z "$PVE_STORAGE" ]] && PVE_STORAGE="local-lvm"
[[ -z "$PVE_ISO_STORAGE" ]] && PVE_ISO_STORAGE="local"
[[ -z "$PVE_BRIDGE" ]] && PVE_BRIDGE="vmbr0"
[[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root"
[[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh"
# Step 4: CLI flags override everything
if [[ -n "$METHOD" ]]; then
PVE_METHOD="$METHOD"
fi
# Host override
if [[ -n "$HOST_OVERRIDE" ]]; then
PVE_HOST="$HOST_OVERRIDE"
fi
@ -1315,7 +1517,7 @@ pve_run() {
local cmd="$1"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] ${PVE_METHOD}: ${cmd}"
info "[dry-run] ${PVE_METHOD}: ${cmd}"
return 0
fi
@ -1345,7 +1547,7 @@ pve_api() {
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET:-}"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] API ${method} ${endpoint}"
info "[dry-run] API ${method} ${endpoint}"
if [[ -n "$data" ]]; then
detail "[dry-run] data: ${data}"
fi
@ -1387,7 +1589,7 @@ pve_upload() {
local remote_filename="$2"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] upload: ${local_path} -> ${PVE_ISO_STORAGE}:iso/${remote_filename}"
info "[dry-run] upload: ${local_path} -> ${PVE_ISO_STORAGE}:iso/${remote_filename}"
return 0
fi
@ -1529,7 +1731,7 @@ pve_create_vm() {
fi
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] ${cmd}"
info "[dry-run] ${cmd}"
return 0
fi
@ -2498,6 +2700,11 @@ phase_validate() {
step "Phase 1: Validate"
echo ""
# Load proxmox.yaml base values (before lab config overlay)
if find_proxmox_config; then
load_proxmox_config "$PROXMOX_CONFIG_RESOLVED"
fi
parse_config
load_config
validate_config
@ -2527,7 +2734,7 @@ phase_prepare() {
fi
elif [[ "$DRY_RUN" == true ]]; then
ISO_FILENAME="IncusOS_latest.iso"
detail "[dry-run] Would fetch latest version from CDN and download ISO"
info "[dry-run] Would fetch latest version from CDN and download ISO"
else
# Check if ISO already exists on Proxmox
fetch_latest_version
@ -2581,7 +2788,7 @@ phase_prepare() {
fi
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] ${seed_cmd[*]}"
info "[dry-run] ${seed_cmd[*]}"
else
if ! "${seed_cmd[@]}"; then
error "Failed to generate seed for VM '${name}'"
@ -2708,6 +2915,78 @@ phase_create() {
success "All VMs created"
}
# ---------------------------------------------------------------------------
# Remote registration
# ---------------------------------------------------------------------------
# Register an incus remote for a deployed VM.
# Handles stale remotes: if a remote exists but points to a different IP,
# it is removed and re-added with the correct IP (DHCP may reassign addresses
# between deploys).
# Usage: register_remote <name> <ip>
register_remote() {
local name="$1"
local ip="$2"
# Only register if incus client is available and a cert was injected
if ! command -v incus &>/dev/null; then
return
fi
if [[ -z "$CERT_FILE_RESOLVED" ]]; then
return
fi
# Check if remote already exists
local existing_url=""
existing_url=$(incus remote list --format csv 2>/dev/null | \
awk -F',' -v name="$name" '$1 == name {print $2; exit}') || true
if [[ -n "$existing_url" ]]; then
# Remote exists -- check if the IP matches
if echo "$existing_url" | grep -q "${ip}"; then
detail "Remote '${name}' already configured at ${ip}"
return
fi
# Stale remote -- IP changed (DHCP reassignment between deploys)
if [[ "$DRY_RUN" == true ]]; then
info "[dry-run] Would update stale remote '${name}': ${existing_url} -> ${ip}"
return
fi
warn "Remote '${name}' points to ${existing_url} but VM is at ${ip} -- updating"
# Switch away first -- can't remove the current/default remote
local current_remote=""
current_remote=$(incus remote get-default 2>/dev/null) || true
if [[ "$current_remote" == "$name" ]]; then
incus remote switch local 2>/dev/null || true
fi
incus remote remove "$name" 2>/dev/null || true
fi
if [[ "$DRY_RUN" == true ]]; then
info "[dry-run] Would add incus remote '${name}' at ${ip}"
return
fi
# Wait for port 8443 to become reachable (up to 30s)
local attempt=0
while [[ $attempt -lt 6 ]]; do
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
break
fi
sleep 5
attempt=$((attempt + 1))
done
if incus remote add "$name" "$ip" --accept-certificate --quiet 2>/dev/null; then
success "Remote '${name}' added (${ip})"
else
warn "Could not add remote '${name}' at ${ip} -- add manually:"
warn " incus remote add ${name} ${ip} --accept-certificate"
fi
}
# ---------------------------------------------------------------------------
# Phase 5: Install
# ---------------------------------------------------------------------------
@ -2730,7 +3009,7 @@ phase_install() {
if [[ "$DRY_RUN" == true ]]; then
step "Starting VM '${name}' (VMID ${vmid})"
detail "[dry-run] Would start VM, swap boot order, wait for install, clean up"
info "[dry-run] Would start VM, swap boot order, wait for install, clean up"
i=$((i + 1))
continue
fi
@ -2743,12 +3022,13 @@ phase_install() {
pve_vm_has_install_media "$vmid" || has_media=false
if [[ "$cur_status" == "running" ]] && [[ "$has_media" == false ]]; then
# Already installed and running — detect IP and skip
# Already installed and running — detect IP, register remote, skip
success "VM '${name}' (VMID ${vmid}) is already installed and running"
local ip
ip=$(pve_vm_ip "$vmid") || ip=""
if [[ -n "$ip" ]]; then
detail "IP: ${ip}"
register_remote "$name" "$ip"
fi
i=$((i + 1))
continue
@ -2763,6 +3043,7 @@ phase_install() {
ip=$(pve_vm_ip "$vmid") || ip=""
if [[ -n "$ip" ]]; then
success "VM '${name}' started at ${ip}"
register_remote "$name" "$ip"
else
success "VM '${name}' started from disk"
fi
@ -2805,7 +3086,7 @@ phase_install() {
# installation is complete. We then stop the VM, remove install media,
# and start cleanly from disk.
step "Waiting for '${name}' to complete installation"
detail "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)"
info "Monitoring disk write activity (polling every 5s, timeout: 10 minutes)"
local timeout=600
local interval=5
@ -2840,14 +3121,14 @@ phase_install() {
local wr_mb=$(( wr_bytes / 1048576 ))
# Show progress every 15s
if [[ $((elapsed % 15)) -eq 0 ]]; then
detail "Installing... (${wr_mb} MiB written) [${elapsed}s]"
info "Installing... (${wr_mb} MiB written) [${elapsed}s]"
fi
elif [[ "$writes_started" == true ]]; then
# Writes have stopped after being active
stable_count=$((stable_count + 1))
if [[ $stable_count -ge $STABLE_THRESHOLD ]]; then
local wr_mb=$(( wr_bytes / 1048576 ))
detail "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]"
info "Disk activity stopped (${wr_mb} MiB written total) [${elapsed}s]"
install_done=true
break
fi
@ -2873,7 +3154,7 @@ phase_install() {
local cur_status
cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
if [[ "$cur_status" == "running" ]]; then
detail "Stopping VM to remove install media..."
info "Stopping VM to remove install media..."
pve_stop_vm "$vmid" || true
sleep 5
fi
@ -2892,7 +3173,7 @@ phase_install() {
fi
# 5. Wait for services to come up
detail "Waiting for services to start..."
info "Waiting for services to start..."
sleep 30
# 6. Try to detect IP
@ -2902,12 +3183,10 @@ phase_install() {
echo ""
if [[ -n "$ip" ]]; then
success "VM '${name}' installed and running at ${ip}"
if [[ -n "$CERT_FILE_RESOLVED" ]]; then
detail "Add remote: incus remote add ${name} ${ip}"
fi
register_remote "$name" "$ip"
else
success "VM '${name}' installed and running"
detail "Check Proxmox console for IP address"
info "Check Proxmox console for IP address"
fi
echo ""
@ -2988,7 +3267,7 @@ phase_status() {
fi
echo -e " Network: ${ip} port 8443: ${port_status}"
# Incus remote check
# Incus remote check + auto-register
if command -v incus &>/dev/null; then
local remote_name=""
remote_name=$(incus remote list --format csv 2>/dev/null | \
@ -2996,7 +3275,15 @@ phase_status() {
if [[ -n "$remote_name" ]]; then
echo -e " Remote: ${GREEN}incus remote '${remote_name}' configured${RESET}"
else
echo -e " Remote: ${YELLOW}no incus remote for ${ip}${RESET}"
register_remote "$name" "$ip"
# Re-check after registration attempt
remote_name=$(incus remote list --format csv 2>/dev/null | \
awk -F',' -v addr="$ip" '$2 ~ addr {print $1; exit}') || true
if [[ -n "$remote_name" ]]; then
echo -e " Remote: ${GREEN}incus remote '${remote_name}' configured${RESET}"
else
echo -e " Remote: ${YELLOW}no incus remote for ${ip}${RESET}"
fi
fi
fi
else
@ -3032,7 +3319,7 @@ phase_post_deploy() {
local app="${VM_APPS[$i]}"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would check '${name}' (VMID ${vmid})"
info "[dry-run] Would check '${name}' (VMID ${vmid})"
i=$((i + 1))
continue
fi
@ -3089,13 +3376,22 @@ phase_post_deploy() {
all_ok=false
fi
else
detail "VM '${name}': no incus remote configured"
detail " Add with: incus remote add ${name} ${ip} --accept-certificate"
# Auto-register the remote
register_remote "$name" "$ip"
# Verify
remote_name=$(incus remote list --format csv 2>/dev/null | \
awk -F',' -v addr="$ip" '$2 ~ addr {print $1; exit}') || true
if [[ -n "$remote_name" ]]; then
success "VM '${name}': incus remote '${remote_name}' connected"
else
info "VM '${name}': no incus remote configured"
info " Add with: incus remote add ${name} ${ip} --accept-certificate"
fi
fi
elif [[ "$app" == "operations-center" ]]; then
success "VM '${name}': Operations Center at https://${ip}:8443"
detail " Import PKCS#12 cert in browser for web UI access:"
detail " openssl pkcs12 -export -inkey ~/.config/incus/client.key -in ~/.config/incus/client.crt -out client.pfx"
info " Import PKCS#12 cert in browser for web UI access:"
info " openssl pkcs12 -export -inkey ~/.config/incus/client.key -in ~/.config/incus/client.crt -out client.pfx"
fi
i=$((i + 1))
@ -3163,7 +3459,7 @@ do_cleanup() {
step "Destroying VM '${name}' (VMID ${vmid})"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would stop and destroy VM ${vmid}"
info "[dry-run] Would stop and destroy VM ${vmid}"
destroyed=$((destroyed + 1))
i=$((i + 1))
continue
@ -3173,7 +3469,7 @@ do_cleanup() {
local status
status=$(pve_vm_status "$vmid") || status="unknown"
if [[ "$status" == "running" ]]; then
detail "Stopping VM..."
info "Stopping VM..."
pve_stop_vm "$vmid" || true
sleep 5
fi
@ -3201,9 +3497,67 @@ do_cleanup() {
success "$summary"
}
# Deep cleanup: remove ISOs, seeds, remotes, and local cache
# Detect which IncusOS ISO was used by checking ide2 config on existing VMs.
# Sets DEPLOYMENT_ISO to the filename (e.g. "IncusOS_25.03.iso") or "" if unknown.
DEPLOYMENT_ISO=""
detect_deployment_iso() {
DEPLOYMENT_ISO=""
if [[ "$DRY_RUN" == true ]]; then
return
fi
local count=${#VM_NAMES[@]}
local i=0
while [[ $i -lt $count ]]; do
local vmid="${VM_VMIDS[$i]}"
if pve_vm_exists "$vmid"; then
local config=""
case "$PVE_METHOD" in
ssh)
config=$(pve_run "qm config ${vmid} 2>/dev/null" 2>/dev/null) || config=""
;;
api)
config=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null) || config=""
;;
esac
if [[ -n "$config" ]]; then
local iso_name
# Extract IncusOS ISO name from ide2 line or JSON
iso_name=$(echo "$config" | python3 -c "
import sys, json, re
text = sys.stdin.read()
# Try JSON (API response)
try:
data = json.loads(text)
ide2 = data.get('data', data).get('ide2', '')
except (json.JSONDecodeError, AttributeError):
ide2 = ''
for line in text.splitlines():
if line.startswith('ide2:'):
ide2 = line.split(':', 1)[1].strip()
break
m = re.search(r'(IncusOS_[^,]+\.iso)', ide2)
if m:
print(m.group(1))
" 2>/dev/null) || iso_name=""
if [[ -n "$iso_name" ]]; then
DEPLOYMENT_ISO="$iso_name"
detail "Detected deployment ISO: ${DEPLOYMENT_ISO} (from VM ${vmid})"
return
fi
fi
fi
i=$((i + 1))
done
}
# Deep cleanup: remove seeds and the specific IncusOS ISO used by this deployment
cleanup_storage() {
step "Deep cleanup: removing ISOs and seeds from Proxmox storage"
step "Deep cleanup: removing seeds and deployment ISO from Proxmox storage"
local count=${#VM_NAMES[@]}
local i=0
@ -3218,7 +3572,7 @@ cleanup_storage() {
local seed_file="seed-${name}.iso"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would delete ${PVE_ISO_STORAGE}:iso/${seed_file}"
info "[dry-run] Would delete ${PVE_ISO_STORAGE}:iso/${seed_file}"
else
case "$PVE_METHOD" in
ssh)
@ -3228,18 +3582,46 @@ cleanup_storage() {
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${PVE_ISO_STORAGE}:iso/${seed_file}" &>/dev/null || true
;;
esac
detail "Deleted seed: ${seed_file}"
info "Deleted seed: ${seed_file}"
fi
i=$((i + 1))
done
# Remove IncusOS ISOs from Proxmox storage
# Remove only the specific IncusOS ISO used by this deployment
if [[ -n "$DEPLOYMENT_ISO" ]]; then
local volid="${PVE_ISO_STORAGE}:iso/${DEPLOYMENT_ISO}"
if [[ "$DRY_RUN" == true ]]; then
info "[dry-run] Would delete ${volid}"
else
case "$PVE_METHOD" in
ssh)
pve_run "pvesm free ${volid} 2>/dev/null" &>/dev/null || true
;;
api)
local encoded_volid
encoded_volid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${volid}', safe=''))" 2>/dev/null)
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true
;;
esac
info "Deleted ISO: ${DEPLOYMENT_ISO}"
fi
else
detail "No deployment ISO detected (VMs may already be destroyed)"
fi
success "Storage cleanup complete"
}
# Aggressive cleanup: remove ALL IncusOS ISOs and seed ISOs from storage
cleanup_storage_all() {
step "Deep cleanup: removing ALL IncusOS ISOs and seeds from Proxmox storage"
local iso_list=""
if [[ "$DRY_RUN" != true ]]; then
case "$PVE_METHOD" in
ssh)
iso_list=$(pve_run "pvesm list ${PVE_ISO_STORAGE} --content iso 2>/dev/null" 2>/dev/null | grep "IncusOS_" | awk '{print $1}') || iso_list=""
iso_list=$(pve_run "pvesm list ${PVE_ISO_STORAGE} --content iso 2>/dev/null" 2>/dev/null | grep -E "(IncusOS_|seed-)" | awk '{print $1}') || iso_list=""
;;
api)
iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null | \
@ -3248,7 +3630,7 @@ import json, sys
data = json.load(sys.stdin)
for item in data.get('data', []):
volid = item.get('volid', '')
if 'IncusOS_' in volid:
if 'IncusOS_' in volid or 'seed-' in volid:
print(volid)
" 2>/dev/null) || iso_list=""
;;
@ -3259,7 +3641,7 @@ for item in data.get('data', []):
while IFS= read -r volid; do
[[ -z "$volid" ]] && continue
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would delete ${volid}"
info "[dry-run] Would delete ${volid}"
else
case "$PVE_METHOD" in
ssh)
@ -3271,14 +3653,18 @@ for item in data.get('data', []):
pve_api DELETE "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content/${encoded_volid}" &>/dev/null || true
;;
esac
detail "Deleted ISO: ${volid}"
info "Deleted: ${volid}"
fi
done <<< "$iso_list"
else
info "No IncusOS ISOs or seeds found in storage"
fi
success "Storage cleanup complete"
}
# Remove incus remotes. Uses VM_NAMES (config-based cleanup) or
# DESTROYED_VM_NAMES (pool-wide cleanup) depending on what's populated.
cleanup_remotes() {
step "Deep cleanup: removing incus remotes"
@ -3287,27 +3673,40 @@ cleanup_remotes() {
return
fi
local count=${#VM_NAMES[@]}
local i=0
while [[ $i -lt $count ]]; do
local name="${VM_NAMES[$i]}"
if [[ -n "$VM_FILTER" ]] && [[ "$name" != "$VM_FILTER" ]]; then
# Determine which name list to use
local names_to_check=()
if [[ ${#DESTROYED_VM_NAMES[@]} -gt 0 ]]; then
# Pool-wide cleanup: use names of VMs actually destroyed
names_to_check=("${DESTROYED_VM_NAMES[@]}")
elif [[ ${#VM_NAMES[@]} -gt 0 ]]; then
# Config-based cleanup
local i=0
while [[ $i -lt ${#VM_NAMES[@]} ]]; do
if [[ -n "$VM_FILTER" ]] && [[ "${VM_NAMES[$i]}" != "$VM_FILTER" ]]; then
i=$((i + 1))
continue
fi
names_to_check+=("${VM_NAMES[$i]}")
i=$((i + 1))
continue
fi
done
fi
if [[ ${#names_to_check[@]} -eq 0 ]]; then
detail "No VM names to check for remote cleanup"
success "Remote cleanup complete"
return
fi
for name in "${names_to_check[@]}"; do
# Check if a remote with this name exists
if incus remote list --format csv 2>/dev/null | grep -q "^${name},"; then
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would remove incus remote '${name}'"
info "[dry-run] Would remove incus remote '${name}'"
else
incus remote remove "$name" 2>/dev/null || true
detail "Removed incus remote: ${name}"
info "Removed incus remote: ${name}"
fi
fi
i=$((i + 1))
done
success "Remote cleanup complete"
@ -3319,10 +3718,10 @@ cleanup_local_cache() {
local cache_base="${XDG_CACHE_HOME:-${HOME}/.cache}/${SCRIPT_NAME}/iso-cache"
if [[ -d "$cache_base" ]]; then
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would clear ${cache_base}"
info "[dry-run] Would clear ${cache_base}"
else
rm -rf "$cache_base"
detail "Cleared cache: ${cache_base}"
info "Cleared cache: ${cache_base}"
fi
else
detail "No local cache to clear"
@ -3331,13 +3730,17 @@ cleanup_local_cache() {
success "Local cache cleanup complete"
}
# Names of VMs destroyed during cleanup-all (used by cleanup_remotes)
declare -a DESTROYED_VM_NAMES=()
# Pool-wide cleanup: destroy ALL managed VMs in the resource pool
do_cleanup_all() {
step "Pool-wide cleanup: destroying all managed VMs"
echo ""
if [[ -z "$PVE_POOL" ]]; then
error "Pool-wide cleanup requires a resource pool (proxmox.pool) in the config"
error "Pool-wide cleanup requires a resource pool (proxmox.pool)"
error "Set pool in proxmox.yaml or pass a config with proxmox.pool"
error "Without a pool, there is no safe way to scope which VMs to destroy"
exit 1
fi
@ -3389,8 +3792,9 @@ do_cleanup_all() {
step "Destroying VM '${name}' (VMID ${vmid})"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would stop and destroy VM ${vmid}"
info "[dry-run] Would stop and destroy VM ${vmid}"
destroyed=$((destroyed + 1))
DESTROYED_VM_NAMES+=("$name")
idx=$((idx + 1))
continue
fi
@ -3398,7 +3802,7 @@ do_cleanup_all() {
local status
status=$(pve_vm_status "$vmid") || status="unknown"
if [[ "$status" == "running" ]]; then
detail "Stopping VM..."
info "Stopping VM..."
pve_stop_vm "$vmid" || true
sleep 5
fi
@ -3406,6 +3810,7 @@ do_cleanup_all() {
if pve_destroy_vm "$vmid"; then
success "VM '${name}' (VMID ${vmid}) destroyed"
destroyed=$((destroyed + 1))
DESTROYED_VM_NAMES+=("$name")
else
error "Failed to destroy VM '${name}' (VMID ${vmid})"
fi
@ -3546,6 +3951,40 @@ main() {
setup_workdir
# Handle cleanup-all without a config file (only needs proxmox.yaml)
if [[ "$CLEANUP_ALL" == true ]] && [[ -z "$CONFIG_FILE" ]]; then
if ! find_proxmox_config; then
error "No proxmox.yaml found and no config file specified"
error "Create proxmox.yaml or use --proxmox FILE"
exit 1
fi
load_proxmox_config "$PROXMOX_CONFIG_RESOLVED"
# Apply defaults for anything still unset
[[ -z "$PVE_NODE" ]] && PVE_NODE="pve"
[[ -z "$PVE_STORAGE" ]] && PVE_STORAGE="local-lvm"
[[ -z "$PVE_ISO_STORAGE" ]] && PVE_ISO_STORAGE="local"
[[ -z "$PVE_BRIDGE" ]] && PVE_BRIDGE="vmbr0"
[[ -z "$PVE_SSH_USER" ]] && PVE_SSH_USER="root"
[[ -z "$PVE_METHOD" ]] && PVE_METHOD="ssh"
[[ -n "$METHOD" ]] && PVE_METHOD="$METHOD"
[[ -n "$HOST_OVERRIDE" ]] && PVE_HOST="$HOST_OVERRIDE"
if [[ -z "$PVE_HOST" ]]; then
error "proxmox.yaml must contain 'host:'"
exit 1
fi
do_cleanup_all
if [[ "$DEEP_CLEANUP" == true ]]; then
echo ""
cleanup_storage_all
cleanup_remotes
cleanup_local_cache
fi
return
fi
if [[ "$DRY_RUN" != true ]]; then
check_dependencies
fi
@ -3554,12 +3993,12 @@ main() {
# Always run validate first
phase_validate
# Handle cleanup-all mode (pool-wide, not config-specific VMs)
# Handle cleanup-all mode (with config file for additional context)
if [[ "$CLEANUP_ALL" == true ]]; then
do_cleanup_all
if [[ "$DEEP_CLEANUP" == true ]]; then
echo ""
cleanup_storage
cleanup_storage_all
cleanup_remotes
cleanup_local_cache
fi
@ -3568,6 +4007,10 @@ main() {
# Handle cleanup mode
if [[ "$CLEANUP" == true ]]; then
if [[ "$DEEP_CLEANUP" == true ]]; then
# Detect which ISO was used before destroying VMs
detect_deployment_iso
fi
confirm "Destroy VMs listed above?"
do_cleanup
if [[ "$DEEP_CLEANUP" == true ]]; then

1020
notes/clustering-guide.md Normal file
View File

File diff suppressed because it is too large Load Diff