Safety fix: --cleanup-all, --resources, --labs silently failed when
PROXMOX_TOKEN_SECRET was not set, reporting "No VMs found" instead of
an auth error. Added require_api_auth() gate and error propagation in
pve_list_vms() to fail loudly on API auth failures.
Also:
- Standardize role name to IncusOSDeployer across all docs and scripts
- Add Sys.Audit privilege to all setup command examples
- Document /nodes/pve ACL requirement for scoped permissions
- Update lab-cluster.yaml: apply_defaults: false for joining nodes
(upstream IncusOS recommendation to avoid 8-command cleanup)
- Add full cluster automation to lab-test phase_cluster:
core.https_address setup, remote cert fixup, automated join prompts,
conditional defaults cleanup for apply_defaults: true nodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
lab-replace.yaml (single replacement node) for advanced migration and
cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
formation, storage pool conflict fix, join workflow, VM live migration
with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
cluster enable syntax fix (TWO args not one), cluster remove syntax,
agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements the first wave of lab validation infrastructure:
incusos-proxmox enhancements:
- --doctor flag: standalone environment health check (tool versions,
IncusOS CDN availability, Proxmox connectivity). Works with or
without a config file.
- --cleanup --deep: extended cleanup that also removes seed ISOs,
IncusOS ISOs from Proxmox storage, matching incus remotes, and
the local ISO cache (~/.cache/incusos-proxmox/iso-cache/).
- --cleanup-all: pool-wide cleanup that destroys ALL VMs with the
[incusos-lab:managed] marker in the configured resource pool.
Requires a pool to be set in config for safety scoping.
New script -- lab-test:
- Guided lab validation with 5 phases: deploy & verify, single-node
workloads (container + VM lifecycle), cluster formation (enable +
join via incus remotes), cluster workloads (scheduler-placed +
targeted), and migration & evacuation (stop/move + evacuate/restore).
- Interactive by default with --yes for unattended runs.
- Reports PASS/FAIL/SKIP per test with a summary.
- Supports --dry-run, --skip-deploy, --cleanup, per-phase execution.
- Reads the same YAML config format as incusos-proxmox.
New example configs:
- lab-cluster.yaml: 3-node Incus cluster (VMID 900+) for testing
clustering, workloads, and migration.
- lab-oc.yaml: Operations Center + 3 Incus nodes (VMID 910+) for
testing OC-managed provisioning and cluster formation.
Documentation:
- CLAUDE.md: added clustering workflow (incus cluster enable/add/join
via remotes), lab-test overview, Operations Center CLI notes,
doctor and cleanup feature descriptions.
- README.md: added lab-test section with usage/phases/options, doctor
check docs, deep cleanup docs, new example files in listing.
- TESTING.md: added full lab validation guide (quick start, phase-by-
phase testing, manual cluster formation alternative, OC lab workflow
with CLI setup and provisioning steps), cleanup instructions.
Not yet tested against live infrastructure -- next step is to deploy
lab-cluster.yaml, add incus remotes, and run lab-test end-to-end.
Known areas to validate: cluster join syntax with remotes, VM nested
virt support, evacuation behavior, apply_defaults consistency across
cluster members.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten