Fix critical auth validation, role naming, Sys.Audit, and apply_defaults

Safety fix: --cleanup-all, --resources, --labs silently failed when
PROXMOX_TOKEN_SECRET was not set, reporting "No VMs found" instead of
an auth error. Added require_api_auth() gate and error propagation in
pve_list_vms() to fail loudly on API auth failures.

Also:
- Standardize role name to IncusOSDeployer across all docs and scripts
- Add Sys.Audit privilege to all setup command examples
- Document /nodes/pve ACL requirement for scoped permissions
- Update lab-cluster.yaml: apply_defaults: false for joining nodes
  (upstream IncusOS recommendation to avoid 8-command cleanup)
- Add full cluster automation to lab-test phase_cluster:
  core.https_address setup, remote cert fixup, automated join prompts,
  conditional defaults cleanup for apply_defaults: true nodes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -222,16 +222,23 @@ incu-contrib/
 - The cert trust on the server side (client → server) is unaffected -- it's
   stored by fingerprint in the Incus database, independent of listen address.
 
-#### Joining nodes: the apply_defaults storage pool conflict
+#### Joining nodes: apply_defaults and the storage pool conflict
 
-- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0`
+- **Upstream recommendation**: use `apply_defaults: false` for nodes destined
-  network bridge on each node at install time. When joining a cluster that
+  to join a cluster. The official IncusOS clustering tutorial states joining
-  already has a `local` pool defined, the join wizard asks for `source` and
+  servers "cannot have preexisting networks or storage pools defined." With
-  `zfs.pool_name` properties but then tries to set them as cluster-wide config.
+  `apply_defaults: false`, the node still listens on port 8443, still trusts
-  Incus rejects this because they are **member-specific** keys.
+  preseed certificates, and the underlying ZFS dataset (`local/incus`) still
-- **The fix**: delete the storage pool and network on the joining node before
+  exists -- but no Incus storage pool or network metadata is created, so
-  joining. The join process then creates the member-specific pool entry
+  the join process works cleanly.
-  correctly.
+- **`apply_defaults: true` on joining nodes** is also functional but requires
+  an 8-command cleanup per node before join (delete pool, network, volumes,
+  profile devices). This is automated in `lab-test` but adds complexity.
+- **Recommended seed pattern for clusters**:
+  - Bootstrap/init node: `apply_defaults: true` (needs pool and network)
+  - Joining nodes: `apply_defaults: false` (join process creates member-specific entries)
+  - Standalone nodes: `apply_defaults: true` (needs pool and network to be functional)
+- **If `apply_defaults: true` was used**, the cleanup before join is:
   ```bash
   # 1. Remove config references
   incus config unset <remote>: storage.backups_volume
@@ -425,7 +432,8 @@ incu-contrib/
   application → Terraform config. Adds `meshbr0` network.
 - **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`,
   nodes already have storage pool/network/cert. OC's Terraform fails with
-  "already exists" errors but the cluster forms successfully.
+  "already exists" errors but the cluster forms successfully. Use
+  `apply_defaults: false` for OC-managed nodes to avoid this.
 
 #### Tested limitations
 

incusos/README.md

@@ -434,12 +434,12 @@ ssh root@192.168.1.10 pvesh get /version
 **API:** Requires an API token. Set up minimal privileges:
 
 ```bash
-pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
+pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \
     VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
     VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
-    Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use"
+    Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use Sys.Audit"
 pveum user add automation@pve
-pveum aclmod / -user automation@pve -role IncusOSDeploy
+pveum aclmod / -user automation@pve -role IncusOSDeployer
 pveum user token add automation@pve deploy --privsep 0
 
 export PROXMOX_TOKEN_SECRET="<secret-from-above>"
@@ -454,7 +454,7 @@ token to it instead of `/`:
 # On Proxmox (one-time setup)
 pveum pool add IncusLab --comment "IncusOS Lab VMs"
 pveum pool modify IncusLab --storage local-lvm,local
-pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
+pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
 ```
 
 Then in your config file:

incusos/TESTING.md

@@ -67,20 +67,21 @@ required by at least one operation in the script:
 | `Datastore.AllocateTemplate` | Upload ISOs to storage |
 | `Datastore.Audit` | List and check storage pools |
 | `SDN.Use` | Use network bridges |
+| `Sys.Audit` | Query node status (RAM, CPU, uptime for `--resources`) |
 
 None of these can be removed without breaking functionality.
 
 #### Setup Commands (Run on Proxmox)
 
 ```bash
-pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
+pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \
     VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
     VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
     Datastore.AllocateSpace Datastore.AllocateTemplate \
-    Datastore.Audit SDN.Use"
+    Datastore.Audit SDN.Use Sys.Audit"
 
 pveum user add automation@pve
-pveum aclmod / -user automation@pve -role IncusOSDeploy
+pveum aclmod / -user automation@pve -role IncusOSDeployer
 pveum user token add automation@pve deploy --privsep 0
 # Save the displayed secret!
 ```
@@ -90,9 +91,16 @@ pveum user token add automation@pve deploy --privsep 0
 For tighter access control, scope the ACL to specific paths instead of `/`:
 
 ```bash
-pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
+pveum aclmod /storage/local -user automation@pve -role IncusOSDeployer
-pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
+pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeployer
-pveum aclmod /vms -user automation@pve -role IncusOSDeploy
+pveum aclmod /vms -user automation@pve -role IncusOSDeployer
+```
+
+**Note:** `Sys.Audit` requires an ACL on the node path, not covered by pool
+or storage scopes:
+
+```bash
+pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
 ```
 
 For a home lab, `/` (root) is simpler and perfectly acceptable.
@@ -286,13 +294,13 @@ After single-VM success with SSH, switch to API tokens.
 **On Proxmox** (one-time setup via SSH or web shell):
 
 ```bash
-pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
+pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \
     VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
     VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
     Datastore.AllocateSpace Datastore.AllocateTemplate \
-    Datastore.Audit SDN.Use"
+    Datastore.Audit SDN.Use Sys.Audit"
 pveum user add automation@pve
-pveum aclmod / -user automation@pve -role IncusOSDeploy
+pveum aclmod / -user automation@pve -role IncusOSDeployer
 pveum user token add automation@pve deploy --privsep 0
 # ^^^ Save the displayed secret!
 ```
@@ -323,7 +331,9 @@ For API method with pool isolation:
 ```bash
 pveum pool add IncusLab --comment "IncusOS Lab VMs"
 pveum pool modify IncusLab --storage local-lvm,local
-pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
+pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
+# For --resources (Sys.Audit on node path):
+pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
 ```
 
 Add to your config:

incusos/examples/lab-cluster.yaml

@@ -1,8 +1,10 @@
 # lab-cluster.yaml - 3-node IncusOS cluster for lab testing
 #
 # Deploys 3 standalone Incus nodes that can be clustered after deployment.
-# All nodes get apply_defaults: true so they have matching storage pool
+# The init node gets apply_defaults: true (storage pool + network bridge).
-# and network bridge configurations.
+# Joining nodes use apply_defaults: false -- the cluster join process creates
+# member-specific pool entries. This follows the upstream IncusOS recommendation
+# and avoids the 8-command cleanup per joining node.
 #
 # Usage:
 #   incusos-proxmox --dry-run examples/lab-cluster.yaml   # Preview
@@ -26,12 +28,12 @@ defaults:
 vms:
   - name: incus-lab-01
     app: incus
-    apply_defaults: true
+    apply_defaults: true    # init node: needs storage pool + network
 
   - name: incus-lab-02
     app: incus
-    apply_defaults: true
+    apply_defaults: false   # joining node: cluster join creates pool entry
 
   - name: incus-lab-03
     app: incus
-    apply_defaults: true
+    apply_defaults: false   # joining node: cluster join creates pool entry

incusos/incusos-proxmox

@@ -196,19 +196,19 @@ ${BOLD}PROXMOX SETUP${RESET}
     ${BOLD}API method:${RESET}
       Requires an API token. Set up minimal privileges:
 
-        pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \\
+        pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \\
             VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \\
             VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \\
             Datastore.AllocateSpace Datastore.AllocateTemplate \\
-            Datastore.Audit SDN.Use"
+            Datastore.Audit SDN.Use Sys.Audit"
         pveum user add automation@pve
-        pveum aclmod / -user automation@pve -role IncusOSDeploy
+        pveum aclmod / -user automation@pve -role IncusOSDeployer
         pveum user token add automation@pve deploy --privsep 0
 
       For resource pool isolation (recommended):
         pveum pool add IncusLab --comment "IncusOS Lab VMs"
         pveum pool modify IncusLab --storage local-lvm,local
-        pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
+        pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
 
       Set the token secret:
         export PROXMOX_TOKEN_SECRET="<secret-from-above>"
@@ -1576,6 +1576,34 @@ pve_run() {
     esac
 }
 
+# Validate API auth prerequisites and connectivity.
+# Call this before any code path that uses pve_api() outside of phase_validate().
+# Exits with error if credentials are missing or API is unreachable.
+require_api_auth() {
+    if [[ "$PVE_METHOD" != "api" ]]; then
+        return 0
+    fi
+    if [[ -z "$PVE_API_TOKEN_ID" ]]; then
+        error "API token ID is required for API method"
+        error "Set proxmox.api_token_id in proxmox.yaml"
+        exit 1
+    fi
+    if [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]]; then
+        error "PROXMOX_TOKEN_SECRET environment variable is not set"
+        error "Source your env file:  source env"
+        error "Or set it directly:    export PROXMOX_TOKEN_SECRET=\"<your-token-secret>\""
+        exit 1
+    fi
+    # Verify we can actually reach the API with these credentials
+    local url="https://${PVE_HOST}:8006/api2/json/version"
+    local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
+    if ! curl -fsSk --connect-timeout 5 -H "$auth" "$url" &>/dev/null; then
+        error "Cannot authenticate to Proxmox API at https://${PVE_HOST}:8006"
+        error "Check that the host is reachable, token ID is correct, and secret is valid"
+        exit 1
+    fi
+}
+
 # Execute a Proxmox API call (API method)
 # Usage: pve_api METHOD ENDPOINT [DATA] [json]
 # Pass "json" as 4th arg to send data as application/json instead of form-encoded.
@@ -2175,18 +2203,24 @@ pve_vm_has_install_media() {
 }
 
 # List all VMs as "vmid name" pairs (one per line)
+# Returns non-zero and prints error message on API/SSH failure.
 pve_list_vms() {
     if [[ "$DRY_RUN" == true ]]; then
         echo ""
         return 0
     fi
 
+    local raw_response
+
     # When a resource pool is configured, list only pool members (type=qemu)
     if [[ -n "$PVE_POOL" ]]; then
         case "$PVE_METHOD" in
             ssh)
-                pve_run "pvesh get /pools/${PVE_POOL} --output-format json 2>/dev/null" | \
+                if ! raw_response=$(pve_run "pvesh get /pools/${PVE_POOL} --output-format json 2>/dev/null" 2>/dev/null); then
-                    python3 -c "
+                    echo "[error] Failed to list VMs via SSH" >&2
+                    return 1
+                fi
+                echo "$raw_response" | python3 -c "
 import json, sys
 data = json.load(sys.stdin)
 for m in data.get('members', []):
@@ -2195,8 +2229,11 @@ for m in data.get('members', []):
 " 2>/dev/null || true
                 ;;
             api)
-                pve_api GET "/pools/${PVE_POOL}" 2>/dev/null | \
+                if ! raw_response=$(pve_api GET "/pools/${PVE_POOL}" 2>/dev/null); then
-                    python3 -c "
+                    echo "[error] Failed to list VMs from pool '${PVE_POOL}' via API (auth failure or pool not found)" >&2
+                    return 1
+                fi
+                echo "$raw_response" | python3 -c "
 import json, sys
 data = json.load(sys.stdin).get('data', {})
 for m in data.get('members', []):
@@ -2213,8 +2250,11 @@ for m in data.get('members', []):
             pve_run "qm list 2>/dev/null" | awk 'NR>1 {print $1, $2}' || true
             ;;
         api)
-            pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null | \
+            if ! raw_response=$(pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null); then
-                python3 -c "
+                echo "[error] Failed to list VMs via API (auth failure or node not found)" >&2
+                return 1
+            fi
+            echo "$raw_response" | python3 -c "
 import json, sys
 data = json.load(sys.stdin)
 for vm in data.get('data', []):
@@ -2351,7 +2391,7 @@ for s in data.get('data', []):
             if [[ "$PVE_METHOD" == "api" ]] && [[ -n "$PVE_API_TOKEN_ID" ]]; then
                 local api_user
                 api_user=$(echo "$PVE_API_TOKEN_ID" | cut -d'!' -f1)
-                error "  pveum aclmod /pool/${PVE_POOL} -user ${api_user} -role IncusOSDeploy"
+                error "  pveum aclmod /pool/${PVE_POOL} -user ${api_user} -role IncusOSDeployer"
             fi
             errors=$((errors + 1))
         else
@@ -4156,7 +4196,11 @@ do_cleanup_all() {
 
     # Get all VMs in the pool
     local vm_list
-    vm_list=$(pve_list_vms)
+    if ! vm_list=$(pve_list_vms); then
+        error "Failed to list VMs in pool '${PVE_POOL}'"
+        error "Check Proxmox connectivity and API credentials"
+        exit 1
+    fi
 
     if [[ -z "$vm_list" ]]; then
         info "No VMs found in pool '${PVE_POOL}'"
@@ -4383,6 +4427,7 @@ main() {
             exit 1
         fi
 
+        require_api_auth
         [[ "$RESOURCES" == true ]] && do_resources
         [[ "$LIST_LABS" == true ]] && do_list_labs
         return 0
@@ -4412,6 +4457,7 @@ main() {
             exit 1
         fi
 
+        require_api_auth
         do_cleanup_all
         if [[ "$DEEP_CLEANUP" == true ]]; then
             echo ""

incusos/lab-test

@@ -612,13 +612,50 @@ phase_cluster() {
         fi
     fi
 
-    # Step 1: Enable clustering on init node
+    # Step 1: Set core.https_address on all nodes (required for clustering)
+    step "Setting core.https_address on all nodes"
+    local -A NODE_IPS=()
+    if [[ "$DRY_RUN" == true ]]; then
+        detail "[dry-run] Would query and set core.https_address on all nodes"
+    else
+        local n=0
+        while [[ $n -lt ${#VM_NAMES[@]} ]]; do
+            local node="${VM_NAMES[$n]}"
+            local node_ip
+            node_ip=$(incus query "${node}:/1.0" 2>/dev/null | \
+                python3 -c "
+import sys, json
+d = json.load(sys.stdin)
+for a in d.get('environment', {}).get('addresses', []):
+    if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('['):
+        print(a)
+        break
+" 2>/dev/null) || node_ip=""
+            if [[ -z "$node_ip" ]]; then
+                fail "Cannot determine IP for ${node}"
+                return 1
+            fi
+            NODE_IPS["$node"]="$node_ip"
+            incus config set "${node}:" core.https_address "${node_ip}:8443" 2>/dev/null
+            detail "${node}: ${node_ip}:8443"
+            n=$((n + 1))
+        done
+        pass "core.https_address set on all nodes"
+    fi
+    echo ""
+
+    # Step 2: Enable clustering on init node
     step "Enabling clustering on ${init_node}"
     if [[ "$DRY_RUN" == true ]]; then
         detail "[dry-run] incus cluster enable ${init_node}: ${init_node}"
     else
         if incus cluster enable "${init_node}:" "${init_node}" 2>/dev/null; then
             pass "Clustering enabled on ${init_node}"
+            # Fix remote (cluster enable regenerates TLS cert)
+            incus remote remove "${init_node}" 2>/dev/null || true
+            incus remote add "${init_node}" "https://${NODE_IPS[$init_node]}:8443" \
+                --accept-certificate 2>/dev/null || true
+            detail "Remote ${init_node} re-added with new cluster cert"
         else
             fail "Failed to enable clustering on ${init_node}"
             return 1
@@ -626,7 +663,7 @@ phase_cluster() {
     fi
     echo ""
 
-    # Step 2-N: Join remaining nodes
+    # Step 3-N: Join remaining nodes
     local i=1
     while [[ $i -lt ${#VM_NAMES[@]} ]]; do
         local join_node="${VM_NAMES[$i]}"
@@ -644,6 +681,21 @@ phase_cluster() {
                 continue
             fi
 
+            # Check if joining node has storage pool (apply_defaults: true)
+            # and clean up if needed for cluster join
+            if incus storage show "${join_node}:local" &>/dev/null 2>&1; then
+                detail "Cleaning up defaults on ${join_node} for cluster join"
+                incus config unset "${join_node}:" storage.backups_volume 2>/dev/null || true
+                incus config unset "${join_node}:" storage.images_volume 2>/dev/null || true
+                incus storage volume delete "${join_node}:local" backups 2>/dev/null || true
+                incus storage volume delete "${join_node}:local" images 2>/dev/null || true
+                incus profile device remove "${join_node}:default" root 2>/dev/null || true
+                incus profile device remove "${join_node}:default" eth0 2>/dev/null || true
+                incus storage delete "${join_node}:local" 2>/dev/null || true
+                incus network delete "${join_node}:incusbr0" 2>/dev/null || true
+                detail "Defaults cleaned up"
+            fi
+
             # Generate join token
             local token
             token=$(incus cluster add "${init_node}:" "${join_node}" 2>/dev/null) || token=""
@@ -654,9 +706,15 @@ phase_cluster() {
             fi
             detail "Join token generated"
 
-            # Join the cluster
+            # Join the cluster (automated: accept defaults, answer prompts)
-            if incus cluster join "${init_node}:" "${join_node}:" 2>/dev/null; then
+            if printf '\n\nyes\nlocal/incus\nlocal/incus\n' | \
+                incus cluster join "${init_node}:" "${join_node}:" 2>/dev/null; then
                 pass "Node ${join_node} joined the cluster"
+                # Fix remote (join regenerates TLS cert)
+                incus remote remove "${join_node}" 2>/dev/null || true
+                incus remote add "${join_node}" "https://${NODE_IPS[$join_node]}:8443" \
+                    --accept-certificate 2>/dev/null || true
+                detail "Remote ${join_node} re-added with cluster cert"
             else
                 fail "Failed to join ${join_node} to cluster"
             fi

notes/clustering-guide.md

@@ -28,7 +28,7 @@ With 3 nodes, you get a fully HA database quorum.
 ## Prerequisites
 
 - Incus client version **6.20+** (required for `incus cluster join` via remotes)
-- All nodes deployed with `apply_defaults: true` (matching pool/network names)
+- Init node deployed with `apply_defaults: true`; joining nodes with `apply_defaults: false` (recommended) or `true` (requires cleanup before join)
 - Remotes configured for all nodes (`incus remote add <name> https://<ip>:8443`)
 - Client certificate trusted on all nodes (injected via seed or added manually)
 
@@ -147,9 +147,34 @@ Expected output: one member (`incus-lab-01`), status `ONLINE`, role
 
 ## Step 3: Prepare Joining Nodes
 
-### The apply_defaults storage pool conflict
+### Recommended: deploy joining nodes with apply_defaults: false
 
-Nodes deployed with `apply_defaults: true` already have:
+The upstream IncusOS clustering tutorial recommends deploying joining nodes
+**without** default settings. In the YAML config:
+
+```yaml
+vms:
+  - name: incus-lab-01
+    app: incus
+    apply_defaults: true    # init node: needs storage pool + network
+
+  - name: incus-lab-02
+    app: incus
+    apply_defaults: false   # joining node
+
+  - name: incus-lab-03
+    app: incus
+    apply_defaults: false   # joining node
+```
+
+With `apply_defaults: false`, the node still listens on port 8443, trusts
+preseed certificates, and the underlying ZFS dataset (`local/incus`) exists.
+But no Incus storage pool or network metadata is created, so the join process
+works cleanly without cleanup. Skip to Step 4 if using this approach.
+
+### Alternative: cleanup when apply_defaults: true was used
+
+If joining nodes were deployed with `apply_defaults: true`, they already have:
 - A `local` ZFS storage pool (source: `local/incus`)
 - An `incusbr0` network bridge
 - A `default` profile referencing both