diff --git a/CLAUDE.md b/CLAUDE.md index 4bab33a..4f65eba 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -222,16 +222,23 @@ incu-contrib/ - The cert trust on the server side (client → server) is unaffected -- it's stored by fingerprint in the Incus database, independent of listen address. -#### Joining nodes: the apply_defaults storage pool conflict +#### Joining nodes: apply_defaults and the storage pool conflict -- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0` - network bridge on each node at install time. When joining a cluster that - already has a `local` pool defined, the join wizard asks for `source` and - `zfs.pool_name` properties but then tries to set them as cluster-wide config. - Incus rejects this because they are **member-specific** keys. -- **The fix**: delete the storage pool and network on the joining node before - joining. The join process then creates the member-specific pool entry - correctly. +- **Upstream recommendation**: use `apply_defaults: false` for nodes destined + to join a cluster. The official IncusOS clustering tutorial states joining + servers "cannot have preexisting networks or storage pools defined." With + `apply_defaults: false`, the node still listens on port 8443, still trusts + preseed certificates, and the underlying ZFS dataset (`local/incus`) still + exists -- but no Incus storage pool or network metadata is created, so + the join process works cleanly. +- **`apply_defaults: true` on joining nodes** is also functional but requires + an 8-command cleanup per node before join (delete pool, network, volumes, + profile devices). This is automated in `lab-test` but adds complexity. +- **Recommended seed pattern for clusters**: + - Bootstrap/init node: `apply_defaults: true` (needs pool and network) + - Joining nodes: `apply_defaults: false` (join process creates member-specific entries) + - Standalone nodes: `apply_defaults: true` (needs pool and network to be functional) +- **If `apply_defaults: true` was used**, the cleanup before join is: ```bash # 1. Remove config references incus config unset : storage.backups_volume @@ -425,7 +432,8 @@ incu-contrib/ application → Terraform config. Adds `meshbr0` network. - **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`, nodes already have storage pool/network/cert. OC's Terraform fails with - "already exists" errors but the cluster forms successfully. + "already exists" errors but the cluster forms successfully. Use + `apply_defaults: false` for OC-managed nodes to avoid this. #### Tested limitations diff --git a/incusos/README.md b/incusos/README.md index 8c95b74..2dabc3d 100644 --- a/incusos/README.md +++ b/incusos/README.md @@ -434,12 +434,12 @@ ssh root@192.168.1.10 pvesh get /version **API:** Requires an API token. Set up minimal privileges: ```bash -pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \ +pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \ VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \ VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \ - Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use" + Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use Sys.Audit" pveum user add automation@pve -pveum aclmod / -user automation@pve -role IncusOSDeploy +pveum aclmod / -user automation@pve -role IncusOSDeployer pveum user token add automation@pve deploy --privsep 0 export PROXMOX_TOKEN_SECRET="" @@ -454,7 +454,7 @@ token to it instead of `/`: # On Proxmox (one-time setup) pveum pool add IncusLab --comment "IncusOS Lab VMs" pveum pool modify IncusLab --storage local-lvm,local -pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy +pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer ``` Then in your config file: diff --git a/incusos/TESTING.md b/incusos/TESTING.md index 3301f47..727952d 100644 --- a/incusos/TESTING.md +++ b/incusos/TESTING.md @@ -67,20 +67,21 @@ required by at least one operation in the script: | `Datastore.AllocateTemplate` | Upload ISOs to storage | | `Datastore.Audit` | List and check storage pools | | `SDN.Use` | Use network bridges | +| `Sys.Audit` | Query node status (RAM, CPU, uptime for `--resources`) | None of these can be removed without breaking functionality. #### Setup Commands (Run on Proxmox) ```bash -pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \ +pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \ VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \ VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \ Datastore.AllocateSpace Datastore.AllocateTemplate \ - Datastore.Audit SDN.Use" + Datastore.Audit SDN.Use Sys.Audit" pveum user add automation@pve -pveum aclmod / -user automation@pve -role IncusOSDeploy +pveum aclmod / -user automation@pve -role IncusOSDeployer pveum user token add automation@pve deploy --privsep 0 # Save the displayed secret! ``` @@ -90,9 +91,16 @@ pveum user token add automation@pve deploy --privsep 0 For tighter access control, scope the ACL to specific paths instead of `/`: ```bash -pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy -pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy -pveum aclmod /vms -user automation@pve -role IncusOSDeploy +pveum aclmod /storage/local -user automation@pve -role IncusOSDeployer +pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeployer +pveum aclmod /vms -user automation@pve -role IncusOSDeployer +``` + +**Note:** `Sys.Audit` requires an ACL on the node path, not covered by pool +or storage scopes: + +```bash +pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0 ``` For a home lab, `/` (root) is simpler and perfectly acceptable. @@ -286,13 +294,13 @@ After single-VM success with SSH, switch to API tokens. **On Proxmox** (one-time setup via SSH or web shell): ```bash -pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \ +pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \ VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \ VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \ Datastore.AllocateSpace Datastore.AllocateTemplate \ - Datastore.Audit SDN.Use" + Datastore.Audit SDN.Use Sys.Audit" pveum user add automation@pve -pveum aclmod / -user automation@pve -role IncusOSDeploy +pveum aclmod / -user automation@pve -role IncusOSDeployer pveum user token add automation@pve deploy --privsep 0 # ^^^ Save the displayed secret! ``` @@ -323,7 +331,9 @@ For API method with pool isolation: ```bash pveum pool add IncusLab --comment "IncusOS Lab VMs" pveum pool modify IncusLab --storage local-lvm,local -pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy +pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer +# For --resources (Sys.Audit on node path): +pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0 ``` Add to your config: diff --git a/incusos/examples/lab-cluster.yaml b/incusos/examples/lab-cluster.yaml index 5684106..ec580c4 100644 --- a/incusos/examples/lab-cluster.yaml +++ b/incusos/examples/lab-cluster.yaml @@ -1,8 +1,10 @@ # lab-cluster.yaml - 3-node IncusOS cluster for lab testing # # Deploys 3 standalone Incus nodes that can be clustered after deployment. -# All nodes get apply_defaults: true so they have matching storage pool -# and network bridge configurations. +# The init node gets apply_defaults: true (storage pool + network bridge). +# Joining nodes use apply_defaults: false -- the cluster join process creates +# member-specific pool entries. This follows the upstream IncusOS recommendation +# and avoids the 8-command cleanup per joining node. # # Usage: # incusos-proxmox --dry-run examples/lab-cluster.yaml # Preview @@ -26,12 +28,12 @@ defaults: vms: - name: incus-lab-01 app: incus - apply_defaults: true + apply_defaults: true # init node: needs storage pool + network - name: incus-lab-02 app: incus - apply_defaults: true + apply_defaults: false # joining node: cluster join creates pool entry - name: incus-lab-03 app: incus - apply_defaults: true + apply_defaults: false # joining node: cluster join creates pool entry diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index 5ffd1bf..9629143 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -196,19 +196,19 @@ ${BOLD}PROXMOX SETUP${RESET} ${BOLD}API method:${RESET} Requires an API token. Set up minimal privileges: - pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \\ + pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \\ VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \\ VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \\ Datastore.AllocateSpace Datastore.AllocateTemplate \\ - Datastore.Audit SDN.Use" + Datastore.Audit SDN.Use Sys.Audit" pveum user add automation@pve - pveum aclmod / -user automation@pve -role IncusOSDeploy + pveum aclmod / -user automation@pve -role IncusOSDeployer pveum user token add automation@pve deploy --privsep 0 For resource pool isolation (recommended): pveum pool add IncusLab --comment "IncusOS Lab VMs" pveum pool modify IncusLab --storage local-lvm,local - pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy + pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer Set the token secret: export PROXMOX_TOKEN_SECRET="" @@ -1576,6 +1576,34 @@ pve_run() { esac } +# Validate API auth prerequisites and connectivity. +# Call this before any code path that uses pve_api() outside of phase_validate(). +# Exits with error if credentials are missing or API is unreachable. +require_api_auth() { + if [[ "$PVE_METHOD" != "api" ]]; then + return 0 + fi + if [[ -z "$PVE_API_TOKEN_ID" ]]; then + error "API token ID is required for API method" + error "Set proxmox.api_token_id in proxmox.yaml" + exit 1 + fi + if [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]]; then + error "PROXMOX_TOKEN_SECRET environment variable is not set" + error "Source your env file: source env" + error "Or set it directly: export PROXMOX_TOKEN_SECRET=\"\"" + exit 1 + fi + # Verify we can actually reach the API with these credentials + local url="https://${PVE_HOST}:8006/api2/json/version" + local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}" + if ! curl -fsSk --connect-timeout 5 -H "$auth" "$url" &>/dev/null; then + error "Cannot authenticate to Proxmox API at https://${PVE_HOST}:8006" + error "Check that the host is reachable, token ID is correct, and secret is valid" + exit 1 + fi +} + # Execute a Proxmox API call (API method) # Usage: pve_api METHOD ENDPOINT [DATA] [json] # Pass "json" as 4th arg to send data as application/json instead of form-encoded. @@ -2175,18 +2203,24 @@ pve_vm_has_install_media() { } # List all VMs as "vmid name" pairs (one per line) +# Returns non-zero and prints error message on API/SSH failure. pve_list_vms() { if [[ "$DRY_RUN" == true ]]; then echo "" return 0 fi + local raw_response + # When a resource pool is configured, list only pool members (type=qemu) if [[ -n "$PVE_POOL" ]]; then case "$PVE_METHOD" in ssh) - pve_run "pvesh get /pools/${PVE_POOL} --output-format json 2>/dev/null" | \ - python3 -c " + if ! raw_response=$(pve_run "pvesh get /pools/${PVE_POOL} --output-format json 2>/dev/null" 2>/dev/null); then + echo "[error] Failed to list VMs via SSH" >&2 + return 1 + fi + echo "$raw_response" | python3 -c " import json, sys data = json.load(sys.stdin) for m in data.get('members', []): @@ -2195,8 +2229,11 @@ for m in data.get('members', []): " 2>/dev/null || true ;; api) - pve_api GET "/pools/${PVE_POOL}" 2>/dev/null | \ - python3 -c " + if ! raw_response=$(pve_api GET "/pools/${PVE_POOL}" 2>/dev/null); then + echo "[error] Failed to list VMs from pool '${PVE_POOL}' via API (auth failure or pool not found)" >&2 + return 1 + fi + echo "$raw_response" | python3 -c " import json, sys data = json.load(sys.stdin).get('data', {}) for m in data.get('members', []): @@ -2213,8 +2250,11 @@ for m in data.get('members', []): pve_run "qm list 2>/dev/null" | awk 'NR>1 {print $1, $2}' || true ;; api) - pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null | \ - python3 -c " + if ! raw_response=$(pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null); then + echo "[error] Failed to list VMs via API (auth failure or node not found)" >&2 + return 1 + fi + echo "$raw_response" | python3 -c " import json, sys data = json.load(sys.stdin) for vm in data.get('data', []): @@ -2351,7 +2391,7 @@ for s in data.get('data', []): if [[ "$PVE_METHOD" == "api" ]] && [[ -n "$PVE_API_TOKEN_ID" ]]; then local api_user api_user=$(echo "$PVE_API_TOKEN_ID" | cut -d'!' -f1) - error " pveum aclmod /pool/${PVE_POOL} -user ${api_user} -role IncusOSDeploy" + error " pveum aclmod /pool/${PVE_POOL} -user ${api_user} -role IncusOSDeployer" fi errors=$((errors + 1)) else @@ -4156,7 +4196,11 @@ do_cleanup_all() { # Get all VMs in the pool local vm_list - vm_list=$(pve_list_vms) + if ! vm_list=$(pve_list_vms); then + error "Failed to list VMs in pool '${PVE_POOL}'" + error "Check Proxmox connectivity and API credentials" + exit 1 + fi if [[ -z "$vm_list" ]]; then info "No VMs found in pool '${PVE_POOL}'" @@ -4383,6 +4427,7 @@ main() { exit 1 fi + require_api_auth [[ "$RESOURCES" == true ]] && do_resources [[ "$LIST_LABS" == true ]] && do_list_labs return 0 @@ -4412,6 +4457,7 @@ main() { exit 1 fi + require_api_auth do_cleanup_all if [[ "$DEEP_CLEANUP" == true ]]; then echo "" diff --git a/incusos/lab-test b/incusos/lab-test index 8899d03..fcba83e 100755 --- a/incusos/lab-test +++ b/incusos/lab-test @@ -612,13 +612,50 @@ phase_cluster() { fi fi - # Step 1: Enable clustering on init node + # Step 1: Set core.https_address on all nodes (required for clustering) + step "Setting core.https_address on all nodes" + local -A NODE_IPS=() + if [[ "$DRY_RUN" == true ]]; then + detail "[dry-run] Would query and set core.https_address on all nodes" + else + local n=0 + while [[ $n -lt ${#VM_NAMES[@]} ]]; do + local node="${VM_NAMES[$n]}" + local node_ip + node_ip=$(incus query "${node}:/1.0" 2>/dev/null | \ + python3 -c " +import sys, json +d = json.load(sys.stdin) +for a in d.get('environment', {}).get('addresses', []): + if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('['): + print(a) + break +" 2>/dev/null) || node_ip="" + if [[ -z "$node_ip" ]]; then + fail "Cannot determine IP for ${node}" + return 1 + fi + NODE_IPS["$node"]="$node_ip" + incus config set "${node}:" core.https_address "${node_ip}:8443" 2>/dev/null + detail "${node}: ${node_ip}:8443" + n=$((n + 1)) + done + pass "core.https_address set on all nodes" + fi + echo "" + + # Step 2: Enable clustering on init node step "Enabling clustering on ${init_node}" if [[ "$DRY_RUN" == true ]]; then detail "[dry-run] incus cluster enable ${init_node}: ${init_node}" else if incus cluster enable "${init_node}:" "${init_node}" 2>/dev/null; then pass "Clustering enabled on ${init_node}" + # Fix remote (cluster enable regenerates TLS cert) + incus remote remove "${init_node}" 2>/dev/null || true + incus remote add "${init_node}" "https://${NODE_IPS[$init_node]}:8443" \ + --accept-certificate 2>/dev/null || true + detail "Remote ${init_node} re-added with new cluster cert" else fail "Failed to enable clustering on ${init_node}" return 1 @@ -626,7 +663,7 @@ phase_cluster() { fi echo "" - # Step 2-N: Join remaining nodes + # Step 3-N: Join remaining nodes local i=1 while [[ $i -lt ${#VM_NAMES[@]} ]]; do local join_node="${VM_NAMES[$i]}" @@ -644,6 +681,21 @@ phase_cluster() { continue fi + # Check if joining node has storage pool (apply_defaults: true) + # and clean up if needed for cluster join + if incus storage show "${join_node}:local" &>/dev/null 2>&1; then + detail "Cleaning up defaults on ${join_node} for cluster join" + incus config unset "${join_node}:" storage.backups_volume 2>/dev/null || true + incus config unset "${join_node}:" storage.images_volume 2>/dev/null || true + incus storage volume delete "${join_node}:local" backups 2>/dev/null || true + incus storage volume delete "${join_node}:local" images 2>/dev/null || true + incus profile device remove "${join_node}:default" root 2>/dev/null || true + incus profile device remove "${join_node}:default" eth0 2>/dev/null || true + incus storage delete "${join_node}:local" 2>/dev/null || true + incus network delete "${join_node}:incusbr0" 2>/dev/null || true + detail "Defaults cleaned up" + fi + # Generate join token local token token=$(incus cluster add "${init_node}:" "${join_node}" 2>/dev/null) || token="" @@ -654,9 +706,15 @@ phase_cluster() { fi detail "Join token generated" - # Join the cluster - if incus cluster join "${init_node}:" "${join_node}:" 2>/dev/null; then + # Join the cluster (automated: accept defaults, answer prompts) + if printf '\n\nyes\nlocal/incus\nlocal/incus\n' | \ + incus cluster join "${init_node}:" "${join_node}:" 2>/dev/null; then pass "Node ${join_node} joined the cluster" + # Fix remote (join regenerates TLS cert) + incus remote remove "${join_node}" 2>/dev/null || true + incus remote add "${join_node}" "https://${NODE_IPS[$join_node]}:8443" \ + --accept-certificate 2>/dev/null || true + detail "Remote ${join_node} re-added with cluster cert" else fail "Failed to join ${join_node} to cluster" fi diff --git a/notes/clustering-guide.md b/notes/clustering-guide.md index 9f8e666..4245613 100644 --- a/notes/clustering-guide.md +++ b/notes/clustering-guide.md @@ -28,7 +28,7 @@ With 3 nodes, you get a fully HA database quorum. ## Prerequisites - Incus client version **6.20+** (required for `incus cluster join` via remotes) -- All nodes deployed with `apply_defaults: true` (matching pool/network names) +- Init node deployed with `apply_defaults: true`; joining nodes with `apply_defaults: false` (recommended) or `true` (requires cleanup before join) - Remotes configured for all nodes (`incus remote add https://:8443`) - Client certificate trusted on all nodes (injected via seed or added manually) @@ -147,9 +147,34 @@ Expected output: one member (`incus-lab-01`), status `ONLINE`, role ## Step 3: Prepare Joining Nodes -### The apply_defaults storage pool conflict +### Recommended: deploy joining nodes with apply_defaults: false -Nodes deployed with `apply_defaults: true` already have: +The upstream IncusOS clustering tutorial recommends deploying joining nodes +**without** default settings. In the YAML config: + +```yaml +vms: + - name: incus-lab-01 + app: incus + apply_defaults: true # init node: needs storage pool + network + + - name: incus-lab-02 + app: incus + apply_defaults: false # joining node + + - name: incus-lab-03 + app: incus + apply_defaults: false # joining node +``` + +With `apply_defaults: false`, the node still listens on port 8443, trusts +preseed certificates, and the underlying ZFS dataset (`local/incus`) exists. +But no Incus storage pool or network metadata is created, so the join process +works cleanly without cleanup. Skip to Step 4 if using this approach. + +### Alternative: cleanup when apply_defaults: true was used + +If joining nodes were deployed with `apply_defaults: true`, they already have: - A `local` ZFS storage pool (source: `local/incus`) - An `incusbr0` network bridge - A `default` profile referencing both