maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix critical auth validation, role naming, Sys.Audit, and apply_defaults 

Safety fix: --cleanup-all, --resources, --labs silently failed when
PROXMOX_TOKEN_SECRET was not set, reporting "No VMs found" instead of
an auth error. Added require_api_auth() gate and error propagation in
pve_list_vms() to fail loudly on API auth failures.

Also:
- Standardize role name to IncusOSDeployer across all docs and scripts
- Add Sys.Audit privilege to all setup command examples
- Document /nodes/pve ACL requirement for scoped permissions
- Update lab-cluster.yaml: apply_defaults: false for joining nodes
  (upstream IncusOS recommendation to avoid 8-command cleanup)
- Add full cluster automation to lab-test phase_cluster:
  core.https_address setup, remote cert fixup, automated join prompts,
  conditional defaults cleanup for apply_defaults: true nodes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-21 08:51:52 +01:00
parent ca660585dc
commit e0faba8980
7 changed files with 197 additions and 48 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

28
CLAUDE.md
View File

@ -222,16 +222,23 @@ incu-contrib/
- The cert trust on the server side (client → server) is unaffected -- it's
stored by fingerprint in the Incus database, independent of listen address.
#### Joining nodes: the apply_defaults storage pool conflict
#### Joining nodes: apply_defaults and the storage pool conflict
- **`apply_defaults: true`** creates a `local` ZFS storage pool and `incusbr0`
network bridge on each node at install time. When joining a cluster that
already has a `local` pool defined, the join wizard asks for `source` and
`zfs.pool_name` properties but then tries to set them as cluster-wide config.
Incus rejects this because they are **member-specific** keys.
- **The fix**: delete the storage pool and network on the joining node before
joining. The join process then creates the member-specific pool entry
correctly.
- **Upstream recommendation**: use `apply_defaults: false` for nodes destined
to join a cluster. The official IncusOS clustering tutorial states joining
servers "cannot have preexisting networks or storage pools defined." With
`apply_defaults: false`, the node still listens on port 8443, still trusts
preseed certificates, and the underlying ZFS dataset (`local/incus`) still
exists -- but no Incus storage pool or network metadata is created, so
the join process works cleanly.
- **`apply_defaults: true` on joining nodes** is also functional but requires
an 8-command cleanup per node before join (delete pool, network, volumes,
profile devices). This is automated in `lab-test` but adds complexity.
- **Recommended seed pattern for clusters**:
- Bootstrap/init node: `apply_defaults: true` (needs pool and network)
- Joining nodes: `apply_defaults: false` (join process creates member-specific entries)
- Standalone nodes: `apply_defaults: true` (needs pool and network to be functional)
- **If `apply_defaults: true` was used**, the cleanup before join is:
```bash
# 1. Remove config references
incus config unset <remote>: storage.backups_volume
@ -425,7 +432,8 @@ incu-contrib/
application → Terraform config. Adds `meshbr0` network.
- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`,
nodes already have storage pool/network/cert. OC's Terraform fails with
"already exists" errors but the cluster forms successfully.
"already exists" errors but the cluster forms successfully. Use
`apply_defaults: false` for OC-managed nodes to avoid this.
#### Tested limitations

8
incusos/README.md
View File

@ -434,12 +434,12 @@ ssh root@192.168.1.10 pvesh get /version
**API:** Requires an API token. Set up minimal privileges:
```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \
VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use"
Datastore.AllocateSpace Datastore.AllocateTemplate Datastore.Audit SDN.Use Sys.Audit"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum aclmod / -user automation@pve -role IncusOSDeployer
pveum user token add automation@pve deploy --privsep 0
export PROXMOX_TOKEN_SECRET="<secret-from-above>"
@ -454,7 +454,7 @@ token to it instead of `/`:
# On Proxmox (one-time setup)
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
```
Then in your config file:

30
incusos/TESTING.md
View File

@ -67,20 +67,21 @@ required by at least one operation in the script:
| `Datastore.AllocateTemplate` | Upload ISOs to storage |
| `Datastore.Audit` | List and check storage pools |
| `SDN.Use` | Use network bridges |
| `Sys.Audit` | Query node status (RAM, CPU, uptime for `--resources`) |
None of these can be removed without breaking functionality.
#### Setup Commands (Run on Proxmox)
```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \
VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
Datastore.AllocateSpace Datastore.AllocateTemplate \
Datastore.Audit SDN.Use"
Datastore.Audit SDN.Use Sys.Audit"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum aclmod / -user automation@pve -role IncusOSDeployer
pveum user token add automation@pve deploy --privsep 0
# Save the displayed secret!
```
@ -90,9 +91,16 @@ pveum user token add automation@pve deploy --privsep 0
For tighter access control, scope the ACL to specific paths instead of `/`:
```bash
pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy
pveum aclmod /vms -user automation@pve -role IncusOSDeploy
pveum aclmod /storage/local -user automation@pve -role IncusOSDeployer
pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeployer
pveum aclmod /vms -user automation@pve -role IncusOSDeployer
```
**Note:** `Sys.Audit` requires an ACL on the node path, not covered by pool
or storage scopes:
```bash
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
```
For a home lab, `/` (root) is simpler and perfectly acceptable.
@ -286,13 +294,13 @@ After single-VM success with SSH, switch to API tokens.
**On Proxmox** (one-time setup via SSH or web shell):
```bash
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \
pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \
VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \
VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \
Datastore.AllocateSpace Datastore.AllocateTemplate \
Datastore.Audit SDN.Use"
Datastore.Audit SDN.Use Sys.Audit"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum aclmod / -user automation@pve -role IncusOSDeployer
pveum user token add automation@pve deploy --privsep 0
# ^^^ Save the displayed secret!
```
@ -323,7 +331,9 @@ For API method with pool isolation:
```bash
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
# For --resources (Sys.Audit on node path):
pveum aclmod /nodes/pve -user automation@pve -role IncusOSDeployer -propagate 0
```
Add to your config:

12
incusos/examples/lab-cluster.yaml
View File

@ -1,8 +1,10 @@
# lab-cluster.yaml - 3-node IncusOS cluster for lab testing
#
# Deploys 3 standalone Incus nodes that can be clustered after deployment.
# All nodes get apply_defaults: true so they have matching storage pool
# and network bridge configurations.
# The init node gets apply_defaults: true (storage pool + network bridge).
# Joining nodes use apply_defaults: false -- the cluster join process creates
# member-specific pool entries. This follows the upstream IncusOS recommendation
# and avoids the 8-command cleanup per joining node.
#
# Usage:
# incusos-proxmox --dry-run examples/lab-cluster.yaml # Preview
@ -26,12 +28,12 @@ defaults:
vms:
- name: incus-lab-01
app: incus
apply_defaults: true
apply_defaults: true # init node: needs storage pool + network
- name: incus-lab-02
app: incus
apply_defaults: true
apply_defaults: false # joining node: cluster join creates pool entry
- name: incus-lab-03
app: incus
apply_defaults: true
apply_defaults: false # joining node: cluster join creates pool entry

70
incusos/incusos-proxmox
View File

@ -196,19 +196,19 @@ ${BOLD}PROXMOX SETUP${RESET}
${BOLD}API method:${RESET}
Requires an API token. Set up minimal privileges:
pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \\
pveum role add IncusOSDeployer --privs "VM.Allocate VM.Config.Disk \\
VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \\
VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \\
Datastore.AllocateSpace Datastore.AllocateTemplate \\
Datastore.Audit SDN.Use"
Datastore.Audit SDN.Use Sys.Audit"
pveum user add automation@pve
pveum aclmod / -user automation@pve -role IncusOSDeploy
pveum aclmod / -user automation@pve -role IncusOSDeployer
pveum user token add automation@pve deploy --privsep 0
For resource pool isolation (recommended):
pveum pool add IncusLab --comment "IncusOS Lab VMs"
pveum pool modify IncusLab --storage local-lvm,local
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeploy
pveum aclmod /pool/IncusLab -user automation@pve -role IncusOSDeployer
Set the token secret:
export PROXMOX_TOKEN_SECRET="<secret-from-above>"
@ -1576,6 +1576,34 @@ pve_run() {
esac
}
# Validate API auth prerequisites and connectivity.
# Call this before any code path that uses pve_api() outside of phase_validate().
# Exits with error if credentials are missing or API is unreachable.
require_api_auth() {
if [[ "$PVE_METHOD" != "api" ]]; then
return 0
fi
if [[ -z "$PVE_API_TOKEN_ID" ]]; then
error "API token ID is required for API method"
error "Set proxmox.api_token_id in proxmox.yaml"
exit 1
fi
if [[ -z "${PROXMOX_TOKEN_SECRET:-}" ]]; then
error "PROXMOX_TOKEN_SECRET environment variable is not set"
error "Source your env file: source env"
error "Or set it directly: export PROXMOX_TOKEN_SECRET=\"<your-token-secret>\""
exit 1
fi
# Verify we can actually reach the API with these credentials
local url="https://${PVE_HOST}:8006/api2/json/version"
local auth="Authorization: PVEAPIToken=${PVE_API_TOKEN_ID}=${PROXMOX_TOKEN_SECRET}"
if ! curl -fsSk --connect-timeout 5 -H "$auth" "$url" &>/dev/null; then
error "Cannot authenticate to Proxmox API at https://${PVE_HOST}:8006"
error "Check that the host is reachable, token ID is correct, and secret is valid"
exit 1
fi
}
# Execute a Proxmox API call (API method)
# Usage: pve_api METHOD ENDPOINT [DATA] [json]
# Pass "json" as 4th arg to send data as application/json instead of form-encoded.
@ -2175,18 +2203,24 @@ pve_vm_has_install_media() {
}
# List all VMs as "vmid name" pairs (one per line)
# Returns non-zero and prints error message on API/SSH failure.
pve_list_vms() {
if [[ "$DRY_RUN" == true ]]; then
echo ""
return 0
fi
local raw_response
# When a resource pool is configured, list only pool members (type=qemu)
if [[ -n "$PVE_POOL" ]]; then
case "$PVE_METHOD" in
ssh)
pve_run "pvesh get /pools/${PVE_POOL} --output-format json 2>/dev/null" | \
python3 -c "
if ! raw_response=$(pve_run "pvesh get /pools/${PVE_POOL} --output-format json 2>/dev/null" 2>/dev/null); then
echo "[error] Failed to list VMs via SSH" >&2
return 1
fi
echo "$raw_response" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for m in data.get('members', []):
@ -2195,8 +2229,11 @@ for m in data.get('members', []):
" 2>/dev/null || true
;;
api)
pve_api GET "/pools/${PVE_POOL}" 2>/dev/null | \
python3 -c "
if ! raw_response=$(pve_api GET "/pools/${PVE_POOL}" 2>/dev/null); then
echo "[error] Failed to list VMs from pool '${PVE_POOL}' via API (auth failure or pool not found)" >&2
return 1
fi
echo "$raw_response" | python3 -c "
import json, sys
data = json.load(sys.stdin).get('data', {})
for m in data.get('members', []):
@ -2213,8 +2250,11 @@ for m in data.get('members', []):
pve_run "qm list 2>/dev/null" | awk 'NR>1 {print $1, $2}' || true
;;
api)
pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null | \
python3 -c "
if ! raw_response=$(pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null); then
echo "[error] Failed to list VMs via API (auth failure or node not found)" >&2
return 1
fi
echo "$raw_response" | python3 -c "
import json, sys
data = json.load(sys.stdin)
for vm in data.get('data', []):
@ -2351,7 +2391,7 @@ for s in data.get('data', []):
if [[ "$PVE_METHOD" == "api" ]] && [[ -n "$PVE_API_TOKEN_ID" ]]; then
local api_user
api_user=$(echo "$PVE_API_TOKEN_ID" | cut -d'!' -f1)
error " pveum aclmod /pool/${PVE_POOL} -user ${api_user} -role IncusOSDeploy"
error " pveum aclmod /pool/${PVE_POOL} -user ${api_user} -role IncusOSDeployer"
fi
errors=$((errors + 1))
else
@ -4156,7 +4196,11 @@ do_cleanup_all() {
# Get all VMs in the pool
local vm_list
vm_list=$(pve_list_vms)
if ! vm_list=$(pve_list_vms); then
error "Failed to list VMs in pool '${PVE_POOL}'"
error "Check Proxmox connectivity and API credentials"
exit 1
fi
if [[ -z "$vm_list" ]]; then
info "No VMs found in pool '${PVE_POOL}'"
@ -4383,6 +4427,7 @@ main() {
exit 1
fi
require_api_auth
[[ "$RESOURCES" == true ]] && do_resources
[[ "$LIST_LABS" == true ]] && do_list_labs
return 0
@ -4412,6 +4457,7 @@ main() {
exit 1
fi
require_api_auth
do_cleanup_all
if [[ "$DEEP_CLEANUP" == true ]]; then
echo ""

66
incusos/lab-test
View File

@ -612,13 +612,50 @@ phase_cluster() {
fi
fi
# Step 1: Enable clustering on init node
# Step 1: Set core.https_address on all nodes (required for clustering)
step "Setting core.https_address on all nodes"
local -A NODE_IPS=()
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] Would query and set core.https_address on all nodes"
else
local n=0
while [[ $n -lt ${#VM_NAMES[@]} ]]; do
local node="${VM_NAMES[$n]}"
local node_ip
node_ip=$(incus query "${node}:/1.0" 2>/dev/null | \
python3 -c "
import sys, json
d = json.load(sys.stdin)
for a in d.get('environment', {}).get('addresses', []):
if not a.startswith('10.') and not a.startswith('fd42:') and not a.startswith('['):
print(a)
break
" 2>/dev/null) || node_ip=""
if [[ -z "$node_ip" ]]; then
fail "Cannot determine IP for ${node}"
return 1
fi
NODE_IPS["$node"]="$node_ip"
incus config set "${node}:" core.https_address "${node_ip}:8443" 2>/dev/null
detail "${node}: ${node_ip}:8443"
n=$((n + 1))
done
pass "core.https_address set on all nodes"
fi
echo ""
# Step 2: Enable clustering on init node
step "Enabling clustering on ${init_node}"
if [[ "$DRY_RUN" == true ]]; then
detail "[dry-run] incus cluster enable ${init_node}: ${init_node}"
else
if incus cluster enable "${init_node}:" "${init_node}" 2>/dev/null; then
pass "Clustering enabled on ${init_node}"
# Fix remote (cluster enable regenerates TLS cert)
incus remote remove "${init_node}" 2>/dev/null || true
incus remote add "${init_node}" "https://${NODE_IPS[$init_node]}:8443" \
--accept-certificate 2>/dev/null || true
detail "Remote ${init_node} re-added with new cluster cert"
else
fail "Failed to enable clustering on ${init_node}"
return 1
@ -626,7 +663,7 @@ phase_cluster() {
fi
echo ""
# Step 2-N: Join remaining nodes
# Step 3-N: Join remaining nodes
local i=1
while [[ $i -lt ${#VM_NAMES[@]} ]]; do
local join_node="${VM_NAMES[$i]}"
@ -644,6 +681,21 @@ phase_cluster() {
continue
fi
# Check if joining node has storage pool (apply_defaults: true)
# and clean up if needed for cluster join
if incus storage show "${join_node}:local" &>/dev/null 2>&1; then
detail "Cleaning up defaults on ${join_node} for cluster join"
incus config unset "${join_node}:" storage.backups_volume 2>/dev/null || true
incus config unset "${join_node}:" storage.images_volume 2>/dev/null || true
incus storage volume delete "${join_node}:local" backups 2>/dev/null || true
incus storage volume delete "${join_node}:local" images 2>/dev/null || true
incus profile device remove "${join_node}:default" root 2>/dev/null || true
incus profile device remove "${join_node}:default" eth0 2>/dev/null || true
incus storage delete "${join_node}:local" 2>/dev/null || true
incus network delete "${join_node}:incusbr0" 2>/dev/null || true
detail "Defaults cleaned up"
fi
# Generate join token
local token
token=$(incus cluster add "${init_node}:" "${join_node}" 2>/dev/null) || token=""
@ -654,9 +706,15 @@ phase_cluster() {
fi
detail "Join token generated"
# Join the cluster
if incus cluster join "${init_node}:" "${join_node}:" 2>/dev/null; then
# Join the cluster (automated: accept defaults, answer prompts)
if printf '\n\nyes\nlocal/incus\nlocal/incus\n' | \
incus cluster join "${init_node}:" "${join_node}:" 2>/dev/null; then
pass "Node ${join_node} joined the cluster"
# Fix remote (join regenerates TLS cert)
incus remote remove "${join_node}" 2>/dev/null || true
incus remote add "${join_node}" "https://${NODE_IPS[$join_node]}:8443" \
--accept-certificate 2>/dev/null || true
detail "Remote ${join_node} re-added with cluster cert"
else
fail "Failed to join ${join_node} to cluster"
fi

31
notes/clustering-guide.md
View File

@ -28,7 +28,7 @@ With 3 nodes, you get a fully HA database quorum.
## Prerequisites
- Incus client version **6.20+** (required for `incus cluster join` via remotes)
- All nodes deployed with `apply_defaults: true` (matching pool/network names)
- Init node deployed with `apply_defaults: true`; joining nodes with `apply_defaults: false` (recommended) or `true` (requires cleanup before join)
- Remotes configured for all nodes (`incus remote add <name> https://<ip>:8443`)
- Client certificate trusted on all nodes (injected via seed or added manually)
@ -147,9 +147,34 @@ Expected output: one member (`incus-lab-01`), status `ONLINE`, role
## Step 3: Prepare Joining Nodes
### The apply_defaults storage pool conflict
### Recommended: deploy joining nodes with apply_defaults: false
Nodes deployed with `apply_defaults: true` already have:
The upstream IncusOS clustering tutorial recommends deploying joining nodes
**without** default settings. In the YAML config:
```yaml
vms:
- name: incus-lab-01
app: incus
apply_defaults: true # init node: needs storage pool + network
- name: incus-lab-02
app: incus
apply_defaults: false # joining node
- name: incus-lab-03
app: incus
apply_defaults: false # joining node
```
With `apply_defaults: false`, the node still listens on port 8443, trusts
preseed certificates, and the underlying ZFS dataset (`local/incus`) exists.
But no Incus storage pool or network metadata is created, so the join process
works cleanly without cleanup. Skip to Step 4 if using this approach.
### Alternative: cleanup when apply_defaults: true was used
If joining nodes were deployed with `apply_defaults: true`, they already have:
- A `local` ZFS storage pool (source: `local/incus`)
- An `incusbr0` network bridge
- A `default` profile referencing both