diff --git a/incusos/TESTING.md b/incusos/TESTING.md new file mode 100644 index 0000000..354ad0e --- /dev/null +++ b/incusos/TESTING.md @@ -0,0 +1,298 @@ +# Testing Guide: incusos-proxmox + +A step-by-step guide for deploying IncusOS VMs to Proxmox VE using +`incusos-proxmox`. + +## Implementation Status + +The three scripts (`incusos-proxmox`, `incusos-seed`, `incusos-iso`) implement +the full lifecycle: + +- **Config parsing** -- YAML with PyYAML or built-in fallback parser +- **ISO download** -- fetches latest IncusOS from the CDN by channel + (stable/testing) +- **Per-VM seed generation** -- calls `incusos-seed --format fat` for each VM +- **Upload to Proxmox** -- SCP (SSH method) or multipart POST (API method) +- **VM creation** -- all IncusOS-required settings enforced (UEFI, TPM, + VirtIO-SCSI, etc.) +- **Automated install monitoring** -- state-machine polling with 10-minute + timeout +- **Cleanup** -- safe teardown with managed-VM marker checks + +### Cluster Formation (Manual Step) + +The `cluster` and `cluster_role` fields are accepted in config for +documentation purposes, but automatic cluster join is not yet implemented. +After deployment, join nodes to the cluster manually: + +```bash +# On the init node: +incus cluster add +# Copy the join token, then on the joining node's preseed or CLI +``` + +--- + +## Authentication + +### SSH (Default) + +Uses key-based SSH as `root@`. Requires no setup beyond +`ssh-copy-id`. + +### API Token (Recommended for Production) + +Avoids root SSH entirely. Uses a dedicated Proxmox user with minimal +privileges. + +#### Minimum Required Privileges + +The role below is already the minimum viable set -- every privilege is +required by at least one operation in the script: + +| Privilege | Used For | +|-----------|----------| +| `VM.Allocate` | Create and destroy VMs | +| `VM.Config.Disk` | scsi0, efidisk0, tpmstate0 | +| `VM.Config.CPU` | Set cores, cpu=host | +| `VM.Config.Memory` | Set memory, balloon=0 | +| `VM.Config.Network` | Attach VM to bridge | +| `VM.Config.CDROM` | Attach ISO (ide2) and seed (ide3) | +| `VM.Config.Options` | Set boot order, description, agent flag | +| `VM.Config.HWType` | Set bios=ovmf, machine=q35, scsihw | +| `VM.PowerMgmt` | Start, stop, and reboot VMs | +| `VM.Audit` | Query VM status and config | +| `Datastore.AllocateSpace` | Create disks in storage pool | +| `Datastore.AllocateTemplate` | Upload ISOs to storage | +| `Datastore.Audit` | List and check storage pools | +| `SDN.Use` | Use network bridges | + +None of these can be removed without breaking functionality. + +#### Setup Commands (Run on Proxmox) + +```bash +pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \ + VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \ + VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \ + Datastore.AllocateSpace Datastore.AllocateTemplate \ + Datastore.Audit SDN.Use" + +pveum user add automation@pve +pveum aclmod / -user automation@pve -role IncusOSDeploy +pveum user token add automation@pve deploy --privsep 0 +# Save the displayed secret! +``` + +#### Optional: Scoped ACLs + +For tighter access control, scope the ACL to specific paths instead of `/`: + +```bash +pveum aclmod /storage/local -user automation@pve -role IncusOSDeploy +pveum aclmod /storage/local-lvm -user automation@pve -role IncusOSDeploy +pveum aclmod /vms -user automation@pve -role IncusOSDeploy +``` + +For a home lab, `/` (root) is simpler and perfectly acceptable. + +--- + +## Guided Test Scenario + +### Prerequisites + +**On your workstation** (where you run the script): + +```bash +# Verify tools +bash --version # Need 4+ +python3 --version # For YAML parsing +which jq curl ssh scp # All needed for SSH method +which mcopy mkfs.fat # Needed for seed FAT images (mtools + dosfstools) + +# Verify the Incus client has generated a keypair +ls ~/.config/incus/client.crt ~/.config/incus/client.key +# If missing, run: incus remote list (triggers auto-generation) +``` + +**On your Proxmox host**: nothing to install -- just ensure SSH key auth +works. + +### Step 1: Verify SSH Connectivity + +```bash +ssh root@ pvesh get /version +``` + +You should see JSON with the Proxmox VE version. If this fails, fix SSH +key auth first (`ssh-copy-id root@`). + +### Step 2: Create a Test Config + +```bash +cp incusos/examples/proxmox-minimal.yaml my-test.yaml +``` + +Edit `my-test.yaml` -- change only the host IP: + +```yaml +proxmox: + host: # <-- change this + +defaults: + start_vmid: 900 # High range to avoid collisions + +vms: + - name: incus-test + app: incus + apply_defaults: true + cores: 4 + memory: 8192 + disk: 64 +``` + +Using VMID 900+ avoids clashing with existing VMs. + +### Step 3: Dry Run + +```bash +./incusos/incusos-proxmox --dry-run my-test.yaml +``` + +**Verify in the output:** + +- Config parsed correctly (host, node, storage shown) +- Client certificate detected (`~/.config/incus/client.crt`) +- Storage pools `local-lvm` and `local` found (adjust for your setup) +- Network bridge `vmbr0` found +- VMID 900 allocated for `incus-test` +- ISO version resolved from CDN (e.g., `IncusOS_25.03.iso`) +- `[dry-run]` prefix on all actions -- nothing actually executed +- No errors + +### Step 4: Deploy (Single VM) + +```bash +./incusos/incusos-proxmox my-test.yaml +``` + +**Watch the phases:** + +1. **Validate** -- connects to Proxmox, checks storage/bridge/tools +2. **Confirm** -- type `y` to proceed +3. **Prepare** -- downloads IncusOS ISO from CDN (~500 MB), generates seed +4. **Upload** -- SCPs ISO + seed to Proxmox +5. **Create** -- runs `qm create 900 ...` with all IncusOS settings +6. **Install** -- starts VM, monitors install completion (~3-5 min) + +**Simultaneously on the Proxmox web UI:** + +- Open `https://:8006` +- VM 900 (`incus-test`) should appear +- Open its Console tab +- Watch the IncusOS installer boot from ISO, read the seed, install to + disk, and reboot +- After reboot you should see the IncusOS login prompt + +### Step 5: Verify the Deployed VM + +Once the script reports success with an IP address: + +```bash +# The cert was injected via seed, so it's already trusted +incus remote add incus-test https://:8443 --accept-certificate + +# Test connectivity +incus remote list +incus info incus-test: + +# If apply_defaults was true, there should be a ZFS pool and bridge +incus storage list incus-test: +incus network list incus-test: +``` + +### Step 6: Clean Up the Test VM + +```bash +./incusos/incusos-proxmox --cleanup my-test.yaml +``` + +- Confirms before destroying +- Checks the management marker (`[incusos-lab:managed]` in the VM + description) +- Stops and destroys with `--purge` + +Remove the remote too: + +```bash +incus remote remove incus-test +``` + +### Step 7: (Optional) Test API Method + +After single-VM success with SSH, switch to API tokens. + +**On Proxmox** (one-time setup via SSH or web shell): + +```bash +pveum role add IncusOSDeploy --privs "VM.Allocate VM.Config.Disk \ + VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.CDROM \ + VM.Config.Options VM.Config.HWType VM.PowerMgmt VM.Audit \ + Datastore.AllocateSpace Datastore.AllocateTemplate \ + Datastore.Audit SDN.Use" +pveum user add automation@pve +pveum aclmod / -user automation@pve -role IncusOSDeploy +pveum user token add automation@pve deploy --privsep 0 +# ^^^ Save the displayed secret! +``` + +**On your workstation:** + +```bash +export PROXMOX_TOKEN_SECRET="" +``` + +Update `my-test.yaml`: + +```yaml +proxmox: + host: + method: api + api_token_id: automation@pve!deploy +``` + +Repeat steps 3-6 using the API method. + +### Step 8: (Optional) Full Lab Deployment + +Once single-VM tests pass, deploy the full 4-VM lab: + +```bash +cp incusos/examples/proxmox-lab.yaml my-lab.yaml +# Edit: change host IP, adjust start_vmid if needed +./incusos/incusos-proxmox --dry-run my-lab.yaml +./incusos/incusos-proxmox my-lab.yaml +``` + +This deploys 1 Operations Center + 3 Incus nodes. After deployment: + +1. Access Operations Center via browser at its reported IP +2. On the init node, run `incus cluster add ` to get join tokens +3. Join the other two nodes using the tokens + +--- + +## Troubleshooting + +| Symptom | Likely Cause | Fix | +|---------|-------------|-----| +| SSH connection refused | Key auth not set up | `ssh-copy-id root@` | +| Storage pool not found | Wrong pool name in config | Check `pvesm status` on Proxmox | +| ISO download fails | CDN unreachable | Download manually, use `--iso path/to/file.iso` | +| VM creation fails | VMID collision | Use higher `start_vmid` or check `qm list` | +| Install times out (10 min) | Slow I/O or boot issue | Check VM console in Proxmox web UI | +| No IP reported | Guest agent not ready | Wait a minute, check Proxmox UI for IP | +| Cert not auto-detected | No Incus keypair | Run `incus remote list` to generate one | +| `mcopy` not found | mtools not installed | Install `mtools` package | +| `mkfs.fat` not found | dosfstools not installed | Install `dosfstools` package | diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index 995ff47..d01112a 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -17,6 +17,7 @@ readonly CDN_INDEX="https://images.linuxcontainers.org/os/index.json" readonly CDN_BASE="https://images.linuxcontainers.org/os" readonly MIN_DISK_GIB=50 readonly MIN_MEMORY_MIB=4096 +readonly MANAGED_MARKER="[incusos-lab:managed]" # --------------------------------------------------------------------------- # Color and formatting @@ -84,7 +85,8 @@ ${BOLD}OPTIONS${RESET} ${BOLD}--no-cert${RESET} Skip certificate injection ${BOLD}--vm${RESET} NAME Operate on a single VM only ${BOLD}-y, --yes${RESET} Skip confirmation prompts - ${BOLD}--cleanup${RESET} Destroy VMs defined in config + ${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only) + ${BOLD}--force-cleanup${RESET} Destroy VMs even without management marker ${BOLD}--dry-run${RESET} Preview actions without executing ${BOLD}-q, --quiet${RESET} Suppress informational output ${BOLD}-h, --help${RESET} Show this help message @@ -196,6 +198,7 @@ NO_CERT=false VM_FILTER="" YES=false CLEANUP=false +FORCE_CLEANUP=false DRY_RUN=false QUIET=false CONFIG_FILE="" @@ -239,6 +242,11 @@ parse_args() { CLEANUP=true shift ;; + --force-cleanup) + CLEANUP=true + FORCE_CLEANUP=true + shift + ;; --dry-run) DRY_RUN=true shift @@ -1137,8 +1145,12 @@ pve_create_vm() { local iso_filename="$6" local seed_filename="$7" + local description + description=$(build_vm_description) + local cmd="qm create ${vmid}" cmd+=" --name ${name}" + cmd+=" --description '${description}'" cmd+=" --ostype l26" cmd+=" --bios ovmf" cmd+=" --machine q35" @@ -1166,10 +1178,15 @@ pve_create_vm() { pve_run "$cmd" ;; api) + # URL-encode the description for the API + local url_encoded_desc + url_encoded_desc=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${description}'))" 2>/dev/null) + # Build API parameters local params="" params+="vmid=${vmid}" params+="&name=${name}" + params+="&description=${url_encoded_desc}" params+="&ostype=l26" params+="&bios=ovmf" params+="&machine=q35" @@ -1314,6 +1331,174 @@ except: esac } +# --------------------------------------------------------------------------- +# VM management marker +# --------------------------------------------------------------------------- + +# Build the description string for a managed VM +build_vm_description() { + local config_basename + config_basename=$(basename "$CONFIG_FILE") + local timestamp + timestamp=$(date -u +"%Y-%m-%dT%H:%M:%SZ") + echo "${MANAGED_MARKER} config=${config_basename} created=${timestamp} by=${SCRIPT_NAME}/${VERSION}" +} + +# Check if a VM has our management marker in its description +pve_vm_is_managed() { + local vmid="$1" + + if [[ "$DRY_RUN" == true ]]; then + return 1 + fi + + local config_output="" + case "$PVE_METHOD" in + ssh) + config_output=$(pve_run "qm config ${vmid}" 2>/dev/null) || return 1 + ;; + api) + config_output=$(pve_api GET "/nodes/${PVE_NODE}/qemu/${vmid}/config" 2>/dev/null) || return 1 + ;; + esac + + echo "$config_output" | grep -qF "$MANAGED_MARKER" +} + +# List all VMs as "vmid name" pairs (one per line) +pve_list_vms() { + if [[ "$DRY_RUN" == true ]]; then + echo "" + return 0 + fi + + case "$PVE_METHOD" in + ssh) + pve_run "qm list 2>/dev/null" | awk 'NR>1 {print $1, $2}' || true + ;; + api) + pve_api GET "/nodes/${PVE_NODE}/qemu" 2>/dev/null | \ + python3 -c " +import json, sys +data = json.load(sys.stdin) +for vm in data.get('data', []): + print('{} {}'.format(vm.get('vmid', ''), vm.get('name', ''))) +" 2>/dev/null || true + ;; + esac +} + +# --------------------------------------------------------------------------- +# Pre-flight checks +# --------------------------------------------------------------------------- + +preflight_checks() { + step "Running pre-flight checks" + + local errors=0 + local warnings=0 + + # 1. Check that storage pools exist on Proxmox + if [[ "$DRY_RUN" != true ]]; then + case "$PVE_METHOD" in + ssh) + if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_STORAGE"; then + error "Storage pool '${PVE_STORAGE}' not found on Proxmox" + error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')" + error "Fix: update 'proxmox.storage' in your config file" + errors=$((errors + 1)) + fi + if ! pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | grep -qw "$PVE_ISO_STORAGE"; then + error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox" + error "Available pools: $(pve_run "pvesm status 2>/dev/null" | awk 'NR>1 {print \$1}' | tr '\n' ' ')" + error "Fix: update 'proxmox.iso_storage' in your config file" + errors=$((errors + 1)) + fi + ;; + api) + local storage_list + storage_list=$(pve_api GET "/nodes/${PVE_NODE}/storage" 2>/dev/null | \ + python3 -c " +import json, sys +data = json.load(sys.stdin) +for s in data.get('data', []): + print(s.get('storage', '')) +" 2>/dev/null) || storage_list="" + if ! echo "$storage_list" | grep -qw "$PVE_STORAGE"; then + error "Storage pool '${PVE_STORAGE}' not found on Proxmox" + error "Available pools: $(echo "$storage_list" | tr '\n' ' ')" + error "Fix: update 'proxmox.storage' in your config file" + errors=$((errors + 1)) + fi + if ! echo "$storage_list" | grep -qw "$PVE_ISO_STORAGE"; then + error "ISO storage '${PVE_ISO_STORAGE}' not found on Proxmox" + error "Available pools: $(echo "$storage_list" | tr '\n' ' ')" + error "Fix: update 'proxmox.iso_storage' in your config file" + errors=$((errors + 1)) + fi + ;; + esac + + # 2. Network bridge check (SSH only, warn-level) + if [[ "$PVE_METHOD" == "ssh" ]]; then + if ! pve_run "ip link show ${PVE_BRIDGE} 2>/dev/null" &>/dev/null; then + warn "Network bridge '${PVE_BRIDGE}' not found on Proxmox" + warn "VMs may fail to get network connectivity" + warn "Fix: update 'proxmox.bridge' in your config file" + warnings=$((warnings + 1)) + fi + fi + + # 3. ISO availability: check if already on Proxmox or CDN reachable + if [[ -z "$LOCAL_ISO" ]]; then + local any_iso_present=false + local iso_list="" + case "$PVE_METHOD" in + ssh) + iso_list=$(pve_run "pvesm list ${PVE_ISO_STORAGE} --content iso 2>/dev/null" 2>/dev/null) || iso_list="" + ;; + api) + iso_list=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_ISO_STORAGE}/content?content=iso" 2>/dev/null) || iso_list="" + ;; + esac + if echo "$iso_list" | grep -q "IncusOS_"; then + any_iso_present=true + fi + if [[ "$any_iso_present" != true ]]; then + # No ISO on Proxmox -- check if CDN is reachable + if ! curl -fsSL --connect-timeout 5 -o /dev/null "$CDN_INDEX" 2>/dev/null; then + error "No IncusOS ISO found on Proxmox and CDN is unreachable: ${CDN_INDEX}" + error "Fix: provide a local ISO with --iso FILE, or check internet connectivity" + errors=$((errors + 1)) + fi + fi + fi + fi + + # 4. FAT tools for SEED_DATA generation + if ! command -v mcopy &>/dev/null; then + error "mcopy (mtools) is required for SEED_DATA generation but not found" + error "Install mtools for your platform (e.g. mtools package)" + errors=$((errors + 1)) + fi + if ! command -v mkfs.fat &>/dev/null; then + error "mkfs.fat (dosfstools) is required for SEED_DATA generation but not found" + error "Install dosfstools for your platform (e.g. dosfstools package)" + errors=$((errors + 1)) + fi + + if [[ $errors -gt 0 ]]; then + error "${errors} pre-flight check(s) failed" + exit 1 + fi + + if [[ $warnings -gt 0 ]]; then + success "Pre-flight checks passed with ${warnings} warning(s)" + else + success "Pre-flight checks passed" + fi +} + # --------------------------------------------------------------------------- # Connectivity check # --------------------------------------------------------------------------- @@ -1368,11 +1553,26 @@ check_connectivity() { allocate_vmids() { step "Allocating VMIDs" - local existing_ids - existing_ids=$(pve_list_vmids) + # Get existing VMs as "vmid name" pairs + local vm_list + vm_list=$(pve_list_vms) + + # Build lookup: existing IDs and name-to-VMID map + local existing_ids="" + declare -A existing_name_to_vmid + if [[ -n "$vm_list" ]]; then + while IFS=' ' read -r eid ename; do + [[ -z "$eid" ]] && continue + existing_ids="${existing_ids} ${eid}" + if [[ -n "$ename" ]]; then + existing_name_to_vmid["$ename"]="$eid" + fi + done <<< "$vm_list" + fi local next_id="$DEF_START_VMID" local count=${#VM_NAMES[@]} + local errors=0 local i=0 while [[ $i -lt $count ]]; do @@ -1384,6 +1584,27 @@ allocate_vmids() { continue fi + # Check for name collision with existing Proxmox VMs + if [[ -n "${existing_name_to_vmid[$name]+x}" ]]; then + local collision_vmid="${existing_name_to_vmid[$name]}" + if pve_vm_is_managed "$collision_vmid"; then + warn "VM '${name}' already exists at VMID ${collision_vmid} (managed by ${SCRIPT_NAME})" + warn "Use --cleanup to redeploy, or remove it manually" + # Assign the existing VMID so cleanup/skip works correctly + VM_VMIDS[$i]="$collision_vmid" + i=$((i + 1)) + continue + else + error "VM name '${name}' already exists at VMID ${collision_vmid} but is NOT managed by ${SCRIPT_NAME}" + error "This VM was not created by this tool and will not be modified" + error "Fix: rename the VM in your config file, or manually remove VMID ${collision_vmid}:" + error " ssh ${PVE_SSH_USER}@${PVE_HOST} qm destroy ${collision_vmid} --purge" + errors=$((errors + 1)) + i=$((i + 1)) + continue + fi + fi + if [[ -n "${VM_VMIDS[$i]}" ]]; then # Explicit VMID detail "VM '${name}': VMID ${VM_VMIDS[$i]} (explicit)" @@ -1400,6 +1621,11 @@ allocate_vmids() { i=$((i + 1)) done + if [[ $errors -gt 0 ]]; then + error "${errors} name collision(s) with unmanaged VMs -- cannot continue" + exit 1 + fi + success "VMIDs allocated" } @@ -1625,6 +1851,7 @@ phase_validate() { validate_config check_connectivity allocate_vmids + preflight_checks detect_client_cert print_plan } @@ -1655,7 +1882,7 @@ phase_prepare() { ISO_FILENAME="IncusOS_${LATEST_VERSION}.iso" if pve_iso_exists "$ISO_FILENAME"; then - success "ISO already on Proxmox: ${ISO_FILENAME}" + success "ISO already on Proxmox: ${ISO_FILENAME} (reusing from previous deployment)" else download_iso "$LATEST_VERSION" "${WORKDIR}/${ISO_FILENAME}" fi @@ -1799,8 +2026,14 @@ phase_create() { # Check if VM already exists if pve_vm_exists "$vmid"; then - warn "VM ${vmid} ('${name}') already exists -- skipping" - warn "Use --cleanup first to remove existing VMs" + if pve_vm_is_managed "$vmid"; then + warn "VM ${vmid} ('${name}') already exists (managed) -- skipping" + warn "Use --cleanup to redeploy" + else + warn "VM ${vmid} ('${name}') already exists (NOT managed by ${SCRIPT_NAME}) -- skipping" + warn "Choose a different VMID or remove it manually:" + warn " ssh ${PVE_SSH_USER}@${PVE_HOST} qm destroy ${vmid} --purge" + fi i=$((i + 1)) continue fi @@ -1963,10 +2196,17 @@ phase_install() { do_cleanup() { step "Cleanup: destroying VMs defined in config" + if [[ "$FORCE_CLEANUP" == true ]]; then + warn "Force cleanup enabled -- will destroy VMs even without management marker" + fi echo "" local count=${#VM_NAMES[@]} + local destroyed=0 + local not_found=0 + local skipped_unmanaged=0 local i=0 + while [[ $i -lt $count ]]; do local name="${VM_NAMES[$i]}" @@ -1977,16 +2217,37 @@ do_cleanup() { local vmid="${VM_VMIDS[$i]}" + # 1. Check if VM exists if ! pve_vm_exists "$vmid"; then - detail "VM '${name}' (VMID ${vmid}): does not exist, skipping" + detail "VM '${name}' (VMID ${vmid}): does not exist" + not_found=$((not_found + 1)) i=$((i + 1)) continue fi + # 2. Check if VM is managed by us + if [[ "$DRY_RUN" != true ]]; then + if ! pve_vm_is_managed "$vmid"; then + if [[ "$FORCE_CLEANUP" == true ]]; then + warn "VM '${name}' (VMID ${vmid}): NOT managed by ${SCRIPT_NAME} -- destroying anyway (--force-cleanup)" + else + warn "VM '${name}' (VMID ${vmid}): NOT managed by ${SCRIPT_NAME} -- refusing to destroy" + warn " This VM was not created by this tool. To destroy it manually:" + warn " ssh ${PVE_SSH_USER}@${PVE_HOST} qm stop ${vmid} && ssh ${PVE_SSH_USER}@${PVE_HOST} qm destroy ${vmid} --purge" + warn " Or use --force-cleanup to override this safety check" + skipped_unmanaged=$((skipped_unmanaged + 1)) + i=$((i + 1)) + continue + fi + fi + fi + + # 3. Destroy the VM step "Destroying VM '${name}' (VMID ${vmid})" if [[ "$DRY_RUN" == true ]]; then detail "[dry-run] Would stop and destroy VM ${vmid}" + destroyed=$((destroyed + 1)) i=$((i + 1)) continue fi @@ -2003,6 +2264,7 @@ do_cleanup() { # Destroy if pve_destroy_vm "$vmid"; then success "VM '${name}' (VMID ${vmid}) destroyed" + destroyed=$((destroyed + 1)) else error "Failed to destroy VM '${name}' (VMID ${vmid})" fi @@ -2010,8 +2272,16 @@ do_cleanup() { i=$((i + 1)) done + # Summary echo "" - success "Cleanup complete" + local summary="Cleanup complete: ${destroyed} destroyed" + if [[ $not_found -gt 0 ]]; then + summary+=", ${not_found} not found" + fi + if [[ $skipped_unmanaged -gt 0 ]]; then + summary+=", ${skipped_unmanaged} skipped (unmanaged)" + fi + success "$summary" } # ---------------------------------------------------------------------------