From 52d8dbdc89da78d5975dd00231b62d682a4e35d2 Mon Sep 17 00:00:00 2001 From: Maarten Date: Sat, 21 Feb 2026 21:41:34 +0100 Subject: [PATCH] OC v0.3.0 comprehensive test: retry logic, auto-heal, apply_defaults validation Phase 0-10 of OC test plan completed: - Add --retries N flag to incusos-proxmox (default 3, configurable boot retries) - Add fix_scrub_schedule() auto-heal for IncusOS crontab bug via REST API - Validate apply_defaults: false for OC-managed nodes (recommended approach) - Test full OC lifecycle: deploy, provision, cluster, workloads, migration, evacuation, node failure/recovery, inventory query, REST API - Update CLAUDE.md: OC v0.3.0 commands, --retries, scrub fix, node recovery - Update operations-center-guide.md: v0.3.0, apply_defaults, scrub fix, node failure simulation, IncusOS REST API docs, inventory query Key findings: - apply_defaults: false lets OC Terraform handle everything cleanly - fix_scrub_schedule() proactively heals ~50% crontab boot failure - OC does not detect node failures (always shows "ready") - inventory query (v0.3.0) provides cross-resource tree views - Node crash recovery: auto-rejoin in ~60s, containers auto-start Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 38 ++++-- incusos/README.md | 1 + incusos/examples/lab-oc-nodes.yaml | 6 +- incusos/incusos-proxmox | 169 +++++++++++++++++++------ notes/operations-center-guide.md | 190 ++++++++++++++++++++++------- 5 files changed, 309 insertions(+), 95 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 185d8a2..0ba4d83 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -419,8 +419,14 @@ incu-contrib/ as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`). - **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI. - **Browser access**: requires PKCS#12 client certificate (`client.pfx`) - imported into the browser. -- **OC is under active development** (v0.2.2). Commands and APIs may change. + imported into the browser. Web UI is a React SPA at `/ui/`. +- **OC is under active development** (v0.3.0). Commands and APIs may change. + Use `operations-center --version` (not `version` subcommand) to check. +- **v0.3.0 new commands**: `provisioning channel` (add/list/show), + `cluster update/update-certificate/rename`, expanded `inventory` + (network-acl, address-sets, load-balancers, peers, zones, integrations, + forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource + tree view with filters and Go templates), `system certificate set`. - The `--doctor` command on `incusos-proxmox` reports whether the CLI is installed. @@ -452,10 +458,11 @@ incu-contrib/ ``` OC handles: `core.https_address` → cluster enable → joins → seed application → Terraform config. Adds `meshbr0` network. -- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`, - nodes already have storage pool/network/cert. OC's Terraform fails with - "already exists" errors but the cluster forms successfully. Use - `apply_defaults: false` for OC-managed nodes to avoid this. +- **apply_defaults: false is recommended** (tested): use `apply_defaults: false` + for OC-managed nodes. OC's Terraform handles storage pool, network, and cert + creation cleanly. With `apply_defaults: true`, nodes already have these + resources and OC's Terraform fails with "already exists" errors (cluster still + forms, but Terraform artifacts are empty). #### Tested limitations @@ -468,9 +475,13 @@ incu-contrib/ crash the daemon. See "IncusOS boot failure" section below for full analysis. Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe. - **No cluster member state tracking** -- OC always shows `ready` even for - EVACUATED/OFFLINE nodes + EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat + detects failure in ~40s; OC has no equivalent). - **Stale entries** from out-of-band cluster changes persist after resync - **Server removal blocked** if server is part of an OC cluster +- **Node failure recovery**: Proxmox hard-stop simulates crash. After restart, + node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly + if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`). - See `notes/operations-center-guide.md` for full tested OC reference. @@ -490,6 +501,12 @@ incu-contrib/ - **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default output is concise (step names + results). `--quiet` suppresses everything except warnings and errors. +- **`--retries N`**: number of stop+start retries for VMs that fail to boot + (port 8443 not reachable). Default: 3. `--retries 0` disables retries. + Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443. + With ~50% boot failure rate from the crontab bug, 3 retries gives ~93.75% + success rate. After retries, `fix_scrub_schedule()` proactively heals the + root cause via the IncusOS REST API. ### IncusOS first-boot sequence (observed via console screenshots) @@ -627,6 +644,13 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`. - **Detection**: `phase_install` checks port 8443 after starting from disk (30s wait + up to 60s polling). If first check fails, automatic retry adds ~100s. Failed VMs after retry are reported with remediation. +- **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes + `fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on + every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix + proxies to the IncusOS daemon API through Incus. Safe to call on every + node — returns early if schedule is already set. Called on both initial + success and retry success paths. +- **Filed upstream**: IncusOS issue #843. - **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`, `updateChecker()`, `startup()`, `firstBootActions()`), `incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`, diff --git a/incusos/README.md b/incusos/README.md index 2dabc3d..f2f8e5e 100644 --- a/incusos/README.md +++ b/incusos/README.md @@ -295,6 +295,7 @@ This script ensures all settings are correct, preventing install failures. -c, --cert FILE Client certificate PEM file (default: auto-detect) --no-cert Skip certificate injection --vm NAME Operate on a single VM only + --retries N Boot retries for first-boot failures (default: 3) -y, --yes Skip confirmation prompts --proxmox FILE Proxmox connection config (default: auto-discover) --doctor Check tool versions and environment health diff --git a/incusos/examples/lab-oc-nodes.yaml b/incusos/examples/lab-oc-nodes.yaml index 08b079d..021c457 100644 --- a/incusos/examples/lab-oc-nodes.yaml +++ b/incusos/examples/lab-oc-nodes.yaml @@ -23,12 +23,12 @@ defaults: vms: - name: oc-node-01 app: incus - apply_defaults: true + apply_defaults: false - name: oc-node-02 app: incus - apply_defaults: true + apply_defaults: false - name: oc-node-03 app: incus - apply_defaults: true + apply_defaults: false diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index b28b124..a3b1799 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -135,6 +135,7 @@ ${BOLD}OPTIONS${RESET} ${BOLD}-c, --cert${RESET} FILE Client certificate PEM file ${BOLD}--no-cert${RESET} Skip certificate injection ${BOLD}--vm${RESET} NAME Operate on a single VM only + ${BOLD}--retries${RESET} N Boot retries for first-boot failures (default: 3) ${BOLD}-y, --yes${RESET} Skip confirmation prompts ${BOLD}--doctor${RESET} Check tool versions and environment health ${BOLD}--cleanup${RESET} Destroy VMs defined in config (managed only) @@ -304,6 +305,7 @@ LIST_LABS=false DRY_RUN=false QUIET=false VERBOSE=false +INSTALL_RETRIES=3 PROXMOX_CONFIG="" CONFIG_FILE="" @@ -342,6 +344,14 @@ parse_args() { YES=true shift ;; + --retries) + INSTALL_RETRIES="${2:?'--retries requires a number'}" + if ! [[ "$INSTALL_RETRIES" =~ ^[0-9]+$ ]]; then + error "--retries must be a non-negative integer" + exit 1 + fi + shift 2 + ;; --status) PHASE="status" shift @@ -3221,6 +3231,72 @@ register_remote() { fi } +# --------------------------------------------------------------------------- +# fix_scrub_schedule -- proactively fix empty scrub_schedule via IncusOS API +# --------------------------------------------------------------------------- +# IncusOS has an intermittent bug where state.txt is written with an empty +# scrub_schedule field (the encoder skips zero values). When registerJobs() +# runs, gocron.IsValid("") fails and the daemon crashes in a loop. The REST +# API is briefly reachable during each cycle (Incus starts before the +# scheduler), so we can fix it via PUT /os/1.0/system/storage. +# +# This function is safe to call on every node -- if scrub_schedule is already +# set correctly, the GET check returns early without making any changes. +# +# Usage: fix_scrub_schedule +# Returns: 0 on success or not needed, 1 on failure +fix_scrub_schedule() { + local ip="$1" + local name="$2" + + # Need client cert for the IncusOS API + if [[ -z "$CERT_FILE_RESOLVED" ]]; then + detail "No client certificate available — skipping scrub_schedule check" + return 0 + fi + + local cert="$CERT_FILE_RESOLVED" + local key="${CERT_FILE_RESOLVED%.crt}.key" + + if [[ ! -f "$key" ]]; then + detail "Client key not found at ${key} — skipping scrub_schedule check" + return 0 + fi + + # GET current storage config + local response + response=$(curl -sk --connect-timeout 5 --max-time 10 \ + --cert "$cert" --key "$key" \ + "https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || { + detail "Could not query IncusOS storage API on '${name}' — skipping scrub_schedule check" + return 0 + } + + # Check if scrub_schedule is empty or missing + local current_schedule + current_schedule=$(printf '%s' "$response" | \ + python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('metadata',{}).get('config',{}).get('scrub_schedule',''))" 2>/dev/null) || current_schedule="" + + if [[ -n "$current_schedule" ]]; then + detail "scrub_schedule on '${name}' is already set: ${current_schedule}" + return 0 + fi + + # scrub_schedule is empty — fix it + local put_response + put_response=$(curl -sk --connect-timeout 5 --max-time 10 \ + --cert "$cert" --key "$key" \ + -X PUT -H "Content-Type: application/json" \ + -d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \ + "https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || { + warn "Failed to fix scrub_schedule on '${name}' via IncusOS API" + return 1 + } + + info "Fixed empty scrub_schedule on '${name}' via IncusOS API" + return 0 +} + # --------------------------------------------------------------------------- # Phase 5: Install # --------------------------------------------------------------------------- @@ -3431,53 +3507,66 @@ phase_install() { echo "" if [[ "$healthy" == true ]]; then + # Proactively fix empty scrub_schedule (IncusOS crontab bug). + # Safe to call on every node -- returns early if already set. + fix_scrub_schedule "$ip" "$name" || true success "VM '${name}' installed and running at ${ip}" register_remote "$name" "$ip" else - # Retry once: IncusOS has an intermittent "invalid crontab expression" + # Retry loop: IncusOS has an intermittent "invalid crontab expression" # bug (~50% first-boot hit rate) where registerJobs() fails during - # startup(). Most likely a race condition between the REST API and - # startup sequence, or a gocron library edge case. A stop+start - # cycle resolves it because the transient condition clears on retry. - if [[ -n "$ip" ]]; then - warn "VM '${name}': port 8443 not reachable at ${ip} — retrying (stop+start)" - else - warn "VM '${name}': could not detect IP — retrying (stop+start)" - fi - info "Stopping VM for retry..." - pve_stop_vm "$vmid" || true - sleep 10 - info "Starting VM (retry)..." - pve_start_vm "$vmid" || true - sleep 30 + # startup(). A stop+start cycle resolves it because the hard power-off + # may discard the corrupt state.txt before it's committed to disk, + # allowing initialize() to recreate it cleanly on next boot. + local retry=0 + local retry_ok=false + while [[ $retry -lt $INSTALL_RETRIES ]]; do + retry=$((retry + 1)) + if [[ -n "$ip" ]]; then + warn "VM '${name}': port 8443 not reachable at ${ip} — retry ${retry}/${INSTALL_RETRIES} (stop+start)" + else + warn "VM '${name}': could not detect IP — retry ${retry}/${INSTALL_RETRIES} (stop+start)" + fi + info "Stopping VM for retry..." + pve_stop_vm "$vmid" || true + sleep 10 + info "Starting VM (retry ${retry}/${INSTALL_RETRIES})..." + pve_start_vm "$vmid" || true + sleep 30 - # Re-detect IP (may have changed after restart) - ip=$(pve_vm_ip "$vmid") || ip="" + # Re-detect IP (may have changed after restart) + ip=$(pve_vm_ip "$vmid") || ip="" - local retry_healthy=false - if [[ -n "$ip" ]]; then - local retry_poll=0 - while [[ $retry_poll -lt 12 ]]; do - if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then - retry_healthy=true - break - fi - sleep 5 - retry_poll=$((retry_poll + 1)) - done - fi + if [[ -n "$ip" ]]; then + local retry_poll=0 + while [[ $retry_poll -lt 12 ]]; do + if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then + retry_ok=true + break + fi + sleep 5 + retry_poll=$((retry_poll + 1)) + done + fi - if [[ "$retry_healthy" == true ]]; then - success "VM '${name}' recovered after retry — running at ${ip}" - register_remote "$name" "$ip" - elif [[ -n "$ip" ]]; then - warn "VM '${name}' installed but port 8443 not reachable at ${ip} (even after retry)" - warn "Likely IncusOS boot failure (check Proxmox console for errors)" - warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}" - install_failures=$((install_failures + 1)) - else - warn "VM '${name}' installed but could not detect IP (even after retry)" - warn "Check Proxmox console for status" + if [[ "$retry_ok" == true ]]; then + # Proactively fix empty scrub_schedule (IncusOS crontab bug). + fix_scrub_schedule "$ip" "$name" || true + success "VM '${name}' recovered after retry ${retry}/${INSTALL_RETRIES} — running at ${ip}" + register_remote "$name" "$ip" + break + fi + done + + if [[ "$retry_ok" != true ]]; then + if [[ -n "$ip" ]]; then + warn "VM '${name}' installed but port 8443 not reachable at ${ip} (after ${INSTALL_RETRIES} retries)" + warn "Likely IncusOS boot failure (check Proxmox console for errors)" + warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}" + else + warn "VM '${name}' installed but could not detect IP (after ${INSTALL_RETRIES} retries)" + warn "Check Proxmox console for status" + fi install_failures=$((install_failures + 1)) fi fi diff --git a/notes/operations-center-guide.md b/notes/operations-center-guide.md index be56f1e..ddcdc7a 100644 --- a/notes/operations-center-guide.md +++ b/notes/operations-center-guide.md @@ -5,7 +5,7 @@ a centralized management layer for IncusOS/Incus deployments. OC handles provisioning, clustering, inventory, and updates through a REST API, CLI, and web UI. -All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553 +All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553 on Proxmox VE 9.1.5 (nested virtualization on Intel). --- @@ -36,10 +36,10 @@ containing a provisioning token, then self-register on first boot. ```bash # Check for latest release at https://github.com/FuturFusion/operations-center/releases -wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64 +wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64 chmod +x operations-center_linux_x86_64 sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center -operations-center version +operations-center --version # Note: use --version, not 'version' subcommand ``` ### Set up OC client certificates @@ -233,15 +233,15 @@ defaults: vms: - name: oc-node-01 app: incus - apply_defaults: true + apply_defaults: false # OC handles storage/network via Terraform - name: oc-node-02 app: incus - apply_defaults: true + apply_defaults: false - name: oc-node-03 app: incus - apply_defaults: true + apply_defaults: false ``` **VMID note**: if your Proxmox API token is scoped to a resource pool, it @@ -316,29 +316,35 @@ OC orchestrates the full cluster formation: 4. Applies the application seed (injects client cert) 5. Runs Terraform/OpenTofu for post-cluster configuration -### apply_defaults conflict (tested finding) +### apply_defaults: false (tested, recommended) -When using `apply_defaults: true` in the SEED_DATA (via `incusos-proxmox`), -each node already has a `local` ZFS storage pool, `incusbr0` network bridge, -and the client certificate. OC's Terraform post-clustering step then tries to -create the same resources and fails with: +**Use `apply_defaults: false`** for OC-managed nodes. This is now the tested +and recommended approach. With `apply_defaults: false`: + +- Nodes still listen on port 8443 and trust preseed certificates +- The underlying ZFS dataset (`local/incus`) still exists +- But no Incus storage pool, network, or profile devices are created +- OC's Terraform handles all resource creation cleanly during cluster formation + +**Tested result**: cluster forms cleanly with `apply_defaults: false`. OC's +Terraform creates the storage pool (`local`), networks (`incusbr0`, `meshbr0`), +and injects the client certificate via `--application-seed-config`. The only +Terraform error is "Certificate already in trust store" when the cert was also +injected via SEED_DATA -- avoid this by only injecting the cert through the +application seed config, not the SEED_DATA. + +### apply_defaults: true (also works, with caveats) + +When using `apply_defaults: true`, each node already has a `local` ZFS storage +pool, `incusbr0` network bridge, and the client certificate. OC's Terraform +then fails with: - "Certificate already in trust store" - "Network is not in pending state" - "Storage pool is not in pending state" -**The cluster itself forms successfully** despite these Terraform errors. All -3 nodes join the cluster correctly. The Terraform errors are non-fatal because -the resources already exist. However, the `cluster artifact list` will be empty -since Terraform didn't complete. - -**Recommended approaches** for future deployments: -1. **Keep apply_defaults** (current tested approach): cluster forms fine, ignore - Terraform errors. Skip `--application-seed-config` since the cert is already - injected by the seed. -2. **Skip apply_defaults**: deploy nodes without `apply_defaults: true` and - without cert injection in the seed. Let OC handle everything via Terraform. - This is the "clean" approach but hasn't been tested yet. +**The cluster itself forms successfully** despite these Terraform errors. +However, `cluster artifact list` will be empty since Terraform didn't complete. ### Verify cluster (tested) @@ -433,6 +439,59 @@ The proper lifecycle for maintenance is: 4. Wait for IncusOS to boot (~60-90s) 5. `incus cluster restore : --force` +### IncusOS scrub_schedule fix via API (tested) + +IncusOS has an intermittent bug (~50% of deploys) where `state.txt` is written +with an empty `scrub_schedule` field. The `registerJobs()` function calls +`gocron.IsValid("")` which fails, crashing the daemon in a loop. The REST API +is briefly reachable during each crash cycle (Incus starts before the scheduler). + +**Fix via IncusOS REST API** (through the Incus proxy at `/os/` prefix): + +```bash +# Check current storage config +curl -sk --cert ~/.config/incus/client.crt \ + --key ~/.config/incus/client.key \ + https://:8443/os/1.0/system/storage + +# If scrub_schedule is empty, fix it +curl -sk --cert ~/.config/incus/client.crt \ + --key ~/.config/incus/client.key \ + -X PUT -H "Content-Type: application/json" \ + -d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \ + https://:8443/os/1.0/system/storage +``` + +**Key details**: +- The `/os/` prefix is required — this proxies to the IncusOS daemon API +- PATCH returns 501 (not implemented); use PUT instead +- `incus admin os system storage edit` fails because it reads the current + (corrupt) config and validates it before allowing edits +- `incusos-proxmox` now includes `fix_scrub_schedule()` which proactively + fixes this on every deployed node (safe — returns early if already set) +- Filed upstream as IncusOS issue #843 + +### Node failure and recovery (tested) + +Proxmox hard-stop simulates node crash. Incus cluster detects failure via +heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot): + +```bash +# Simulate crash: Proxmox API stop +curl -sk -X POST ".../nodes/pve/qemu//status/stop" ... + +# After ~40s: incus cluster list shows OFFLINE with heartbeat info + +# Restart: Proxmox API start +curl -sk -X POST ".../nodes/pve/qemu//status/start" ... + +# After ~60s boot: node auto-rejoins cluster (ONLINE) +# Containers on the node auto-start after recovery +``` + +**OC does not detect node failures**: `provisioning server show` still shows +`Status: ready` for an OFFLINE/crashed node. `Last Seen` timestamp is stale. + **OC does not track Incus cluster member state**: the server list always shows `ready` status even when the Incus cluster reports `EVACUATED`. @@ -542,6 +601,7 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force # 2. Destroy broken VM via Proxmox (stop + delete) # 3. Redeploy via incusos-proxmox --iso (single-node config) +# Use apply_defaults: false for OC-managed nodes ./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml # 4. New node auto-registers with OC (same hostname, new IP) @@ -550,18 +610,10 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force # 5. Manual cluster join (since node was force-removed) incus config set oc-node-03: core.https_address :8443 -# Remove apply_defaults resources -incus config unset oc-node-03: storage.backups_volume -incus config unset oc-node-03: storage.images_volume -incus storage volume delete oc-node-03:local backups -incus storage volume delete oc-node-03:local images -incus profile device remove oc-node-03:default root -incus profile device remove oc-node-03:default eth0 -incus storage delete oc-node-03:local -incus network delete oc-node-03:incusbr0 +# With apply_defaults: false, no cleanup needed (no pool/network to delete) # Generate token and join incus cluster add oc-node-01:oc-node-03 -printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03: +printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03: # Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering) # 6. Fix remote after cert change @@ -569,6 +621,21 @@ incus remote remove oc-node-03 incus remote add oc-node-03 https://:8443 --accept-certificate ``` +### Node failure simulation (tested) + +Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s): + +```bash +# Simulate crash +curl -sk -X POST ".../nodes/pve/qemu//status/stop" ... +# incus cluster list shows OFFLINE after ~40s + +# Restart +curl -sk -X POST ".../nodes/pve/qemu//status/start" ... +# Node auto-rejoins after ~60s boot. Containers auto-start. +# OC still shows "ready" (does not detect node failures) +``` + ### Factory reset (not tested) ```bash @@ -650,10 +717,20 @@ operations-center provisioning cluster factory-reset oc-cluster | `inventory instance list` | List all instances | | `inventory instance show ` | Instance details (by UUID, not name) | | `inventory storage-pool list` | List storage pools | +| `inventory storage-volume list` | List storage volumes | +| `inventory storage-bucket list` | List storage buckets | | `inventory network list` | List networks | +| `inventory network-acl list` | List network ACLs | +| `inventory network-address-set list` | List network address sets | +| `inventory network-forward list` | List network forwards | +| `inventory network-integration list` | List network integrations | +| `inventory network-load-balancer list` | List network load balancers | +| `inventory network-peer list` | List network peers | +| `inventory network-zone list` | List network zones | | `inventory profile list` | List profiles | | `inventory project list` | List projects | | `inventory image list` | List images | +| `inventory query --cluster ` | Cross-resource tree view (v0.3.0+) | ### Updates @@ -683,6 +760,8 @@ operations-center provisioning cluster factory-reset oc-cluster | `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) | | `admin os service show ` | Service config | | `admin os debug log` | Debug logs | +| `admin os debug processes` | System process list | +| `admin os debug secureboot` | Secure Boot debug info | All commands prefixed with `operations-center`. @@ -713,6 +792,19 @@ curl -sk --cert ~/.config/incus/client.crt \ | GET | `/1.0/provisioning/clusters` | List clusters | | POST | `/1.0/provisioning/clusters` | Form cluster | | GET | `/1.0/inventory/instances` | List all instances | +| GET | `/1.0/inventory/instance/` | Instance details | +| GET | `/1.0/provisioning/servers/` | Server details + hardware inventory | + +### IncusOS REST API (via Incus proxy) + +IncusOS nodes expose their own REST API through the Incus proxy at the `/os/` +prefix on port 8443: + +| Method | Endpoint | Description | +|--------|----------|-------------| +| GET | `/os/1.0/system/storage` | Storage config (scrub_schedule) | +| PUT | `/os/1.0/system/storage` | Update storage config | +| GET | `/os/1.0` | IncusOS system info | --- @@ -771,7 +863,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}' ## Known Limitations (Tested Findings) -### Confirmed limitations (v0.2.2) +### Confirmed limitations (v0.3.0) - **No brownfield adoption**: nodes must boot from OC-provisioned ISO - **No real-time inventory sync**: requires explicit `cluster resync` @@ -869,20 +961,24 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}' ## Open Research Items -These are observed behaviors whose root cause is not yet confirmed: - - **OC reboot failure mechanism** (resolved): guest reboot is safe on standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover - in ~50s). The failure is OC-specific: the OC agent runs on every boot and - fails with configuration errors (invalid crontab), re-registration - conflicts (constraint violation), or cluster quorum loss (peers in boot - loops). The root cause is NOT dqlite corruption or TPM issues. + in ~50s). The failure is OC-specific: the OC agent pushes config via the + IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid + values (e.g., cron expression where Go duration is expected) crash the + daemon. Root cause is NOT dqlite corruption or TPM issues. -- **apply_defaults conflict with OC Terraform**: when nodes have - `apply_defaults: true`, OC's post-clustering Terraform step fails with - "already in trust store" / "not in pending state" errors. The cluster forms - successfully despite these errors. Unclear whether skipping `apply_defaults` - would allow clean Terraform completion (untested). +- **apply_defaults conflict with OC Terraform** (resolved): `apply_defaults: false` + is now the tested and recommended approach. OC's Terraform handles storage pool, + network, and cert creation cleanly when nodes start without defaults. + `apply_defaults: true` also works but produces non-fatal Terraform errors. + +- **IncusOS scrub_schedule bug** (workaround found, upstream filed as #843): + ~50% of deploys hit an empty `scrub_schedule` in `state.txt`. Root cause is + likely a race condition between REST API startup and `registerJobs()`, or a + zero-value encoding bug in `encodeHelper()`. Workaround: `PUT /os/1.0/system/storage` + with `{"config":{"scrub_schedule":"0 4 * * 0"}}`. `incusos-proxmox` now + auto-heals this proactively on every deploy. - **Stale server entries after out-of-band changes**: when cluster members are removed via `incus cluster remove` (not through OC), stale entries persist in @@ -893,6 +989,10 @@ These are observed behaviors whose root cause is not yet confirmed: to pick up changes made via the `incus` CLI. No webhook or event-driven sync. This applies to instance creation, migration, evacuation, and deletion. +- **Node failure detection**: OC does not detect node failures. `provisioning + server show` continues to report `Status: ready` for OFFLINE/crashed nodes. + Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent. + --- ## References