OC v0.3.0 comprehensive test: retry logic, auto-heal, apply_defaults validation

Phase 0-10 of OC test plan completed:
- Add --retries N flag to incusos-proxmox (default 3, configurable boot retries)
- Add fix_scrub_schedule() auto-heal for IncusOS crontab bug via REST API
- Validate apply_defaults: false for OC-managed nodes (recommended approach)
- Test full OC lifecycle: deploy, provision, cluster, workloads, migration,
  evacuation, node failure/recovery, inventory query, REST API
- Update CLAUDE.md: OC v0.3.0 commands, --retries, scrub fix, node recovery
- Update operations-center-guide.md: v0.3.0, apply_defaults, scrub fix,
  node failure simulation, IncusOS REST API docs, inventory query

Key findings:
- apply_defaults: false lets OC Terraform handle everything cleanly
- fix_scrub_schedule() proactively heals ~50% crontab boot failure
- OC does not detect node failures (always shows "ready")
- inventory query (v0.3.0) provides cross-resource tree views
- Node crash recovery: auto-rejoin in ~60s, containers auto-start

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -419,8 +419,14 @@ incu-contrib/
   as Incus: copy `client.crt` and `client.key` from `~/.config/incus/`).
 - **Port**: 8443 (same as Incus on IncusOS) for API, CLI, and web UI.
 - **Browser access**: requires PKCS#12 client certificate (`client.pfx`)
-  imported into the browser.
+  imported into the browser. Web UI is a React SPA at `/ui/`.
-- **OC is under active development** (v0.2.2). Commands and APIs may change.
+- **OC is under active development** (v0.3.0). Commands and APIs may change.
+  Use `operations-center --version` (not `version` subcommand) to check.
+- **v0.3.0 new commands**: `provisioning channel` (add/list/show),
+  `cluster update/update-certificate/rename`, expanded `inventory`
+  (network-acl, address-sets, load-balancers, peers, zones, integrations,
+  forwards, storage-buckets, storage-volumes), `inventory query` (cross-resource
+  tree view with filters and Go templates), `system certificate set`.
 - The `--doctor` command on `incusos-proxmox` reports whether the CLI is
   installed.
 
@@ -452,10 +458,11 @@ incu-contrib/
   ```
   OC handles: `core.https_address` → cluster enable → joins → seed
   application → Terraform config. Adds `meshbr0` network.
-- **apply_defaults conflict** (tested): if SEED_DATA has `apply_defaults: true`,
+- **apply_defaults: false is recommended** (tested): use `apply_defaults: false`
-  nodes already have storage pool/network/cert. OC's Terraform fails with
+  for OC-managed nodes. OC's Terraform handles storage pool, network, and cert
-  "already exists" errors but the cluster forms successfully. Use
+  creation cleanly. With `apply_defaults: true`, nodes already have these
-  `apply_defaults: false` for OC-managed nodes to avoid this.
+  resources and OC's Terraform fails with "already exists" errors (cluster still
+  forms, but Terraform artifacts are empty).
 
 #### Tested limitations
 
@@ -468,9 +475,13 @@ incu-contrib/
   crash the daemon. See "IncusOS boot failure" section below for full analysis.
   Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
 - **No cluster member state tracking** -- OC always shows `ready` even for
-  EVACUATED/OFFLINE nodes
+  EVACUATED/OFFLINE nodes. Does not detect node crashes (Incus heartbeat
+  detects failure in ~40s; OC has no equivalent).
 - **Stale entries** from out-of-band cluster changes persist after resync
 - **Server removal blocked** if server is part of an OC cluster
+- **Node failure recovery**: Proxmox hard-stop simulates crash. After restart,
+  node auto-rejoins cluster in ~60s. Containers auto-start. Works cleanly
+  if crontab bug doesn't hit (auto-healed by `fix_scrub_schedule()`).
 
 - See `notes/operations-center-guide.md` for full tested OC reference.
 
@@ -490,6 +501,12 @@ incu-contrib/
 - **`--verbose` / `-v`**: shows detailed output (tool paths, API calls). Default
   output is concise (step names + results). `--quiet` suppresses everything
   except warnings and errors.
+- **`--retries N`**: number of stop+start retries for VMs that fail to boot
+  (port 8443 not reachable). Default: 3. `--retries 0` disables retries.
+  Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443.
+  With ~50% boot failure rate from the crontab bug, 3 retries gives ~93.75%
+  success rate. After retries, `fix_scrub_schedule()` proactively heals the
+  root cause via the IncusOS REST API.
 
 ### IncusOS first-boot sequence (observed via console screenshots)
 
@@ -627,6 +644,13 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
 - **Detection**: `phase_install` checks port 8443 after starting from disk
   (30s wait + up to 60s polling). If first check fails, automatic retry
   adds ~100s. Failed VMs after retry are reported with remediation.
+- **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes
+  `fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on
+  every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix
+  proxies to the IncusOS daemon API through Incus. Safe to call on every
+  node — returns early if schedule is already set. Called on both initial
+  success and retry success paths.
+- **Filed upstream**: IncusOS issue #843.
 - **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`,
   `updateChecker()`, `startup()`, `firstBootActions()`),
   `incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`,

incusos/README.md

@@ -295,6 +295,7 @@ This script ensures all settings are correct, preventing install failures.
 -c, --cert FILE         Client certificate PEM file (default: auto-detect)
     --no-cert           Skip certificate injection
     --vm NAME           Operate on a single VM only
+    --retries N         Boot retries for first-boot failures (default: 3)
 -y, --yes               Skip confirmation prompts
     --proxmox FILE      Proxmox connection config (default: auto-discover)
     --doctor            Check tool versions and environment health

incusos/examples/lab-oc-nodes.yaml

@@ -23,12 +23,12 @@ defaults:
 vms:
   - name: oc-node-01
     app: incus
-    apply_defaults: true
+    apply_defaults: false
 
   - name: oc-node-02
     app: incus
-    apply_defaults: true
+    apply_defaults: false
 
   - name: oc-node-03
     app: incus
-    apply_defaults: true
+    apply_defaults: false

incusos/incusos-proxmox

@@ -135,6 +135,7 @@ ${BOLD}OPTIONS${RESET}
     ${BOLD}-c, --cert${RESET} FILE         Client certificate PEM file
         ${BOLD}--no-cert${RESET}            Skip certificate injection
         ${BOLD}--vm${RESET} NAME            Operate on a single VM only
+        ${BOLD}--retries${RESET} N          Boot retries for first-boot failures (default: 3)
     ${BOLD}-y, --yes${RESET}               Skip confirmation prompts
         ${BOLD}--doctor${RESET}             Check tool versions and environment health
         ${BOLD}--cleanup${RESET}            Destroy VMs defined in config (managed only)
@@ -304,6 +305,7 @@ LIST_LABS=false
 DRY_RUN=false
 QUIET=false
 VERBOSE=false
+INSTALL_RETRIES=3
 PROXMOX_CONFIG=""
 CONFIG_FILE=""
 
@@ -342,6 +344,14 @@ parse_args() {
                 YES=true
                 shift
                 ;;
+            --retries)
+                INSTALL_RETRIES="${2:?'--retries requires a number'}"
+                if ! [[ "$INSTALL_RETRIES" =~ ^[0-9]+$ ]]; then
+                    error "--retries must be a non-negative integer"
+                    exit 1
+                fi
+                shift 2
+                ;;
             --status)
                 PHASE="status"
                 shift
@@ -3221,6 +3231,72 @@ register_remote() {
     fi
 }
 
+# ---------------------------------------------------------------------------
+# fix_scrub_schedule -- proactively fix empty scrub_schedule via IncusOS API
+# ---------------------------------------------------------------------------
+# IncusOS has an intermittent bug where state.txt is written with an empty
+# scrub_schedule field (the encoder skips zero values). When registerJobs()
+# runs, gocron.IsValid("") fails and the daemon crashes in a loop. The REST
+# API is briefly reachable during each cycle (Incus starts before the
+# scheduler), so we can fix it via PUT /os/1.0/system/storage.
+#
+# This function is safe to call on every node -- if scrub_schedule is already
+# set correctly, the GET check returns early without making any changes.
+#
+# Usage: fix_scrub_schedule <ip> <name>
+# Returns: 0 on success or not needed, 1 on failure
+fix_scrub_schedule() {
+    local ip="$1"
+    local name="$2"
+
+    # Need client cert for the IncusOS API
+    if [[ -z "$CERT_FILE_RESOLVED" ]]; then
+        detail "No client certificate available — skipping scrub_schedule check"
+        return 0
+    fi
+
+    local cert="$CERT_FILE_RESOLVED"
+    local key="${CERT_FILE_RESOLVED%.crt}.key"
+
+    if [[ ! -f "$key" ]]; then
+        detail "Client key not found at ${key} — skipping scrub_schedule check"
+        return 0
+    fi
+
+    # GET current storage config
+    local response
+    response=$(curl -sk --connect-timeout 5 --max-time 10 \
+        --cert "$cert" --key "$key" \
+        "https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
+        detail "Could not query IncusOS storage API on '${name}' — skipping scrub_schedule check"
+        return 0
+    }
+
+    # Check if scrub_schedule is empty or missing
+    local current_schedule
+    current_schedule=$(printf '%s' "$response" | \
+        python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('metadata',{}).get('config',{}).get('scrub_schedule',''))" 2>/dev/null) || current_schedule=""
+
+    if [[ -n "$current_schedule" ]]; then
+        detail "scrub_schedule on '${name}' is already set: ${current_schedule}"
+        return 0
+    fi
+
+    # scrub_schedule is empty — fix it
+    local put_response
+    put_response=$(curl -sk --connect-timeout 5 --max-time 10 \
+        --cert "$cert" --key "$key" \
+        -X PUT -H "Content-Type: application/json" \
+        -d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
+        "https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
+        warn "Failed to fix scrub_schedule on '${name}' via IncusOS API"
+        return 1
+    }
+
+    info "Fixed empty scrub_schedule on '${name}' via IncusOS API"
+    return 0
+}
+
 # ---------------------------------------------------------------------------
 # Phase 5: Install
 # ---------------------------------------------------------------------------
@@ -3431,53 +3507,66 @@ phase_install() {
 
         echo ""
         if [[ "$healthy" == true ]]; then
+            # Proactively fix empty scrub_schedule (IncusOS crontab bug).
+            # Safe to call on every node -- returns early if already set.
+            fix_scrub_schedule "$ip" "$name" || true
             success "VM '${name}' installed and running at ${ip}"
             register_remote "$name" "$ip"
         else
-            # Retry once: IncusOS has an intermittent "invalid crontab expression"
+            # Retry loop: IncusOS has an intermittent "invalid crontab expression"
             # bug (~50% first-boot hit rate) where registerJobs() fails during
-            # startup(). Most likely a race condition between the REST API and
+            # startup(). A stop+start cycle resolves it because the hard power-off
-            # startup sequence, or a gocron library edge case. A stop+start
+            # may discard the corrupt state.txt before it's committed to disk,
-            # cycle resolves it because the transient condition clears on retry.
+            # allowing initialize() to recreate it cleanly on next boot.
-            if [[ -n "$ip" ]]; then
+            local retry=0
-                warn "VM '${name}': port 8443 not reachable at ${ip} — retrying (stop+start)"
+            local retry_ok=false
-            else
+            while [[ $retry -lt $INSTALL_RETRIES ]]; do
-                warn "VM '${name}': could not detect IP — retrying (stop+start)"
+                retry=$((retry + 1))
-            fi
+                if [[ -n "$ip" ]]; then
-            info "Stopping VM for retry..."
+                    warn "VM '${name}': port 8443 not reachable at ${ip} — retry ${retry}/${INSTALL_RETRIES} (stop+start)"
-            pve_stop_vm "$vmid" || true
+                else
-            sleep 10
+                    warn "VM '${name}': could not detect IP — retry ${retry}/${INSTALL_RETRIES} (stop+start)"
-            info "Starting VM (retry)..."
+                fi
-            pve_start_vm "$vmid" || true
+                info "Stopping VM for retry..."
-            sleep 30
+                pve_stop_vm "$vmid" || true
+                sleep 10
+                info "Starting VM (retry ${retry}/${INSTALL_RETRIES})..."
+                pve_start_vm "$vmid" || true
+                sleep 30
 
-            # Re-detect IP (may have changed after restart)
+                # Re-detect IP (may have changed after restart)
-            ip=$(pve_vm_ip "$vmid") || ip=""
+                ip=$(pve_vm_ip "$vmid") || ip=""
 
-            local retry_healthy=false
+                if [[ -n "$ip" ]]; then
-            if [[ -n "$ip" ]]; then
+                    local retry_poll=0
-                local retry_poll=0
+                    while [[ $retry_poll -lt 12 ]]; do
-                while [[ $retry_poll -lt 12 ]]; do
+                        if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
-                    if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
+                            retry_ok=true
-                        retry_healthy=true
+                            break
-                        break
+                        fi
-                    fi
+                        sleep 5
-                    sleep 5
+                        retry_poll=$((retry_poll + 1))
-                    retry_poll=$((retry_poll + 1))
+                    done
-                done
+                fi
-            fi
 
-            if [[ "$retry_healthy" == true ]]; then
+                if [[ "$retry_ok" == true ]]; then
-                success "VM '${name}' recovered after retry — running at ${ip}"
+                    # Proactively fix empty scrub_schedule (IncusOS crontab bug).
-                register_remote "$name" "$ip"
+                    fix_scrub_schedule "$ip" "$name" || true
-            elif [[ -n "$ip" ]]; then
+                    success "VM '${name}' recovered after retry ${retry}/${INSTALL_RETRIES} — running at ${ip}"
-                warn "VM '${name}' installed but port 8443 not reachable at ${ip} (even after retry)"
+                    register_remote "$name" "$ip"
-                warn "Likely IncusOS boot failure (check Proxmox console for errors)"
+                    break
-                warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}"
+                fi
-                install_failures=$((install_failures + 1))
+            done
-            else
+
-                warn "VM '${name}' installed but could not detect IP (even after retry)"
+            if [[ "$retry_ok" != true ]]; then
-                warn "Check Proxmox console for status"
+                if [[ -n "$ip" ]]; then
+                    warn "VM '${name}' installed but port 8443 not reachable at ${ip} (after ${INSTALL_RETRIES} retries)"
+                    warn "Likely IncusOS boot failure (check Proxmox console for errors)"
+                    warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}"
+                else
+                    warn "VM '${name}' installed but could not detect IP (after ${INSTALL_RETRIES} retries)"
+                    warn "Check Proxmox console for status"
+                fi
                 install_failures=$((install_failures + 1))
             fi
         fi

notes/operations-center-guide.md

@@ -5,7 +5,7 @@ a centralized management layer for IncusOS/Incus deployments. OC handles
 provisioning, clustering, inventory, and updates through a REST API, CLI, and
 web UI.
 
-All commands tested with Operations Center v0.2.2 and IncusOS build 202602200553
+All commands tested with Operations Center v0.3.0 and IncusOS build 202602200553
 on Proxmox VE 9.1.5 (nested virtualization on Intel).
 
 ---
@@ -36,10 +36,10 @@ containing a provisioning token, then self-register on first boot.
 
 ```bash
 # Check for latest release at https://github.com/FuturFusion/operations-center/releases
-wget https://github.com/FuturFusion/operations-center/releases/download/v0.2.2/operations-center_linux_x86_64
+wget https://github.com/FuturFusion/operations-center/releases/download/v0.3.0/operations-center_linux_x86_64
 chmod +x operations-center_linux_x86_64
 sudo mv operations-center_linux_x86_64 /usr/local/bin/operations-center
-operations-center version
+operations-center --version  # Note: use --version, not 'version' subcommand
 ```
 
 ### Set up OC client certificates
@@ -233,15 +233,15 @@ defaults:
 vms:
   - name: oc-node-01
     app: incus
-    apply_defaults: true
+    apply_defaults: false   # OC handles storage/network via Terraform
 
   - name: oc-node-02
     app: incus
-    apply_defaults: true
+    apply_defaults: false
 
   - name: oc-node-03
     app: incus
-    apply_defaults: true
+    apply_defaults: false
 ```
 
 **VMID note**: if your Proxmox API token is scoped to a resource pool, it
@@ -316,29 +316,35 @@ OC orchestrates the full cluster formation:
 4. Applies the application seed (injects client cert)
 5. Runs Terraform/OpenTofu for post-cluster configuration
 
-### apply_defaults conflict (tested finding)
+### apply_defaults: false (tested, recommended)
 
-When using `apply_defaults: true` in the SEED_DATA (via `incusos-proxmox`),
+**Use `apply_defaults: false`** for OC-managed nodes. This is now the tested
-each node already has a `local` ZFS storage pool, `incusbr0` network bridge,
+and recommended approach. With `apply_defaults: false`:
-and the client certificate. OC's Terraform post-clustering step then tries to
+
-create the same resources and fails with:
+- Nodes still listen on port 8443 and trust preseed certificates
+- The underlying ZFS dataset (`local/incus`) still exists
+- But no Incus storage pool, network, or profile devices are created
+- OC's Terraform handles all resource creation cleanly during cluster formation
+
+**Tested result**: cluster forms cleanly with `apply_defaults: false`. OC's
+Terraform creates the storage pool (`local`), networks (`incusbr0`, `meshbr0`),
+and injects the client certificate via `--application-seed-config`. The only
+Terraform error is "Certificate already in trust store" when the cert was also
+injected via SEED_DATA -- avoid this by only injecting the cert through the
+application seed config, not the SEED_DATA.
+
+### apply_defaults: true (also works, with caveats)
+
+When using `apply_defaults: true`, each node already has a `local` ZFS storage
+pool, `incusbr0` network bridge, and the client certificate. OC's Terraform
+then fails with:
 
 - "Certificate already in trust store"
 - "Network is not in pending state"
 - "Storage pool is not in pending state"
 
-**The cluster itself forms successfully** despite these Terraform errors. All
+**The cluster itself forms successfully** despite these Terraform errors.
-3 nodes join the cluster correctly. The Terraform errors are non-fatal because
+However, `cluster artifact list` will be empty since Terraform didn't complete.
-the resources already exist. However, the `cluster artifact list` will be empty
-since Terraform didn't complete.
-
-**Recommended approaches** for future deployments:
-1. **Keep apply_defaults** (current tested approach): cluster forms fine, ignore
-   Terraform errors. Skip `--application-seed-config` since the cert is already
-   injected by the seed.
-2. **Skip apply_defaults**: deploy nodes without `apply_defaults: true` and
-   without cert injection in the seed. Let OC handle everything via Terraform.
-   This is the "clean" approach but hasn't been tested yet.
 
 ### Verify cluster (tested)
 
@@ -433,6 +439,59 @@ The proper lifecycle for maintenance is:
 4. Wait for IncusOS to boot (~60-90s)
 5. `incus cluster restore <remote>:<member> --force`
 
+### IncusOS scrub_schedule fix via API (tested)
+
+IncusOS has an intermittent bug (~50% of deploys) where `state.txt` is written
+with an empty `scrub_schedule` field. The `registerJobs()` function calls
+`gocron.IsValid("")` which fails, crashing the daemon in a loop. The REST API
+is briefly reachable during each crash cycle (Incus starts before the scheduler).
+
+**Fix via IncusOS REST API** (through the Incus proxy at `/os/` prefix):
+
+```bash
+# Check current storage config
+curl -sk --cert ~/.config/incus/client.crt \
+     --key ~/.config/incus/client.key \
+     https://<NODE_IP>:8443/os/1.0/system/storage
+
+# If scrub_schedule is empty, fix it
+curl -sk --cert ~/.config/incus/client.crt \
+     --key ~/.config/incus/client.key \
+     -X PUT -H "Content-Type: application/json" \
+     -d '{"config":{"scrub_schedule":"0 4 * * 0"}}' \
+     https://<NODE_IP>:8443/os/1.0/system/storage
+```
+
+**Key details**:
+- The `/os/` prefix is required — this proxies to the IncusOS daemon API
+- PATCH returns 501 (not implemented); use PUT instead
+- `incus admin os system storage edit` fails because it reads the current
+  (corrupt) config and validates it before allowing edits
+- `incusos-proxmox` now includes `fix_scrub_schedule()` which proactively
+  fixes this on every deployed node (safe — returns early if already set)
+- Filed upstream as IncusOS issue #843
+
+### Node failure and recovery (tested)
+
+Proxmox hard-stop simulates node crash. Incus cluster detects failure via
+heartbeat timeout (~30-40s). Node auto-rejoins after Proxmox restart (~60s boot):
+
+```bash
+# Simulate crash: Proxmox API stop
+curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
+
+# After ~40s: incus cluster list shows OFFLINE with heartbeat info
+
+# Restart: Proxmox API start
+curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
+
+# After ~60s boot: node auto-rejoins cluster (ONLINE)
+# Containers on the node auto-start after recovery
+```
+
+**OC does not detect node failures**: `provisioning server show` still shows
+`Status: ready` for an OFFLINE/crashed node. `Last Seen` timestamp is stale.
+
 **OC does not track Incus cluster member state**: the server list always
 shows `ready` status even when the Incus cluster reports `EVACUATED`.
 
@@ -542,6 +601,7 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
 # 2. Destroy broken VM via Proxmox (stop + delete)
 
 # 3. Redeploy via incusos-proxmox --iso (single-node config)
+#    Use apply_defaults: false for OC-managed nodes
 ./incusos/incusos-proxmox --iso /tmp/IncusOS-oc.iso --yes /tmp/redeploy-node-03.yaml
 
 # 4. New node auto-registers with OC (same hostname, new IP)
@@ -550,18 +610,10 @@ printf "yes\n" | incus cluster remove oc-node-01:oc-node-03 --force
 
 # 5. Manual cluster join (since node was force-removed)
 incus config set oc-node-03: core.https_address <NEW_IP>:8443
-# Remove apply_defaults resources
+# With apply_defaults: false, no cleanup needed (no pool/network to delete)
-incus config unset oc-node-03: storage.backups_volume
-incus config unset oc-node-03: storage.images_volume
-incus storage volume delete oc-node-03:local backups
-incus storage volume delete oc-node-03:local images
-incus profile device remove oc-node-03:default root
-incus profile device remove oc-node-03:default eth0
-incus storage delete oc-node-03:local
-incus network delete oc-node-03:incusbr0
 # Generate token and join
 incus cluster add oc-node-01:oc-node-03
-printf '\n\nyes\n\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
+printf '\n\nyes\nlocal/incus\nlocal/incus\n' | incus cluster join oc-node-01: oc-node-03:
 # Note: extra \n for meshbr0 tunnel.mesh.interface prompt (added by OC clustering)
 
 # 6. Fix remote after cert change
@@ -569,6 +621,21 @@ incus remote remove oc-node-03
 incus remote add oc-node-03 https://<NEW_IP>:8443 --accept-certificate
 ```
 
+### Node failure simulation (tested)
+
+Proxmox hard-stop simulates node crash. Incus detects via heartbeat (~40s):
+
+```bash
+# Simulate crash
+curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/stop" ...
+# incus cluster list shows OFFLINE after ~40s
+
+# Restart
+curl -sk -X POST ".../nodes/pve/qemu/<VMID>/status/start" ...
+# Node auto-rejoins after ~60s boot. Containers auto-start.
+# OC still shows "ready" (does not detect node failures)
+```
+
 ### Factory reset (not tested)
 
 ```bash
@@ -650,10 +717,20 @@ operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
 | `inventory instance list` | List all instances |
 | `inventory instance show <UUID>` | Instance details (by UUID, not name) |
 | `inventory storage-pool list` | List storage pools |
+| `inventory storage-volume list` | List storage volumes |
+| `inventory storage-bucket list` | List storage buckets |
 | `inventory network list` | List networks |
+| `inventory network-acl list` | List network ACLs |
+| `inventory network-address-set list` | List network address sets |
+| `inventory network-forward list` | List network forwards |
+| `inventory network-integration list` | List network integrations |
+| `inventory network-load-balancer list` | List network load balancers |
+| `inventory network-peer list` | List network peers |
+| `inventory network-zone list` | List network zones |
 | `inventory profile list` | List profiles |
 | `inventory project list` | List projects |
 | `inventory image list` | List images |
+| `inventory query --cluster <name>` | Cross-resource tree view (v0.3.0+) |
 
 ### Updates
 
@@ -683,6 +760,8 @@ operations-center provisioning cluster factory-reset oc-cluster <token-uuid>
 | `admin os service list` | List services (iscsi, lvm, nvme, ovn, tailscale, ...) |
 | `admin os service show <name>` | Service config |
 | `admin os debug log` | Debug logs |
+| `admin os debug processes` | System process list |
+| `admin os debug secureboot` | Secure Boot debug info |
 
 All commands prefixed with `operations-center`.
 
@@ -713,6 +792,19 @@ curl -sk --cert ~/.config/incus/client.crt \
 | GET | `/1.0/provisioning/clusters` | List clusters |
 | POST | `/1.0/provisioning/clusters` | Form cluster |
 | GET | `/1.0/inventory/instances` | List all instances |
+| GET | `/1.0/inventory/instance/<UUID>` | Instance details |
+| GET | `/1.0/provisioning/servers/<name>` | Server details + hardware inventory |
+
+### IncusOS REST API (via Incus proxy)
+
+IncusOS nodes expose their own REST API through the Incus proxy at the `/os/`
+prefix on port 8443:
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/os/1.0/system/storage` | Storage config (scrub_schedule) |
+| PUT | `/os/1.0/system/storage` | Update storage config |
+| GET | `/os/1.0` | IncusOS system info |
 
 ---
 
@@ -771,7 +863,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
 
 ## Known Limitations (Tested Findings)
 
-### Confirmed limitations (v0.2.2)
+### Confirmed limitations (v0.3.0)
 
 - **No brownfield adoption**: nodes must boot from OC-provisioned ISO
 - **No real-time inventory sync**: requires explicit `cluster resync`
@@ -869,20 +961,24 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
 
 ## Open Research Items
 
-These are observed behaviors whose root cause is not yet confirmed:
-
 - **OC reboot failure mechanism** (resolved): guest reboot is safe on
   standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
-  in ~50s). The failure is OC-specific: the OC agent runs on every boot and
+  in ~50s). The failure is OC-specific: the OC agent pushes config via the
-  fails with configuration errors (invalid crontab), re-registration
+  IncusOS REST API that gets persisted to `state.txt`. On reboot, invalid
-  conflicts (constraint violation), or cluster quorum loss (peers in boot
+  values (e.g., cron expression where Go duration is expected) crash the
-  loops). The root cause is NOT dqlite corruption or TPM issues.
+  daemon. Root cause is NOT dqlite corruption or TPM issues.
 
-- **apply_defaults conflict with OC Terraform**: when nodes have
+- **apply_defaults conflict with OC Terraform** (resolved): `apply_defaults: false`
-  `apply_defaults: true`, OC's post-clustering Terraform step fails with
+  is now the tested and recommended approach. OC's Terraform handles storage pool,
-  "already in trust store" / "not in pending state" errors. The cluster forms
+  network, and cert creation cleanly when nodes start without defaults.
-  successfully despite these errors. Unclear whether skipping `apply_defaults`
+  `apply_defaults: true` also works but produces non-fatal Terraform errors.
-  would allow clean Terraform completion (untested).
+
+- **IncusOS scrub_schedule bug** (workaround found, upstream filed as #843):
+  ~50% of deploys hit an empty `scrub_schedule` in `state.txt`. Root cause is
+  likely a race condition between REST API startup and `registerJobs()`, or a
+  zero-value encoding bug in `encodeHelper()`. Workaround: `PUT /os/1.0/system/storage`
+  with `{"config":{"scrub_schedule":"0 4 * * 0"}}`. `incusos-proxmox` now
+  auto-heals this proactively on every deploy.
 
 - **Stale server entries after out-of-band changes**: when cluster members are
   removed via `incus cluster remove` (not through OC), stale entries persist in
@@ -893,6 +989,10 @@ These are observed behaviors whose root cause is not yet confirmed:
   to pick up changes made via the `incus` CLI. No webhook or event-driven sync.
   This applies to instance creation, migration, evacuation, and deletion.
 
+- **Node failure detection**: OC does not detect node failures. `provisioning
+  server show` continues to report `Status: ready` for OFFLINE/crashed nodes.
+  Incus cluster detects failures via heartbeat (~40s), but OC has no equivalent.
+
 ---
 
 ## References