From ca660585dc21508e6a11ba723911aebf9b470c17 Mon Sep 17 00:00:00 2001 From: Maarten Date: Sat, 21 Feb 2026 00:48:06 +0100 Subject: [PATCH] Correct reboot diagnosis: OC-specific, not dqlite/TPM corruption Volume testing with live Proxmox infrastructure proved that guest reboot does NOT break standalone IncusOS nodes. Tested: - 4 consecutive Proxmox stop/start cycles: all clean, data intact - Single-node guest reboot: recovered in ~50s, containers auto-started - Simultaneous 3-node guest reboot: all recovered, data integrity verified The "reboot breaks IncusOS" issue is OC-specific. Console errors showed: - Node 1: "failed to parse update frequency" (OC pushed bad crontab) - Node 2: stuck at "starting application" (waiting for peers in loops) - Node 3: "constraint violation" (OC re-registering existing server) Updated all 8 references across CLAUDE.md, operations-center-guide.md, and TESTING.md to reflect the corrected diagnosis. Also fixed one more bare return bug in do_list_labs (empty pool path). Co-Authored-By: Claude Opus 4.6 --- CLAUDE.md | 11 +++++--- incusos/TESTING.md | 14 ++++++---- incusos/incusos-proxmox | 2 +- notes/operations-center-guide.md | 48 ++++++++++++++++++++------------ 4 files changed, 46 insertions(+), 29 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index b195a3d..4bab33a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -430,10 +430,13 @@ incu-contrib/ #### Tested limitations - **Inventory is NOT real-time** -- requires explicit `cluster resync` -- **OC reboot BREAKS IncusOS on Proxmox** -- guest reboot likely corrupts - the dqlite/Incus database through unclean shutdown. OS boots (pingable) - but Incus daemon never starts. Open research item. Only fix: destroy - and redeploy. Use Proxmox stop/start instead. +- **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe + on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in + ~50s with data intact). The failure is OC-specific: the OC agent runs on + boot and fails with errors like "invalid crontab expression" (bad update + config), "constraint violation" (re-registration of existing server), or + Incus stuck at "starting application" (waiting for peers in boot loops). + Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe. - **No cluster member state tracking** -- OC always shows `ready` even for EVACUATED/OFFLINE nodes - **Stale entries** from out-of-band cluster changes persist after resync diff --git a/incusos/TESTING.md b/incusos/TESTING.md index f0bb4ca..3301f47 100644 --- a/incusos/TESTING.md +++ b/incusos/TESTING.md @@ -832,11 +832,13 @@ operations-center inventory instance list # shows relocated workloads incus cluster restore oc-node-01:oc-node-01 --force ``` -**WARNING**: do NOT use `operations-center provisioning server system reboot`. -It sends a guest-level reboot that likely corrupts the dqlite/Incus database -through unclean shutdown on Proxmox VMs. The node becomes pingable (OS boots) -but Incus daemon never starts (port 8443 closed). This is an open research -item. The node must be destroyed/redeployed. +**WARNING**: do NOT use `operations-center provisioning server system reboot` +on OC-managed nodes. Guest reboot is safe on standalone IncusOS (tested: +simultaneous 3-node reboot, all recover in ~50s with data intact). But on +OC-managed nodes, the OC agent runs on boot and fails with errors like +"invalid crontab expression", "constraint violation" (re-registration), or +Incus hangs at "starting application" (peers in boot loops). The node must +be destroyed/redeployed. ### Cleanup @@ -889,7 +891,7 @@ operations-center remote remove oc-lab 2>/dev/null | Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable | | OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root | | OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` | -| OC reboot kills IncusOS node (port 8443 closed forever) | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy VM. Use Proxmox stop/start instead of OC reboot | +| OC reboot kills OC-managed node (port 8443 closed) | OC agent boot failures: invalid crontab, re-registration conflict, or peers in boot loops (standalone reboot is safe) | Destroy and redeploy OC-managed VM. Guest reboot works fine on standalone IncusOS | | OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync ` | | OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist | | OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management | diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index e60965d..5ffd1bf 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -3561,7 +3561,7 @@ print(desc) if [[ ${#lab_configs[@]} -eq 0 ]]; then info "No managed labs found" [[ $unmanaged -gt 0 ]] && detail "${unmanaged} unmanaged VMs in pool" - return + return 0 fi local sorted_configs diff --git a/notes/operations-center-guide.md b/notes/operations-center-guide.md index 1e8189f..be56f1e 100644 --- a/notes/operations-center-guide.md +++ b/notes/operations-center-guide.md @@ -407,16 +407,25 @@ operations-center provisioning server system reboot # WARNING: see bel operations-center provisioning server system poweroff ``` -**WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot` -command sends a guest-level (ACPI) reboot. On Proxmox VMs, this likely -corrupts the dqlite/Incus database through unclean shutdown. The IncusOS -node becomes pingable (TPM disk decryption succeeds, OS boots) but the -Incus daemon never starts (port 8443 never opens). Even a Proxmox-level -stop/start does not recover the node. The only fix is to destroy and -redeploy the VM. **Note**: this is an open research item -- the exact -failure mechanism is unconfirmed. +**WARNING -- OC reboot breaks OC-managed nodes on Proxmox**: the `system +reboot` command sends a guest-level (ACPI) reboot. On standalone IncusOS +nodes, guest reboot works perfectly (tested: simultaneous 3-node reboot, +all recover in ~50s with containers and data intact). The failure is +**OC-specific**: the OC agent runs on every boot and can fail with: +- "failed to parse update frequency - invalid crontab expression" (OC + pushed invalid update config) +- "error from operations center: forbidden: failed creating server: + constraint violation" (OC trying to re-register an existing server) +- Incus stuck at "starting application name=incus" (cluster peers are in + boot loops, so quorum cannot form) -**Safe reboot alternative**: use Proxmox stop/start instead of OC reboot. +These failures cause boot loops or hung services. Even Proxmox stop/start +does not recover because the OC state on disk remains broken. The only +fix is to destroy and redeploy the VM. + +**Safe reboot**: Proxmox stop/start and guest reboot both work on +standalone (non-OC) IncusOS nodes. Avoid OC `system reboot` on +OC-managed nodes. The proper lifecycle for maintenance is: 1. `incus cluster evacuate : --force` 2. Proxmox stop the VM @@ -770,8 +779,9 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}' EVACUATED or OFFLINE nodes - **Stale server entries**: out-of-band cluster changes (via `incus` CLI) create stale entries that `resync` doesn't fix -- **OC reboot breaks Proxmox VMs**: guest reboot likely corrupts dqlite/Incus - database (open research item -- OS boots but Incus daemon doesn't start) +- **OC reboot breaks OC-managed Proxmox VMs**: OC agent boot failures cause + boot loops or hung services (not a dqlite or TPM issue -- standalone nodes + reboot fine) - **Token seed format quirk**: requires structured YAML with section keys, not flat format - **Expired tokens not auto-cleaned**: remain in list after expiry @@ -801,7 +811,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}' | OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` | | Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors | | `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove ` then `incus remote add https://IP:8443 --accept-certificate` | -| OC reboot kills IncusOS node | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. | +| OC reboot kills OC-managed node | OC agent boot failures: invalid crontab, re-registration constraint violation, or cluster peers in boot loops | Destroy and redeploy the VM. Guest reboot is safe on standalone (non-OC) nodes. | | OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. | | OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) | | Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove ` | @@ -850,7 +860,8 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}' - Manual: full control, no extra server, works with any Incus version - OC hybrid: auto-registration, single-command clustering, centralized inventory/updates, but requires OC server + OC-provisioned ISOs -- OC reboot command is **dangerous** on Proxmox (likely corrupts dqlite database) +- OC reboot command is **dangerous** on Proxmox (OC agent fails on boot, causing + loops -- standalone reboot is safe) - OC inventory requires manual resync (not real-time) - Out-of-band cluster changes (via `incus` CLI) create stale OC state @@ -860,11 +871,12 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}' These are observed behaviors whose root cause is not yet confirmed: -- **OC reboot failure mechanism**: guest-level (ACPI) reboot on Proxmox VMs - causes the Incus daemon to never start. The OS boots (node is pingable, - TPM disk decryption succeeds) but port 8443 never opens. Likely cause: - dqlite database corruption from unclean shutdown. Proxmox stop/start does - not recover the node. Needs investigation with IncusOS debug logs. +- **OC reboot failure mechanism** (resolved): guest reboot is safe on + standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover + in ~50s). The failure is OC-specific: the OC agent runs on every boot and + fails with configuration errors (invalid crontab), re-registration + conflicts (constraint violation), or cluster quorum loss (peers in boot + loops). The root cause is NOT dqlite corruption or TPM issues. - **apply_defaults conflict with OC Terraform**: when nodes have `apply_defaults: true`, OC's post-clustering Terraform step fails with