Correct reboot diagnosis: OC-specific, not dqlite/TPM corruption

Volume testing with live Proxmox infrastructure proved that guest reboot
does NOT break standalone IncusOS nodes. Tested:
- 4 consecutive Proxmox stop/start cycles: all clean, data intact
- Single-node guest reboot: recovered in ~50s, containers auto-started
- Simultaneous 3-node guest reboot: all recovered, data integrity verified

The "reboot breaks IncusOS" issue is OC-specific. Console errors showed:
- Node 1: "failed to parse update frequency" (OC pushed bad crontab)
- Node 2: stuck at "starting application" (waiting for peers in loops)
- Node 3: "constraint violation" (OC re-registering existing server)

Updated all 8 references across CLAUDE.md, operations-center-guide.md,
and TESTING.md to reflect the corrected diagnosis.

Also fixed one more bare return bug in do_list_labs (empty pool path).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-21 00:48:06 +01:00
parent 37d63c35a6
commit ca660585dc
4 changed files with 46 additions and 29 deletions

View File

@ -430,10 +430,13 @@ incu-contrib/
#### Tested limitations #### Tested limitations
- **Inventory is NOT real-time** -- requires explicit `cluster resync` - **Inventory is NOT real-time** -- requires explicit `cluster resync`
- **OC reboot BREAKS IncusOS on Proxmox** -- guest reboot likely corrupts - **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe
the dqlite/Incus database through unclean shutdown. OS boots (pingable) on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in
but Incus daemon never starts. Open research item. Only fix: destroy ~50s with data intact). The failure is OC-specific: the OC agent runs on
and redeploy. Use Proxmox stop/start instead. boot and fails with errors like "invalid crontab expression" (bad update
config), "constraint violation" (re-registration of existing server), or
Incus stuck at "starting application" (waiting for peers in boot loops).
Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
- **No cluster member state tracking** -- OC always shows `ready` even for - **No cluster member state tracking** -- OC always shows `ready` even for
EVACUATED/OFFLINE nodes EVACUATED/OFFLINE nodes
- **Stale entries** from out-of-band cluster changes persist after resync - **Stale entries** from out-of-band cluster changes persist after resync

View File

@ -832,11 +832,13 @@ operations-center inventory instance list # shows relocated workloads
incus cluster restore oc-node-01:oc-node-01 --force incus cluster restore oc-node-01:oc-node-01 --force
``` ```
**WARNING**: do NOT use `operations-center provisioning server system reboot`. **WARNING**: do NOT use `operations-center provisioning server system reboot`
It sends a guest-level reboot that likely corrupts the dqlite/Incus database on OC-managed nodes. Guest reboot is safe on standalone IncusOS (tested:
through unclean shutdown on Proxmox VMs. The node becomes pingable (OS boots) simultaneous 3-node reboot, all recover in ~50s with data intact). But on
but Incus daemon never starts (port 8443 closed). This is an open research OC-managed nodes, the OC agent runs on boot and fails with errors like
item. The node must be destroyed/redeployed. "invalid crontab expression", "constraint violation" (re-registration), or
Incus hangs at "starting application" (peers in boot loops). The node must
be destroyed/redeployed.
### Cleanup ### Cleanup
@ -889,7 +891,7 @@ operations-center remote remove oc-lab 2>/dev/null
| Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable | | Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable |
| OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root | | OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root |
| OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` | | OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` |
| OC reboot kills IncusOS node (port 8443 closed forever) | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy VM. Use Proxmox stop/start instead of OC reboot | | OC reboot kills OC-managed node (port 8443 closed) | OC agent boot failures: invalid crontab, re-registration conflict, or peers in boot loops (standalone reboot is safe) | Destroy and redeploy OC-managed VM. Guest reboot works fine on standalone IncusOS |
| OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync <name>` | | OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync <name>` |
| OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist | | OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist |
| OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management | | OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management |

View File

@ -3561,7 +3561,7 @@ print(desc)
if [[ ${#lab_configs[@]} -eq 0 ]]; then if [[ ${#lab_configs[@]} -eq 0 ]]; then
info "No managed labs found" info "No managed labs found"
[[ $unmanaged -gt 0 ]] && detail "${unmanaged} unmanaged VMs in pool" [[ $unmanaged -gt 0 ]] && detail "${unmanaged} unmanaged VMs in pool"
return return 0
fi fi
local sorted_configs local sorted_configs

View File

@ -407,16 +407,25 @@ operations-center provisioning server system reboot <name> # WARNING: see bel
operations-center provisioning server system poweroff <name> operations-center provisioning server system poweroff <name>
``` ```
**WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot` **WARNING -- OC reboot breaks OC-managed nodes on Proxmox**: the `system
command sends a guest-level (ACPI) reboot. On Proxmox VMs, this likely reboot` command sends a guest-level (ACPI) reboot. On standalone IncusOS
corrupts the dqlite/Incus database through unclean shutdown. The IncusOS nodes, guest reboot works perfectly (tested: simultaneous 3-node reboot,
node becomes pingable (TPM disk decryption succeeds, OS boots) but the all recover in ~50s with containers and data intact). The failure is
Incus daemon never starts (port 8443 never opens). Even a Proxmox-level **OC-specific**: the OC agent runs on every boot and can fail with:
stop/start does not recover the node. The only fix is to destroy and - "failed to parse update frequency - invalid crontab expression" (OC
redeploy the VM. **Note**: this is an open research item -- the exact pushed invalid update config)
failure mechanism is unconfirmed. - "error from operations center: forbidden: failed creating server:
constraint violation" (OC trying to re-register an existing server)
- Incus stuck at "starting application name=incus" (cluster peers are in
boot loops, so quorum cannot form)
**Safe reboot alternative**: use Proxmox stop/start instead of OC reboot. These failures cause boot loops or hung services. Even Proxmox stop/start
does not recover because the OC state on disk remains broken. The only
fix is to destroy and redeploy the VM.
**Safe reboot**: Proxmox stop/start and guest reboot both work on
standalone (non-OC) IncusOS nodes. Avoid OC `system reboot` on
OC-managed nodes.
The proper lifecycle for maintenance is: The proper lifecycle for maintenance is:
1. `incus cluster evacuate <remote>:<member> --force` 1. `incus cluster evacuate <remote>:<member> --force`
2. Proxmox stop the VM 2. Proxmox stop the VM
@ -770,8 +779,9 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
EVACUATED or OFFLINE nodes EVACUATED or OFFLINE nodes
- **Stale server entries**: out-of-band cluster changes (via `incus` CLI) - **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
create stale entries that `resync` doesn't fix create stale entries that `resync` doesn't fix
- **OC reboot breaks Proxmox VMs**: guest reboot likely corrupts dqlite/Incus - **OC reboot breaks OC-managed Proxmox VMs**: OC agent boot failures cause
database (open research item -- OS boots but Incus daemon doesn't start) boot loops or hung services (not a dqlite or TPM issue -- standalone nodes
reboot fine)
- **Token seed format quirk**: requires structured YAML with section keys, - **Token seed format quirk**: requires structured YAML with section keys,
not flat format not flat format
- **Expired tokens not auto-cleaned**: remain in list after expiry - **Expired tokens not auto-cleaned**: remain in list after expiry
@ -801,7 +811,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` | | OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors | | Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` | | `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
| OC reboot kills IncusOS node | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. | | OC reboot kills OC-managed node | OC agent boot failures: invalid crontab, re-registration constraint violation, or cluster peers in boot loops | Destroy and redeploy the VM. Guest reboot is safe on standalone (non-OC) nodes. |
| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. | | OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) | | OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` | | Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
@ -850,7 +860,8 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
- Manual: full control, no extra server, works with any Incus version - Manual: full control, no extra server, works with any Incus version
- OC hybrid: auto-registration, single-command clustering, centralized - OC hybrid: auto-registration, single-command clustering, centralized
inventory/updates, but requires OC server + OC-provisioned ISOs inventory/updates, but requires OC server + OC-provisioned ISOs
- OC reboot command is **dangerous** on Proxmox (likely corrupts dqlite database) - OC reboot command is **dangerous** on Proxmox (OC agent fails on boot, causing
loops -- standalone reboot is safe)
- OC inventory requires manual resync (not real-time) - OC inventory requires manual resync (not real-time)
- Out-of-band cluster changes (via `incus` CLI) create stale OC state - Out-of-band cluster changes (via `incus` CLI) create stale OC state
@ -860,11 +871,12 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
These are observed behaviors whose root cause is not yet confirmed: These are observed behaviors whose root cause is not yet confirmed:
- **OC reboot failure mechanism**: guest-level (ACPI) reboot on Proxmox VMs - **OC reboot failure mechanism** (resolved): guest reboot is safe on
causes the Incus daemon to never start. The OS boots (node is pingable, standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
TPM disk decryption succeeds) but port 8443 never opens. Likely cause: in ~50s). The failure is OC-specific: the OC agent runs on every boot and
dqlite database corruption from unclean shutdown. Proxmox stop/start does fails with configuration errors (invalid crontab), re-registration
not recover the node. Needs investigation with IncusOS debug logs. conflicts (constraint violation), or cluster quorum loss (peers in boot
loops). The root cause is NOT dqlite corruption or TPM issues.
- **apply_defaults conflict with OC Terraform**: when nodes have - **apply_defaults conflict with OC Terraform**: when nodes have
`apply_defaults: true`, OC's post-clustering Terraform step fails with `apply_defaults: true`, OC's post-clustering Terraform step fails with