Correct reboot diagnosis: OC-specific, not dqlite/TPM corruption
Volume testing with live Proxmox infrastructure proved that guest reboot does NOT break standalone IncusOS nodes. Tested: - 4 consecutive Proxmox stop/start cycles: all clean, data intact - Single-node guest reboot: recovered in ~50s, containers auto-started - Simultaneous 3-node guest reboot: all recovered, data integrity verified The "reboot breaks IncusOS" issue is OC-specific. Console errors showed: - Node 1: "failed to parse update frequency" (OC pushed bad crontab) - Node 2: stuck at "starting application" (waiting for peers in loops) - Node 3: "constraint violation" (OC re-registering existing server) Updated all 8 references across CLAUDE.md, operations-center-guide.md, and TESTING.md to reflect the corrected diagnosis. Also fixed one more bare return bug in do_list_labs (empty pool path). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
37d63c35a6
commit
ca660585dc
11
CLAUDE.md
11
CLAUDE.md
|
|
@ -430,10 +430,13 @@ incu-contrib/
|
|||
#### Tested limitations
|
||||
|
||||
- **Inventory is NOT real-time** -- requires explicit `cluster resync`
|
||||
- **OC reboot BREAKS IncusOS on Proxmox** -- guest reboot likely corrupts
|
||||
the dqlite/Incus database through unclean shutdown. OS boots (pingable)
|
||||
but Incus daemon never starts. Open research item. Only fix: destroy
|
||||
and redeploy. Use Proxmox stop/start instead.
|
||||
- **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe
|
||||
on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in
|
||||
~50s with data intact). The failure is OC-specific: the OC agent runs on
|
||||
boot and fails with errors like "invalid crontab expression" (bad update
|
||||
config), "constraint violation" (re-registration of existing server), or
|
||||
Incus stuck at "starting application" (waiting for peers in boot loops).
|
||||
Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
|
||||
- **No cluster member state tracking** -- OC always shows `ready` even for
|
||||
EVACUATED/OFFLINE nodes
|
||||
- **Stale entries** from out-of-band cluster changes persist after resync
|
||||
|
|
|
|||
|
|
@ -832,11 +832,13 @@ operations-center inventory instance list # shows relocated workloads
|
|||
incus cluster restore oc-node-01:oc-node-01 --force
|
||||
```
|
||||
|
||||
**WARNING**: do NOT use `operations-center provisioning server system reboot`.
|
||||
It sends a guest-level reboot that likely corrupts the dqlite/Incus database
|
||||
through unclean shutdown on Proxmox VMs. The node becomes pingable (OS boots)
|
||||
but Incus daemon never starts (port 8443 closed). This is an open research
|
||||
item. The node must be destroyed/redeployed.
|
||||
**WARNING**: do NOT use `operations-center provisioning server system reboot`
|
||||
on OC-managed nodes. Guest reboot is safe on standalone IncusOS (tested:
|
||||
simultaneous 3-node reboot, all recover in ~50s with data intact). But on
|
||||
OC-managed nodes, the OC agent runs on boot and fails with errors like
|
||||
"invalid crontab expression", "constraint violation" (re-registration), or
|
||||
Incus hangs at "starting application" (peers in boot loops). The node must
|
||||
be destroyed/redeployed.
|
||||
|
||||
### Cleanup
|
||||
|
||||
|
|
@ -889,7 +891,7 @@ operations-center remote remove oc-lab 2>/dev/null
|
|||
| Bash mangles Proxmox API token with `!` | History expansion | Use single quotes around token value or store in variable |
|
||||
| OC token seed fields all `{}` | Flat YAML format used | Use structured format: `install: { force_reboot: true }` not `force_reboot: true` at root |
|
||||
| OC cluster Terraform errors ("already in trust store") | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite errors. Skip `--application-seed-config` or skip `apply_defaults` |
|
||||
| OC reboot kills IncusOS node (port 8443 closed forever) | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy VM. Use Proxmox stop/start instead of OC reboot |
|
||||
| OC reboot kills OC-managed node (port 8443 closed) | OC agent boot failures: invalid crontab, re-registration conflict, or peers in boot loops (standalone reboot is safe) | Destroy and redeploy OC-managed VM. Guest reboot works fine on standalone IncusOS |
|
||||
| OC inventory empty after creating workloads | Inventory not real-time | Run `operations-center provisioning cluster resync <name>` |
|
||||
| OC shows stale server entry after node replacement | Out-of-band cluster change | OC can't detect `incus cluster remove`; stale entries persist |
|
||||
| OC `server remove` fails ("part of cluster") | Server is in OC cluster | Must remove cluster first or handle via OC cluster management |
|
||||
|
|
|
|||
|
|
@ -3561,7 +3561,7 @@ print(desc)
|
|||
if [[ ${#lab_configs[@]} -eq 0 ]]; then
|
||||
info "No managed labs found"
|
||||
[[ $unmanaged -gt 0 ]] && detail "${unmanaged} unmanaged VMs in pool"
|
||||
return
|
||||
return 0
|
||||
fi
|
||||
|
||||
local sorted_configs
|
||||
|
|
|
|||
|
|
@ -407,16 +407,25 @@ operations-center provisioning server system reboot <name> # WARNING: see bel
|
|||
operations-center provisioning server system poweroff <name>
|
||||
```
|
||||
|
||||
**WARNING -- OC reboot breaks IncusOS on Proxmox**: the `system reboot`
|
||||
command sends a guest-level (ACPI) reboot. On Proxmox VMs, this likely
|
||||
corrupts the dqlite/Incus database through unclean shutdown. The IncusOS
|
||||
node becomes pingable (TPM disk decryption succeeds, OS boots) but the
|
||||
Incus daemon never starts (port 8443 never opens). Even a Proxmox-level
|
||||
stop/start does not recover the node. The only fix is to destroy and
|
||||
redeploy the VM. **Note**: this is an open research item -- the exact
|
||||
failure mechanism is unconfirmed.
|
||||
**WARNING -- OC reboot breaks OC-managed nodes on Proxmox**: the `system
|
||||
reboot` command sends a guest-level (ACPI) reboot. On standalone IncusOS
|
||||
nodes, guest reboot works perfectly (tested: simultaneous 3-node reboot,
|
||||
all recover in ~50s with containers and data intact). The failure is
|
||||
**OC-specific**: the OC agent runs on every boot and can fail with:
|
||||
- "failed to parse update frequency - invalid crontab expression" (OC
|
||||
pushed invalid update config)
|
||||
- "error from operations center: forbidden: failed creating server:
|
||||
constraint violation" (OC trying to re-register an existing server)
|
||||
- Incus stuck at "starting application name=incus" (cluster peers are in
|
||||
boot loops, so quorum cannot form)
|
||||
|
||||
**Safe reboot alternative**: use Proxmox stop/start instead of OC reboot.
|
||||
These failures cause boot loops or hung services. Even Proxmox stop/start
|
||||
does not recover because the OC state on disk remains broken. The only
|
||||
fix is to destroy and redeploy the VM.
|
||||
|
||||
**Safe reboot**: Proxmox stop/start and guest reboot both work on
|
||||
standalone (non-OC) IncusOS nodes. Avoid OC `system reboot` on
|
||||
OC-managed nodes.
|
||||
The proper lifecycle for maintenance is:
|
||||
1. `incus cluster evacuate <remote>:<member> --force`
|
||||
2. Proxmox stop the VM
|
||||
|
|
@ -770,8 +779,9 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
|
|||
EVACUATED or OFFLINE nodes
|
||||
- **Stale server entries**: out-of-band cluster changes (via `incus` CLI)
|
||||
create stale entries that `resync` doesn't fix
|
||||
- **OC reboot breaks Proxmox VMs**: guest reboot likely corrupts dqlite/Incus
|
||||
database (open research item -- OS boots but Incus daemon doesn't start)
|
||||
- **OC reboot breaks OC-managed Proxmox VMs**: OC agent boot failures cause
|
||||
boot loops or hung services (not a dqlite or TPM issue -- standalone nodes
|
||||
reboot fine)
|
||||
- **Token seed format quirk**: requires structured YAML with section keys,
|
||||
not flat format
|
||||
- **Expired tokens not auto-cleaned**: remain in list after expiry
|
||||
|
|
@ -801,7 +811,7 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
|
|||
| OC CLI "not authorized" | Wrong certs in `~/.config/operations-center/` | Copy `client.crt` and `client.key` from `~/.config/incus/` |
|
||||
| Cluster Terraform errors | `apply_defaults: true` pre-created resources | Non-fatal -- cluster forms despite "already in trust store" / "not in pending state" errors |
|
||||
| `incus` TLS error after clustering | Cluster cert regenerated | `incus remote remove <r>` then `incus remote add <r> https://IP:8443 --accept-certificate` |
|
||||
| OC reboot kills IncusOS node | Likely dqlite/Incus database corruption from unclean shutdown (open research item) | Destroy and redeploy the VM. Use Proxmox stop/start instead of OC reboot. |
|
||||
| OC reboot kills OC-managed node | OC agent boot failures: invalid crontab, re-registration constraint violation, or cluster peers in boot loops | Destroy and redeploy the VM. Guest reboot is safe on standalone (non-OC) nodes. |
|
||||
| OC stale server entry after node replace | OC doesn't detect out-of-band cluster changes | `cluster resync` doesn't fix stale entries. Must manage cluster lifecycle through OC. |
|
||||
| OC shows old IP for replaced node | Re-registration blocked by existing entry | Remove stale entry (requires cluster dissociation first) |
|
||||
| Token expired but still listed | Tokens don't auto-clean | Remove with `provisioning token remove <UUID>` |
|
||||
|
|
@ -850,7 +860,8 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
|
|||
- Manual: full control, no extra server, works with any Incus version
|
||||
- OC hybrid: auto-registration, single-command clustering, centralized
|
||||
inventory/updates, but requires OC server + OC-provisioned ISOs
|
||||
- OC reboot command is **dangerous** on Proxmox (likely corrupts dqlite database)
|
||||
- OC reboot command is **dangerous** on Proxmox (OC agent fails on boot, causing
|
||||
loops -- standalone reboot is safe)
|
||||
- OC inventory requires manual resync (not real-time)
|
||||
- Out-of-band cluster changes (via `incus` CLI) create stale OC state
|
||||
|
||||
|
|
@ -860,11 +871,12 @@ ip neigh show | grep -i "$MAC" | awk '{print $1}'
|
|||
|
||||
These are observed behaviors whose root cause is not yet confirmed:
|
||||
|
||||
- **OC reboot failure mechanism**: guest-level (ACPI) reboot on Proxmox VMs
|
||||
causes the Incus daemon to never start. The OS boots (node is pingable,
|
||||
TPM disk decryption succeeds) but port 8443 never opens. Likely cause:
|
||||
dqlite database corruption from unclean shutdown. Proxmox stop/start does
|
||||
not recover the node. Needs investigation with IncusOS debug logs.
|
||||
- **OC reboot failure mechanism** (resolved): guest reboot is safe on
|
||||
standalone IncusOS nodes (tested: simultaneous 3-node reboot, all recover
|
||||
in ~50s). The failure is OC-specific: the OC agent runs on every boot and
|
||||
fails with configuration errors (invalid crontab), re-registration
|
||||
conflicts (constraint violation), or cluster quorum loss (peers in boot
|
||||
loops). The root cause is NOT dqlite corruption or TPM issues.
|
||||
|
||||
- **apply_defaults conflict with OC Terraform**: when nodes have
|
||||
`apply_defaults: true`, OC's post-clustering Terraform step fails with
|
||||
|
|
|
|||
Loading…
Reference in New Issue