Update boot failure analysis: error is persistent within VM session

Verified across 6 observe-deploy runs (3/6 hit crontab error, all 6 PASS).
Key finding: the error persists across systemd daemon restarts because the
deferred Save() writes corrupt state.txt (encoder skips zero-value
ScrubSchedule). Recovery requires Proxmox stop+start (hard power-off may
lose uncommitted corrupt write). Updated CLAUDE.md accordingly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-21 18:16:34 +01:00
parent 1499dd7072
commit b8e7a7ca62
1 changed files with 12 additions and 6 deletions

View File

@ -612,12 +612,18 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
when empty (to preserve the default through encode/decode cycles),
(3) add mutex protection on State fields shared between API handlers
and startup().
- **Recovery via restart**: `phase_install` in `incusos-proxmox` now
automatically retries once (stop → 10s wait → start → re-check).
After the first crash, the deferred `s.Save()` writes state. On the
second boot, systemd restarts the daemon; if the root cause was
transient (race condition or gocron edge case), the second boot
succeeds.
- **Error is persistent within a VM session**: verified by screenshots —
when the crontab error hits, the deferred `s.Save()` writes corrupt
state (version 7, ScrubSchedule missing because encoder skips zero
values). Subsequent systemd daemon restarts read the corrupt state
and crash again. The error does NOT self-heal via daemon restart alone.
- **Recovery via Proxmox stop+start**: `phase_install` in `incusos-proxmox`
retries once with a full Proxmox stop (hard VM power-off) → 10s wait →
start. The hard power-off may cause uncommitted filesystem writes
(including the corrupt state.txt) to be lost, restoring the correct
state from `initialize()`. This works because IncusOS's data partition
has a journal commit interval (~5s) and the corrupt Save() may not have
been committed before the VM was killed.
- **Detection**: `phase_install` checks port 8443 after starting from disk
(30s wait + up to 60s polling). If first check fails, automatic retry
adds ~100s. Failed VMs after retry are reported with remediation.