Observational single-VM deploy tool (observe-deploy) captures frame-by-frame
console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot
failures. Four deploys revealed ~50% "invalid crontab expression" hit rate
with no correlation to update.yaml presence.
Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go)
traced the full crash path: registerJobs() validates ScrubSchedule via gocron
at step 19 of startup(), after applications start at step 16 and REST API
starts even earlier. Most likely a race condition or gocron edge case.
Key fixes:
- incusos-proxmox: automatic retry on first-boot failure (stop+start cycle)
- incusos-proxmox: pve_destroy_vm now waits for async UPID task completion
- observe-deploy: same UPID wait for destroy, full screenshot pipeline
- incusos-seed: always generate update.yaml with explicit check_frequency
- CLAUDE.md: complete boot sequence timeline, source-level root cause analysis,
proposed upstream fix for IncusOS registerJobs() defensive validation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Major changes to incusos-proxmox and incusos-seed to achieve fully automated
IncusOS deployment to Proxmox VE. Fixes multiple blockers discovered during
testing: FAT→ISO 9660 seed format (CD-ROM label detection), preseed YAML
structure (Go yaml:",inline"), disk target auto-detection, install completion
monitoring via blockstat disk I/O, install media ejection, and ARP-based IP
detection for the agent-less IncusOS.
Key changes:
- incusos-seed: add --format iso using genisoimage, fix preseed structure
- incusos-proxmox: ISO seeds, blockstat install detection, media ejection,
ARP-based IP lookup, force_reboot requirement
- Documentation: comprehensive updates to CLAUDE.md, README.md, TESTING.md
- Examples: fix preseed.certificates path in all YAML examples
- Notes: full session progress with all bugs found and fixed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten