incus-contrib

Commit Graph

Author	SHA1	Message	Date
Maarten	088baf6807	Fix IncusOS deploy: eliminate crontab bug and TPM corruption Root cause #1: force_reboot triggers SysRq-B intermediate boot that races with IncusOS state initialization, causing ~50% crontab bug rate. Fix: omit force_reboot from seed — VM sits at "please remove media" after install, blockstat detects idle cleanly, no intermediate boot. Validated: 15/15 PASS (100%) on 4-core VMs. Root cause #2: 90s boot timeout was too short for first-boot sysext download under load. Premature retry hard-stops the VM during first boot, corrupting the TPM encryption key permanently. Fix: increase boot timeout to 180s (60s sleep + 120s polling). Changes: - incusos-proxmox: remove --force-reboot from seed generation, increase boot timeout from 90s to 180s, update retry comments - incusos-seed: add --ip/--gateway/--dns flags for static IP seeds, add network.yaml generation - observe-deploy: add --no-force-reboot flag, --runs N mode, VLAN support, scrub_schedule health check, 1s screenshot interval - lab configs: add static IPs for VLAN 69 (192.168.102.x/22) - proxmox.yaml.example: add vlan, gateway, dns fields - CLAUDE.md: document findings, boot timeout, TPM corruption risk Tested: single VM, 3-node cluster, OC + 3 nodes — all 100% success. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-22 21:50:02 +01:00
Maarten	1499dd7072	Add observe-deploy diagnostics, boot failure analysis, and deploy reliability fixes Observational single-VM deploy tool (observe-deploy) captures frame-by-frame console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot failures. Four deploys revealed ~50% "invalid crontab expression" hit rate with no correlation to update.yaml presence. Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go) traced the full crash path: registerJobs() validates ScrubSchedule via gocron at step 19 of startup(), after applications start at step 16 and REST API starts even earlier. Most likely a race condition or gocron edge case. Key fixes: - incusos-proxmox: automatic retry on first-boot failure (stop+start cycle) - incusos-proxmox: pve_destroy_vm now waits for async UPID task completion - observe-deploy: same UPID wait for destroy, full screenshot pipeline - incusos-seed: always generate update.yaml with explicit check_frequency - CLAUDE.md: complete boot sequence timeline, source-level root cause analysis, proposed upstream fix for IncusOS registerJobs() defensive validation Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>	2026-02-21 13:42:51 +01:00

Author

SHA1

Message

Date

Maarten

088baf6807

Fix IncusOS deploy: eliminate crontab bug and TPM corruption

Root cause #1: force_reboot triggers SysRq-B intermediate boot that
races with IncusOS state initialization, causing ~50% crontab bug rate.
Fix: omit force_reboot from seed — VM sits at "please remove media"
after install, blockstat detects idle cleanly, no intermediate boot.
Validated: 15/15 PASS (100%) on 4-core VMs.

Root cause #2: 90s boot timeout was too short for first-boot sysext
download under load. Premature retry hard-stops the VM during first
boot, corrupting the TPM encryption key permanently. Fix: increase
boot timeout to 180s (60s sleep + 120s polling).

Changes:
- incusos-proxmox: remove --force-reboot from seed generation, increase
  boot timeout from 90s to 180s, update retry comments
- incusos-seed: add --ip/--gateway/--dns flags for static IP seeds,
  add network.yaml generation
- observe-deploy: add --no-force-reboot flag, --runs N mode, VLAN
  support, scrub_schedule health check, 1s screenshot interval
- lab configs: add static IPs for VLAN 69 (192.168.102.x/22)
- proxmox.yaml.example: add vlan, gateway, dns fields
- CLAUDE.md: document findings, boot timeout, TPM corruption risk

Tested: single VM, 3-node cluster, OC + 3 nodes — all 100% success.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-22 21:50:02 +01:00

Maarten

1499dd7072

Add observe-deploy diagnostics, boot failure analysis, and deploy reliability fixes

Observational single-VM deploy tool (observe-deploy) captures frame-by-frame
console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot
failures. Four deploys revealed ~50% "invalid crontab expression" hit rate
with no correlation to update.yaml presence.

Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go)
traced the full crash path: registerJobs() validates ScrubSchedule via gocron
at step 19 of startup(), after applications start at step 16 and REST API
starts even earlier. Most likely a race condition or gocron edge case.

Key fixes:
- incusos-proxmox: automatic retry on first-boot failure (stop+start cycle)
- incusos-proxmox: pve_destroy_vm now waits for async UPID task completion
- observe-deploy: same UPID wait for destroy, full screenshot pipeline
- incusos-seed: always generate update.yaml with explicit check_frequency
- CLAUDE.md: complete boot sequence timeline, source-level root cause analysis,
  proposed upstream fix for IncusOS registerJobs() defensive validation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-02-21 13:42:51 +01:00

2 Commits