Aether deployed on oc-node-01, connected to Incus cluster (trust token)
and Operations Center (mutual TLS). Extracted all 16 built-in help pages
(426 KB) covering cluster management, storage, profiles, instances,
HAProxy LB, firewall ACLs, blueprints, and health monitoring.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Observational single-VM deploy tool (observe-deploy) captures frame-by-frame
console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot
failures. Four deploys revealed ~50% "invalid crontab expression" hit rate
with no correlation to update.yaml presence.
Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go)
traced the full crash path: registerJobs() validates ScrubSchedule via gocron
at step 19 of startup(), after applications start at step 16 and REST API
starts even earlier. Most likely a race condition or gocron edge case.
Key fixes:
- incusos-proxmox: automatic retry on first-boot failure (stop+start cycle)
- incusos-proxmox: pve_destroy_vm now waits for async UPID task completion
- observe-deploy: same UPID wait for destroy, full screenshot pipeline
- incusos-seed: always generate update.yaml with explicit check_frequency
- CLAUDE.md: complete boot sequence timeline, source-level root cause analysis,
proposed upstream fix for IncusOS registerJobs() defensive validation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New lab-advanced.yaml (3-node 6/4/4 core heterogeneous cluster) and
lab-replace.yaml (single replacement node) for advanced migration and
cluster lifecycle testing
- New notes/clustering-guide.md with full clustering walkthrough: cluster
formation, storage pool conflict fix, join workflow, VM live migration
with limits.cpu range fix, multi-vCPU test results (2/3/4 vCPUs),
edge cases (I/O, memory, concurrent), and node replacement lifecycle
- Updated CLAUDE.md with clustering docs, rebalancing config keys,
cluster enable syntax fix (TWO args not one), cluster remove syntax,
agent reconnect timing (3-4s not 1-2s), and lifecycle procedure
- proxmox.yaml.example for Proxmox connection config reference
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Maarten