diff --git a/.gitignore b/.gitignore index 250da7a..112ad4f 100644 --- a/.gitignore +++ b/.gitignore @@ -31,3 +31,6 @@ proxmox.yaml # Environment secrets env + +# Observational deploy output (screenshots, logs) +observe-runs/ diff --git a/CLAUDE.md b/CLAUDE.md index 6bd8fe5..65c20cf 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -19,6 +19,8 @@ incu-contrib/ │ ├── incusos-seed # Seed archive generator (cross-platform: Linux + macOS) │ ├── incusos-proxmox # Declarative Proxmox VM deployment + lab lifecycle │ ├── lab-test # Guided lab validation (12 test phases) +│ ├── observe-deploy # Single-VM deploy with rapid console screenshots +│ ├── observe-runs/ # Screenshot output from observe-deploy (gitignored) │ ├── proxmox.yaml # Proxmox connection config (gitignored, contains credentials) │ ├── ../env # PROXMOX_TOKEN_SECRET (gitignored); source with: source env │ ├── TESTING.md # Testing guide for incusos-proxmox and lab-test @@ -180,6 +182,23 @@ incu-contrib/ | 920-929 | OC server standalone | | 930-939 | Advanced / heterogeneous | +### Resource constraints for multi-lab + +- **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node + cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource + where you can actually run out. +- **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default + (`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk. + LZ4 compression provides ~1.5-1.8x ratio on OS data. +- **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs + can share CPUs without contention. +- **Pre-deploy checks**: `incusos-proxmox` warns during preflight if + requested RAM exceeds available host RAM (API method only). +- **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to + run the full deploy pipeline (auto-accepts with `--yes`). +- **`--resources` actual disk**: shows actual vs allocated disk for pool VMs + (via storage content API) and notes thin provisioning for ZFS/LVM-thin. + ### Incus clustering via remotes - **Cluster formation** is done entirely through the `incus` CLI using remotes. @@ -443,10 +462,10 @@ incu-contrib/ - **Inventory is NOT real-time** -- requires explicit `cluster resync` - **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in - ~50s with data intact). The failure is OC-specific: the OC agent runs on - boot and fails with errors like "invalid crontab expression" (bad update - config), "constraint violation" (re-registration of existing server), or - Incus stuck at "starting application" (waiting for peers in boot loops). + ~50s with data intact). The failure is OC-specific: the OC agent pushes + config via the IncusOS REST API that gets persisted to `state.txt`. On + reboot, invalid values (e.g., cron expression where Go duration is expected) + crash the daemon. See "IncusOS boot failure" section below for full analysis. Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe. - **No cluster member state tracking** -- OC always shows `ready` even for EVACUATED/OFFLINE nodes @@ -472,6 +491,194 @@ incu-contrib/ output is concise (step names + results). `--quiet` suppresses everything except warnings and errors. +### IncusOS first-boot sequence (observed via console screenshots) + +Observed using `observe-deploy` with 2-second screenshot intervals. The +complete lifecycle from Proxmox VM start to port 8443 ready: + +**Phase 1: ISO boot and installation (~60-85s)** +1. UEFI firmware → "Guest has not initialized the display (yet)" (~2s) +2. IncusOS boot menu: "IncusOS 202602200553" selected, "Boot in 1 s." (~10s) +3. IncusOS Install TUI: "Starting install of IncusOS to local disk" (~24s) + - "Installing IncusOS source=/dev/mapper/sr0 target=/dev/sda" + - "Cloning GPT partitions" → progress bar → complete +4. `force_reboot: true` triggers **SysRq-B** (raw kernel reboot via + `/proc/sysrq-trigger`, preceded by `unix.Sync()` + 5s sleep). Not + a graceful systemd reboot. This is intentional — the install + environment is minimal and may not have systemd running. +5. VM reboots but ISO still attached → boots from ISO again +6. "System check error: install media detected, but the system is already + installed; please remove USB/CDROM and reboot the system" (~40s). + IncusOS daemon sleeps **1 hour** at this error, then exits. The + `state.txt` on the installed disk (scsi0 data partition) is **NOT + touched** — this boot runs from the ISO rootfs (tmpfs). +7. Stays at this error until blockstat detects stable writes (3×5s = 15s idle) + +**Phase 2: Transition (~15s)** +1. Proxmox stops VM +2. Remove ide2 (ISO) and ide3 (seed), set `boot: order=scsi0` +3. Start VM from disk + +**Phase 3: First boot from disk (~50s to port 8443)** +1. UEFI → "IncusOS is starting..." (~8s) +2. `state.LoadOrCreate()` → state.txt doesn't exist → `initialize()` sets + defaults: `ScrubSchedule="0 4 * * 0"`, `CheckFrequency="6h"`, + `Channel="stable"` → `Save()` writes state.txt to data partition +3. "Auto-generating encryption recovery key, this may take a few seconds" (~10s) +4. "System is starting up machine-id=... mode=production" (~10s after key) +5. "Bringing up the network" (immediate) +6. "Downloading SecureBoot update" → "Applying Secure Boot certificate" (~12s) +7. "Downloading application update application=incus" + progress bar (~33-38s) +8. "Bringing up the local storage" (after download) +9. "Starting application" → "Initializing application" → TLS cert fingerprint +10. `registerJobs()` validates ScrubSchedule with gocron +11. "System is ready" — port 8443 now reachable + +**Total: ~130-215s from VM create to port 8443 ready.** The biggest variable +is the application download (33-38s on first boot; skipped if already cached). + +**Important**: `DoInstall()` does NOT create `state.txt`. It writes the +IncusOS image to disk, processes seed files (including deleting `install.yaml` +from the target's seed partition via `CleanupPostInstall()`), and triggers +the SysRq-B reboot. The `state.txt` is created exclusively on first boot +from the installed disk by `state.LoadOrCreate()` → `initialize()`. + +### IncusOS boot failure (crontab / update frequency error) + +- **Symptom**: "ERROR invalid crontab expression" appears on the console + during first boot. Intermittent — observed in ~50% of deploys (2/4 + observational deploys). Error does NOT correlate with `update.yaml` + presence/absence. +- **Two distinct error paths in IncusOS**: + - `registerJobs()` validates `ScrubSchedule` (5-field crontab expression + like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire + daemon exits — **this is the fatal one**. Port 8443 never opens + (or opens briefly during the 15-second error sleep). + - `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()` + format like `"6h"`). If invalid, it logs an error but does **not** crash. +- **Not always fatal** (key finding from observational testing): + - The REST API starts BEFORE `startup()` completes. Applications (Incus) + start at step ~16 of `startup()`, while `registerJobs()` is at step ~19. + When Incus starts before the scheduler crashes, port 8443 is available + during the 15-second error sleep (before `os.Exit(1)`). + - `observe-deploy` detects port 8443 during this window → reports PASS. + - "System is ready" is NOT shown on console (only appears after + `registerJobs()` succeeds). +- **Source code analysis** (root cause investigation): + - `state.txt` is created FRESH on first boot from disk by + `LoadOrCreate()` → `initialize()`. The `initialize()` function + correctly sets `ScrubSchedule = "0 4 * * 0"`. + - The "install media detected" boot (#2, ISO still attached) runs from + the ISO rootfs. It creates state.txt in tmpfs, NOT on the installed + disk. The installed disk's state.txt is untouched. + - `DoInstall()` does not create state.txt. It only writes the OS image + and processes seed files. + - **The encoder skips zero values**: `encodeHelper()` returns early on + `v.IsZero()`. An empty ScrubSchedule string is NOT written to file. + If state.txt is read with a missing ScrubSchedule, the field stays + at Go's zero value (empty string) → `gocron.IsValid("")` fails. + - **Non-atomic state writes**: `Save()` uses `os.WriteFile()` which + truncates then writes. Crash during write → partial file. + - **Race window**: The REST API server starts BEFORE `startup()` + completes. Applications start before `registerJobs()`. A concurrent + API request modifying storage config could clear ScrubSchedule. + - **Most likely root cause**: either (a) a race condition between the + REST API/application startup and `registerJobs()` that clears + `ScrubSchedule`, or (b) a gocron library edge case where + `IsValid("0 4 * * 0", time.UTC, time.Now())` intermittently fails. + Both explain the ~50% hit rate and lack of correlation with seed + content. +- **Proposed upstream fix** (for `incus-os` project): + ```go + // In registerJobs(): validate and fall back to default + func registerJobs(s *state.State) error { + schedule := s.System.Storage.Config.ScrubSchedule + if schedule == "" { + schedule = "0 4 * * 0" + s.System.Storage.Config.ScrubSchedule = schedule + } + err := s.JobScheduler.RegisterJob(zfs.PoolScrubJob, schedule, zfs.ScrubAllPools) + if err != nil { + // Fall back to default instead of crashing the daemon + slog.Warn("Invalid scrub schedule, using default", "schedule", schedule) + s.System.Storage.Config.ScrubSchedule = "0 4 * * 0" + return s.JobScheduler.RegisterJob(zfs.PoolScrubJob, "0 4 * * 0", zfs.ScrubAllPools) + } + return nil + } + ``` + Additional recommendations: (1) use atomic writes in `Save()` (write + to temp file + `os.Rename()`), (2) always encode `ScrubSchedule` even + when empty (to preserve the default through encode/decode cycles), + (3) add mutex protection on State fields shared between API handlers + and startup(). +- **Recovery via restart**: `phase_install` in `incusos-proxmox` now + automatically retries once (stop → 10s wait → start → re-check). + After the first crash, the deferred `s.Save()` writes state. On the + second boot, systemd restarts the daemon; if the root cause was + transient (race condition or gocron edge case), the second boot + succeeds. +- **Detection**: `phase_install` checks port 8443 after starting from disk + (30s wait + up to 60s polling). If first check fails, automatic retry + adds ~100s. Failed VMs after retry are reported with remediation. +- **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`, + `updateChecker()`, `startup()`, `firstBootActions()`), + `incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`, + `ErrInvalidCronTab`), `incus-osd/internal/state/file.go` + (`LoadOrCreate()`, `Save()`, `initialize()`), + `incus-osd/internal/state/encode.go` (`encodeHelper()` zero-value skip), + `incus-osd/internal/install/install.go` (`DoInstall()`, + `rebootUponDeviceRemoval()` SysRq-B). + +### Proxmox SSH root access — strict rules + +Root SSH access to the Proxmox host is available for **diagnostics only**. +The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`. + +**MANDATORY RULES — violation of any rule is unacceptable:** + +1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM + console screenshots via `qm monitor screendump`. No other use + without explicit user instruction. +2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869) + or VMs in the IncusLab pool that were created by our scripts. +3. **No modifications**: NEVER run any command that modifies, stops, starts, + or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`, + `pct` commands, `zfs` commands, `systemctl` commands, or ANY write + operation on the host. +4. **Do not touch the dev VM**: the user's dev VM runs on the same host. + Do not interact with it in any way — do not even `qm status` it. +5. **No config access**: do not read or modify `/etc/pve/`, storage configs, + network configs, user/ACL settings, or any host-level configuration. +6. **Transparency**: every SSH command is visible in tool output. If the + user rejects a command, do not retry it. +7. **Only during active tests**: only SSH during test runs where test VMs + have been deployed, and only to screenshot those test VMs. + +**Permitted commands (exhaustive list):** +```bash +# Take screenshot of a test VM (VMID in 850-869 range) +sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ + root@ "echo 'screendump /tmp/vm--screen.ppm' | qm monitor " + +# Retrieve screenshot +sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \ + root@:/tmp/vm--screen.ppm /tmp/ + +# Cleanup screenshot on remote +sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \ + root@ "rm -f /tmp/vm--screen.ppm" +``` + +**Technical notes:** +- QEMU's `screendump` is root-only in PVE 9's HMP permission model. + API tokens cannot execute it regardless of privileges. +- Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng, + which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`). +- PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`: + `python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"` + PNG output is ~20 KB and can be read directly by Claude Code (multimodal). + ## Coding conventions for scripts - **Shell**: bash with `set -euo pipefail` diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index 556d3b4..b28b124 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -1937,13 +1937,35 @@ pve_stop_vm() { esac } -# Destroy a VM +# Destroy a VM (waits for async task to complete) pve_destroy_vm() { local vmid="$1" case "$PVE_METHOD" in ssh) pve_run "qm destroy ${vmid} --purge" ;; - api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" >/dev/null ;; + api) + local destroy_response + destroy_response=$(pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1" 2>/dev/null) || return 1 + # VM destruction is async -- wait for the UPID task to finish + # to avoid VMID/disk reuse issues in rapid destroy+create cycles. + local upid + upid=$(echo "$destroy_response" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',''))" 2>/dev/null) || upid="" + if [[ -n "$upid" ]]; then + local encoded_upid + encoded_upid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${upid}', safe=''))" 2>/dev/null) + local wait_elapsed=0 + while [[ $wait_elapsed -lt 30 ]]; do + sleep 2 + wait_elapsed=$((wait_elapsed + 2)) + local task_status + task_status=$(pve_api GET "/nodes/${PVE_NODE}/tasks/${encoded_upid}/status" 2>/dev/null | \ + python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null) || task_status="" + if [[ "$task_status" == "stopped" ]]; then + break + fi + done + fi + ;; esac } @@ -2458,6 +2480,70 @@ for s in data.get('data', []): errors=$((errors + 1)) fi + # 6. Host resource feasibility (API method only) + if [[ "$PVE_METHOD" == "api" ]] && [[ "$DRY_RUN" != true ]] && [[ ${#VM_MEMORY[@]} -gt 0 ]]; then + local total_ram_mib=0 + local total_disk_gib=0 + local total_cores=0 + local ri=0 + while [[ $ri -lt ${#VM_MEMORY[@]} ]]; do + total_ram_mib=$((total_ram_mib + VM_MEMORY[$ri])) + total_disk_gib=$((total_disk_gib + VM_DISK[$ri])) + total_cores=$((total_cores + VM_CORES[$ri])) + ri=$((ri + 1)) + done + + local res_node_status + res_node_status=$(pve_api GET "/nodes/${PVE_NODE}/status" 2>/dev/null) || res_node_status="" + if [[ -n "$res_node_status" ]]; then + local resource_lines + resource_lines=$(python3 -c " +import json, sys +data = json.loads('''${res_node_status}''').get('data', {}) +mem = data.get('memory', {}) +mem_total = mem.get('total', 0) / (1024**3) +mem_used = mem.get('used', 0) / (1024**3) +mem_free = mem_total - mem_used +cpu_count = data.get('cpuinfo', {}).get('cpus', 0) +req_ram = ${total_ram_mib} / 1024 +req_cores = ${total_cores} + +# RAM check +if mem_free > 0: + pct = req_ram / mem_free * 100 + if req_ram > mem_free: + print(f'error|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free -- not enough RAM') + elif pct > 70: + print(f'warn|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free ({pct:.0f}% of free) -- tight fit') + else: + print(f'ok|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free ({pct:.0f}% of free)') + +# CPU (info only) +print(f'ok|CPU: {req_cores} cores requested, {cpu_count} available') +" 2>/dev/null) || resource_lines="" + + local line + while IFS= read -r line; do + [[ -z "$line" ]] && continue + local level="${line%%|*}" + local msg="${line#*|}" + case "$level" in + error) + error "$msg" + errors=$((errors + 1)) + ;; + warn) + warn "$msg" + warnings=$((warnings + 1)) + ;; + ok) + detail "$msg" + ;; + esac + done <<< "$resource_lines" + fi + fi + if [[ $errors -gt 0 ]]; then error "${errors} pre-flight check(s) failed" exit 1 @@ -2826,6 +2912,20 @@ confirm() { fi } +# Like confirm(), but returns 0/1 instead of exiting on "no" +confirm_yn() { + local message="$1" + + if [[ "$YES" == true ]]; then + return 0 + fi + + echo -n -e "${BOLD}${message} [y/N] ${RESET}" + local reply + read -r reply + [[ "$reply" =~ ^[Yy]$ ]] +} + # --------------------------------------------------------------------------- # Phase 1: Validate # --------------------------------------------------------------------------- @@ -3130,6 +3230,7 @@ phase_install() { echo "" local count=${#VM_NAMES[@]} + local install_failures=0 local i=0 while [[ $i -lt $count ]]; do local name="${VM_NAMES[$i]}" @@ -3306,28 +3407,92 @@ phase_install() { continue fi - # 5. Wait for services to come up + # 5. Wait for services to come up and verify health info "Waiting for services to start..." sleep 30 - # 6. Try to detect IP + # 6. Detect IP and check port 8443 local ip ip=$(pve_vm_ip "$vmid") || ip="" - echo "" + local healthy=false if [[ -n "$ip" ]]; then + # Poll port 8443 for up to 60s (the 30s sleep above + 60s = 90s total) + local poll=0 + while [[ $poll -lt 12 ]]; do + if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then + healthy=true + break + fi + sleep 5 + poll=$((poll + 1)) + done + fi + + echo "" + if [[ "$healthy" == true ]]; then success "VM '${name}' installed and running at ${ip}" register_remote "$name" "$ip" else - success "VM '${name}' installed and running" - info "Check Proxmox console for IP address" + # Retry once: IncusOS has an intermittent "invalid crontab expression" + # bug (~50% first-boot hit rate) where registerJobs() fails during + # startup(). Most likely a race condition between the REST API and + # startup sequence, or a gocron library edge case. A stop+start + # cycle resolves it because the transient condition clears on retry. + if [[ -n "$ip" ]]; then + warn "VM '${name}': port 8443 not reachable at ${ip} — retrying (stop+start)" + else + warn "VM '${name}': could not detect IP — retrying (stop+start)" + fi + info "Stopping VM for retry..." + pve_stop_vm "$vmid" || true + sleep 10 + info "Starting VM (retry)..." + pve_start_vm "$vmid" || true + sleep 30 + + # Re-detect IP (may have changed after restart) + ip=$(pve_vm_ip "$vmid") || ip="" + + local retry_healthy=false + if [[ -n "$ip" ]]; then + local retry_poll=0 + while [[ $retry_poll -lt 12 ]]; do + if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then + retry_healthy=true + break + fi + sleep 5 + retry_poll=$((retry_poll + 1)) + done + fi + + if [[ "$retry_healthy" == true ]]; then + success "VM '${name}' recovered after retry — running at ${ip}" + register_remote "$name" "$ip" + elif [[ -n "$ip" ]]; then + warn "VM '${name}' installed but port 8443 not reachable at ${ip} (even after retry)" + warn "Likely IncusOS boot failure (check Proxmox console for errors)" + warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}" + install_failures=$((install_failures + 1)) + else + warn "VM '${name}' installed but could not detect IP (even after retry)" + warn "Check Proxmox console for status" + install_failures=$((install_failures + 1)) + fi fi echo "" i=$((i + 1)) done - success "Installation phase complete" + if [[ $install_failures -gt 0 ]]; then + warn "Installation phase complete: ${install_failures} VM(s) failed health check" + warn "Failed VMs are likely stuck in an IncusOS boot loop" + warn "Re-run deployment to redeploy, or use --cleanup to start fresh" + else + success "Installation phase complete" + fi } # --------------------------------------------------------------------------- @@ -3393,6 +3558,48 @@ phase_lab_up() { local count=${#VM_NAMES[@]} local started=0 local skipped=0 + local missing=0 + + # First pass: check how many VMs exist + if [[ "$DRY_RUN" != true ]]; then + local i=0 + while [[ $i -lt $count ]]; do + if ! pve_vm_exists "${VM_VMIDS[$i]}"; then + missing=$((missing + 1)) + fi + i=$((i + 1)) + done + + # If ALL VMs are missing, offer to deploy + if [[ $missing -eq $count ]]; then + echo "" + warn "No VMs found -- lab has not been deployed yet" + if confirm_yn "Deploy lab now?"; then + echo "" + phase_prepare + echo "" + phase_upload + echo "" + phase_create + echo "" + phase_install + echo "" + phase_post_deploy + return + else + info "Aborted" + return + fi + fi + + # If SOME VMs are missing, warn but don't offer auto-deploy + if [[ $missing -gt 0 ]]; then + warn "${missing} of ${count} VMs not found -- partial state detected" + warn "Use --cleanup then redeploy, or deploy missing VMs manually" + fi + fi + + # Second pass: start existing VMs local i=0 while [[ $i -lt $count ]]; do local name="${VM_NAMES[$i]}" @@ -3505,6 +3712,7 @@ print(f' Uptime: {days}d {hours}h') python3 -c " import json, sys data = json.loads('''${storage_list}''').get('data', []) +thin_types = {'zfspool', 'lvmthin'} for s in sorted(data, key=lambda x: x.get('storage', '')): name = s.get('storage', '?') total = s.get('total', 0) / (1024**3) @@ -3513,10 +3721,11 @@ for s in sorted(data, key=lambda x: x.get('storage', '')): pct = (used / total * 100) if total > 0 else 0 stype = s.get('type', '?') content = s.get('content', '?') + thin = ' [thin]' if stype in thin_types else '' if total > 0: - print(f' {name:<16} {stype:<12} {used:.1f}G / {total:.1f}G ({pct:.0f}%) -- {content}') + print(f' {name:<16} {stype:<12} {used:.1f}G / {total:.1f}G ({pct:.0f}%) -- {content}{thin}') else: - print(f' {name:<16} {stype:<12} (no capacity info) -- {content}') + print(f' {name:<16} {stype:<12} (no capacity info) -- {content}{thin}') " 2>/dev/null || warn "Failed to parse storage list" fi @@ -3540,6 +3749,57 @@ print(f' VMs: {len(vms)} total ({running} running, {stopped} stopped)') print(f' Allocated: {total_mem:.1f} GiB RAM, {total_disk:.0f} GiB disk') " 2>/dev/null || warn "Failed to parse pool data" fi + + # Actual disk usage via storage content API + if [[ -n "$pool_data" ]]; then + local storage_content + storage_content=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_STORAGE}/content" 2>/dev/null) || storage_content="" + local storage_status_json + storage_status_json=$(echo "$storage_list" | python3 -c " +import json, sys +data = json.load(sys.stdin).get('data', []) +for s in data: + if s.get('storage') == '${PVE_STORAGE}': + print(json.dumps(s)) + break +" 2>/dev/null) || storage_status_json="" + + if [[ -n "$storage_content" ]] && [[ -n "$pool_data" ]]; then + python3 -c " +import json, sys +content = json.loads('''${storage_content}''').get('data', []) +pool = json.loads('''${pool_data}''').get('data', {}) +vm_vmids = {str(m.get('vmid', '')) for m in pool.get('members', []) if m.get('type') == 'qemu'} + +# Sum actual (used) bytes for volumes belonging to pool VMs +actual_bytes = 0 +allocated_bytes = 0 +for vol in content: + vmid = str(vol.get('vmid', '')) + if vmid in vm_vmids: + actual_bytes += vol.get('used', 0) + allocated_bytes += vol.get('size', 0) + +if allocated_bytes > 0: + actual_gib = actual_bytes / (1024**3) + allocated_gib = allocated_bytes / (1024**3) + ratio = (actual_bytes / allocated_bytes * 100) if allocated_bytes > 0 else 0 + print(f' Actual: {actual_gib:.1f} GiB disk used ({ratio:.0f}% of allocated, thin provisioned)') +" 2>/dev/null || true + fi + + # Storage type note + if [[ -n "$storage_status_json" ]]; then + python3 -c " +import json +s = json.loads('''${storage_status_json}''') +stype = s.get('type', '') +notes = {'zfspool': 'ZFS (thin provisioned, lz4 compression)', 'lvmthin': 'LVM-thin (thin provisioned)'} +if stype in notes: + print(f' Storage: {notes[stype]}') +" 2>/dev/null || true + fi + fi fi echo "" diff --git a/incusos/incusos-seed b/incusos/incusos-seed index 74eb56d..72150be 100755 --- a/incusos/incusos-seed +++ b/incusos/incusos-seed @@ -616,17 +616,13 @@ generate_network_yaml() { } generate_update_yaml() { - # Only generate if non-default values were specified - if [[ "$CHANNEL" == "stable" ]] && [[ "$AUTO_REBOOT" == false ]]; then - return - fi - local file="${WORKDIR}/update.yaml" step "Generating update.yaml" { echo "# Update configuration" + echo "# Always generated to ensure explicit check_frequency on first boot" echo "channel: \"${CHANNEL}\"" echo "auto_reboot: ${AUTO_REBOOT}" echo "check_frequency: \"6h\"" diff --git a/incusos/observe-deploy b/incusos/observe-deploy new file mode 100755 index 0000000..bfe1ce1 --- /dev/null +++ b/incusos/observe-deploy @@ -0,0 +1,1041 @@ +#!/usr/bin/env bash +# observe-deploy - Single-VM IncusOS deploy with rapid console screenshots +# +# Captures the complete IncusOS lifecycle on screen — from ISO boot through +# installation, reboot, first-boot sequence, and port 8443 readiness. Takes +# screenshots every 2 seconds via Proxmox SSH root access (screendump). +# +# Purpose: diagnose boot failures by building a frame-by-frame timeline of +# exactly what IncusOS shows at every stage. NOT a batch test — one VM at +# a time, maximum visibility. +# +# Usage: +# ./observe-deploy # Standard deploy with screenshots +# ./observe-deploy --no-update-yaml # Omit update.yaml from seed +# ./observe-deploy --label run-01 # Custom run label +# ./observe-deploy --vmid 855 # Use a specific VMID +# ./observe-deploy --dry-run # Preview without deploying +# +# Output: timestamped directory with PNG frames + deploy.log +# observe-runs/YYYY-MM-DD_HH-MM-SS_