Add observe-deploy diagnostics, boot failure analysis, and deploy reliability fixes

Observational single-VM deploy tool (observe-deploy) captures frame-by-frame
console screenshots via Proxmox SSH root screendump to diagnose IncusOS boot
failures. Four deploys revealed ~50% "invalid crontab expression" hit rate
with no correlation to update.yaml presence.

Deep analysis of IncusOS source code (main.go, state/, scheduling/, install.go)
traced the full crash path: registerJobs() validates ScrubSchedule via gocron
at step 19 of startup(), after applications start at step 16 and REST API
starts even earlier. Most likely a race condition or gocron edge case.

Key fixes:
- incusos-proxmox: automatic retry on first-boot failure (stop+start cycle)
- incusos-proxmox: pve_destroy_vm now waits for async UPID task completion
- observe-deploy: same UPID wait for destroy, full screenshot pipeline
- incusos-seed: always generate update.yaml with explicit check_frequency
- CLAUDE.md: complete boot sequence timeline, source-level root cause analysis,
  proposed upstream fix for IncusOS registerJobs() defensive validation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.gitignore

@@ -31,3 +31,6 @@ proxmox.yaml
 
 # Environment secrets
 env
+
+# Observational deploy output (screenshots, logs)
+observe-runs/

CLAUDE.md

@@ -19,6 +19,8 @@ incu-contrib/
 │   ├── incusos-seed       # Seed archive generator (cross-platform: Linux + macOS)
 │   ├── incusos-proxmox    # Declarative Proxmox VM deployment + lab lifecycle
 │   ├── lab-test           # Guided lab validation (12 test phases)
+│   ├── observe-deploy     # Single-VM deploy with rapid console screenshots
+│   ├── observe-runs/      # Screenshot output from observe-deploy (gitignored)
 │   ├── proxmox.yaml       # Proxmox connection config (gitignored, contains credentials)
 │   ├── ../env             # PROXMOX_TOKEN_SECRET (gitignored); source with: source env
 │   ├── TESTING.md         # Testing guide for incusos-proxmox and lab-test
@@ -180,6 +182,23 @@ incu-contrib/
   | 920-929 | OC server standalone |
   | 930-939 | Advanced / heterogeneous |
 
+### Resource constraints for multi-lab
+
+- **RAM is the bottleneck**: each IncusOS VM needs 4 GiB minimum. A 3-node
+  cluster = 12 GiB, OC lab (4 VMs) = 28 GiB. RAM is the only resource
+  where you can actually run out.
+- **Storage is not a concern**: Proxmox ZFS uses thin provisioning by default
+  (`sparse` in storage.cfg). 3x 50 GiB VMs use ~7-8 GiB actual disk.
+  LZ4 compression provides ~1.5-1.8x ratio on OS data.
+- **CPU is plentiful**: 4 cores per VM, 20 cores on host. Multiple labs
+  can share CPUs without contention.
+- **Pre-deploy checks**: `incusos-proxmox` warns during preflight if
+  requested RAM exceeds available host RAM (API method only).
+- **`--lab-up` auto-deploy**: if no VMs exist yet, `--lab-up` offers to
+  run the full deploy pipeline (auto-accepts with `--yes`).
+- **`--resources` actual disk**: shows actual vs allocated disk for pool VMs
+  (via storage content API) and notes thin provisioning for ZFS/LVM-thin.
+
 ### Incus clustering via remotes
 
 - **Cluster formation** is done entirely through the `incus` CLI using remotes.
@@ -443,10 +462,10 @@ incu-contrib/
 - **Inventory is NOT real-time** -- requires explicit `cluster resync`
 - **OC reboot breaks OC-managed nodes on Proxmox** -- guest reboot is safe
   on standalone IncusOS (tested: simultaneous 3-node reboot, all recover in
-  ~50s with data intact). The failure is OC-specific: the OC agent runs on
+  ~50s with data intact). The failure is OC-specific: the OC agent pushes
-  boot and fails with errors like "invalid crontab expression" (bad update
+  config via the IncusOS REST API that gets persisted to `state.txt`. On
-  config), "constraint violation" (re-registration of existing server), or
+  reboot, invalid values (e.g., cron expression where Go duration is expected)
-  Incus stuck at "starting application" (waiting for peers in boot loops).
+  crash the daemon. See "IncusOS boot failure" section below for full analysis.
   Fix: destroy and redeploy OC-managed nodes. Proxmox stop/start is safe.
 - **No cluster member state tracking** -- OC always shows `ready` even for
   EVACUATED/OFFLINE nodes
@@ -472,6 +491,194 @@ incu-contrib/
   output is concise (step names + results). `--quiet` suppresses everything
   except warnings and errors.
 
+### IncusOS first-boot sequence (observed via console screenshots)
+
+Observed using `observe-deploy` with 2-second screenshot intervals. The
+complete lifecycle from Proxmox VM start to port 8443 ready:
+
+**Phase 1: ISO boot and installation (~60-85s)**
+1. UEFI firmware → "Guest has not initialized the display (yet)" (~2s)
+2. IncusOS boot menu: "IncusOS 202602200553" selected, "Boot in 1 s." (~10s)
+3. IncusOS Install TUI: "Starting install of IncusOS to local disk" (~24s)
+   - "Installing IncusOS source=/dev/mapper/sr0 target=/dev/sda"
+   - "Cloning GPT partitions" → progress bar → complete
+4. `force_reboot: true` triggers **SysRq-B** (raw kernel reboot via
+   `/proc/sysrq-trigger`, preceded by `unix.Sync()` + 5s sleep). Not
+   a graceful systemd reboot. This is intentional — the install
+   environment is minimal and may not have systemd running.
+5. VM reboots but ISO still attached → boots from ISO again
+6. "System check error: install media detected, but the system is already
+   installed; please remove USB/CDROM and reboot the system" (~40s).
+   IncusOS daemon sleeps **1 hour** at this error, then exits. The
+   `state.txt` on the installed disk (scsi0 data partition) is **NOT
+   touched** — this boot runs from the ISO rootfs (tmpfs).
+7. Stays at this error until blockstat detects stable writes (3×5s = 15s idle)
+
+**Phase 2: Transition (~15s)**
+1. Proxmox stops VM
+2. Remove ide2 (ISO) and ide3 (seed), set `boot: order=scsi0`
+3. Start VM from disk
+
+**Phase 3: First boot from disk (~50s to port 8443)**
+1. UEFI → "IncusOS is starting..." (~8s)
+2. `state.LoadOrCreate()` → state.txt doesn't exist → `initialize()` sets
+   defaults: `ScrubSchedule="0 4 * * 0"`, `CheckFrequency="6h"`,
+   `Channel="stable"` → `Save()` writes state.txt to data partition
+3. "Auto-generating encryption recovery key, this may take a few seconds" (~10s)
+4. "System is starting up machine-id=... mode=production" (~10s after key)
+5. "Bringing up the network" (immediate)
+6. "Downloading SecureBoot update" → "Applying Secure Boot certificate" (~12s)
+7. "Downloading application update application=incus" + progress bar (~33-38s)
+8. "Bringing up the local storage" (after download)
+9. "Starting application" → "Initializing application" → TLS cert fingerprint
+10. `registerJobs()` validates ScrubSchedule with gocron
+11. "System is ready" — port 8443 now reachable
+
+**Total: ~130-215s from VM create to port 8443 ready.** The biggest variable
+is the application download (33-38s on first boot; skipped if already cached).
+
+**Important**: `DoInstall()` does NOT create `state.txt`. It writes the
+IncusOS image to disk, processes seed files (including deleting `install.yaml`
+from the target's seed partition via `CleanupPostInstall()`), and triggers
+the SysRq-B reboot. The `state.txt` is created exclusively on first boot
+from the installed disk by `state.LoadOrCreate()` → `initialize()`.
+
+### IncusOS boot failure (crontab / update frequency error)
+
+- **Symptom**: "ERROR invalid crontab expression" appears on the console
+  during first boot. Intermittent — observed in ~50% of deploys (2/4
+  observational deploys). Error does NOT correlate with `update.yaml`
+  presence/absence.
+- **Two distinct error paths in IncusOS**:
+  - `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
+    like `"0 4 * * 0"`) using `gocron.IsValid()`. If invalid, the entire
+    daemon exits — **this is the fatal one**. Port 8443 never opens
+    (or opens briefly during the 15-second error sleep).
+  - `updateChecker()` validates `CheckFrequency` (Go `time.ParseDuration()`
+    format like `"6h"`). If invalid, it logs an error but does **not** crash.
+- **Not always fatal** (key finding from observational testing):
+  - The REST API starts BEFORE `startup()` completes. Applications (Incus)
+    start at step ~16 of `startup()`, while `registerJobs()` is at step ~19.
+    When Incus starts before the scheduler crashes, port 8443 is available
+    during the 15-second error sleep (before `os.Exit(1)`).
+  - `observe-deploy` detects port 8443 during this window → reports PASS.
+  - "System is ready" is NOT shown on console (only appears after
+    `registerJobs()` succeeds).
+- **Source code analysis** (root cause investigation):
+  - `state.txt` is created FRESH on first boot from disk by
+    `LoadOrCreate()` → `initialize()`. The `initialize()` function
+    correctly sets `ScrubSchedule = "0 4 * * 0"`.
+  - The "install media detected" boot (#2, ISO still attached) runs from
+    the ISO rootfs. It creates state.txt in tmpfs, NOT on the installed
+    disk. The installed disk's state.txt is untouched.
+  - `DoInstall()` does not create state.txt. It only writes the OS image
+    and processes seed files.
+  - **The encoder skips zero values**: `encodeHelper()` returns early on
+    `v.IsZero()`. An empty ScrubSchedule string is NOT written to file.
+    If state.txt is read with a missing ScrubSchedule, the field stays
+    at Go's zero value (empty string) → `gocron.IsValid("")` fails.
+  - **Non-atomic state writes**: `Save()` uses `os.WriteFile()` which
+    truncates then writes. Crash during write → partial file.
+  - **Race window**: The REST API server starts BEFORE `startup()`
+    completes. Applications start before `registerJobs()`. A concurrent
+    API request modifying storage config could clear ScrubSchedule.
+  - **Most likely root cause**: either (a) a race condition between the
+    REST API/application startup and `registerJobs()` that clears
+    `ScrubSchedule`, or (b) a gocron library edge case where
+    `IsValid("0 4 * * 0", time.UTC, time.Now())` intermittently fails.
+    Both explain the ~50% hit rate and lack of correlation with seed
+    content.
+- **Proposed upstream fix** (for `incus-os` project):
+  ```go
+  // In registerJobs(): validate and fall back to default
+  func registerJobs(s *state.State) error {
+      schedule := s.System.Storage.Config.ScrubSchedule
+      if schedule == "" {
+          schedule = "0 4 * * 0"
+          s.System.Storage.Config.ScrubSchedule = schedule
+      }
+      err := s.JobScheduler.RegisterJob(zfs.PoolScrubJob, schedule, zfs.ScrubAllPools)
+      if err != nil {
+          // Fall back to default instead of crashing the daemon
+          slog.Warn("Invalid scrub schedule, using default", "schedule", schedule)
+          s.System.Storage.Config.ScrubSchedule = "0 4 * * 0"
+          return s.JobScheduler.RegisterJob(zfs.PoolScrubJob, "0 4 * * 0", zfs.ScrubAllPools)
+      }
+      return nil
+  }
+  ```
+  Additional recommendations: (1) use atomic writes in `Save()` (write
+  to temp file + `os.Rename()`), (2) always encode `ScrubSchedule` even
+  when empty (to preserve the default through encode/decode cycles),
+  (3) add mutex protection on State fields shared between API handlers
+  and startup().
+- **Recovery via restart**: `phase_install` in `incusos-proxmox` now
+  automatically retries once (stop → 10s wait → start → re-check).
+  After the first crash, the deferred `s.Save()` writes state. On the
+  second boot, systemd restarts the daemon; if the root cause was
+  transient (race condition or gocron edge case), the second boot
+  succeeds.
+- **Detection**: `phase_install` checks port 8443 after starting from disk
+  (30s wait + up to 60s polling). If first check fails, automatic retry
+  adds ~100s. Failed VMs after retry are reported with remediation.
+- **Source files**: `incus-osd/cmd/incus-osd/main.go` (`registerJobs()`,
+  `updateChecker()`, `startup()`, `firstBootActions()`),
+  `incus-osd/internal/scheduling/scheduling.go` (`RegisterJob()`,
+  `ErrInvalidCronTab`), `incus-osd/internal/state/file.go`
+  (`LoadOrCreate()`, `Save()`, `initialize()`),
+  `incus-osd/internal/state/encode.go` (`encodeHelper()` zero-value skip),
+  `incus-osd/internal/install/install.go` (`DoInstall()`,
+  `rebootUponDeviceRemoval()` SysRq-B).
+
+### Proxmox SSH root access — strict rules
+
+Root SSH access to the Proxmox host is available for **diagnostics only**.
+The password is stored in the `env` file as `PROXMOX_ROOT_PASSWORD`.
+
+**MANDATORY RULES — violation of any rule is unacceptable:**
+
+1. **Screenshots only**: the ONLY permitted use of root SSH is taking VM
+   console screenshots via `qm monitor <vmid> screendump`. No other use
+   without explicit user instruction.
+2. **Test VMs only**: only screenshot VMs in the test VMID range (850-869)
+   or VMs in the IncusLab pool that were created by our scripts.
+3. **No modifications**: NEVER run any command that modifies, stops, starts,
+   or deletes any VM. No `qm set`, `qm stop`, `qm start`, `qm destroy`,
+   `pct` commands, `zfs` commands, `systemctl` commands, or ANY write
+   operation on the host.
+4. **Do not touch the dev VM**: the user's dev VM runs on the same host.
+   Do not interact with it in any way — do not even `qm status` it.
+5. **No config access**: do not read or modify `/etc/pve/`, storage configs,
+   network configs, user/ACL settings, or any host-level configuration.
+6. **Transparency**: every SSH command is visible in tool output. If the
+   user rejects a command, do not retry it.
+7. **Only during active tests**: only SSH during test runs where test VMs
+   have been deployed, and only to screenshot those test VMs.
+
+**Permitted commands (exhaustive list):**
+```bash
+# Take screenshot of a test VM (VMID in 850-869 range)
+sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
+    root@<host> "echo 'screendump /tmp/vm-<vmid>-screen.ppm' | qm monitor <vmid>"
+
+# Retrieve screenshot
+sshpass -p "$PROXMOX_ROOT_PASSWORD" scp -o StrictHostKeyChecking=no \
+    root@<host>:/tmp/vm-<vmid>-screen.ppm /tmp/
+
+# Cleanup screenshot on remote
+sshpass -p "$PROXMOX_ROOT_PASSWORD" ssh -o StrictHostKeyChecking=no \
+    root@<host> "rm -f /tmp/vm-<vmid>-screen.ppm"
+```
+
+**Technical notes:**
+- QEMU's `screendump` is root-only in PVE 9's HMP permission model.
+  API tokens cannot execute it regardless of privileges.
+- Use `.ppm` format — PNG (`-f png`) requires QEMU compiled with libpng,
+  which PVE 9.1 does **not** have (`Error: Enable PNG support with libpng`).
+- PPM files are ~3 MB (1280x800). Convert to PNG with `python3-pil`:
+  `python3 -c "from PIL import Image; Image.open('f.ppm').save('f.png')"`
+  PNG output is ~20 KB and can be read directly by Claude Code (multimodal).
+
 ## Coding conventions for scripts
 
 - **Shell**: bash with `set -euo pipefail`

incusos/incusos-proxmox

@@ -1937,13 +1937,35 @@ pve_stop_vm() {
     esac
 }
 
-# Destroy a VM
+# Destroy a VM (waits for async task to complete)
 pve_destroy_vm() {
     local vmid="$1"
 
     case "$PVE_METHOD" in
         ssh) pve_run "qm destroy ${vmid} --purge" ;;
-        api) pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1" >/dev/null ;;
+        api)
+            local destroy_response
+            destroy_response=$(pve_api DELETE "/nodes/${PVE_NODE}/qemu/${vmid}?purge=1&destroy-unreferenced-disks=1" 2>/dev/null) || return 1
+            # VM destruction is async -- wait for the UPID task to finish
+            # to avoid VMID/disk reuse issues in rapid destroy+create cycles.
+            local upid
+            upid=$(echo "$destroy_response" | python3 -c "import json,sys; print(json.load(sys.stdin).get('data',''))" 2>/dev/null) || upid=""
+            if [[ -n "$upid" ]]; then
+                local encoded_upid
+                encoded_upid=$(python3 -c "import urllib.parse; print(urllib.parse.quote('${upid}', safe=''))" 2>/dev/null)
+                local wait_elapsed=0
+                while [[ $wait_elapsed -lt 30 ]]; do
+                    sleep 2
+                    wait_elapsed=$((wait_elapsed + 2))
+                    local task_status
+                    task_status=$(pve_api GET "/nodes/${PVE_NODE}/tasks/${encoded_upid}/status" 2>/dev/null | \
+                        python3 -c "import json,sys; print(json.load(sys.stdin).get('data',{}).get('status',''))" 2>/dev/null) || task_status=""
+                    if [[ "$task_status" == "stopped" ]]; then
+                        break
+                    fi
+                done
+            fi
+            ;;
     esac
 }
 
@@ -2458,6 +2480,70 @@ for s in data.get('data', []):
         errors=$((errors + 1))
     fi
 
+    # 6. Host resource feasibility (API method only)
+    if [[ "$PVE_METHOD" == "api" ]] && [[ "$DRY_RUN" != true ]] && [[ ${#VM_MEMORY[@]} -gt 0 ]]; then
+        local total_ram_mib=0
+        local total_disk_gib=0
+        local total_cores=0
+        local ri=0
+        while [[ $ri -lt ${#VM_MEMORY[@]} ]]; do
+            total_ram_mib=$((total_ram_mib + VM_MEMORY[$ri]))
+            total_disk_gib=$((total_disk_gib + VM_DISK[$ri]))
+            total_cores=$((total_cores + VM_CORES[$ri]))
+            ri=$((ri + 1))
+        done
+
+        local res_node_status
+        res_node_status=$(pve_api GET "/nodes/${PVE_NODE}/status" 2>/dev/null) || res_node_status=""
+        if [[ -n "$res_node_status" ]]; then
+            local resource_lines
+            resource_lines=$(python3 -c "
+import json, sys
+data = json.loads('''${res_node_status}''').get('data', {})
+mem = data.get('memory', {})
+mem_total = mem.get('total', 0) / (1024**3)
+mem_used = mem.get('used', 0) / (1024**3)
+mem_free = mem_total - mem_used
+cpu_count = data.get('cpuinfo', {}).get('cpus', 0)
+req_ram = ${total_ram_mib} / 1024
+req_cores = ${total_cores}
+
+# RAM check
+if mem_free > 0:
+    pct = req_ram / mem_free * 100
+    if req_ram > mem_free:
+        print(f'error|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free -- not enough RAM')
+    elif pct > 70:
+        print(f'warn|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free ({pct:.0f}% of free) -- tight fit')
+    else:
+        print(f'ok|RAM: {req_ram:.1f} GiB requested, {mem_free:.1f} GiB free ({pct:.0f}% of free)')
+
+# CPU (info only)
+print(f'ok|CPU: {req_cores} cores requested, {cpu_count} available')
+" 2>/dev/null) || resource_lines=""
+
+            local line
+            while IFS= read -r line; do
+                [[ -z "$line" ]] && continue
+                local level="${line%%|*}"
+                local msg="${line#*|}"
+                case "$level" in
+                    error)
+                        error "$msg"
+                        errors=$((errors + 1))
+                        ;;
+                    warn)
+                        warn "$msg"
+                        warnings=$((warnings + 1))
+                        ;;
+                    ok)
+                        detail "$msg"
+                        ;;
+                esac
+            done <<< "$resource_lines"
+        fi
+    fi
+
     if [[ $errors -gt 0 ]]; then
         error "${errors} pre-flight check(s) failed"
         exit 1
@@ -2826,6 +2912,20 @@ confirm() {
     fi
 }
 
+# Like confirm(), but returns 0/1 instead of exiting on "no"
+confirm_yn() {
+    local message="$1"
+
+    if [[ "$YES" == true ]]; then
+        return 0
+    fi
+
+    echo -n -e "${BOLD}${message} [y/N] ${RESET}"
+    local reply
+    read -r reply
+    [[ "$reply" =~ ^[Yy]$ ]]
+}
+
 # ---------------------------------------------------------------------------
 # Phase 1: Validate
 # ---------------------------------------------------------------------------
@@ -3130,6 +3230,7 @@ phase_install() {
     echo ""
 
     local count=${#VM_NAMES[@]}
+    local install_failures=0
     local i=0
     while [[ $i -lt $count ]]; do
         local name="${VM_NAMES[$i]}"
@@ -3306,28 +3407,92 @@ phase_install() {
             continue
         fi
 
-        # 5. Wait for services to come up
+        # 5. Wait for services to come up and verify health
         info "Waiting for services to start..."
         sleep 30
 
-        # 6. Try to detect IP
+        # 6. Detect IP and check port 8443
         local ip
         ip=$(pve_vm_ip "$vmid") || ip=""
 
-        echo ""
+        local healthy=false
         if [[ -n "$ip" ]]; then
+            # Poll port 8443 for up to 60s (the 30s sleep above + 60s = 90s total)
+            local poll=0
+            while [[ $poll -lt 12 ]]; do
+                if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
+                    healthy=true
+                    break
+                fi
+                sleep 5
+                poll=$((poll + 1))
+            done
+        fi
+
+        echo ""
+        if [[ "$healthy" == true ]]; then
             success "VM '${name}' installed and running at ${ip}"
             register_remote "$name" "$ip"
         else
-            success "VM '${name}' installed and running"
+            # Retry once: IncusOS has an intermittent "invalid crontab expression"
-            info "Check Proxmox console for IP address"
+            # bug (~50% first-boot hit rate) where registerJobs() fails during
+            # startup(). Most likely a race condition between the REST API and
+            # startup sequence, or a gocron library edge case. A stop+start
+            # cycle resolves it because the transient condition clears on retry.
+            if [[ -n "$ip" ]]; then
+                warn "VM '${name}': port 8443 not reachable at ${ip} — retrying (stop+start)"
+            else
+                warn "VM '${name}': could not detect IP — retrying (stop+start)"
+            fi
+            info "Stopping VM for retry..."
+            pve_stop_vm "$vmid" || true
+            sleep 10
+            info "Starting VM (retry)..."
+            pve_start_vm "$vmid" || true
+            sleep 30
+
+            # Re-detect IP (may have changed after restart)
+            ip=$(pve_vm_ip "$vmid") || ip=""
+
+            local retry_healthy=false
+            if [[ -n "$ip" ]]; then
+                local retry_poll=0
+                while [[ $retry_poll -lt 12 ]]; do
+                    if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
+                        retry_healthy=true
+                        break
+                    fi
+                    sleep 5
+                    retry_poll=$((retry_poll + 1))
+                done
+            fi
+
+            if [[ "$retry_healthy" == true ]]; then
+                success "VM '${name}' recovered after retry — running at ${ip}"
+                register_remote "$name" "$ip"
+            elif [[ -n "$ip" ]]; then
+                warn "VM '${name}' installed but port 8443 not reachable at ${ip} (even after retry)"
+                warn "Likely IncusOS boot failure (check Proxmox console for errors)"
+                warn "Redeploy this VM with: ${SCRIPT_NAME} --cleanup --yes ${CONFIG_FILE} && ${SCRIPT_NAME} --yes ${CONFIG_FILE}"
+                install_failures=$((install_failures + 1))
+            else
+                warn "VM '${name}' installed but could not detect IP (even after retry)"
+                warn "Check Proxmox console for status"
+                install_failures=$((install_failures + 1))
+            fi
         fi
         echo ""
 
         i=$((i + 1))
     done
 
+    if [[ $install_failures -gt 0 ]]; then
+        warn "Installation phase complete: ${install_failures} VM(s) failed health check"
+        warn "Failed VMs are likely stuck in an IncusOS boot loop"
+        warn "Re-run deployment to redeploy, or use --cleanup to start fresh"
+    else
         success "Installation phase complete"
+    fi
 }
 
 # ---------------------------------------------------------------------------
@@ -3393,6 +3558,48 @@ phase_lab_up() {
     local count=${#VM_NAMES[@]}
     local started=0
     local skipped=0
+    local missing=0
+
+    # First pass: check how many VMs exist
+    if [[ "$DRY_RUN" != true ]]; then
+        local i=0
+        while [[ $i -lt $count ]]; do
+            if ! pve_vm_exists "${VM_VMIDS[$i]}"; then
+                missing=$((missing + 1))
+            fi
+            i=$((i + 1))
+        done
+
+        # If ALL VMs are missing, offer to deploy
+        if [[ $missing -eq $count ]]; then
+            echo ""
+            warn "No VMs found -- lab has not been deployed yet"
+            if confirm_yn "Deploy lab now?"; then
+                echo ""
+                phase_prepare
+                echo ""
+                phase_upload
+                echo ""
+                phase_create
+                echo ""
+                phase_install
+                echo ""
+                phase_post_deploy
+                return
+            else
+                info "Aborted"
+                return
+            fi
+        fi
+
+        # If SOME VMs are missing, warn but don't offer auto-deploy
+        if [[ $missing -gt 0 ]]; then
+            warn "${missing} of ${count} VMs not found -- partial state detected"
+            warn "Use --cleanup then redeploy, or deploy missing VMs manually"
+        fi
+    fi
+
+    # Second pass: start existing VMs
     local i=0
     while [[ $i -lt $count ]]; do
         local name="${VM_NAMES[$i]}"
@@ -3505,6 +3712,7 @@ print(f'  Uptime:  {days}d {hours}h')
         python3 -c "
 import json, sys
 data = json.loads('''${storage_list}''').get('data', [])
+thin_types = {'zfspool', 'lvmthin'}
 for s in sorted(data, key=lambda x: x.get('storage', '')):
     name = s.get('storage', '?')
     total = s.get('total', 0) / (1024**3)
@@ -3513,10 +3721,11 @@ for s in sorted(data, key=lambda x: x.get('storage', '')):
     pct = (used / total * 100) if total > 0 else 0
     stype = s.get('type', '?')
     content = s.get('content', '?')
+    thin = ' [thin]' if stype in thin_types else ''
     if total > 0:
-        print(f'    {name:<16} {stype:<12} {used:.1f}G / {total:.1f}G ({pct:.0f}%) -- {content}')
+        print(f'    {name:<16} {stype:<12} {used:.1f}G / {total:.1f}G ({pct:.0f}%) -- {content}{thin}')
     else:
-        print(f'    {name:<16} {stype:<12} (no capacity info) -- {content}')
+        print(f'    {name:<16} {stype:<12} (no capacity info) -- {content}{thin}')
 " 2>/dev/null || warn "Failed to parse storage list"
     fi
 
@@ -3540,6 +3749,57 @@ print(f'    VMs:       {len(vms)} total ({running} running, {stopped} stopped)')
 print(f'    Allocated: {total_mem:.1f} GiB RAM, {total_disk:.0f} GiB disk')
 " 2>/dev/null || warn "Failed to parse pool data"
         fi
+
+        # Actual disk usage via storage content API
+        if [[ -n "$pool_data" ]]; then
+            local storage_content
+            storage_content=$(pve_api GET "/nodes/${PVE_NODE}/storage/${PVE_STORAGE}/content" 2>/dev/null) || storage_content=""
+            local storage_status_json
+            storage_status_json=$(echo "$storage_list" | python3 -c "
+import json, sys
+data = json.load(sys.stdin).get('data', [])
+for s in data:
+    if s.get('storage') == '${PVE_STORAGE}':
+        print(json.dumps(s))
+        break
+" 2>/dev/null) || storage_status_json=""
+
+            if [[ -n "$storage_content" ]] && [[ -n "$pool_data" ]]; then
+                python3 -c "
+import json, sys
+content = json.loads('''${storage_content}''').get('data', [])
+pool = json.loads('''${pool_data}''').get('data', {})
+vm_vmids = {str(m.get('vmid', '')) for m in pool.get('members', []) if m.get('type') == 'qemu'}
+
+# Sum actual (used) bytes for volumes belonging to pool VMs
+actual_bytes = 0
+allocated_bytes = 0
+for vol in content:
+    vmid = str(vol.get('vmid', ''))
+    if vmid in vm_vmids:
+        actual_bytes += vol.get('used', 0)
+        allocated_bytes += vol.get('size', 0)
+
+if allocated_bytes > 0:
+    actual_gib = actual_bytes / (1024**3)
+    allocated_gib = allocated_bytes / (1024**3)
+    ratio = (actual_bytes / allocated_bytes * 100) if allocated_bytes > 0 else 0
+    print(f'    Actual:    {actual_gib:.1f} GiB disk used ({ratio:.0f}% of allocated, thin provisioned)')
+" 2>/dev/null || true
+            fi
+
+            # Storage type note
+            if [[ -n "$storage_status_json" ]]; then
+                python3 -c "
+import json
+s = json.loads('''${storage_status_json}''')
+stype = s.get('type', '')
+notes = {'zfspool': 'ZFS (thin provisioned, lz4 compression)', 'lvmthin': 'LVM-thin (thin provisioned)'}
+if stype in notes:
+    print(f'    Storage:   {notes[stype]}')
+" 2>/dev/null || true
+            fi
+        fi
     fi
 
     echo ""

incusos/incusos-seed

@@ -616,17 +616,13 @@ generate_network_yaml() {
 }
 
 generate_update_yaml() {
-    # Only generate if non-default values were specified
-    if [[ "$CHANNEL" == "stable" ]] && [[ "$AUTO_REBOOT" == false ]]; then
-        return
-    fi
-
     local file="${WORKDIR}/update.yaml"
 
     step "Generating update.yaml"
 
     {
         echo "# Update configuration"
+        echo "# Always generated to ensure explicit check_frequency on first boot"
         echo "channel: \"${CHANNEL}\""
         echo "auto_reboot: ${AUTO_REBOOT}"
         echo "check_frequency: \"6h\""