Fix IncusOS deploy: eliminate crontab bug and TPM corruption

Root cause #1: force_reboot triggers SysRq-B intermediate boot that
races with IncusOS state initialization, causing ~50% crontab bug rate.
Fix: omit force_reboot from seed — VM sits at "please remove media"
after install, blockstat detects idle cleanly, no intermediate boot.
Validated: 15/15 PASS (100%) on 4-core VMs.

Root cause #2: 90s boot timeout was too short for first-boot sysext
download under load. Premature retry hard-stops the VM during first
boot, corrupting the TPM encryption key permanently. Fix: increase
boot timeout to 180s (60s sleep + 120s polling).

Changes:
- incusos-proxmox: remove --force-reboot from seed generation, increase
  boot timeout from 90s to 180s, update retry comments
- incusos-seed: add --ip/--gateway/--dns flags for static IP seeds,
  add network.yaml generation
- observe-deploy: add --no-force-reboot flag, --runs N mode, VLAN
  support, scrub_schedule health check, 1s screenshot interval
- lab configs: add static IPs for VLAN 69 (192.168.102.x/22)
- proxmox.yaml.example: add vlan, gateway, dns fields
- CLAUDE.md: document findings, boot timeout, TPM corruption risk

Tested: single VM, 3-node cluster, OC + 3 nodes — all 100% success.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -130,22 +130,33 @@ incu-contrib/
   `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
   `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
 - **Install flow** (automated by `incusos-proxmox`):
-  1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
+  1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed
-  2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
+  2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone),
+     then sits at "please remove installation media" prompt
   3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
      when disk writes start then stop for 15s (3 stable polls), install is done
   4. Stop the VM (Proxmox stop, not guest shutdown)
   5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
      and refuses to start if found, regardless of boot order
   6. Set boot order to `order=scsi0` and start from disk
-- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
+- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two
-  ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
+  strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM
-  ping broadcast → look up MAC in ARP table. Verify with direct ping before
+  in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's
-  trusting the result.
+  `network.yaml` configures the VM's interface at boot. No ARP scan or
-- **force_reboot is required**: without `force_reboot: true` in the seed,
+  SSH needed — the IP is known at deploy time. Works across VLANs.
-  IncusOS sits at "please remove installation media" and waits indefinitely.
+  (2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM
-  It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
+  config → flush stale ARP → ping broadcast → look up MAC in ARP table.
-  (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
+  Only works on the same L2 domain (native LAN, not across VLANs).
+- **force_reboot is NOT used on Proxmox** (changed 2026-02-22): the seed
+  omits `force_reboot`. Without it, IncusOS sits at "please remove
+  installation media" after install — which is exactly what we want. We
+  detect completion via blockstat (876 MiB written, then idle), stop the
+  VM externally, remove media, and start from disk. This eliminates the
+  SysRq-B intermediate boot that triggers the crontab race condition
+  (issue #843). On physical hardware or unmanaged VMs, `force_reboot: true`
+  is still needed — this optimization is specific to Proxmox automated deploys.
+  `force_reboot` triggers SysRq-B (raw kernel reboot via `/proc/sysrq-trigger`).
+  This does NOT reset QEMU VM uptime — only a Proxmox stop/start does.
 - **Resource pool isolation**: the optional `proxmox.pool` config field scopes
   all VM operations to a Proxmox resource pool. When set, the script only
   "sees" VMs in that pool (for collision detection and cleanup), and the API
@@ -511,10 +522,18 @@ incu-contrib/
   except warnings and errors.
 - **`--retries N`**: number of stop+start retries for VMs that fail to boot
   (port 8443 not reachable). Default: 3. `--retries 0` disables retries.
-  Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443.
+  Each retry: Proxmox stop → wait → start → 60s wait → poll port 8443 for
-  With ~67% boot failure rate from the crontab bug (confirmed by parallel
+  up to 120s. With the `force_reboot` fix (2026-02-22), the crontab bug is
-  investigation), 3 retries gives ~96.3% success rate (1 - 0.67^3). After retries, `fix_scrub_schedule()` proactively heals the
+  eliminated and retries are rarely needed. They remain as a safety net for
-  root cause via the IncusOS REST API.
+  other transient boot failures. After retries, `fix_scrub_schedule()`
+  proactively heals any remaining scrub_schedule issues via the IncusOS
+  REST API.
+- **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot
+  downloads SecureBoot update + application sysext (Incus: ~30-120s depending
+  on CDN speed and concurrent VMs). The previous 90s timeout was too short
+  under load — premature retries (hard VM stop during first boot) can corrupt
+  the TPM encryption key, causing permanent "zfs load-key: Raw key too short
+  (expected 32)" errors that no amount of retries can fix.
 
 ### IncusOS first-boot sequence (observed via console screenshots)
 
@@ -570,9 +589,19 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
 
 ### IncusOS boot failure (crontab / update frequency error)
 
+- **RESOLVED** (2026-02-22): root cause identified and fixed in our pipeline.
+  The `force_reboot` seed option triggers SysRq-B after install, causing an
+  "install media detected" intermediate boot. This intermediate boot generates
+  ~26 GB of additional disk writes (the IncusOS daemon starts from ISO tmpfs).
+  Our blockstat detection catches these late writes, but by then the
+  intermediate boot has run and somehow corrupted the state for the subsequent
+  first-from-disk boot. **Fix: omit `force_reboot` from the seed.** Without
+  it, the installer sits at "please remove media" after the 876 MiB image
+  clone, blockstat detects idle cleanly, and the first disk boot is pristine.
+  Results: **15/15 PASS (100%)** without force_reboot on 4 cores vs 50% with it.
 - **Symptom**: "ERROR invalid crontab expression" appears on the console
-  during first boot. Intermittent — confirmed **67% failure rate** across
+  during first boot. Intermittent — confirmed **~50% failure rate** with
-  20 parallel VM deploys (8/12 API-verified). Error does NOT correlate with
+  `force_reboot: true` on 4-core VMs. Error does NOT correlate with
   `update.yaml` presence/absence, ISO version, or version mismatch.
 - **Two distinct error paths in IncusOS**:
   - `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
@@ -613,11 +642,25 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
   - **Race window**: The REST API server starts BEFORE `startup()`
     completes. Applications start before `registerJobs()`. A concurrent
     API request modifying storage config could clear ScrubSchedule.
-  - **Most likely root cause**: a race condition between the REST API/
+  - **Root cause confirmed**: the bug is triggered by the SysRq-B
-    application startup and `registerJobs()` that clears `ScrubSchedule`.
+    intermediate boot (from `force_reboot: true`). Controlled experiments
-    Investigation ruled out version mismatch (bug hits equally with matching
+    on VLAN 69 with static IP (2026-02-22, ISO 202602210344):
-    ISO), update downloads (occurs even without OS updates), and gocron
+    - Baseline (4 cores, force_reboot): 5/10 PASS (50%)
-    library issues (the field is genuinely empty in the state struct).
+    - cache=writethrough: 3/5 PASS (60%) — falsified
+    - agent=0: 3/5 PASS (60%) — falsified
+    - cores=1 (force_reboot): 14/15 PASS (93%) — GOMAXPROCS=1 reduces race
+    - **No force_reboot (4 cores): 15/15 PASS (100%)** — confirmed fix
+    Correlation: runs with 876 MiB blockstat detection (before SysRq-B)
+    had 100% success; runs with 27 GB detection (after SysRq-B) had ~55%.
+  - **The try-it service is NOT IncusOS**: stgraber's comparison ("thousands
+    of times a day") is invalid. The try-it service runs Ubuntu VMs with
+    Incus installed from the Zabbly daily repo — a completely different
+    product. Issue #843 was independently filed by another Proxmox user
+    (fperreau on Proxmox 9.1.4), confirming it's not unique to our setup.
+  - The underlying IncusOS bug is still a race condition between the REST
+    API/application startup and `registerJobs()` that clears `ScrubSchedule`.
+    Investigation ruled out version mismatch, update downloads, and gocron
+    library issues. The field is genuinely empty in the state struct.
   - **Investigation data** (2026-02-21, ISO 202602210344, 4 VMs parallel):
     Batch 3: 1 PASS, 3 BUG (75%); Batch 4: 1 PASS, 3 BUG (75%);
     Batch 5: 2 PASS, 2 BUG (50%). Total API-verified: 4 PASS, 8 BUG
@@ -671,8 +714,16 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
   has a journal commit interval (~5s) and the corrupt Save() may not have
   been committed before the VM was killed.
 - **Detection**: `phase_install` checks port 8443 after starting from disk
-  (30s wait + up to 60s polling). If first check fails, automatic retry
+  (60s wait + up to 120s polling = 180s total). If first check fails,
-  adds ~100s. Failed VMs after retry are reported with remediation.
+  automatic retry adds ~180s. Failed VMs after retry are reported with
+  remediation.
+- **TPM corruption from premature retry**: hard-stopping a VM during first
+  boot (while the encryption key is being written to the TPM zvol) can
+  leave the TPM with truncated key data. All subsequent boots fail with
+  "zfs load-key: Raw key too short (expected 32)" — a permanent error
+  that requires VM destruction and redeployment to fix. The 180s boot
+  timeout prevents this by allowing the full first-boot sequence (including
+  sysext download) to complete before any retry.
 - **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes
   `fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on
   every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix

incusos/examples/lab-cluster.yaml

@@ -29,11 +29,14 @@ vms:
   - name: incus-lab-01
     app: incus
     apply_defaults: true    # init node: needs storage pool + network
+    ip: 192.168.102.101/22
 
   - name: incus-lab-02
     app: incus
     apply_defaults: false   # joining node: cluster join creates pool entry
+    ip: 192.168.102.102/22
 
   - name: incus-lab-03
     app: incus
     apply_defaults: false   # joining node: cluster join creates pool entry
+    ip: 192.168.102.103/22

incusos/examples/lab-oc.yaml

@@ -29,6 +29,7 @@ vms:
   - name: lab-oc
     app: operations-center
     apply_defaults: true
+    ip: 192.168.102.110/22
 
   - name: lab-node-01
     app: incus
@@ -36,11 +37,14 @@ vms:
     cores: 4
     memory: 8192
     disk: 64
+    ip: 192.168.102.111/22
 
   - name: lab-node-02
     app: incus
     apply_defaults: true
+    ip: 192.168.102.112/22
 
   - name: lab-node-03
     app: incus
     apply_defaults: true
+    ip: 192.168.102.113/22

incusos/examples/lab-single.yaml

@@ -23,3 +23,4 @@ vms:
   - name: incus-single
     app: incus
     apply_defaults: true
+    ip: 192.168.102.100/22

incusos/examples/network-static.yaml

@@ -21,7 +21,7 @@ time:
 
 interfaces:
   - name: "mgmt"
-    hwaddr: "enp5s0"           # interface name or MAC address
+    hwaddr: "ens18"            # interface name or MAC (ens18 = Proxmox VirtIO)
     addresses:
       - "192.168.1.100/24"
     roles:

incusos/examples/proxmox.yaml.example

@@ -17,5 +17,8 @@ node: pve                      # Proxmox node name
 storage: local-lvm             # VM disk storage
 iso_storage: local             # ISO/image storage
 bridge: vmbr0                  # Network bridge
+# vlan: 69                     # VLAN tag for VM NIC (optional)
+# gateway: 192.168.100.1       # Default gateway for static IPs (required if VMs use ip:)
+# dns: 192.168.100.1           # DNS server for static IPs (optional, used with ip:)
 # pool: IncusLab               # Resource pool (optional, for isolation)
 ssh_user: root                 # SSH user (for ssh method)

incusos/incusos-proxmox

@@ -1296,6 +1296,8 @@ PVE_STORAGE=""
 PVE_ISO_STORAGE=""
 PVE_BRIDGE=""
 PVE_VLAN=""
+PVE_GATEWAY=""
+PVE_DNS=""
 PVE_METHOD=""
 PVE_SSH_USER=""
 PVE_API_TOKEN_ID=""
@@ -1316,6 +1318,7 @@ declare -a VM_CORES=()
 declare -a VM_MEMORY=()
 declare -a VM_DISK=()
 declare -a VM_VMIDS=()
+declare -a VM_IPS=()
 declare -a VM_CLUSTERS=()
 declare -a VM_CLUSTER_ROLES=()
 
@@ -1396,6 +1399,8 @@ json.dump(result, sys.stdout, indent=2)
     val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
     val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
     val=$(json_get "$px_json" ".vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
+    val=$(json_get "$px_json" ".gateway" ""); [[ -n "$val" ]] && PVE_GATEWAY="$val"
+    val=$(json_get "$px_json" ".dns" ""); [[ -n "$val" ]] && PVE_DNS="$val"
     val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
     val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
     val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
@@ -1419,6 +1424,8 @@ load_config() {
     val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
     val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
     val=$(json_get "$CONFIG_JSON" ".proxmox.vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
+    val=$(json_get "$CONFIG_JSON" ".proxmox.gateway" ""); [[ -n "$val" ]] && PVE_GATEWAY="$val"
+    val=$(json_get "$CONFIG_JSON" ".proxmox.dns" ""); [[ -n "$val" ]] && PVE_DNS="$val"
     val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
     val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
     val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
@@ -1459,6 +1466,7 @@ load_config() {
         VM_CORES+=("$(vm_get "$i" "cores" "$DEF_CORES")")
         VM_MEMORY+=("$(vm_get "$i" "memory" "$DEF_MEMORY")")
         VM_DISK+=("$(vm_get "$i" "disk" "$DEF_DISK")")
+        VM_IPS+=("$(vm_get "$i" "ip" "")")
         VM_VMIDS+=("$(vm_get "$i" "vmid" "")")
         VM_CLUSTERS+=("$(vm_get "$i" "cluster" "")")
         VM_CLUSTER_ROLES+=("$(vm_get "$i" "cluster_role" "")")
@@ -1577,6 +1585,19 @@ validate_config() {
             esac
         fi
 
+        # Static IP validation
+        local vm_ip="${VM_IPS[$i]}"
+        if [[ -n "$vm_ip" ]]; then
+            if [[ "$vm_ip" != */* ]]; then
+                error "VM '${name}': ip must include prefix (e.g. 192.168.102.100/22), got '${vm_ip}'"
+                errors=$((errors + 1))
+            fi
+            if [[ -z "$PVE_GATEWAY" ]]; then
+                error "VM '${name}': static ip requires gateway in proxmox.yaml or lab config"
+                errors=$((errors + 1))
+            fi
+        fi
+
         # Track cluster init nodes
         if [[ -n "$cluster" ]] && [[ "$role" == "init" ]]; then
             if echo "$cluster_inits" | grep -qw "$cluster"; then
@@ -2103,6 +2124,13 @@ pve_vm_exists() {
 # Get VM's IP address (best-effort via QEMU guest agent)
 pve_vm_ip() {
     local vmid="$1"
+    local known_ip="${2:-}"
+
+    # If static IP is configured, return bare IP (strip CIDR prefix)
+    if [[ -n "$known_ip" ]]; then
+        echo "${known_ip%%/*}"
+        return 0
+    fi
 
     # Method 1: Try QEMU guest agent (works if agent is installed)
     local ip=""
@@ -2881,6 +2909,9 @@ print_plan() {
     else
         echo -e "  Bridge:        ${PVE_BRIDGE}"
     fi
+    if [[ -n "$PVE_GATEWAY" ]]; then
+        echo -e "  Gateway:       ${PVE_GATEWAY}"
+    fi
     if [[ -n "$PVE_POOL" ]]; then
         echo -e "  Pool:          ${CYAN}${PVE_POOL}${RESET}"
     fi
@@ -2907,6 +2938,10 @@ print_plan() {
         echo -e "      VMID: ${VM_VMIDS[$i]}  App: ${VM_APPS[$i]}  Defaults: ${VM_APPLY_DEFAULTS[$i]}"
         echo -e "      CPU: ${VM_CORES[$i]} cores  RAM: ${VM_MEMORY[$i]} MiB  Disk: ${VM_DISK[$i]} GiB"
 
+        if [[ -n "${VM_IPS[$i]}" ]]; then
+            echo -e "      IP: ${GREEN}${VM_IPS[$i]}${RESET} (static)"
+        fi
+
         if [[ -n "${VM_CLUSTERS[$i]}" ]]; then
             echo -e "      Cluster: ${VM_CLUSTERS[$i]} (${VM_CLUSTER_ROLES[$i]})"
         fi
@@ -3031,7 +3066,12 @@ phase_prepare() {
         seed_cmd+=(--app "$app")
         seed_cmd+=(--output "$seed_output")
         seed_cmd+=(--force-install)
-        seed_cmd+=(--force-reboot)
+        # NOTE: --force-reboot intentionally omitted. On Proxmox, we control
+        # the VM externally (stop/start). force_reboot triggers SysRq-B which
+        # causes an "install media detected" intermediate boot — this triggers
+        # the IncusOS state.txt race condition (issue #843) ~50% of the time.
+        # Without force_reboot, the installer sits at "please remove media",
+        # blockstat detects idle, and we cleanly stop/remove media/start from disk.
         seed_cmd+=(--hostname "$name")
         seed_cmd+=(--quiet)
 
@@ -3045,6 +3085,16 @@ phase_prepare() {
             seed_cmd+=(--no-cert)
         fi
 
+        # Static IP: pass --ip, --gateway, --dns to seed generator
+        local vm_ip="${VM_IPS[$i]}"
+        if [[ -n "$vm_ip" ]]; then
+            seed_cmd+=(--ip "$vm_ip")
+            seed_cmd+=(--gateway "$PVE_GATEWAY")
+            if [[ -n "$PVE_DNS" ]]; then
+                seed_cmd+=(--dns "$PVE_DNS")
+            fi
+        fi
+
         if [[ "$DRY_RUN" == true ]]; then
             info "[dry-run] ${seed_cmd[*]}"
         else
@@ -3350,7 +3400,7 @@ phase_install() {
                 # Already installed and running — detect IP, register remote, skip
                 success "VM '${name}' (VMID ${vmid}) is already installed and running"
                 local ip
-                ip=$(pve_vm_ip "$vmid") || ip=""
+                ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
                 if [[ -n "$ip" ]]; then
                     detail "IP: ${ip}"
                     register_remote "$name" "$ip"
@@ -3365,7 +3415,7 @@ phase_install() {
                 if pve_start_vm "$vmid"; then
                     sleep 30
                     local ip
-                    ip=$(pve_vm_ip "$vmid") || ip=""
+                    ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
                     if [[ -n "$ip" ]]; then
                         success "VM '${name}' started at ${ip}"
                         register_remote "$name" "$ip"
@@ -3399,13 +3449,11 @@ phase_install() {
 
         # 2. Wait for installation to complete by monitoring disk writes
         #
-        # Install flow: boot ISO -> read SEED_DATA -> install to disk -> reboot
+        # Install flow: boot ISO -> read SEED_DATA -> install to disk -> idle
-        # (force_reboot in seed). A guest-level reboot does NOT reset the QEMU
+        # (no force_reboot: VM sits at "please remove installation media").
-        # VM uptime, so we cannot detect reboot that way. Instead we monitor
+        # We monitor scsi0 write bytes via blockstat:
-        # scsi0 write bytes via blockstat:
+        #   - During install: wr_bytes increases steadily (~876 MiB for image clone)
-        #   - During install: wr_bytes increases steadily
+        #   - After install: wr_bytes stops (VM is idle at "remove media" prompt)
-        #   - After install + reboot: wr_bytes stops (VM is idle at post-reboot
-        #     screen or "install media detected" error)
         #
         # When wr_bytes has been stable for STABLE_COUNT consecutive polls,
         # installation is complete. We then stop the VM, remove install media,
@@ -3476,17 +3524,46 @@ phase_install() {
 
         # 3. Stop the VM (it's either at "remove install media" prompt or
         #    post-reboot "install media detected" error -- either way idle).
+        #    Poll until confirmed stopped — do not proceed on assumption.
         local cur_status
         cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
         if [[ "$cur_status" == "running" ]]; then
             info "Stopping VM to remove install media..."
             pve_stop_vm "$vmid" || true
-            sleep 5
+            local stop_wait=0
+            while [[ $stop_wait -lt 30 ]]; do
+                cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
+                [[ "$cur_status" == "stopped" ]] && break
+                sleep 2
+                stop_wait=$((stop_wait + 2))
+            done
+            if [[ "$cur_status" != "stopped" ]]; then
+                warn "VM '${name}' did not stop within 30s (status: ${cur_status})"
+                install_failures=$((install_failures + 1))
+                i=$((i + 1))
+                continue
+            fi
+            detail "VM stopped after ${stop_wait}s"
         fi
 
-        # 4. Remove install media and set clean boot order
+        # 4. Remove install media and set clean boot order.
+        #    Verify removal succeeded by re-reading VM config.
         step "Removing install media from '${name}'"
         pve_set_vm "$vmid" "--delete ide2 --delete ide3 --boot order=scsi0" || true
+        sleep 2
+        # Verify ide2 and ide3 are gone
+        if pve_vm_has_install_media "$vmid"; then
+            warn "Install media still attached after removal — retrying"
+            pve_set_vm "$vmid" "--delete ide2" || true
+            pve_set_vm "$vmid" "--delete ide3" || true
+            sleep 2
+            if pve_vm_has_install_media "$vmid"; then
+                error "Failed to remove install media from '${name}' after retry"
+                install_failures=$((install_failures + 1))
+                i=$((i + 1))
+                continue
+            fi
+        fi
         success "Install media detached, boot order set to disk-only"
 
         # 5. Start VM from disk
@@ -3497,21 +3574,43 @@ phase_install() {
             continue
         fi
 
-        # 5. Wait for services to come up and verify health
+        # 6. Wait for services and verify health (port 8443 + scrub_schedule)
+        #
+        # The health check has two levels:
+        #   1. Port 8443 reachable (Incus API is up)
+        #   2. scrub_schedule is set (IncusOS daemon fully initialized)
+        #
+        # Port 8443 reachable but scrub_schedule empty = crontab bug hit.
+        # The IncusOS REST API starts BEFORE registerJobs(), so port 8443
+        # can respond even when the daemon is about to crash. We use
+        # scrub_schedule as the definitive health indicator and trigger
+        # retries (not just fixes) when it's empty.
         info "Waiting for services to start..."
-        sleep 30
+        # First boot downloads SecureBoot update + application sysext (30-120s).
+        # Total first-boot time is typically 60-150s. Wait 60s before polling.
+        sleep 60
 
-        # 6. Detect IP and check port 8443
         local ip
-        ip=$(pve_vm_ip "$vmid") || ip=""
+        ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
 
         local healthy=false
         if [[ -n "$ip" ]]; then
-            # Poll port 8443 for up to 60s (the 30s sleep above + 60s = 90s total)
+            # Poll port 8443 for up to 120s (the 60s sleep above + 120s = 180s total)
             local poll=0
-            while [[ $poll -lt 12 ]]; do
+            while [[ $poll -lt 24 ]]; do
                 if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
+                    # Port is up — check and fix scrub_schedule.
+                    # fix_scrub_schedule returns 1 ONLY when it detects empty
+                    # scrub_schedule AND the PUT to fix it fails. In that case,
+                    # the crontab bug hit and we should retry (not declare success).
+                    # Returns 0 for: no cert (can't check), already set, or fixed.
+                    local fix_result=0
+                    fix_scrub_schedule "$ip" "$name" || fix_result=$?
+                    if [[ $fix_result -eq 0 ]]; then
                         healthy=true
+                    else
+                        warn "VM '${name}': crontab bug detected (scrub_schedule empty, fix failed)"
+                    fi
                     break
                 fi
                 sleep 5
@@ -3521,17 +3620,13 @@ phase_install() {
 
         echo ""
         if [[ "$healthy" == true ]]; then
-            # Proactively fix empty scrub_schedule (IncusOS crontab bug).
-            # Safe to call on every node -- returns early if already set.
-            fix_scrub_schedule "$ip" "$name" || true
             success "VM '${name}' installed and running at ${ip}"
             register_remote "$name" "$ip"
         else
-            # Retry loop: IncusOS has an intermittent "invalid crontab expression"
+            # Retry loop: if port 8443 didn't come up within 180s, something
-            # bug (~50% first-boot hit rate) where registerJobs() fails during
+            # went wrong during first boot. A stop+start cycle can help with
-            # startup(). A stop+start cycle resolves it because the hard power-off
+            # transient failures. CAUTION: hard-stopping a VM during first boot
-            # may discard the corrupt state.txt before it's committed to disk,
+            # can corrupt the TPM encryption key — only retry if genuinely stuck.
-            # allowing initialize() to recreate it cleanly on next boot.
             local retry=0
             local retry_ok=false
             while [[ $retry -lt $INSTALL_RETRIES ]]; do
@@ -3543,19 +3638,32 @@ phase_install() {
                 fi
                 info "Stopping VM for retry..."
                 pve_stop_vm "$vmid" || true
-                sleep 10
+                # Poll until confirmed stopped (not just sleep)
+                local retry_stop_wait=0
+                while [[ $retry_stop_wait -lt 30 ]]; do
+                    local retry_status
+                    retry_status=$(pve_vm_status "$vmid") || retry_status="unknown"
+                    [[ "$retry_status" == "stopped" ]] && break
+                    sleep 2
+                    retry_stop_wait=$((retry_stop_wait + 2))
+                done
                 info "Starting VM (retry ${retry}/${INSTALL_RETRIES})..."
                 pve_start_vm "$vmid" || true
-                sleep 30
+                sleep 60
 
                 # Re-detect IP (may have changed after restart)
-                ip=$(pve_vm_ip "$vmid") || ip=""
+                ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
 
                 if [[ -n "$ip" ]]; then
                     local retry_poll=0
-                    while [[ $retry_poll -lt 12 ]]; do
+                    while [[ $retry_poll -lt 24 ]]; do
                         if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
+                            # Port is up — check and fix scrub_schedule
+                            local retry_fix=0
+                            fix_scrub_schedule "$ip" "$name" || retry_fix=$?
+                            if [[ $retry_fix -eq 0 ]]; then
                                 retry_ok=true
+                            fi
                             break
                         fi
                         sleep 5
@@ -3564,8 +3672,6 @@ phase_install() {
                 fi
 
                 if [[ "$retry_ok" == true ]]; then
-                    # Proactively fix empty scrub_schedule (IncusOS crontab bug).
-                    fix_scrub_schedule "$ip" "$name" || true
                     success "VM '${name}' recovered after retry ${retry}/${INSTALL_RETRIES} — running at ${ip}"
                     register_remote "$name" "$ip"
                     break
@@ -4119,7 +4225,7 @@ phase_status() {
         # Network and service checks (only for running, installed VMs)
         if [[ "$status" == "running" ]] && [[ "$has_media" == false ]]; then
             local ip
-            ip=$(pve_vm_ip "$vmid") || ip=""
+            ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
 
             if [[ -n "$ip" ]]; then
                 local port_status="${RED}closed${RESET}"
@@ -4206,7 +4312,7 @@ phase_post_deploy() {
         fi
 
         local ip
-        ip=$(pve_vm_ip "$vmid") || ip=""
+        ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
         if [[ -z "$ip" ]]; then
             warn "VM '${name}': could not detect IP address"
             all_ok=false

incusos/incusos-seed

@@ -70,6 +70,9 @@ ${BOLD}OPTIONS${RESET}
         ${BOLD}--hostname${RESET} NAME      Set system hostname
         ${BOLD}--domain${RESET} NAME        Set system domain
         ${BOLD}--dns${RESET} SERVERS        Comma-separated DNS servers (e.g. "8.8.8.8,1.1.1.1")
+        ${BOLD}--ip${RESET} ADDR/PREFIX    Static IP in CIDR notation (e.g. "192.168.102.100/22")
+        ${BOLD}--gateway${RESET} ADDR       Default gateway (required with --ip)
+        ${BOLD}--iface${RESET} NAME        Network interface name (default: ens18 for Proxmox VMs)
 
   ${BOLD}Update options:${RESET}
         ${BOLD}--channel${RESET} CHANNEL    Update channel: ${CYAN}stable${RESET}, ${CYAN}testing${RESET} (default: stable)
@@ -114,7 +117,7 @@ ${BOLD}SEED FILES REFERENCE${RESET}
     ${CYAN}applications.yaml${RESET}         Application selection (incus, operations-center)
     ${CYAN}incus.yaml${RESET}                Incus preseed (defaults, certificates)
     ${CYAN}operations-center.yaml${RESET}    Operations Center config (when --app=operations-center)
-    ${CYAN}network.yaml${RESET}              Network configuration (hostname, DNS)
+    ${CYAN}network.yaml${RESET}              Network configuration (hostname, DNS, static IP)
     ${CYAN}update.yaml${RESET}               Update channel and reboot policy
 
 EOF
@@ -139,6 +142,9 @@ MISSING_SECUREBOOT=false
 HOSTNAME_OPT=""
 DOMAIN_OPT=""
 DNS_SERVERS=""
+STATIC_IP=""
+GATEWAY=""
+IFACE="ens18"
 CHANNEL="stable"
 AUTO_REBOOT=false
 FORMAT="tar"
@@ -201,6 +207,18 @@ parse_args() {
                 DNS_SERVERS="${2:?'--dns requires a value'}"
                 shift 2
                 ;;
+            --ip)
+                STATIC_IP="${2:?'--ip requires ADDR/PREFIX (e.g. 192.168.102.100/22)'}"
+                shift 2
+                ;;
+            --gateway)
+                GATEWAY="${2:?'--gateway requires an address'}"
+                shift 2
+                ;;
+            --iface)
+                IFACE="${2:?'--iface requires a name'}"
+                shift 2
+                ;;
             --channel)
                 CHANNEL="${2:?'--channel requires a value'}"
                 shift 2
@@ -322,6 +340,16 @@ validate_args() {
         exit 1
     fi
 
+    if [[ -n "$STATIC_IP" ]] && [[ -z "$GATEWAY" ]]; then
+        error "--ip requires --gateway"
+        exit 1
+    fi
+
+    if [[ -n "$STATIC_IP" ]] && [[ "$STATIC_IP" != */* ]]; then
+        error "--ip must include prefix length (e.g. 192.168.102.100/22)"
+        exit 1
+    fi
+
     if [[ -n "$CERT_FILE" ]] && [[ ! -f "$CERT_FILE" ]]; then
         error "Certificate file not found: ${CERT_FILE}"
         exit 1
@@ -578,7 +606,7 @@ generate_ops_center_yaml() {
 
 generate_network_yaml() {
     # Only generate if network options were specified
-    if [[ -z "$HOSTNAME_OPT" ]] && [[ -z "$DOMAIN_OPT" ]] && [[ -z "$DNS_SERVERS" ]]; then
+    if [[ -z "$HOSTNAME_OPT" ]] && [[ -z "$DOMAIN_OPT" ]] && [[ -z "$DNS_SERVERS" ]] && [[ -z "$STATIC_IP" ]]; then
         return
     fi
 
@@ -602,6 +630,21 @@ generate_network_yaml() {
                 done
             fi
         fi
+
+        # Static IP configuration
+        if [[ -n "$STATIC_IP" ]]; then
+            echo ""
+            echo "interfaces:"
+            echo "  - name: \"mgmt\""
+            echo "    hwaddr: \"${IFACE}\""
+            echo "    addresses:"
+            echo "      - \"${STATIC_IP}\""
+            echo "    roles:"
+            echo "      - management"
+            echo "    routes:"
+            echo "      - to: \"0.0.0.0/0\""
+            echo "        via: \"${GATEWAY}\""
+        fi
     } > "$file"
 
     if [[ "$DRY_RUN" == true ]]; then
@@ -610,6 +653,7 @@ generate_network_yaml() {
     else
         local desc=""
         [[ -n "$HOSTNAME_OPT" ]] && desc+="hostname=${HOSTNAME_OPT}"
+        [[ -n "$STATIC_IP" ]]   && desc+=", ip=${STATIC_IP}"
         [[ -n "$DNS_SERVERS" ]]  && desc+=", dns=${DNS_SERVERS}"
         success "network.yaml (${desc})"
     fi

incusos/observe-deploy

@@ -13,7 +13,9 @@
 #   ./observe-deploy                          # Standard deploy with screenshots
 #   ./observe-deploy --no-update-yaml         # Omit update.yaml from seed
 #   ./observe-deploy --label run-01           # Custom run label
+#   ./observe-deploy --runs 10 --label baseline  # 10 sequential cycles with stats
 #   ./observe-deploy --vmid 855              # Use a specific VMID
+#   ./observe-deploy --interval 1            # 1-second screenshots
 #   ./observe-deploy --dry-run               # Preview without deploying
 #
 # Output: timestamped directory with PNG frames + deploy.log
@@ -37,7 +39,12 @@ SEED_TOOL="${SCRIPT_DIR}/incusos-seed"
 VMID=850
 VM_NAME="observe-deploy"
 LABEL=""
+RUNS=1                      # number of sequential deploy+destroy cycles
 NO_UPDATE_YAML=false
+NO_VLAN=false
+STATIC_IP=""                # Static IP in CIDR notation (e.g. 192.168.102.100/22)
+VM_EXTRA=""                 # Extra VM creation overrides (key=value,key=value)
+NO_FORCE_REBOOT=false       # Skip force_reboot in seed (no SysRq-B reboot after install)
 DRY_RUN=false
 SCREENSHOT_INTERVAL=2       # seconds between screenshots
 INSTALL_TIMEOUT=600         # seconds to wait for install completion
@@ -52,6 +59,9 @@ PVE_STORAGE=""
 PVE_ISO_STORAGE=""
 PVE_BRIDGE=""
 PVE_POOL=""
+PVE_VLAN=""
+PVE_GATEWAY=""
+PVE_DNS=""
 PVE_API_TOKEN_ID=""
 
 # IncusOS ISO
@@ -67,6 +77,8 @@ PHASE="init"   # init → iso_boot → installing → install_done → transitio
 PREV_WR_BYTES=0
 WRITES_STARTED=false
 STABLE_COUNT=0
+SCRUB_SCHEDULE=""   # Filled after port 8443 check: "0 4 * * 0" = healthy, "" = crontab bug
+DEPLOY_IP=""        # IP of the deployed VM (set during disk_boot phase)
 
 # ---------------------------------------------------------------------------
 # Colors
@@ -119,7 +131,12 @@ while [[ $# -gt 0 ]]; do
     case "$1" in
         --vmid)           VMID="${2:?'--vmid requires a number'}"; shift 2 ;;
         --label)          LABEL="${2:?'--label requires a string'}"; shift 2 ;;
+        --runs)           RUNS="${2:?'--runs requires a number'}"; shift 2 ;;
         --no-update-yaml) NO_UPDATE_YAML=true; shift ;;
+        --no-force-reboot) NO_FORCE_REBOOT=true; shift ;;
+        --no-vlan)        NO_VLAN=true; shift ;;
+        --ip)             STATIC_IP="${2:?'--ip requires ADDR/PREFIX'}"; shift 2 ;;
+        --vm-extra)       VM_EXTRA="${2:?'--vm-extra requires key=value pairs'}"; shift 2 ;;
         --interval)       SCREENSHOT_INTERVAL="${2:?'--interval requires seconds'}"; shift 2 ;;
         --dry-run)        DRY_RUN=true; shift ;;
         -h|--help)
@@ -132,7 +149,12 @@ ${BOLD}USAGE${RESET}
 ${BOLD}OPTIONS${RESET}
     --vmid N              VMID to use (default: 850, must be 850-869)
     --label NAME          Run label for output directory (default: auto)
+    --runs N              Run N sequential deploy+destroy cycles with stats (default: 1)
     --no-update-yaml      Omit update.yaml from seed (Group A equivalent)
+    --no-force-reboot     Omit force_reboot from seed (no SysRq-B after install)
+    --no-vlan             Skip VLAN tag (use native LAN for IP detection)
+    --ip ADDR/PREFIX      Static IP for the VM (e.g. 192.168.102.100/22)
+                          Skips ARP detection; uses known IP for health checks
     --interval N          Screenshot interval in seconds (default: 2)
     --dry-run             Preview without deploying
     -h, --help            Show this help
@@ -215,6 +237,12 @@ load_proxmox_yaml() {
     PVE_ISO_STORAGE=$(yaml_get "$yaml_file" "iso_storage" "local")
     PVE_BRIDGE=$(yaml_get "$yaml_file" "bridge" "vmbr0")
     PVE_POOL=$(yaml_get "$yaml_file" "pool")
+    PVE_VLAN=$(yaml_get "$yaml_file" "vlan")
+    PVE_GATEWAY=$(yaml_get "$yaml_file" "gateway")
+    PVE_DNS=$(yaml_get "$yaml_file" "dns")
+    if [[ "$NO_VLAN" == true ]]; then
+        PVE_VLAN=""
+    fi
     PVE_API_TOKEN_ID=$(yaml_get "$yaml_file" "api_token_id")
 
     if [[ -z "$PVE_HOST" ]]; then
@@ -319,6 +347,13 @@ print(m.group(1).lower() if m else '')
 
 pve_vm_ip() {
     local vmid="$1"
+
+    # If static IP is configured, return the bare IP (strip CIDR prefix)
+    if [[ -n "$STATIC_IP" ]]; then
+        echo "${STATIC_IP%%/*}"
+        return 0
+    fi
+
     local mac
     mac=$(pve_vm_mac "$vmid") || return 1
     [[ -z "$mac" ]] && return 1
@@ -373,6 +408,56 @@ for h in net.hosts():
     return 1
 }
 
+# ---------------------------------------------------------------------------
+# scrub_schedule health check (definitive crontab bug indicator)
+# ---------------------------------------------------------------------------
+
+check_scrub_schedule() {
+    # Query the IncusOS REST API for the scrub_schedule field.
+    # This is the definitive health indicator:
+    #   "0 4 * * 0"  → genuine success
+    #   ""           → crontab bug hit (even if port 8443 responds)
+    #   unreachable  → daemon crashed before API started
+    #
+    # Sets global SCRUB_SCHEDULE and returns 0 on success, 1 on bug/failure.
+    local ip="$1"
+
+    local cert="${HOME}/.config/incus/client.crt"
+    local key="${HOME}/.config/incus/client.key"
+
+    if [[ ! -f "$cert" ]] || [[ ! -f "$key" ]]; then
+        log "No client certificate — skipping scrub_schedule check"
+        SCRUB_SCHEDULE="unchecked"
+        return 0
+    fi
+
+    local response
+    response=$(curl -sk --connect-timeout 5 --max-time 10 \
+        --cert "$cert" --key "$key" \
+        "https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
+        log "Could not query IncusOS storage API at ${ip}"
+        SCRUB_SCHEDULE="unreachable"
+        return 1
+    }
+
+    SCRUB_SCHEDULE=$(printf '%s' "$response" | python3 -c "
+import sys, json
+try:
+    d = json.load(sys.stdin)
+    print(d.get('metadata', {}).get('config', {}).get('scrub_schedule', ''))
+except:
+    print('')
+" 2>/dev/null) || SCRUB_SCHEDULE=""
+
+    if [[ -n "$SCRUB_SCHEDULE" ]]; then
+        log "scrub_schedule = '${SCRUB_SCHEDULE}' → HEALTHY"
+        return 0
+    else
+        log "scrub_schedule is EMPTY → crontab bug detected"
+        return 1
+    fi
+}
+
 # ---------------------------------------------------------------------------
 # Screenshot capture
 # ---------------------------------------------------------------------------
@@ -478,9 +563,18 @@ generate_seed() {
         info "Omitting update.yaml from seed (Group A equivalent)"
 
         # Generate full seed as tar, extract, remove update.yaml, create ISO
+        local seed_extra=()
+        if [[ -n "$STATIC_IP" ]]; then
+            seed_extra+=(--ip "$STATIC_IP" --gateway "$PVE_GATEWAY")
+            [[ -n "$PVE_DNS" ]] && seed_extra+=(--dns "$PVE_DNS")
+        fi
+
+        local reboot_flag=()
+        [[ "$NO_FORCE_REBOOT" != true ]] && reboot_flag=(--force-reboot)
+
         "$SEED_TOOL" --format tar --app incus --output "${SEED_TMPDIR}/seed.tar" \
-            --force-install --force-reboot --hostname "$VM_NAME" \
+            --force-install "${reboot_flag[@]+"${reboot_flag[@]}"}" --hostname "$VM_NAME" \
-            --defaults --quiet &>/dev/null
+            --defaults "${seed_extra[@]+"${seed_extra[@]}"}" --quiet &>/dev/null
 
         local seeddir="${SEED_TMPDIR}/seedfiles"
         mkdir -p "$seeddir"
@@ -505,11 +599,20 @@ generate_seed() {
     else
         info "Standard seed (with update.yaml, check_frequency=6h)"
 
-        "$SEED_TOOL" --format iso --app incus --output "$local_path" \
+        local seed_extra=()
-            --force-install --force-reboot --hostname "$VM_NAME" \
+        if [[ -n "$STATIC_IP" ]]; then
-            --defaults --quiet &>/dev/null
+            seed_extra+=(--ip "$STATIC_IP" --gateway "$PVE_GATEWAY")
+            [[ -n "$PVE_DNS" ]] && seed_extra+=(--dns "$PVE_DNS")
+        fi
 
-        log "Seed: standard incusos-seed output"
+        local reboot_flag=()
+        [[ "$NO_FORCE_REBOOT" != true ]] && reboot_flag=(--force-reboot)
+
+        "$SEED_TOOL" --format iso --app incus --output "$local_path" \
+            --force-install "${reboot_flag[@]+"${reboot_flag[@]}"}" --hostname "$VM_NAME" \
+            --defaults "${seed_extra[@]+"${seed_extra[@]}"}" --quiet &>/dev/null
+
+        log "Seed: standard incusos-seed output (force_reboot=$([[ "$NO_FORCE_REBOOT" == true ]] && echo false || echo true))"
     fi
 
     success "Seed ISO: ${seed_name}"
@@ -554,13 +657,25 @@ payload = {
     'scsi0': '${PVE_STORAGE}:50,iothread=1',
     'ide2': '${PVE_ISO_STORAGE}:iso/${ISO_FILENAME},media=cdrom',
     'ide3': '${PVE_ISO_STORAGE}:iso/${SEED_ISO},media=cdrom',
-    'net0': 'virtio,bridge=${PVE_BRIDGE}',
+    'net0': 'virtio,bridge=${PVE_BRIDGE}' + (',tag=${PVE_VLAN}' if '${PVE_VLAN}' else ''),
     'boot': 'order=ide2;scsi0',
     'agent': 1,
 }
 pool = '${PVE_POOL}'
 if pool:
     payload['pool'] = pool
+# Apply --vm-extra overrides (e.g. 'scsi0=local-zfs:50,cache=writethrough,iothread=1')
+vm_extra = '${VM_EXTRA}'
+if vm_extra:
+    for pair in vm_extra.split(',,,'):  # triple-comma separator for multi-value pairs
+        if '=' in pair:
+            k, v = pair.split('=', 1)
+            # Convert numeric strings to ints for JSON
+            try:
+                v = int(v)
+            except (ValueError, TypeError):
+                pass
+            payload[k.strip()] = v
 print(json.dumps(payload))
 ")
 
@@ -793,6 +908,7 @@ run_deploy() {
                 # Check port 8443
                 if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
                     log "Port 8443 responding at ${ip} (${boot_elapsed}s into disk boot)"
+                    DEPLOY_IP="$ip"
                     PHASE="ready"
                     break
                 else
@@ -825,11 +941,20 @@ run_deploy() {
         done
     fi
 
+    # Definitive health check: query scrub_schedule via IncusOS REST API
+    if [[ "$PHASE" == "ready" ]] && [[ -n "$DEPLOY_IP" ]]; then
+        step "Checking scrub_schedule (definitive health indicator)"
+        if ! check_scrub_schedule "$DEPLOY_IP"; then
+            PHASE="crontab_bug"
+            log "Deploy reached port 8443 but scrub_schedule is empty — crontab bug"
+        fi
+    fi
+
     local deploy_end
     deploy_end=$(date +%s)
     local total_seconds=$((deploy_end - deploy_start))
 
-    log "Deploy finished: phase=${PHASE}, total_time=${total_seconds}s, frames=${FRAME_NUM}"
+    log "Deploy finished: phase=${PHASE}, scrub_schedule=${SCRUB_SCHEDULE:-unset}, total_time=${total_seconds}s, frames=${FRAME_NUM}"
 
     if [[ "$PHASE" == "ready" ]]; then
         return 0
@@ -931,7 +1056,7 @@ if [[ "$DRY_RUN" == true ]]; then
     step "Would load proxmox.yaml"
     load_proxmox_yaml
     detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}"
-    detail "Pool: ${PVE_POOL}, Token: ${PVE_API_TOKEN_ID}"
+    detail "Pool: ${PVE_POOL}, VLAN: ${PVE_VLAN:-none}, Token: ${PVE_API_TOKEN_ID}"
 
     echo ""
     step "Would check:"
@@ -949,8 +1074,12 @@ if [[ "$DRY_RUN" == true ]]; then
     detail "5. Monitor blockstat for install completion"
     detail "6. Stop VM, remove media, start from disk"
     detail "7. Continue screenshots until port 8443 responds (or ${BOOT_TIMEOUT}s timeout)"
-    detail "8. Save frames + deploy.log"
+    detail "8. Check scrub_schedule via IncusOS REST API"
-    detail "9. Cleanup VM"
+    detail "9. Save frames + deploy.log"
+    detail "10. Cleanup VM"
+    if [[ $RUNS -gt 1 ]]; then
+        detail "Repeat ${RUNS} times with aggregate statistics"
+    fi
 
     echo ""
     info "Dry run complete"
@@ -960,7 +1089,14 @@ fi
 # Load configuration
 step "Loading configuration"
 load_proxmox_yaml
-detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}"
+detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}, VLAN: ${PVE_VLAN:-none}"
+if [[ -n "$STATIC_IP" ]]; then
+    detail "Static IP: ${STATIC_IP}, Gateway: ${PVE_GATEWAY:-MISSING}"
+    if [[ -z "$PVE_GATEWAY" ]]; then
+        error "--ip requires 'gateway' in proxmox.yaml"
+        exit 1
+    fi
+fi
 
 # Run preflight checks
 preflight
@@ -968,32 +1104,6 @@ preflight
 # Find IncusOS ISO
 find_incusos_iso
 
-# Setup output directory
-setup_output
-
-# Log configuration
-log "=== observe-deploy v${VERSION} ==="
-log "VMID: ${VMID}"
-log "VM name: ${VM_NAME}"
-log "ISO: ${ISO_FILENAME}"
-log "Update.yaml: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "OMITTED"; else echo "included (6h)"; fi)"
-log "Screenshot interval: ${SCREENSHOT_INTERVAL}s"
-log "Install timeout: ${INSTALL_TIMEOUT}s"
-log "Boot timeout: ${BOOT_TIMEOUT}s"
-
-# Generate and upload seed
-generate_seed
-
-# Create VM
-create_vm
-
-echo ""
-echo -e "${BOLD}Starting observational deploy${RESET}"
-echo "  Frames saved to: ${RUN_DIR}/"
-echo "  Log: ${RUN_DIR}/deploy.log"
-echo "  Press Ctrl+C to abort (VM will be cleaned up)"
-echo ""
-
 # Trap for clean exit
 trap_cleanup() {
     echo ""
@@ -1003,39 +1113,213 @@ trap_cleanup() {
 }
 trap trap_cleanup INT TERM
 
-# Run the deploy with screenshots
+# ---------------------------------------------------------------------------
-deploy_result=0
+# Run loop (single run or multi-run with --runs N)
-run_deploy || deploy_result=$?
+# ---------------------------------------------------------------------------
 
-echo ""
+run_single_cycle() {
-echo -e "${BOLD}========================================${RESET}"
+    # Runs one complete deploy+destroy cycle. Returns 0 on success, 1 on failure.
-echo -e "${BOLD}  Deploy Results${RESET}"
+    # Sets globals: PHASE, SCRUB_SCHEDULE, DEPLOY_IP, FRAME_NUM
-echo -e "${BOLD}========================================${RESET}"
-echo ""
-echo "  VMID:       ${VMID}"
-echo "  Seed:       $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "no update.yaml"; else echo "standard (check_frequency=6h)"; fi)"
-echo "  Outcome:    $(if [[ $deploy_result -eq 0 ]]; then echo -e "${GREEN}PASS${RESET} — port 8443 reachable"; else echo -e "${RED}FAIL${RESET} — phase: ${PHASE}"; fi)"
-echo "  Frames:     ${FRAME_NUM}"
-echo "  Output:     ${RUN_DIR}/"
-echo ""
 
-# Cleanup
+    # Reset state for this run
-step "Post-deploy cleanup"
+    PHASE="init"
-cleanup_vm
+    PREV_WR_BYTES=0
+    WRITES_STARTED=false
+    STABLE_COUNT=0
+    SCRUB_SCHEDULE=""
+    DEPLOY_IP=""
+    FRAME_NUM=0
 
-if [[ $deploy_result -eq 0 ]]; then
+    # Setup per-run output directory
+    setup_output
+
+    # Log configuration
+    log "=== observe-deploy v${VERSION} ==="
+    log "VMID: ${VMID}"
+    log "VM name: ${VM_NAME}"
+    log "ISO: ${ISO_FILENAME}"
+    log "Update.yaml: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "OMITTED"; else echo "included (6h)"; fi)"
+    log "Screenshot interval: ${SCREENSHOT_INTERVAL}s"
+    log "Install timeout: ${INSTALL_TIMEOUT}s"
+    log "Boot timeout: ${BOOT_TIMEOUT}s"
+
+    # Generate and upload seed
+    generate_seed
+
+    # Create VM
+    create_vm
+
+    echo ""
+    echo -e "${BOLD}Starting observational deploy${RESET}"
+    echo "  Frames saved to: ${RUN_DIR}/"
+    echo "  Log: ${RUN_DIR}/deploy.log"
+    echo "  Press Ctrl+C to abort (VM will be cleaned up)"
+    echo ""
+
+    # Run the deploy with screenshots
+    local result=0
+    run_deploy || result=$?
+
+    echo ""
+    echo -e "${BOLD}========================================${RESET}"
+    echo -e "${BOLD}  Deploy Results${RESET}"
+    echo -e "${BOLD}========================================${RESET}"
+    echo ""
+    echo "  VMID:       ${VMID}"
+    echo "  Seed:       $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "no update.yaml"; else echo "standard (check_frequency=6h)"; fi)"
+    if [[ $result -eq 0 ]]; then
+        echo -e "  Outcome:    ${GREEN}PASS${RESET} — port 8443 reachable, scrub_schedule='${SCRUB_SCHEDULE}'"
+    elif [[ "$PHASE" == "crontab_bug" ]]; then
+        echo -e "  Outcome:    ${RED}FAIL${RESET} — crontab bug (port 8443 up, scrub_schedule empty)"
+    else
+        echo -e "  Outcome:    ${RED}FAIL${RESET} — phase: ${PHASE}"
+    fi
+    echo "  Frames:     ${FRAME_NUM}"
+    echo "  Output:     ${RUN_DIR}/"
+    echo ""
+
+    # Cleanup
+    step "Post-deploy cleanup"
+    cleanup_vm
+
+    return $result
+}
+
+if [[ $RUNS -le 1 ]]; then
+    # Single-run mode (original behavior)
+    deploy_result=0
+    run_single_cycle || deploy_result=$?
+
+    if [[ $deploy_result -eq 0 ]]; then
         success "Observational deploy complete — ${FRAME_NUM} frames captured"
-else
+    else
         warn "Deploy failed at phase: ${PHASE}"
         warn "Check screenshots and log for diagnosis:"
         warn "  ${RUN_DIR}/deploy.log"
         warn "  ${RUN_DIR}/frame-*.png"
+    fi
+
+    echo ""
+    echo "To view key frames:"
+    echo "  ls ${RUN_DIR}/frame-*.png"
+    echo "  # View in image viewer or analyze with Claude Code"
+    echo ""
+
+    exit $deploy_result
+else
+    # Multi-run mode: N sequential deploy+destroy cycles with aggregate stats
+    echo ""
+    echo -e "${BOLD}Multi-run mode: ${RUNS} sequential deploy cycles${RESET}"
+    echo ""
+
+    # Compute base label before the loop (LABEL may be empty at this point)
+    if [[ -z "$LABEL" ]]; then
+        if [[ "$NO_UPDATE_YAML" == true ]]; then
+            BASE_LABEL="no-update-yaml"
+        else
+            BASE_LABEL="standard"
+        fi
+    else
+        BASE_LABEL="$LABEL"
+    fi
+
+    total_pass=0
+    total_fail=0
+    total_crontab_bug=0
+    total_timeout=0
+    total_time=0
+    run_results=()   # Array of "PASS|FAIL|BUG" per run
+
+    overall_start=$(date +%s)
+
+    run_num=0
+    while [[ $run_num -lt $RUNS ]]; do
+        run_num=$((run_num + 1))
+        echo ""
+        echo -e "${BOLD}╔══════════════════════════════════════╗${RESET}"
+        echo -e "${BOLD}║  Run ${run_num}/${RUNS}${RESET}"
+        echo -e "${BOLD}╚══════════════════════════════════════╝${RESET}"
+        echo ""
+
+        # Set per-run label
+        LABEL="${BASE_LABEL}_run-$(printf '%02d' "$run_num")"
+
+        local_start=$(date +%s)
+
+        deploy_result=0
+        run_single_cycle || deploy_result=$?
+
+        local_end=$(date +%s)
+        local_duration=$((local_end - local_start))
+        total_time=$((total_time + local_duration))
+
+        if [[ $deploy_result -eq 0 ]]; then
+            total_pass=$((total_pass + 1))
+            run_results+=("PASS")
+            success "Run ${run_num}/${RUNS}: PASS (${local_duration}s)"
+        elif [[ "$PHASE" == "crontab_bug" ]]; then
+            total_fail=$((total_fail + 1))
+            total_crontab_bug=$((total_crontab_bug + 1))
+            run_results+=("BUG")
+            warn "Run ${run_num}/${RUNS}: FAIL — crontab bug (${local_duration}s)"
+        else
+            total_fail=$((total_fail + 1))
+            total_timeout=$((total_timeout + 1))
+            run_results+=("FAIL")
+            warn "Run ${run_num}/${RUNS}: FAIL — ${PHASE} (${local_duration}s)"
+        fi
+
+        # Brief pause between runs to let Proxmox settle
+        if [[ $run_num -lt $RUNS ]]; then
+            info "Pausing 10s before next run..."
+            sleep 10
+        fi
+    done
+
+    overall_end=$(date +%s)
+    overall_duration=$((overall_end - overall_start))
+    avg_time=$((total_time / RUNS))
+
+    echo ""
+    echo -e "${BOLD}╔══════════════════════════════════════════════════╗${RESET}"
+    echo -e "${BOLD}║  Multi-Run Summary (${RUNS} cycles)${RESET}"
+    echo -e "${BOLD}╚══════════════════════════════════════════════════╝${RESET}"
+    echo ""
+    echo -e "  ${GREEN}PASS:${RESET}           ${total_pass}/${RUNS}"
+    echo -e "  ${RED}FAIL:${RESET}           ${total_fail}/${RUNS}"
+    if [[ $total_crontab_bug -gt 0 ]]; then
+        echo -e "    ${RED}crontab bug:${RESET}  ${total_crontab_bug}"
+    fi
+    if [[ $total_timeout -gt 0 ]]; then
+        echo -e "    ${RED}timeout/other:${RESET} ${total_timeout}"
+    fi
+    echo ""
+    echo "  Avg time:     ${avg_time}s per run"
+    echo "  Total time:   ${overall_duration}s"
+    echo ""
+
+    # Per-run results
+    echo "  Per-run: ${run_results[*]}"
+    echo ""
+
+    # Success rate
+    if [[ $RUNS -gt 0 ]]; then
+        pct=$((total_pass * 100 / RUNS))
+        echo -e "  Success rate: ${pct}%"
+        if [[ $pct -ge 95 ]]; then
+            echo -e "  ${GREEN}Target met (≥95%)${RESET}"
+        else
+            echo -e "  ${RED}Below target (<95%)${RESET}"
+        fi
+    fi
+    echo ""
+
+    # Output directories
+    echo "  Output: ${SCRIPT_DIR}/observe-runs/*${BASE_LABEL}*"
+    echo ""
+
+    if [[ $total_fail -gt 0 ]]; then
+        exit 1
+    fi
+    exit 0
 fi
-
-echo ""
-echo "To view key frames:"
-echo "  ls ${RUN_DIR}/frame-*.png"
-echo "  # View in image viewer or analyze with Claude Code"
-echo ""
-
-exit $deploy_result