diff --git a/CLAUDE.md b/CLAUDE.md index 0e60e66..d5caf44 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -130,22 +130,33 @@ incu-contrib/ `scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit `disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs). - **Install flow** (automated by `incusos-proxmox`): - 1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed - 2. IncusOS reads seed, installs to disk (scsi0), auto-reboots + 1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed + 2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone), + then sits at "please remove installation media" prompt 3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API -- when disk writes start then stop for 15s (3 stable polls), install is done 4. Stop the VM (Proxmox stop, not guest shutdown) 5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot and refuses to start if found, regardless of boot order 6. Set boot order to `order=scsi0` and start from disk -- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use - ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP → - ping broadcast → look up MAC in ARP table. Verify with direct ping before - trusting the result. -- **force_reboot is required**: without `force_reboot: true` in the seed, - IncusOS sits at "please remove installation media" and waits indefinitely. - It does NOT halt automatically. `force_reboot` triggers a guest-level reboot - (note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does). +- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two + strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM + in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's + `network.yaml` configures the VM's interface at boot. No ARP scan or + SSH needed — the IP is known at deploy time. Works across VLANs. + (2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM + config → flush stale ARP → ping broadcast → look up MAC in ARP table. + Only works on the same L2 domain (native LAN, not across VLANs). +- **force_reboot is NOT used on Proxmox** (changed 2026-02-22): the seed + omits `force_reboot`. Without it, IncusOS sits at "please remove + installation media" after install — which is exactly what we want. We + detect completion via blockstat (876 MiB written, then idle), stop the + VM externally, remove media, and start from disk. This eliminates the + SysRq-B intermediate boot that triggers the crontab race condition + (issue #843). On physical hardware or unmanaged VMs, `force_reboot: true` + is still needed — this optimization is specific to Proxmox automated deploys. + `force_reboot` triggers SysRq-B (raw kernel reboot via `/proc/sysrq-trigger`). + This does NOT reset QEMU VM uptime — only a Proxmox stop/start does. - **Resource pool isolation**: the optional `proxmox.pool` config field scopes all VM operations to a Proxmox resource pool. When set, the script only "sees" VMs in that pool (for collision detection and cleanup), and the API @@ -511,10 +522,18 @@ incu-contrib/ except warnings and errors. - **`--retries N`**: number of stop+start retries for VMs that fail to boot (port 8443 not reachable). Default: 3. `--retries 0` disables retries. - Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443. - With ~67% boot failure rate from the crontab bug (confirmed by parallel - investigation), 3 retries gives ~96.3% success rate (1 - 0.67^3). After retries, `fix_scrub_schedule()` proactively heals the - root cause via the IncusOS REST API. + Each retry: Proxmox stop → wait → start → 60s wait → poll port 8443 for + up to 120s. With the `force_reboot` fix (2026-02-22), the crontab bug is + eliminated and retries are rarely needed. They remain as a safety net for + other transient boot failures. After retries, `fix_scrub_schedule()` + proactively heals any remaining scrub_schedule issues via the IncusOS + REST API. +- **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot + downloads SecureBoot update + application sysext (Incus: ~30-120s depending + on CDN speed and concurrent VMs). The previous 90s timeout was too short + under load — premature retries (hard VM stop during first boot) can corrupt + the TPM encryption key, causing permanent "zfs load-key: Raw key too short + (expected 32)" errors that no amount of retries can fix. ### IncusOS first-boot sequence (observed via console screenshots) @@ -570,9 +589,19 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`. ### IncusOS boot failure (crontab / update frequency error) +- **RESOLVED** (2026-02-22): root cause identified and fixed in our pipeline. + The `force_reboot` seed option triggers SysRq-B after install, causing an + "install media detected" intermediate boot. This intermediate boot generates + ~26 GB of additional disk writes (the IncusOS daemon starts from ISO tmpfs). + Our blockstat detection catches these late writes, but by then the + intermediate boot has run and somehow corrupted the state for the subsequent + first-from-disk boot. **Fix: omit `force_reboot` from the seed.** Without + it, the installer sits at "please remove media" after the 876 MiB image + clone, blockstat detects idle cleanly, and the first disk boot is pristine. + Results: **15/15 PASS (100%)** without force_reboot on 4 cores vs 50% with it. - **Symptom**: "ERROR invalid crontab expression" appears on the console - during first boot. Intermittent — confirmed **67% failure rate** across - 20 parallel VM deploys (8/12 API-verified). Error does NOT correlate with + during first boot. Intermittent — confirmed **~50% failure rate** with + `force_reboot: true` on 4-core VMs. Error does NOT correlate with `update.yaml` presence/absence, ISO version, or version mismatch. - **Two distinct error paths in IncusOS**: - `registerJobs()` validates `ScrubSchedule` (5-field crontab expression @@ -613,11 +642,25 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`. - **Race window**: The REST API server starts BEFORE `startup()` completes. Applications start before `registerJobs()`. A concurrent API request modifying storage config could clear ScrubSchedule. - - **Most likely root cause**: a race condition between the REST API/ - application startup and `registerJobs()` that clears `ScrubSchedule`. - Investigation ruled out version mismatch (bug hits equally with matching - ISO), update downloads (occurs even without OS updates), and gocron - library issues (the field is genuinely empty in the state struct). + - **Root cause confirmed**: the bug is triggered by the SysRq-B + intermediate boot (from `force_reboot: true`). Controlled experiments + on VLAN 69 with static IP (2026-02-22, ISO 202602210344): + - Baseline (4 cores, force_reboot): 5/10 PASS (50%) + - cache=writethrough: 3/5 PASS (60%) — falsified + - agent=0: 3/5 PASS (60%) — falsified + - cores=1 (force_reboot): 14/15 PASS (93%) — GOMAXPROCS=1 reduces race + - **No force_reboot (4 cores): 15/15 PASS (100%)** — confirmed fix + Correlation: runs with 876 MiB blockstat detection (before SysRq-B) + had 100% success; runs with 27 GB detection (after SysRq-B) had ~55%. + - **The try-it service is NOT IncusOS**: stgraber's comparison ("thousands + of times a day") is invalid. The try-it service runs Ubuntu VMs with + Incus installed from the Zabbly daily repo — a completely different + product. Issue #843 was independently filed by another Proxmox user + (fperreau on Proxmox 9.1.4), confirming it's not unique to our setup. + - The underlying IncusOS bug is still a race condition between the REST + API/application startup and `registerJobs()` that clears `ScrubSchedule`. + Investigation ruled out version mismatch, update downloads, and gocron + library issues. The field is genuinely empty in the state struct. - **Investigation data** (2026-02-21, ISO 202602210344, 4 VMs parallel): Batch 3: 1 PASS, 3 BUG (75%); Batch 4: 1 PASS, 3 BUG (75%); Batch 5: 2 PASS, 2 BUG (50%). Total API-verified: 4 PASS, 8 BUG @@ -671,8 +714,16 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`. has a journal commit interval (~5s) and the corrupt Save() may not have been committed before the VM was killed. - **Detection**: `phase_install` checks port 8443 after starting from disk - (30s wait + up to 60s polling). If first check fails, automatic retry - adds ~100s. Failed VMs after retry are reported with remediation. + (60s wait + up to 120s polling = 180s total). If first check fails, + automatic retry adds ~180s. Failed VMs after retry are reported with + remediation. +- **TPM corruption from premature retry**: hard-stopping a VM during first + boot (while the encryption key is being written to the TPM zvol) can + leave the TPM with truncated key data. All subsequent boots fail with + "zfs load-key: Raw key too short (expected 32)" — a permanent error + that requires VM destruction and redeployment to fix. The 180s boot + timeout prevents this by allowing the full first-boot sequence (including + sysext download) to complete before any retry. - **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes `fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix diff --git a/incusos/examples/lab-cluster.yaml b/incusos/examples/lab-cluster.yaml index ec580c4..a94ae72 100644 --- a/incusos/examples/lab-cluster.yaml +++ b/incusos/examples/lab-cluster.yaml @@ -29,11 +29,14 @@ vms: - name: incus-lab-01 app: incus apply_defaults: true # init node: needs storage pool + network + ip: 192.168.102.101/22 - name: incus-lab-02 app: incus apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.102/22 - name: incus-lab-03 app: incus apply_defaults: false # joining node: cluster join creates pool entry + ip: 192.168.102.103/22 diff --git a/incusos/examples/lab-oc.yaml b/incusos/examples/lab-oc.yaml index 9362abc..c9ad71d 100644 --- a/incusos/examples/lab-oc.yaml +++ b/incusos/examples/lab-oc.yaml @@ -29,6 +29,7 @@ vms: - name: lab-oc app: operations-center apply_defaults: true + ip: 192.168.102.110/22 - name: lab-node-01 app: incus @@ -36,11 +37,14 @@ vms: cores: 4 memory: 8192 disk: 64 + ip: 192.168.102.111/22 - name: lab-node-02 app: incus apply_defaults: true + ip: 192.168.102.112/22 - name: lab-node-03 app: incus apply_defaults: true + ip: 192.168.102.113/22 diff --git a/incusos/examples/lab-single.yaml b/incusos/examples/lab-single.yaml index c8a14f4..df35d31 100644 --- a/incusos/examples/lab-single.yaml +++ b/incusos/examples/lab-single.yaml @@ -23,3 +23,4 @@ vms: - name: incus-single app: incus apply_defaults: true + ip: 192.168.102.100/22 diff --git a/incusos/examples/network-static.yaml b/incusos/examples/network-static.yaml index 2adbd1a..cb9c2dc 100644 --- a/incusos/examples/network-static.yaml +++ b/incusos/examples/network-static.yaml @@ -21,7 +21,7 @@ time: interfaces: - name: "mgmt" - hwaddr: "enp5s0" # interface name or MAC address + hwaddr: "ens18" # interface name or MAC (ens18 = Proxmox VirtIO) addresses: - "192.168.1.100/24" roles: diff --git a/incusos/examples/proxmox.yaml.example b/incusos/examples/proxmox.yaml.example index 8342be4..d1398b0 100644 --- a/incusos/examples/proxmox.yaml.example +++ b/incusos/examples/proxmox.yaml.example @@ -17,5 +17,8 @@ node: pve # Proxmox node name storage: local-lvm # VM disk storage iso_storage: local # ISO/image storage bridge: vmbr0 # Network bridge +# vlan: 69 # VLAN tag for VM NIC (optional) +# gateway: 192.168.100.1 # Default gateway for static IPs (required if VMs use ip:) +# dns: 192.168.100.1 # DNS server for static IPs (optional, used with ip:) # pool: IncusLab # Resource pool (optional, for isolation) ssh_user: root # SSH user (for ssh method) diff --git a/incusos/incusos-proxmox b/incusos/incusos-proxmox index c539aad..86c4a44 100755 --- a/incusos/incusos-proxmox +++ b/incusos/incusos-proxmox @@ -1296,6 +1296,8 @@ PVE_STORAGE="" PVE_ISO_STORAGE="" PVE_BRIDGE="" PVE_VLAN="" +PVE_GATEWAY="" +PVE_DNS="" PVE_METHOD="" PVE_SSH_USER="" PVE_API_TOKEN_ID="" @@ -1316,6 +1318,7 @@ declare -a VM_CORES=() declare -a VM_MEMORY=() declare -a VM_DISK=() declare -a VM_VMIDS=() +declare -a VM_IPS=() declare -a VM_CLUSTERS=() declare -a VM_CLUSTER_ROLES=() @@ -1396,6 +1399,8 @@ json.dump(result, sys.stdout, indent=2) val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val" val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val" val=$(json_get "$px_json" ".vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val" + val=$(json_get "$px_json" ".gateway" ""); [[ -n "$val" ]] && PVE_GATEWAY="$val" + val=$(json_get "$px_json" ".dns" ""); [[ -n "$val" ]] && PVE_DNS="$val" val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val" val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val" val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val" @@ -1419,6 +1424,8 @@ load_config() { val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.gateway" ""); [[ -n "$val" ]] && PVE_GATEWAY="$val" + val=$(json_get "$CONFIG_JSON" ".proxmox.dns" ""); [[ -n "$val" ]] && PVE_DNS="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val" val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val" @@ -1459,6 +1466,7 @@ load_config() { VM_CORES+=("$(vm_get "$i" "cores" "$DEF_CORES")") VM_MEMORY+=("$(vm_get "$i" "memory" "$DEF_MEMORY")") VM_DISK+=("$(vm_get "$i" "disk" "$DEF_DISK")") + VM_IPS+=("$(vm_get "$i" "ip" "")") VM_VMIDS+=("$(vm_get "$i" "vmid" "")") VM_CLUSTERS+=("$(vm_get "$i" "cluster" "")") VM_CLUSTER_ROLES+=("$(vm_get "$i" "cluster_role" "")") @@ -1577,6 +1585,19 @@ validate_config() { esac fi + # Static IP validation + local vm_ip="${VM_IPS[$i]}" + if [[ -n "$vm_ip" ]]; then + if [[ "$vm_ip" != */* ]]; then + error "VM '${name}': ip must include prefix (e.g. 192.168.102.100/22), got '${vm_ip}'" + errors=$((errors + 1)) + fi + if [[ -z "$PVE_GATEWAY" ]]; then + error "VM '${name}': static ip requires gateway in proxmox.yaml or lab config" + errors=$((errors + 1)) + fi + fi + # Track cluster init nodes if [[ -n "$cluster" ]] && [[ "$role" == "init" ]]; then if echo "$cluster_inits" | grep -qw "$cluster"; then @@ -2103,6 +2124,13 @@ pve_vm_exists() { # Get VM's IP address (best-effort via QEMU guest agent) pve_vm_ip() { local vmid="$1" + local known_ip="${2:-}" + + # If static IP is configured, return bare IP (strip CIDR prefix) + if [[ -n "$known_ip" ]]; then + echo "${known_ip%%/*}" + return 0 + fi # Method 1: Try QEMU guest agent (works if agent is installed) local ip="" @@ -2881,6 +2909,9 @@ print_plan() { else echo -e " Bridge: ${PVE_BRIDGE}" fi + if [[ -n "$PVE_GATEWAY" ]]; then + echo -e " Gateway: ${PVE_GATEWAY}" + fi if [[ -n "$PVE_POOL" ]]; then echo -e " Pool: ${CYAN}${PVE_POOL}${RESET}" fi @@ -2907,6 +2938,10 @@ print_plan() { echo -e " VMID: ${VM_VMIDS[$i]} App: ${VM_APPS[$i]} Defaults: ${VM_APPLY_DEFAULTS[$i]}" echo -e " CPU: ${VM_CORES[$i]} cores RAM: ${VM_MEMORY[$i]} MiB Disk: ${VM_DISK[$i]} GiB" + if [[ -n "${VM_IPS[$i]}" ]]; then + echo -e " IP: ${GREEN}${VM_IPS[$i]}${RESET} (static)" + fi + if [[ -n "${VM_CLUSTERS[$i]}" ]]; then echo -e " Cluster: ${VM_CLUSTERS[$i]} (${VM_CLUSTER_ROLES[$i]})" fi @@ -3031,7 +3066,12 @@ phase_prepare() { seed_cmd+=(--app "$app") seed_cmd+=(--output "$seed_output") seed_cmd+=(--force-install) - seed_cmd+=(--force-reboot) + # NOTE: --force-reboot intentionally omitted. On Proxmox, we control + # the VM externally (stop/start). force_reboot triggers SysRq-B which + # causes an "install media detected" intermediate boot — this triggers + # the IncusOS state.txt race condition (issue #843) ~50% of the time. + # Without force_reboot, the installer sits at "please remove media", + # blockstat detects idle, and we cleanly stop/remove media/start from disk. seed_cmd+=(--hostname "$name") seed_cmd+=(--quiet) @@ -3045,6 +3085,16 @@ phase_prepare() { seed_cmd+=(--no-cert) fi + # Static IP: pass --ip, --gateway, --dns to seed generator + local vm_ip="${VM_IPS[$i]}" + if [[ -n "$vm_ip" ]]; then + seed_cmd+=(--ip "$vm_ip") + seed_cmd+=(--gateway "$PVE_GATEWAY") + if [[ -n "$PVE_DNS" ]]; then + seed_cmd+=(--dns "$PVE_DNS") + fi + fi + if [[ "$DRY_RUN" == true ]]; then info "[dry-run] ${seed_cmd[*]}" else @@ -3350,7 +3400,7 @@ phase_install() { # Already installed and running — detect IP, register remote, skip success "VM '${name}' (VMID ${vmid}) is already installed and running" local ip - ip=$(pve_vm_ip "$vmid") || ip="" + ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip="" if [[ -n "$ip" ]]; then detail "IP: ${ip}" register_remote "$name" "$ip" @@ -3365,7 +3415,7 @@ phase_install() { if pve_start_vm "$vmid"; then sleep 30 local ip - ip=$(pve_vm_ip "$vmid") || ip="" + ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip="" if [[ -n "$ip" ]]; then success "VM '${name}' started at ${ip}" register_remote "$name" "$ip" @@ -3399,13 +3449,11 @@ phase_install() { # 2. Wait for installation to complete by monitoring disk writes # - # Install flow: boot ISO -> read SEED_DATA -> install to disk -> reboot - # (force_reboot in seed). A guest-level reboot does NOT reset the QEMU - # VM uptime, so we cannot detect reboot that way. Instead we monitor - # scsi0 write bytes via blockstat: - # - During install: wr_bytes increases steadily - # - After install + reboot: wr_bytes stops (VM is idle at post-reboot - # screen or "install media detected" error) + # Install flow: boot ISO -> read SEED_DATA -> install to disk -> idle + # (no force_reboot: VM sits at "please remove installation media"). + # We monitor scsi0 write bytes via blockstat: + # - During install: wr_bytes increases steadily (~876 MiB for image clone) + # - After install: wr_bytes stops (VM is idle at "remove media" prompt) # # When wr_bytes has been stable for STABLE_COUNT consecutive polls, # installation is complete. We then stop the VM, remove install media, @@ -3476,17 +3524,46 @@ phase_install() { # 3. Stop the VM (it's either at "remove install media" prompt or # post-reboot "install media detected" error -- either way idle). + # Poll until confirmed stopped — do not proceed on assumption. local cur_status cur_status=$(pve_vm_status "$vmid") || cur_status="unknown" if [[ "$cur_status" == "running" ]]; then info "Stopping VM to remove install media..." pve_stop_vm "$vmid" || true - sleep 5 + local stop_wait=0 + while [[ $stop_wait -lt 30 ]]; do + cur_status=$(pve_vm_status "$vmid") || cur_status="unknown" + [[ "$cur_status" == "stopped" ]] && break + sleep 2 + stop_wait=$((stop_wait + 2)) + done + if [[ "$cur_status" != "stopped" ]]; then + warn "VM '${name}' did not stop within 30s (status: ${cur_status})" + install_failures=$((install_failures + 1)) + i=$((i + 1)) + continue + fi + detail "VM stopped after ${stop_wait}s" fi - # 4. Remove install media and set clean boot order + # 4. Remove install media and set clean boot order. + # Verify removal succeeded by re-reading VM config. step "Removing install media from '${name}'" pve_set_vm "$vmid" "--delete ide2 --delete ide3 --boot order=scsi0" || true + sleep 2 + # Verify ide2 and ide3 are gone + if pve_vm_has_install_media "$vmid"; then + warn "Install media still attached after removal — retrying" + pve_set_vm "$vmid" "--delete ide2" || true + pve_set_vm "$vmid" "--delete ide3" || true + sleep 2 + if pve_vm_has_install_media "$vmid"; then + error "Failed to remove install media from '${name}' after retry" + install_failures=$((install_failures + 1)) + i=$((i + 1)) + continue + fi + fi success "Install media detached, boot order set to disk-only" # 5. Start VM from disk @@ -3497,21 +3574,43 @@ phase_install() { continue fi - # 5. Wait for services to come up and verify health + # 6. Wait for services and verify health (port 8443 + scrub_schedule) + # + # The health check has two levels: + # 1. Port 8443 reachable (Incus API is up) + # 2. scrub_schedule is set (IncusOS daemon fully initialized) + # + # Port 8443 reachable but scrub_schedule empty = crontab bug hit. + # The IncusOS REST API starts BEFORE registerJobs(), so port 8443 + # can respond even when the daemon is about to crash. We use + # scrub_schedule as the definitive health indicator and trigger + # retries (not just fixes) when it's empty. info "Waiting for services to start..." - sleep 30 + # First boot downloads SecureBoot update + application sysext (30-120s). + # Total first-boot time is typically 60-150s. Wait 60s before polling. + sleep 60 - # 6. Detect IP and check port 8443 local ip - ip=$(pve_vm_ip "$vmid") || ip="" + ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip="" local healthy=false if [[ -n "$ip" ]]; then - # Poll port 8443 for up to 60s (the 30s sleep above + 60s = 90s total) + # Poll port 8443 for up to 120s (the 60s sleep above + 120s = 180s total) local poll=0 - while [[ $poll -lt 12 ]]; do + while [[ $poll -lt 24 ]]; do if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then - healthy=true + # Port is up — check and fix scrub_schedule. + # fix_scrub_schedule returns 1 ONLY when it detects empty + # scrub_schedule AND the PUT to fix it fails. In that case, + # the crontab bug hit and we should retry (not declare success). + # Returns 0 for: no cert (can't check), already set, or fixed. + local fix_result=0 + fix_scrub_schedule "$ip" "$name" || fix_result=$? + if [[ $fix_result -eq 0 ]]; then + healthy=true + else + warn "VM '${name}': crontab bug detected (scrub_schedule empty, fix failed)" + fi break fi sleep 5 @@ -3521,17 +3620,13 @@ phase_install() { echo "" if [[ "$healthy" == true ]]; then - # Proactively fix empty scrub_schedule (IncusOS crontab bug). - # Safe to call on every node -- returns early if already set. - fix_scrub_schedule "$ip" "$name" || true success "VM '${name}' installed and running at ${ip}" register_remote "$name" "$ip" else - # Retry loop: IncusOS has an intermittent "invalid crontab expression" - # bug (~50% first-boot hit rate) where registerJobs() fails during - # startup(). A stop+start cycle resolves it because the hard power-off - # may discard the corrupt state.txt before it's committed to disk, - # allowing initialize() to recreate it cleanly on next boot. + # Retry loop: if port 8443 didn't come up within 180s, something + # went wrong during first boot. A stop+start cycle can help with + # transient failures. CAUTION: hard-stopping a VM during first boot + # can corrupt the TPM encryption key — only retry if genuinely stuck. local retry=0 local retry_ok=false while [[ $retry -lt $INSTALL_RETRIES ]]; do @@ -3543,19 +3638,32 @@ phase_install() { fi info "Stopping VM for retry..." pve_stop_vm "$vmid" || true - sleep 10 + # Poll until confirmed stopped (not just sleep) + local retry_stop_wait=0 + while [[ $retry_stop_wait -lt 30 ]]; do + local retry_status + retry_status=$(pve_vm_status "$vmid") || retry_status="unknown" + [[ "$retry_status" == "stopped" ]] && break + sleep 2 + retry_stop_wait=$((retry_stop_wait + 2)) + done info "Starting VM (retry ${retry}/${INSTALL_RETRIES})..." pve_start_vm "$vmid" || true - sleep 30 + sleep 60 # Re-detect IP (may have changed after restart) - ip=$(pve_vm_ip "$vmid") || ip="" + ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip="" if [[ -n "$ip" ]]; then local retry_poll=0 - while [[ $retry_poll -lt 12 ]]; do + while [[ $retry_poll -lt 24 ]]; do if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then - retry_ok=true + # Port is up — check and fix scrub_schedule + local retry_fix=0 + fix_scrub_schedule "$ip" "$name" || retry_fix=$? + if [[ $retry_fix -eq 0 ]]; then + retry_ok=true + fi break fi sleep 5 @@ -3564,8 +3672,6 @@ phase_install() { fi if [[ "$retry_ok" == true ]]; then - # Proactively fix empty scrub_schedule (IncusOS crontab bug). - fix_scrub_schedule "$ip" "$name" || true success "VM '${name}' recovered after retry ${retry}/${INSTALL_RETRIES} — running at ${ip}" register_remote "$name" "$ip" break @@ -4119,7 +4225,7 @@ phase_status() { # Network and service checks (only for running, installed VMs) if [[ "$status" == "running" ]] && [[ "$has_media" == false ]]; then local ip - ip=$(pve_vm_ip "$vmid") || ip="" + ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip="" if [[ -n "$ip" ]]; then local port_status="${RED}closed${RESET}" @@ -4206,7 +4312,7 @@ phase_post_deploy() { fi local ip - ip=$(pve_vm_ip "$vmid") || ip="" + ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip="" if [[ -z "$ip" ]]; then warn "VM '${name}': could not detect IP address" all_ok=false diff --git a/incusos/incusos-seed b/incusos/incusos-seed index 72150be..d049083 100755 --- a/incusos/incusos-seed +++ b/incusos/incusos-seed @@ -70,6 +70,9 @@ ${BOLD}OPTIONS${RESET} ${BOLD}--hostname${RESET} NAME Set system hostname ${BOLD}--domain${RESET} NAME Set system domain ${BOLD}--dns${RESET} SERVERS Comma-separated DNS servers (e.g. "8.8.8.8,1.1.1.1") + ${BOLD}--ip${RESET} ADDR/PREFIX Static IP in CIDR notation (e.g. "192.168.102.100/22") + ${BOLD}--gateway${RESET} ADDR Default gateway (required with --ip) + ${BOLD}--iface${RESET} NAME Network interface name (default: ens18 for Proxmox VMs) ${BOLD}Update options:${RESET} ${BOLD}--channel${RESET} CHANNEL Update channel: ${CYAN}stable${RESET}, ${CYAN}testing${RESET} (default: stable) @@ -114,7 +117,7 @@ ${BOLD}SEED FILES REFERENCE${RESET} ${CYAN}applications.yaml${RESET} Application selection (incus, operations-center) ${CYAN}incus.yaml${RESET} Incus preseed (defaults, certificates) ${CYAN}operations-center.yaml${RESET} Operations Center config (when --app=operations-center) - ${CYAN}network.yaml${RESET} Network configuration (hostname, DNS) + ${CYAN}network.yaml${RESET} Network configuration (hostname, DNS, static IP) ${CYAN}update.yaml${RESET} Update channel and reboot policy EOF @@ -139,6 +142,9 @@ MISSING_SECUREBOOT=false HOSTNAME_OPT="" DOMAIN_OPT="" DNS_SERVERS="" +STATIC_IP="" +GATEWAY="" +IFACE="ens18" CHANNEL="stable" AUTO_REBOOT=false FORMAT="tar" @@ -201,6 +207,18 @@ parse_args() { DNS_SERVERS="${2:?'--dns requires a value'}" shift 2 ;; + --ip) + STATIC_IP="${2:?'--ip requires ADDR/PREFIX (e.g. 192.168.102.100/22)'}" + shift 2 + ;; + --gateway) + GATEWAY="${2:?'--gateway requires an address'}" + shift 2 + ;; + --iface) + IFACE="${2:?'--iface requires a name'}" + shift 2 + ;; --channel) CHANNEL="${2:?'--channel requires a value'}" shift 2 @@ -322,6 +340,16 @@ validate_args() { exit 1 fi + if [[ -n "$STATIC_IP" ]] && [[ -z "$GATEWAY" ]]; then + error "--ip requires --gateway" + exit 1 + fi + + if [[ -n "$STATIC_IP" ]] && [[ "$STATIC_IP" != */* ]]; then + error "--ip must include prefix length (e.g. 192.168.102.100/22)" + exit 1 + fi + if [[ -n "$CERT_FILE" ]] && [[ ! -f "$CERT_FILE" ]]; then error "Certificate file not found: ${CERT_FILE}" exit 1 @@ -578,7 +606,7 @@ generate_ops_center_yaml() { generate_network_yaml() { # Only generate if network options were specified - if [[ -z "$HOSTNAME_OPT" ]] && [[ -z "$DOMAIN_OPT" ]] && [[ -z "$DNS_SERVERS" ]]; then + if [[ -z "$HOSTNAME_OPT" ]] && [[ -z "$DOMAIN_OPT" ]] && [[ -z "$DNS_SERVERS" ]] && [[ -z "$STATIC_IP" ]]; then return fi @@ -602,6 +630,21 @@ generate_network_yaml() { done fi fi + + # Static IP configuration + if [[ -n "$STATIC_IP" ]]; then + echo "" + echo "interfaces:" + echo " - name: \"mgmt\"" + echo " hwaddr: \"${IFACE}\"" + echo " addresses:" + echo " - \"${STATIC_IP}\"" + echo " roles:" + echo " - management" + echo " routes:" + echo " - to: \"0.0.0.0/0\"" + echo " via: \"${GATEWAY}\"" + fi } > "$file" if [[ "$DRY_RUN" == true ]]; then @@ -610,6 +653,7 @@ generate_network_yaml() { else local desc="" [[ -n "$HOSTNAME_OPT" ]] && desc+="hostname=${HOSTNAME_OPT}" + [[ -n "$STATIC_IP" ]] && desc+=", ip=${STATIC_IP}" [[ -n "$DNS_SERVERS" ]] && desc+=", dns=${DNS_SERVERS}" success "network.yaml (${desc})" fi diff --git a/incusos/observe-deploy b/incusos/observe-deploy index bfe1ce1..1cad9b7 100755 --- a/incusos/observe-deploy +++ b/incusos/observe-deploy @@ -13,7 +13,9 @@ # ./observe-deploy # Standard deploy with screenshots # ./observe-deploy --no-update-yaml # Omit update.yaml from seed # ./observe-deploy --label run-01 # Custom run label +# ./observe-deploy --runs 10 --label baseline # 10 sequential cycles with stats # ./observe-deploy --vmid 855 # Use a specific VMID +# ./observe-deploy --interval 1 # 1-second screenshots # ./observe-deploy --dry-run # Preview without deploying # # Output: timestamped directory with PNG frames + deploy.log @@ -37,7 +39,12 @@ SEED_TOOL="${SCRIPT_DIR}/incusos-seed" VMID=850 VM_NAME="observe-deploy" LABEL="" +RUNS=1 # number of sequential deploy+destroy cycles NO_UPDATE_YAML=false +NO_VLAN=false +STATIC_IP="" # Static IP in CIDR notation (e.g. 192.168.102.100/22) +VM_EXTRA="" # Extra VM creation overrides (key=value,key=value) +NO_FORCE_REBOOT=false # Skip force_reboot in seed (no SysRq-B reboot after install) DRY_RUN=false SCREENSHOT_INTERVAL=2 # seconds between screenshots INSTALL_TIMEOUT=600 # seconds to wait for install completion @@ -52,6 +59,9 @@ PVE_STORAGE="" PVE_ISO_STORAGE="" PVE_BRIDGE="" PVE_POOL="" +PVE_VLAN="" +PVE_GATEWAY="" +PVE_DNS="" PVE_API_TOKEN_ID="" # IncusOS ISO @@ -67,6 +77,8 @@ PHASE="init" # init → iso_boot → installing → install_done → transitio PREV_WR_BYTES=0 WRITES_STARTED=false STABLE_COUNT=0 +SCRUB_SCHEDULE="" # Filled after port 8443 check: "0 4 * * 0" = healthy, "" = crontab bug +DEPLOY_IP="" # IP of the deployed VM (set during disk_boot phase) # --------------------------------------------------------------------------- # Colors @@ -119,7 +131,12 @@ while [[ $# -gt 0 ]]; do case "$1" in --vmid) VMID="${2:?'--vmid requires a number'}"; shift 2 ;; --label) LABEL="${2:?'--label requires a string'}"; shift 2 ;; + --runs) RUNS="${2:?'--runs requires a number'}"; shift 2 ;; --no-update-yaml) NO_UPDATE_YAML=true; shift ;; + --no-force-reboot) NO_FORCE_REBOOT=true; shift ;; + --no-vlan) NO_VLAN=true; shift ;; + --ip) STATIC_IP="${2:?'--ip requires ADDR/PREFIX'}"; shift 2 ;; + --vm-extra) VM_EXTRA="${2:?'--vm-extra requires key=value pairs'}"; shift 2 ;; --interval) SCREENSHOT_INTERVAL="${2:?'--interval requires seconds'}"; shift 2 ;; --dry-run) DRY_RUN=true; shift ;; -h|--help) @@ -132,7 +149,12 @@ ${BOLD}USAGE${RESET} ${BOLD}OPTIONS${RESET} --vmid N VMID to use (default: 850, must be 850-869) --label NAME Run label for output directory (default: auto) + --runs N Run N sequential deploy+destroy cycles with stats (default: 1) --no-update-yaml Omit update.yaml from seed (Group A equivalent) + --no-force-reboot Omit force_reboot from seed (no SysRq-B after install) + --no-vlan Skip VLAN tag (use native LAN for IP detection) + --ip ADDR/PREFIX Static IP for the VM (e.g. 192.168.102.100/22) + Skips ARP detection; uses known IP for health checks --interval N Screenshot interval in seconds (default: 2) --dry-run Preview without deploying -h, --help Show this help @@ -215,6 +237,12 @@ load_proxmox_yaml() { PVE_ISO_STORAGE=$(yaml_get "$yaml_file" "iso_storage" "local") PVE_BRIDGE=$(yaml_get "$yaml_file" "bridge" "vmbr0") PVE_POOL=$(yaml_get "$yaml_file" "pool") + PVE_VLAN=$(yaml_get "$yaml_file" "vlan") + PVE_GATEWAY=$(yaml_get "$yaml_file" "gateway") + PVE_DNS=$(yaml_get "$yaml_file" "dns") + if [[ "$NO_VLAN" == true ]]; then + PVE_VLAN="" + fi PVE_API_TOKEN_ID=$(yaml_get "$yaml_file" "api_token_id") if [[ -z "$PVE_HOST" ]]; then @@ -319,6 +347,13 @@ print(m.group(1).lower() if m else '') pve_vm_ip() { local vmid="$1" + + # If static IP is configured, return the bare IP (strip CIDR prefix) + if [[ -n "$STATIC_IP" ]]; then + echo "${STATIC_IP%%/*}" + return 0 + fi + local mac mac=$(pve_vm_mac "$vmid") || return 1 [[ -z "$mac" ]] && return 1 @@ -373,6 +408,56 @@ for h in net.hosts(): return 1 } +# --------------------------------------------------------------------------- +# scrub_schedule health check (definitive crontab bug indicator) +# --------------------------------------------------------------------------- + +check_scrub_schedule() { + # Query the IncusOS REST API for the scrub_schedule field. + # This is the definitive health indicator: + # "0 4 * * 0" → genuine success + # "" → crontab bug hit (even if port 8443 responds) + # unreachable → daemon crashed before API started + # + # Sets global SCRUB_SCHEDULE and returns 0 on success, 1 on bug/failure. + local ip="$1" + + local cert="${HOME}/.config/incus/client.crt" + local key="${HOME}/.config/incus/client.key" + + if [[ ! -f "$cert" ]] || [[ ! -f "$key" ]]; then + log "No client certificate — skipping scrub_schedule check" + SCRUB_SCHEDULE="unchecked" + return 0 + fi + + local response + response=$(curl -sk --connect-timeout 5 --max-time 10 \ + --cert "$cert" --key "$key" \ + "https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || { + log "Could not query IncusOS storage API at ${ip}" + SCRUB_SCHEDULE="unreachable" + return 1 + } + + SCRUB_SCHEDULE=$(printf '%s' "$response" | python3 -c " +import sys, json +try: + d = json.load(sys.stdin) + print(d.get('metadata', {}).get('config', {}).get('scrub_schedule', '')) +except: + print('') +" 2>/dev/null) || SCRUB_SCHEDULE="" + + if [[ -n "$SCRUB_SCHEDULE" ]]; then + log "scrub_schedule = '${SCRUB_SCHEDULE}' → HEALTHY" + return 0 + else + log "scrub_schedule is EMPTY → crontab bug detected" + return 1 + fi +} + # --------------------------------------------------------------------------- # Screenshot capture # --------------------------------------------------------------------------- @@ -478,9 +563,18 @@ generate_seed() { info "Omitting update.yaml from seed (Group A equivalent)" # Generate full seed as tar, extract, remove update.yaml, create ISO + local seed_extra=() + if [[ -n "$STATIC_IP" ]]; then + seed_extra+=(--ip "$STATIC_IP" --gateway "$PVE_GATEWAY") + [[ -n "$PVE_DNS" ]] && seed_extra+=(--dns "$PVE_DNS") + fi + + local reboot_flag=() + [[ "$NO_FORCE_REBOOT" != true ]] && reboot_flag=(--force-reboot) + "$SEED_TOOL" --format tar --app incus --output "${SEED_TMPDIR}/seed.tar" \ - --force-install --force-reboot --hostname "$VM_NAME" \ - --defaults --quiet &>/dev/null + --force-install "${reboot_flag[@]+"${reboot_flag[@]}"}" --hostname "$VM_NAME" \ + --defaults "${seed_extra[@]+"${seed_extra[@]}"}" --quiet &>/dev/null local seeddir="${SEED_TMPDIR}/seedfiles" mkdir -p "$seeddir" @@ -505,11 +599,20 @@ generate_seed() { else info "Standard seed (with update.yaml, check_frequency=6h)" - "$SEED_TOOL" --format iso --app incus --output "$local_path" \ - --force-install --force-reboot --hostname "$VM_NAME" \ - --defaults --quiet &>/dev/null + local seed_extra=() + if [[ -n "$STATIC_IP" ]]; then + seed_extra+=(--ip "$STATIC_IP" --gateway "$PVE_GATEWAY") + [[ -n "$PVE_DNS" ]] && seed_extra+=(--dns "$PVE_DNS") + fi - log "Seed: standard incusos-seed output" + local reboot_flag=() + [[ "$NO_FORCE_REBOOT" != true ]] && reboot_flag=(--force-reboot) + + "$SEED_TOOL" --format iso --app incus --output "$local_path" \ + --force-install "${reboot_flag[@]+"${reboot_flag[@]}"}" --hostname "$VM_NAME" \ + --defaults "${seed_extra[@]+"${seed_extra[@]}"}" --quiet &>/dev/null + + log "Seed: standard incusos-seed output (force_reboot=$([[ "$NO_FORCE_REBOOT" == true ]] && echo false || echo true))" fi success "Seed ISO: ${seed_name}" @@ -554,13 +657,25 @@ payload = { 'scsi0': '${PVE_STORAGE}:50,iothread=1', 'ide2': '${PVE_ISO_STORAGE}:iso/${ISO_FILENAME},media=cdrom', 'ide3': '${PVE_ISO_STORAGE}:iso/${SEED_ISO},media=cdrom', - 'net0': 'virtio,bridge=${PVE_BRIDGE}', + 'net0': 'virtio,bridge=${PVE_BRIDGE}' + (',tag=${PVE_VLAN}' if '${PVE_VLAN}' else ''), 'boot': 'order=ide2;scsi0', 'agent': 1, } pool = '${PVE_POOL}' if pool: payload['pool'] = pool +# Apply --vm-extra overrides (e.g. 'scsi0=local-zfs:50,cache=writethrough,iothread=1') +vm_extra = '${VM_EXTRA}' +if vm_extra: + for pair in vm_extra.split(',,,'): # triple-comma separator for multi-value pairs + if '=' in pair: + k, v = pair.split('=', 1) + # Convert numeric strings to ints for JSON + try: + v = int(v) + except (ValueError, TypeError): + pass + payload[k.strip()] = v print(json.dumps(payload)) ") @@ -793,6 +908,7 @@ run_deploy() { # Check port 8443 if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then log "Port 8443 responding at ${ip} (${boot_elapsed}s into disk boot)" + DEPLOY_IP="$ip" PHASE="ready" break else @@ -825,11 +941,20 @@ run_deploy() { done fi + # Definitive health check: query scrub_schedule via IncusOS REST API + if [[ "$PHASE" == "ready" ]] && [[ -n "$DEPLOY_IP" ]]; then + step "Checking scrub_schedule (definitive health indicator)" + if ! check_scrub_schedule "$DEPLOY_IP"; then + PHASE="crontab_bug" + log "Deploy reached port 8443 but scrub_schedule is empty — crontab bug" + fi + fi + local deploy_end deploy_end=$(date +%s) local total_seconds=$((deploy_end - deploy_start)) - log "Deploy finished: phase=${PHASE}, total_time=${total_seconds}s, frames=${FRAME_NUM}" + log "Deploy finished: phase=${PHASE}, scrub_schedule=${SCRUB_SCHEDULE:-unset}, total_time=${total_seconds}s, frames=${FRAME_NUM}" if [[ "$PHASE" == "ready" ]]; then return 0 @@ -931,7 +1056,7 @@ if [[ "$DRY_RUN" == true ]]; then step "Would load proxmox.yaml" load_proxmox_yaml detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}" - detail "Pool: ${PVE_POOL}, Token: ${PVE_API_TOKEN_ID}" + detail "Pool: ${PVE_POOL}, VLAN: ${PVE_VLAN:-none}, Token: ${PVE_API_TOKEN_ID}" echo "" step "Would check:" @@ -949,8 +1074,12 @@ if [[ "$DRY_RUN" == true ]]; then detail "5. Monitor blockstat for install completion" detail "6. Stop VM, remove media, start from disk" detail "7. Continue screenshots until port 8443 responds (or ${BOOT_TIMEOUT}s timeout)" - detail "8. Save frames + deploy.log" - detail "9. Cleanup VM" + detail "8. Check scrub_schedule via IncusOS REST API" + detail "9. Save frames + deploy.log" + detail "10. Cleanup VM" + if [[ $RUNS -gt 1 ]]; then + detail "Repeat ${RUNS} times with aggregate statistics" + fi echo "" info "Dry run complete" @@ -960,7 +1089,14 @@ fi # Load configuration step "Loading configuration" load_proxmox_yaml -detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}" +detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}, VLAN: ${PVE_VLAN:-none}" +if [[ -n "$STATIC_IP" ]]; then + detail "Static IP: ${STATIC_IP}, Gateway: ${PVE_GATEWAY:-MISSING}" + if [[ -z "$PVE_GATEWAY" ]]; then + error "--ip requires 'gateway' in proxmox.yaml" + exit 1 + fi +fi # Run preflight checks preflight @@ -968,32 +1104,6 @@ preflight # Find IncusOS ISO find_incusos_iso -# Setup output directory -setup_output - -# Log configuration -log "=== observe-deploy v${VERSION} ===" -log "VMID: ${VMID}" -log "VM name: ${VM_NAME}" -log "ISO: ${ISO_FILENAME}" -log "Update.yaml: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "OMITTED"; else echo "included (6h)"; fi)" -log "Screenshot interval: ${SCREENSHOT_INTERVAL}s" -log "Install timeout: ${INSTALL_TIMEOUT}s" -log "Boot timeout: ${BOOT_TIMEOUT}s" - -# Generate and upload seed -generate_seed - -# Create VM -create_vm - -echo "" -echo -e "${BOLD}Starting observational deploy${RESET}" -echo " Frames saved to: ${RUN_DIR}/" -echo " Log: ${RUN_DIR}/deploy.log" -echo " Press Ctrl+C to abort (VM will be cleaned up)" -echo "" - # Trap for clean exit trap_cleanup() { echo "" @@ -1003,39 +1113,213 @@ trap_cleanup() { } trap trap_cleanup INT TERM -# Run the deploy with screenshots -deploy_result=0 -run_deploy || deploy_result=$? +# --------------------------------------------------------------------------- +# Run loop (single run or multi-run with --runs N) +# --------------------------------------------------------------------------- -echo "" -echo -e "${BOLD}========================================${RESET}" -echo -e "${BOLD} Deploy Results${RESET}" -echo -e "${BOLD}========================================${RESET}" -echo "" -echo " VMID: ${VMID}" -echo " Seed: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "no update.yaml"; else echo "standard (check_frequency=6h)"; fi)" -echo " Outcome: $(if [[ $deploy_result -eq 0 ]]; then echo -e "${GREEN}PASS${RESET} — port 8443 reachable"; else echo -e "${RED}FAIL${RESET} — phase: ${PHASE}"; fi)" -echo " Frames: ${FRAME_NUM}" -echo " Output: ${RUN_DIR}/" -echo "" +run_single_cycle() { + # Runs one complete deploy+destroy cycle. Returns 0 on success, 1 on failure. + # Sets globals: PHASE, SCRUB_SCHEDULE, DEPLOY_IP, FRAME_NUM -# Cleanup -step "Post-deploy cleanup" -cleanup_vm + # Reset state for this run + PHASE="init" + PREV_WR_BYTES=0 + WRITES_STARTED=false + STABLE_COUNT=0 + SCRUB_SCHEDULE="" + DEPLOY_IP="" + FRAME_NUM=0 -if [[ $deploy_result -eq 0 ]]; then - success "Observational deploy complete — ${FRAME_NUM} frames captured" + # Setup per-run output directory + setup_output + + # Log configuration + log "=== observe-deploy v${VERSION} ===" + log "VMID: ${VMID}" + log "VM name: ${VM_NAME}" + log "ISO: ${ISO_FILENAME}" + log "Update.yaml: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "OMITTED"; else echo "included (6h)"; fi)" + log "Screenshot interval: ${SCREENSHOT_INTERVAL}s" + log "Install timeout: ${INSTALL_TIMEOUT}s" + log "Boot timeout: ${BOOT_TIMEOUT}s" + + # Generate and upload seed + generate_seed + + # Create VM + create_vm + + echo "" + echo -e "${BOLD}Starting observational deploy${RESET}" + echo " Frames saved to: ${RUN_DIR}/" + echo " Log: ${RUN_DIR}/deploy.log" + echo " Press Ctrl+C to abort (VM will be cleaned up)" + echo "" + + # Run the deploy with screenshots + local result=0 + run_deploy || result=$? + + echo "" + echo -e "${BOLD}========================================${RESET}" + echo -e "${BOLD} Deploy Results${RESET}" + echo -e "${BOLD}========================================${RESET}" + echo "" + echo " VMID: ${VMID}" + echo " Seed: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "no update.yaml"; else echo "standard (check_frequency=6h)"; fi)" + if [[ $result -eq 0 ]]; then + echo -e " Outcome: ${GREEN}PASS${RESET} — port 8443 reachable, scrub_schedule='${SCRUB_SCHEDULE}'" + elif [[ "$PHASE" == "crontab_bug" ]]; then + echo -e " Outcome: ${RED}FAIL${RESET} — crontab bug (port 8443 up, scrub_schedule empty)" + else + echo -e " Outcome: ${RED}FAIL${RESET} — phase: ${PHASE}" + fi + echo " Frames: ${FRAME_NUM}" + echo " Output: ${RUN_DIR}/" + echo "" + + # Cleanup + step "Post-deploy cleanup" + cleanup_vm + + return $result +} + +if [[ $RUNS -le 1 ]]; then + # Single-run mode (original behavior) + deploy_result=0 + run_single_cycle || deploy_result=$? + + if [[ $deploy_result -eq 0 ]]; then + success "Observational deploy complete — ${FRAME_NUM} frames captured" + else + warn "Deploy failed at phase: ${PHASE}" + warn "Check screenshots and log for diagnosis:" + warn " ${RUN_DIR}/deploy.log" + warn " ${RUN_DIR}/frame-*.png" + fi + + echo "" + echo "To view key frames:" + echo " ls ${RUN_DIR}/frame-*.png" + echo " # View in image viewer or analyze with Claude Code" + echo "" + + exit $deploy_result else - warn "Deploy failed at phase: ${PHASE}" - warn "Check screenshots and log for diagnosis:" - warn " ${RUN_DIR}/deploy.log" - warn " ${RUN_DIR}/frame-*.png" + # Multi-run mode: N sequential deploy+destroy cycles with aggregate stats + echo "" + echo -e "${BOLD}Multi-run mode: ${RUNS} sequential deploy cycles${RESET}" + echo "" + + # Compute base label before the loop (LABEL may be empty at this point) + if [[ -z "$LABEL" ]]; then + if [[ "$NO_UPDATE_YAML" == true ]]; then + BASE_LABEL="no-update-yaml" + else + BASE_LABEL="standard" + fi + else + BASE_LABEL="$LABEL" + fi + + total_pass=0 + total_fail=0 + total_crontab_bug=0 + total_timeout=0 + total_time=0 + run_results=() # Array of "PASS|FAIL|BUG" per run + + overall_start=$(date +%s) + + run_num=0 + while [[ $run_num -lt $RUNS ]]; do + run_num=$((run_num + 1)) + echo "" + echo -e "${BOLD}╔══════════════════════════════════════╗${RESET}" + echo -e "${BOLD}║ Run ${run_num}/${RUNS}${RESET}" + echo -e "${BOLD}╚══════════════════════════════════════╝${RESET}" + echo "" + + # Set per-run label + LABEL="${BASE_LABEL}_run-$(printf '%02d' "$run_num")" + + local_start=$(date +%s) + + deploy_result=0 + run_single_cycle || deploy_result=$? + + local_end=$(date +%s) + local_duration=$((local_end - local_start)) + total_time=$((total_time + local_duration)) + + if [[ $deploy_result -eq 0 ]]; then + total_pass=$((total_pass + 1)) + run_results+=("PASS") + success "Run ${run_num}/${RUNS}: PASS (${local_duration}s)" + elif [[ "$PHASE" == "crontab_bug" ]]; then + total_fail=$((total_fail + 1)) + total_crontab_bug=$((total_crontab_bug + 1)) + run_results+=("BUG") + warn "Run ${run_num}/${RUNS}: FAIL — crontab bug (${local_duration}s)" + else + total_fail=$((total_fail + 1)) + total_timeout=$((total_timeout + 1)) + run_results+=("FAIL") + warn "Run ${run_num}/${RUNS}: FAIL — ${PHASE} (${local_duration}s)" + fi + + # Brief pause between runs to let Proxmox settle + if [[ $run_num -lt $RUNS ]]; then + info "Pausing 10s before next run..." + sleep 10 + fi + done + + overall_end=$(date +%s) + overall_duration=$((overall_end - overall_start)) + avg_time=$((total_time / RUNS)) + + echo "" + echo -e "${BOLD}╔══════════════════════════════════════════════════╗${RESET}" + echo -e "${BOLD}║ Multi-Run Summary (${RUNS} cycles)${RESET}" + echo -e "${BOLD}╚══════════════════════════════════════════════════╝${RESET}" + echo "" + echo -e " ${GREEN}PASS:${RESET} ${total_pass}/${RUNS}" + echo -e " ${RED}FAIL:${RESET} ${total_fail}/${RUNS}" + if [[ $total_crontab_bug -gt 0 ]]; then + echo -e " ${RED}crontab bug:${RESET} ${total_crontab_bug}" + fi + if [[ $total_timeout -gt 0 ]]; then + echo -e " ${RED}timeout/other:${RESET} ${total_timeout}" + fi + echo "" + echo " Avg time: ${avg_time}s per run" + echo " Total time: ${overall_duration}s" + echo "" + + # Per-run results + echo " Per-run: ${run_results[*]}" + echo "" + + # Success rate + if [[ $RUNS -gt 0 ]]; then + pct=$((total_pass * 100 / RUNS)) + echo -e " Success rate: ${pct}%" + if [[ $pct -ge 95 ]]; then + echo -e " ${GREEN}Target met (≥95%)${RESET}" + else + echo -e " ${RED}Below target (<95%)${RESET}" + fi + fi + echo "" + + # Output directories + echo " Output: ${SCRIPT_DIR}/observe-runs/*${BASE_LABEL}*" + echo "" + + if [[ $total_fail -gt 0 ]]; then + exit 1 + fi + exit 0 fi - -echo "" -echo "To view key frames:" -echo " ls ${RUN_DIR}/frame-*.png" -echo " # View in image viewer or analyze with Claude Code" -echo "" - -exit $deploy_result