maarten
/
incus-contrib
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix IncusOS deploy: eliminate crontab bug and TPM corruption 

Root cause #1: force_reboot triggers SysRq-B intermediate boot that
races with IncusOS state initialization, causing ~50% crontab bug rate.
Fix: omit force_reboot from seed — VM sits at "please remove media"
after install, blockstat detects idle cleanly, no intermediate boot.
Validated: 15/15 PASS (100%) on 4-core VMs.

Root cause #2: 90s boot timeout was too short for first-boot sysext
download under load. Premature retry hard-stops the VM during first
boot, corrupting the TPM encryption key permanently. Fix: increase
boot timeout to 180s (60s sleep + 120s polling).

Changes:
- incusos-proxmox: remove --force-reboot from seed generation, increase
  boot timeout from 90s to 180s, update retry comments
- incusos-seed: add --ip/--gateway/--dns flags for static IP seeds,
  add network.yaml generation
- observe-deploy: add --no-force-reboot flag, --runs N mode, VLAN
  support, scrub_schedule health check, 1s screenshot interval
- lab configs: add static IPs for VLAN 69 (192.168.102.x/22)
- proxmox.yaml.example: add vlan, gateway, dns fields
- CLAUDE.md: document findings, boot timeout, TPM corruption risk

Tested: single VM, 3-node cluster, OC + 3 nodes — all 100% success.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Maarten 2026-02-22 21:50:02 +01:00
parent 9e3201393e
commit 088baf6807
9 changed files with 627 additions and 131 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

97
CLAUDE.md
View File

@ -130,22 +130,33 @@ incu-contrib/
`scsi-*` does NOT match `scsi-0QEMU_QEMU_HARDDISK_drive-scsi0`. Omit
`disk-target` entirely and let IncusOS auto-detect (works for single-disk VMs).
- **Install flow** (automated by `incusos-proxmox`):
1. Boot VM with ISO (ide2) + SEED_DATA (ide3) + `force_reboot: true` in seed
2. IncusOS reads seed, installs to disk (scsi0), auto-reboots
1. Boot VM with ISO (ide2) + SEED_DATA (ide3), **no** `force_reboot` in seed
2. IncusOS reads seed, installs to disk (scsi0, ~876 MiB image clone),
then sits at "please remove installation media" prompt
3. Detect install completion by polling `blockstat.scsi0.wr_bytes` via API --
when disk writes start then stop for 15s (3 stable polls), install is done
4. Stop the VM (Proxmox stop, not guest shutdown)
5. **Delete ide2 and ide3** -- IncusOS checks for install media at every boot
and refuses to start if found, regardless of boot order
6. Set boot order to `order=scsi0` and start from disk
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Use
ARP-based lookup: get MAC from Proxmox VM config → flush stale ARP →
ping broadcast → look up MAC in ARP table. Verify with direct ping before
trusting the result.
- **force_reboot is required**: without `force_reboot: true` in the seed,
IncusOS sits at "please remove installation media" and waits indefinitely.
It does NOT halt automatically. `force_reboot` triggers a guest-level reboot
(note: this does NOT reset QEMU VM uptime -- only a Proxmox stop/start does).
- **IP detection**: IncusOS is immutable and has no QEMU guest agent. Two
strategies: (1) **Static IP** (preferred): set `ip: ADDR/PREFIX` per VM
in the lab config + `gateway:` and `dns:` in proxmox.yaml. The seed's
`network.yaml` configures the VM's interface at boot. No ARP scan or
SSH needed — the IP is known at deploy time. Works across VLANs.
(2) **ARP-based lookup** (fallback for DHCP): get MAC from Proxmox VM
config → flush stale ARP → ping broadcast → look up MAC in ARP table.
Only works on the same L2 domain (native LAN, not across VLANs).
- **force_reboot is NOT used on Proxmox** (changed 2026-02-22): the seed
omits `force_reboot`. Without it, IncusOS sits at "please remove
installation media" after install — which is exactly what we want. We
detect completion via blockstat (876 MiB written, then idle), stop the
VM externally, remove media, and start from disk. This eliminates the
SysRq-B intermediate boot that triggers the crontab race condition
(issue #843). On physical hardware or unmanaged VMs, `force_reboot: true`
is still needed — this optimization is specific to Proxmox automated deploys.
`force_reboot` triggers SysRq-B (raw kernel reboot via `/proc/sysrq-trigger`).
This does NOT reset QEMU VM uptime — only a Proxmox stop/start does.
- **Resource pool isolation**: the optional `proxmox.pool` config field scopes
all VM operations to a Proxmox resource pool. When set, the script only
"sees" VMs in that pool (for collision detection and cleanup), and the API
@ -511,10 +522,18 @@ incu-contrib/
except warnings and errors.
- **`--retries N`**: number of stop+start retries for VMs that fail to boot
(port 8443 not reachable). Default: 3. `--retries 0` disables retries.
Each retry: Proxmox stop → 10s wait → start → 30s wait → poll port 8443.
With ~67% boot failure rate from the crontab bug (confirmed by parallel
investigation), 3 retries gives ~96.3% success rate (1 - 0.67^3). After retries, `fix_scrub_schedule()` proactively heals the
root cause via the IncusOS REST API.
Each retry: Proxmox stop → wait → start → 60s wait → poll port 8443 for
up to 120s. With the `force_reboot` fix (2026-02-22), the crontab bug is
eliminated and retries are rarely needed. They remain as a safety net for
other transient boot failures. After retries, `fix_scrub_schedule()`
proactively heals any remaining scrub_schedule issues via the IncusOS
REST API.
- **Boot timeout is 180s** (60s initial sleep + 120s polling): first boot
downloads SecureBoot update + application sysext (Incus: ~30-120s depending
on CDN speed and concurrent VMs). The previous 90s timeout was too short
under load — premature retries (hard VM stop during first boot) can corrupt
the TPM encryption key, causing permanent "zfs load-key: Raw key too short
(expected 32)" errors that no amount of retries can fix.
### IncusOS first-boot sequence (observed via console screenshots)
@ -570,9 +589,19 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
### IncusOS boot failure (crontab / update frequency error)
- **RESOLVED** (2026-02-22): root cause identified and fixed in our pipeline.
The `force_reboot` seed option triggers SysRq-B after install, causing an
"install media detected" intermediate boot. This intermediate boot generates
~26 GB of additional disk writes (the IncusOS daemon starts from ISO tmpfs).
Our blockstat detection catches these late writes, but by then the
intermediate boot has run and somehow corrupted the state for the subsequent
first-from-disk boot. **Fix: omit `force_reboot` from the seed.** Without
it, the installer sits at "please remove media" after the 876 MiB image
clone, blockstat detects idle cleanly, and the first disk boot is pristine.
Results: **15/15 PASS (100%)** without force_reboot on 4 cores vs 50% with it.
- **Symptom**: "ERROR invalid crontab expression" appears on the console
during first boot. Intermittent — confirmed **67% failure rate** across
20 parallel VM deploys (8/12 API-verified). Error does NOT correlate with
during first boot. Intermittent — confirmed **~50% failure rate** with
`force_reboot: true` on 4-core VMs. Error does NOT correlate with
`update.yaml` presence/absence, ISO version, or version mismatch.
- **Two distinct error paths in IncusOS**:
- `registerJobs()` validates `ScrubSchedule` (5-field crontab expression
@ -613,11 +642,25 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
- **Race window**: The REST API server starts BEFORE `startup()`
completes. Applications start before `registerJobs()`. A concurrent
API request modifying storage config could clear ScrubSchedule.
- **Most likely root cause**: a race condition between the REST API/
application startup and `registerJobs()` that clears `ScrubSchedule`.
Investigation ruled out version mismatch (bug hits equally with matching
ISO), update downloads (occurs even without OS updates), and gocron
library issues (the field is genuinely empty in the state struct).
- **Root cause confirmed**: the bug is triggered by the SysRq-B
intermediate boot (from `force_reboot: true`). Controlled experiments
on VLAN 69 with static IP (2026-02-22, ISO 202602210344):
- Baseline (4 cores, force_reboot): 5/10 PASS (50%)
- cache=writethrough: 3/5 PASS (60%) — falsified
- agent=0: 3/5 PASS (60%) — falsified
- cores=1 (force_reboot): 14/15 PASS (93%) — GOMAXPROCS=1 reduces race
- **No force_reboot (4 cores): 15/15 PASS (100%)** — confirmed fix
Correlation: runs with 876 MiB blockstat detection (before SysRq-B)
had 100% success; runs with 27 GB detection (after SysRq-B) had ~55%.
- **The try-it service is NOT IncusOS**: stgraber's comparison ("thousands
of times a day") is invalid. The try-it service runs Ubuntu VMs with
Incus installed from the Zabbly daily repo — a completely different
product. Issue #843 was independently filed by another Proxmox user
(fperreau on Proxmox 9.1.4), confirming it's not unique to our setup.
- The underlying IncusOS bug is still a race condition between the REST
API/application startup and `registerJobs()` that clears `ScrubSchedule`.
Investigation ruled out version mismatch, update downloads, and gocron
library issues. The field is genuinely empty in the state struct.
- **Investigation data** (2026-02-21, ISO 202602210344, 4 VMs parallel):
Batch 3: 1 PASS, 3 BUG (75%); Batch 4: 1 PASS, 3 BUG (75%);
Batch 5: 2 PASS, 2 BUG (50%). Total API-verified: 4 PASS, 8 BUG
@ -671,8 +714,16 @@ from the installed disk by `state.LoadOrCreate()` → `initialize()`.
has a journal commit interval (~5s) and the corrupt Save() may not have
been committed before the VM was killed.
- **Detection**: `phase_install` checks port 8443 after starting from disk
(30s wait + up to 60s polling). If first check fails, automatic retry
adds ~100s. Failed VMs after retry are reported with remediation.
(60s wait + up to 120s polling = 180s total). If first check fails,
automatic retry adds ~180s. Failed VMs after retry are reported with
remediation.
- **TPM corruption from premature retry**: hard-stopping a VM during first
boot (while the encryption key is being written to the TPM zvol) can
leave the TPM with truncated key data. All subsequent boots fail with
"zfs load-key: Raw key too short (expected 32)" — a permanent error
that requires VM destruction and redeployment to fix. The 180s boot
timeout prevents this by allowing the full first-boot sequence (including
sysext download) to complete before any retry.
- **Auto-heal via IncusOS REST API**: `incusos-proxmox` includes
`fix_scrub_schedule()` which proactively fixes empty `scrub_schedule` on
every deployed node via `PUT /os/1.0/system/storage`. The `/os/` prefix

3
incusos/examples/lab-cluster.yaml
View File

@ -29,11 +29,14 @@ vms:
- name: incus-lab-01
app: incus
apply_defaults: true # init node: needs storage pool + network
ip: 192.168.102.101/22
- name: incus-lab-02
app: incus
apply_defaults: false # joining node: cluster join creates pool entry
ip: 192.168.102.102/22
- name: incus-lab-03
app: incus
apply_defaults: false # joining node: cluster join creates pool entry
ip: 192.168.102.103/22

4
incusos/examples/lab-oc.yaml
View File

@ -29,6 +29,7 @@ vms:
- name: lab-oc
app: operations-center
apply_defaults: true
ip: 192.168.102.110/22
- name: lab-node-01
app: incus
@ -36,11 +37,14 @@ vms:
cores: 4
memory: 8192
disk: 64
ip: 192.168.102.111/22
- name: lab-node-02
app: incus
apply_defaults: true
ip: 192.168.102.112/22
- name: lab-node-03
app: incus
apply_defaults: true
ip: 192.168.102.113/22

1
incusos/examples/lab-single.yaml
View File

@ -23,3 +23,4 @@ vms:
- name: incus-single
app: incus
apply_defaults: true
ip: 192.168.102.100/22

2
incusos/examples/network-static.yaml
View File

@ -21,7 +21,7 @@ time:
interfaces:
- name: "mgmt"
hwaddr: "enp5s0" # interface name or MAC address
hwaddr: "ens18" # interface name or MAC (ens18 = Proxmox VirtIO)
addresses:
- "192.168.1.100/24"
roles:

3
incusos/examples/proxmox.yaml.example
View File

@ -17,5 +17,8 @@ node: pve # Proxmox node name
storage: local-lvm # VM disk storage
iso_storage: local # ISO/image storage
bridge: vmbr0 # Network bridge
# vlan: 69 # VLAN tag for VM NIC (optional)
# gateway: 192.168.100.1 # Default gateway for static IPs (required if VMs use ip:)
# dns: 192.168.100.1 # DNS server for static IPs (optional, used with ip:)
# pool: IncusLab # Resource pool (optional, for isolation)
ssh_user: root # SSH user (for ssh method)

178
incusos/incusos-proxmox
View File

@ -1296,6 +1296,8 @@ PVE_STORAGE=""
PVE_ISO_STORAGE=""
PVE_BRIDGE=""
PVE_VLAN=""
PVE_GATEWAY=""
PVE_DNS=""
PVE_METHOD=""
PVE_SSH_USER=""
PVE_API_TOKEN_ID=""
@ -1316,6 +1318,7 @@ declare -a VM_CORES=()
declare -a VM_MEMORY=()
declare -a VM_DISK=()
declare -a VM_VMIDS=()
declare -a VM_IPS=()
declare -a VM_CLUSTERS=()
declare -a VM_CLUSTER_ROLES=()
@ -1396,6 +1399,8 @@ json.dump(result, sys.stdout, indent=2)
val=$(json_get "$px_json" ".iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
val=$(json_get "$px_json" ".bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
val=$(json_get "$px_json" ".vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
val=$(json_get "$px_json" ".gateway" ""); [[ -n "$val" ]] && PVE_GATEWAY="$val"
val=$(json_get "$px_json" ".dns" ""); [[ -n "$val" ]] && PVE_DNS="$val"
val=$(json_get "$px_json" ".ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
val=$(json_get "$px_json" ".api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
val=$(json_get "$px_json" ".pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
@ -1419,6 +1424,8 @@ load_config() {
val=$(json_get "$CONFIG_JSON" ".proxmox.iso_storage" ""); [[ -n "$val" ]] && PVE_ISO_STORAGE="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.bridge" ""); [[ -n "$val" ]] && PVE_BRIDGE="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.vlan" ""); [[ -n "$val" ]] && PVE_VLAN="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.gateway" ""); [[ -n "$val" ]] && PVE_GATEWAY="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.dns" ""); [[ -n "$val" ]] && PVE_DNS="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.ssh_user" ""); [[ -n "$val" ]] && PVE_SSH_USER="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.api_token_id" ""); [[ -n "$val" ]] && PVE_API_TOKEN_ID="$val"
val=$(json_get "$CONFIG_JSON" ".proxmox.pool" ""); [[ -n "$val" ]] && PVE_POOL="$val"
@ -1459,6 +1466,7 @@ load_config() {
VM_CORES+=("$(vm_get "$i" "cores" "$DEF_CORES")")
VM_MEMORY+=("$(vm_get "$i" "memory" "$DEF_MEMORY")")
VM_DISK+=("$(vm_get "$i" "disk" "$DEF_DISK")")
VM_IPS+=("$(vm_get "$i" "ip" "")")
VM_VMIDS+=("$(vm_get "$i" "vmid" "")")
VM_CLUSTERS+=("$(vm_get "$i" "cluster" "")")
VM_CLUSTER_ROLES+=("$(vm_get "$i" "cluster_role" "")")
@ -1577,6 +1585,19 @@ validate_config() {
esac
fi
# Static IP validation
local vm_ip="${VM_IPS[$i]}"
if [[ -n "$vm_ip" ]]; then
if [[ "$vm_ip" != */* ]]; then
error "VM '${name}': ip must include prefix (e.g. 192.168.102.100/22), got '${vm_ip}'"
errors=$((errors + 1))
fi
if [[ -z "$PVE_GATEWAY" ]]; then
error "VM '${name}': static ip requires gateway in proxmox.yaml or lab config"
errors=$((errors + 1))
fi
fi
# Track cluster init nodes
if [[ -n "$cluster" ]] && [[ "$role" == "init" ]]; then
if echo "$cluster_inits" | grep -qw "$cluster"; then
@ -2103,6 +2124,13 @@ pve_vm_exists() {
# Get VM's IP address (best-effort via QEMU guest agent)
pve_vm_ip() {
local vmid="$1"
local known_ip="${2:-}"
# If static IP is configured, return bare IP (strip CIDR prefix)
if [[ -n "$known_ip" ]]; then
echo "${known_ip%%/*}"
return 0
fi
# Method 1: Try QEMU guest agent (works if agent is installed)
local ip=""
@ -2881,6 +2909,9 @@ print_plan() {
else
echo -e " Bridge: ${PVE_BRIDGE}"
fi
if [[ -n "$PVE_GATEWAY" ]]; then
echo -e " Gateway: ${PVE_GATEWAY}"
fi
if [[ -n "$PVE_POOL" ]]; then
echo -e " Pool: ${CYAN}${PVE_POOL}${RESET}"
fi
@ -2907,6 +2938,10 @@ print_plan() {
echo -e " VMID: ${VM_VMIDS[$i]} App: ${VM_APPS[$i]} Defaults: ${VM_APPLY_DEFAULTS[$i]}"
echo -e " CPU: ${VM_CORES[$i]} cores RAM: ${VM_MEMORY[$i]} MiB Disk: ${VM_DISK[$i]} GiB"
if [[ -n "${VM_IPS[$i]}" ]]; then
echo -e " IP: ${GREEN}${VM_IPS[$i]}${RESET} (static)"
fi
if [[ -n "${VM_CLUSTERS[$i]}" ]]; then
echo -e " Cluster: ${VM_CLUSTERS[$i]} (${VM_CLUSTER_ROLES[$i]})"
fi
@ -3031,7 +3066,12 @@ phase_prepare() {
seed_cmd+=(--app "$app")
seed_cmd+=(--output "$seed_output")
seed_cmd+=(--force-install)
seed_cmd+=(--force-reboot)
# NOTE: --force-reboot intentionally omitted. On Proxmox, we control
# the VM externally (stop/start). force_reboot triggers SysRq-B which
# causes an "install media detected" intermediate boot — this triggers
# the IncusOS state.txt race condition (issue #843) ~50% of the time.
# Without force_reboot, the installer sits at "please remove media",
# blockstat detects idle, and we cleanly stop/remove media/start from disk.
seed_cmd+=(--hostname "$name")
seed_cmd+=(--quiet)
@ -3045,6 +3085,16 @@ phase_prepare() {
seed_cmd+=(--no-cert)
fi
# Static IP: pass --ip, --gateway, --dns to seed generator
local vm_ip="${VM_IPS[$i]}"
if [[ -n "$vm_ip" ]]; then
seed_cmd+=(--ip "$vm_ip")
seed_cmd+=(--gateway "$PVE_GATEWAY")
if [[ -n "$PVE_DNS" ]]; then
seed_cmd+=(--dns "$PVE_DNS")
fi
fi
if [[ "$DRY_RUN" == true ]]; then
info "[dry-run] ${seed_cmd[*]}"
else
@ -3350,7 +3400,7 @@ phase_install() {
# Already installed and running — detect IP, register remote, skip
success "VM '${name}' (VMID ${vmid}) is already installed and running"
local ip
ip=$(pve_vm_ip "$vmid") || ip=""
ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
if [[ -n "$ip" ]]; then
detail "IP: ${ip}"
register_remote "$name" "$ip"
@ -3365,7 +3415,7 @@ phase_install() {
if pve_start_vm "$vmid"; then
sleep 30
local ip
ip=$(pve_vm_ip "$vmid") || ip=""
ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
if [[ -n "$ip" ]]; then
success "VM '${name}' started at ${ip}"
register_remote "$name" "$ip"
@ -3399,13 +3449,11 @@ phase_install() {
# 2. Wait for installation to complete by monitoring disk writes
#
# Install flow: boot ISO -> read SEED_DATA -> install to disk -> reboot
# (force_reboot in seed). A guest-level reboot does NOT reset the QEMU
# VM uptime, so we cannot detect reboot that way. Instead we monitor
# scsi0 write bytes via blockstat:
# - During install: wr_bytes increases steadily
# - After install + reboot: wr_bytes stops (VM is idle at post-reboot
# screen or "install media detected" error)
# Install flow: boot ISO -> read SEED_DATA -> install to disk -> idle
# (no force_reboot: VM sits at "please remove installation media").
# We monitor scsi0 write bytes via blockstat:
# - During install: wr_bytes increases steadily (~876 MiB for image clone)
# - After install: wr_bytes stops (VM is idle at "remove media" prompt)
#
# When wr_bytes has been stable for STABLE_COUNT consecutive polls,
# installation is complete. We then stop the VM, remove install media,
@ -3476,17 +3524,46 @@ phase_install() {
# 3. Stop the VM (it's either at "remove install media" prompt or
# post-reboot "install media detected" error -- either way idle).
# Poll until confirmed stopped — do not proceed on assumption.
local cur_status
cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
if [[ "$cur_status" == "running" ]]; then
info "Stopping VM to remove install media..."
pve_stop_vm "$vmid" || true
sleep 5
local stop_wait=0
while [[ $stop_wait -lt 30 ]]; do
cur_status=$(pve_vm_status "$vmid") || cur_status="unknown"
[[ "$cur_status" == "stopped" ]] && break
sleep 2
stop_wait=$((stop_wait + 2))
done
if [[ "$cur_status" != "stopped" ]]; then
warn "VM '${name}' did not stop within 30s (status: ${cur_status})"
install_failures=$((install_failures + 1))
i=$((i + 1))
continue
fi
detail "VM stopped after ${stop_wait}s"
fi
# 4. Remove install media and set clean boot order
# 4. Remove install media and set clean boot order.
# Verify removal succeeded by re-reading VM config.
step "Removing install media from '${name}'"
pve_set_vm "$vmid" "--delete ide2 --delete ide3 --boot order=scsi0" || true
sleep 2
# Verify ide2 and ide3 are gone
if pve_vm_has_install_media "$vmid"; then
warn "Install media still attached after removal — retrying"
pve_set_vm "$vmid" "--delete ide2" || true
pve_set_vm "$vmid" "--delete ide3" || true
sleep 2
if pve_vm_has_install_media "$vmid"; then
error "Failed to remove install media from '${name}' after retry"
install_failures=$((install_failures + 1))
i=$((i + 1))
continue
fi
fi
success "Install media detached, boot order set to disk-only"
# 5. Start VM from disk
@ -3497,21 +3574,43 @@ phase_install() {
continue
fi
# 5. Wait for services to come up and verify health
# 6. Wait for services and verify health (port 8443 + scrub_schedule)
#
# The health check has two levels:
# 1. Port 8443 reachable (Incus API is up)
# 2. scrub_schedule is set (IncusOS daemon fully initialized)
#
# Port 8443 reachable but scrub_schedule empty = crontab bug hit.
# The IncusOS REST API starts BEFORE registerJobs(), so port 8443
# can respond even when the daemon is about to crash. We use
# scrub_schedule as the definitive health indicator and trigger
# retries (not just fixes) when it's empty.
info "Waiting for services to start..."
sleep 30
# First boot downloads SecureBoot update + application sysext (30-120s).
# Total first-boot time is typically 60-150s. Wait 60s before polling.
sleep 60
# 6. Detect IP and check port 8443
local ip
ip=$(pve_vm_ip "$vmid") || ip=""
ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
local healthy=false
if [[ -n "$ip" ]]; then
# Poll port 8443 for up to 60s (the 30s sleep above + 60s = 90s total)
# Poll port 8443 for up to 120s (the 60s sleep above + 120s = 180s total)
local poll=0
while [[ $poll -lt 12 ]]; do
while [[ $poll -lt 24 ]]; do
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
healthy=true
# Port is up — check and fix scrub_schedule.
# fix_scrub_schedule returns 1 ONLY when it detects empty
# scrub_schedule AND the PUT to fix it fails. In that case,
# the crontab bug hit and we should retry (not declare success).
# Returns 0 for: no cert (can't check), already set, or fixed.
local fix_result=0
fix_scrub_schedule "$ip" "$name" || fix_result=$?
if [[ $fix_result -eq 0 ]]; then
healthy=true
else
warn "VM '${name}': crontab bug detected (scrub_schedule empty, fix failed)"
fi
break
fi
sleep 5
@ -3521,17 +3620,13 @@ phase_install() {
echo ""
if [[ "$healthy" == true ]]; then
# Proactively fix empty scrub_schedule (IncusOS crontab bug).
# Safe to call on every node -- returns early if already set.
fix_scrub_schedule "$ip" "$name" || true
success "VM '${name}' installed and running at ${ip}"
register_remote "$name" "$ip"
else
# Retry loop: IncusOS has an intermittent "invalid crontab expression"
# bug (~50% first-boot hit rate) where registerJobs() fails during
# startup(). A stop+start cycle resolves it because the hard power-off
# may discard the corrupt state.txt before it's committed to disk,
# allowing initialize() to recreate it cleanly on next boot.
# Retry loop: if port 8443 didn't come up within 180s, something
# went wrong during first boot. A stop+start cycle can help with
# transient failures. CAUTION: hard-stopping a VM during first boot
# can corrupt the TPM encryption key — only retry if genuinely stuck.
local retry=0
local retry_ok=false
while [[ $retry -lt $INSTALL_RETRIES ]]; do
@ -3543,19 +3638,32 @@ phase_install() {
fi
info "Stopping VM for retry..."
pve_stop_vm "$vmid" || true
sleep 10
# Poll until confirmed stopped (not just sleep)
local retry_stop_wait=0
while [[ $retry_stop_wait -lt 30 ]]; do
local retry_status
retry_status=$(pve_vm_status "$vmid") || retry_status="unknown"
[[ "$retry_status" == "stopped" ]] && break
sleep 2
retry_stop_wait=$((retry_stop_wait + 2))
done
info "Starting VM (retry ${retry}/${INSTALL_RETRIES})..."
pve_start_vm "$vmid" || true
sleep 30
sleep 60
# Re-detect IP (may have changed after restart)
ip=$(pve_vm_ip "$vmid") || ip=""
ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
if [[ -n "$ip" ]]; then
local retry_poll=0
while [[ $retry_poll -lt 12 ]]; do
while [[ $retry_poll -lt 24 ]]; do
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
retry_ok=true
# Port is up — check and fix scrub_schedule
local retry_fix=0
fix_scrub_schedule "$ip" "$name" || retry_fix=$?
if [[ $retry_fix -eq 0 ]]; then
retry_ok=true
fi
break
fi
sleep 5
@ -3564,8 +3672,6 @@ phase_install() {
fi
if [[ "$retry_ok" == true ]]; then
# Proactively fix empty scrub_schedule (IncusOS crontab bug).
fix_scrub_schedule "$ip" "$name" || true
success "VM '${name}' recovered after retry ${retry}/${INSTALL_RETRIES} — running at ${ip}"
register_remote "$name" "$ip"
break
@ -4119,7 +4225,7 @@ phase_status() {
# Network and service checks (only for running, installed VMs)
if [[ "$status" == "running" ]] && [[ "$has_media" == false ]]; then
local ip
ip=$(pve_vm_ip "$vmid") || ip=""
ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
if [[ -n "$ip" ]]; then
local port_status="${RED}closed${RESET}"
@ -4206,7 +4312,7 @@ phase_post_deploy() {
fi
local ip
ip=$(pve_vm_ip "$vmid") || ip=""
ip=$(pve_vm_ip "$vmid" "${VM_IPS[$i]:-}") || ip=""
if [[ -z "$ip" ]]; then
warn "VM '${name}': could not detect IP address"
all_ok=false

48
incusos/incusos-seed
View File

@ -70,6 +70,9 @@ ${BOLD}OPTIONS${RESET}
${BOLD}--hostname${RESET} NAME Set system hostname
${BOLD}--domain${RESET} NAME Set system domain
${BOLD}--dns${RESET} SERVERS Comma-separated DNS servers (e.g. "8.8.8.8,1.1.1.1")
${BOLD}--ip${RESET} ADDR/PREFIX Static IP in CIDR notation (e.g. "192.168.102.100/22")
${BOLD}--gateway${RESET} ADDR Default gateway (required with --ip)
${BOLD}--iface${RESET} NAME Network interface name (default: ens18 for Proxmox VMs)
${BOLD}Update options:${RESET}
${BOLD}--channel${RESET} CHANNEL Update channel: ${CYAN}stable${RESET}, ${CYAN}testing${RESET} (default: stable)
@ -114,7 +117,7 @@ ${BOLD}SEED FILES REFERENCE${RESET}
${CYAN}applications.yaml${RESET} Application selection (incus, operations-center)
${CYAN}incus.yaml${RESET} Incus preseed (defaults, certificates)
${CYAN}operations-center.yaml${RESET} Operations Center config (when --app=operations-center)
${CYAN}network.yaml${RESET} Network configuration (hostname, DNS)
${CYAN}network.yaml${RESET} Network configuration (hostname, DNS, static IP)
${CYAN}update.yaml${RESET} Update channel and reboot policy
EOF
@ -139,6 +142,9 @@ MISSING_SECUREBOOT=false
HOSTNAME_OPT=""
DOMAIN_OPT=""
DNS_SERVERS=""
STATIC_IP=""
GATEWAY=""
IFACE="ens18"
CHANNEL="stable"
AUTO_REBOOT=false
FORMAT="tar"
@ -201,6 +207,18 @@ parse_args() {
DNS_SERVERS="${2:?'--dns requires a value'}"
shift 2
;;
--ip)
STATIC_IP="${2:?'--ip requires ADDR/PREFIX (e.g. 192.168.102.100/22)'}"
shift 2
;;
--gateway)
GATEWAY="${2:?'--gateway requires an address'}"
shift 2
;;
--iface)
IFACE="${2:?'--iface requires a name'}"
shift 2
;;
--channel)
CHANNEL="${2:?'--channel requires a value'}"
shift 2
@ -322,6 +340,16 @@ validate_args() {
exit 1
fi
if [[ -n "$STATIC_IP" ]] && [[ -z "$GATEWAY" ]]; then
error "--ip requires --gateway"
exit 1
fi
if [[ -n "$STATIC_IP" ]] && [[ "$STATIC_IP" != */* ]]; then
error "--ip must include prefix length (e.g. 192.168.102.100/22)"
exit 1
fi
if [[ -n "$CERT_FILE" ]] && [[ ! -f "$CERT_FILE" ]]; then
error "Certificate file not found: ${CERT_FILE}"
exit 1
@ -578,7 +606,7 @@ generate_ops_center_yaml() {
generate_network_yaml() {
# Only generate if network options were specified
if [[ -z "$HOSTNAME_OPT" ]] && [[ -z "$DOMAIN_OPT" ]] && [[ -z "$DNS_SERVERS" ]]; then
if [[ -z "$HOSTNAME_OPT" ]] && [[ -z "$DOMAIN_OPT" ]] && [[ -z "$DNS_SERVERS" ]] && [[ -z "$STATIC_IP" ]]; then
return
fi
@ -602,6 +630,21 @@ generate_network_yaml() {
done
fi
fi
# Static IP configuration
if [[ -n "$STATIC_IP" ]]; then
echo ""
echo "interfaces:"
echo " - name: \"mgmt\""
echo " hwaddr: \"${IFACE}\""
echo " addresses:"
echo " - \"${STATIC_IP}\""
echo " roles:"
echo " - management"
echo " routes:"
echo " - to: \"0.0.0.0/0\""
echo " via: \"${GATEWAY}\""
fi
} > "$file"
if [[ "$DRY_RUN" == true ]]; then
@ -610,6 +653,7 @@ generate_network_yaml() {
else
local desc=""
[[ -n "$HOSTNAME_OPT" ]] && desc+="hostname=${HOSTNAME_OPT}"
[[ -n "$STATIC_IP" ]] && desc+=", ip=${STATIC_IP}"
[[ -n "$DNS_SERVERS" ]] && desc+=", dns=${DNS_SERVERS}"
success "network.yaml (${desc})"
fi

422
incusos/observe-deploy
View File

@ -13,7 +13,9 @@
# ./observe-deploy # Standard deploy with screenshots
# ./observe-deploy --no-update-yaml # Omit update.yaml from seed
# ./observe-deploy --label run-01 # Custom run label
# ./observe-deploy --runs 10 --label baseline # 10 sequential cycles with stats
# ./observe-deploy --vmid 855 # Use a specific VMID
# ./observe-deploy --interval 1 # 1-second screenshots
# ./observe-deploy --dry-run # Preview without deploying
#
# Output: timestamped directory with PNG frames + deploy.log
@ -37,7 +39,12 @@ SEED_TOOL="${SCRIPT_DIR}/incusos-seed"
VMID=850
VM_NAME="observe-deploy"
LABEL=""
RUNS=1 # number of sequential deploy+destroy cycles
NO_UPDATE_YAML=false
NO_VLAN=false
STATIC_IP="" # Static IP in CIDR notation (e.g. 192.168.102.100/22)
VM_EXTRA="" # Extra VM creation overrides (key=value,key=value)
NO_FORCE_REBOOT=false # Skip force_reboot in seed (no SysRq-B reboot after install)
DRY_RUN=false
SCREENSHOT_INTERVAL=2 # seconds between screenshots
INSTALL_TIMEOUT=600 # seconds to wait for install completion
@ -52,6 +59,9 @@ PVE_STORAGE=""
PVE_ISO_STORAGE=""
PVE_BRIDGE=""
PVE_POOL=""
PVE_VLAN=""
PVE_GATEWAY=""
PVE_DNS=""
PVE_API_TOKEN_ID=""
# IncusOS ISO
@ -67,6 +77,8 @@ PHASE="init" # init → iso_boot → installing → install_done → transitio
PREV_WR_BYTES=0
WRITES_STARTED=false
STABLE_COUNT=0
SCRUB_SCHEDULE="" # Filled after port 8443 check: "0 4 * * 0" = healthy, "" = crontab bug
DEPLOY_IP="" # IP of the deployed VM (set during disk_boot phase)
# ---------------------------------------------------------------------------
# Colors
@ -119,7 +131,12 @@ while [[ $# -gt 0 ]]; do
case "$1" in
--vmid) VMID="${2:?'--vmid requires a number'}"; shift 2 ;;
--label) LABEL="${2:?'--label requires a string'}"; shift 2 ;;
--runs) RUNS="${2:?'--runs requires a number'}"; shift 2 ;;
--no-update-yaml) NO_UPDATE_YAML=true; shift ;;
--no-force-reboot) NO_FORCE_REBOOT=true; shift ;;
--no-vlan) NO_VLAN=true; shift ;;
--ip) STATIC_IP="${2:?'--ip requires ADDR/PREFIX'}"; shift 2 ;;
--vm-extra) VM_EXTRA="${2:?'--vm-extra requires key=value pairs'}"; shift 2 ;;
--interval) SCREENSHOT_INTERVAL="${2:?'--interval requires seconds'}"; shift 2 ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help)
@ -132,7 +149,12 @@ ${BOLD}USAGE${RESET}
${BOLD}OPTIONS${RESET}
--vmid N VMID to use (default: 850, must be 850-869)
--label NAME Run label for output directory (default: auto)
--runs N Run N sequential deploy+destroy cycles with stats (default: 1)
--no-update-yaml Omit update.yaml from seed (Group A equivalent)
--no-force-reboot Omit force_reboot from seed (no SysRq-B after install)
--no-vlan Skip VLAN tag (use native LAN for IP detection)
--ip ADDR/PREFIX Static IP for the VM (e.g. 192.168.102.100/22)
Skips ARP detection; uses known IP for health checks
--interval N Screenshot interval in seconds (default: 2)
--dry-run Preview without deploying
-h, --help Show this help
@ -215,6 +237,12 @@ load_proxmox_yaml() {
PVE_ISO_STORAGE=$(yaml_get "$yaml_file" "iso_storage" "local")
PVE_BRIDGE=$(yaml_get "$yaml_file" "bridge" "vmbr0")
PVE_POOL=$(yaml_get "$yaml_file" "pool")
PVE_VLAN=$(yaml_get "$yaml_file" "vlan")
PVE_GATEWAY=$(yaml_get "$yaml_file" "gateway")
PVE_DNS=$(yaml_get "$yaml_file" "dns")
if [[ "$NO_VLAN" == true ]]; then
PVE_VLAN=""
fi
PVE_API_TOKEN_ID=$(yaml_get "$yaml_file" "api_token_id")
if [[ -z "$PVE_HOST" ]]; then
@ -319,6 +347,13 @@ print(m.group(1).lower() if m else '')
pve_vm_ip() {
local vmid="$1"
# If static IP is configured, return the bare IP (strip CIDR prefix)
if [[ -n "$STATIC_IP" ]]; then
echo "${STATIC_IP%%/*}"
return 0
fi
local mac
mac=$(pve_vm_mac "$vmid") || return 1
[[ -z "$mac" ]] && return 1
@ -373,6 +408,56 @@ for h in net.hosts():
return 1
}
# ---------------------------------------------------------------------------
# scrub_schedule health check (definitive crontab bug indicator)
# ---------------------------------------------------------------------------
check_scrub_schedule() {
# Query the IncusOS REST API for the scrub_schedule field.
# This is the definitive health indicator:
# "0 4 * * 0" → genuine success
# "" → crontab bug hit (even if port 8443 responds)
# unreachable → daemon crashed before API started
#
# Sets global SCRUB_SCHEDULE and returns 0 on success, 1 on bug/failure.
local ip="$1"
local cert="${HOME}/.config/incus/client.crt"
local key="${HOME}/.config/incus/client.key"
if [[ ! -f "$cert" ]] || [[ ! -f "$key" ]]; then
log "No client certificate — skipping scrub_schedule check"
SCRUB_SCHEDULE="unchecked"
return 0
fi
local response
response=$(curl -sk --connect-timeout 5 --max-time 10 \
--cert "$cert" --key "$key" \
"https://${ip}:8443/os/1.0/system/storage" 2>/dev/null) || {
log "Could not query IncusOS storage API at ${ip}"
SCRUB_SCHEDULE="unreachable"
return 1
}
SCRUB_SCHEDULE=$(printf '%s' "$response" | python3 -c "
import sys, json
try:
d = json.load(sys.stdin)
print(d.get('metadata', {}).get('config', {}).get('scrub_schedule', ''))
except:
print('')
" 2>/dev/null) || SCRUB_SCHEDULE=""
if [[ -n "$SCRUB_SCHEDULE" ]]; then
log "scrub_schedule = '${SCRUB_SCHEDULE}' → HEALTHY"
return 0
else
log "scrub_schedule is EMPTY → crontab bug detected"
return 1
fi
}
# ---------------------------------------------------------------------------
# Screenshot capture
# ---------------------------------------------------------------------------
@ -478,9 +563,18 @@ generate_seed() {
info "Omitting update.yaml from seed (Group A equivalent)"
# Generate full seed as tar, extract, remove update.yaml, create ISO
local seed_extra=()
if [[ -n "$STATIC_IP" ]]; then
seed_extra+=(--ip "$STATIC_IP" --gateway "$PVE_GATEWAY")
[[ -n "$PVE_DNS" ]] && seed_extra+=(--dns "$PVE_DNS")
fi
local reboot_flag=()
[[ "$NO_FORCE_REBOOT" != true ]] && reboot_flag=(--force-reboot)
"$SEED_TOOL" --format tar --app incus --output "${SEED_TMPDIR}/seed.tar" \
--force-install --force-reboot --hostname "$VM_NAME" \
--defaults --quiet &>/dev/null
--force-install "${reboot_flag[@]+"${reboot_flag[@]}"}" --hostname "$VM_NAME" \
--defaults "${seed_extra[@]+"${seed_extra[@]}"}" --quiet &>/dev/null
local seeddir="${SEED_TMPDIR}/seedfiles"
mkdir -p "$seeddir"
@ -505,11 +599,20 @@ generate_seed() {
else
info "Standard seed (with update.yaml, check_frequency=6h)"
"$SEED_TOOL" --format iso --app incus --output "$local_path" \
--force-install --force-reboot --hostname "$VM_NAME" \
--defaults --quiet &>/dev/null
local seed_extra=()
if [[ -n "$STATIC_IP" ]]; then
seed_extra+=(--ip "$STATIC_IP" --gateway "$PVE_GATEWAY")
[[ -n "$PVE_DNS" ]] && seed_extra+=(--dns "$PVE_DNS")
fi
log "Seed: standard incusos-seed output"
local reboot_flag=()
[[ "$NO_FORCE_REBOOT" != true ]] && reboot_flag=(--force-reboot)
"$SEED_TOOL" --format iso --app incus --output "$local_path" \
--force-install "${reboot_flag[@]+"${reboot_flag[@]}"}" --hostname "$VM_NAME" \
--defaults "${seed_extra[@]+"${seed_extra[@]}"}" --quiet &>/dev/null
log "Seed: standard incusos-seed output (force_reboot=$([[ "$NO_FORCE_REBOOT" == true ]] && echo false || echo true))"
fi
success "Seed ISO: ${seed_name}"
@ -554,13 +657,25 @@ payload = {
'scsi0': '${PVE_STORAGE}:50,iothread=1',
'ide2': '${PVE_ISO_STORAGE}:iso/${ISO_FILENAME},media=cdrom',
'ide3': '${PVE_ISO_STORAGE}:iso/${SEED_ISO},media=cdrom',
'net0': 'virtio,bridge=${PVE_BRIDGE}',
'net0': 'virtio,bridge=${PVE_BRIDGE}' + (',tag=${PVE_VLAN}' if '${PVE_VLAN}' else ''),
'boot': 'order=ide2;scsi0',
'agent': 1,
}
pool = '${PVE_POOL}'
if pool:
payload['pool'] = pool
# Apply --vm-extra overrides (e.g. 'scsi0=local-zfs:50,cache=writethrough,iothread=1')
vm_extra = '${VM_EXTRA}'
if vm_extra:
for pair in vm_extra.split(',,,'): # triple-comma separator for multi-value pairs
if '=' in pair:
k, v = pair.split('=', 1)
# Convert numeric strings to ints for JSON
try:
v = int(v)
except (ValueError, TypeError):
pass
payload[k.strip()] = v
print(json.dumps(payload))
")
@ -793,6 +908,7 @@ run_deploy() {
# Check port 8443
if curl -sk --connect-timeout 3 "https://${ip}:8443" &>/dev/null; then
log "Port 8443 responding at ${ip} (${boot_elapsed}s into disk boot)"
DEPLOY_IP="$ip"
PHASE="ready"
break
else
@ -825,11 +941,20 @@ run_deploy() {
done
fi
# Definitive health check: query scrub_schedule via IncusOS REST API
if [[ "$PHASE" == "ready" ]] && [[ -n "$DEPLOY_IP" ]]; then
step "Checking scrub_schedule (definitive health indicator)"
if ! check_scrub_schedule "$DEPLOY_IP"; then
PHASE="crontab_bug"
log "Deploy reached port 8443 but scrub_schedule is empty — crontab bug"
fi
fi
local deploy_end
deploy_end=$(date +%s)
local total_seconds=$((deploy_end - deploy_start))
log "Deploy finished: phase=${PHASE}, total_time=${total_seconds}s, frames=${FRAME_NUM}"
log "Deploy finished: phase=${PHASE}, scrub_schedule=${SCRUB_SCHEDULE:-unset}, total_time=${total_seconds}s, frames=${FRAME_NUM}"
if [[ "$PHASE" == "ready" ]]; then
return 0
@ -931,7 +1056,7 @@ if [[ "$DRY_RUN" == true ]]; then
step "Would load proxmox.yaml"
load_proxmox_yaml
detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}"
detail "Pool: ${PVE_POOL}, Token: ${PVE_API_TOKEN_ID}"
detail "Pool: ${PVE_POOL}, VLAN: ${PVE_VLAN:-none}, Token: ${PVE_API_TOKEN_ID}"
echo ""
step "Would check:"
@ -949,8 +1074,12 @@ if [[ "$DRY_RUN" == true ]]; then
detail "5. Monitor blockstat for install completion"
detail "6. Stop VM, remove media, start from disk"
detail "7. Continue screenshots until port 8443 responds (or ${BOOT_TIMEOUT}s timeout)"
detail "8. Save frames + deploy.log"
detail "9. Cleanup VM"
detail "8. Check scrub_schedule via IncusOS REST API"
detail "9. Save frames + deploy.log"
detail "10. Cleanup VM"
if [[ $RUNS -gt 1 ]]; then
detail "Repeat ${RUNS} times with aggregate statistics"
fi
echo ""
info "Dry run complete"
@ -960,7 +1089,14 @@ fi
# Load configuration
step "Loading configuration"
load_proxmox_yaml
detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}"
detail "Host: ${PVE_HOST}, Node: ${PVE_NODE}, Storage: ${PVE_STORAGE}, VLAN: ${PVE_VLAN:-none}"
if [[ -n "$STATIC_IP" ]]; then
detail "Static IP: ${STATIC_IP}, Gateway: ${PVE_GATEWAY:-MISSING}"
if [[ -z "$PVE_GATEWAY" ]]; then
error "--ip requires 'gateway' in proxmox.yaml"
exit 1
fi
fi
# Run preflight checks
preflight
@ -968,32 +1104,6 @@ preflight
# Find IncusOS ISO
find_incusos_iso
# Setup output directory
setup_output
# Log configuration
log "=== observe-deploy v${VERSION} ==="
log "VMID: ${VMID}"
log "VM name: ${VM_NAME}"
log "ISO: ${ISO_FILENAME}"
log "Update.yaml: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "OMITTED"; else echo "included (6h)"; fi)"
log "Screenshot interval: ${SCREENSHOT_INTERVAL}s"
log "Install timeout: ${INSTALL_TIMEOUT}s"
log "Boot timeout: ${BOOT_TIMEOUT}s"
# Generate and upload seed
generate_seed
# Create VM
create_vm
echo ""
echo -e "${BOLD}Starting observational deploy${RESET}"
echo " Frames saved to: ${RUN_DIR}/"
echo " Log: ${RUN_DIR}/deploy.log"
echo " Press Ctrl+C to abort (VM will be cleaned up)"
echo ""
# Trap for clean exit
trap_cleanup() {
echo ""
@ -1003,39 +1113,213 @@ trap_cleanup() {
}
trap trap_cleanup INT TERM
# Run the deploy with screenshots
deploy_result=0
run_deploy || deploy_result=$?
# ---------------------------------------------------------------------------
# Run loop (single run or multi-run with --runs N)
# ---------------------------------------------------------------------------
echo ""
echo -e "${BOLD}========================================${RESET}"
echo -e "${BOLD} Deploy Results${RESET}"
echo -e "${BOLD}========================================${RESET}"
echo ""
echo " VMID: ${VMID}"
echo " Seed: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "no update.yaml"; else echo "standard (check_frequency=6h)"; fi)"
echo " Outcome: $(if [[ $deploy_result -eq 0 ]]; then echo -e "${GREEN}PASS${RESET} — port 8443 reachable"; else echo -e "${RED}FAIL${RESET} — phase: ${PHASE}"; fi)"
echo " Frames: ${FRAME_NUM}"
echo " Output: ${RUN_DIR}/"
echo ""
run_single_cycle() {
# Runs one complete deploy+destroy cycle. Returns 0 on success, 1 on failure.
# Sets globals: PHASE, SCRUB_SCHEDULE, DEPLOY_IP, FRAME_NUM
# Cleanup
step "Post-deploy cleanup"
cleanup_vm
# Reset state for this run
PHASE="init"
PREV_WR_BYTES=0
WRITES_STARTED=false
STABLE_COUNT=0
SCRUB_SCHEDULE=""
DEPLOY_IP=""
FRAME_NUM=0
if [[ $deploy_result -eq 0 ]]; then
success "Observational deploy complete — ${FRAME_NUM} frames captured"
# Setup per-run output directory
setup_output
# Log configuration
log "=== observe-deploy v${VERSION} ==="
log "VMID: ${VMID}"
log "VM name: ${VM_NAME}"
log "ISO: ${ISO_FILENAME}"
log "Update.yaml: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "OMITTED"; else echo "included (6h)"; fi)"
log "Screenshot interval: ${SCREENSHOT_INTERVAL}s"
log "Install timeout: ${INSTALL_TIMEOUT}s"
log "Boot timeout: ${BOOT_TIMEOUT}s"
# Generate and upload seed
generate_seed
# Create VM
create_vm
echo ""
echo -e "${BOLD}Starting observational deploy${RESET}"
echo " Frames saved to: ${RUN_DIR}/"
echo " Log: ${RUN_DIR}/deploy.log"
echo " Press Ctrl+C to abort (VM will be cleaned up)"
echo ""
# Run the deploy with screenshots
local result=0
run_deploy || result=$?
echo ""
echo -e "${BOLD}========================================${RESET}"
echo -e "${BOLD} Deploy Results${RESET}"
echo -e "${BOLD}========================================${RESET}"
echo ""
echo " VMID: ${VMID}"
echo " Seed: $(if [[ "$NO_UPDATE_YAML" == true ]]; then echo "no update.yaml"; else echo "standard (check_frequency=6h)"; fi)"
if [[ $result -eq 0 ]]; then
echo -e " Outcome: ${GREEN}PASS${RESET} — port 8443 reachable, scrub_schedule='${SCRUB_SCHEDULE}'"
elif [[ "$PHASE" == "crontab_bug" ]]; then
echo -e " Outcome: ${RED}FAIL${RESET} — crontab bug (port 8443 up, scrub_schedule empty)"
else
echo -e " Outcome: ${RED}FAIL${RESET} — phase: ${PHASE}"
fi
echo " Frames: ${FRAME_NUM}"
echo " Output: ${RUN_DIR}/"
echo ""
# Cleanup
step "Post-deploy cleanup"
cleanup_vm
return $result
}
if [[ $RUNS -le 1 ]]; then
# Single-run mode (original behavior)
deploy_result=0
run_single_cycle || deploy_result=$?
if [[ $deploy_result -eq 0 ]]; then
success "Observational deploy complete — ${FRAME_NUM} frames captured"
else
warn "Deploy failed at phase: ${PHASE}"
warn "Check screenshots and log for diagnosis:"
warn " ${RUN_DIR}/deploy.log"
warn " ${RUN_DIR}/frame-*.png"
fi
echo ""
echo "To view key frames:"
echo " ls ${RUN_DIR}/frame-*.png"
echo " # View in image viewer or analyze with Claude Code"
echo ""
exit $deploy_result
else
warn "Deploy failed at phase: ${PHASE}"
warn "Check screenshots and log for diagnosis:"
warn " ${RUN_DIR}/deploy.log"
warn " ${RUN_DIR}/frame-*.png"
# Multi-run mode: N sequential deploy+destroy cycles with aggregate stats
echo ""
echo -e "${BOLD}Multi-run mode: ${RUNS} sequential deploy cycles${RESET}"
echo ""
# Compute base label before the loop (LABEL may be empty at this point)
if [[ -z "$LABEL" ]]; then
if [[ "$NO_UPDATE_YAML" == true ]]; then
BASE_LABEL="no-update-yaml"
else
BASE_LABEL="standard"
fi
else
BASE_LABEL="$LABEL"
fi
total_pass=0
total_fail=0
total_crontab_bug=0
total_timeout=0
total_time=0
run_results=() # Array of "PASS|FAIL|BUG" per run
overall_start=$(date +%s)
run_num=0
while [[ $run_num -lt $RUNS ]]; do
run_num=$((run_num + 1))
echo ""
echo -e "${BOLD}╔══════════════════════════════════════╗${RESET}"
echo -e "${BOLD}║ Run ${run_num}/${RUNS}${RESET}"
echo -e "${BOLD}╚══════════════════════════════════════╝${RESET}"
echo ""
# Set per-run label
LABEL="${BASE_LABEL}_run-$(printf '%02d' "$run_num")"
local_start=$(date +%s)
deploy_result=0
run_single_cycle || deploy_result=$?
local_end=$(date +%s)
local_duration=$((local_end - local_start))
total_time=$((total_time + local_duration))
if [[ $deploy_result -eq 0 ]]; then
total_pass=$((total_pass + 1))
run_results+=("PASS")
success "Run ${run_num}/${RUNS}: PASS (${local_duration}s)"
elif [[ "$PHASE" == "crontab_bug" ]]; then
total_fail=$((total_fail + 1))
total_crontab_bug=$((total_crontab_bug + 1))
run_results+=("BUG")
warn "Run ${run_num}/${RUNS}: FAIL — crontab bug (${local_duration}s)"
else
total_fail=$((total_fail + 1))
total_timeout=$((total_timeout + 1))
run_results+=("FAIL")
warn "Run ${run_num}/${RUNS}: FAIL — ${PHASE} (${local_duration}s)"
fi
# Brief pause between runs to let Proxmox settle
if [[ $run_num -lt $RUNS ]]; then
info "Pausing 10s before next run..."
sleep 10
fi
done
overall_end=$(date +%s)
overall_duration=$((overall_end - overall_start))
avg_time=$((total_time / RUNS))
echo ""
echo -e "${BOLD}╔══════════════════════════════════════════════════╗${RESET}"
echo -e "${BOLD}║ Multi-Run Summary (${RUNS} cycles)${RESET}"
echo -e "${BOLD}╚══════════════════════════════════════════════════╝${RESET}"
echo ""
echo -e " ${GREEN}PASS:${RESET} ${total_pass}/${RUNS}"
echo -e " ${RED}FAIL:${RESET} ${total_fail}/${RUNS}"
if [[ $total_crontab_bug -gt 0 ]]; then
echo -e " ${RED}crontab bug:${RESET} ${total_crontab_bug}"
fi
if [[ $total_timeout -gt 0 ]]; then
echo -e " ${RED}timeout/other:${RESET} ${total_timeout}"
fi
echo ""
echo " Avg time: ${avg_time}s per run"
echo " Total time: ${overall_duration}s"
echo ""
# Per-run results
echo " Per-run: ${run_results[*]}"
echo ""
# Success rate
if [[ $RUNS -gt 0 ]]; then
pct=$((total_pass * 100 / RUNS))
echo -e " Success rate: ${pct}%"
if [[ $pct -ge 95 ]]; then
echo -e " ${GREEN}Target met (≥95%)${RESET}"
else
echo -e " ${RED}Below target (<95%)${RESET}"
fi
fi
echo ""
# Output directories
echo " Output: ${SCRIPT_DIR}/observe-runs/*${BASE_LABEL}*"
echo ""
if [[ $total_fail -gt 0 ]]; then
exit 1
fi
exit 0
fi
echo ""
echo "To view key frames:"
echo " ls ${RUN_DIR}/frame-*.png"
echo " # View in image viewer or analyze with Claude Code"
echo ""
exit $deploy_result